Save yourself some trouble and remap Mail's "Add Sender to Contacts" shortcut
There is a shortcut in Mail, in case you didn’t know, to “Add Sender to Contacts”. I just had the joy of pressing this shortcut ⇧⌘Y, while instead I was aiming for ⇧⌘U (“Mark As Read”). This was such a pleasant experience, because I happened to have all mails from Trash selected, which made Mail add 1230 contacts. Sadly not undoable. Why are these shortcuts so close to each other? Also why do the contacts not show up in Last Import in Contacts? *hrmpf*
Anyway. To remove all contacts create a new Smart Group:
Remove all contacts from there manually. Finally you may want to go to System Preferences and change that shortcut to something you will not press by accident:
Less learned, I guess.