Do not cache already searched fprs for p15

This might lead to unexpected problems during testing
when users delete keys plug out / plug in smartcards etc.
An ldap query for a fingerprint is reasonable to do because
setCard is rarely called and should only be called when a new
card is detected.