This was initially reported as a KMail bug: https://bugs.kde.org/show_bug.cgi?id=327986

IMO the issue here is: If you get a message for hidden recipients you are asked to unlock every secret key you have. Even if the session key could already be decrypted with one of them.
I can reproduce it by:

• Encrypt a message to hidden recipients including a key that is already unlocked / passphrase entered.
• Decrypt the message.

Result is that I'm asked for the passphrase of all my keys.

IMO the priority here is Whishlist because hidden recipients is an expert / non standard feature for which we don't need good usability but might wish to improve usability for.