Nevermind, I did not realize that passwd does not only operate on the selected key but on all keys (subkeys) in sequence.

I tried to remove the passphrase on my authentication subkey but the same issue seems to still be present in version 2.2.2.