By the KEYGRIP centric support in scdaemon, we can move on to the next step;

Using a key on card, we still have the access pattern: firstly identify the key on card or identify card by SERIALNO and then, access by $AUTHKEYID, $ENCRKEYID, or $SIGNKEYID.

It is more natural for scdaemon (and gpg-agent) to expose a capability of AUTH/ENCR/SIGN of key, say, as a flag, when it handles KEYINFO command.

Then, we can remove the access by SERIALNO to change it access by KEYGRIP.