Because of PQ crypto which size is larger, the data to be signed (with Ed25519) is too large (something like 330-byte). It's not OpenPGP which does double-hashing, the protocol uses bare data to be signed (not hashed beforehand).
While it can be signed by token/smartcard (at least with newer Gnuk), as long as it is smaller than the buffer size, scdaemon itself rejects, not asking token/smartcard.