Thanks. Adding 'PubkeyAuthentication unbound' to my ~/.ssh/config seems to workaround it for me on openssh-9.1p1-3 (arch). I don't quite follow what the implications of that setting are though.

In T5931#165009, @alexk wrote:

A workaround you can add the following line to ~/.ssh/config or /etc/ssh/ssh_config:

KexAlgorithms -sntrup761x25519-sha512@openssh.com

For me ssh -o KexAlgorithms=-sntrup761x25519-sha512@openssh.com ... does work as well.

Its a nitrokey start. I gave it another spin just to make sure, and again when updating to openssh 9.0 and "gpg (GnuPG) 2.3.6-unknown", it fails (again with careful gpgconf --kill gpg-agent etc. Double checked the downloaded source code by arch's makepkg, appears to have that patch applied. Also tried adding -o KexAlgorithms=-sntrup761x25519-sha512@openssh.com to the ssh command, which didn't help.

FYI, I built 2.3.6 using a modified archlinux PKGBUILD (& disabling patches to avoid conflicts), then did:
gpgconf --kill gpg-agent
gpgconf --launch gpg-agent
but ssh still fails as before