@werner That is not helpful. I tried 4 or 5 different readers. And the Reiner SCT cyberjack is the one that works best out of all of them on both Windows and Linux.

Some more info: OpenVPN does not care about the second reader only gnupg agent is sensitive to what is present when it is started. So a workaround that I just found is to disable the Virtual Smartcard reader first so that only the ReinerSCT smartcard reader with an OpenPGP V3.4 card is present. Make sure to open an SSH connection. Then reconnect the second reader. And reconnect to VPN. After the PIN for the OpenPGP V3.4 card is already cached and a connection to the card established I can also open more SSH connections with the second reader attached and disconnect and reconnect the VPN as I want.
Even removing the smartcard from the ReinerSCT reader and plugging it back in works and I can still authenticate with new SSH tunnels and both readers present. So it seems it is actually only important which readers are present when the agent connects for the first time.
So this is a practical woraround. Although disabling the TPM backed reader temporarily needs Admin rights and is really janky.

I am on Windows 10 21H1 and I using gnupg-w32-2.3.3_20211012 from here [1]
Together with win-gpg-agent, which extends gnupg to play nicely with Windows sockets. [2]