After a lot of digging I finally found the problem. It's actually not Gpg4win/GnuPG, but it's the Bitwarden desktop app. They recently added support for it to function as an SSH agent, and even though I have not enabled that feature, it's hijacking the socket anyways. When I close Bitwarden the issue disappears. The issue is logged in bitwarden/clients#13150.

@werner Thank you for the response. Is there a nightly build or similar that I can grab from somewhere to see if using the latest master branch solves the issue?