Current backlog for listening gpg-agent socket is hard-coded and equals 5. It's too low value for running parralel ssh while using gpg-agent as ssh-agent. Default value for ssh-agent is 128 and it's ok.
My suggestion is to change hard-coded backlog value to configurable one and set 128 as default.