Thank you, your suggestion inspired me to experiment a bit further and I found the problem - I needed in fact to delete the line from my ssh config, no idea why:

Match host * exec "gpg-connect-agent UPDATESTARTUPTTY /bye"

Now I update startup tty only on terminal start and it seems to be working. Still a bit strange.