There is a fairly obvious null pointer dereference in gpg-agent, file agent/command.c, line 1958. (Variable "pi" is null.)
This (at least on my machine) breaks symmetric decryption.
The bug is obvious, but the fix is less obvious. I tried lifting the memory allocation for "pi" out of the "if" statement, and passing pi to "agent_get_passphrase" (line 1955), but that broke symmetric encryption even worse. I commented out line 1958 and 1966, and that seems to work okay.