The gnupg system tries to transmit all relevant information from the
gnupg front-ends through the agent to the pinentry so that it can
communicate with the user. For connections to the X server, this is
currently limited to the value of the DISPLAY environment variable.
This is not quite enough, though.

X clients usually also require authorization information. This
information is usually taken from ~/.Xauthority. However, sometimes the
user uses a different file for that and indicates this with the
XAUTHORITY environment variable. If pinentry's environment has the
wrong value for XAUTHORITY it cannot connect to the X server even if
DISPLAY is set correctly.

In many cases relying on the implicit use of ~/.Xauthority works fine.
But there's at least one scenario where it isn't: Using GDM to log in and
the Debian gnupg packages.

The Debian gnupg packages install a script that is run when the user
logs in via GDM. This script starts the gpg agent if it is not yet
running and otherwise reuse the already running one. Now, at least in
some circumstances, GDM puts the authorization information into a file
in /tmp/ and sets XAUTHORITY accordingly. This variable is inherited by
the agent if it is started during login, otherwise the agent will likely
have a different and wrong value for XAUTHORITY. This means that the
agent cal only start the pinentry program if it has been started during
the same GDM login session. A reused agent cannot start pinentry
properly.

One workaround is to make sure that either the agent is always started
during login (and old agents are killed once they're no longer needed).