https://bugs.debian.org/635935 reports that pinentry-gtk-2 (and presumably other
graphical pinentries) fail hard if $DISPLAY is set but cannot be opened.
They fall back to curses if $DISPLAY is unset. Maybe they should fallback to
curses if they know they are connected to a TTY *and* $DISPLAY is set but
unreachable?
Below is an example that could have succeeded if this workaround was in place:
0 dkg@alice:~$ DISPLAY=:noexist pinentry-gtk-2
(pinentry-gtk-2:16257): Gtk-WARNING **: cannot open display: :noexist
1 dkg@alice:~$