Afaict neither QT nor FLTK offer an equivalent to gtk-2's gtk_init_check() so there is no trivial change to get the same functionality.

I am pretty sure the heuristics used by pinentry-qt (5) are not correct ...

const bool hasWaylandDisplay = qEnvironmentVariableIsSet("WAYLAND_DISPLAY");
const bool isWaylandSessionType = qgetenv("XDG_SESSION_TYPE") == "wayland";
const bool hasX11Display = pinentry_have_display(argc, argv);
const bool isX11SessionType = qgetenv("XDG_SESSION_TYPE") == "x11";
const bool isGUISession = hasWaylandDisplay || isWaylandSessionType || hasX11Display || isX11SessionType;

... X11 does not work with $DISPLAY even if XDG_SESSION_TYPE=x11. $internetsearch suggests that XDG_SESSION_TYPE=wayland without WAYLAND_DISPLAY should fall back to a default and indeed work. So perhaps the hasX11SessionType should simply be removed.

We talked about this in our developer meeting on Monday. I have never experienced the problem because I use the Qt version only on Windows and for my own use I use the Gtk version. In any case I think that Qt and fltk should fallback to curses to cover the case of using the Pinentry for a system startup on the console (e.g. the g13 case) with later switching to a GUI. And of course for those users who switch between GUI and console.

As noted by @ametzler1 pinentry-qt has such a fallback. Of course, we can try to improve the heuristics pinentry-qt uses.

I'm not sure if we should consider env DISPLAY=invalid pinentry-qt a valid test. The generic pinentry_have_display in pinentry.c checks for a --display argument or for non-empty DISPLAY.

On the other hand, I guess XDG_SESSION_TYPE=x11 env -u DISPLAY pinentry-qt should fallback to TTY, i.e. for X11 pinentry-qt should require a DISPLAY. But that would break on Unix systems that set DISPLAY, but don't set XDG_SESSION_TYPE.

So, I guess, @ametzler1's suggestion to remove the check for isX11SessionType is the correct solution. DISPLAY=invalid would still not work, but I think that's acceptable.

In T8156#216401, @ikloecker wrote:

I'm not sure if we should consider env DISPLAY=invalid pinentry-qt a valid test.

[...]

So, I guess, @ametzler1's suggestion to remove the check for isX11SessionType is the correct solution. DISPLAY=invalid would still not work, but I think that's acceptable.

Please also handle invalid DISPLAY correctly, according to the original submitter this happens in real-world usage with GNU Screen and tmux.

How is "invalid DISPLAY" defined? DISPLAY=invalid? Anything that's not DISPLAY=:<some number>? Why do screen and tmux have to use an extra-wurst?

Apparently, DISPLAY is hostname:displaynumber.screennumber where hostname and .screennumber are optional and where hostname is a hostname or maybe host/unix. Does hostname include IPv6 address literals? Anyway, I guess the only sensible heuristic is to consider any DISPLAY value that contains : as valid.

"ikloecker (Ingo Klöcker)" wrote:

How is "invalid DISPLAY" defined? `DISPLAY=invalid`? Anything that's not `DISPLAY=:<some number>`? Why do screen and tmux have to use an extra-wurst?

[...]

There is is no special format for invalid DISPLAY strings, the essence is that you will get an error when you try to access the X server that $DISPLAY points to. e.g. it might have been accessible when you started the screen session but stopped being so when the session is continued.

Well, I don't think we'll add platform-specific X11 code to pinentry-qt just to check for an invalid DISPLAY. We are using Qt so that we don't have to deal with platform-specific stuff. I have no intention to look into this and, given Wayland, investing any more time in X11 feels wasted. We might accept a patch that can be used by all GUI pinentries to check for a usable DISPLAY.