Pushed the change into master of libassuan (to be 3.0).

Pushed to GnuPG master. Let us test. For my machine of Debian GNU/Linux, Wine emulation (Windows 32-bit, Windows 64-bit), make check goes all well.
After confirming the implementation, I'd like to put it into gpgrt.

Implemented in rA9110945ce625: Implement timeout in assuan_sock_connect_byname..

Actually, it's not 'connect' system call, but 'CONNECT' request which matters. The use of SOCKS in libassuan is that it always connects to SOCKS server at localhost.
So, other than the special case of erroneous configuration of TOR, introducing timeout handling to the initial connection to the SOCKS server makes less sense.

I see the point of use of int.
For backward compatibility, the semantics of 0 should remain as default timeout (let kernel decide == 120 sec, usually), -1 would be meaning immediately (only success when local).

Why can't we keep the signed int? Do we ever need such a long timeout. We could for example define -1 as use default timeout.

assuan_sock_connect_byname may be extended to change the third argument (now int reserved) to unsigned int timeout.
It's a kind of API change, but ABI wise, the impact is minimum.

My understanding is that: selftest in Kleo does call assuan_socket_connect (possibly in kleopatra/src/libkleopatraclient/core/command.cpp), and it didn't send nonce correctly.

Thanks! I don't understand why your fix helps as Kleopatra never calls assuan_socket_connect on the server side, but it helps. :) So this is probably called indirectly and I don't see it.

I found an issue in the assuan code of client side. This might be the cause of the server failure for nonce.

The followup of this issue for libassuan is: https://dev.gnupg.org/T6324

What I mean is that our socket emulation is encapsulated in libgcrypt and details should not be visible to the caller. Further libassuan and kleopatra might be build against different libc versions and thus the used structures might also differ.

From the NEWS assuan_sock_set_sockaddr_un was only added in 2014, years after the UIServer code was really last modified.

The question is why Kleopatra does not use assuan_sock_set_sockaddr_un as we do in GnuPG. See for example
https://dev.gnupg.org/source/gnupg/browse/master/kbx/keyboxd.c$1124 - was this a workaround back when we had no support for Unicode? assuan_sock_set_sockaddr_un and assuan_sock_get_nonce work together and their internal workings should be opaque to the caller.

Btw. This is how Kleopatra creates the socket: https://dev.gnupg.org/source/kleo/browse/master/src/uiserver/uiserver_win.cpp$34 which does not use a function that would set is_socket=1. My naive fix would be:

works

Fixed by reverting to 2.5.5

Problem is that Kleopatra uses the latest libassuan from Git and not the released version. Replacing the libassuan DLL with the one from the gnupg directory fixes the bug.

On Windows, closing/inheriting handles is different to POSIX.
https://devblogs.microsoft.com/oldnewthing/20111216-00/?p=8873
https://devblogs.microsoft.com/oldnewthing/20130426-00/?p=4543
https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessw
https://learn.microsoft.com/en-us/windows/win32/api/winbase/ns-winbase-startupinfoexw
https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-initializeprocthreadattributelist
https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-updateprocthreadattribute
https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-deleteprocthreadattributelist

We should consider to break the Assuan API maybe we can do that without too many problems for the current use cases.

I'm going to use gnupg_process_* prefix for the functions.

Last two points are for future changes of assuan internal; For the case of controlling fds in detail, it is possible to use spawn callback controlling fds by the routine and let no-touching (inherit) by the spawn function.

I don't understand the last two points: This is only about the three standard descriptors but how shall we supply more descriptors? At least in GPGME we definitely need more.

Evaluating again, I'd like to change spawn functions like this one in libgpg-error:

Examining again, I realized that the current spawn API (not published yet, only available in libgpg-error master) is not that useful in general (or difficult to use), while it works somehow.

Here is the change of GnuPG to use new spawn functions from libgpg-error:

Merged PIPE connection part into master.