Turns out the cause is a wrong entry in the gpgsm.conf. Setting "dbug-level basic" without specifying an output file.
And encrypting a file is likewise affected.

This setting can be generated with the configure dialog in Kleopatra without any warning, I might add.

Setting a log file makes everything work again, as does removing the debug-level setting.

Do you have a gpgme log?

That look s like a problems with logging to stderr in --server mode. On Windows fds 0,1,2 are special.

The issue also occurs in VSD-3.3.2 and 4win-4.4.1 but not in VSD 3.1.26

Although in VSD 3.2.2 we get no warning when configuring S/MIME debugging wrong we then get a nice message "Configuration error" when trying to encrypt with S/MIME, instead of gpgsm hanging without any message at all:

I guess the disappearance of the message is most likely a regression which happened with version 3.3.0.

Another data point is that the faulty versions use libassuan 3 with a slightly changed API. May one of the follwing chnages cause the problem?

• API change: For new code, which uses libassuan with nPTH, please use gpgrt_get_syscall_clamp and assuan_control, instead of the system_hooks API. Use of ASSUAN_SYSTEM_NPTH is deprecated with new API version 3. If it's really needed to keep using old implementation of ASSUAN_SYSTEM_NPTH, you need to change your your application code, to define ASSUAN_REALLY_REQUIRE_V2_NPTH_SYSTEM_HOOKS before including <assuan.h>. For an application which uses version 2 API (NEED_LIBASSUAN_API=2 in its configure.ac), use of ASSUAN_SYSTEM_NPTH is still supported. [T5914]
• New functions assuan_pipe_wait_server_termination and assuan_pipe_kill_server to support abstraction of process. [T6487]

A search on codesearch.debian.net shows that the two new funtions from T6487 are not yet used by any code.

I wonder whether rA3bccb33ccd9028ff505d9979fd6c8a37393b892d which changes Assuan's waitpid function for Windows is well aligned with the my_waitpid in gpgme's assuan-support.c (which does nothing). gpgme creates a detached process in most cases but for gpgsm assuan_pipe_connect is used without the ASSUAN_PIPE_CONNECT_DETACHED flag.

A quick check with passing ASSUAN_PIPE_CONNECT_DETACHED does not changed anything.

Hypothetical scenario (gpgsm --server + dirmngr):
(0) It may hang when much debug messages are generated by libassuan to the pipe of --logger (diag_cb).
(1) In gpgme, by engine-gpgsm, gpgsm is invoked with --logger.
(2) If it's the case of standard gpgme interactions which uses gpgsm_io_event, no problem. Because the data on diag_cb is consumed well.
(3) In case of gpgsm_encrypt (or other commands), it uses gpgsm_assuan_simple_command which does not consume the data on diag_cb pipe at all.
(4) In particular, in set_recipients, gpgsm_assuan_simple_command is called by the number of recipients times.
(5) IIUC, in the server side, to handle RECIPIENT command, dirmngr is used by the call chain of:

• cmd_recipient
• gpgsm_add_to_certlist
• gpgsm_validate_chain...
• gpgsm_dirmngr_isvalid

(6) In gpgsm_dirmngr_isvalid function, libassuan is used as client side, it generates debug messages.
(7) When there are many recipients, the debug message may be big enough to fill the pipe.
(8) When pipe is filled, it blocks at log_string in my_libassuan_log_handler, waiting the data in pipe is consumed.
(9) ... it hangs.

Similar situation could happen with gpgsm + gpg-agent, when gpg-agent is invoked by gpgsm.
(1) No gpg-agent.
(2) In gpgme, by engine-gpgsm, gpgsm is invoked with --logger.
(3) In gpgsm_keylist, it makes sure gpg-agent is available by GETINFO agent-check, using gpgsm_assuan_simple_command.
(4) In the server side, it tries to connect gpg-agent, invokes gpg-agent, and connect to the agent again.
(5) On Windows, it may takes time to invoke gpg-agent. And it may try to connect multiple times. Each trial may generate debug messages.
(6) When it takes too much time, the debug messages are too much. It may fill the pipe.
(7) And it blocks at log_string in my_libassuan_log_handler.
(8) ... it hangs.

I fixed the problem (which I identified above) in gniibe/t7759 branch. There might be other causes/problems for the particular symptom, so, I don't know the fix resolves the symptom or not, though. Anyhow, I believe that this is an improvement.

The commits are two:

• rM52bde50b879c: gpgsm: Introduce two phase interactions to consume diag output.
• rMcff9df67d34f: gpgsm: Extend gpgsm_assuan_simple_command to consule diag output.

Those changes make gpgsm_assuan_simple_command go through even when there are much debug output in diag pipe.
It now selects on two pipes of status and diag. Since debug output are correctly consumed in the client side, the server side won't block at the debug output.

Looks good to me on gpg4win-5.0.0-beta369 @ win10.
Can't reproduce it anymore, message is S/MIME decrypted instantly: