Turns out the cause is a wrong entry in the gpgsm.conf. Setting "dbug-level basic" without specifying an output file.
And encrypting a file is likewise affected.

This setting can be generated with the configure dialog in Kleopatra without any warning, I might add.

Setting a log file makes everything work again, as does removing the debug-level setting.

Do you have a gpgme log?

That look s like a problems with logging to stderr in --server mode. On Windows fds 0,1,2 are special.

The issue also occurs in VSD-3.3.2 and 4win-4.4.1 but not in VSD 3.1.26

Although in VSD 3.2.2 we get no warning when configuring S/MIME debugging wrong we then get a nice message "Configuration error" when trying to encrypt with S/MIME, instead of gpgsm hanging without any message at all:

I guess the disappearance of the message is most likely a regression which happened with version 3.3.0.

Another data point is that the faulty versions use libassuan 3 with a slightly changed API. May one of the follwing chnages cause the problem?

• API change: For new code, which uses libassuan with nPTH, please use gpgrt_get_syscall_clamp and assuan_control, instead of the system_hooks API. Use of ASSUAN_SYSTEM_NPTH is deprecated with new API version 3. If it's really needed to keep using old implementation of ASSUAN_SYSTEM_NPTH, you need to change your your application code, to define ASSUAN_REALLY_REQUIRE_V2_NPTH_SYSTEM_HOOKS before including <assuan.h>. For an application which uses version 2 API (NEED_LIBASSUAN_API=2 in its configure.ac), use of ASSUAN_SYSTEM_NPTH is still supported. [T5914]
• New functions assuan_pipe_wait_server_termination and assuan_pipe_kill_server to support abstraction of process. [T6487]

A search on codesearch.debian.net shows that the two new funtions from T6487 are not yet used by any code.

I wonder whether rA3bccb33ccd9028ff505d9979fd6c8a37393b892d which changes Assuan's waitpid function for Windows is well aligned with the my_waitpid in gpgme's assuan-support.c (which does nothing). gpgme creates a detached process in most cases but for gpgsm assuan_pipe_connect is used without the ASSUAN_PIPE_CONNECT_DETACHED flag.

A quick check with passing ASSUAN_PIPE_CONNECT_DETACHED does not changed anything.