I found a race condition between KILLAGENT command and accepting another request.
Here is a patch to replicate the race condition :

Because of this, gpgconf --kill all may be failed, since it firstly invokes gpg-connect-agent KILLAGENT then secondly invokes gpg-connect-agent "GETINFO scd_running" .... While first invocation asks killing agent and it may take time, second invocation may connect successfully but during processing the GETINFO command the process of gpg-agent exits.

Correct solution is to implement KILLAGENT synchronously, but it's somehow harder to implement.
Easier workaround is modifying gpgconf like:

diff --git a/tools/gpgconf-comp.c b/tools/gpgconf-comp.c
index d4b3ebd29..1ba242387 100644
--- a/tools/gpgconf-comp.c
+++ b/tools/gpgconf-comp.c
@@ -1339,7 +1339,7 @@ gc_component_kill (int component)
     }
 
   /* Do the restart for the selected backends.  */
-  for (backend = 0; backend < GC_BACKEND_NR; backend++)
+  for (backend = GC_BACKEND_NR-1; backend; backend--)
     {
       if (runtime[backend] && gc_backend[backend].runtime_change)
         (*gc_backend[backend].runtime_change) (1);

@gniibe, thanks for the diagnosis! I agree that restarting or shutting down the backends should be done in the reverse order as a simple workaround.

We've known for some time that shutting down the background daemons is problematic. while they're shutting down, they're still holding the listening socket open, but not answering new requests on it. during that time, we see this particular race.

Interestingly, once the agent completely *does* terminate, we *don't* get the problem, because the client will just auto-launch another agent.

It's never been clear to me what the value of of this window of inaccessibility is.

I note that if the agent was to stop accept()ing on the listening socket and close it as soon as the agent plans to shut down, then the window would be significantly smaller (i'm not sure how to eliminate the window entirely).

There are two different cases: (1) By SIGTERM and (2) By KILLAGENT. It's true that the agent stops accepting on the listening socket for (1), but it's not the case for (2).
This particular problem is for the case (2).

For the window between two SIGTERMs, my interpretation is that it is an intentional behavior for debug purpose. Well, I also don't fully understand how it's useful, though.

For fully fixing (2), I think that it requires a bit of larger surgery of the agent. Currently, threads are dispatched upon accepting on the listening socket, and the main loop does no check for those children threads. This structure should be changed (or we need to introduce interaction between child thread to the main loop), so that the main loop can synchronously receive the notification of KILLAGENT (to shutdown the process), and the child thread can defer answer to the client until it can make sure gracefully shutting down.

Hm, T4521 suggests that the two different cases should not be treated differently. If you think that they *should* cause distinct behavior, please do mention it over there!

@werner, perhaps you can comment on whether there is a design goal for this window of inaccessibility?

I'm not sure i understand the options you lay out for dealing with case (2), @gniibe, but "or we need to introduce interaction between child thread to the main loop" sounds like a patch we've been shipping in debian for a while that allows a client to send a signal to the parent. I don't know how to make it work on Windows though -- perhaps @aheinecke can suggest how it would work on that platform?

@dkg, for your patch, it can be improved for Windows by using its event mechanism. You can see gnupg/scd/scdaemon.c.

Somehow related things...

Last year, actually, I tried to merge your patch, when I needed similar for scdaemon.
For scdaemon, currently, we use pipe instead of signal, because I encountered a symptom: pselect doesn't return EINTR by the signal

This use of pipe is needed for NetBSD, and it's also needed for GNU/Linux (other than ARM) in a corner case of suspend/resume.
In suspend/resume on GNU/Linux machine, SIGCONT is used internally to wake up frozen processes, and there is a race condition where a scdaemon thread of USB sending SIGCONT and pselect is restarted wrongly, never returns EINTR.

I sent my fix to the kernel: https://lkml.org/lkml/2018/3/21/909
I had expected that fix were accepted soonish, so that I could merge your patch with no change for GNU/Linux.

Perhaps, it's time to re-evaluate.

I'm unlikely to put a windows-specific patch into the debian source, as
i have no good way of testing it, and it wouldn't affect any binary that
we ship.

If you wanted to imlement this functionality in gpg-agent in a way that
i can drop the debian patches, i would be very happy, though :)