Thanks for your fast reply.
Sadly I have not much time these days... but I have done what you suggested.
Honestly the log files dont tell me much. One thing I recognised is sometimes the logfiles end with "Fatal: libgcrypt
problem: out of core in secure memory" and sometimes they dont (I have not copied every log file here, this might
be to much).

This is what I got with debug memstat and the recent version of gnupg in Arch.

2016-11-30 21:18:35 gpg-agent[5516] listening on socket '/run/user/1000/gnupg/S.gpg-agent'
2016-11-30 21:18:35 gpg-agent[5517] gpg-agent (GnuPG) 2.1.15 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe1d0e5700 for fd 5 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe1c8e4700 for fd 6 started
2016-11-30 21:18:45 gpg-agent[5517] starting a new PIN Entry
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe17fff700 for fd 8 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe177fe700 for fd 9 started
2016-11-30 21:18:51 gpg-agent[5517] handler 0x7efe1d0e5700 for fd 5 terminated
2016-11-30 21:18:52 gpg-agent[5517] Fatal: out of core in secure memory while allocating 512 bytes

2016-11-30 21:18:52 gpg-agent[5517] Fatal: libgcrypt problem: out of core in secure memory
2016-11-30 21:18:35 gpg-agent[5516] listening on socket '/run/user/1000/gnupg/S.gpg-agent'
2016-11-30 21:18:35 gpg-agent[5517] gpg-agent (GnuPG) 2.1.15 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe1d0e5700 for fd 5 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe1c8e4700 for fd 6 started
2016-11-30 21:18:45 gpg-agent[5517] starting a new PIN Entry
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe17fff700 for fd 8 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe177fe700 for fd 9 started
2016-11-30 21:18:51 gpg-agent[5517] handler 0x7efe1d0e5700 for fd 5 terminated
2016-11-30 21:18:52 gpg-agent[5517] Fatal: out of core in secure memory while allocating 512 bytes

2016-11-30 21:18:52 gpg-agent[5517] Fatal: libgcrypt problem: out of core in secure memory
v2016-11-30 21:18:35 gpg-agent[5516] listening on socket '/run/user/1000/gnupg/S.gpg-agent'
2016-11-30 21:18:35 gpg-agent[5517] gpg-agent (GnuPG) 2.1.15 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe1d0e5700 for fd 5 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe1c8e4700 for fd 6 started
2016-11-30 21:18:45 gpg-agent[5517] starting a new PIN Entry
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe17fff700 for fd 8 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe177fe700 for fd 9 started
2016-11-30 21:18:51 gpg-agent[5517] handler 0x7efe1d0e5700 for fd 5 terminated
2016-11-30 21:18:52 gpg-agent[5517] Fatal: out of core in secure memory while allocating 512 bytes

2016-11-30 21:18:52 gpg-agent[5517] Fatal: libgcrypt problem: out of core in secure memory

This is what I got with debuging memstat and downgraded gnupg to gnupg-2.1.15-2-x86_64.pkg.tar.xz

2016-11-30 21:28:30 gpg-agent[5953] listening on socket '/run/user/1000/gnupg/S.gpg-agent'
2016-11-30 21:28:30 gpg-agent[5954] gpg-agent (GnuPG) 2.1.15 started
2016-11-30 21:28:37 gpg-agent[5954] handler 0x7fd6fedca700 for fd 5 started
2016-11-30 21:28:37 gpg-agent[5954] starting a new PIN Entry
2016-11-30 21:28:42 gpg-agent[5954] handler 0x7fd6fedca700 for fd 5 terminated
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fedca700 for fd 5 started
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fe5c9700 for fd 7 started
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fddc8700 for fd 8 started
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fd5c7700 for fd 9 started
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fedca700 for fd 5 terminated
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fddc8700 for fd 8 terminated
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fe5c9700 for fd 7 terminated
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fd5c7700 for fd 9 terminated
2016-11-30 21:29:32 gpg-agent[5954] handler 0x7fd6fe5c9700 for fd 7 started
2016-11-30 21:29:32 gpg-agent[5954] handler 0x7fd6fe5c9700 for fd 7 terminated
2016-11-30 21:30:10 gpg-agent[5954] handler 0x7fd6fe5c9700 for fd 5 started
2016-11-30 21:30:10 gpg-agent[5954] handler 0x7fd6fd5c7700 for fd 7 started
2016-11-30 21:30:10 gpg-agent[5954] handler 0x7fd6fddc8700 for fd 8 started
2016-11-30 21:30:10 gpg-agent[5954] handler 0x7fd6fedca700 for fd 9 started
2016-11-30 21:30:10 gpg-agent[5954] Fatal: out of core in secure memory while allocating 512 bytes

2016-11-30 21:30:10 gpg-agent[5954] Fatal: libgcrypt problem: out of core in secure memory

And this is what I got from gdb (I am quite unfamiliar with gdb, so maybe i done something wrong)

Reading symbols from gpg-agent...(no debugging symbols found)...done.
Attaching to program: /usr/bin/gpg-agent, process 3492
Reading symbols from /usr/lib/libgcrypt.so.20...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libgpg-error.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libassuan.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libnpth.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Reading symbols from /usr/lib/libc.so.6...(no debugging symbols found)...done.
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
0x00007f05452cd18c in pselect () from /usr/lib/libc.so.6
(gdb) break log_fatal
Function "log_fatal" not defined.
Make breakpoint pending on future shared library load? (y or [n])
(gdb) c
Continuing.
[New Thread 0x7f053ffff700 (LWP 4687)]
[New Thread 0x7f05449ea700 (LWP 4698)]
[New Thread 0x7f053f7fe700 (LWP 4699)]
[New Thread 0x7f05451eb700 (LWP 4700)]
[Thread 0x7f053ffff700 (LWP 4687) exited]
[Thread 0x7f053f7fe700 (LWP 4699) exited]
[Thread 0x7f05451eb700 (LWP 4700) exited]
[Thread 0x7f05449ea700 (LWP 4698) exited]
[New Thread 0x7f05449ea700 (LWP 4733)]
[New Thread 0x7f05451eb700 (LWP 4745)]
[New Thread 0x7f053f7fe700 (LWP 4746)]
[New Thread 0x7f053ffff700 (LWP 4747)]
[Thread 0x7f053f7fe700 (LWP 4746) exited]
[Thread 0x7f05449ea700 (LWP 4733) exited]
[Thread 0x7f05451eb700 (LWP 4745) exited]
[Thread 0x7f053ffff700 (LWP 4747) exited]
[New Thread 0x7f053ffff700 (LWP 4775)]
[New Thread 0x7f05451eb700 (LWP 4776)]
[Thread 0x7f053ffff700 (LWP 4775) exited]

Thread 11 "gpg-agent" received signal SIGPIPE, Broken pipe.
[Switching to Thread 0x7f05451eb700 (LWP 4776)]

0x00007f054559a16d in write () from /usr/lib/libpthread.so.0

This should be fixed in: 2f27cb12e30c9f6e780354eecc3ff0039ed52c63 .

Applied to 2.0, too. Will be in 2.0.31.

Fixed in 2.1.16.

Fixed in 2.1.11 and 2.0.30.

Fixed in 2.1.16.

Fixed in 2.1.16. Will be in 2.0.31 as the fix is in the git repo already.

Fixed with nPth 1.3.

Fixed in 1.3.

Fixed with nPth 1.3.

Fixed in STABLE-BRANCH-2-0 branch of git repo, as of the commit:
5c599e4f6edd288f4759c9fc2bcf9fe87dee1836

While looking at the problem I found a corner case related to a shutdown and
fixed that.

I also tried to close the listening socket after the first shutdown event. I
reverted that because the effect is that a client trying to connect immediately
gets a failure and then starts a new dirmngr - which is not the idea of a shutdown.

The man pages notes:

SIGTERM

   Shuts down the process but waits until all current requests are
   fulfilled.  If the process has received 3  of these signals
   and requests are still pending, a shutdown is forced.  You may
   also use
         gpgconf --kill dirmngr
   instead of this signal

thus this is by design and identical to what gpg-agent does. IIRC, there was a
regression for some time, fixed in 2.1.16. So this fixed regression is what you
see as a bug.

However, the process should not anymore listen for new connections as soon as a
shutdown is pending. That needs to be fixed.

Yeah, lets do that. Commit 8489b12 to go into 2.1.17. Thanks.

commit a5910e00ace882b8a17169faf4607163ab454af9 should fix that. Will go into
2.1.17.

What about putting in the suggested patch as an intermediate step towards a full
solution?

Addressed in 9fb5e9c14557f7567cbc7c50b9881b7d7bfa2f12.

Is that sufficient?

On Windows especially the initial keylist is very slow, subsequent keylists are
okish (less then 10 seconds) I don't think it's as big a problem anymore.
Listing a specific key is ~100ms. And that is with a large keyring (~18mb) on a
VM with a fairly slow harddisk.

For me this would be good enough to use tofu on windows. So it can be resolved
if you do not think the performance (especially of the initial listing) can be
improved or should have been better.

PS C:\Users\aheinecke> Measure-Command -Expression { gpg --no-auto-check-trustdb
--with-colons --trust-model tofu --list-keys --with-colons > $null }
gpg: NOTE: THIS IS A DEVELOPMENT VERSION!
gpg: It is only intended for test purposes and should NOT be
gpg: used in a production environment or with production keys!
gpg: please do a --check-trustdb
gpg: public key 60041E4EC03449C4 is 39 seconds newer than the signature
gpg: public key 60041E4EC03449C4 is 39 seconds newer than the signature

Days : 0
Hours : 0
Minutes : 1
Seconds : 14
Milliseconds : 785
Ticks : 747854659
TotalDays : 0.000865572521990741
TotalHours : 0.0207737405277778
TotalMinutes : 1.24642443166667
TotalSeconds : 74.7854659
TotalMilliseconds : 74785.4659

PS C:\Users\aheinecke> Measure-Command -Expression { gpg --no-auto-check-trustdb
--with-colons --trust-model tofu --list-keys --with-colons > $null }
gpg: NOTE: THIS IS A DEVELOPMENT VERSION!
gpg: It is only intended for test purposes and should NOT be
gpg: used in a production environment or with production keys!
gpg: please do a --check-trustdb
gpg: public key 60041E4EC03449C4 is 39 seconds newer than the signature

Days : 0
Hours : 0
Minutes : 0
Seconds : 7
Milliseconds : 812
Ticks : 78128420
TotalDays : 9.0426412037037E-05
TotalHours : 0.00217023388888889
TotalMinutes : 0.130214033333333
TotalSeconds : 7.812842
TotalMilliseconds : 7812.842

PS C:\Users\aheinecke> Measure-Command -Expression { gpg --no-auto-check-trustdb
--with-colons --trust-model pgp --list-keys --with-colons > $null }
gpg: NOTE: THIS IS A DEVELOPMENT VERSION!
gpg: It is only intended for test purposes and should NOT be
gpg: used in a production environment or with production keys!
gpg: public key 60041E4EC03449C4 is 39 seconds newer than the signature

Days : 0
Hours : 0
Minutes : 0
Seconds : 1
Milliseconds : 369
Ticks : 13697177
TotalDays : 1.58532141203704E-05
TotalHours : 0.000380477138888889
TotalMinutes : 0.0228286283333333
TotalSeconds : 1.3697177
TotalMilliseconds : 1369.7177

PS C:\Users\aheinecke> gpg --version
gpg (GnuPG) 2.1.17-beta30
libgcrypt 1.7.3

Home: C:/Users/aheinecke/AppData/Roaming/gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,

CAMELLIA128, CAMELLIA192, CAMELLIA256

Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

Released with 2.1.16.

all done.

Patrick also mentioned this on the ML. I am not sure whether this has been
fixed. Can you please check tools/Makefile.am and close this bug if -lintl has
not yet been added.

FWIW, we are running build tests now on macOS Sierra w/o problems.

Patch 0001 should be applied to 2.3

Please do not use "checking-upstream-swdb" patch.

Sure, for Debian and other distros the version number is of no use and should
not be used (I am still annoyed by xlockscreen thing). However disabling this
in dirmngr is the wrong approach. It should be disabled in tools which actually
use that service (e.g. KMail). The SWDB file carries more version information
than just GPA and is thus useful for developers who build their own version of
GPA or their own Windows installer. It has also nothing to do with the wakeups.

Having a dirmngr installed which does not work as described is a bad idea.

BTW: although we won't be able to implement key retrieval queueing into dirmngr
(e.g. for use with --auto-key-retrieve) in time for the Debain freeze, we will
add this later so that it may be available in a later point release. Obviously
this needs regualr wakeups to test for network connectivity and to process the
queue.

I just pushed the LDAP reaper patch for 2.1.17.

The LDAP stuff is mainly used for CRLs and is often hard to deploy because often
proxies are needed etc. I don't know a public one which is reliable enough for
testing. The one I used mostly was related to certain smartcards but those
cards expire faster than software can be deployed. Fortunately most public CRLs
are available via HTTP.

Another use are LDAP keyservers. I do not know a public service, Some
keyserver operators run them privately and Ireply on them to test GnUPG's support.

Sorry, I have not used those conf files suffixed for a long time.

gpg-agent sets 32k aside for so called secure memory. It seems Libgcrypt runs
out of memory during computations with private key parameters.

Please put "debug memstat" into gpg-agent.conf which should print two lines of
info at process termination. If possible do the same with the old version and
compare.

Another thing you can do is to start gpg-agent ("gpgconf --launch gpg-agent"),
then look for its PID and attach gdb:

  $ gpg gpg-agent PID
  gdb> break log_fatal
  gdb> c

after you hit the breakpoint enter "bt".

Thank you for your report.
In 2.1.x, I fixed scdaemon so that card removal works fine.
I'll backport to 2.0.

I could reproduce this by opening two crypto mails in multiple windows this
reliably triggered the crash.

I have not fully understood the crash as it crashed in the close invocation in
outlook. After various trys and improvements to our code (there were some fishy
cleanups) i was able to fix this by closing the inspector of the mailobject
before closing the mail. Outlook apprarently did not like it if I closed a mail
that was active in an inspector but that is a bit speculation.

Also:

$ ssh -V
OpenSSH_7.2p2, LibreSSL 2.4.1

Let's use T2425 for the tar failure, and T2847 for the ssh failure. The
log you posted here shows exactly the same problem as in T2847.

Do you also see tar failing?

You can use

make -Ctests/openpgp check XTESTS="gpgtar.scm gpgtar.scm gpgtar.scm gpgtar.scm
gpgtar.scm"

to run the same test over and over again. That is how I measured how often we
see the failure. We updated our box since, and I haven't tried it again yet.

Thanks for the report.

I changed the title to reflect what I learned from the log.

Our test runs fine, here a recent the log:

http://jenkins.gnupg.org/job/gnupg/501/XTARGET=native,label=macos/consoleFull

I don't know how to compare the OS versions, but this is what I see:

$ uname -a
Darwin ... 16.0.0 Darwin Kernel Version 16.0.0: Mon Aug 29 17:56:20 PDT 2016;
root:xnu-3789.1.32~3/RELEASE_X86_64 x86_64
$ shasum /usr/bin/ssh-add
bdb1005292b0891edba78b3f1f00fe036c4e60f9 /usr/bin/ssh-add

Could you please arrange the tests to be called using 'make check verbose=2',
and post
the generated ssh.scm.log file? For reference, here is our log:

http://jenkins.gnupg.org/job/gnupg/XTARGET=native,label=macos/ws/obj/tests/openpgp/ssh-import.scm.log/*view*/

(Note that I just renamed the test to 'ssh-import.scm'.)

Fixed in 4db9a425644dccaf81b51ebc97b32a9cc21941a4.
Test for --export-ssh-key added in 47b8b9e2ce5af7fba117ae0b00e10bec414dcfb0.