What about putting in the suggested patch as an intermediate step towards a full
solution?

Addressed in 9fb5e9c14557f7567cbc7c50b9881b7d7bfa2f12.

Is that sufficient?

On Windows especially the initial keylist is very slow, subsequent keylists are
okish (less then 10 seconds) I don't think it's as big a problem anymore.
Listing a specific key is ~100ms. And that is with a large keyring (~18mb) on a
VM with a fairly slow harddisk.

For me this would be good enough to use tofu on windows. So it can be resolved
if you do not think the performance (especially of the initial listing) can be
improved or should have been better.

PS C:\Users\aheinecke> Measure-Command -Expression { gpg --no-auto-check-trustdb
--with-colons --trust-model tofu --list-keys --with-colons > $null }
gpg: NOTE: THIS IS A DEVELOPMENT VERSION!
gpg: It is only intended for test purposes and should NOT be
gpg: used in a production environment or with production keys!
gpg: please do a --check-trustdb
gpg: public key 60041E4EC03449C4 is 39 seconds newer than the signature
gpg: public key 60041E4EC03449C4 is 39 seconds newer than the signature

Days : 0
Hours : 0
Minutes : 1
Seconds : 14
Milliseconds : 785
Ticks : 747854659
TotalDays : 0.000865572521990741
TotalHours : 0.0207737405277778
TotalMinutes : 1.24642443166667
TotalSeconds : 74.7854659
TotalMilliseconds : 74785.4659

PS C:\Users\aheinecke> Measure-Command -Expression { gpg --no-auto-check-trustdb
--with-colons --trust-model tofu --list-keys --with-colons > $null }
gpg: NOTE: THIS IS A DEVELOPMENT VERSION!
gpg: It is only intended for test purposes and should NOT be
gpg: used in a production environment or with production keys!
gpg: please do a --check-trustdb
gpg: public key 60041E4EC03449C4 is 39 seconds newer than the signature

Days : 0
Hours : 0
Minutes : 0
Seconds : 7
Milliseconds : 812
Ticks : 78128420
TotalDays : 9.0426412037037E-05
TotalHours : 0.00217023388888889
TotalMinutes : 0.130214033333333
TotalSeconds : 7.812842
TotalMilliseconds : 7812.842

PS C:\Users\aheinecke> Measure-Command -Expression { gpg --no-auto-check-trustdb
--with-colons --trust-model pgp --list-keys --with-colons > $null }
gpg: NOTE: THIS IS A DEVELOPMENT VERSION!
gpg: It is only intended for test purposes and should NOT be
gpg: used in a production environment or with production keys!
gpg: public key 60041E4EC03449C4 is 39 seconds newer than the signature

Days : 0
Hours : 0
Minutes : 0
Seconds : 1
Milliseconds : 369
Ticks : 13697177
TotalDays : 1.58532141203704E-05
TotalHours : 0.000380477138888889
TotalMinutes : 0.0228286283333333
TotalSeconds : 1.3697177
TotalMilliseconds : 1369.7177

PS C:\Users\aheinecke> gpg --version
gpg (GnuPG) 2.1.17-beta30
libgcrypt 1.7.3

Home: C:/Users/aheinecke/AppData/Roaming/gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,

CAMELLIA128, CAMELLIA192, CAMELLIA256

Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

Released with 2.1.16.

all done.

Patrick also mentioned this on the ML. I am not sure whether this has been
fixed. Can you please check tools/Makefile.am and close this bug if -lintl has
not yet been added.

FWIW, we are running build tests now on macOS Sierra w/o problems.

Patch 0001 should be applied to 2.3

Please do not use "checking-upstream-swdb" patch.

Sure, for Debian and other distros the version number is of no use and should
not be used (I am still annoyed by xlockscreen thing). However disabling this
in dirmngr is the wrong approach. It should be disabled in tools which actually
use that service (e.g. KMail). The SWDB file carries more version information
than just GPA and is thus useful for developers who build their own version of
GPA or their own Windows installer. It has also nothing to do with the wakeups.

Having a dirmngr installed which does not work as described is a bad idea.

BTW: although we won't be able to implement key retrieval queueing into dirmngr
(e.g. for use with --auto-key-retrieve) in time for the Debain freeze, we will
add this later so that it may be available in a later point release. Obviously
this needs regualr wakeups to test for network connectivity and to process the
queue.

I just pushed the LDAP reaper patch for 2.1.17.

The LDAP stuff is mainly used for CRLs and is often hard to deploy because often
proxies are needed etc. I don't know a public one which is reliable enough for
testing. The one I used mostly was related to certain smartcards but those
cards expire faster than software can be deployed. Fortunately most public CRLs
are available via HTTP.

Another use are LDAP keyservers. I do not know a public service, Some
keyserver operators run them privately and Ireply on them to test GnUPG's support.

Sorry, I have not used those conf files suffixed for a long time.

gpg-agent sets 32k aside for so called secure memory. It seems Libgcrypt runs
out of memory during computations with private key parameters.

Please put "debug memstat" into gpg-agent.conf which should print two lines of
info at process termination. If possible do the same with the old version and
compare.

Another thing you can do is to start gpg-agent ("gpgconf --launch gpg-agent"),
then look for its PID and attach gdb:

  $ gpg gpg-agent PID
  gdb> break log_fatal
  gdb> c

after you hit the breakpoint enter "bt".

Thank you for your report.
In 2.1.x, I fixed scdaemon so that card removal works fine.
I'll backport to 2.0.

I could reproduce this by opening two crypto mails in multiple windows this
reliably triggered the crash.

I have not fully understood the crash as it crashed in the close invocation in
outlook. After various trys and improvements to our code (there were some fishy
cleanups) i was able to fix this by closing the inspector of the mailobject
before closing the mail. Outlook apprarently did not like it if I closed a mail
that was active in an inspector but that is a bit speculation.

Also:

$ ssh -V
OpenSSH_7.2p2, LibreSSL 2.4.1

Let's use T2425 for the tar failure, and T2847 for the ssh failure. The
log you posted here shows exactly the same problem as in T2847.

Do you also see tar failing?

You can use

make -Ctests/openpgp check XTESTS="gpgtar.scm gpgtar.scm gpgtar.scm gpgtar.scm
gpgtar.scm"

to run the same test over and over again. That is how I measured how often we
see the failure. We updated our box since, and I haven't tried it again yet.

Thanks for the report.

I changed the title to reflect what I learned from the log.

Our test runs fine, here a recent the log:

http://jenkins.gnupg.org/job/gnupg/501/XTARGET=native,label=macos/consoleFull

I don't know how to compare the OS versions, but this is what I see:

$ uname -a
Darwin ... 16.0.0 Darwin Kernel Version 16.0.0: Mon Aug 29 17:56:20 PDT 2016;
root:xnu-3789.1.32~3/RELEASE_X86_64 x86_64
$ shasum /usr/bin/ssh-add
bdb1005292b0891edba78b3f1f00fe036c4e60f9 /usr/bin/ssh-add

Could you please arrange the tests to be called using 'make check verbose=2',
and post
the generated ssh.scm.log file? For reference, here is our log:

http://jenkins.gnupg.org/job/gnupg/XTARGET=native,label=macos/ws/obj/tests/openpgp/ssh-import.scm.log/*view*/

(Note that I just renamed the test to 'ssh-import.scm'.)

Fixed in 4db9a425644dccaf81b51ebc97b32a9cc21941a4.
Test for --export-ssh-key added in 47b8b9e2ce5af7fba117ae0b00e10bec414dcfb0.

Just for the record:
It's gpg.conf-1 or gpg.conf-2 and not gpg.conf.1

My workaround for this problem also was to have a gpg.conf-2 which is then used
by gpgconf and a gpg.conf that is used by gpg 1.

gpgconf, which is a gnupg 2 tool, can't work with gpg version 1. As soon as you
use options not available in gpg 1 you will run into problems for which there
may or may not be a workaround.

The easy workaround is to use gpg.conf.1 which will be used by gpg 1 instead of
gpg.conf.

What you describe is a standard requirement for many low level libraries and
tool when cross-compiling.

Please do not use the bug tracker for discussions but use gnupg-devel instead.
Thanks.

sorry, but it is not acceptable to copy executables via ssh to native computers
and execute them there and copy the result back to the build machine during
crosscompilation.
the whole arch-specific stuff is unnecessary when you just use pthread_mutex_t
directly and be done with it. patch attached.
in case there's a system that doesnt use pthreads, fine, then you can do the
arch-specific dance there, but please do not ruin the buildprocess for anyone
using a POSIX conforming system for this madness.

D399: 921_0001-fix-libgpg-error-lock-madness.patch

Werner, you closed this issue with (the now removed) T1448 (wk on Jun 24 2014, 01:42 PM / Roundup) stating:
"You may use --ignore-invalid-option to list options which are only implemented
by gpg2."

This option seems only to be supported in gpg.conf, not on the command line.
(but this is no problem for me)

And it generally works fine (thank you!), just not in this special case here,
becaue gpg1 accepts the option "--debug-level" as valid, but does not allow
any arguments (neither numbers nor e.g. "basic").

The result (with "debug-level basic" in line 42) is:

$ gpg
gpg: /home/thomas/.gnupg/gpg.conf:42: argument not expected

I'm currently using gpg (GnuPG) 1.4.18 from Debian jessie.

As I understand it, "debug-level" is intended to just be a dummy option in
gpg1 to avoid problems with this option appearing in gpg.conf, correct?
So we have two possible solutions:

• either remove option "debug-level" (and rely on "ignore-invalid-option debug-level")
• or accept an argument for "debug-level"

Indeed, I confirm that the newly updated version 2.1.16 fix this issue, thanks a
lot for doing this portability work!

The same problem reproduces with gnupg2 installed from Homebrew (w/o GPGTools patches).

I've updated the patch series here to the series we're using in debian for 2.1.16.

D389: 918_0005-dirmngr-Drop-useless-housekeeping.patch

D390: 917_0004-dirmngr-Avoid-automatically-checking-upstream-swdb.patch

D391: 916_0003-dirmngr-Lazily-launch-ldap-reaper-thread.patch

D392: 915_0002-dimrngr-Avoid-need-for-hkp-housekeeping.patch

D393: 914_0001-dirmngr-hkp-Avoid-potential-race-condition-when-some.patch

In practice, dirmngr from git master still wakes up every few seconds due to the
ldap-reaper thread, even if no connections to ldap have ever happened.

the patch dirmngr-Lazily-launch-ldap-reaper-thread.patch avoids this additional
wakeup at least for those dirmngr instances that have never used LDAP.

Fixed in 03a65a5. The time for doing a tofu --with-tofu-info --with-colons
listing is now similar to doing a pgp listing.

Please reopen if there are still unresolved issues.

$ time gpg2 --with-tofu-info --with-colons --no-auto-check-trustdb
--no-default-keyring --keyring /usr/share/keyrings/debian-keyring.gpg
--trust-model pgp -k >/dev/null
gpg: Note: signatures using the MD5 algorithm are rejected

real 0m1.972s
user 0m1.940s
sys 0m0.028s
$ time gpg2 --with-tofu-info --with-colons --no-auto-check-trustdb
--no-default-keyring --keyring /usr/share/keyrings/debian-keyring.gpg
--trust-model tofu -k >/dev/null
gpg: Note: signatures using the MD5 algorithm are rejected

real 0m2.252s
user 0m2.172s
sys 0m0.020s

I suspect that the problem is the same as T2817.

Andre and I chatted about this issue offline, and I now understand what the
problem is. The TOFU_STATS status line (as documented in gnupg/doc/DETAILS) has
a "validity" field that is a number between 0 and 4 where 1 to 4 indicate how
confident we are that the binding is valid, and 0 means that the binding has an
unresolved conflict. The problem that Andre has observed is that this field is
not set to 0 if there is a conflict.

As a matter of fact, the validity field is never set to 0. This is completely
redundant as the same TOFU_STATS status line has a policy parameter, which is
"ask" if there is a conflict. Moreover, overloading this field in this way
causes a loss of information. Just because there is a conflict doesn't mean
that gpg shouldn't report the validity, or that the client can't made use of it.

Thus, in my opinion, the right thing to do is to simply use the <policy> field
to detect whether there is a conflict. Werner has suggested that this is wrong,
but I couldn't follow his logic. Thus, I'm adding him to the nosy list and I
hope he can clarify what he wants here.

The ssh.scm failure is still happening intermittently with 2.1.16

https://bot.brew.sh/job/Homebrew%20Versions%20Pull%20Requests/1733/version=yosemite/console

$ ssh -V
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011

Ah I spoke too soon. Just got the ssh.scm:
https://bot.brew.sh/job/Homebrew%20Versions%20Pull%20Requests/1733/version=yosemite/console

No problem. Thanks for looking into it.

My fault. Sorry.

Everything looks fine now that I removed all of the dependencies and started
from a blank slate. Sorry for the noise.

So far I'm not seeing the old "FAIL: gpgtar.scm" and "FAIL: ssh.scm"

Were those specifically fixed in some new commit(s), or am I just lucky so far?

This has been changed in 2.1.16 to happen only every minute. Along with the
wakeup being done at the full second (as has been agreed upon for other
daemons), this should be more of an annoyance than a real problem.

It has been confirmed that 2.1.16 solves the problem.

The reason for the crash is that 2.1.15 is calling gpgrt_set_syscall_clamp
before nPth is initialized. The nPth initialization was changed in 2.1.15 so to
solve problems on some other platforms.