Tested this with keybox and it appears to be working. When running a keylist
while importing the import holds for a bit and continues after the keylist.
Not tested this with keyring yet.

Several such bugs have been fixed since 2.1.2. Thus I assume this has been
resolved.

D321: 754_0001-Fix-possible-_SC_OPEN_MAX-max-problem-on-AIX.patch

D322: 753_0001-Fix-possible-AIX-problem-with-sysconf-in-rndunix.patch

Just in case you are not using /dev/random, the "Fix-posssible..." patch may
help for 1.4.

D323: 752_0001-common-Cope-with-AIX-problem-on-number-of-open-files.patch

Indeed the sysconf patch had not been backported to gnupg 2.0. Please
try the attached patch for 2.0.x (sorry for the trailing white space changes).

(newer?) AIX versions have the bug that sysconf returns INT_MAX32 for the number
of max. open file descriptors. That leads to a long delay due to closing all
possible open file descriptors. See T1778.

That bug has been fixed in GnuPG 2.1 but not in libassuan. I have fixed it with
commit 7101fcb for libassuan which wilk go into libassaun 2.4.3. This might
also help with GnuPG 2.0.26 but I need to check that.

This was implemented for 2.1. We won't backport it to 1.4 or 2.0.

Right, getkey_next had a somewhat surprising semantic. I fixed that with commit
b280aa6.

It also works with ECDSA keys.

Current master b2da3951 segfaults on me.
Btw. I think this is likely because i have a local ID without an Authentication
subkey for aheinecke@gnupg.org

(gdb) run --export-ssh-key aheinecke@gnupg.org
Starting program: /opt/gnupg/bin/gpg2 --export-ssh-key aheinecke@gnupg.org
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
gpg: NOTE: THIS IS A DEVELOPMENT VERSION!
gpg: It is only intended for test purposes and should NOT be
gpg: used in a production environment or with production keys!
gpg: enabled debug flags: memstat

Program received signal SIGSEGV, Segmentation fault.
lookup (ctx=ctx@entry=0x6dd940, ret_keyblock=ret_keyblock@entry=0x0,
ret_found_key=ret_found_key@entry=0x7fffffffd998,

want_secret=<optimized out>) at ../../g10/getkey.c:3116

3116 *ret_keyblock = keyblock; /* Return the keyblock. */
(gdb) bt full
#0 lookup (ctx=ctx@entry=0x6dd940, ret_keyblock=ret_keyblock@entry=0x0,
ret_found_key=ret_found_key@entry=0x7fffffffd998,

want_secret=<optimized out>) at ../../g10/getkey.c:3116
    rc = 0
    no_suitable_key = 0
    keyblock = 0x0
    found_key = 0x701980

#1 0x0000000000415bb6 in getkey_next (ctx=0x6dd940, pk=0x0, ret_keyblock=0x0)
at ../../g10/getkey.c:1636

rc = <optimized out>
found_key = 0x0

#2 0x000000000045713a in export_ssh_key (ctrl=0x6dd810, userid=0x7fffffffe420
"aheinecke@gnupg.org") at ../../g10/export.c:1437

getkeyctx = 0x6dd940
keyblock = 0x6fd160
desc = {mode = KEYDB_SEARCH_MODE_SUBSTR, skipfnc = 0x0, skipfncvalue =

0x0, sn = 0x0, snlen = 0, u = {

name = 0x7fffffffe420 "aheinecke@gnupg.org", fpr = "

\344\377\377\377\177", '\000' <repeats 17 times>, kid = {

4294960160, 32767}, grip = " \344\377\377\377\177", '\000'

<repeats 13 times>}, exact = 0}

curtime = 1452288169
pk = 0x0
identifier = 0x6ddb80 ""
mb = {len = 0, size = 4096, buf = 0x6e5d70 "", out_of_core = 0}
fp = 0x6dd810
b64_state = {flags = 7199040, idx = 0, quad_count = -153676256, fp =

0x10, stream = 0x6dd800, title = 0x6ddb80 "",

radbuf = "\000\000\000", crc = 0, stop_seen = -1, invalid_encoding =

0, lasterr = 0}

fname = 0x7fffffffe420 "aheinecke@gnupg.org"

#3 0x000000000040dc00 in main (argc=1, argv=0x7fffffffdfe8) at ../../g10/gpg.c:4193

pargs = {argc = 0x7fffffffdb9c, argv = 0x7fffffffdb90, flags = 32769,

err = 0, r_opt = 0, r_type = 0, r = {ret_int = 0,

ret_long = 0, ret_ulong = 0, ret_str = 0x0}, internal = {idx = 2,

inarg = 0, stopped = 1,

last = 0x7fffffffe420 "aheinecke@gnupg.org", aliases = 0x0,

cur_alias = 0x0, iio_list = 0x0}}

a = 0x6dd800
orig_argc = 0
orig_argv = 0x6ddb80
fname = 0x7fffffffe420 "aheinecke@gnupg.org"
sl = 0x0
remusr = 0x6ddb40
locusr = 0x0
nrings = 0x0
afx = 0x7fffffffe420
configfp = 0x7fffffffe420
configlineno = 27
parse_debug = 7198720
cmd = aExportSshKey
malloc_hooks = {malloc = 0x405ee0 <gcry_malloc@plt>, realloc = 0x406d40

<gcry_realloc@plt>, free = 0x406290 <gcry_free@plt>}

ctrl = 0x6dd810

backported by dkg with commit 0c3d764 for 1.4.19

New Debian package is uploaded based on master branch of Poldi.
Problem has gone, so, closing this issue.

Thank you. There is no reason. It is fixed in:

http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=commit;h=4ee881bff4c8fdfa4b3b7a4b7afab611471e97f1

I've tried to improve the web page.

http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg-doc.git;a=commit;h=fa61217e26a97c4b9f3294746a581aee5eb47ad8

Since Werner needs to check this, I'm changing the status of this issue to
testing and adding him to the cc.

@Reuben: If you have some ideas of additional improvements, I'd be grateful.
Thanks.

Based on Werner's comment, this issue has been addressed. As such, I'm closing
this bug report.

I'm closing this issue. As Andre already said, a bug report is probably not a
good place to track multiple issues. Further, there hasn't been any activity
for the past year so I'm guessing interest has been lost.

Thanks.

Committed (ec409e6).

Verifying the unwrapped data also works:

$ gpg2 --decrypt --unwrap /tmp/a > /tmp/b
Please enter the passphrase to unlock the OpenPGP secret key:
"Testing (insecure!)"
1024-bit RSA key, ID 6EA74366,
created 2015-09-18 (main key ID EE37CF96).

Passphrase:
gpg: encrypted with 1024-bit RSA key, ID 6EA74366, created 2015-09-18

"Testing (insecure!)"

$ gpg2 --verify /tmp/b
gpg: Signature made Wed 04 Nov 2015 01:53:31 PM CET using RSA key ID EE37CF96
gpg: Good signature from "Testing (insecure!)" [full]
gpg: Verified 7 messages signed by "Testing (insecure!)" (key: 362D 3527 F53A
AD19 71AA FDE6 5885 9975 EE37 CF96, policy: good) in the past 1 day, 20 hours.
The most recent message was verified 22 hours, 40 minutes ago.

Fixed in 6897bbf.

$ gpg2 --multifile --sign --encrypt-files /tmp/foo /tmp/bar
gpg: NOTE: THIS IS A DEVELOPMENT VERSION!
gpg: It is only intended for test purposes and should NOT be
gpg: used in a production environment or with production keys!
gpg: --sign --encrypt does not yet work with --multifile

Thank you for pointing out. It was long standing mistake.
Fixed in the repo.

For 1.6, please see:

    commit d501cc4edd55d3953d7581b3f8ff0c348df31ef0
    commit 24f6c65e36edec13aa781862ff1ff45ca3e99b99

Please test.

Fixed with commit d06a5f5.

Solution (c) will be used for 2.1.8.

Won't fix in 1.4 because that version is mostly useful on old systems and those
don't have proper utf-8 supoort anyway.

2.0.29-beta has a fix for this. See also T1823.

We should really do that for 1.7. SHA512 is important enough these days to
require a 64 bit type.

The GIT master and also the 1.6 branch now has a fix for that problem. A 1.6.4
release sill be done soon.

This problem still occurs with libgcrypt from current master:
libgcrypt 1.7.0-beta259

#0 0x655f24a7 in _gcry_aes_aesni_do_setkey (ctx=0x97f868,

key=0x656621b4 <key_128>

"\350\351\352\353\355\356\357\360\362\363\364\365\367\370\371\372\001K\257\"x\246\235\063\035Q\200\020\066C\351\232gC\303\321Q\232\264\362͚x\253\t\245\021\275]\036\362\r\316ּ\274\022\023\032\307\305G\210\252\b\016\225\027\353\026wq\232\317r\200\206\004",
<incomplete sequence \343>) at rijndael-aesni.c:117

Ooops - I should know it is my installer :-(
1.6.3.

IIRC, we fixed the alignment in Libgcrypt but I am not sure whether this has
been backported to Libgcrypt 1.6. Which libgcrypt version is used?

Backtrace with debug symbols:

(gdb) bt full
#0 0x655ea3e9 in aesni_do_setkey (ctx=0xc6f868,

key=0x6565dc10 <key_128.65421>

"\350\351\352\353\355\356\357\360\362\363\364\365\367\370\371\372\001K\257\"x\246\235\063\035Q\200\020\066C\351\232gC\303\321Q\232\264\362͚x\253\t\245\021\275]\036\362\r\316ּ\274\022\023\032\307\305G\210\252\b\016\225\027\353\026wq\232\317r\200\206\004",
<incomplete sequence \343>)

at

/home/aheinecke/arbeit/gpg4win/src/gnupg-w32-2.1.7/PLAY/src/libgcrypt/cipher/rijndael.c:248
No locals.
#1 0x655ead8a in do_setkey (ctx=0xc6f868,

key=0x6565dc10 <key_128.65421>

"\350\351\352\353\355\356\357\360\362\363\364\365\367\370\371\372\001K\257\"x\246\235\063\035Q\200\020\066C\351\232gC\303\321Q\232\264\362͚x\253\t\245\021\275]\036\362\r\316ּ\274\022\023\032\307\305G\210\252\b\016\225\027\353\026wq\232\317r\200\206\004",
<incomplete sequence \343>, keylen=16)

at

/home/aheinecke/arbeit/gpg4win/src/gnupg-w32-2.1.7/PLAY/src/libgcrypt/cipher/rijndael.c:569

initialized = 1
selftest_failed = 0x0
rounds = 10
i = 1
j = 1
r = 1
t = 13813018
rconpointer = 0
KC = 4
hwfeatures = 1472

#2 0x655eb2b1 in rijndael_setkey (context=0xc6f868,

key=0x6565dc10 <key_128.65421>

"\350\351\352\353\355\356\357\360\362\363\364\365\367\370\371\372\001K\257\"x\246\235\063\035Q\200\020\066C\351\232gC\303\321Q\232\264\362͚x\253\t\245\021\275]\036\362\r\316ּ\274\022\023\032\307\305G\210\252\b\016\225\027\353\026wq\232\317r\200\206\004",
<incomplete sequence \343>, keylen=16)

at

/home/aheinecke/arbeit/gpg4win/src/gnupg-w32-2.1.7/PLAY/src/libgcrypt/cipher/rijndael.c:668

ctx = 0xc6f868

...

info registers
eax 0x6565dc10 1701174288
ecx 0xd25110 13783312
edx 0xc6f868 13039720
ebx 0x0 0
esp 0xc6f760 0xc6f760
ebp 0xc6f760 0xc6f760
esi 0x0 0
edi 0xd24478 13780088
eip 0x655ea3e9 0x655ea3e9 <aesni_do_setkey+31>
eflags 0x10297 [ CF PF AF SF IF RF ]
cs 0x1b 27
ss 0x23 35
ds 0x23 35
es 0x23 35
fs 0x3b 59
gs 0x0 0

disas 0x655ea3e2,0x655ea3ff

Dump of assembler code from 0x655ea3e2 to 0x655ea3ff:

0x655ea3e2 <aesni_do_setkey+24>:     mov    0xc(%ebp),%eax
0x655ea3e5 <aesni_do_setkey+27>:     movdqu (%eax),%xmm1

> 0x655ea3e9 <aesni_do_setkey+31>: movdqa %xmm1,(%edx)

   0x655ea3ed <aesni_do_setkey+35>:     aeskeygenassist $0x1,%xmm1,%xmm2
   0x655ea3f3 <aesni_do_setkey+41>:     pshufd $0xff,%xmm2,%xmm2
   0x655ea3f8 <aesni_do_setkey+46>:     movdqa %xmm1,%xmm3
   0x655ea3fc <aesni_do_setkey+50>:     pslldq $0x4,%xmm3

It appears to be that this is crash is due to the fact that windows uses a 4
Byte stack alignment and the movdqa call expects 16 byte alignment.

I've found some info on this here:
http://www.peterstock.co.uk/games/mingw_sse/

I also confirmed that with "-mstackrealign" the crash no longer happens.

Werner: should we add this globaly to the configure options of gcrypt or do you
have a better fix for this?

...
Or printf debugging was the wrong approach here.

Attaching gdb to the agent led to the following backtrace:

#0 0x655ea3e9 in aesni_do_setkey () from C:\Program
Files\GnuPG\bin\libgcrypt-20.dll
#1 0x655ead8a in do_setkey () from C:\Program Files\GnuPG\bin\libgcrypt-20.dll
#2 0x655eb2b1 in rijndael_setkey () from C:\Program
Files\GnuPG\bin\libgcrypt-20.dll
#3 0x655edadd in selftest_basic_128 () from C:\Program
Files\GnuPG\bin\libgcrypt-20.dll
#4 0x655ede09 in selftest () from C:\Program Files\GnuPG\bin\libgcrypt-20.dll
#5 0x655eabfc in do_setkey () from C:\Program Files\GnuPG\bin\libgcrypt-20.dll
#6 0x655eb2b1 in rijndael_setkey () from C:\Program
Files\GnuPG\bin\libgcrypt-20.dll
#7 0x655cd4ae in cipher_setkey () from C:\Program Files\GnuPG\bin\libgcrypt-20.dll
#8 0x655ce076 in _gcry_cipher_setkey () from C:\Program
Files\GnuPG\bin\libgcrypt-20.dll
#9 0x655c2308 in gcry_cipher_setkey () from C:\Program
Files\GnuPG\bin\libgcrypt-20.dll
#10 0x0041aea8 in agent_protect ()
#11 0x004189a9 in store_key ()
#12 0x0041950b in agent_genkey ()
#13 0x00407a5e in cmd_genkey ()

So I've built libgcrypt again with --disable-aesni-support (Which is also what
gpg4win uses). And the crash goes away.

Surprise. This issue is weird.

Agent calls: hash_passphrase in agent/protect.c:do_encryption
I've added a load of debug output there but this is where it crashes.
I've moved the get_standard_s2k_count out of that call to verify that this is
not he crashing part.

My code looks like this:

  log_debug ("%s:%s: Line: %d", __FILE__, __func__, __LINE__);
  unsigned long s2kcnt = get_standard_s2k_count();
  log_debug ("%s:%s: Line: %d", __FILE__, __func__, __LINE__);
  rc = hash_passphrase (passphrase, GCRY_MD_SHA1,
                        3, iv+2*blklen,
                        s2kcnt,
			key, keylen);
  log_debug ("%s:%s: Line: %d", __FILE__, __func__, __LINE__);

The debug output after the hash_passphrase is not reached. The line before is.

But now this is where it gets weird.

With (debug enhanced):

static int
hash_passphrase (const char *passphrase, int hashalgo,

int s2kmode,
const unsigned char *s2ksalt,
unsigned long s2kcount,
unsigned char *key, size_t keylen)

{

  /* The key derive function does not support a zero length string for
     the passphrase in the S2K modes.  Return a better suited error
     code than GPG_ERR_INV_DATA.  */
  int ret;
  log_debug ("%s:%s: Line: %d", __FILE__, __func__, __LINE__);
  if (!passphrase || !*passphrase)
    return gpg_error (GPG_ERR_NO_PASSPHRASE);
  log_debug ("%s:%s: Line: %d", __FILE__, __func__, __LINE__);
  ret = gcry_kdf_derive (passphrase, strlen (passphrase),
                         s2kmode == 3? GCRY_KDF_ITERSALTED_S2K :
                         s2kmode == 1? GCRY_KDF_SALTED_S2K :
                         s2kmode == 0? GCRY_KDF_SIMPLE_S2K : GCRY_KDF_NONE,
                         hashalgo, s2ksalt, 8, s2kcount,
                         keylen, key);
  log_debug ("%s:%s: Line: %d", __FILE__, __func__, __LINE__);
  log_debug ("ret: %i ", ret);

  return ret;

}

I can see the debug line above the return statement is executed and that it
returns 0! But i don't see the call returning to do_encryption.

The only idea explaining this behavior that i have so far is some kind of stack
corruption where has_passphrase tries to return to an invalid pointer. But i
don't see the problem atm.

Dup key fixed with commit 4a326d7

Yes they are, the output of `$GNUPGBIN --homedir $GNUPGHOME --with-fingerprint
--with-colons --list-key | grep ^fpr` is identical for both (GNUPGBIN=gpg,
GNUPGHOME=/run/shm/gnupg1) and (GNUPGBIN=gpg2, GNUPGHOME=/run/shm/gnupg2).

Also, the keystores were created with

$ mkdir -m0700 /run/shm/gnupg{1,2}

$ echo 'trust-model always' | tee /run/shm/gnupg1/gpg.conf

/run/shm/gnupg2/gpg.conf

  $ echo no-autostart >>/run/shm/gnupg2/gpg.conf
  $ gpg2 --export | gpg  --homedir /run/shm/gnupg1 --import
  $ gpg2 --export | gpg2 --homedir /run/shm/gnupg2 --import

(Each keystore currently contains 1291 public keys and 235302 signatures, 161385
of which from unknown signers.)

Actually 2.1.6 on the keybox does about 4x more read(2) than 1.4.19 or 2.1.6 on
the legacy format.
Here are some figures using the i7:

$ strace -co /tmp/sigs_gpg_gnupg1 gpg --homedir /run/shm/gnupg1 --with-colons

--list-sigs >/dev/null

$ strace -co /tmp/sigs_gpg2_gnupg1 gpg2 --homedir /run/shm/gnupg1

--with-colons --list-sigs >/dev/null

$ strace -co /tmp/sigs_gpg2_gnupg2 gpg2 --homedir /run/shm/gnupg2

--with-colons --list-sigs >/dev/null

  $ head -5 /tmp/sigs_*
  ==> /tmp/sigs_gpg_gnupg1 <==
  % time     seconds  usecs/call     calls    errors syscall
  ------ ----------- ----------- --------- --------- ----------------
   71.05    0.010748           0   7480899           read
   15.64    0.002366           0   2233564           getrusage
   12.94    0.001958           0   2233564           clock_gettime

  ==> /tmp/sigs_gpg2_gnupg1 <==
  % time     seconds  usecs/call     calls    errors syscall
  ------ ----------- ----------- --------- --------- ----------------
   99.92    0.010204           0   7480899           read
    0.04    0.000004           0      4608           write
    0.04    0.000004           0     20438           lseek

  ==> /tmp/sigs_gpg2_gnupg2 <==
  % time     seconds  usecs/call     calls    errors syscall
  ------ ----------- ----------- --------- --------- ----------------
   99.94    1.724001           0  29610764           read
    0.02    0.000386           0     15307         9 open
    0.02    0.000376           0     15290           munmap

Interestingly, with --list-keys 2.1.6 is actually faster on /run/shm/gnupg2:

  $ time gpg --homedir /run/shm/gnupg1 --with-colons --list-keys >/dev/null
  0:02.01 (1.99 user, 0.02 sys)  11004k maxres
  $ time gpg2 --homedir /run/shm/gnupg1 --with-colons --list-keys >/dev/null
  0:02.03 (2.01 user, 0.02 sys)  11456k maxres
  $ time gpg2 --homedir /run/shm/gnupg2 --with-colons --list-keys >/dev/null
  0:00.60 (0.58 user, 0.02 sys)  16568k maxres

It also does about 10x less read(2):

$ head -5 /tmp/keys*

==> /tmp/keys_gpg2_gnupg1 <==

% time     seconds  usecs/call     calls    errors syscall

------ ----------- ----------- --------- --------- ----------------

100.00    0.000004           0     21011           read

  0.00    0.000000           0       166           write

  0.00    0.000000           0        20         3 open



==> /tmp/keys_gpg2_gnupg2 <==

% time     seconds  usecs/call     calls    errors syscall

------ ----------- ----------- --------- --------- ----------------

  0.00    0.000000           0      2036           read

  0.00    0.000000           0       166           write

  0.00    0.000000           0        19         3 open

Sure it does syscalls because it reads the keybox file for each signature. The
order of the keys in the keybox can make a big difference when checksing
signatures. Arge the keys store in the same order? "gpg -k" walks the
keyring/keybox sequentially.

Thanks for trying to fix that bug, Werner! However, while your patch does
indeed make 2.1.6 quite a bit faster than 2.1.5, it's still much much slower
than the venarable 1.4.19. And since with a keybox 2.1.x spends most of its
time doing syscalls, I believe there is something odd in the way it reads
keyboxes hence I'm reopening the issue.

With an x60s (i686, 2x Core Duo L2400 @ 1.66GHz, 3GiB DDR2 PC2-5300):

  $ time gpg --homedir /run/shm/gnupg1 --list-sigs >/dev/null
  2:16.41 (91.60 user, 43.06 sys)  9444k maxres
  $ time gpg2 --homedir /run/shm/gnupg1 --list-sigs >/dev/null
  2:14.45 (97.77 user, 35.41 sys)  10364k maxres
  $ time gpg2 --homedir /run/shm/gnupg2 --list-sigs >/dev/null
  19:40.61 (80.14 user, 1088.58 sys)  15016k maxres

With a desktop (x86_64, 4x Core i7-4770S @ 3.1GHz, 16GiB DDR3 PC3-12800):

  $ time gpg --homedir /run/shm/gnupg1 --list-sigs >/dev/null
  0:30.74 (24.32 user, 6.40 sys)  11700k maxres
  $ time gpg2 --homedir /run/shm/gnupg1 --list-sigs >/dev/null
  0:26.34 (20.24 user, 6.08 sys)  13132k maxres
  $ time gpg2 --homedir /run/shm/gnupg2 --list-sigs >/dev/null
  2:23.97 (14.39 user, 129.52 sys)  18800k maxres

(On both machines, '/run/shm' is mounted in memory. '/run/shm/gnupg1' contains a
keyring in legacy format, while '/run/shm/gnupg2' contains the same keyring in
keybox format.)

So on the fast machine, --list-sigs is still 5x slower on a keybox. Note the
proportion of the execution spent in kernel mode: 93% on the L2400, 90% on the
i7! (Vs. 26% and 23% with a legacy .gpg format.) I have also a hunch that this
might be the reason why updating the trustdb takes that long with 2.1.x and a
keybox.

Simpler: The code does only handle one usage flag and ignores combinations.

I now keep a list of not found keyids in the keydb modules, thus all lookups by
keyid will benefit from it. It also works for keybox and keyring. My tests
shows a speedup from 28 minutes to 3 minutes. That test with 14000 not
available keys requires about 220 KiB exra RAM, which should not be a problem.
See commit 6500f33.

The attached callgraph for g10/gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID4

Of course it should be g10/gpg2 --homedir /tmp/gnupg2 --list-sigs $keyID4.
(/tmp/gnupg1 doesn't contain a keybox.) Sorry for the confusion.

The attached callgraph for g10/gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID4
(without the patch) shows that a significant amount of time is spent
fread(3)'ing each blob of the keybox (for every single sig encountered).

A caching mechanism (ideally two hashtables, the first mapping 64-bits key IDs
to their primary UID, the second keeping track of unknown signer's 64-bits key
IDs) would certainly speed up things, but as shown in T1938 (guilhem on Apr 01 2015, 04:12 PM / Roundup) read(2) causes
overhead on a keybox for *both* --list-keys and --list-sigs, so it wouldn't
solve the core of the issue. I would therefore agree with dkg's T1938 (dkg on Apr 01 2015, 09:40 PM / Roundup) here:
it's probably worth trying to mmap(3) the keybox instead of fread(3)'ing it.

Aren't the second tests (gpg2 --homedir /tmp/gnupg1 --list-sigs) precisely
what you want? This suggests that the bug is triggered by get_user_id, but
only when used on a keybox.

Using master (6619ead) without your patch yields timings comparable to 2.1.2
with both pubring.kbx and pubring.gpg. As I hinted at in the ExtLink, I also
noticed that adding --fast-list-mode makes --list-sigs as fast as --list-keys
(which is not surprising, since it removes the quadratic behavior). Since it
also doesn't show any significant difference between pubring.gpg and
pubring.kbx, it confirms that the bug is triggered by get_user_id when used on
a keybox (for some reason read(2) seems to be much much more expensive on a
keybox than on the legacy format).

I have no immediate idea why 1.4 is faster than 2.1. To sort out whether it is
a problem of the gnupg version or of the keyring format you may replace
pubring.kbx in gnupg2 by pubring.gpg from gnupg1 and check whether there is a
substantial difference between the evrsions using the same pubring.gpg.

Given the number of keys the default cache sizes should work.

I am not testing with 2.1.2 but with master which might have some updates. With
-fast-list-mode --lits-sigs is really fast.

Taking the primary user id from the keybox meta data is not a good idea because
it violates the layering.

Indeed, your patch speeds up the listing, but doesn't seem to fully solve the
regression and in particular doesn't make the keybox setting as fast as the
legacy pubring.gpg one.

Here is my test environement:

• fresh identical keyrings /tmp/gnupg1 and /tmp/gnupg2, the former with a

pubring.gpg, the latter with a pubring.kbx.

• 1246 public keys: `gpg2 --homedir /tmp/gnupg1 --with-colons --list-sigs |

grep -c '^pub:'`

• 228176 sigs: `gpg2 --homedir /tmp/gnupg1 --with-colons --list-sigs | grep -Ec

'^(sig|rev):'`

• 156625 sigs from an unknown signer: `gpg2 --homedir /tmp/gnupg1 --with-colons

--list-sigs | grep -Ec '^(sig|rev):([^:]*:){8}\[User ID not found\]:'`

• 13754 unknown signers: `gpg2 --homedir /tmp/gnupg1 --with-colons --list-sigs

found\]:.*/\3/p' | sort -u | wc -l`

• g10/gpg2 is the current HEAD (6619ead) with your patch on top; gpg2 is

2.1.2; gpg is 1.4.18.

• $keyID1: 216 sigs (98 unique signers), 178 from an unknown signer (87 unique

unknown signers)

`gpg --homedir /tmp/gnupg1 --list-sigs $keyID1`:      0:07.17 (5.24 user,

1.82 sys) 7956k maxres

`gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID1`:     0:07.62 (5.47 user,

1.53 sys) 8312k maxres

`gpg2 --homedir /tmp/gnupg2 --list-sigs $keyID1`:     0:58.27 (3.74 user,

51.69 sys) 11836k maxres

`g10/gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID1`: 0:06.71 (5.03 user,

1.58 sys) 8224k maxres

`g10/gpg2 --homedir /tmp/gnupg2 --list-sigs $keyID1`: 0:17.03 (1.50 user,

14.92 sys) 11812k maxres

• $keyID2: 1026 sigs (250 unique signers), 716 from an unknown signer (181

unique unknown signers)

`gpg --homedir /tmp/gnupg1 --list-sigs $keyID2`:      0:07.29 (5.33 user,

1.86 sys) 7940k maxres

`gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID2`:     0:07.53 (5.68 user,

1.64 sys) 8256k maxres

`gpg2 --homedir /tmp/gnupg2 --list-sigs $keyID2`:     1:01.09 (3.59 user,

53.18 sys) 11784k maxres

`g10/gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID2`: 0:06.66 (4.98 user,

1.55 sys) 8052k maxres

`g10/gpg2 --homedir /tmp/gnupg2 --list-sigs $keyID2`: 0:17.11 (1.50 user,

15.16 sys) 11812k maxres

• $keyID3: 1334 sigs (302 unique signers), 374 from an unknown signer (80 unique

unknown signers)

`gpg --homedir /tmp/gnupg1 --list-sigs $keyID3`:      0:18.31 (12.18 user,

5.71 sys) 8928k maxres

`gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID3`:     0:17.73 (12.80 user,

4.68 sys) 9380k maxres

`gpg2 --homedir /tmp/gnupg2 --list-sigs $keyID3`:     0:40.33 ( 3.70 user,

34.58 sys) 14576k maxres

`g10/gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID3`: 0:16.38 (11.03 user,

4.85 sys) 9280k maxres

`g10/gpg2 --homedir /tmp/gnupg2 --list-sigs $keyID3`: 0:17.83 ( 2.16 user,

13.79 sys) 14612k maxres

• $keyID4: 1439 sigs (643 unique signers), 1168 from an unknown signer (555

unique unknown signers)

`gpg --homedir /tmp/gnupg1 --list-sigs $keyID4`:      0:08.94 (6.64 user,

2.22 sys) 7952k maxres

`gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID4`:     0:09.61 (7.26 user,

1.93 sys) 8444k maxres

`gpg2 --homedir /tmp/gnupg2 --list-sigs $keyID4`:     1:34.20 (5.98 user,

85.03 sys) 11724k maxres

`g10/gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID4`: 0:08.50 (6.37 user,

1.94 sys) 8092k maxres

`g10/gpg2 --homedir /tmp/gnupg2 --list-sigs $keyID4`: 0:50.71 (3.20 user,

44.63 sys) 11568k maxres

Furthermore, the patch brings down --list-sigs on the entire keybox to 17min
from 2h25 (vs 2min on a pubring.gpg).

So all in a all, while this patch significantly improves the slowness of
--list-sigs with a keybox, it doesn't generally (except on $keyID3) make it as
fast as when used with the legacy pubring.gpg format.

Also, I'd like to point out that since the nested lookups in get_user_id are
only meant to retrieve the signer's primary UID, and as it seems to be part of
the keybox's metadata (if I got T1939 right, that's why kbxutil is lightning
fast), it should be possible avoid the parsing (which I guess is causing most of
the overhead) to get the UID right? Of course it can't hurt to have a caching
mechanism on top of that, though ;-)

D294: 597_0001-gpg-TEST-with-no-uid-cache.patch

Just to be clear (it isn't clear to me that these issues are the same),
issue1938 is related to --list-sigs being extremely slow and spending all its
time in kernel mode, while in T1939 I merely wish that doing a keyring dump
with --list-keys was as fast as using kbxutils.