In T1938#99890, @marcus wrote:

It's unclear from the discussion if this issue has been resolved.

Hi Neal,

Thanks for the patch, works great on the couple of keys I tried it on.
Unfortunately I'm unsure how to build OpenPGP keys with deliberately wrongly
ordered packets, so my tests are probably not exhaustive :-( But looking at
your code (from an outsider's perspective), I can't see how revocation
certificates etc would be handled differently from certificate signatures.

I found two issues though:

+ ndataa = pubkey_get_nsig (a->pubkey_algo);
+ ndatab = pubkey_get_nsig (a->pubkey_algo);

I guess it should be "b->pubkey_algo" on the second line.

Also, since the "check" command of the GnuPG prompt can modify the keyblock, it
should set "modify" accordingly:

-8<----------------------------------------------------------------------------------->8-
diff --git a/g10/keyedit.c b/g10/keyedit.c
index d7c2a4b..ede350a 100644

• a/g10/keyedit.c

+++ b/g10/keyedit.c
@@ -2190,8 +2190,9 @@ keyedit_menu (ctrl_t ctrl, const char *username, strlist_t
locusr,

         break;

       case cmdCHECK:

• check_all_keysigs (keyblock, count_selected_uids (keyblock),
• !strcmp (arg_string, "selfsig"));

+ if (check_all_keysigs (keyblock, count_selected_uids (keyblock),
+ !strcmp (arg_string, "selfsig")))
+ modified = 1;

         break;

       case

cmdSIGN:-8<----------------------------------------------------------------------------------->8-

I understand that by default only selfsigs are reordered for performance
reasons. May I suggest to also consider the key to sign with (for instance
specified with "--local-user")? This can be useful, otherwise in order to avoid
potential duplicates signers might have to type "check" before signing a key.

Also (repeating what we discussed about on IRC so it gets indexed on the web :-)
Due to the append-only nature of keyservers, an uploaded badly ordered key
can't be fixed on the keyserver. As a consequence, with the current algorithm
each refresh would undo fixing the packets' order and removing the duplicates.
Ideally keys would be reordered upon import, and the merge algorithm would avoid
duplicate (for instance it could assume the local copy to be properly ordered,
and not add a packet to the local copy if said packet was found elsewhere on the
keyblock).

I found the following comment above g10/keyedit.c' fix_key_signature_order
function, which is responsible for printing the "moving a key signature to the
correct place" lines upon --edit-key.

  /*
   * There are some keys out (due to a bug in gnupg), where the sequence
   * of the packets is wrong.  This function fixes that.
   * Returns: true if the keyblock has been fixed.
   *        
   * Note:  This function does not work if there is more than one user ID.
   */

When was the mentioned bug introduced and fixed? git blame tells me the comment
predates Thu Jun 5 07:14:21 2003 +0000, date when cvs2svn created branch
'GNUPG-1-9-BRANCH' (commit 72503314). But Lucas' key was generated much later,
on 2009-07-01.

Of the 100 keys with lowest MSD http://pgp.cs.uu.nl/doc/top_1000.html, 27 have
badly ordered packets; 26 of these have multiple UIDs; and only 20 of the 120
uids with a creation date have been generated before 2009-07-01.

Is there a technical reason which "This function does not work if there is more
than one user ID"? (I guess gpg could try to verify the sig against each
UID/UAT to find out which one it binds to.) Or is Luca's key (among many
others) broken from a WoT perspective due to the multiple UIDs?

In fact Luca' key can currently be found on the keyserver pool with badly
ordered packets (I can provide a copy if need be):

~$ gpg2 --homedir="$gnupghome" --keyserver-options import-minimal --keyserver

hkp://pool.sks-keyservers.net --recv-key "$key"

~$ gpg2 --homedir="$gnupghome" --with-colons --list-sigs "$key" | grep -E

'^(pub|sub|uid|sig:([^:]*:){3}(06EAA066E397832F|39278DA8109E6244)):'

pub:-:4096:1:06EAA066E397832F:1246459499:::-:::scESCA:::::::
uid:-::::1286747091::B41FA634ADD68A6717D380A790190CB3BC80005B::Luca Capello

<luca@pca.it>:::::::::

sig:::1:06EAA066E397832F:1286747091::::Luca Capello <luca@pca.it>:13x:::::8:
uid:-::::1286747538::3590ECEB44695F2B0D4E5B2E85EDBBF99C3A90C6::Luca Capello

<gismo@debian.org>:::::::::

sig:::1:06EAA066E397832F:1286747538::::Luca Capello <luca@pca.it>:13x:::::8:
uid:-::::1453646682::8523545E8C0C86F63F6FC3387DE2D188A55481AF::Luca Capello

<luca.capello@infomaniak.ch>:::::::::

sig:::1:06EAA066E397832F:1453646682::::Luca Capello <luca@pca.it>:13x:::::10:
uid:-::::1454107799::45C4E00E6D5D53EDE22B1CC8D2B44DCE3E3E93B5::Luca Capello

<luca.capello@infomaniak.com>:::::::::

  sig:::1:06EAA066E397832F:1454107799::::Luca Capello <luca@pca.it>:13x:::::10:
  sub:-:4096:1:90C02DEC2BB95F4B:1246460155::::::e::::::
  sig:::1:06EAA066E397832F:1246460155::::Luca Capello <luca@pca.it>:18x:::::8:
  sub:-:4096:1:D91D57A03BE9F36D:1246460943::::::esa::::::
  sig:::1:39278DA8109E6244:1360031056::::[User ID not found]:10x:::::10:
  sig:::1:06EAA066E397832F:1246459499::::Luca Capello <luca@pca.it>:13x:::::8:
  sig:::1:06EAA066E397832F:1246460297::::Luca Capello <luca@pca.it>:13x:::::8:
  sig:::1:06EAA066E397832F:1286747091::::Luca Capello <luca@pca.it>:13x:::::8:
  sig:::1:06EAA066E397832F:1246460943::::Luca Capello <luca@pca.it>:18x:::::8:

Is there any reason why --import/--recv-key didn't move the packets to their
proper place? After all the keyring is then open in write mode.

Moreover while --edit attempts to reorder the packets, it places the signature
packets under the wrong UID:

~$ gpg2 --homedir="$gnupghome" --edit "$key" save
[…]
gpg: moving a key signature to the correct place
~$ gpg2 --homedir="$gnupghome" --with-colons --list-sigs "$key" | grep -E

'^(pub|sub|uid|sig:([^:]*:){3}(06EAA066E397832F|39278DA8109E6244)):'

pub:-:4096:1:06EAA066E397832F:1246459499:::-:::scESCA:::::::
uid:-::::1286747091::B41FA634ADD68A6717D380A790190CB3BC80005B::Luca Capello

<luca@pca.it>:::::::::

sig:::1:06EAA066E397832F:1286747091::::Luca Capello <luca@pca.it>:13x:::::8:
uid:-::::1286747538::3590ECEB44695F2B0D4E5B2E85EDBBF99C3A90C6::Luca Capello

<gismo@debian.org>:::::::::

sig:::1:06EAA066E397832F:1286747538::::Luca Capello <luca@pca.it>:13x:::::8:
uid:-::::1453646682::8523545E8C0C86F63F6FC3387DE2D188A55481AF::Luca Capello

<luca.capello@infomaniak.ch>:::::::::

sig:::1:06EAA066E397832F:1453646682::::Luca Capello <luca@pca.it>:13x:::::10:
uid:-::::1454107799::45C4E00E6D5D53EDE22B1CC8D2B44DCE3E3E93B5::Luca Capello

<luca.capello@infomaniak.com>:::::::::

  sig:::1:06EAA066E397832F:1454107799::::Luca Capello <luca@pca.it>:13x:::::10:
  sig:::1:06EAA066E397832F:1286747091::::Luca Capello <luca@pca.it>:13x:::::8:
  sig:::1:06EAA066E397832F:1246460297::::Luca Capello <luca@pca.it>:13x:::::8:
  sig:::1:06EAA066E397832F:1246459499::::Luca Capello <luca@pca.it>:13x:::::8:
  sig:::1:39278DA8109E6244:1360031056::::[User ID not found]:10x:::::10:
  sub:-:4096:1:90C02DEC2BB95F4B:1246460155::::::e::::::
  sig:::1:06EAA066E397832F:1246460155::::Luca Capello <luca@pca.it>:18x:::::8:
  sub:-:4096:1:D91D57A03BE9F36D:1246460943::::::esa::::::
  sig:::1:06EAA066E397832F:1246460943::::Luca Capello <luca@pca.it>:18x:::::8:

I (0x39278DA8109E6244) did *not* sign Luca's 4th UID. I'm unsure (based on
--list-packets' output) which of the 2 first UIDs my signature applies to, but
certainly not to the last two, which were created 3 years after my sig was
issued.

In fact this is reproducible with Luca's key (but strangely not with mine):

~$ gpg2 --version
gpg (GnuPG) 2.1.11
~$ gnupghome=$(mktemp -d)
~$ key=0x06EAA066E397832F
~$ gpg2 --homedir="$gnupghome" --keyserver hkp://pool.sks-keyservers.net

--recv-key "$key"

~$ gpg2 --homedir="$gnupghome" --edit-key "$key" minimize 4 deluid save
~$ gpg2 --homedir="$gnupghome" --keyserver hkp://pool.sks-keyservers.net

--recv-key "$key"

  ~$ gpg2 --homedir="$gnupghome" --with-colons --list-sigs "$key"

The last command shows a lot of signatures under the last subkey. This not only
messes up the parsing, but also confuses GnuPG: for instance it refuses to let
me sign the 4th UID because it thinks I already did.

I confirm that I'm not able to reproduce T2176 (guilhem on Dec 11 2015, 02:21 PM / Roundup) nor T2176 (guilhem on Dec 11 2015, 01:07 PM / Roundup) with 4ffe44c, so
I'm changing the issue to “resolved”. Thanks for the prompt action!

I was able to reproduce by creating two keys with the default settings, adding a
signing subkey to the second one and then removing the private part of the
second master key.

Starting with two news keys in a fresh GNUPGHOME:

  ~$ mkdir -m0700 /run/shm/gnupg
  ~$ gpg2 --homedir /run/shm/gnupg --gen-key
  […]
  pub   rsa2048/86874ACC 2015-12-11
        Key fingerprint = 769C 9102 D5C2 2C35 9521  B4C9 C386 39F6 8687 4ACC
  uid         [ultimate] This is the 1st key
  sub   rsa2048/1B30F327 2015-12-11
  ~$ gpg2 --homedir /run/shm/gnupg --gen-key
  […]
  pub   rsa2048/FB0BE35A 2015-12-11
        Key fingerprint = B340 0EFB 296D EA60 6A86  D273 25C8 9A38 FB0B E35A
  uid         [ultimate] This is the 2nd key
  sub   rsa2048/3F1ADEB2 2015-12-11

I add a signing subkey to the second one

  ~$ gpg2 --homedir /run/shm/gnupg --expert --edit-key FB0BE35A addkey
  […]
  Possible actions for a RSA key: Sign Encrypt Authenticate
  Current allowed actions: Sign 
  […]
  gpg> save

Then I remove the private of the second master key

  ~$ gpg2 --homedir /run/shm/gnupg --with-keygrip --list-key FB0BE35A
  pub   rsa2048/FB0BE35A 2015-12-11
        Keygrip = 2B862E8DF6928622DD65D1BCDD5BF0F73FAE19CC
  uid         [ultimate] This is the 2nd key
  sub   rsa2048/3F1ADEB2 2015-12-11
        Keygrip = C4119344DBB7F874BDBC3FF706DB3D53FA8894C5
  sub   rsa2048/F9F86760 2015-12-11
        Keygrip = 30376FE31C5B6211728F8E2A4F595AF5D0F9F578
  ~$ gpg2 --homedir /run/shm/gnupg -K FB0BE35A
  sec#  rsa2048/FB0BE35A 2015-12-11
  uid         [ultimate] This is the 2nd key
  ssb   rsa2048/3F1ADEB2 2015-12-11
  ssb   rsa2048/F9F86760 2015-12-11

I want to signing something with the second key (using its signing subkey),
which I select with --default-key. But the first key is chosen instead:

~$ gpg2 --homedir /run/shm/gnupg --default-key FB0BE35A -a --sign </dev/null |

gpg --list-packets | grep keyid

  :onepass_sig packet: keyid C38639F686874ACC
  :signature packet: algo 1, keyid C38639F686874ACC

(Using --default-key 'F9F86760! doesn't help.)

The problem is that FPR2 actually appears in your keyring twice! Are you use

a keyring or a keybox?

Thank you for looking into this, Neal. I'm using a keybox, but I don't have the
same output:

~$ FPR2=7420DF86BCE15A458DCE997639278DA8109E6244
~$ gpg2 --debug=lookup -k $FPR2
gpg: reading options from '/home/guilhem/.gnupg/gpg.conf'
gpg: enabled debug flags: lookup
gpg: DBG: keydb_search: 1 search descriptions:
gpg: DBG: keydb_search   0: FPR20: '7420 DF86 BCE1 5A45 8DCE  9976 3927 8DA8

7420 DF8'

gpg: DBG: keydb_search: searching keybox (resource 0 of 1)
gpg: DBG: keydb_search: searched keybox (resource 0 of 1) => Success
gpg: DBG: finish_lookup: checking key 109E6244 (all)(req_usage=0)
gpg: DBG: 	using key 109E6244
gpg: DBG: keydb_search: 1 search descriptions:
gpg: DBG: keydb_search   0: FPR20: '7420 DF86 BCE1 5A45 8DCE  9976 3927 8DA8

7420 DF8'

  gpg: DBG: keydb_search: searching keybox (resource 0 of 1)
  gpg: DBG: keydb_search: searched keybox (resource 0 of 1) => EOF
  gpg: secmem usage: 0/65536 bytes in 0 blocks
  pub   rsa4096/0x39278DA8109E6244 2012-11-05 [expires: 2017-12-02]
  uid                   [ unknown] Guilhem Moulin <guilhem@fripost.org>
  uid                   [ unknown] Guilhem Moulin <guilhem.moulin@chalmers.se>
  uid                   [ unknown] Guilhem Moulin <guilhem.moulin@fripost.org>
  uid                   [ unknown] Guilhem Moulin <guilhem.moulin@ens-lyon.org>
  uid                   [ unknown] Guilhem Moulin <guilhem@guilhem.org>
  sub   rsa4096/0xD39A499C3C21A552 2012-11-05 [expires: 2016-10-31]
  sub   rsa4096/0xC27306B86774D6F7 2012-11-05 [expires: 2016-10-31]
  sub   rsa4096/0x73FB3BE5DB686A30 2012-11-05 [expires: 2016-10-31]

I've also tried to replicate the behavior in a clean GNUPGHOME:

  ~$ mkdir -m0700 /run/shm/gnupg
  ~$ ln -s ~/.gnupg/S.gpg-agent /run/shm/gnupg
  ~$ gpg2 --export $FPR1 | gpg2 --homedir /run/shm/gnupg --import
  ~$ gpg2 --export $FPR2 | gpg2 --homedir /run/shm/gnupg --import

(So /run/shm/gnupg contains a keybox, with $FPR1 and $FPR2 in that order.) The
following ouput should rule out a problem with a not properly merged key

~$ gpg2 --homedir /run/shm/gnupg --export | gpg --list-packets | grep -c

'^:public key'

  2
  ~$ kbxutil /run/shm/gnupg/pubring.kbx | grep '^Key-Fpr\[0\]:'
  Key-Fpr[0]: $FPR1
  Key-Fpr[0]: 7420DF86BCE15A458DCE997639278DA8109E6244

  ~$ gpg2 --homedir /run/shm/gnupg -a --sign </dev/null
  # selects $FPR1 (expected)

  ~$ gpg2 --homedir /run/shm/gnupg --default-key $FPR2 -a --sign </dev/null
  # selects $FPR1 (unexpected)

  ~$ gpg2 --homedir /run/shm/gnupg --local-user $FPR2 -a --sign </dev/null
  # selects $FPR2 (expected)

  ~$ echo default-key $FPR2 >/run/shm/gnupg/gpg.conf
  ~$ gpg2 --homedir /run/shm/gnupg -a --sign </dev/null
  # selects $FPR1 (unexpected)

  ~$ echo local-user $FPR2 >/run/shm/gnupg/gpg.conf
  ~$ gpg2 --homedir /run/shm/gnupg -a --sign </dev/null
  gpg: key specification '7420DF86BCE15A458DCE997639278DA8109E6244' is ambiguous
  gpg: (check argument of option '--local-user')
  gpg: '7420DF86BCE15A458DCE997639278DA8109E6244' matches at least:
  gpg:   7420DF86BCE15A458DCE997639278DA8109E6244
  gpg:   7420DF86BCE15A458DCE997639278DA8109E6244

In the meantime, a workaround is to add an udev rule to scdaemon whenever the
Yubikey is plugged in:
https://github.com/Yubico/yubikey-personalization/issues/36 . I guess it doesn't
work on Windows, though.

Yes they are, the output of `$GNUPGBIN --homedir $GNUPGHOME --with-fingerprint
--with-colons --list-key | grep ^fpr` is identical for both (GNUPGBIN=gpg,
GNUPGHOME=/run/shm/gnupg1) and (GNUPGBIN=gpg2, GNUPGHOME=/run/shm/gnupg2).

Also, the keystores were created with

$ mkdir -m0700 /run/shm/gnupg{1,2}

$ echo 'trust-model always' | tee /run/shm/gnupg1/gpg.conf

/run/shm/gnupg2/gpg.conf

  $ echo no-autostart >>/run/shm/gnupg2/gpg.conf
  $ gpg2 --export | gpg  --homedir /run/shm/gnupg1 --import
  $ gpg2 --export | gpg2 --homedir /run/shm/gnupg2 --import

(Each keystore currently contains 1291 public keys and 235302 signatures, 161385
of which from unknown signers.)

Actually 2.1.6 on the keybox does about 4x more read(2) than 1.4.19 or 2.1.6 on
the legacy format.
Here are some figures using the i7:

$ strace -co /tmp/sigs_gpg_gnupg1 gpg --homedir /run/shm/gnupg1 --with-colons

--list-sigs >/dev/null

$ strace -co /tmp/sigs_gpg2_gnupg1 gpg2 --homedir /run/shm/gnupg1

--with-colons --list-sigs >/dev/null

$ strace -co /tmp/sigs_gpg2_gnupg2 gpg2 --homedir /run/shm/gnupg2

--with-colons --list-sigs >/dev/null

  $ head -5 /tmp/sigs_*
  ==> /tmp/sigs_gpg_gnupg1 <==
  % time     seconds  usecs/call     calls    errors syscall
  ------ ----------- ----------- --------- --------- ----------------
   71.05    0.010748           0   7480899           read
   15.64    0.002366           0   2233564           getrusage
   12.94    0.001958           0   2233564           clock_gettime

  ==> /tmp/sigs_gpg2_gnupg1 <==
  % time     seconds  usecs/call     calls    errors syscall
  ------ ----------- ----------- --------- --------- ----------------
   99.92    0.010204           0   7480899           read
    0.04    0.000004           0      4608           write
    0.04    0.000004           0     20438           lseek

  ==> /tmp/sigs_gpg2_gnupg2 <==
  % time     seconds  usecs/call     calls    errors syscall
  ------ ----------- ----------- --------- --------- ----------------
   99.94    1.724001           0  29610764           read
    0.02    0.000386           0     15307         9 open
    0.02    0.000376           0     15290           munmap

Interestingly, with --list-keys 2.1.6 is actually faster on /run/shm/gnupg2:

  $ time gpg --homedir /run/shm/gnupg1 --with-colons --list-keys >/dev/null
  0:02.01 (1.99 user, 0.02 sys)  11004k maxres
  $ time gpg2 --homedir /run/shm/gnupg1 --with-colons --list-keys >/dev/null
  0:02.03 (2.01 user, 0.02 sys)  11456k maxres
  $ time gpg2 --homedir /run/shm/gnupg2 --with-colons --list-keys >/dev/null
  0:00.60 (0.58 user, 0.02 sys)  16568k maxres

It also does about 10x less read(2):

$ head -5 /tmp/keys*

==> /tmp/keys_gpg2_gnupg1 <==

% time     seconds  usecs/call     calls    errors syscall

------ ----------- ----------- --------- --------- ----------------

100.00    0.000004           0     21011           read

  0.00    0.000000           0       166           write

  0.00    0.000000           0        20         3 open



==> /tmp/keys_gpg2_gnupg2 <==

% time     seconds  usecs/call     calls    errors syscall

------ ----------- ----------- --------- --------- ----------------

  0.00    0.000000           0      2036           read

  0.00    0.000000           0       166           write

  0.00    0.000000           0        19         3 open

Thanks for trying to fix that bug, Werner! However, while your patch does
indeed make 2.1.6 quite a bit faster than 2.1.5, it's still much much slower
than the venarable 1.4.19. And since with a keybox 2.1.x spends most of its
time doing syscalls, I believe there is something odd in the way it reads
keyboxes hence I'm reopening the issue.

With an x60s (i686, 2x Core Duo L2400 @ 1.66GHz, 3GiB DDR2 PC2-5300):

  $ time gpg --homedir /run/shm/gnupg1 --list-sigs >/dev/null
  2:16.41 (91.60 user, 43.06 sys)  9444k maxres
  $ time gpg2 --homedir /run/shm/gnupg1 --list-sigs >/dev/null
  2:14.45 (97.77 user, 35.41 sys)  10364k maxres
  $ time gpg2 --homedir /run/shm/gnupg2 --list-sigs >/dev/null
  19:40.61 (80.14 user, 1088.58 sys)  15016k maxres

With a desktop (x86_64, 4x Core i7-4770S @ 3.1GHz, 16GiB DDR3 PC3-12800):

  $ time gpg --homedir /run/shm/gnupg1 --list-sigs >/dev/null
  0:30.74 (24.32 user, 6.40 sys)  11700k maxres
  $ time gpg2 --homedir /run/shm/gnupg1 --list-sigs >/dev/null
  0:26.34 (20.24 user, 6.08 sys)  13132k maxres
  $ time gpg2 --homedir /run/shm/gnupg2 --list-sigs >/dev/null
  2:23.97 (14.39 user, 129.52 sys)  18800k maxres

(On both machines, '/run/shm' is mounted in memory. '/run/shm/gnupg1' contains a
keyring in legacy format, while '/run/shm/gnupg2' contains the same keyring in
keybox format.)

So on the fast machine, --list-sigs is still 5x slower on a keybox. Note the
proportion of the execution spent in kernel mode: 93% on the L2400, 90% on the
i7! (Vs. 26% and 23% with a legacy .gpg format.) I have also a hunch that this
might be the reason why updating the trustdb takes that long with 2.1.x and a
keybox.

The attached callgraph for g10/gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID4

Of course it should be g10/gpg2 --homedir /tmp/gnupg2 --list-sigs $keyID4.
(/tmp/gnupg1 doesn't contain a keybox.) Sorry for the confusion.

The attached callgraph for g10/gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID4
(without the patch) shows that a significant amount of time is spent
fread(3)'ing each blob of the keybox (for every single sig encountered).

A caching mechanism (ideally two hashtables, the first mapping 64-bits key IDs
to their primary UID, the second keeping track of unknown signer's 64-bits key
IDs) would certainly speed up things, but as shown in T1938 (guilhem on Apr 01 2015, 04:12 PM / Roundup) read(2) causes
overhead on a keybox for *both* --list-keys and --list-sigs, so it wouldn't
solve the core of the issue. I would therefore agree with dkg's T1938 (dkg on Apr 01 2015, 09:40 PM / Roundup) here:
it's probably worth trying to mmap(3) the keybox instead of fread(3)'ing it.

Aren't the second tests (gpg2 --homedir /tmp/gnupg1 --list-sigs) precisely
what you want? This suggests that the bug is triggered by get_user_id, but
only when used on a keybox.

Using master (6619ead) without your patch yields timings comparable to 2.1.2
with both pubring.kbx and pubring.gpg. As I hinted at in the ExtLink, I also
noticed that adding --fast-list-mode makes --list-sigs as fast as --list-keys
(which is not surprising, since it removes the quadratic behavior). Since it
also doesn't show any significant difference between pubring.gpg and
pubring.kbx, it confirms that the bug is triggered by get_user_id when used on
a keybox (for some reason read(2) seems to be much much more expensive on a
keybox than on the legacy format).

Indeed, your patch speeds up the listing, but doesn't seem to fully solve the
regression and in particular doesn't make the keybox setting as fast as the
legacy pubring.gpg one.

Here is my test environement:

• fresh identical keyrings /tmp/gnupg1 and /tmp/gnupg2, the former with a

pubring.gpg, the latter with a pubring.kbx.

• 1246 public keys: `gpg2 --homedir /tmp/gnupg1 --with-colons --list-sigs |

grep -c '^pub:'`

• 228176 sigs: `gpg2 --homedir /tmp/gnupg1 --with-colons --list-sigs | grep -Ec

'^(sig|rev):'`

• 156625 sigs from an unknown signer: `gpg2 --homedir /tmp/gnupg1 --with-colons

--list-sigs | grep -Ec '^(sig|rev):([^:]*:){8}\[User ID not found\]:'`

• 13754 unknown signers: `gpg2 --homedir /tmp/gnupg1 --with-colons --list-sigs

found\]:.*/\3/p' | sort -u | wc -l`

• g10/gpg2 is the current HEAD (6619ead) with your patch on top; gpg2 is

2.1.2; gpg is 1.4.18.

• $keyID1: 216 sigs (98 unique signers), 178 from an unknown signer (87 unique

unknown signers)

`gpg --homedir /tmp/gnupg1 --list-sigs $keyID1`:      0:07.17 (5.24 user,

1.82 sys) 7956k maxres

`gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID1`:     0:07.62 (5.47 user,

1.53 sys) 8312k maxres

`gpg2 --homedir /tmp/gnupg2 --list-sigs $keyID1`:     0:58.27 (3.74 user,

51.69 sys) 11836k maxres

`g10/gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID1`: 0:06.71 (5.03 user,

1.58 sys) 8224k maxres

`g10/gpg2 --homedir /tmp/gnupg2 --list-sigs $keyID1`: 0:17.03 (1.50 user,

14.92 sys) 11812k maxres

• $keyID2: 1026 sigs (250 unique signers), 716 from an unknown signer (181

unique unknown signers)

`gpg --homedir /tmp/gnupg1 --list-sigs $keyID2`:      0:07.29 (5.33 user,

1.86 sys) 7940k maxres

`gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID2`:     0:07.53 (5.68 user,

1.64 sys) 8256k maxres

`gpg2 --homedir /tmp/gnupg2 --list-sigs $keyID2`:     1:01.09 (3.59 user,

53.18 sys) 11784k maxres

`g10/gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID2`: 0:06.66 (4.98 user,

1.55 sys) 8052k maxres

`g10/gpg2 --homedir /tmp/gnupg2 --list-sigs $keyID2`: 0:17.11 (1.50 user,

15.16 sys) 11812k maxres

• $keyID3: 1334 sigs (302 unique signers), 374 from an unknown signer (80 unique

unknown signers)

`gpg --homedir /tmp/gnupg1 --list-sigs $keyID3`:      0:18.31 (12.18 user,

5.71 sys) 8928k maxres

`gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID3`:     0:17.73 (12.80 user,

4.68 sys) 9380k maxres

`gpg2 --homedir /tmp/gnupg2 --list-sigs $keyID3`:     0:40.33 ( 3.70 user,

34.58 sys) 14576k maxres

`g10/gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID3`: 0:16.38 (11.03 user,

4.85 sys) 9280k maxres

`g10/gpg2 --homedir /tmp/gnupg2 --list-sigs $keyID3`: 0:17.83 ( 2.16 user,

13.79 sys) 14612k maxres

• $keyID4: 1439 sigs (643 unique signers), 1168 from an unknown signer (555

unique unknown signers)

`gpg --homedir /tmp/gnupg1 --list-sigs $keyID4`:      0:08.94 (6.64 user,

2.22 sys) 7952k maxres

`gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID4`:     0:09.61 (7.26 user,

1.93 sys) 8444k maxres

`gpg2 --homedir /tmp/gnupg2 --list-sigs $keyID4`:     1:34.20 (5.98 user,

85.03 sys) 11724k maxres

`g10/gpg2 --homedir /tmp/gnupg1 --list-sigs $keyID4`: 0:08.50 (6.37 user,

1.94 sys) 8092k maxres

`g10/gpg2 --homedir /tmp/gnupg2 --list-sigs $keyID4`: 0:50.71 (3.20 user,

44.63 sys) 11568k maxres

Furthermore, the patch brings down --list-sigs on the entire keybox to 17min
from 2h25 (vs 2min on a pubring.gpg).

So all in a all, while this patch significantly improves the slowness of
--list-sigs with a keybox, it doesn't generally (except on $keyID3) make it as
fast as when used with the legacy pubring.gpg format.

Also, I'd like to point out that since the nested lookups in get_user_id are
only meant to retrieve the signer's primary UID, and as it seems to be part of
the keybox's metadata (if I got T1939 right, that's why kbxutil is lightning
fast), it should be possible avoid the parsing (which I guess is causing most of
the overhead) to get the UID right? Of course it can't hurt to have a caching
mechanism on top of that, though ;-)

Done in c238340:

http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=commit;h=c2383407bba5eefea486464a31e02846124c9da5

Just to be clear (it isn't clear to me that these issues are the same),
issue1938 is related to --list-sigs being extremely slow and spending all its
time in kernel mode, while in T1939 I merely wish that doing a keyring dump
with --list-keys was as fast as using kbxutils.

I created (1938) a new issue for the extreme slowness of --list-sigs on a
keybox. 1938 is most likely a bug, while 1710 is merely a quickfix for an
algorithmic issue in --list-sigs. However if with keybox “random access to the
keys is now really fast”, maybe it a proper fix could easily be implemented
instead. See also

http://lists.gnupg.org/pipermail/gnupg-devel/2015-February/029541.html

With slightly over 1000 keys in my keyring, it is surprisingly slow to list all
865 sigs on my key:

    $ time gpg --list-sigs --with-colon $keyID >/dev/null
    real    0m22.061s

At first I naively thought the major bottleneck was caused by the trust
calculation, but changing the trust model doesn't seem to help:

    $ time gpg --trust-model=always --list-sigs --with-colon $keyID >/dev/null
    real    0m22.157s

While --fast-list-mode downs the execution time by a factor of 100 (0m0.220s),
the key UIDS are not displayed (which makes it impossible to know on which of
them the signature was added). I didn't benchmark the difference myself but I
don't expect it to be significant, as if I got RFC 4880 right, the UID packets
are being looped over when gpg inspects the signature packets. (OTOH the
primary UID of the signing keys are not printed either when --fast-list-mode is
set, which I can understand as doing so would require costly lookups through
the keyring.)

I wish the user had more fine-grain control on what to skip with
--fast-list-mode.

Thanks!

Guilhem.