If we can assume ret_keyblock != NULL (it is, in current implementation), it can be as simple as:

diff --git a/g10/getkey.c b/g10/getkey.c
index 6802026f6..27bbd354c 100644
--- a/g10/getkey.c
+++ b/g10/getkey.c
@@ -1354,6 +1354,8 @@ get_best_pubkey_byname (ctrl_t ctrl, enum get_pubkey_modes mode,
   int is_mbox = is_valid_mailbox (name);
   int wkd_tried = 0;

Agreed.

It has now been more than a month since:

Thanks.

i've just pushed rGc4b9eba1d6a63b73238dcbb644b365dc53563f3d to the dkg-fix-T4682 branch resolve this.

Sounds interesting @stm! Are there technical documents or specifications I could read to dig into details?

@dkg First step toward the canonical OpenPGP certificate export: http://git.savannah.nongnu.org/cgit/libtmcg.git/commit/?id=75372cac01501ae427dec1ae18805449bf28d087

@wiktor-k Thanks for your interest.

@werner wrote:

Other tasks in master are right now more important.

Other tasks in master are right now more important. You need to wait a bit more.

So, what about this? If I recall correctly, we had agreed in the call to merge this patch, at least into master?

@gniibe, thank you for backporting this to STABLE-BRANCH-2-2!

@dkg You are right. The term "issuer" was too ambiguous.
I like your proposal and would try to implement it. However, "export" of dkg-keycheck and other programs from DKGPG are very limited.

@stm it kind of is a last-resort already, given that it's only in the event where the signature creation dates are equal, but sure, i wouldn't mind adjusting the proposal to say that (sigs) means "sort by date, then issuer, then binary content" -- but what do we think "sort by issuer" means?

does the removal of the gpg22 tag mean that it will not be possible to rely on colon-delimited output for the gpg 2.2 series?

I am proposing to backport rG33c17a8008c3ba3bb740069f9f97c7467f156b54 and rGa7a043e82555a9da984c6fb01bfec4990d904690 to STABLE-BRANCH-2-2 as they represent a significant performance improvement in several specific use cases and appear to have no downsides.

@gniibe, the documentation (at least on the stable branch) says that --fast-import is just a synonym for --import. is that incorrect?

About importing, there are two other works: repairing and trustdb update. We can figure out the difference by the --import-options of no-repair-keys and fast-import (to skip those works).
I think that both can be O(N^2) for number of signatures.

A linked list of 100000 items is not a usable data structure. The problem however is not the linked list but the DoS due to the number of signatures being well beyond the design limit. 1000 key signatures is already a large number and only few people have them. We need to put a limit on them.

with @gniibe's patches applied, i profiled the --import, since that is where the largest CPU cost remains. I tried two different times:

Okay, for 100000 signature this is clearly a win if no key lookup is needed.

i also checked the CPU time for git tag -v, whether @gniibe's patches were applied or not.

fwiw, i tried gpg --import on the ascii-armored version of my C4BC2DDB38CCE96485EBE9C2F20691179038E5C6 OpenPGP certificate (22895014 octets, 54614 certifications), followed by gpg --list-keys and gpg --export | wc. I was comparing 2.2.17-1 (from the debian package in unstable) with the exact same source, just with @gniibe's two patches rG33c17a8008c3 and rGa7a043e82555 applied as well. I did this with GNUPGHOME set to an otherwise empty directory, where i had done touch pubring.gpg to avoid the keybox format. (the two runs did not share a GNUPGHOME).

For the particular problem of --list-key with pubring.gpg, I think we can say it's fixed.

@werner : Yes, the way to go is having something like a server for keys; It can remove all unnecessary search/lookup all together.

(i think that rG33c17a8008c3ba3bb740069f9f97c7467f156b54 is also relevant, though it was not tagged with this ticket)

@gniibe -- thank you very much for tracking down these O(N^2) operations and cleaning them up. I will profile the effect of those changes and report my findings.

We as GPGTools would also like to see this addition being integrated into GnuPG, since we do plan to switch to keys.openpgp.org in the near future, as we have long been hoping for a key server with better performance and among other things email verification. Without this change, revocations would not work as expected in combination with hagrid however. Preferably of course in the 2.2.X branch.

@gniibe: I doubt that your fix really makes a difference. The majority of time is spend on searching the keyring for keys. This is why I have the gpgk thing in the works.

then they are sorted by their binary content.

and from my understanding they are sending the self-signatures anyway.

This is not just about keys.openpgp.org. It's about any keystore that implements user id redaction, for whatever reason. When you say "what they can do is accept only user ids which…" i think you mean "the userid-redacting keystores can instead redistribute user ids which …". Is that right?

Given the recent problems with the keyservers, I expect that the keyserver feature will go away anyway and thus I do not think we will put any more effort into this. Thus I re-tag this as gpg 2.3.

I tried to implement this but this is troublesome for other programs using the interface because a common patter is to use --search-keys to get a listing and then use --recv-key to import the keys - That won't work and will require changes to --recv-key too. Thus this change will not go into 2.2. Anyway, it is not dangerous to have --search-keys because the new default for import from keyservers will be to strip all key-signatures.

I'm also interested in fine details especially w.r.t. interfacing with GnuPG. I've seen multiple timestamping standards starting from RFC3161, to blockchains or secure time protocols even (ab)using Certificate Transparency logs and ideas on how to append the signature (timestamp flag vs unhashed notations) so I'll be eager to hear the details on the ML @stm!

Indeed we are in urgent need for a timestamping service. I was already pondering with the idea to integrate existing X.509 stamping services into OpenPGP signatures. Please write to gnupg-devel if you want to reach a wider audience. Unfortunately I need to abstain for getting involved in your project; there are too many other things to do.

Recently, I started a new project at savannah for developing free software and documentation in order to operate a Distributed OpenPGP Timestamping Service. Everyone is welcome to join.

Just importing a ~666KiB certificate when this monster certificate is in the keyring consumes over 10m of CPU time:

Verifying a git tag from the "clean" version of this certificate takes ~225ms of CPU time. Verifying the same git tag from a keyring that contains the flooded version of the certificate takes ~145s. This is factor of more than 600×. Any automated git tag verification system can probably be DoSed by this behavior.

I didn't mean to suggest that switching to sqlite was the only way to fix this, but if it is a promising way to fix it, that would be great. I'm sure there are other ways.

We know that. The problem is that we can't simply switch to sqlite for key storage because it is common that dozens of gpg processes are accessing the key data base. At least at some points we need proper transactional behaviour and Sqlite implements that by talking a temporary copy of the database - not an option for large keyrings.

FWIW, I disabled @aa7356 because he again started to troll.

Snap question regards to the clock;

I anyway plan to extend the --quick-gen-key parameters to allow the specification of several subkeys on the command line.

Users keep showing up in our support, confused by this inconsistency. This problem continues in 2020. What's holding this back?

I did lot of tests in the last weeks while working on gpg-card.

Well, it took long to fix. My original plan was to fix it while reworking getkey.c but that I have not yet come to work on that.

This is largely solved.

fwiw, a comment over on T4422 contains a bash script that tries to force GnuPG to do its certificate/signature re-ordering. this doesn't produce anything canonical yet, but it's the closest i've come so far to getting GnuPG to do something repeatable with a certificate after merging (but even that is not quite stable).

The issue for the quality indication is: T2103

In T4346#122371, @gouttegd wrote:

Regarding the quality evaluation, several months ago I proposed to optionally delegate that task to an external tool (specified by a new gpg-agent option passphrase-checker). I posted a first draft as D442 and then submitted a proper patchset to gnupg-devel, but although @werner expressed interest it was never merged. I have just checked that the patchset still applies cleanly to both the master branch and the STABLE-BRANCH-2-2. I can re-submit it to the mailing list if needed.

FWIW I like @gouttegd 's patchset.

In T4346#122098, @werner wrote:

The quality bar is switched off by default. That feature including the quality was ordered and accepted by a client. I don't like it either and thus the new default of having it disabled is a useful solution.

I reviewed the multibyte handling in GnuPG and you are right, there is a general problem because we use ReadConsoleA and basically GetCommandLineA, so there is no way for multibyte input unless a parameter file is used. Output is also broken, but that is easier to fix iff the input case has been fixed.

Changes backported to 2.2

Thanks for fixing that.

That's my badness. In wait_child_thread, assuan_release may cause thread context switch to agent_reset_scd which accesses scd_local_list; This access should be serialized.
And... in start_scd, calling unlock_scd should be after unlocking start_scd_lock.

Builds fine now with GCC 9. Thanks for looking into this so quickly.

Does not happen in 2.2. Additional requirement to test this bug in master: Another connection to the scdaemon must be open. For example running scute or, easier, call "gpg --card-edit" and keep it open.

Fixed in master, by removing use of compound literals. Compound literals are not portable feature (even for C99 code), so, it's good to avoid when we can.
Still dns.c uses C99 features of struct initializer with name.

Libdns is not our own code and our intention was to keep it in sync with upstream. However, after some initial success the upstream author lost interest. We now consider to rework the code to remove a bit of the more creative use of C99 and maybe even get rid of some of the used C99 features (gnupg is mainly C90 with some exceptions).

I can't tell whether this bug report is about all the ways that we wish that GnuPG's default password process was better, or whether it's about one specific change.

Regarding the quality evaluation, several months ago I proposed to optionally delegate that task to an external tool (specified by a new gpg-agent option passphrase-checker). I posted a first draft as D442 and then submitted a proper patchset to gnupg-devel, but although @werner expressed interest it was never merged. I have just checked that the patchset still applies cleanly to both the master branch and the STABLE-BRANCH-2-2. I can re-submit it to the mailing list if needed.

First of all I find PIN a very bad term. "Personal Identification Number" for example for my Gnuk token is confusing. I use a string there,... So let us use PIN only where it really has to be a number. Otherwise it is a Password.

Despite that I created this task, I am still not not convinced that removing the term passphrase is a good idea. If we do this in gnupg we would need to change all strings to make it clear that the passphrase is used to protect one's own key and has nothing to do with encryption etc. In fact the term PIN would be better because it is common knowledge that you use a PIN to get access to something you own. There would be less confusion on the purpose of the passphrase. Sure PIN is usually considered to be a number. However my bank allows a string to be used as, what they call, PIN.

There has been some progress here. At least we no longer use "passphrase" in new code. We still have not yet replaced all old occurances.