While I understand incorrectness, the risk in practice is not that high. So, I put this as "normal" priority.

In the current implementation of GnuPG, multiple packets of Symmetric-Key Encrypted Session Key Packet are not handled very well.

Pushed the change to master as well as 2.2 branch.

I think dropping import-clean from the default keyserver options is the right way to go. It is not clear what additional benefit import-clean provides given that we are already using self-sigs-only. And the idea of non-additive behavior to the local keyring when pulling from a keyserver is a deeply surprising change for multiple users i've talked to.

The fact that import-clean modifies already-held certifications makes me think it is inappropriate to have as the default for keyserver access (see T4628 for more details).

I am proposing to backport rG33c17a8008c3ba3bb740069f9f97c7467f156b54 and rGa7a043e82555a9da984c6fb01bfec4990d904690 to STABLE-BRANCH-2-2 as they represent a significant performance improvement in several specific use cases and appear to have no downsides.

If you're on a platform that has awk available (any GNU/Linux and macOS should provide it), you can scan for the largest OpenPGP certificate in your keyring with an awk script i posted over at https://dev.gnupg.org/T3972#127356

How to find out which keys are affected?

You need to delete the flooded keys to make things go faster.

After waiting for far over an hour, Kleopatra read the keys. Now, things go faster (also in LibreOffice), but it still takes around 30 seconds, which is quite long.

gpg4win 3.1.10 did not fix this issue for me, neither in Kleopatra nor in LibreOffice.

The card frame works received a lot of changes in master but we won't backport it to 2.2. Sorry.

@gniibe, the documentation (at least on the stable branch) says that --fast-import is just a synonym for --import. is that incorrect?

This is fixed.

This should be fixed.

About importing, there are two other works: repairing and trustdb update. We can figure out the difference by the --import-options of no-repair-keys and fast-import (to skip those works).
I think that both can be O(N^2) for number of signatures.

A linked list of 100000 items is not a usable data structure. The problem however is not the linked list but the DoS due to the number of signatures being well beyond the design limit. 1000 key signatures is already a large number and only few people have them. We need to put a limit on them.

with @gniibe's patches applied, i profiled the --import, since that is where the largest CPU cost remains. I tried two different times:

I disabled the dependency rules for the figures (it's only enabled for maintainers).

@gniibe: We move this issue over to mail. I'll forward it to you.

Okay, for 100000 signature this is clearly a win if no key lookup is needed.

Fixed.

i also checked the CPU time for git tag -v, whether @gniibe's patches were applied or not.

fwiw, i tried gpg --import on the ascii-armored version of my C4BC2DDB38CCE96485EBE9C2F20691179038E5C6 OpenPGP certificate (22895014 octets, 54614 certifications), followed by gpg --list-keys and gpg --export | wc. I was comparing 2.2.17-1 (from the debian package in unstable) with the exact same source, just with @gniibe's two patches rG33c17a8008c3 and rGa7a043e82555 applied as well. I did this with GNUPGHOME set to an otherwise empty directory, where i had done touch pubring.gpg to avoid the keybox format. (the two runs did not share a GNUPGHOME).

If I were testing more, I would generate many (say, 1000, or more, for example) encrypted message by the tool (IBM Encryption Facility), to examine by GnuPG and figure out some patterns of failure.

While I only observed the output of --list-packet, what I see are:

With NTBTLS, it seems it works correctly.

Which SSH client are you using?

For the particular problem of --list-key with pubring.gpg, I think we can say it's fixed.

@werner : Yes, the way to go is having something like a server for keys; It can remove all unnecessary search/lookup all together.

Check out the mailing list gcrypt-devel@

Folks, I was just wondering if I could get an update on where we are with this bug. It seems we aren't sure if it's a real issue or not. What's the latest thought?

(i think that rG33c17a8008c3ba3bb740069f9f97c7467f156b54 is also relevant, though it was not tagged with this ticket)

@gniibe -- thank you very much for tracking down these O(N^2) operations and cleaning them up. I will profile the effect of those changes and report my findings.

Hi Maximilian,

Hi, @JW-D, as the 'fixed' version of mailvelope has been released, could you please confirm if the issue is solved for you with mailvelope 3.3.1, or if you're still affected? Thank you.

@gniibe: I doubt that your fix really makes a difference. The majority of time is spend on searching the keyring for keys. This is why I have the gpgk thing in the works.

You probably have one of the spammed keys in your keyring. This is a problem with the keyserver networks. Do not use --auto-key-retrieve and avoid using the keyservers until we provide a mitigation with the next gpg4win/gnupg release. See also T4591

yes, python2.7 and python3.7

Using several python versions?

rM7d0a979c07d2 disabled the test for this. @werner says:

Because this is a GPGME bug.

why is this fix not relevant for the 2.2 stable branch? I've had no feedback on this proposed patch.

Quiet tricky to get right; needs some rework.

I closed this as a duplicate of a newer task because I did not find this issue when creating T4561 and there was already work done for T4561.

Not every incoming certificate that has no user ID will lack a user ID once it is merged with the local copy of the same certificate. T4393 describes that use case, so if you're interested in receiving user-ID-lacking updates to certificates that you already have a copy of, @jaymzh, you should follow up on that ticket.

Once a revocation is added (to any part of the certificate), perhaps all the certification packets that are clearly made obsolete by the revocation could be dropped from the certificate? That would certainly free up space to be able to import additional revocations if needed.

Given the recent problems with the keyservers, I expect that the keyserver feature will go away anyway and thus I do not think we will put any more effort into this. Thus I re-tag this as gpg 2.3.

And of course, thanks for your fix.

Applied to both branches. I have run no tests myself, though.

Fix will be in 2.2.17

Fix will be in 2.2.17.
See T4612 for the revocation case.

Re 1.: I don't view this as a bug. gpg prints stats on what it has been done and clearly it has processed a key. If it would have imported the key you would see another stat line telling about this. There was however a bug in the stats output which has been fixed.

Because we use dot-locking in GnuPG and copy-update-write for keyrings. Granted: For gpgv this is not required but the code is identical to the gpg code and adding new code does not make much sense. After all gpgv is a stripped down version of gpg I once wrote for Debian. I see your use case but tehre are other ways to do this and thus anthing here has low priority.

Aha, thank you. Sorry I saw the original post about the flood attacks (https://gist.github.com/rjhansen/67ab921ffb4084c865b3618d6955275f) which said to change your keyserver and I did, but I hadn't realized there were such significant differences.

I think what you're missing is the keys.openpgp.org documentation which makes it clear that they will not distribute identity information (read: "User IDs") without an explicit confirmation by the operator of the e-mail address named in the User ID. They strip down the certificate pretty significantly before redistribution, especially if the e-mail address hasn't been confirmed directly with the operators of that server.

I know the keyservers have been under attack, I'm using 'keys.openpgp.org' which is supposed to be more resilient to these, as I understand it?

out of curiosity, why does gpgv need the name of the file?

In that case, you can treat this ticket as a bug in the documentation, which still needs to be resolved.

We need random access and the name of the file. Thus a file descriptor is not sufficient.

Okay, if an attacker exactly hist that limit your case is valid. I see no easy fix here, though. What we can do is what is done on Unix file systems to give average users a disk full erroreven if there a few percent of the disk is free; root can use that extra space then. Revocation certificates would be what root is on Unix file systems.

That was pretty easy to reproduce thanks to your still not working server.

I did some manual tests using netcat and KS_FETCH to test the redirection.

I think you're suggesting accepting *any* path if the hostname of the proposed redirection matches openpgpkey.example.org when querying the WKD direct URL for an @example.org address. That would also be a fine solution from my point of view.

I head the same idea when I read your configuration. Given that the advanced lookup was not reallydeployed (see T4590) I also expect that we will receive complains now that it works. Thus white listing any "openpgpkey." seems to me a reasonable easy solution.

Will be in 2.2.17

Oh dear, that happens if one is always on master. I simply forgot to cherry pick the change from master back in November.
Two commits, though.

my initial scenario is where an adversarial keystore floods a certificate right up to (but within) the 5MiB boundary, so that the user has stored it in the keyring already. Then, the user encounters the certificate again, with revocation attached.

@werner, thanks for the pointer to the report, that's certainly useful. And i'm happy that organizations like SektionEins are doing GnuPG audits and publishing their results regardless of who paid for them.