@werner wrote:

Other tasks in master are right now more important.

I've just pushed rE732855a483709345a5c0f49504f45cb8da3f883a to dkg-fix-T4643 in the gpg-error git repository. I don't know why it is not yet visible here.

CC_FOR_BUILD is defined in configure.ac as build system C compiler, not build system C compiler and flags.

I'm aware of you releasing an RC for comments, and i apologize for not catching this particular case earlier. As you know from T4607, i was even advocating for it. i didn't understand the full implications of the "import-then-clean" approach at the time, and was thinking it would only apply to the incoming material, not the stored material.

i've merged a variant of rGbe99eec2b105eb5f8e3759147ae351dcc40560ad into the GnuPG packaging in debian unstable as of version 2.2.17-3 to avoid the risks of data loss and signature verification failures. I'll revert it if i see the concern addressed upstream.

@gniibe, thank you for backporting this to STABLE-BRANCH-2-2!

I don't know why dkg-fix-T4641 is not showing up here on the assuan git repo.

I've just pushed rA45f01593d4ce794ae3562359aee2ff80c97e368e to the dkg-fix-T4641 branch that resolves this.

Thanks for the feedback. I'll go ahead and close any tickets that come in via debian that expect to be able to cross compile without having at least once had a native compiler on the platform to generate the appropriate lock-obj-pub-*.h.

@stm it kind of is a last-resort already, given that it's only in the event where the signature creation dates are equal, but sure, i wouldn't mind adjusting the proposal to say that (sigs) means "sort by date, then issuer, then binary content" -- but what do we think "sort by issuer" means?

does the removal of the gpg22 tag mean that it will not be possible to rely on colon-delimited output for the gpg 2.2 series?

Just a note that we're now shipping this patch in debian unstable. It would be great if it was merged upstream.

that pseudocode is strange to me -- it looks like you have (two) duplicate calls to clean_key (imported_keyblock) (though maybe i just don't know what .... means in this pseudocode).

I think dropping import-clean from the default keyserver options is the right way to go. It is not clear what additional benefit import-clean provides given that we are already using self-sigs-only. And the idea of non-additive behavior to the local keyring when pulling from a keyserver is a deeply surprising change for multiple users i've talked to.

The fact that import-clean modifies already-held certifications makes me think it is inappropriate to have as the default for keyserver access (see T4628 for more details).

Due to T4628, i no longer think that import-clean is a good idea by default.

I am proposing to backport rG33c17a8008c3ba3bb740069f9f97c7467f156b54 and rGa7a043e82555a9da984c6fb01bfec4990d904690 to STABLE-BRANCH-2-2 as they represent a significant performance improvement in several specific use cases and appear to have no downsides.

If you're on a platform that has awk available (any GNU/Linux and macOS should provide it), you can scan for the largest OpenPGP certificate in your keyring with an awk script i posted over at https://dev.gnupg.org/T3972#127356

@gniibe, the documentation (at least on the stable branch) says that --fast-import is just a synonym for --import. is that incorrect?

with @gniibe's patches applied, i profiled the --import, since that is where the largest CPU cost remains. I tried two different times:

i also checked the CPU time for git tag -v, whether @gniibe's patches were applied or not.

fwiw, i tried gpg --import on the ascii-armored version of my C4BC2DDB38CCE96485EBE9C2F20691179038E5C6 OpenPGP certificate (22895014 octets, 54614 certifications), followed by gpg --list-keys and gpg --export | wc. I was comparing 2.2.17-1 (from the debian package in unstable) with the exact same source, just with @gniibe's two patches rG33c17a8008c3 and rGa7a043e82555 applied as well. I did this with GNUPGHOME set to an otherwise empty directory, where i had done touch pubring.gpg to avoid the keybox format. (the two runs did not share a GNUPGHOME).

I agree, many currently-shipped DNS client library implementations do not provide DNSSEC validity checks.

(i think that rG33c17a8008c3ba3bb740069f9f97c7467f156b54 is also relevant, though it was not tagged with this ticket)

@gniibe -- thank you very much for tracking down these O(N^2) operations and cleaning them up. I will profile the effect of those changes and report my findings.

aiui, a keyserver scheme of https:// implies that the specific URL is to be queried directly, not using any of the HKPS URL path patterns.

yes, python2.7 and python3.7

rM7d0a979c07d2 disabled the test for this. @werner says:

This is especially relevant if you are not going to implement the fallback to import-clean that was proposed in T4591.

I see that you have lowered the WKD limit to 64KiB with 6396f8d115f21ae15571b683e9ac9d1d7e3f44f4 -- i think this is a mistake, as reasonable certificates can be several times that size (e.g. zack's cleaned certificate, mentioned above). I'd prefer a limit of 256KiB.

why is this fix not relevant for the 2.2 stable branch? I've had no feedback on this proposed patch.

This is not just about keys.openpgp.org. It's about any keystore that implements user id redaction, for whatever reason. When you say "what they can do is accept only user ids which…" i think you mean "the userid-redacting keystores can instead redistribute user ids which …". Is that right?

@werner, i don't think there is a 64K limit either, at least not in 2.2.16. Here is 2.2.16 with an empty homedir fetching Zack's certificate here which is > 97KiB:

Not every incoming certificate that has no user ID will lack a user ID once it is merged with the local copy of the same certificate. T4393 describes that use case, so if you're interested in receiving user-ID-lacking updates to certificates that you already have a copy of, @jaymzh, you should follow up on that ticket.

Once a revocation is added (to any part of the certificate), perhaps all the certification packets that are clearly made obsolete by the revocation could be dropped from the certificate? That would certainly free up space to be able to import additional revocations if needed.

I think what you're missing is the keys.openpgp.org documentation which makes it clear that they will not distribute identity information (read: "User IDs") without an explicit confirmation by the operator of the e-mail address named in the User ID. They strip down the certificate pretty significantly before redistribution, especially if the e-mail address hasn't been confirmed directly with the operators of that server.

out of curiosity, why does gpgv need the name of the file?

in 2.2.16, anyway, gnupg does not appear to apply import-minimal for WKD.

In that case, you can treat this ticket as a bug in the documentation, which still needs to be resolved.

auto-key-retrieve happens in the context of signature verification when the certificate is missing. If no signer User ID subpacket is present in the signature, then WKD simply won't work.

hm, i see your point. If you could spell out what the specific regression(s) in more detail, though, that might help us to reason about their impact.

if you want to add a separate subcommand for that, i would be happy to abandon migrate-pubring-from-classic-gpg.

I think you're suggesting accepting *any* path if the hostname of the proposed redirection matches openpgpkey.example.org when querying the WKD direct URL for an @example.org address. That would also be a fine solution from my point of view.

my initial scenario is where an adversarial keystore floods a certificate right up to (but within) the 5MiB boundary, so that the user has stored it in the keyring already. Then, the user encounters the certificate again, with revocation attached.

@werner, thanks for the pointer to the report, that's certainly useful. And i'm happy that organizations like SektionEins are doing GnuPG audits and publishing their results regardless of who paid for them.

@jaymzh, I agree with @werner that without more information, this bug report isn't particularly actionable.

as a separate variant: if the attacker floods the certificate with bogus self-signatures -- that is, certifications that have an issuer fingerprint or issuer key id subpacket, whether hashed or unhashed -- will that make it impossible to import any of them?

Thanks for working on this fallback, Werner.

Thanks for the pointer, @werner. Certainly we want T4590 fixed.

I should add that i don't really care whose fault it is if the software is broken by some downstream. if it harms any users, and we can fix it, we should fix it, especially if the fix is easy.

We're writing free software, which we know that people use and modify downstream. if we know that the software has a particular sharp edge that people who are modifying it are likely to cut themselves on, we have two options:

If the default keyserver is not hkps.pool.sks-keyservers.net, then @kristianf's CA certificate has no business certifying it.

So this is a defense against an adversary capable of creating a pinentry-wrapper somewhere in $PATH, but not capable of modifying gpg-agent.conf? It sounds to me like this is a defense against a very unusually-constrained attacker, at the expense of regular, common bug reports and user confusion.

thanks for working on this @werner. rG2e349bb61737 is definitely not useful for me. If i am going to tell anyone "hey, do this weird thing differently in order to fetch my key", i will tell them "pull it from https://dkg.fifthhorseman.net/dkg-openpgp.key". I will never tell anyone to use import-self-sigs-only.

To be clear, this would allow the least competent CA in the system root trust anchor list to certify an arbitrary server as a member of hkps.pool.sks-keyservers.net. So it is in some sense a security vulnerability -- it allows for a bypass of the correct authority.

I've just pushed 1c9cc97e9d47d73763810dcb4a36b6cdf31a2254 to the branch dkg-fix-T4593

Note also that some keyservers like keys.openpgp.org will distribute only verified self-sigs (including revocations and subkey updates) without distributing the floodable third-party certifications. We can and should distinguish "updates-only" keyservers from discovery-by-address mecahnisms.

Just importing a ~666KiB certificate when this monster certificate is in the keyring consumes over 10m of CPU time:

Verifying a git tag from the "clean" version of this certificate takes ~225ms of CPU time. Verifying the same git tag from a keyring that contains the flooded version of the certificate takes ~145s. This is factor of more than 600×. Any automated git tag verification system can probably be DoSed by this behavior.

I didn't mean to suggest that switching to sqlite was the only way to fix this, but if it is a promising way to fix it, that would be great. I'm sure there are other ways.

I recognize that adding network activity to the test suite can be complicated (not all test suites are run with functional network access), but if it is possible to have a unit test or something (that doesn't do network access, but just looks at what the dirmngr *would* have tried somehow?), that would be great. Thanks for looking into this!

i'm aware of the filters you're using, but they are not a principled response to this kind of certificate flooding attack. An attacker who wants to be really abusive can easily create certifications that bypass any import-filter gpg is capable of.

sorry to keep pinging this, but given the ongoing flooding attacks (e.g. T4591) and how SKS and similar keyservers are unable to safely transmit flooded certificates, i think this kind of fix is urgent if we expect gpg to be able to retrieve revocations safely. What's the status here?

Please see T4592 where i've reported this particular performance concern in more detail, including profiling data.

For folks who encounter this problem in the future, i recommend that you first check whether you have a pubring.gpg instead of (or in addition to) your pubring.kbx. If you do have pubring.gpg, you should be able to run the pipeline to the awk script described above with just the pubring directly, which omits the time-consuming gpg --export step above. so i think that would look like: