This is not just about keys.openpgp.org. It's about any keystore that implements user id redaction, for whatever reason. When you say "what they can do is accept only user ids which…" i think you mean "the userid-redacting keystores can instead redistribute user ids which …". Is that right?

@werner, i don't think there is a 64K limit either, at least not in 2.2.16. Here is 2.2.16 with an empty homedir fetching Zack's certificate here which is > 97KiB:

Not every incoming certificate that has no user ID will lack a user ID once it is merged with the local copy of the same certificate. T4393 describes that use case, so if you're interested in receiving user-ID-lacking updates to certificates that you already have a copy of, @jaymzh, you should follow up on that ticket.

Once a revocation is added (to any part of the certificate), perhaps all the certification packets that are clearly made obsolete by the revocation could be dropped from the certificate? That would certainly free up space to be able to import additional revocations if needed.

I think what you're missing is the keys.openpgp.org documentation which makes it clear that they will not distribute identity information (read: "User IDs") without an explicit confirmation by the operator of the e-mail address named in the User ID. They strip down the certificate pretty significantly before redistribution, especially if the e-mail address hasn't been confirmed directly with the operators of that server.

out of curiosity, why does gpgv need the name of the file?

in 2.2.16, anyway, gnupg does not appear to apply import-minimal for WKD.

In that case, you can treat this ticket as a bug in the documentation, which still needs to be resolved.

auto-key-retrieve happens in the context of signature verification when the certificate is missing. If no signer User ID subpacket is present in the signature, then WKD simply won't work.

hm, i see your point. If you could spell out what the specific regression(s) in more detail, though, that might help us to reason about their impact.

if you want to add a separate subcommand for that, i would be happy to abandon migrate-pubring-from-classic-gpg.

I think you're suggesting accepting *any* path if the hostname of the proposed redirection matches openpgpkey.example.org when querying the WKD direct URL for an @example.org address. That would also be a fine solution from my point of view.

my initial scenario is where an adversarial keystore floods a certificate right up to (but within) the 5MiB boundary, so that the user has stored it in the keyring already. Then, the user encounters the certificate again, with revocation attached.

@werner, thanks for the pointer to the report, that's certainly useful. And i'm happy that organizations like SektionEins are doing GnuPG audits and publishing their results regardless of who paid for them.

@jaymzh, I agree with @werner that without more information, this bug report isn't particularly actionable.

as a separate variant: if the attacker floods the certificate with bogus self-signatures -- that is, certifications that have an issuer fingerprint or issuer key id subpacket, whether hashed or unhashed -- will that make it impossible to import any of them?

Thanks for working on this fallback, Werner.

Thanks for the pointer, @werner. Certainly we want T4590 fixed.

I should add that i don't really care whose fault it is if the software is broken by some downstream. if it harms any users, and we can fix it, we should fix it, especially if the fix is easy.

We're writing free software, which we know that people use and modify downstream. if we know that the software has a particular sharp edge that people who are modifying it are likely to cut themselves on, we have two options:

If the default keyserver is not hkps.pool.sks-keyservers.net, then @kristianf's CA certificate has no business certifying it.

So this is a defense against an adversary capable of creating a pinentry-wrapper somewhere in $PATH, but not capable of modifying gpg-agent.conf? It sounds to me like this is a defense against a very unusually-constrained attacker, at the expense of regular, common bug reports and user confusion.

thanks for working on this @werner. rG2e349bb61737 is definitely not useful for me. If i am going to tell anyone "hey, do this weird thing differently in order to fetch my key", i will tell them "pull it from https://dkg.fifthhorseman.net/dkg-openpgp.key". I will never tell anyone to use import-self-sigs-only.

To be clear, this would allow the least competent CA in the system root trust anchor list to certify an arbitrary server as a member of hkps.pool.sks-keyservers.net. So it is in some sense a security vulnerability -- it allows for a bypass of the correct authority.

I've just pushed 1c9cc97e9d47d73763810dcb4a36b6cdf31a2254 to the branch dkg-fix-T4593

Note also that some keyservers like keys.openpgp.org will distribute only verified self-sigs (including revocations and subkey updates) without distributing the floodable third-party certifications. We can and should distinguish "updates-only" keyservers from discovery-by-address mecahnisms.

Just importing a ~666KiB certificate when this monster certificate is in the keyring consumes over 10m of CPU time:

Verifying a git tag from the "clean" version of this certificate takes ~225ms of CPU time. Verifying the same git tag from a keyring that contains the flooded version of the certificate takes ~145s. This is factor of more than 600×. Any automated git tag verification system can probably be DoSed by this behavior.

I didn't mean to suggest that switching to sqlite was the only way to fix this, but if it is a promising way to fix it, that would be great. I'm sure there are other ways.

I recognize that adding network activity to the test suite can be complicated (not all test suites are run with functional network access), but if it is possible to have a unit test or something (that doesn't do network access, but just looks at what the dirmngr *would* have tried somehow?), that would be great. Thanks for looking into this!

i'm aware of the filters you're using, but they are not a principled response to this kind of certificate flooding attack. An attacker who wants to be really abusive can easily create certifications that bypass any import-filter gpg is capable of.

sorry to keep pinging this, but given the ongoing flooding attacks (e.g. T4591) and how SKS and similar keyservers are unable to safely transmit flooded certificates, i think this kind of fix is urgent if we expect gpg to be able to retrieve revocations safely. What's the status here?

Please see T4592 where i've reported this particular performance concern in more detail, including profiling data.

For folks who encounter this problem in the future, i recommend that you first check whether you have a pubring.gpg instead of (or in addition to) your pubring.kbx. If you do have pubring.gpg, you should be able to run the pipeline to the awk script described above with just the pubring directly, which omits the time-consuming gpg --export step above. so i think that would look like:

wow, 46MiB, that's even worse than mine. :( thanks for sharing the update, @jackalope. I'm glad you've worked around it for now, but sadly this kind of certificate flooding could happen at any time if you're using the SKS keyserver network :(

That's a great question, @jackalope. I found this in a different misbehaving keyring recently by basically deleting keys by hand until only one was left. surprise, it was mine (ugh)! But that process is pretty slow and manual and tedious. Let me see if i can do better.

@jackalope, the place where the output is hanging is likely due to output buffering (i have been able to replicate the same problem, and the output hangs at intervals of 8192 octets). So while it is giving you a clue about where the hang is, it's not a very precise clue.

Thanks for the feedback, @werner. I think I understand the reasons that we've gotten to this place -- but that doesn't mean i think it's ok to stay here. In this bug report, i'm pointing out that the documentation and the feedback/error reporting is misleading, which leads to difficulty in debugging. We need to do something about it.

I note that this is likely happening because we are using gcr's system-modal prompter. I haven't looked into whether it's even possible to use gcr in a non-system-modal way, but i'd welcome pointers.

i think this might be a duplicate of T4496

I'm unlikely to put a windows-specific patch into the debian source, as
i have no good way of testing it, and it wouldn't affect any binary that
we ship.

Hm, T4521 suggests that the two different cases should not be treated differently. If you think that they *should* cause distinct behavior, please do mention it over there!

@gniibe, thanks for the diagnosis! I agree that restarting or shutting down the backends should be done in the reverse order as a simple workaround.

without feedback, i have no idea what you want to do here as upstream. I believe this issue has identified a specific failing use case, and it has a patch that fixes the problem. if there's a problem, please let me know what it is. If there's no problem, please consider merging.

I note that "the best" seems like it might be a pretty subjective thing. The standard GnuPG framing asks about the validity of keys for the User ID in question. Perhaps the caller could indicate whether they want to require full validity for each key to make this key selection more strict.

The function would do something like:

• from msg, extract all e-mail addresses from to, cc, bcc fields
• find "the best" keys that match these addresses, storing them in keylist
• copy msg to tmp, remove bcc header from tmp
• wrap armored output of gpg.Context.encrypt(bytes(tmp), recipients=keylist) in the necessary RFC 3156 cladding, copying most headers from msg (maybe stubbing out the subject), producing an email.message.EmailMessage object.

Any word on this? i've pushed a fix for this into debian experimental as a part of 2.2.16-2, but i am concerned that there's no adoption from upstream. If there's a reason that this is the wrong fix, please do let me know!

we now have a DCO from @Valodim

If we only need it for backward compatibility, then the configuration in gpg.conf should *not* be overriding the preferred, forward-looking form of the configuration (in dirmngr.conf). If it is low priority to fix this, then there will be a generation of GnuPG users and toolchains which deliberately configure the value in gpg.conf instead of dirmngr.conf because they'll know that's the more robust way to do it.

@werner, My usual approach for private branches is to prefix with dkg/, but (a) playfair rejects branch names with a /, and (b) i'm not the author of these patches, and i didn't want to claim credit that doesn't belong to me.

I've pushed @Valodim's proposed patches to the fix-4393 branch in our git repo. they look good to me, and i think they should be merged to master.

I think this commit should be reverted -- if the test fails we should figure out why and fix it, because the logic of the test is correct.

It also passes for me with python 2.7.16 (debian package 2.7.16-2).

i think you mean t-decrypt-verify.py, right? That seems to indicate a problem on the targeted system that we ought to fix, rather than just commenting out the test. t-decrypt-verify.py passes for me when i test it with python 3.7.3 (debian python 3.7.3-1). what version of python are you testing with?

Sorry for the truncated commit. the sentence should have been:

@gouttegd good catch!

fwiw, the bug looks like it's in send_request in ks-engine-hkp.c, which re-uses the http_session object without re-initializing its tls_session member.

thanks for the triage, @werner!

thanks for fixing that error message, @werner. As @Valodim points out in discusson about hagrid, a gpg.conf keyserver option (deprecated according to the documentation) overrides the dirmngr.conf keyserver option (not deprecated according to the documentation.