Ok, I managed to find 48504E974C0DAC5B5CD476C8202274B24C8C7172 via Powershell. It was in the CA store of my non-privileged user and since I always checked the certificate store as administrator it did not show up there. After removal of this intermediate certificate I am able to use hkps://keyserver.ubuntu.com.

Ok, so order of loading is not a problem since the cache does not store them by insertion order, but instead indexes them by the first byte of their fingerprint.
So, I think the problem here is that the expired intermediate certificate (48504E974C0DAC5B5CD476C8202274B24C8C7172) is somehow loaded in Windows and since its fingerprint's first byte is less than the server-supplied intermediate (A053375BFE84E8B748782C7CEE15827A6AF5A405) Windows chooses this one. I can see that the expired intermediate certificate is indeed loaded on Windows if I increase verbosity of dirmngr logs. However, I am still unsure where this certificate lives. The log says it comes from the "CA" store, but searching for it visually or by fingerprint search in Windows Certificates Snap-In (MMC) does not let me find it.
I will keep looking, but if you want to reproduce in your VMs, I suppose adding the expired intermediate certificate and the expired root certificate to the system store should make this reproducible.

@TheParanoidProgrammer thanks for investigating further. It is highly appreciated!

On a side note, it turns out that Ubuntu Maintainers ship gpg with GnuTLS dynamically linked, so that's why I went down that road first. I compiled gpg from source for Ubuntu with ntbtls for further tests. Interesting insight is that find_cert_bysubject returns different certificates on first try on my Ubuntu Machine compared to my Windows 10 Machine:

Ok, I may see three potential problems in dirmngr->validate.c->validate_cert_chain(), but it may also be my limited familiarity with the gnupg source.

• Here we leave the certificate validation loop at the first trusted root certificate, even if it is expired as we only mark this fact for later evaluation.
• Here we seem to only ever go up the chain, never sideways as is the case in the original patch for this bug.
• And probably most impactful, here we fail the whole validation if any of the previously checked certificates is expired, so that even if we would fix the second point by checking sibling certificates, we would still get an overall failure.

What I wonder is: In a number of tests in our machines (mostly virtual machines), the TLS access to keyserver.ubuntu.com does work. I have yet to see a VM where it does not. So there must be a difference.

Not a solution yet, but some more insights.
Starting from @NoSubstitute 's log output and from @bernhard 's statement that we use ntbTLS I verified that my dirmngr.exe was indeed compiled with NTBTLS 0.2.0. I did so by running strings "C:\Program Files (x86)\GnuPG\bin\dirmngr.exe" | grep TLS which returned "This is NTBTLS 0.2.0 - Not Too Bad TLS" among other strings. I also grepped for some debug strings introduced in newer commits to verify that the NTBTLS version used is not the current HEAD of master, but at least some commit before 64f895dba734802662cbb81b64cd0b4af198ee71. I will just assume it is the actual 0.2.0 release for now.

@NoSubstitute It is okay for me to keep this issue, if most people prefer it this way, was just asking.

@bernard - well, that's the kicker, isn't it.

Ah, just seeing that this issue is resolved. Shall we open a new one to be well structured?
(If we reopen this one, there is a lot of old information in here that does not apply anymore before the fixes that went into dirmngr/gnupg).

Does gpg4win ship a TLS library with gpg or does it use a system default?

Alright, in the hope it helps to pin this down, trying to sum up what I tried during and after my conversation with @bernhard so far:

• Windows 10 keeps both the old and new root CA in the store and manual edits to the root certificate store are undone by the OS sooner or later
• ignoring the intermediate certificate with dirmngr --ignore-cert 48504E974C0DAC5B5CD476C8202274B24C8C7172 fixes the problem as a workaround, but is not a satisfying solution
• I cloned the repository and took a look at the original patch; while it seems that we only check validity of certificates without considering the expiration date, the patch does fix the original bug which I confirmed by compiling gpg from source at the commit containing the patch and another version at the commit prior to the patch. That is, the patch successfully fixes this on my Ubuntu machine. On my Windows 10 machine the bug persists no matter if using a self-compiled version from those commits or the official versions from gpg4win.
• During exploring the source code and finding out how to compile and test from source I found out that I can reproduce the bug on Ubuntu if I compile gpg with the patch applied, but with a GnuTLS version that does not have their patch for this issue. Since this is the case with the default GnuTLS dev sources in Ubuntu 20.04., I had to get GnuTLS library from the project itself in order to successfully compile gpg for Ubuntu. For Windows the problem persists, however. I did not find GnuTLS or any other TLS library in the application directory of the GPG install on Windows nor in the installer itself. So I'm wondering if the remaining issue on Windows is actually with the used TLS library there. Does gpg4win ship a TLS library with gpg or does it use a system default?
• The fixed version of GnuTLS is 3.6.14 for the project itself, there are backports of this patch for Ubuntu Xenial and Bionic, unfortunately not for Focal yet.

Hello.
@bernard has been so kind to try and help me with this exact issue over in the gpg4win forum, and it seems I'm not the only one who still has problems with the "broken" LE certificate chain and hkps://keyserver.ubuntu.com.

Ubuntu have been syncing since 7th December: https://www.mail-archive.com/sks-devel@nongnu.org/msg07174.html

Ubuntu have been syncing since 7th December: https://www.mail-archive.com/sks-devel@nongnu.org/msg07174.html

Why the Ubuntu server? AFAIU it does not sync with other servers and it has some tained pubkeys (which is both fine as a choice of this service, it just does not seem to fit the purposes best).

For the next release I'll change the gnupg.net mappings to use the Ubuntu server also for non-TLS connections.

@ikloecker yes sorry ok

@bernard Right sorry. I have sent request to mailing lists

@alexnadtoka, please stop adding the same information to two different issues. Let's use T5744: Issue with connecting to GPG server for any further comments.

@alexnadtoka wrote:

both versions had issues(( and send two requests to RU and EN comunity . No answer for two days already

@bernhard yeah thank you. both versions had issues(( and send two requests to RU and EN comunity . No answer for two days already
The log clearlys says certificate is expired(( but it is not at least for keyserver... May be it is reffering to gpg key... I dont know... but it is not expired either. Probably I am missing something. Will try to contact community again.

@alexnadtoka When using Gpg4win-4.0.0 or 3.3.16 with an updated GnuPG the validation of dirmngr works fine with the Let's encrypt certificates again. If you have one of these versions, and you still have problems, you need to be more specific about which connection you are referring to.
Maybe it is best to ask on one of community channels (e.g. the gnupg-users mailinglist, see https://gnupg.org/documentation/mailing-lists.html )

Do you have a ballpark figure for the install base (not including variants such as debian with modified defaults)? That might help us decide what counts as "overloading".

The problem is just that there are not that much keyservers left and thus I added those running by large organisations. I really don't want to overload your servers. I would also trust nlnet more than canoncial which is why I started with them.
Its all a mess. Maybe no keyserver should be the default.

@alexnadtoka, did you do what Werner wrote in T5639#150626?

Guys I am facing similar issue but my Lets ecnrypt certificates are all ok. What is the problem with my gpg4win client? When connecting to openpgp server it says certificate is expired. Anybody can help me?

ikloecker: Please go ahead

IIRC, the problem is/was that this breaks some old keyservers. But there are no more old keyservers - if there are useful keyservers at all.

Proposed patch:

Hi guys, I just tested the git version (426d82fcf1c133bfc1d5c931109d71db3f3815a9) and it works well thank you.

Fixed in 2.2.33.

Might be a TOR Thing?

Any news here? Is this issue going to be fixed or not? It's really annoying.

This has not been set high on the priorities, because keyserver access works for most with Gpg4win (and thus GnuPG) on windows. A recent exception has been occurred about a month ago with Let's encrypt expired root certificate. So currently for Gpg4win 3.1.16 you need to update to a newer GnuPG (Version 2.2.32 at time of writing), by installing the simple installer,e.g. https://gnupg.org/ftp/gcrypt/binary/gnupg-w32-2.2.32_20211006.exe

I don't know if it's same in your case, but to fix my case, I pushed a change rG48359c723206: dns: Make reading resolv.conf more robust.

I managed to create a case. Put a line:

BTW, in your screen shot (log is preferred here), it shows 1c00, that must be actually written as AAAA (0x1c). In the bug T3803, we saw byte sequence like that, additional 00 was added then resulted malformed DNS packet.

dots are not allowed in hostnames.

OK, I'll gdb in there to see what happens. My domain is a classic pgp.domain.com

Ah, other possible case is .. in hostname.

It's hard to investigate your problem, with no information of host for the query.
I mean, there is no case to replicate (for us).

There won't be any other 3.1 release - install GnuPG 2.2.32 on top of Gpg4win 3.1.16

My experience on a Window 10 system (with Gpg4win 3.1.15 which has GnuPG 2.2.27) was, that removing the expired root certificate did not help with https://keyserver.ubuntu.com and the intermediate certificate was not in the windows store, so it could not be removed.

Removing an intermediate cert from your local system doesn't help because any correctly configured server will send you all necessary intermediate certs together with the server cert. You'd have to remove the expired root certificate instead (see Workaround 1 on https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/). The problem is that this will break certificate verification for any servers that still use the old intermediate cert, e.g. keyserver.ubuntu.com.

The LE web site has instruction on how to do this. However, it is complicated and depends on your system. The intermediate cert you listed is signed by the expired old root cert. If you remove this intermediate cert the other root cert will be found and we are done. The old LE certs had a 4 tier chain and the new one a 3 tier.
See https://dev.gnupg.org/rG341ab0123a8fa386565ecf13f6462a73a137e6a4 and https://letsencrypt.org/images/isrg-hierarchy.png

One problem I see is that keyserver.ubuntu.com delivers a problematic intermediate(?) certificate:

If there is no easy way to install a new version of GnuPG, e.g. for Gpg4win or for GNU/Linux distributions: It may make sense to have instructions for the workaround ready.

This has been solved in 2.2.26 commit rGc75fd75532

This has been fixed in version 2.2.16.

dirmngr needs to be killed for this. gpgconf --kill dirmngr.

The now used /var/run thingy solves all these problems nicely. In fact we may eventually remove the use fallback of using sockets in the GNUPGHOMEDIR.

The other patches don't make sense because of future plans for dirmngr.

The code has been reworked to also support the updated schema which also stores the fingerprints and a parsed down mail address. See gnupg/doc/ldap/ . These changes are in master and 2.2.26. Sorry for taking so long to fix that.

Granted I'm not familiar with the functions and it may not be applicable, but the DNS resolver functions in the GNU C Library have semi-recently gained parameters (RES_USE_DNSSEC) to check for DNSSEC validation IIRC. Recent versions of glibc also don't trust the 'ad' bit unless an indication of its trustworthiness is set in /etc/resolv.conf, say if using a local validating resolver, so one can be sure that it's trustworthy. It also appears musl libc may support this.

Just reading this issue in detail.

Pushed fix to master and STABLE-BRANCH-2-2.

Thanks for your log.

$ cat /run/user/1000/dirmngr.log

2020-07-11 19:33:44 dirmngr[2305.0] permanently loaded certificates: 140
2020-07-11 19:33:44 dirmngr[2305.0]     runtime cached certificates: 0
2020-07-11 19:33:44 dirmngr[2305.0]            trusted certificates: 140 (139,0,0,1)
2020-07-11 19:39:24 dirmngr[2305.6] force-crl-refresh active for issuer id CE04B58CBA5B8069AA0D503634B861593BE86F20; update required
2020-07-11 19:39:24 dirmngr[2305.6] number of system provided CAs: 148
2020-07-11 19:39:24 dirmngr[2305.6] error creating socket: Address family not supported by protocol
2020-07-11 19:39:24 dirmngr[2305.6] error connecting to 'http://cdp1.pca.dfn.de/global-root-g2-ca/pub/crl/cacrl.crl': Address family not supported by protocol
2020-07-11 19:39:24 dirmngr[2305.6] error retrieving 'http://cdp1.pca.dfn.de/global-root-g2-ca/pub/crl/cacrl.crl': Address family not supported by protocol
2020-07-11 19:39:24 dirmngr[2305.6] crl_fetch via DP failed: Address family not supported by protocol
2020-07-11 19:39:24 dirmngr[2305.6] command 'ISVALID' failed: Address family not supported by protocol
2020-07-11 19:39:24 dirmngr[2305.6] force-crl-refresh active for issuer id 3476EB7C1E02B3BAF954EEE2EFD321F7B8E49D18; update required
2020-07-11 19:39:24 dirmngr[2305.6] error creating socket: Address family not supported by protocol
2020-07-11 19:39:24 dirmngr[2305.6] error connecting to 'http://pki0336.telesec.de/rl/TeleSec_GlobalRoot_Class_2.crl': Address family not supported by protocol
2020-07-11 19:39:24 dirmngr[2305.6] error retrieving 'http://pki0336.telesec.de/rl/TeleSec_GlobalRoot_Class_2.crl': Address family not supported by protocol
2020-07-11 19:39:24 dirmngr[2305.6] crl_fetch via DP failed: Address family not supported by protocol
2020-07-11 19:39:24 dirmngr[2305.6] command 'ISVALID' failed: Address family not supported by protocol
2020-07-11 19:39:24 dirmngr[2305.6] force-crl-refresh active for issuer id 70F42DB9235EC84DC35D445B3407CABF4324291C; update required
2020-07-11 19:39:24 dirmngr[2305.6] error creating socket: Address family not supported by protocol

@gniibe I saw that you didn't understand what I meant by "dirmngr stops working properly" in E663.
Have a look at this post in Archlinux forum.

Fixed; In master the code already uses our generic scheme parser.

DANE for OpenPGP is an experimental RFC (RFC-7929) and it is likely that we will remove the support because it is too hard for most users to add keys to a zone. Further a validating resolver on the desktop is too hard to maintain and the cause of too many other failures. And no, unbound etc is not an option because it is not usable by the majority of GnuPG users.

The same concern has been reported at https://bugs.debian.org/964033 -- if dirmngr is not going to follow the specification, it should at least document (and maybe warn?) about how it is divergent.

When I test it on Debian, disabling by,

Please get log of dirmngr, by putting

log-file /run/user/<YOURNUMBER-LIKE-1000>/dirmngr.log

Can you characterize the failure when ipv6.disable=1 ? The straightforward failure (connect() fails with EHOSTUNREACH after a few seconds) should presumably be treated the same as if some other host happened to be offline. That should result in dirmngr failing over to the next available address for the configured keyserver, right?

The problem is that I have not yet found a _portable_ way to detect proper working v6 or v4 networking without doing a test connection. For privacy reasons we don't want to do that.

Fixed in master and applied to 2.2 branch too.