When disabling IPv6 on boot (see this) with kernel parameter ipv6.disable=1, dirmngr stops working properly.
This is fixed just by adding disable-ipv6 option to ~/.gnupg/dirmngr.conf but it can be hard to debug if the problem is noticed in other applications using gnupg (e.g. Kleopatra showing invalid trust levels on certificates, Kmail not being able to sign emails, etc.).
I guess I would expect dirmngr not to use IPv6 if IPv6 module is disabled. I found this relevant issue T4165 but it looks like it was only fixed for Windows in 2.2.18. It would be nice to implement something similar for GNU/Linux.
I am currently using Archlinux with linux 5.7.4 and GnuPG 2.2.20. But I can replicate this issue in other distros as well.