I agree on the first part. This needs to be fixed.

I do not understand wht you think "no public key" is the wrong message. We have
always used this message if the public key is not available for verification.
Do you think the text should be changed to "public key not found" ? That would
be a simple change in libgpg-error.

Libgpg-error has a GPG_ERR_MISSING_KEY but that code indicates wrong usage of
functions or bad data structures.

for cases (1), (2), and (3) it sounds like you don't need the PTR at all. right?

For your case (4), i think we should reject hkps via literal IP addresses. It's
not a real-world use case, and if you want to test/experiment with hkps as a
developer, you should have at least the capacity to edit /etc/hosts (or whatever
your system's equivalent is). Anyway, trying to support this case for the
purposes of debugging doesn't make sense if support for this case is the cause
of the bugs in the first place ;)

re: duplicate hosts: I live in a part of the world where dual-stack
connectivity is sketchy at best. And, when connecting to things over Tor, it's
possible that connections to IPv4 hosts will have a different failure rate than
IPv6 connections.
So unless you already know that the host itself is down, why would you avoid
trying the other routes you have to it?

Look at it another way: when trying to reach host X, you discover that X has two
IP addresses, A and B. You try to reach A and it's not available. Isn't it
better to try B instead, rather than to avoid trying B at all just because A was
unreachable?

In a pool scenario, you might want to try to cluster addresses together by
perceived identity so that you can try an entirely different host first, rather
than a different address for the same host who happens to be in the pool twice.
But that strikes me as a very narrow optimization, certainly something that'd
only be worth implementing after we've squeezed the last bit of performance out
of other parts of the code (parallel connections, "happy eyeballs", etc).
Definitely not something to bother with at the outset. So i'd say drop that
optimization for simplicity's sake.

So the simplest approach is:

a) know the configured name of the keysserver
b) resolve it to a set of addresses
c) try to connect to those addresses, using the configured name of the server
for SNI and HTTP Host:

This is all that's needed for cases (1) and (3), and it could also be used in
case (2) if you see (b) as a two-stage resolution process (name→SRV→A/AAAA),
discarding the intermediate names from the SRV. Given that some people may
access the pool via case (1), and servers in the pool won't be able to
distinguish between how they were selected (SRV vs. A/AAAA), they'll still
accept the connections.

If you decide the additional complexity is worthwhile for tracking the
intermediate names in the SRV records, you can always propagate the intermediate
names wherever you like locally without changing the "simplest" algorithm.

If you really want to use the names from the SRV in collecting, then the
algorithm should change to:

a) know the configured name of the keyserver
b) resolve it to a set of intermediate names
c) resolve the intermediate names to a set of addresses
d) try to connect to those addresses, using the intermediate name of the server
for SNI and HTTP host.

But still, no PTR records are needed.

Okay, I get this error now. I had to implement a new option --disable-ipv4 to
make testing easier.

I have never seen the no permission message but only a general connection failed
error. I can try your suggestion of setting an explicit NoIPv6Traffic

We have several cases:

1. A pool accessed via round-robin A/AAAA record: We do not use the canonical hostname (i.e. from the PTR) but the name of the pool for the certificate. This is the classical way how keyserver pools.

2. A pool access via SRV records: The SRV record has the canonical name and thus we do not need a PTR lookup. But we need a address lookup.

3. A keyserver specified by its name: We alread have the name thus no need for PTR lookup.

4. A keyserver specified by literal IP address: We need a host name for the certificate. Either we take it from the PTR record or we reject TLS access. I don't think that is is a real world use case but for debugging it is/was really helpful. Should we reject hkps via literal IP addresses?

It is quite possible that some of these cases do not work right. I
have done only manual testing and the matrix is pretty complex: We
have all combinations of direct/Tor, v4 only, v6 only, v4, v6,
interface up, network down.

Right, by "duplicate host", I mean hosts reachable by several addresses
and in particular by v4 and v6. My test back when I originally
implemented the code showed that when hosts are down their other
addresses are also down. Without marking the host dead, the code
would have tried the same request on another address and would run
into the next timeout.

I also think that most delays are due to connection problems and not due to DNS
problems. And most connection problems are due to lost network access. There
we might need to tweak the code a bit similar to what I did for ADNS.

Test added in 5aafa56dffefe3fac55b9d0555c7c86e8a07f072.

Here's a concrete example of how using PTR records gets things mixed up.

keyserver.stack.nl offers keyserver service on port 443.

It has an A record at 131.155.141.70.

But the ptr is to mud.stack.nl:

70.141.155.131.in-addr.arpa. 69674 IN PTR mud.stack.nl.

and the https SNI and HTTP Host: directives provide an entirely different
website depending on whether you access it with:

  https://mud.stack.nl/

or

  https://keyserver.stack.nl/

If you access it as https://hkps.pool.sks-keyservers.net/, you get the
"keyserver" view. But if you access it by the name in the PTR record
("mud.stack.nl") then you get the mud view (and a 404 on any /pks URLs)

Even more troubling is that dirmngr successfully connects to mud.stack.nl and
does the query, even though it is configured to only talk to
hkps.pool.sks-keyservers.net

This suggests that anyone able to spoof a PTR record to me can get my dirmngr to
send my potentially-sensitive keyserver queries to an entirely different webserver.

Fix is in 2.1.18

Fixed in 2.1.18

Should be fixed in the just released 2.1.18

Released with 2.1.18

I've moved the discussion about the need for PTR over to
T2928

In this ticket, let's focus on what happens when Tor has the NoIPv6Traffic flag
set. How should dirmngr respond in that case?

I think if it gets a "permission denied" from its tor socket (or from any proxy)
when it's trying to connect to a host, it should treat that host as dead and
move to try next one. If dirmngr knows that it is using tor, and it knows that
the address it is trying is also IPv6, it could also log a message about the
IPv6Traffic flag.

does that seem like the right set of changes needed?

if you add NoIPv6Traffic to your torrc, and restart tor, can you replicate the
problem?

I nearly filed this as a minor bug to start with. Apologies for the
mis-classification.

My thinking was that there are a few rarer cases on unattended/shared
environments where this may be an issue. Scripts may deliberately be using a
umask that allows write permission to files it's creating, not expecting that
pubring (or other keyring) changes will create a new file. Other users/services
may need read permission to those keyrings, and actually end up with write
permission. This is potentially a problem despite the data not being secret.

Granted, the above hypothetical situation is uncommon and easily worked around
with better design/testing, but it might catch people out.

I don't consider this a minor bug.

The pubring does not contain secret information but only sensitive data, like
many files in a user's $HOME. The umask is the standard Unix way of restricting
access for new files. For files holding secret data we explicitly set the
permissions.

To answer your question:

  Can you explain why dirmngr does the DNS roundtrip lookup, mapping
  from the pool's A and AAAA addresses back to names?  It seems like
  it'd be a lot simpler (and faster, and less error-prone) to avoid
  the PTR lookups if we have the IP addresses already.

If it is a plain server and not a pool, looking up the PTR is
necessary to get the hostname for SNI and possible also for Host:
header.

For a pool we would not need the name because the already known name
of the pool is used for SNI. However, to find duplicate hosts in the
hosttable it is useful to have the hostname. We also return the
actual used hostname to gpg for information purposes and to eventually
store this with the key as meta info.

Yes, we could the PTR lookup of pools faster or in the background -
but for now a simple appraoch is better for debugging.

FWIW, I am using Libassuan 2.4.3 plus one portability fix for BSDs.

And here is a log when using

  keyserver hkp://oteiza.siccegge.de

in dirmngr.conf (and also use-tor) of course:

DBG: chan_7 -> OK Dirmngr 2.1.18-beta67 at your service
connection from process 24314 (1000:1000)
DBG: chan_7 <- keyserver --resolve --hosttable
DBG: dns: libdns initialized (tor mode)
DBG: dns: getsrv(_pgpkey-http._tcp.oteiza.siccegge.de) -> 0 records
DBG: dns: libdns initialized (tor mode)
DBG: dns: resolve_dns_name(oteiza.siccegge.de): Success
resolve_dns_addr for 'oteiza.siccegge.de': 'oteiza.siccegge.de' [already known]
DBG: dns: resolve_dns_addr(): Success
DBG: chan_7 -> S # http://oteiza.siccegge.de:11371
DBG: chan_7 -> S # hosttable (idx, ipv6, ipv4, dead, name, time):
DBG: chan_7 -> S # 0 6 oteiza.siccegge.de
v6=[2a01:4a0:59:1000:223:9eff:fe00:100f]
DBG: chan_7 -> OK
DBG: chan_7 <- [eof]
handler for fd 7 terminated
handler for fd 7 started
DBG: chan_7 -> # Home: /home/wk/.gnupg
DBG: chan_7 -> # Config: /home/wk/.gnupg/dirmngr.conf
DBG: chan_7 -> OK Dirmngr 2.1.18-beta67 at your service
connection from process 24325 (1000:1000)
DBG: chan_7 <- GETINFO version
DBG: chan_7 -> D 2.1.18-beta67
DBG: chan_7 -> OK
DBG: chan_7 <- KS_GET -- 0xDEADBEEF
number of system provided CAs: 173
DBG: http.c:connect_server: trying name='oteiza.siccegge.de' port=11371
DBG: dns: resolve_dns_name(oteiza.siccegge.de): Success
DBG: http.c:1706:socket_new: object 0x00007f1420453df0 for fd 8 created
DBG: http.c:request:
DBG: >> GET /pks/lookup?op=get&options=mr&search=0xDEADBEEF HTTP/1.0\r\n
DBG: >> Host: oteiza.siccegge.de:11371\r\n
DBG: http.c:request-header:
DBG: >> \r\n
DBG: chan_7 -> S PROGRESS tick ? 0 0
DBG: chan_7 -> S SOURCE http://oteiza.siccegge.de:11371
DBG: (27779 bytes sent via D lines not shown)
DBG: chan_7 -> OK

I tried this with and without my local v6 interface up; both are
obviously the same. Bote that in both cases my resolver is on the
local network and accessed via v4 - but it should not matter because
due to use-tor 8.8.8.8 is used anyway.

Even with the Tor from testing I am stil having the IPv6Traffic flag
in my torrc - I am not sure whether this is still required.

Using a configuration with only "use-tor" and debug options, and no
keyserver nor certificates defined I used

  gpg-connect-agent --dirmngr 'keyserver --resolve --hosttable' /bye

several times until I got oteiza.siccegge.de as keyserver:

DBG: chan_7 <- keyserver --resolve --hosttable
DBG: chan_7 -> S # https://oteiza.siccegge.de:443
DBG: chan_7 -> S # hosttable (idx, ipv6, ipv4, dead, name, time):
DBG: chan_7 -> S # 0 hkps.pool.sks-keyservers.net
DBG: chan_7 -> S # . hkps.pool.sks-keyservers.net
DBG: chan_7 -> S # . --> 2 10 6 4 9 5 8 7 1* 3
DBG: chan_7 -> S # 1 6 4 oteiza.siccegge.de
v6=[2a01:4a0:59:1000:223:9eff:fe00:100f] v4=92.43.111.21
DBG: chan_7 -> S # 2 6 4 bone.digitalis.org v6=[2a00:14b0:4200:3000:27::27]
v4=212.12.48.27
DBG: chan_7 -> S # 3 6 prod00.keyserver.dca.witopia.net
v6=[2606:9500:201:1::141]
DBG: chan_7 -> S # 4 6 4 gpg.NebrWesleyan.edu v6=[2606:1c00:2802::b]
v4=192.94.109.73
DBG: chan_7 -> S # 5 6 4 hufu.ki.iif.hu v6=[2001:738:0:600:216:3eff:fe02:42]
v4=193.224.163.43
DBG: chan_7 -> S # 6 6 4 gozer.rediris.es v6=[2001:720:418:caf1::8]
v4=130.206.1.8
DBG: chan_7 -> S # 7 6 4 mud.stack.nl v6=[2001:610:1108:5011::70]
v4=131.155.141.70
DBG: chan_7 -> S # 8 4 ip-209-135-211-141.ragingwire.net v4=209.135.211.141
DBG: chan_7 -> S # 9 4 host-37-191-238-78.lynet.no v4=37.191.238.78
DBG: chan_7 -> S # 10 4 cryptonomicon.mit.edu v4=18.9.60.141
DBG: chan_7 -> OK

and then "gpg --recv-key deadbeef":

DBG: chan_7 <- KS_GET -- 0xDEADBEEF
DBG: http.c:connect_server: trying name='oteiza.siccegge.de' port=443
DBG: dns: resolve_dns_name(oteiza.siccegge.de): Success
DBG: http.c:1706:socket_new: object 0x00007f5d5000bea0 for fd 9 created
DBG: http.c:request:
DBG: >> GET /pks/lookup?op=get&options=mr&search=0xDEADBEEF HTTP/1.0\r\n
DBG: >> Host: hkps.pool.sks-keyservers.net:443\r\n
DBG: http.c:request-header:
DBG: >> \r\n
DBG: chan_7 -> S PROGRESS tick ? 0 0
DBG: chan_7 -> S SOURCE https://oteiza.siccegge.de:443
DBG: (27779 bytes sent via D lines not shown)

I did my test with tor 2.5.12-4 (jessie). I will ungrade to testing now and redo.

Fixed in 34fa2d79a07a079be472c3ff486debfdac8c6070.

here's the example run from my modified reproducer script that focuses on
oteiza.siccegge.de:

gpg: keybox '/home/dkg/tmp/tmp.XgzSpI4Oy0/gpg/pubring.kbx' created
gpg: keyserver receive failed: Permission denied
S # hosttable (idx, ipv6, ipv4, dead, name, time):
S # 0 6 oteiza.siccegge.de v6=[2a01:4a0:59:1000:223:9eff:fe00:100f]
OK
2017-01-18 03:19:00 dirmngr[30881] listening on socket
'/home/dkg/tmp/tmp.XgzSpI4Oy0/gpg/S.dirmngr'
2017-01-18 03:19:00 dirmngr[30882.0] permanently loaded certificates: 0
2017-01-18 03:19:00 dirmngr[30882.0] runtime cached certificates: 0
2017-01-18 03:19:00 dirmngr[30882.0] failed to open cache dir file
'/home/dkg/tmp/tmp.XgzSpI4Oy0/gpg/crls.d/DIR.txt': No such file or directory
2017-01-18 03:19:00 dirmngr[30882.0] creating directory
'/home/dkg/tmp/tmp.XgzSpI4Oy0/gpg/crls.d'
2017-01-18 03:19:00 dirmngr[30882.0] new cache dir file
'/home/dkg/tmp/tmp.XgzSpI4Oy0/gpg/crls.d/DIR.txt' created
2017-01-18 03:19:01 dirmngr[30882.6] handler for fd 6 started
2017-01-18 03:19:01 dirmngr[30882.6] connection from process 30879 (1000:1000)
2017-01-18 03:19:01 dirmngr[30882.6] DBG: dns: libdns initialized (tor mode)
2017-01-18 03:19:02 dirmngr[30882.6] DBG: dns:
getsrv(_pgpkey-http._tcp.oteiza.siccegge.de) -> 0 records
2017-01-18 03:19:02 dirmngr[30882.6] DBG: dns: libdns initialized (tor mode)
2017-01-18 03:19:03 dirmngr[30882.6] DBG: dns:
resolve_dns_name(oteiza.siccegge.de): Success
2017-01-18 03:19:03 dirmngr[30882.6] resolve_dns_addr for 'oteiza.siccegge.de':
'oteiza.siccegge.de' [already known]
2017-01-18 03:19:03 dirmngr[30882.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 03:19:03 dirmngr[30882.6] number of system provided CAs: 142
2017-01-18 03:19:03 dirmngr[30882.6] DBG: http.c:connect_server: trying
name='oteiza.siccegge.de' port=11371
2017-01-18 03:19:05 dirmngr[30882.6] DBG: dns:
resolve_dns_name(oteiza.siccegge.de): Success
2017-01-18 03:19:05 dirmngr[30882.6] can't connect to 'oteiza.siccegge.de':
Permission denied
2017-01-18 03:19:05 dirmngr[30882.6] error connecting to
'http://oteiza.siccegge.de:11371': Permission denied
2017-01-18 03:19:05 dirmngr[30882.6] command 'KS_GET' failed: Permission denied
2017-01-18 03:19:05 dirmngr[30882.6] handler for fd 6 terminated
2017-01-18 03:19:05 dirmngr[30882.6] handler for fd 6 started
2017-01-18 03:19:05 dirmngr[30882.6] connection from process 30921 (1000:1000)
2017-01-18 03:19:05 dirmngr[30882.6] handler for fd 6 terminated

fwiw, i'm seeing fewer errors with this version than i was before, perhaps
because we're addressing servers via tor by name instead of by IP address, which
means that we're not tickling the IPv6 issue quite as often?

The failure with oteiza.siccegge.de might actually just be the IPv6 issue
itself, since there is no IPv4 address for it. I can actually force the issue
if i just add the following line to the dirmngr.conf generated in my reproducer
script:

    keyserver hkp://oteiza.siccegge.de

but of course it's a faster failure, because there isn't a dozen DNS A->PTR
round-trips.

Can you explain why dirmngr does the DNS roundtrip lookup, mapping from the
pool's A and AAAA addresses back to names? It seems like it'd be a lot simpler
(and faster, and less error-prone) to avoid the PTR lookups if we have the IP
addresses already.

I note here that the "oteiza.siccegge.de" domain name might be supplied by PTR
records for both its v4 and v6 addresses, and it appears to have a AAAA record,
but it doesn't have any *forward* A record.

I'm baffled by the fact that you're not seeing these errors, and not sure what
to do about it. What version of tor are you running? how is it configured?

i'm running the stock debian tor package, version 0.2.9.8-2.

I've tried with the latest patches and i still see failures :(

gpg: keybox '/home/dkg/tmp/tmp.nchsng7MNY/gpg/pubring.kbx' created
gpg: keyserver receive failed: Permission denied
S # hosttable (idx, ipv6, ipv4, dead, name, time):
S # 0 hkps.pool.sks-keyservers.net
S # . hkps.pool.sks-keyservers.net
S # . --> 9 11 6 4 10 5 7 8 2* 3 1
S # 1 6 4 sks.spodhuis.org v6=[2a02:898:31:0:48:4558:73:6b73] v4=94.142.242.225
S # 2 6 4 oteiza.siccegge.de v6=[2a01:4a0:59:1000:223:9eff:fe00:100f]
v4=92.43.111.21
S # 3 6 prod00.keyserver.dca.witopia.net v6=[2606:9500:201:1::141]
S # 4 6 4 gpg.NebrWesleyan.edu v6=[2606:1c00:2802::b] v4=192.94.109.73
S # 5 6 4 d hufu.ki.iif.hu v6=[2001:738:0:600:216:3eff:fe02:42]
v4=193.224.163.43 (1s)
S # 6 6 4 gozer.rediris.es v6=[2001:720:418:caf1::8] v4=130.206.1.8
S # 7 4 ip-209-135-211-141.ragingwire.net v4=209.135.211.141
S # 8 4 mud.stack.nl v4=131.155.141.70
S # 9 4 ams.sks.heypete.com v4=51.15.53.138
S # 10 4 host-37-191-238-78.lynet.no v4=37.191.238.78
S # 11 4 cryptonomicon.mit.edu v4=18.9.60.141
OK
2017-01-18 02:56:23 dirmngr[9098] listening on socket
'/home/dkg/tmp/tmp.nchsng7MNY/gpg/S.dirmngr'
2017-01-18 02:56:23 dirmngr[9099.0] permanently loaded certificates: 0
2017-01-18 02:56:23 dirmngr[9099.0] runtime cached certificates: 0
2017-01-18 02:56:23 dirmngr[9099.0] failed to open cache dir file
'/home/dkg/tmp/tmp.nchsng7MNY/gpg/crls.d/DIR.txt': No such file or directory
2017-01-18 02:56:23 dirmngr[9099.0] creating directory
'/home/dkg/tmp/tmp.nchsng7MNY/gpg/crls.d'
2017-01-18 02:56:23 dirmngr[9099.0] new cache dir file
'/home/dkg/tmp/tmp.nchsng7MNY/gpg/crls.d/DIR.txt' created
2017-01-18 02:56:24 dirmngr[9099.6] handler for fd 6 started
2017-01-18 02:56:24 dirmngr[9099.6] connection from process 9096 (1000:1000)
2017-01-18 02:56:24 dirmngr[9099.6] DBG: dns: libdns initialized (tor mode)
2017-01-18 02:56:25 dirmngr[9099.6] DBG: dns:
getsrv(_pgpkey-https._tcp.hkps.pool.sks-keyservers.net) -> 0 records
2017-01-18 02:56:25 dirmngr[9099.6] DBG: dns: libdns initialized (tor mode)
2017-01-18 02:56:27 dirmngr[9099.6] DBG: dns:
resolve_dns_name(hkps.pool.sks-keyservers.net): Success
2017-01-18 02:56:28 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:28 dirmngr[9099.6] resolve_dns_addr for
'hkps.pool.sks-keyservers.net': 'sks.spodhuis.org'
2017-01-18 02:56:28 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:29 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:29 dirmngr[9099.6] resolve_dns_addr for
'hkps.pool.sks-keyservers.net': 'oteiza.siccegge.de'
2017-01-18 02:56:29 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:29 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:29 dirmngr[9099.6] resolve_dns_addr for
'hkps.pool.sks-keyservers.net': 'prod00.keyserver.dca.witopia.net'
2017-01-18 02:56:29 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:30 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:30 dirmngr[9099.6] resolve_dns_addr for
'hkps.pool.sks-keyservers.net': 'gpg.NebrWesleyan.edu'
2017-01-18 02:56:30 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:31 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): No name
2017-01-18 02:56:31 dirmngr[9099.6] resolve_dns_addr failed while checking
'hkps.pool.sks-keyservers.net': No name
2017-01-18 02:56:32 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:32 dirmngr[9099.6] resolve_dns_addr for
'hkps.pool.sks-keyservers.net': 'hufu.ki.iif.hu'
2017-01-18 02:56:32 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:33 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:33 dirmngr[9099.6] resolve_dns_addr for
'hkps.pool.sks-keyservers.net': 'gozer.rediris.es'
2017-01-18 02:56:33 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:34 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Connection
closed in DNS
2017-01-18 02:56:34 dirmngr[9099.6] resolve_dns_addr failed while checking
'hkps.pool.sks-keyservers.net': Connection closed in DNS
2017-01-18 02:56:35 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:35 dirmngr[9099.6] resolve_dns_addr for
'hkps.pool.sks-keyservers.net': 'ip-209-135-211-141.ragingwire.net'
2017-01-18 02:56:35 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:36 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:36 dirmngr[9099.6] resolve_dns_addr for
'hkps.pool.sks-keyservers.net': 'hufu.ki.iif.hu' [already known]
2017-01-18 02:56:36 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:37 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:37 dirmngr[9099.6] resolve_dns_addr for
'hkps.pool.sks-keyservers.net': 'gpg.NebrWesleyan.edu' [already known]
2017-01-18 02:56:37 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:38 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:38 dirmngr[9099.6] resolve_dns_addr for
'hkps.pool.sks-keyservers.net': 'mud.stack.nl'
2017-01-18 02:56:38 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:38 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:38 dirmngr[9099.6] resolve_dns_addr for
'hkps.pool.sks-keyservers.net': 'gozer.rediris.es' [already known]
2017-01-18 02:56:38 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:39 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:39 dirmngr[9099.6] resolve_dns_addr for
'hkps.pool.sks-keyservers.net': 'sks.spodhuis.org' [already known]
2017-01-18 02:56:39 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:40 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:40 dirmngr[9099.6] resolve_dns_addr for
'hkps.pool.sks-keyservers.net': 'oteiza.siccegge.de' [already known]
2017-01-18 02:56:40 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:41 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:41 dirmngr[9099.6] resolve_dns_addr for
'hkps.pool.sks-keyservers.net': 'ams.sks.heypete.com'
2017-01-18 02:56:41 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:41 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:41 dirmngr[9099.6] resolve_dns_addr for
'hkps.pool.sks-keyservers.net': 'host-37-191-238-78.lynet.no'
2017-01-18 02:56:41 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:42 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:42 dirmngr[9099.6] resolve_dns_addr for
'hkps.pool.sks-keyservers.net': 'cryptonomicon.mit.edu'
2017-01-18 02:56:42 dirmngr[9099.6] DBG: dns: resolve_dns_addr(): Success
2017-01-18 02:56:42 dirmngr[9099.6] DBG: http.c:connect_server: trying
name='hufu.ki.iif.hu' port=443
2017-01-18 02:56:45 dirmngr[9099.6] DBG: dns: resolve_dns_name(hufu.ki.iif.hu):
Connection closed in DNS
2017-01-18 02:56:45 dirmngr[9099.6] resolving 'hufu.ki.iif.hu' failed:
Connection closed in DNS
2017-01-18 02:56:45 dirmngr[9099.6] can't connect to 'hufu.ki.iif.hu': host not
found
2017-01-18 02:56:45 dirmngr[9099.6] error connecting to
'https://hufu.ki.iif.hu:443': Unknown host
2017-01-18 02:56:45 dirmngr[9099.6] marking host 'hufu.ki.iif.hu' as dead
2017-01-18 02:56:45 dirmngr[9099.6] DBG: http.c:connect_server: trying
name='oteiza.siccegge.de' port=443
2017-01-18 02:56:46 dirmngr[9099.6] DBG: dns:
resolve_dns_name(oteiza.siccegge.de): Success
2017-01-18 02:56:46 dirmngr[9099.6] can't connect to 'oteiza.siccegge.de':
Permission denied
2017-01-18 02:56:46 dirmngr[9099.6] error connecting to
'https://oteiza.siccegge.de:443': Permission denied
2017-01-18 02:56:46 dirmngr[9099.6] command 'KS_GET' failed: Permission denied
2017-01-18 02:56:46 dirmngr[9099.6] handler for fd 6 terminated
2017-01-18 02:56:46 dirmngr[9099.6] handler for fd 6 started
2017-01-18 02:56:46 dirmngr[9099.6] connection from process 9101 (1000:1000)
2017-01-18 02:56:46 dirmngr[9099.6] handler for fd 6 terminated

Thanks for the report. I can replicate this.

What you see are bogus subkey binding signatures. The clean function only
worked on user ID packets and their self signatures. A comment in the code
stated this. However, I see no reason why we should not remove those bogus
signatures.
Commit 3563237 does this now.

Thanks.

Thanks for the dumps. I was not able to get the permission denied errors. The
ENETDOWN errors on my site where due to a IPv6 not being enabled for Tor (during
most of my older tests I used the Torbrowser).

Anyway, With the patches from today, things are working much better now.
Please give it a try.

I'm assuming it is. Feel free to reopen this bug if this still causes problems
for you.

I tried to reproduce this problem, and failed. Can you provide more information
about your build environment, and how to reproduce this problem?

I would also be fine just to add -L to any call to /bin/pwd in our tests. Note
that most tests are in tests/openpgp, and that set of tests changed radically
since 2.1.7, and the new version should not be affected (tests/ and tests/pkits
are mostly stubs anyway).

In the man page of gpg(1) it says:

--export-options parameters

export-minimal
    Export  the  smallest key possible. This removes all signatures except the

most recent self-signature on each user ID. [..]

If you actually do this to my key however, you will see that the resulting
output contains Nicolas' signature on my key, which is against what
"export-minimal" is supposed to do.

This is a bug tracker, not a support forum. If you need commercial support for
GnuPG, we are happy to provide that.

If you want to use GnuPG in your application, it is *strongly* recommended to
use GPGME. Please see https://wiki.gnupg.org/APIs

Note that each of these outputs is preceded by a KEY_CONSIDERED lined (for the
same key). Since the TOFU conflict information is per key, I'd expect an
implementation to say: Oh, there is already some conflict information for key X.
This must be a more up to date version, so I'll delete that first instead of
appending to it. Is this an unreasonable expectation?

In my Opinion it is. There is a technical, (i guess) unintentional, reason for
the multiple outputs, they
don't convey useful information. So I would consider this Output a Bug and
implementations
working like you describe it to be a workaround for that bug.

Getting firs wrong information and later updating it with the correct
information makes implementations
more complicated and error prone and currently is not handled in GPGME.

Also in GPGME we just want to figure out the TOFU Info for all the UID's of the
key used
to check the signature. We don't want information about conflicting keys. We need
a reliable way to filter this out. So I have a patch that ignores all TOFU_USER
lines
that don't match the fingerprint of the signature but still that breaks because
the "Update"
is not handled.

Please describe exactly what you mean. I can't see from this report why it and
what does not work or leak information.

KEY_CONSIDERED is orthogonal to the TOFU stats. Thus GPGME thus not evaluate it
to learn about the TOFU state.

Attached example output after patch is applied. Now User4 has full validity like
expected, and the debug output shows a match for User4's email address (NOTE:
the debug output has 'YES' for no match and 'NO' for successful match)

D406: 944_example.patch

Attached example patch prevents escaping normal lowercase letters.

Note that this isn't a general solution, though it does solve the issue for me.
For example, some email addresses have numbers (I don't know if having backslash
before numbers is an issue like it is for letters)

Attached example are the following setup:

user1 tsign user2 with full trust, depth 1, domain="customer.com". User2 signs
user3 through user5 (regular signatures). User4 is at customer.com, users 3 and
5 are at example.com.

I had a chance to run this test against 2.1.17 today as that version has been recently
released via homebrew. The error is essentially the same, but the debug output is
indeed slightly different. Now the debug line numbers are different and there is the
addition of string=''; in the debug output. I didn't notice anything else that looked
significantly different.

/tmp$ gpg2 --verify TrueTimeStamp-certificate-4793.txt
gpg: Signature made Wed Nov 23 23:08:29 2016 PST
gpg: using DSA key 0x6F3B2E6AB748A8F8
gpg: Good signature from "TrueTimeStamp <signing-department@TrueTimeStamp.org>"
[marginal]
gpg: DBG: tofu.c:3068: strtoul failed for TOFU DB data; returned string (string='';
tail=''): Invalid argument
gpg: DBG: tofu.c:3070: strtoul failed for TOFU DB data; returned string (string='';
tail=''): Invalid argument
gpg: signing-department@truetimestamp.org: Verified 1 signature in the past

5 weeks.  Encrypted 0 messages.

gpg: Warning: we've only seen one message signed using this key and user id!
gpg: Warning: you have yet to encrypt a message to this key!
gpg: Warning: if you think you've seen more signatures by this key and user

id, then this key might be a forgery!  Carefully examine the email address
for small variations.  If the key is suspect, then use
  gpg --tofu-policy bad 83289060F40DED088CF246B56F3B2E6AB748A8F8
to mark it as being bad.

gpg: WARNING: This key is not certified with sufficiently trusted signatures!
gpg: It is not certain that the signature belongs to the owner.
Primary key fingerprint: 8328 9060 F40D ED08 8CF2 46B5 6F3B 2E6A B748 A8F8