ftr, here is the thread I had in mind but couldn't recall above. @aheinecke is that your thinking, or a more pgp/mime bound mechanism as @dkg assumed?

@wiktor-k, "just extend the spec" doesn't necessarily work with existing clients, which might be surprised to find unexpected packets in the signature section of an e-mail. It seems more likely to me that they'd be able to handle (meaning: ignore) an unknown subpacket (as long as it's well-formed) than to handle additional packets. But all of these surmises require testing with existing clients, of course. Has anyone done any of that testing?

This is a nice idea and although it overlaps with Autocrypt it has other uses too: for example verification of signed files that can be vastly simplified (just get the file and the signature, no key fetching needed, downside: the key attached to the signature could be stale).

Ah, thanks for pointing out the subpacket option (i guess it could be hashed or unhashed). i don't think any of the subpackets currently defined in RFC4880 supports this use case -- but i guess you could mint a new one, or use a notation.

Werner said that it's possible in OpenPGP to also put the pubkey into the signature. (...) The nice advantage is that this will also work for files.

Hi @aheinecke, thanks for thinking about this, and thanks for tagging me here too. I'm definitely interested.

In T4513#132777, @Valodim wrote:

But searching on Keyservers is also in my opinion not a common use case for Kleopatra users.

Thanks for engaging constructively.

In T4513#132770, @aheinecke wrote:

Werner could you maybe at least check for an internet connection, I don't know how to do it on Linux but on Windows it's easy because windows has API for that.

But searching on Keyservers is also in my opinion not a common use case for Kleopatra users.

and by that bypassing all key source tracking as done by gpg. In any case searching by name or mail address on a keyserver should not be done - at least not by a GUI tool as used by non experienced users.

I agree that this is a tricky problem, but it should really be improved.

The problem is not to check whether there is a connection but on how to decide whether something is a pool or an explictly added single keyserver and how often should we try to connect or read from it. Without marking hosts as dead the auto search features won't work well.

@Valodim probably not so much as dirmngr might behave differently and not mark hosts as dead.

The proper solution is of course to use pkill instead of killall. SCNR.

I can attest to the "growing bit of popular lore": Roughly half the support requests I get to support@keys.openpgp.org boil down to an exchange of "it just doesn't work with a 'general error' message" -> "try killall dirmngr" -> "that did it". I have heard similar stories from @patrick from Enigmail users, and more than once heard people applying poweruser trickery like "I just have killall dirmngr in my resume.d".

Unusable v6 interfaces are now detected on Windows and then not used.

Ping.

There is a growing bit of popular lore in the GnuPG community that "when keyserver operations fail, you solve that problem with killall dirmngr." I believe this suggestion is potentially damaging (the long-running daemon may be in the middle of operations for a client that you don't know about), but i suspect it is circulating as advice because it resolves the situation outlined in this ticket. For whatever ephemeral reason, dirmngr gets stuck, and fails to notice that this situation has resolved itself.

I'm aware of you releasing an RC for comments, and i apologize for not catching this particular case earlier. As you know from T4607, i was even advocating for it. i didn't understand the full implications of the "import-then-clean" approach at the time, and was thinking it would only apply to the incoming material, not the stored material.

The code has comments why we do a first clean_key on the imported keyblock.

i've merged a variant of rGbe99eec2b105eb5f8e3759147ae351dcc40560ad into the GnuPG packaging in debian unstable as of version 2.2.17-3 to avoid the risks of data loss and signature verification failures. I'll revert it if i see the concern addressed upstream.

that pseudocode is strange to me -- it looks like you have (two) duplicate calls to clean_key (imported_keyblock) (though maybe i just don't know what .... means in this pseudocode).

You are partly right. I missed that we also do clean the original keyblock while updating a key. The code is

I think dropping import-clean from the default keyserver options is the right way to go. It is not clear what additional benefit import-clean provides given that we are already using self-sigs-only. And the idea of non-additive behavior to the local keyring when pulling from a keyserver is a deeply surprising change for multiple users i've talked to.

Ah, that makes sense, good catch. Seems this is just an issue of documentation, then.

We should put it of the agenda od the Brussesl summit in 3 weeks. I have a few ideas what we can do in gpg.

Given the recent problems with the keyservers, I expect that the keyserver feature will go away anyway and thus I do not think we will put any more effort into this. Thus I re-tag this as gpg 2.3.

If we only need it for backward compatibility, then the configuration in gpg.conf should *not* be overriding the preferred, forward-looking form of the configuration (in dirmngr.conf). If it is low priority to fix this, then there will be a generation of GnuPG users and toolchains which deliberately configure the value in gpg.conf instead of dirmngr.conf because they'll know that's the more robust way to do it.

We need --keyserver in gpg for just one reason: backward compatibility.

thanks for fixing that error message, @werner. As @Valodim points out in discusson about hagrid, a gpg.conf keyserver option (deprecated according to the documentation) overrides the dirmngr.conf keyserver option (not deprecated according to the documentation.

I doubt that we are going to implement this.

Thanks

I removed this specialized error message. Thanks for reporting.

This is particularly bad for users who have manually specified a given keyserver in dirmngr.conf, because even a transient failure in that keyserver will prevent them from any future keyserver requests until dirmngr decides that the "death" has worn off.

HTTP/1.1 spec, RFC 7230, Section 5.4, paragraph 2:
https://tools.ietf.org/html/rfc7230#section-5.4

Please be so kind and point me to the specs stating that you should put the IP address into Host:

It's up to GPG to send the Host header that shows the user's intent.

So in short you want:

1. Allow to specify a keyserver by IP without any DNS lookups.
2. When connecting via IP use the IP address for Host:.

There is a solution for it:

So, the keyserver operator had thrown in a hockeypuck server in the pool, causing this.. While the keyserver remains in the exclude list until confirmation it has been resolved, that explains the behavior and it has been made clear that separate software needs to use different names in the future.

@kristianf we talked about this on Saturday evening. Would you be so kind and have a quick look at the problem with the hu server?

Hi Werner and thanks for looking into this.

According to sks-keyservers.net both servers you mention run the very same software. Thus I would like to understand why you think they require the use of a legacy option.

It actually tries several servers but we need to set a limit because we need to cope with longer timeouts. Do you suggest to toggle between v4 and v6 addresses? That is if a v6 address fails, first try the next v4 address and it that fails, another v6 address, etc.

The problem is that the keyserver network is abused as free and
permanent data storage. We can't do much about it without larger
changes on the search capabilities of the keyservers. For more
information see the archives of the sks-devel list starting in July.

Ok. I was not aware that HKPS should already have the highest quality.

hkps pool really should be the most responsive, and it already requires clustered only servers for a couple of weeks to try to increase the responsiveness. Experience has shown that any keyserver with less than 3 nodes in a cluster should not be used towards end-users. But do you have any more debugging output as to the problem at hand?

We assume that this has meanwhile been fixed.

@elonsatoshi: Were you able to check this with 2.2.9 which has a fix for the resolver?

You know... I think connman and DNS have something to do with this. Connman does some weird DNS thing. And it auto-generates /etc/resolv.conf to use localhost as the DNS server.

Well, I'm able to reproduce this issue on Parabola. I was also get a different error when I turn off my vpn: `server indicated a failure```, but now I get the dns error again.

elonsatoshi@tyger ~> gpg -vvv --debug-level guru --search elonsatoshi@riseup.net
gpg: using character set 'utf-8'
gpg: enabled debug flags: packet mpi crypto filter iobuf memory cache memstat trust hashing ipc clock lookup extprog
gpg: DBG: [not enabled in the source] start
gpg: DBG: chan_3 <- # Home: /home/elonsatoshi/.gnupg
gpg: DBG: chan_3 <- # Config: [none]
gpg: DBG: chan_3 <- OK Dirmngr 2.1.23 at your service
gpg: DBG: connection to the dirmngr established
gpg: DBG: chan_3 -> GETINFO version
gpg: DBG: chan_3 <- D 2.1.23
gpg: DBG: chan_3 <- OK
gpg: DBG: chan_3 -> KEYSERVER --clear hkps://pgp.mit.edu/
gpg: DBG: chan_3 <- OK
gpg: DBG: chan_3 -> KS_SEARCH -- elonsatoshi@riseup.net
gpg: DBG: chan_3 <- ERR 167772876 Connection closed in DNS <Dirmngr>
gpg: error searching keyserver: Connection closed in DNS
gpg: keyserver search failed: Connection closed in DNS
gpg: DBG: chan_3 -> BYE
gpg: DBG: [not enabled in the source] stop
gpg: keydb: handles=0 locks=0 parse=0 get=0
gpg:        build=0 update=0 insert=0 delete=0
gpg:        reset=0 found=0 not=0 cache=0 not=0
gpg: kid_not_found_cache: count=0 peak=0 flushes=0
gpg: sig_cache: total=0 cached=0 good=0 bad=0
gpg: random usage: poolsize=600 mixed=0 polls=0/0 added=0/0
              outmix=0 getlvl1=0/0 getlvl2=0/0
gpg: rndjent stat: collector=0x0000000000000000 calls=0 bytes=0
gpg: secmem usage: 0/32768 bytes in 0 blocks
elonsatoshi@tyger ~> sudo rc-service openvpn stop
[sudo] password for elonsatoshi: 
 * WARNING: openvpn is already stopped
elonsatoshi@tyger ~> pidof openvpn
elonsatoshi@tyger ~> gpg -vvv --debug-level guru --search elonsatoshi@riseup.net
gpg: using character set 'utf-8'
gpg: enabled debug flags: packet mpi crypto filter iobuf memory cache memstat trust hashing ipc clock lookup extprog
gpg: DBG: [not enabled in the source] start
gpg: DBG: chan_3 <- # Home: /home/elonsatoshi/.gnupg
gpg: DBG: chan_3 <- # Config: [none]
gpg: DBG: chan_3 <- OK Dirmngr 2.1.23 at your service
gpg: DBG: connection to the dirmngr established
gpg: DBG: chan_3 -> GETINFO version
gpg: DBG: chan_3 <- D 2.1.23
gpg: DBG: chan_3 <- OK
gpg: DBG: chan_3 -> KEYSERVER --clear hkps://pgp.mit.edu/
gpg: DBG: chan_3 <- OK
gpg: DBG: chan_3 -> KS_SEARCH -- elonsatoshi@riseup.net
gpg: DBG: chan_3 <- ERR 167772876 Connection closed in DNS <Dirmngr>
gpg: error searching keyserver: Connection closed in DNS
gpg: keyserver search failed: Connection closed in DNS
gpg: DBG: chan_3 -> BYE
gpg: DBG: [not enabled in the source] stop
gpg: keydb: handles=0 locks=0 parse=0 get=0
gpg:        build=0 update=0 insert=0 delete=0
gpg:        reset=0 found=0 not=0 cache=0 not=0
gpg: kid_not_found_cache: count=0 peak=0 flushes=0
gpg: sig_cache: total=0 cached=0 good=0 bad=0
gpg: random usage: poolsize=600 mixed=0 polls=0/0 added=0/0
              outmix=0 getlvl1=0/0 getlvl2=0/0
gpg: rndjent stat: collector=0x0000000000000000 calls=0 bytes=0
gpg: secmem usage: 0/32768 bytes in 0 blocks

Any update on this?

I was able to reproduce it again. Maybe this bug depends on which keyserver in
the pool answers. The error is the same for Tor and non-Tor connections.

I don't know why, it is not repdroducible anymore.

Are you using tor? if so, is your tor daemon up and running, and actively
connecting to the outside world?

$ gpg2 --keyserver hkps://hkps.sks-keyservers.net --search-keys 2071B08A33BD3F06
gpg: no keyserver known (use option --keyserver)
gpg: keyserver search failed: No keyserver available

WHAT?! I just specified --keyserver!!!??

Relax. You forgot the '.pool' in the url.

:facepalm: ... Apparently I need more coffee -- persistent failure state in user
encountered.

Sorry for the noise.

I do wonder about fault-tolerance, though, if a e.g. non-responsive host creeps
in to the pool.

At any rate, this is mainly a cosmetic issue at this point, and this bug report
probably contains sufficient information to help someone who "encounters" the
condition to resolve the protocol error quickly.

For the records, the suggested way to kill dirmngr is

gpgconf --kill dirmngr

this makes sure that dirmngr will not be started if it is not running.

tl;dr: HKPS handler will die when used with non-HKPS hosts in a given pool.

I think dying is reasonable. Maybe it should return a nicer error
than 'general error' and it shouldn't take 10 seconds to figure out
the protocol error.

Using setup directions from
https://sks-keyservers.net/overview-of-pools.php I assumed that
configuring my GnuPG client to use ipv4.pool.sks-keyservers.net
would provide an appropriate response. It took me quite some time to
determine that HKPS is totally incompatible with the ipv4 (or other)
server pools.

This is further confused by the fact that an older version of the
GnuPG skeleton files which includes a clause with examples that mix
HKPS and hkp servers (skel may not necessarily be updated in a
user's directory):

Sorry about that. I think the current skeleton file is clearer on
this.

As a result, I kept encountering the errors reported in
T1792

I don't see a connection to this bug.

Here's a simple demonstration of the failure case

$ gpg2 --keyserver hkps://ipv4.pool.sks-keyservers.net --search-keys
2071B08A33BD3F06
gpg: error searching keyserver: General error
gpg: keyserver search failed: General error

Contrast with:

$ gpg2 --keyserver hkps://hkps.pool.sks-keyservers.net --search-keys
2071B08A33BD3F06
gpg: data source: https://mud.stack.nl:443
(1) NIIBE Yutaka (GnuPG Release Key) <gniibe@fsij.org>

 	  2048 bit RSA key 2071B08A33BD3F06, created: 2014-10-29, expires: 2020-10-30

PERSISTENT FAILURE CASE:
Now, once the failure condition is encountered, further queries FAIL:

$ pkill dirmngr

A nicer way to kill the dirmngr is:

  gpg-connect-agent --dirmngr 'killdirmngr' /bye

$ gpg2 --keyserver hkp://pool.sks-keyservers.net --search-keys 2071B08A33BD3F06
gpg: error searching keyserver: No route to host
gpg: keyserver search failed: No route to host

This is strange, and looks like it should work. Works over here. Maybe it is
bad luck and you got a bad host from the roundrobin.

$ gpg2 --keyserver hkps://hkps.sks-keyservers.net --search-keys 2071B08A33BD3F06
gpg: no keyserver known (use option --keyserver)
gpg: keyserver search failed: No keyserver available

WHAT?! I just specified --keyserver!!!??

Relax. You forgot the '.pool' in the url.

Let's see if this can be rectified with clearing the keyserver:

$ gpg-connect-agent --dirmngr keyserver
> keyserver --clear
OK

$ gpg2 --keyserver hkps://hkps.sks-keyservers.net --search-keys 2071B08A33BD3F06
gpg: no keyserver known (use option --keyserver)
gpg: keyserver search failed: No keyserver available

Likewise.

1. Try this with other VALID --keyserver combinations. Bang head against wall. The ONLY command that seems to fix this persistent failure case: $ gpg2 --search-keys 2071B08A33BD3F06 Suddenly, I can use --keyserver again, after this.

I'm pretty sure you just messed up the urls.

Sorry, I can't see any problem here.

The "priotr-old" key is actually the newer key because an expiration date was
added to that copy of the key (2012-07-09) and that key has meanwhile expired.
Thus you can't encrypt using this key.

When you import the "piotr" key that is actually the same key but w/o the update
with the expiration date. Thus gpg does not chnage the exiting in key because
the existing key has a newer self-signature (where the expiration date is
stored) than the new key. So nothing changes, which is correct.

If you delete the .gnupg directory you don't have the newer key and by importing
the key w/o the expiration date you can encrypt to that key.