I just double checked the code from 2.1 - it looks really okay.
I need to look at the 2.x branch, though.

FWIW, with commit 19545e3a from 2015-09-09 I had bumped the limit up to 20MiB.
This should solve all current practical problems.

Test data from: http://keyserver.borgnet.us/dump/sks-dump-0000.pgp.bz2

In one console window:
mkdir c:\test-issue2135
set GNUPGHOME=c:\test-issue2135
gpg2 --import c:\users\aheinecke\Desktop\sks-dump-0000.pgp

in another:
set GNUPGHOME=c:\test-issue2135
gpg2 -k

Triggers this: (And the error messages also look wrong)

gpg: waiting for lock c:/test-issue2135/pubring.gpg.lock...
gpg: renaming c:/test-issue2135/pubring.gpg' to c:/test-issue2135/pubring.bak'
failed: Permission
denied
gpg: error writing keyring `c:/test-issue2135/pubring.gpg': Permission denied
gpg: key CBB511F4: public key "[User ID not found]" imported
gpg: error reading `c:\\Users\\aheinecke\\Desktop\\sks-dump-0000.pgp':
Permission denied
gpg: import from `c:\\Users\\aheinecke\\Desktop\\sks-dump-0000.pgp' failed:
Permission denied
gpg: Total number processed: 278
gpg: w/o user IDs: 14
gpg: imported: 265 (RSA: 82)
gpg: renaming c:/test-issue2135/pubring.gpg' to c:/test-issue2135/pubring.bak'
failed: Permission
denied
gpg: failed to rebuild keyring cache: Permission denied
gpg: no ultimately trusted keys found

werner: Are you suggesting that there are programs that use an ambiguous
specification in order to delete just one key? I'm trying to imagine how this
is not undesirable behavior. Do you have any examples of this?

neal: What you describe are two a different things/bugs.

1. Indicating that a specification is ambigious. We do this at other places as well

2. Continue to show (with -k) or ask (with --delete-key) the other matching keys.

For the later I don't think that there is a need to do this with --delete-key
because we would have already printed that the specification is ambigious.
However a "gpg -k shortkeyid" should list all matching short keyids. A quick
test with:

0F26563A76B8337A
D1EC50AA76B8337A

shows that this is not the case. Note that we need to take care when modifiying
--delete-key. Not all programs using this are well written but implicity assume
the current behavious. And unfortunately many authors do not use --with-colons
and rely on the human output. Not deleting is better than accidently deleting
something.

In this case I'm pretty sure that it does not. I check that I can come up with a
testcase that does not involve kleo.

Is Kleopatra messing around with files in ~/.gnupg in any way? IIRC, Kleo
sometimes bypasses gpgme. For example does it open pubring.gpg ?

No, keys were never moved. I have a test case ready, but I am trying to get a
hold of a newer AIX box to test it there. I will upload it shortly.

Ivan: Did you create the key on a fast machine and then moved it to a slower
one. Each use of a passphrases takes about 100ms for security reasons. Use
"gpg --passwd KEYID" to change the passphrase and adjust it to the speed of your
system.

A correctly installed system will start the agent either on the fly only the
first time or you need to start gpg-agent manually. See the man page for details.

May this be related to you fix for

if(ai->family != AF_INET6)

commit 23ea641b ?

Please run

  gpg --with-keygrip --with-fingerprint --with-fingerprint -K 30A99F9A

and

  gpg --with-keygrip --with-fingerprint --with-fingerprint -K 9BA84708

If one of the commands does not show a key run it again with -k
(lowercase). Also run

gpg --version

OK, based on my findings, I have to clarify that it is the environment of the
agent program that is cached, thus rendering the pinentry program oblivious to
the further changes of the environment, even though it is those changes that
"guide" it in the retrieval of the correct passphrase.

Attached is a test program. I apologize that it is TCSH, but that is what we use
in our shop (for portability). Running test.csh should produce an output like
this:

    > test.csh < /dev/null
    ---------------------------------------------------
     GPG version:
    ---------------------------------------------------
    gpg (GnuPG) 2.1.9
    libgcrypt 1.6.3
    Copyright (C) 2015 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    
    Home: /tmp/gpg-caching/.gnupg
    Supported algorithms:
    Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
    Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
            CAMELLIA128, CAMELLIA192, CAMELLIA256
    Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
    Compression: Uncompressed, ZIP, ZLIB, BZIP2
    
    ---------------------------------------------------
     Available keys:
    ---------------------------------------------------
    /tmp/gpg-caching/.gnupg/pubring.kbx
    -----------------------------------
    pub   dsa1024/E65831AC 2015-11-24
    uid         [ultimate] user1 <user1@email.com>
    sub   rsa2048/95F85202 2015-11-24
    
    pub   dsa1024/B0731F16 2015-11-24
    uid         [ultimate] user2 <user2@email.com>
    sub   rsa2048/67841B64 2015-11-24
    
    
    ---------------------------------------------------
     Key 1 decrypted using gpg2 directly:
    ---------------------------------------------------
    gpg: encrypted with 2048-bit RSA key, ID 95F85202, created 2015-11-24
          "user1 <user1@email.com>"
    32-bit-long key for user1 (ONE).
    
    ---------------------------------------------------
     Key 2 decrypted using gpg2 directly:
    ---------------------------------------------------
    gpg: encrypted with 2048-bit RSA key, ID 67841B64, created 2015-11-24
          "user2 <user2@email.com>"
    gpg: public key decryption failed: Broken pipe  
    gpg: decryption failed: No secret key
    
    
    ---------------------------------------------------
     Key 1 decrypted using gpgme:
    ---------------------------------------------------
    32-bit-long key for user1 (ONE).
    
    ---------------------------------------------------
     Key 2 decrypted using gpgme:
    ---------------------------------------------------
    Error while accessing key file user2.key: Decryption failed

This test creates two private/public key pairs in the same keyring; the password
to the private key of the first is "password1"; that of the second is
"password2". It then generates two files (user1.key and user2.key), one encrypted
with the first public key, the other with the second. The contents of those files
are "32-bit-long key for user1 (ONE)." and "32-bit-long key for user2 (TWO).",
respectively. The test also involves a simplistic pinentry program that returns
the value of $password to the agent. The point is to demonstrate that once the
gpg agent is started, it caches the environment, such that even updating
$password to the right value is of no use. And that is why only user1.key is
decrypted successfully (the agent was started to decrypt it).

Very conveniently in git:

Found using the Clang Static Analyzer.
Signed-off-by: Justus Winter <justus@g10code.com>

Please describe problems and do not just send a patch. You may however send a
patch to gnupg-devel.

Fixed with commit eb957ffc. Thanks.

However, this is not a problem because that value is not used.

If you check the do-while above you will notice that after the loop LINE is
guaranteed to always end with a LF. Thus strpbrk will always succeed.

Werner, in https://lists.gnupg.org/pipermail/gnupg-users/2015-May/053617.html you wrote:

The real bug is that dirmngr does not mark the v6 address dead and
retry anotyer server (or the v4 address).

I cannot reproduce this. I pointed dirnmngr to ipv6.pool.sks-keyservers.net and servers
got marked as dead as expected.

@guilhem, @dkg I've cc'd you on this since you seem to be interested in this code.

I've just updated the branch with a few small bug fixes. Most importantly, I
fixed the memory problem by limiting the read-ahead cache to 20 MB. @guilhem:
I'd be interested to hear whether this fixes the problem that you observed.

@bernhard Sorry for not getting back to you sooner. If you checkout neal/kdb,
you'll get the latest code for the kdb format. Set --homedir or GNUPGHOME
appropriately, import your keyring and then try some operations:

$ mkdir /tmp/gnupg-kbx

$ gpg2 --export | time gpg2 --homedir /tmp/gnupg-kbx --no-default-keyring

--keyring gnupg-kdb:pubring.kdb --import /tmp/keys 2>/dev/null

This makes 6.5 seconds on my box using the kdb format and just under 2 min using
kdb. See
https://lists.gnupg.org/pipermail/gnupg-devel/2015-November/030525.html for some
example benchmarks.

I'm particularly interested to hear whether this fixes the major performance
problem that you're experiencing.

Thanks.

Fixed in a9e0b1dd.

To be clear: the limitation is that GnuPG doesn't currently allow selecting the
main key and subkeys at the same time.

In b64b33b, I've added the ability to update multiple subkeys at once. Note: it
is still not possible to update the main key and the subkeys at the same time,
but this should be a significant improvement, I think.

I've tried to reproduce this issue by setting max-cache-ttl and default-
cache-ttl to 0 in my gpg-agent.conf and running echo | gpg2 -s -a multiple times
in a row. The behavior is such that I have to enter the password each time. I
also traced the control flow in gpg-agent using gdb. The point where the
password is cached is in agent/cache.c:agent_put_cache. The passed ttl is 0 and
opt.def.cache_ttl is also 0. Thus, the password is not cached in my experiments.

Can you please confirm using the most recent version of GnuPG (2.1.9 as of
today) that a gpg-agent.conf with just the above options set caches the password
for you? Does the problem only occur when you are using your gpgme-based
application?

Thanks!

Thank you for quick response)

I couldn't build and test 2.1.9 right now, but bug is still here in 1.4.16 and
2.0.22.

I've created test script for this case:
https://gist.github.com/anton-ryzhov/a0dcfcaabe18fc6ad35e

Run ./gen.sh in some working folder and then try ./runtest.sh several times,
expire different subkeys, compare the result.

Ivan: it would be very helpful if you could create a small example demonstrating
the problem. Thanks!

Keep the bug open. We won't fix it for the next release.

As Werner stated, this appears to be a user support inquiry, which is better
answered elsewhere. As such, I'm marking this issue as resolved.

I'm marking this issue as resolved.

werner: What is your call to action? Should pinentry always be shutdown or is
the status quo acceptable? Thanks.

dkg: I've now applied your backport to the 1.4 branch. Sorry for not doing this
sooner. I believe that this now completely rectifies this issue. As such, I'm
marking this issue as resolve. Thanks.

It seems this now works. In fact the code suggests that it should have detected
this condition since at least 2013 so I don't know what the actual issue was.

$ echo | gpg2 --pinentry-mode=loopback -s -a
gpg: NOTE: THIS IS A DEVELOPMENT VERSION!
gpg: It is only intended for test purposes and should NOT be
gpg: used in a production environment or with production keys!
gpg: setting pinentry mode 'loopback' failed: Not supported

Werner notes:

There is a comment in mainproc that we need to sort the list of keys and try
them in an order to get a decryption key early. The other thing is about the
meta data for keys. It would be possible to add a priority to the private keys
and use them to prioritise the list of keys to try.

The error has change in 2.1 to:

  gpg: secret key "foo.gpg" not found: Not found

(i.e., it doesn't say Unknown system error any more.)

The fundamental issue is that the argument to --gen-revoke is not a filename,
but a user id (e.g., the key id). I've accordingly change the error message in
46e128d as follows:

$ gpg2 --output revoke.asc --gen-revoke foo.gpg

gpg: no secret key matches the search term "foo.gpg"

gp_ast: have you gotten a chance to try this?

I'm closing this as Werner think it is a problem with Fedora and the original
reporter hasn't suggested this is not the case.

As an additional point, the client max body size in nginx defaults to 1 MiB[0].
Currently no checking is done for larger request bodies for inclusion in the
keyserver pools. Apache does not have such a limit by default.

Reference:
[0] http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size

Fixed in eb54fca.

Kristian Fiskerstrand told me that the SKS keyservers currently have a 5 MB
limit for parsing incoming header, pre-merge.

This tool has now been marked as deprecated in the documentation.

Based on Werner's comment, this issue has been addressed. As such, I'm closing
this bug report.

I now see I misunderstood the problem description.

The point is that a user has a message that is encrypted to key X. After
receiving the message, he wants to allow another key (say Y) to decrypt the
message by adding a symmetrically encrypted data packet to the message for Y,
i.e., without reencrypting the whole thing.

The reporter wasn't to specify the secret key to use. Werner indicated that
--try-secret-key does what the reporter wants in 2.1, but that this won't be
backported to 2.0. As such, I'm marking this issue as resolved.

I reviewed this issue. I've identified three issues that the reporter is
complaining about:

• Can't create a key with a passphrase (this works)
• Can't import a key that is not protected by a passphrase (this works)
• Can't export a key without protecting it with a passphrase (this is not allowed)

I also moved my mouse between screens in my multi-head setup and gpg did not crash.

I'm marking this issue as resolved.

At least with 2.1.9, it is possible to create a key without a passphrase:

$ gpg2 --gen-key
gpg (GnuPG) 2.1.10-beta132; Copyright (C) 2015 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

gpg: NOTE: THIS IS A DEVELOPMENT VERSION!
gpg: It is only intended for test purposes and should NOT be
gpg: used in a production environment or with production keys!
Note: Use "gpg2 --full-gen-key" for a full featured key generation dialog.

GnuPG needs to construct a user ID to identify your key.

Real name: Empty
Email address: empty@testing.org
You selected this USER-ID:

    "Empty <empty@testing.org>"

Change (N)ame, (E)mail, or (O)kay/(Q)uit? o
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
Please enter the passphrase to
protect your new key
Passphrase:
Repeat:
You have not entered a passphrase - this is in general a bad idea!
Please confirm that you do not want to have any protection on your key.

Yes, protection is not needed
Enter new passphrase

[ye]? y

gpg: key BC364B3A marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, TOFU+PGP trust model
gpg: depth: 0 valid: 2 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 2u
gpg: next trustdb check due at 2015-11-18
pub rsa2048/BC364B3A 2015-11-18

Key fingerprint = 6766 A52A 3E04 F09B E6F7  F80C 920C 9361 BC36 4B3A

uid [ultimate] Empty <empty@testing.org>
sub rsa2048/906F39F0 2015-11-18

It is also possible to import a secret key that doesn't have a passphrase:

$ gpg --no-options --gen-key
gpg (GnuPG) 1.4.18; Copyright (C) 2014 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:

(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)

Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 1024
Requested keysize is 1024 bits
Please specify how long the key should be valid.

   0 = key does not expire
<n>  = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years

Key is valid for? (0) 10
Key expires at Sat 28 Nov 2015 01:21:43 PM CET
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:

    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"

Real name: Empty Passphrase
Email address:
Comment:
You selected this USER-ID:

    "Empty Passphrase"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
You need a Passphrase to protect your secret key.

You don't want a passphrase - this is probably a *bad* idea!
I will do it anyway. You can change your passphrase at any time,
using this program with the option "--edit-key".

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

Not enough random bytes available. Please do some other work to give
the OS a chance to collect more entropy! (Need 201 more bytes)
...+++++
..+++++
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
.............+++++
.+++++
gpg: unable to use unknown trust model (7) - assuming PGP trust model
gpg: key 4240CFD8 marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: public key of ultimately trusted key BC364B3A not found
gpg: public key of ultimately trusted key 41A7057B not found
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 3 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 3u
gpg: next trustdb check due at 2015-11-28
pub 1024R/4240CFD8 2015-11-18 [expires: 2015-11-28]

Key fingerprint = 4E0D 8EED 3567 4228 7F44  C7D7 92BE 30B6 4240 CFD8

uid Empty Passphrase
sub 1024R/D6CF583D 2015-11-18 [expires: 2015-11-28]

$ gpg --no-options --export-secret-key 4240CFD8 > 4240CFD8.sec
$ gpg2 --import 4240CFD8.sec
gpg: NOTE: THIS IS A DEVELOPMENT VERSION!
gpg: It is only intended for test purposes and should NOT be
gpg: used in a production environment or with production keys!
gpg: key 4240CFD8: public key "Empty Passphrase" imported
gpg: key 4240CFD8: secret key imported
gpg: Total number processed: 3
gpg: imported: 1
gpg: secret keys read: 3
gpg: secret keys imported: 2
gpg: 3 marginal(s) needed, 1 complete(s) needed, TOFU+PGP trust model
gpg: depth: 0 valid: 3 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 3u
gpg: next trustdb check due at 2015-11-18
$ gpg2 -K 4240CFD8
gpg: NOTE: THIS IS A DEVELOPMENT VERSION!
gpg: It is only intended for test purposes and should NOT be
gpg: used in a production environment or with production keys!
sec rsa1024/4240CFD8 2015-11-18 [expires: 2015-11-28]
uid [ultimate] Empty Passphrase
ssb rsa1024/D6CF583D 2015-11-18 [expires: 2015-11-28]

I'm going to close this. The right forum to address these issues is the OpenPGP
working group.

As I understand the problem, a key appeared in multiple keyrings and this was
causing confusion. I don't think there is a bug here so I'm marking this issue
as resolved.

Fixed in 1e3dbb15.

cd2d685 fixes the assert. I don't see the utility of checking keyid (gpg will
do that). Closing.

Based on Werner's comment, I'm changing this to nobug and marking the issue as
resolved.

Based on Werner's comment, I'm marking this bug as resolved.

(At least) 2.1.9 should support version 3 (see dirmngr/ks-engine-ldap.c:492).
If this is still not working, please reopen this bug. Thanks.

Fixed in 84ebf15. Thanks!

Fixed in 84ebf15. Thanks!

This seems to work with 2.1.9. As such, I'm marking this issue as resolved.

$ gpg2 --list-options no-show-unusable-subkeys -k 4F43C989
pub rsa1024/4F43C989 2015-11-17
uid [ unknown] Testing <testing@testing.com>
sub rsa1024/3CAD33EE 2015-11-17
sub rsa1024/FE39BBA1 2015-11-17

$ gpg2 --list-options show-unusable-subkeys -k 4F43C989
pub rsa1024/4F43C989 2015-11-17
uid [ unknown] Testing <testing@testing.com>
sub rsa1024/3CAD33EE 2015-11-17
sub rsa1024/FE39BBA1 2015-11-17
sub elg1024/A10351BD 2015-11-17 [revoked: 2015-11-17]

It looks like this problem has been fixed in the meantime. As such, I'm marking
this bug as resolved. Thanks.

$ gpg2 --with-fingerprint 4F43C989.txt
pub rsa1024/4F43C989 2015-11-17

Key fingerprint = A8D8 E9B9 D25D 6AB8 9997  AEE4 3817 872D 4F43 C989

uid Testing <testing@testing.com>
sub rsa1024/3CAD33EE 2015-11-17
sub rsa1024/FE39BBA1 2015-11-17
sub elg1024/A10351BD 2015-11-17
$ gpg2 --fingerprint 4F43C989
pub rsa1024/4F43C989 2015-11-17

Key fingerprint = A8D8 E9B9 D25D 6AB8 9997  AEE4 3817 872D 4F43 C989

uid [ unknown] Testing <testing@testing.com>
sub rsa1024/3CAD33EE 2015-11-17
sub rsa1024/FE39BBA1 2015-11-17
sub elg1024/A10351BD 2015-11-17

I've fixed this with commit 0b86c74 by making it possible to select keys using
the key id. Consider:

gpg> key 4BFA08E4

pub rsa4096/D21739E9

created: 2007-06-02  expires: 2016-01-21  usage: SC  
validity: unknown

sub rsa4096/21484CFF

created: 2007-06-02  expired: 2015-02-26  usage: E

sub* rsa2048/4BFA08E4

created: 2008-06-19  expires: 2016-01-21  usage: A

sub rsa4096/1BFDFA5C

created: 2013-03-12  expires: 2016-01-21  usage: S

sub rsa2432/0CA757FB

created: 2013-09-11  expires: 2016-09-14  usage:

sub ed25519/BD7CFAB5

created: 2014-11-07  expired: 2015-05-06  usage: A

sub rsa4096/14D5DA70

created: 2015-01-21  expires: 2016-01-21  usage: E

sub ed25519/BD7CFAB5

created: 2014-11-07  expired: 2015-05-06  usage: A

sub ed25519/BD7CFAB5

created: 2014-11-07  expired: 2015-05-06  usage: A

sub ed25519/BD7CFAB5

created: 2014-11-07  expired: 2015-05-06  usage: A

sub ed25519/BD7CFAB5

created: 2014-11-07  expired: 2015-05-06  usage: A

[ unknown] (1). Daniel Kahn Gillmor <dkg@fifthhorseman.net>
[ unknown] (2) Daniel Kahn Gillmor <dkg@openflows.com>
[ revoked] (3) Daniel Kahn Gillmor <dkg@astro.columbia.edu>
[ revoked] (4) Daniel Kahn Gillmor <dkg-debian.org@fifthhorseman.net>
[ unknown] (5) [jpeg image of size 3515]
[ unknown] (6) Daniel Kahn Gillmor <dkg@debian.org>
[ unknown] (7) Daniel Kahn Gillmor <dkg@aclu.org>

For what it is worth, this does not appear to be an issue for GnuPG 2.1.x. If
the specified home directory does not exist, GnuPG quickly fails:

$ gpg2 --homedir /tmp/gpg-temp --gen-key
gpg (GnuPG) 2.1.10-beta132; Copyright (C) 2015 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

gpg: NOTE: THIS IS A DEVELOPMENT VERSION!
gpg: It is only intended for test purposes and should NOT be
gpg: used in a production environment or with production keys!
gpg: keyblock resource '/tmp/gpg-temp/pubring.kbx': No such file or directory
Note: Use "gpg2 --full-gen-key" for a full featured key generation dialog.

GnuPG needs to construct a user ID to identify your key.

Real name: Foo
Name must be at least 5 characters long
Real name: Foobar
Email address:
You selected this USER-ID:

    "Foobar"

Change (N)ame, (E)mail, or (O)kay/(Q)uit? o
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: failed to create temporary file
'/tmp/gpg-temp/.#lk0x00000000017158f0.grit.10925': No such file or directory
gpg: can't connect to the agent: No such file or directory
gpg: agent_genkey failed: No agent running
Key generation failed: No agent running

I just tried following the steps using gpg2 (2.1.9) and I can't reproduce the
problem. It would be good if we had an exact sequence of commands that
reproduced the problem.