GPG Agent fails in parallel: "gpg: decryption failed: No secret key"
Testing, NormalPublic

Description

Platform: Mac High Sierra

Problem:
Running multiple gpg commands in a terminal in parallel fail with "gpg: decryption failed: No secret key".

One manifestation of this issue is opening multiple shells at the same time. I have a shortcut to spawn a series of tabs, and then they all load, it tries to source my .bashrc in each tab, which tries to run a gpg command, and none of the new shells are set up properly.

gpg-agent --version
gpg-agent (GnuPG/MacGPG2) 2.2.7
libgcrypt 1.8.2

Details

Version
2.2.7
andy created this task.Sep 17 2018, 11:47 PM
werner added a subscriber: werner.

We need a way to replicate your problem, a few questions first:

Did you run gpg with the option -v (or --verbose) to show more detailed information about the failures? For example timeouts?

How many gpg processes did you run in parallel and how did you start them?

andy added a comment.Sep 22 2018, 11:43 PM

I made a large file for testing, but it doesn't matter. There's an arbitrary parallel limit where gpg will crash.

# Make a large? file
dd if=/dev/urandom of=largefile bs=1024 count=1024

# Encrypt it:
gpg --output encrypted.gpg --encrypt --recipient 'your@email.com' largefile

# Decrypt it a few times:
for i in {1..10}; do gpg --use-agent --no-tty --verbose -o - encrypted.gpg > /dev/null & done

Then you get lots of

gpg: public key decryption failed: Cannot allocate memory
gpg: decryption failed: No secret key
[7]   Exit 2                  gpg --use-agent --no-tty --verbose -o - encrypted.gpg > /dev/null

Running with -v would really be helpful.

for i in {1..10}; do gpg  -v --no-tty --verbose -o - encrypted.gpg 2>mylog.$i > /dev/null &  done

How do the mylog.N differ, can you post a sample? Did you notice for a short time a file "gnupg_spawn_agent_sentinel.lock" ~/.gnupg ? Before you test make sure no agent is running (gpgconf --kill gpg-agent)

BenM added a subscriber: BenM.Sep 26 2018, 10:45 PM

I am unable to replicate this on OS X 10.9 Mavericks.

BenM added a comment.Sep 26 2018, 10:58 PM

In the interests of completeness I also tried it on a much larger file (1GB) which was both signed and encrypted. I also set the decryption to show the session key just to confirm it was decrypting since the plaintext was being sent to /dev/null.

This continued without any trouble.

andy added a comment.Sep 26 2018, 11:09 PM

I ran gpgconf --kill gpg-agent and then the suggested command for i in {1..10}; do gpg -v --no-tty --verbose -o - encrypted.gpg 2>mylog.$i > /dev/null & done. (I was already running with --verbose, does -v add something else?)

Here's an example of a full successful output:

gpg: WARNING: no command supplied.  Trying to guess what you mean ...
# off=0 ctb=85 tag=1 hlen=3 plen=524
:pubkey enc packet: version 3, algo 1, keyid xxxx
    data: [4095 bits]
gpg: public key is xxxx
gpg: using subkey xxxx instead of primary key yyyy
gpg: public key encrypted data: good DEK
# off=527 ctb=d2 tag=18 hlen=2 plen=0 partial new-ctb
:encrypted data packet:
    length: unknown
    mdc_method: 2
gpg: using subkey xxxx instead of primary key yyyy
gpg: encrypted with 4096-bit RSA key, ID xxxx, created 2017-01-01
      "My Name <email@email.com>"
gpg: AES256 encrypted data
# off=548 ctb=a3 tag=8 hlen=1 plen=0 indeterminate
:compressed packet: algo=2
# off=550 ctb=ae tag=11 hlen=5 plen=1048591
:literal data packet:
    mode b (62), created 1537652119, name="largefile",
    raw data: 1048576 bytes
gpg: original file name='largefile'
gpg: decryption okay

And here's an example of a failed output:

gpg: WARNING: no command supplied.  Trying to guess what you mean ...
# off=0 ctb=85 tag=1 hlen=3 plen=524
:pubkey enc packet: version 3, algo 1, keyid xxxx
    data: [4095 bits]
gpg: public key is xxxx
gpg: using subkey xxxx instead of primary key yyyy
gpg: pinentry launched (96504 unknown 0.9.7 ? ? ?)
# off=527 ctb=d2 tag=18 hlen=2 plen=0 partial new-ctb
:encrypted data packet:
    length: unknown
    mdc_method: 2
gpg: using subkey xxxx instead of primary key yyyy
gpg: encrypted with 4096-bit RSA key, ID xxxx, created 2017-01-01
      "My Name <email@email.com>"
gpg: public key decryption failed: Cannot allocate memory
gpg: decryption failed: No secret key

When running htop during this process, I see some mild CPU spikes, but memory does not drastically spike, at least according to htop. It stayed at roughly 8/17gb usage.

I don't know if it's relevant but so far this has been 100% reproducible. I'm a machine that has lots of other processes running as well, but it's not currently resource constrained.

Please put

auto-expand-secmem

into gpg-agent.conf. It should work with your versions of gnupg and libgcrypt. See also T3530 .

werner triaged this task as Normal priority.Oct 10 2018, 12:10 PM
werner changed the task status from Open to Testing.