gpg-agent: "<gcrypt> Cannot allocate memory" with 10 threads decrypting OpenPGP
Closed, ResolvedPublic

Description

I'm setting this to needs triage as I'm not sure how important you think threading issues are. It is my opinion that as long as we don't hit system limits like max fd's or max memory the whole GnuPG System ( all components + GPGME ) should be able to handle parallel loads. And 10 operations is quite little. I can see Kleopatra starting 10 decrypts easily.

In this issue "gcrypt's Cannot allocate memory", if it is that limited the error should be handled internally or the number of currently running connections (not accepted connections) be limited.

To reproduce for me:

export GNUPGHOME=$(mktemp -d) 
gpg --yes --pinentry-mode loopback --passphrase "test" --quick-gen-key test@example.com
echo Hello | gpg -er test@example.com > $GNUPGHOME/pgp-enc
# unlock the key
gpg  --pinentry-mode loopback --passphrase "test" -d $GNUPGHOME/pgp-enc
./run-threaded --no-list --threads 10 --data-type 3 --repeat 10 $GNUPGHOME/pgp-enc

When I change the key to future-default it works for me.

Details

Version
STABLE-BRANCH-2-2

Forgot to mention. run-threaded is a new test tool in GPGME.

werner added a subscriber: werner.Nov 15 2018, 3:02 PM

Which Libgcrypt version?

werner triaged this task as High priority.Nov 15 2018, 4:07 PM
werner claimed this task.Mon, Dec 3, 3:05 PM

I can easily replicate this; it is a problem somewhere in the secure memory code of Libgcrypt.

gniibe added a subscriber: gniibe.Wed, Dec 12, 3:47 AM

For my case, with $GNUPGHOME/gpg-agent.conf having debug-all, I observed that rsa_decrypt failes with 'Cannot allocate memory', after debug output of 'res'.
Reading libgcrypt/cipher/rsa.c, it is line 1439, where it calls sexp_build (MPI of PLAIN into SEXP of R_PLAIN).
I think that it does indeed memory failure here.
Having "auto-expand-secmem" in gpg-agent.conf, it goes well.

Not a bug :-). I should have read my own docs before starting a long debug session. The things is that the auto expanding of the secmem area is only done for xmalloc_secure and the internal MPI allocation functions. It is not dne for any memory which is allocated with xtrymalloc becuase those properly return an error to the caller. The idea is that if the caller wants to get an error back he has also the assurance that them memory is allocated in the non-swappable memory (i.e. not in the expanded parts of the secmem).

The mitigation for your problem is to put

auto-expand-secmem

into gpg-agent.conf. Then the pool will we expanded for all types of allocation requests. I just ran a test with 100 threads and repeat 50 without problems. To trigger the problem earlier I also used --debug crypto, which does more allocations.

Uhm, if this option is useful why isn't it default behavior?

I would not know about this.