We need a way to replicate your problem, a few questions first:

Did you run gpg with the option -v (or --verbose) to show more detailed information about the failures? For example timeouts?

How many gpg processes did you run in parallel and how did you start them?

I made a large file for testing, but it doesn't matter. There's an arbitrary parallel limit where gpg will crash.

# Make a large? file
dd if=/dev/urandom of=largefile bs=1024 count=1024

# Encrypt it:
gpg --output encrypted.gpg --encrypt --recipient 'your@email.com' largefile

# Decrypt it a few times:
for i in {1..10}; do gpg --use-agent --no-tty --verbose -o - encrypted.gpg > /dev/null & done

Then you get lots of

gpg: public key decryption failed: Cannot allocate memory
gpg: decryption failed: No secret key
[7]   Exit 2                  gpg --use-agent --no-tty --verbose -o - encrypted.gpg > /dev/null

Running with -v would really be helpful.

for i in {1..10}; do gpg  -v --no-tty --verbose -o - encrypted.gpg 2>mylog.$i > /dev/null &  done

How do the mylog.N differ, can you post a sample? Did you notice for a short time a file "gnupg_spawn_agent_sentinel.lock" ~/.gnupg ? Before you test make sure no agent is running (gpgconf --kill gpg-agent)

I am unable to replicate this on OS X 10.9 Mavericks.

In the interests of completeness I also tried it on a much larger file (1GB) which was both signed and encrypted. I also set the decryption to show the session key just to confirm it was decrypting since the plaintext was being sent to /dev/null.

This continued without any trouble.

I ran gpgconf --kill gpg-agent and then the suggested command for i in {1..10}; do gpg -v --no-tty --verbose -o - encrypted.gpg 2>mylog.$i > /dev/null & done. (I was already running with --verbose, does -v add something else?)

Here's an example of a full successful output:

gpg: WARNING: no command supplied.  Trying to guess what you mean ...
# off=0 ctb=85 tag=1 hlen=3 plen=524
:pubkey enc packet: version 3, algo 1, keyid xxxx
    data: [4095 bits]
gpg: public key is xxxx
gpg: using subkey xxxx instead of primary key yyyy
gpg: public key encrypted data: good DEK
# off=527 ctb=d2 tag=18 hlen=2 plen=0 partial new-ctb
:encrypted data packet:
    length: unknown
    mdc_method: 2
gpg: using subkey xxxx instead of primary key yyyy
gpg: encrypted with 4096-bit RSA key, ID xxxx, created 2017-01-01
      "My Name <email@email.com>"
gpg: AES256 encrypted data
# off=548 ctb=a3 tag=8 hlen=1 plen=0 indeterminate
:compressed packet: algo=2
# off=550 ctb=ae tag=11 hlen=5 plen=1048591
:literal data packet:
    mode b (62), created 1537652119, name="largefile",
    raw data: 1048576 bytes
gpg: original file name='largefile'
gpg: decryption okay

And here's an example of a failed output:

gpg: WARNING: no command supplied.  Trying to guess what you mean ...
# off=0 ctb=85 tag=1 hlen=3 plen=524
:pubkey enc packet: version 3, algo 1, keyid xxxx
    data: [4095 bits]
gpg: public key is xxxx
gpg: using subkey xxxx instead of primary key yyyy
gpg: pinentry launched (96504 unknown 0.9.7 ? ? ?)
# off=527 ctb=d2 tag=18 hlen=2 plen=0 partial new-ctb
:encrypted data packet:
    length: unknown
    mdc_method: 2
gpg: using subkey xxxx instead of primary key yyyy
gpg: encrypted with 4096-bit RSA key, ID xxxx, created 2017-01-01
      "My Name <email@email.com>"
gpg: public key decryption failed: Cannot allocate memory
gpg: decryption failed: No secret key

When running htop during this process, I see some mild CPU spikes, but memory does not drastically spike, at least according to htop. It stayed at roughly 8/17gb usage.

I don't know if it's relevant but so far this has been 100% reproducible. I'm a machine that has lots of other processes running as well, but it's not currently resource constrained.

Please put

auto-expand-secmem

into gpg-agent.conf. It should work with your versions of gnupg and libgcrypt. See also T3530 .

@werner what size of each additionally allocated secure memory area would you recommend? Is this something, that is better to set or leave up to the gpg-agent to decide? Will this additional memory be freed when not needed anymore or will it stay allocated until the process dies? I guess, the documentation could be expanded to answer this.

Allow Libgcrypt to expand its secure memory area as required. The optional value n is a non-negative integer with a suggested size in bytes of each additionally allocated secure memory area. The value is rounded up to the next 32 KiB; usual C style prefixes are allowed. For an heavy loaded gpg-agent with many concurrent connection this option avoids sign or decrypt errors due to out of secure memory error returns.