Backported to LIBGCRYPT-1-7-BRANCH

I have now pushed a change to Libgcrypt master to implement auto-extending of
secre memory pools. Commit b6870cf but there are two cother commits which this
is based upon. My test shows that I can now decrypt a message encrypted to the
test-hugekey.key.

I will port this back to Libgcrypt 1.7.

I will try out the idea of extending the secmem pool even if that means no mlock.

ah right, "ulimit -l" says 64 (kbytes) on my Linux system as well. According to
mlock(2) that's since kernel 2.6.9.

So i think it's worth adopting the supplied patch as a workaround at least (i
can confirm that it resolves the specific use case described in T2857 (dkg on Dec 05 2016, 05:47 PM / Roundup)), and i
agree with you that we should extend libgcrypt to extend secure memory allocation.

it's not clear to me that swap is outside the trust boundary anyway these days,
and modern systems should prefer encrypted swap where possible.

The secmem has two goals:

• Avoid swapping out tehse pages. Thus the mlock.
• Making sure that on free the memory is zeroized.

mlock requires root privileges and thus a special init sequence is required
(install as setuid(root) and gpg-agent drops the privileges direct after
allocating and mlocking the secmem). In the old times, and probably still today
on non-Linux platforms, this is still required. However, Linux turned to
allowing any process to mlock a certain amount (64k on my box).

I tend to suggest that we extend Libgcrypt to extend the secure memory
allocation by not using mlocked memory but keeping the the seroization feature.
The second option from T2857 (wk on Dec 05 2016, 07:11 PM / Roundup).

is the only goal of the secure memory to keep the RAM from being written to
swap, or are there other goals of secure memory? why is it unlikely that a new
block of memory can be mlock'd? what are the consequences of the new block not
being mlock'd? will it still be treated as secure memory?

crashing in the event that we run out of secure memory is simply not acceptable
these days, especially in a model where we have persistent long-term daemons
that people expect to remain running.

I just posted 0001-agent-Respect-enable-large-secmem.patch to gnupg-devel:

https://lists.gnupg.org/pipermail/gnupg-devel/2016-December/032285.html

D400: 927_0001-agent-Respect-enable-large-secmem.patch

Yeah, I saw the Debian bug report. Unfortunately there is no easy
solution to this except for rejecting the use of large secret keys.

The problem here is that the big number library needs to allocate from
a limited secure memory region (32 KiB by default) and terminates on
allocation failure. I know that this is sub-optimal but we are doing
this for 19 years now. Checking for an error after each low-level big
number operation would make the code unreadable and will introduce
bugs. Ideas on what to do:

• On secure memory allocation failure, call the out-of-memory handler which may then free other memory (or purchase new memory). This can be done in the application.
• On secure memory allocation failure, allocate a new block of secure memory and allocate from that one. There are two disadvantages: a) It is unlikely that the new block can be mlock'd. b) A free will be a a little bit slower because it needs to check the list of secure memory blocks and not just one address range. The address range check is needed so that we can figure out whether the freed address is in the secmem range and needs to zeroed out. This requires a new Libgcrypt version, though no ABI change.
• We have a ./configure option --enable-large-secmem which sets 64k instead of 32k aside for the secmem. This is currently only used in gpg to enable gpg's --enable-large-rsa option. Given that in 2.1 we use the gpg-agent for the secret key operations we should have the same options in gpg-agent. However, it is only a kludge, but one we once agreed upon to silence some pretty vocal experts on key size.

fwiw, i'm seeing this too, over at https://bugs.debian.org/846953 , for a user
with an insanely large (10240-bit) RSA key when it is locked with a passphrase.

I'm attaching such an example secret key (with passphrase "abc123"), and you can
trigger the crash with:

gpg --batch --yes --import test-hugekey.key
echo test | gpg -r 861A97D02D4EE690A125DCC156CC9789743D4A89

--encrypt --armor --trust-model=always --batch --yes --output data.gpg

            gpg --decrypt data.gpg

While i think it's fair to say that we need to have some limits on the sizes of
keys we can handle, gpg-agent should not crash when asked to deal with
extra-large keys, it should fail gracefully and return a sensible error code.

Another user reported the same problem on IRC. It seems it is Arch Linux
specific but we don't known for sure. The latest test with re-building
Libgcrypt w/o any special options didn't changed anything.

I need top be able to replicate the problem before I can come up with a solution.

Thanks for your fast reply.
Sadly I have not much time these days... but I have done what you suggested.
Honestly the log files dont tell me much. One thing I recognised is sometimes the logfiles end with "Fatal: libgcrypt
problem: out of core in secure memory" and sometimes they dont (I have not copied every log file here, this might
be to much).

This is what I got with debug memstat and the recent version of gnupg in Arch.

2016-11-30 21:18:35 gpg-agent[5516] listening on socket '/run/user/1000/gnupg/S.gpg-agent'
2016-11-30 21:18:35 gpg-agent[5517] gpg-agent (GnuPG) 2.1.15 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe1d0e5700 for fd 5 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe1c8e4700 for fd 6 started
2016-11-30 21:18:45 gpg-agent[5517] starting a new PIN Entry
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe17fff700 for fd 8 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe177fe700 for fd 9 started
2016-11-30 21:18:51 gpg-agent[5517] handler 0x7efe1d0e5700 for fd 5 terminated
2016-11-30 21:18:52 gpg-agent[5517] Fatal: out of core in secure memory while allocating 512 bytes

2016-11-30 21:18:52 gpg-agent[5517] Fatal: libgcrypt problem: out of core in secure memory
2016-11-30 21:18:35 gpg-agent[5516] listening on socket '/run/user/1000/gnupg/S.gpg-agent'
2016-11-30 21:18:35 gpg-agent[5517] gpg-agent (GnuPG) 2.1.15 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe1d0e5700 for fd 5 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe1c8e4700 for fd 6 started
2016-11-30 21:18:45 gpg-agent[5517] starting a new PIN Entry
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe17fff700 for fd 8 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe177fe700 for fd 9 started
2016-11-30 21:18:51 gpg-agent[5517] handler 0x7efe1d0e5700 for fd 5 terminated
2016-11-30 21:18:52 gpg-agent[5517] Fatal: out of core in secure memory while allocating 512 bytes

2016-11-30 21:18:52 gpg-agent[5517] Fatal: libgcrypt problem: out of core in secure memory
v2016-11-30 21:18:35 gpg-agent[5516] listening on socket '/run/user/1000/gnupg/S.gpg-agent'
2016-11-30 21:18:35 gpg-agent[5517] gpg-agent (GnuPG) 2.1.15 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe1d0e5700 for fd 5 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe1c8e4700 for fd 6 started
2016-11-30 21:18:45 gpg-agent[5517] starting a new PIN Entry
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe17fff700 for fd 8 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe177fe700 for fd 9 started
2016-11-30 21:18:51 gpg-agent[5517] handler 0x7efe1d0e5700 for fd 5 terminated
2016-11-30 21:18:52 gpg-agent[5517] Fatal: out of core in secure memory while allocating 512 bytes

2016-11-30 21:18:52 gpg-agent[5517] Fatal: libgcrypt problem: out of core in secure memory

This is what I got with debuging memstat and downgraded gnupg to gnupg-2.1.15-2-x86_64.pkg.tar.xz

2016-11-30 21:28:30 gpg-agent[5953] listening on socket '/run/user/1000/gnupg/S.gpg-agent'
2016-11-30 21:28:30 gpg-agent[5954] gpg-agent (GnuPG) 2.1.15 started
2016-11-30 21:28:37 gpg-agent[5954] handler 0x7fd6fedca700 for fd 5 started
2016-11-30 21:28:37 gpg-agent[5954] starting a new PIN Entry
2016-11-30 21:28:42 gpg-agent[5954] handler 0x7fd6fedca700 for fd 5 terminated
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fedca700 for fd 5 started
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fe5c9700 for fd 7 started
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fddc8700 for fd 8 started
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fd5c7700 for fd 9 started
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fedca700 for fd 5 terminated
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fddc8700 for fd 8 terminated
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fe5c9700 for fd 7 terminated
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fd5c7700 for fd 9 terminated
2016-11-30 21:29:32 gpg-agent[5954] handler 0x7fd6fe5c9700 for fd 7 started
2016-11-30 21:29:32 gpg-agent[5954] handler 0x7fd6fe5c9700 for fd 7 terminated
2016-11-30 21:30:10 gpg-agent[5954] handler 0x7fd6fe5c9700 for fd 5 started
2016-11-30 21:30:10 gpg-agent[5954] handler 0x7fd6fd5c7700 for fd 7 started
2016-11-30 21:30:10 gpg-agent[5954] handler 0x7fd6fddc8700 for fd 8 started
2016-11-30 21:30:10 gpg-agent[5954] handler 0x7fd6fedca700 for fd 9 started
2016-11-30 21:30:10 gpg-agent[5954] Fatal: out of core in secure memory while allocating 512 bytes

2016-11-30 21:30:10 gpg-agent[5954] Fatal: libgcrypt problem: out of core in secure memory

And this is what I got from gdb (I am quite unfamiliar with gdb, so maybe i done something wrong)

Reading symbols from gpg-agent...(no debugging symbols found)...done.
Attaching to program: /usr/bin/gpg-agent, process 3492
Reading symbols from /usr/lib/libgcrypt.so.20...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libgpg-error.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libassuan.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libnpth.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Reading symbols from /usr/lib/libc.so.6...(no debugging symbols found)...done.
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
0x00007f05452cd18c in pselect () from /usr/lib/libc.so.6
(gdb) break log_fatal
Function "log_fatal" not defined.
Make breakpoint pending on future shared library load? (y or [n])
(gdb) c
Continuing.
[New Thread 0x7f053ffff700 (LWP 4687)]
[New Thread 0x7f05449ea700 (LWP 4698)]
[New Thread 0x7f053f7fe700 (LWP 4699)]
[New Thread 0x7f05451eb700 (LWP 4700)]
[Thread 0x7f053ffff700 (LWP 4687) exited]
[Thread 0x7f053f7fe700 (LWP 4699) exited]
[Thread 0x7f05451eb700 (LWP 4700) exited]
[Thread 0x7f05449ea700 (LWP 4698) exited]
[New Thread 0x7f05449ea700 (LWP 4733)]
[New Thread 0x7f05451eb700 (LWP 4745)]
[New Thread 0x7f053f7fe700 (LWP 4746)]
[New Thread 0x7f053ffff700 (LWP 4747)]
[Thread 0x7f053f7fe700 (LWP 4746) exited]
[Thread 0x7f05449ea700 (LWP 4733) exited]
[Thread 0x7f05451eb700 (LWP 4745) exited]
[Thread 0x7f053ffff700 (LWP 4747) exited]
[New Thread 0x7f053ffff700 (LWP 4775)]
[New Thread 0x7f05451eb700 (LWP 4776)]
[Thread 0x7f053ffff700 (LWP 4775) exited]

Thread 11 "gpg-agent" received signal SIGPIPE, Broken pipe.
[Switching to Thread 0x7f05451eb700 (LWP 4776)]

0x00007f054559a16d in write () from /usr/lib/libpthread.so.0

gpg-agent sets 32k aside for so called secure memory. It seems Libgcrypt runs
out of memory during computations with private key parameters.

Please put "debug memstat" into gpg-agent.conf which should print two lines of
info at process termination. If possible do the same with the old version and
compare.

Another thing you can do is to start gpg-agent ("gpgconf --launch gpg-agent"),
then look for its PID and attach gdb:

  $ gpg gpg-agent PID
  gdb> break log_fatal
  gdb> c

after you hit the breakpoint enter "bt".

John is using 2.1.14, but this bug was fixed in 2.1.15.

This is apparently just re-reported on gnupg-users:

https://lists.gnupg.org/pipermail/gnupg-users/2016-October/056892.html

So i don't think it's fixed.

And fwiw, it seems like a clear bug to me if i use "ssh-add" and then it is not
added to the agent.

From the ssh-add's client's perspective, some keys are magically never added,
but others are. This kind of mystery behavior is confusing and frustrating. If
gpg-agent is going to handle the ssh-agent protocol, it should aim toward behave
as the user of the ssh-agent protocol expects, regardless of whether the user
knows that they're using gpg-agent or some other implementation.

Ah, I misunderstood your problem. In the future, please paste all program interactions in one chunk
in the right order. We did merge some changes related to exporting of secret keys, so it may very
well be solved by that.

Thanks for caring :)

I think the problem is that your key export fails, because you pointed
--homedir at the (presumably) empty directory "%tmp%\_tempKeyring".

The export did not use any filter and tried to export a key as can be seen in
Msg8313 "error receiving key from agent"
The import itself also stated no errors as it can be seen in T2355 (dranft on May 12 2016, 03:00 PM / Roundup), but this
imported secret key cannot be used (or exported) anymore.
Also important: This is no longer reproducible in 2.1.14 (which might be enough
to set the bug to fixed)

I don't believe this demonstrates a bug.

I think the problem is that your key export fails, because you pointed --homedir at the (presumably)
empty directory "%tmp%\_tempKeyring". This leads to the not very helpful error message about the
eof. If the export were successful, gpg would have written the key to stdout.

For reference, here is what I tried. First GNUPGHOME points to a home with the key I want to export:

    $ echo $GNUPGHOME
    /tmp/tmp.T7I4M9RIc3
    $ g10/gpg --list-keys alpha
    gpg: please do a --check-trustdb
    pub   dsa1024 1999-03-08 [SCA]
          A0FF4590BB6122EDEF6E3C542D727CC768697734
    uid           [ unknown] Alfa Test (demo key) <alfa@example.net>
    uid           [ unknown] Alpha Test (demo key) <alpha@example.net>
    uid           [ unknown] Alice (demo key)
    sub   elg1024 1999-03-08 [E]

You need some kind of pinentry program, because you may be asked for the current passphrase or an
export passphrase:

    $ cat $GNUPGHOME/gpg-agent.conf
    pinentry-program /usr/bin/pinentry-x11

Now export the key:

    $ g10/gpg --export-secret-keys alpha >/tmp/alpha.gpg

Now I create an empty home, and import the key in batch mode:

    $ export GNUPGHOME=$(mktemp -d)
    $ g10/gpg --batch --import /tmp/alpha.gpg
    gpg: keybox '/tmp/tmp.bL2caQmZri/pubring.kbx' created
    gpg: /tmp/tmp.bL2caQmZri/trustdb.gpg: trustdb created
    gpg: key 2D727CC768697734: public key "Alfa Test (demo key) <alfa@example.net>" imported
    gpg: key 2D727CC768697734: secret key imported
    gpg: Total number processed: 3
    gpg:               imported: 1
    gpg:       secret keys read: 3
    gpg:   secret keys imported: 2

Could you please check if that works for you?

I do consider it a bug, at least because we did not signal an error to ssh-add.
Fortunately, this was easy to fix.

Fixed in 270f7f7b.

Yes, that is very likely the same bug. Feel free to reopen this report if yuo
can still reproduce it, in which case a backtrace would be very handy.

Fixed in 28fd0ab.

Fixed in f4742493.

Sorry for going AWOL on this, Werner. Do you still need a backtrace from me, or is the
one from 2371 enough?

See bug 2371, which has a backtrace attached.

Duplicate of T2371

Thanks. I need a stack backtrace to find the location of the cause.
Please start gpg-agent using:

  gpg-connect-agent /bye

The figure out the PID of the gpg-agent process and run

  gdb /usr/local/bin/gpg-agent PID

At the gdb prompt enter

  handle SIGPIPE nostop noprint pass
  c

The "c" continues operation of gpg-agent. In another terminal run

  gpg2 --sign

as done in your example. GDB in the first terminal will eventually
stop due to the assert. Enter at the gdb prompt:

  bt

and post the output. I would also like to know which version of
libgpg-error you are using:

  gpg-error --version

should show this (or use gpg-error-config --version).

PS: forget the --homedir thing, it is even reprodicable in the default folder in
%appdata%.

Sorry, forgot my import cmdline:

C:\Program Files (x86)\GNU\GnuPG\2.1.12\bin>gpg --batch --homedir
%tmp%\_tempKeyring --import "P:\2EEC2B65A2B4B3EF.sec.asc"
gpg: Die "Keybox" `C:/Users/ranftd/AppData/Local/Temp/_tempKeyring/pubring.kbx'
wurde erstellt
gpg: C:/Users/ranftd/AppData/Local/Temp/_tempKeyring/trustdb.gpg: trust-db erzeugt
gpg: Schlüssel A2B4B3EF: Öffentlicher Schlüssel "Daniel Ranft (Giegerich &
Partner GmbH)" importiert
gpg: Schlüssel A2B4B3EF: "Daniel Ranft (Giegerich & Partner GmbH)" nicht geändert
gpg: Schlüssel A2B4B3EF: geheimer Schlüssel importiert
gpg: Anzahl insgesamt bearbeiteter Schlüssel: 4
gpg: importiert: 1
gpg: unverändert: 1
gpg: gelesene geheime Schlüssel: 3
gpg: unveränderte geh. Schl.: 2
gpg: keine ultimativ vertrauenswürdigen Schlüssel gefunden

I think I was confused by the fact that I didn't use ssh-add to add the key and
I didn't realize that I could add it manually to sshcontrol. I did that and it
now works as expected. Sorry about the noise.

Although maybe it would be nice to be able to make 'confirm' the default for
keys which are not listed in sshcontrol. But that's a very minor thing.

I would not consider this a bug. sshcontrol is used to enable certain keys for
use with ssh. Updating keys is useless if they are already available.

If you remove the keys from sshcontrol you disable them. I would suggest to put
a '!' in front of the keygrip instead of deleting the line in sshcontrol. This
allows to re-enable a key w/o problems.

The solution is to remove the key in private-keys-v1.d before running ssh-add.

http://superuser.com/a/1064269/216912

gpg-agent does disable core dumps both in the stable and modern version.
Furthermore I have to agree with Werner here, if there is a process that can
ptrace your gpg-agent, then you have already lost anyway.

I've tested to generate an rsa2048 key with backup on a v2.0 card and it works
now. I have not tested restoring from backup etc. But as this report was about
the failed generation, this issue is resolved imo.

Thanks!

I removed the not-working checkbkupkey subcommand in
44aee35e69540510617aea4b886ef845590960fe

Also fixed the bkuptocard subcommand in: 40959add1ba0efc1f4aa87fa075fa42423eff73c

Fixed in
http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=commit;h=b30c15bf7c5336c4abb1f9dcd974cd77ba6c61a7

I'm considering fixing this.

Keep the bug open. We won't fix it for the next release.

werner: What is your call to action? Should pinentry always be shutdown or is
the status quo acceptable? Thanks.

I am pretty sure that there is a race. The two sessions are not mutally locked
and thus the second client may ask for the passprase again.

  Pinentry-1 returns to Client-1
  task switch
  Client-2 now checks the cache again, does not see an update, starts Pinentry-2
  task switch
  Client-1 updates the cache

How ever it will be a rare problem and it would at best be annoying.

I tried running:

  echo | gpg -s -a

in two terminal. In the first terminal, I got a pinentry prompt (I'm using
pinentry-tty) and in the second, gpg2 appeared to freeze. Once I entered my
passphrase correctly in the first terminal, the first gpg2 process split out the
signed message and less than a second later, the second did as well.

The code also suggests that this is fixed (agent/findkey.c)

      /* If the pinentry is currently in use, we wait up to 60 seconds
         for it to close and check the cache again.  This solves a common
         situation where several requests for unprotecting a key have
         been made but the user is still entering the passphrase for
         the first request.  Because all requests to agent_askpin are
         serialized they would then pop up one after the other to
         request the passphrase - despite that the user has already
         entered it and is then available in the cache.  This
         implementation is not race free but in the worst case the
         user has to enter the passphrase only once more. */

Interestingly, this comment is from 2006 (commit: df52700f), which predates this
bug report.

As such, I'm changing this bug's status to needs-eg. Perhaps Werner can shed
some more light on this issue.

Please describe which certifciates you see, which version of GnuPG you are
using, and what error you get when trying to delete a cert.

Are you using an Ed25519 key? There was a regression in 2.1.8 which has
meanwhile be fixed in the repo. See also T2096.

c) Run gpg-agent under gdb
d) Run a modified gpg-agent (rm ~/S.gpg-agent; my-gpg-agent --daemon)
e) Hook into the tty and use pinentry-curses
f) scp ~/.gnupg/private-keys-v1.d/* mybox: and sniff the passphrase.

so far, the proposed mechanisms for getting at gpg-agent's memory from a peer
process running as the same user are:

a) ptrace (e.g. via /usr/bin/gcore or /usr/bin/strace)
b) /proc/$PID/mem, which is owned by the user and mode 0600

DarkStarSword's patch effectively closes (a) (by rejecting ptrace connections)
and appears on my GNU/Linux system to close (b) as well: /proc/$PID/mem is
root-owned when the patch is applied instead of being user-owned.

Are there other channels for per-process memory access that we should be
thinking about?

I agree with Werner and Neal that the UNIX model is probably insufficient to
close all the holes easily, but i also don't think that's a good reason to avoid
closing those holes we can close.

If there are other ways that another process by the same user can get at the
RAM, please point them out and i'll look into ways to address them too.

In the meantime, i'll also look into ways to facilitate running the process as a
separate user account entirely.

I am closing this.

BTW: I can't share DarkStarSword's fear about prioritizing ease of debugging
over security - I would never do that for a real security problem; Neal and me
both explained why this proposed fix does can't help against an attack.