Backported to LIBGCRYPT-1-7-BRANCH

I have now pushed a change to Libgcrypt master to implement auto-extending of
secre memory pools. Commit b6870cf but there are two cother commits which this
is based upon. My test shows that I can now decrypt a message encrypted to the
test-hugekey.key.

I will port this back to Libgcrypt 1.7.

Which version of GnuPG are you using? Do you have scdaemon?

I will try out the idea of extending the secmem pool even if that means no mlock.

ah right, "ulimit -l" says 64 (kbytes) on my Linux system as well. According to
mlock(2) that's since kernel 2.6.9.

So i think it's worth adopting the supplied patch as a workaround at least (i
can confirm that it resolves the specific use case described in T2857 (dkg on Dec 05 2016, 05:47 PM / Roundup)), and i
agree with you that we should extend libgcrypt to extend secure memory allocation.

it's not clear to me that swap is outside the trust boundary anyway these days,
and modern systems should prefer encrypted swap where possible.

The secmem has two goals:

• Avoid swapping out tehse pages. Thus the mlock.
• Making sure that on free the memory is zeroized.

mlock requires root privileges and thus a special init sequence is required
(install as setuid(root) and gpg-agent drops the privileges direct after
allocating and mlocking the secmem). In the old times, and probably still today
on non-Linux platforms, this is still required. However, Linux turned to
allowing any process to mlock a certain amount (64k on my box).

I tend to suggest that we extend Libgcrypt to extend the secure memory
allocation by not using mlocked memory but keeping the the seroization feature.
The second option from T2857 (wk on Dec 05 2016, 07:11 PM / Roundup).

is the only goal of the secure memory to keep the RAM from being written to
swap, or are there other goals of secure memory? why is it unlikely that a new
block of memory can be mlock'd? what are the consequences of the new block not
being mlock'd? will it still be treated as secure memory?

crashing in the event that we run out of secure memory is simply not acceptable
these days, especially in a model where we have persistent long-term daemons
that people expect to remain running.

I just posted 0001-agent-Respect-enable-large-secmem.patch to gnupg-devel:

https://lists.gnupg.org/pipermail/gnupg-devel/2016-December/032285.html

D400: 927_0001-agent-Respect-enable-large-secmem.patch

Thanks! I tried reproducing this issue with your tofu.db (using HEAD), but I
didn't see the warning:

$ gpg --verify /tmp/TrueTimeStamp-certificate-4793.txt
gpg: Signature made Thu 24 Nov 2016 08:08:29 AM CET
gpg: using DSA key 6F3B2E6AB748A8F8
gpg: Good signature from "TrueTimeStamp <signing-department@TrueTimeStamp.org>"
[marginal]
gpg: signing-department@truetimestamp.org: Verified 2 signatures in the past

12 days.  Encrypted 0 messages.

gpg: Warning: you have yet to encrypt a message to this key!
gpg: Warning: if you think you've seen more signatures by this key and user

id, then this key might be a forgery!  Carefully examine the email address
for small variations.  If the key is suspect, then use
  gpg --tofu-policy bad 83289060F40DED088CF246B56F3B2E6AB748A8F8
to mark it as being bad.

gpg: WARNING: This key is not certified with sufficiently trusted signatures!
gpg: It is not certain that the signature belongs to the owner.
Primary key fingerprint: 8328 9060 F40D ED08 8CF2 46B5 6F3B 2E6A B748 A8F8

Most likely, this is because when you verifies the message, the error was fixed.
Can you confirm this for me by trying to reproduce the error with your current
tofu.db? If there is no error, could you send me a copy of the tofu.db from
before the initial verification?

Thanks!

Already fixed in 4db9a425644dccaf81b51ebc97b32a9cc21941a4. Duplicate of T2848.

This is a duplicate of bug 2848

Yeah, I saw the Debian bug report. Unfortunately there is no easy
solution to this except for rejecting the use of large secret keys.

The problem here is that the big number library needs to allocate from
a limited secure memory region (32 KiB by default) and terminates on
allocation failure. I know that this is sub-optimal but we are doing
this for 19 years now. Checking for an error after each low-level big
number operation would make the code unreadable and will introduce
bugs. Ideas on what to do:

• On secure memory allocation failure, call the out-of-memory handler which may then free other memory (or purchase new memory). This can be done in the application.
• On secure memory allocation failure, allocate a new block of secure memory and allocate from that one. There are two disadvantages: a) It is unlikely that the new block can be mlock'd. b) A free will be a a little bit slower because it needs to check the list of secure memory blocks and not just one address range. The address range check is needed so that we can figure out whether the freed address is in the secmem range and needs to zeroed out. This requires a new Libgcrypt version, though no ABI change.
• We have a ./configure option --enable-large-secmem which sets 64k instead of 32k aside for the secmem. This is currently only used in gpg to enable gpg's --enable-large-rsa option. Given that in 2.1 we use the gpg-agent for the secret key operations we should have the same options in gpg-agent. However, it is only a kludge, but one we once agreed upon to silence some pretty vocal experts on key size.

I just tried with latest gpgol (beta 204) and it now seems to work. So bug is
solved already! :)

fwiw, i'm seeing this too, over at https://bugs.debian.org/846953 , for a user
with an insanely large (10240-bit) RSA key when it is locked with a passphrase.

I'm attaching such an example secret key (with passphrase "abc123"), and you can
trigger the crash with:

gpg --batch --yes --import test-hugekey.key
echo test | gpg -r 861A97D02D4EE690A125DCC156CC9789743D4A89

--encrypt --armor --trust-model=always --batch --yes --output data.gpg

            gpg --decrypt data.gpg

While i think it's fair to say that we need to have some limits on the sizes of
keys we can handle, gpg-agent should not crash when asked to deal with
extra-large keys, it should fail gracefully and return a sensible error code.

The only viable solution will be to export the key secret key after key
generation, append that to the %secring given file and delete the key from
gpg-agent's store. Recall that the agent needs to know the secret key so that
gpg is abale to create the self-signatures. Adding a dedicated cache for this
would complicate the gpg-agent code a lot.

Little update with latest 1.0.0 release:

– nothing new regarding pinentry-gnome3, still working nicely;
– nothing new regarding pinentry-gtk-2, but I know why it doesn’t take into
account my dead keys anymore and it’s not an issue on pinentry side;
– thanks to the new “show passphrase” button, I’ve been able to figure where the
issue lies with pinentry-qt: while invoked in the terminal, it does take into
account my dead keys, but while invoked via Thunderbird/Enigmail, it does not
(altought pinentry-gnome3 does).

So I suppose this is in fact an issue with Enigmail… Any hints on what they
could be doing wrong so that I can report this to them?

It's been a year since last update. Still an issue. Maybe someone should send
an email to gnupg-commits-owner@gnupg.org ?

tofu.db sent via encrypted email today.

In general, parallel operations aren't great, but I find that such bad
performance surprising.

If you update a key, only that key's effective policy is rechecked, not all
keys. But, the effective policy of conflicting keys is always rechecked.

Hi,

I think that your assumption is that the local keyring is somehow trusted. In

that case, I think it make sense that deleted keys would clear conflicts.

No, not really I don't think trust plays a role here. It's just a way I think
users may resolve conflicts when they don't know about policies or how things
work internally.

I'm curious when you think people delete keys. My intuition is that it is not

a very common pattern. Do you have any thoughts on this?

As an example: You get a new lock in your front door. Would you remove the key
for the old lock from your keyring or would you paint the old one red as a
marker not to use the old key.

I know that this is not totally applicable because the old key can still be used
for verification etc. But I think that this is the intuitive behavior if a key
changes.

Maybe if GUI offers conflict resolution better this might not be such a big deal
but currently (Kleopatra does not yet have conflict resolution) but for me
during tofu testing I thought I could resolve a the conflict by deleting one of
the keys and found the behavior unexpected.

I encourage you to first try and find a consensus before implementing a

different policy at the higher level.

Indeed. Let's try :-)

No need to apologize for the dup; I was just noting it here for the record.

I think that your assumption is that the local keyring is somehow trusted. In
that case, I think it make sense that deleted keys would clear conflicts.

I'm curious when you think people delete keys. My intuition is that it is not a
very common pattern. Do you have any thoughts on this?

I encourage you to first try and find a consensus before implementing a
different policy at the higher level.

Sorry for the duplicated bug. Although the other issue is older I got more
response here so I keep the discussion here.

In my Optionion it's completely natural for a User to think (I thought this):

• Oh I have two keys that are in conflict: I'll delete the bad one then I don't

have a conflict anymore.

This is very intuitive behavior.

I'm not looking for a solution that works for me but for a solution that I think
would work for other users.

So for me your response ("what you should do") would mean that in Kleopatra on
Key deletion I would need to check for conflicting keys and change their policy
to auto again. Maybe even mark the deleted key as bad before deletion. I would
much prefer it if GnuPG handled this. For me it seems just like an unhandled
state as the error messages also indicate. "Key not found" etc so It's a bug or
maybe missing feature / functionality.

Fwiw I don't see how this can be consistent with WoT behavior as I don't think
WoT has a comparable problem. Can you explain what a comparable problem in WoT is?

If you meant hat the validity of all keys is not updated immediately after key
deletion, and you had some ownertrust to the deleted key ok yes thats probably
also another issue. :-)

Duplicate of T2859

Thanks for reporting this! Unfortunately, I'm not able to reproduce this. I
hope you can help me figure out what is wrong. Would you be willing to share
your tofu.db with me? Feel free to send it to me directly
(8F17777118A33DDA9BA48E62AACB3243630052D9); it contains some privacy sensitive
information (namely, who you communicate with).

Thanks!

This issue has also been reported in https://bugs.gnupg.org/gnupg/Issue2859

Werner replied there and I agree with his conclusion.

Note: this is a dup of T2742

I tend to agree with Werner: if we discover a conflict, it needs to be resolved
and deleting a key is not a sufficient resolution.

Another user reported the same problem on IRC. It seems it is Arch Linux
specific but we don't known for sure. The latest test with re-building
Libgcrypt w/o any special options didn't changed anything.

I need top be able to replicate the problem before I can come up with a solution.

That is consistent with the WoT behaviour. Deleting a key is no solution to a
faked key. It might be re-imported as any time.

What you should do instead is to disable the key so that it won't be used again.

While testing with tofu enabled I sometimes see that some actions take very
long. (>1minute)

Like importing a key in Kleopatra where Kleopatra does an import and starts a
keylist afterwards / in parallel.

I'll try to reproduce this on the command line. Just doing a simple import on
the command line is quick.

Do you have any hint what can take so long?
Like a trigger that would cause a rechecks for cross signatures?

Thanks for your fast reply.
Sadly I have not much time these days... but I have done what you suggested.
Honestly the log files dont tell me much. One thing I recognised is sometimes the logfiles end with "Fatal: libgcrypt
problem: out of core in secure memory" and sometimes they dont (I have not copied every log file here, this might
be to much).

This is what I got with debug memstat and the recent version of gnupg in Arch.

2016-11-30 21:18:35 gpg-agent[5516] listening on socket '/run/user/1000/gnupg/S.gpg-agent'
2016-11-30 21:18:35 gpg-agent[5517] gpg-agent (GnuPG) 2.1.15 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe1d0e5700 for fd 5 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe1c8e4700 for fd 6 started
2016-11-30 21:18:45 gpg-agent[5517] starting a new PIN Entry
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe17fff700 for fd 8 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe177fe700 for fd 9 started
2016-11-30 21:18:51 gpg-agent[5517] handler 0x7efe1d0e5700 for fd 5 terminated
2016-11-30 21:18:52 gpg-agent[5517] Fatal: out of core in secure memory while allocating 512 bytes

2016-11-30 21:18:52 gpg-agent[5517] Fatal: libgcrypt problem: out of core in secure memory
2016-11-30 21:18:35 gpg-agent[5516] listening on socket '/run/user/1000/gnupg/S.gpg-agent'
2016-11-30 21:18:35 gpg-agent[5517] gpg-agent (GnuPG) 2.1.15 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe1d0e5700 for fd 5 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe1c8e4700 for fd 6 started
2016-11-30 21:18:45 gpg-agent[5517] starting a new PIN Entry
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe17fff700 for fd 8 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe177fe700 for fd 9 started
2016-11-30 21:18:51 gpg-agent[5517] handler 0x7efe1d0e5700 for fd 5 terminated
2016-11-30 21:18:52 gpg-agent[5517] Fatal: out of core in secure memory while allocating 512 bytes

2016-11-30 21:18:52 gpg-agent[5517] Fatal: libgcrypt problem: out of core in secure memory
v2016-11-30 21:18:35 gpg-agent[5516] listening on socket '/run/user/1000/gnupg/S.gpg-agent'
2016-11-30 21:18:35 gpg-agent[5517] gpg-agent (GnuPG) 2.1.15 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe1d0e5700 for fd 5 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe1c8e4700 for fd 6 started
2016-11-30 21:18:45 gpg-agent[5517] starting a new PIN Entry
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe17fff700 for fd 8 started
2016-11-30 21:18:45 gpg-agent[5517] handler 0x7efe177fe700 for fd 9 started
2016-11-30 21:18:51 gpg-agent[5517] handler 0x7efe1d0e5700 for fd 5 terminated
2016-11-30 21:18:52 gpg-agent[5517] Fatal: out of core in secure memory while allocating 512 bytes

2016-11-30 21:18:52 gpg-agent[5517] Fatal: libgcrypt problem: out of core in secure memory

This is what I got with debuging memstat and downgraded gnupg to gnupg-2.1.15-2-x86_64.pkg.tar.xz

2016-11-30 21:28:30 gpg-agent[5953] listening on socket '/run/user/1000/gnupg/S.gpg-agent'
2016-11-30 21:28:30 gpg-agent[5954] gpg-agent (GnuPG) 2.1.15 started
2016-11-30 21:28:37 gpg-agent[5954] handler 0x7fd6fedca700 for fd 5 started
2016-11-30 21:28:37 gpg-agent[5954] starting a new PIN Entry
2016-11-30 21:28:42 gpg-agent[5954] handler 0x7fd6fedca700 for fd 5 terminated
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fedca700 for fd 5 started
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fe5c9700 for fd 7 started
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fddc8700 for fd 8 started
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fd5c7700 for fd 9 started
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fedca700 for fd 5 terminated
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fddc8700 for fd 8 terminated
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fe5c9700 for fd 7 terminated
2016-11-30 21:28:47 gpg-agent[5954] handler 0x7fd6fd5c7700 for fd 9 terminated
2016-11-30 21:29:32 gpg-agent[5954] handler 0x7fd6fe5c9700 for fd 7 started
2016-11-30 21:29:32 gpg-agent[5954] handler 0x7fd6fe5c9700 for fd 7 terminated
2016-11-30 21:30:10 gpg-agent[5954] handler 0x7fd6fe5c9700 for fd 5 started
2016-11-30 21:30:10 gpg-agent[5954] handler 0x7fd6fd5c7700 for fd 7 started
2016-11-30 21:30:10 gpg-agent[5954] handler 0x7fd6fddc8700 for fd 8 started
2016-11-30 21:30:10 gpg-agent[5954] handler 0x7fd6fedca700 for fd 9 started
2016-11-30 21:30:10 gpg-agent[5954] Fatal: out of core in secure memory while allocating 512 bytes

2016-11-30 21:30:10 gpg-agent[5954] Fatal: libgcrypt problem: out of core in secure memory

And this is what I got from gdb (I am quite unfamiliar with gdb, so maybe i done something wrong)

Reading symbols from gpg-agent...(no debugging symbols found)...done.
Attaching to program: /usr/bin/gpg-agent, process 3492
Reading symbols from /usr/lib/libgcrypt.so.20...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libgpg-error.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libassuan.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libnpth.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Reading symbols from /usr/lib/libc.so.6...(no debugging symbols found)...done.
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
0x00007f05452cd18c in pselect () from /usr/lib/libc.so.6
(gdb) break log_fatal
Function "log_fatal" not defined.
Make breakpoint pending on future shared library load? (y or [n])
(gdb) c
Continuing.
[New Thread 0x7f053ffff700 (LWP 4687)]
[New Thread 0x7f05449ea700 (LWP 4698)]
[New Thread 0x7f053f7fe700 (LWP 4699)]
[New Thread 0x7f05451eb700 (LWP 4700)]
[Thread 0x7f053ffff700 (LWP 4687) exited]
[Thread 0x7f053f7fe700 (LWP 4699) exited]
[Thread 0x7f05451eb700 (LWP 4700) exited]
[Thread 0x7f05449ea700 (LWP 4698) exited]
[New Thread 0x7f05449ea700 (LWP 4733)]
[New Thread 0x7f05451eb700 (LWP 4745)]
[New Thread 0x7f053f7fe700 (LWP 4746)]
[New Thread 0x7f053ffff700 (LWP 4747)]
[Thread 0x7f053f7fe700 (LWP 4746) exited]
[Thread 0x7f05449ea700 (LWP 4733) exited]
[Thread 0x7f05451eb700 (LWP 4745) exited]
[Thread 0x7f053ffff700 (LWP 4747) exited]
[New Thread 0x7f053ffff700 (LWP 4775)]
[New Thread 0x7f05451eb700 (LWP 4776)]
[Thread 0x7f053ffff700 (LWP 4775) exited]

Thread 11 "gpg-agent" received signal SIGPIPE, Broken pipe.
[Switching to Thread 0x7f05451eb700 (LWP 4776)]

0x00007f054559a16d in write () from /usr/lib/libpthread.so.0

This should be fixed in: 2f27cb12e30c9f6e780354eecc3ff0039ed52c63 .

Applied to 2.0, too. Will be in 2.0.31.

Fixed in 2.1.16.

Fixed in 2.1.11 and 2.0.30.

Fixed in 2.1.16.

Fixed in 2.1.16. Will be in 2.0.31 as the fix is in the git repo already.

Fixed with nPth 1.3.

Fixed in 1.3.

Fixed with nPth 1.3.

Fixed in STABLE-BRANCH-2-0 branch of git repo, as of the commit:
5c599e4f6edd288f4759c9fc2bcf9fe87dee1836

While looking at the problem I found a corner case related to a shutdown and
fixed that.

I also tried to close the listening socket after the first shutdown event. I
reverted that because the effect is that a client trying to connect immediately
gets a failure and then starts a new dirmngr - which is not the idea of a shutdown.

The man pages notes:

SIGTERM

   Shuts down the process but waits until all current requests are
   fulfilled.  If the process has received 3  of these signals
   and requests are still pending, a shutdown is forced.  You may
   also use
         gpgconf --kill dirmngr
   instead of this signal

thus this is by design and identical to what gpg-agent does. IIRC, there was a
regression for some time, fixed in 2.1.16. So this fixed regression is what you
see as a bug.

However, the process should not anymore listen for new connections as soon as a
shutdown is pending. That needs to be fixed.

Yeah, lets do that. Commit 8489b12 to go into 2.1.17. Thanks.

commit a5910e00ace882b8a17169faf4607163ab454af9 should fix that. Will go into
2.1.17.