I'm going to write some documentation about the programmatic use of GnuPG.

Fixed for gpg as of 6b16b02109f4bb5b934e456667ff4c0ba7bc85fd.

--quick-keygen fixed in dd3dde07a9a46130ac01d849f8edf0566e44f11f.

The default expiration interval has been discussed on the mailing list. There
was a rough consensus on two years, which has been challenged by Neal who thinks
it is too short given the current state of the tools, but the ensuing discussion
did not revolve around the time span, so I'm keeping my two years for now. In
any case, it is easy to adjust.

I decided to not change the --full-key-gen, because a) the user asked for it, b)
changing that requires breaking up a large chunk of translated text, and I do
not want to do that right now (a release is imminent).

I just released Libgcrypt 1.7.4 - whcih should fix that bug.

Partially addressed in d568a1561642ed9b7b7b6282b86c56786d10a956.

Backported to LIBGCRYPT-1-7-BRANCH

I have now pushed a change to Libgcrypt master to implement auto-extending of
secre memory pools. Commit b6870cf but there are two cother commits which this
is based upon. My test shows that I can now decrypt a message encrypted to the
test-hugekey.key.

I will port this back to Libgcrypt 1.7.

Would you mind to write to gnupg-devel and ask for comments on your proposal?
In particular on how long the default expiration time shall be. 12, 18, or 24
months?

I will try out the idea of extending the secmem pool even if that means no mlock.

ah right, "ulimit -l" says 64 (kbytes) on my Linux system as well. According to
mlock(2) that's since kernel 2.6.9.

So i think it's worth adopting the supplied patch as a workaround at least (i
can confirm that it resolves the specific use case described in T2857 (dkg on Dec 05 2016, 05:47 PM / Roundup)), and i
agree with you that we should extend libgcrypt to extend secure memory allocation.

it's not clear to me that swap is outside the trust boundary anyway these days,
and modern systems should prefer encrypted swap where possible.

The secmem has two goals:

• Avoid swapping out tehse pages. Thus the mlock.
• Making sure that on free the memory is zeroized.

mlock requires root privileges and thus a special init sequence is required
(install as setuid(root) and gpg-agent drops the privileges direct after
allocating and mlocking the secmem). In the old times, and probably still today
on non-Linux platforms, this is still required. However, Linux turned to
allowing any process to mlock a certain amount (64k on my box).

I tend to suggest that we extend Libgcrypt to extend the secure memory
allocation by not using mlocked memory but keeping the the seroization feature.
The second option from T2857 (wk on Dec 05 2016, 07:11 PM / Roundup).

is the only goal of the secure memory to keep the RAM from being written to
swap, or are there other goals of secure memory? why is it unlikely that a new
block of memory can be mlock'd? what are the consequences of the new block not
being mlock'd? will it still be treated as secure memory?

crashing in the event that we run out of secure memory is simply not acceptable
these days, especially in a model where we have persistent long-term daemons
that people expect to remain running.

I just posted 0001-agent-Respect-enable-large-secmem.patch to gnupg-devel:

https://lists.gnupg.org/pipermail/gnupg-devel/2016-December/032285.html

D400: 927_0001-agent-Respect-enable-large-secmem.patch

Yeah, I saw the Debian bug report. Unfortunately there is no easy
solution to this except for rejecting the use of large secret keys.

The problem here is that the big number library needs to allocate from
a limited secure memory region (32 KiB by default) and terminates on
allocation failure. I know that this is sub-optimal but we are doing
this for 19 years now. Checking for an error after each low-level big
number operation would make the code unreadable and will introduce
bugs. Ideas on what to do:

• On secure memory allocation failure, call the out-of-memory handler which may then free other memory (or purchase new memory). This can be done in the application.
• On secure memory allocation failure, allocate a new block of secure memory and allocate from that one. There are two disadvantages: a) It is unlikely that the new block can be mlock'd. b) A free will be a a little bit slower because it needs to check the list of secure memory blocks and not just one address range. The address range check is needed so that we can figure out whether the freed address is in the secmem range and needs to zeroed out. This requires a new Libgcrypt version, though no ABI change.
• We have a ./configure option --enable-large-secmem which sets 64k instead of 32k aside for the secmem. This is currently only used in gpg to enable gpg's --enable-large-rsa option. Given that in 2.1 we use the gpg-agent for the secret key operations we should have the same options in gpg-agent. However, it is only a kludge, but one we once agreed upon to silence some pretty vocal experts on key size.

Thanks!

--quick-set-expire now available.

I'll take the --quick-set-expire command. -wk

The only viable solution will be to export the key secret key after key
generation, append that to the %secring given file and delete the key from
gpg-agent's store. Recall that the agent needs to know the secret key so that
gpg is abale to create the self-signatures. Adding a dedicated cache for this
would complicate the gpg-agent code a lot.

That is becuase we consider a mail address to be a unique indentifier and thus
and algorithm to figure out the best matching makes sense. Other kinds of user
IDS not not need to be unique and should at best return an ambigious key error.
Well, expired keys and such should be sorted out, though.

Let's use T2425 for the tar failure, and T2847 for the ssh failure. The
log you posted here shows exactly the same problem as in T2847.

Do you also see tar failing?

You can use

make -Ctests/openpgp check XTESTS="gpgtar.scm gpgtar.scm gpgtar.scm gpgtar.scm
gpgtar.scm"

to run the same test over and over again. That is how I measured how often we
see the failure. We updated our box since, and I haven't tried it again yet.

The ssh.scm failure is still happening intermittently with 2.1.16

https://bot.brew.sh/job/Homebrew%20Versions%20Pull%20Requests/1733/version=yosemite/console

$ ssh -V
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011

1.25 has been released.

Fixed in 40e5ff0a0084c0d9521b401db4f38885bfdae233.

For backward compatibily reasons.

Fixed in ab89164be02012f1bf159c971853b8610e966301.

I also don't quite understand why we restrict this to user ids resembling mail
addresses, so I'll keep issue this open for discussion.

I like this work, thanks for it! I wonder whether it would also be useful for
full-match userID, not only for a raw e-mail address?

For example, if i query for '=Peter Palfrader' or '=ssh://host.example', it
ought to give me the key with the highest-validity binding for the requested
user ID.

This is resolved. we have WITH_TOFU_INFO now.

The idea is to change the algorithm in the case that a full mail address is
given - and only a mail address. For both -r and --locate-key. g10/getkey.c
has get_pubkey_byname which implements --locate-key and already checks for a
mail address (IS_MBOX). This function needs to be changed to figure out all
matching keys an return the best one. -r should make use of that function also
if it is a mailbox.

We now have a macOS box, and are building our software on it using Jenkins.

On that box, I also see the gpgtar test failing in about 14% of all runs. There
is something to be learned here.

Please clarify the plan a bit. Shall we use the algorithm currently used by
--recipient, the one used by --locate-key, or implement a new one?

another item for consistency is gpg-agent's different behavior between
--enable-ssh-socket and --extra-socket (and the undocumented --browser-socket,
for that matter, but since it's not documented maybe it's fine to just change
that one).

I have created two sample commits, pushed to

https://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=shortlog;h=refs/heads/justus/issue2700

The second one does indeed change translated strings. If I don't update
translated strings, then the messages will still refer to the old version of the
options, which will still work but won't show up in '--help'. Is there a
problem with updating the strings when I also update the .po files?

Please do that as soon as you have some spare time. Take care not to chnage
translated strings.

By renew you mean prolonging the expiration time?

To add this new default we should first add a --quick-set-expire command to make
it easier to change the expiration time. Or --quick-expire to match the name
used in --edit-key - I don't care. And of course gpgme needs a new API.

According to T1143 (aheinecke on Jun 08 2016, 07:15 PM / Roundup) the plan is that locate-key as well as -r uses a new
mechanism to figure oiut the appropriate key. aheinecke already implemented
this strategy in Kmail but we want to have it in gnupg proper.

If the given key is specified by a mail address the new scheme kicks in for
--locate-key and all keys given with -r. gpg finds all matching non-expired and
suitable keys and then computes the validity (WoT, TOFU, whatever). That is
list ordered and the top ranked key is used. Newer keys/subkeys are preferred
and thus in general there should never be an ambiguity. In case there is an
ambiguity, -r should return an error and --locate-key should return all those keys.

This bug supersedes T1143 and T1232.

I got my hands on a macOS box, and this particular problem is fixed in 2e64ccb0.

I still cannot compile gnupg there, but I'm working on it.

Also, most options join words with hyphens, but some don't.

Thanks - attached please find the results of the grep command on the arpa directory...

Thanks. I believe the relevant part is:

checking whether the resolver is usable... no
checking whether I can make the resolver usable with BIND_8_COMPAT... no

The latter is indeed a MacOS X specific thing.

Would you be so kind to execute the following commands on your machine, and
report the results back?

grep COMPAT /usr/include/arpa/*
grep PACKETSZ /usr/include/arpa/*

actually it was a feature request that a trustdb is not created in case of
--always-trust. But sure, it should not error out.

Thanks for reply - I discovered the MacPG installer, which worked, so no urgency on
this but I provide the output from ./configure in case it helps others. It's in
RichTextFormat (.rtf) -- if you require plain text just let me know.

Thanks for the report. Please attach the full output of configure.

Thanks for clarifying this.

I've update the comment in the test accordingly. This issue is resolved for me.

OpenPGP has a timestamp granularity of one second and thus you can't distinguish
non-RSA signature from each other if they are donewithin the same second.

Waiting a second is an old trick which is even employed somewhere inside gpg.

Thanks, this works now as expected.

While enabling the checks for signcount in gpgme/lang/qt/tests/t-tofuinfo.cpp
I've noticed though that if I sign and verify the same plaintext twice
immediately after another the signcount is not incremented correctly.

In line 266 of that test. The call to signAndVerify leads to an Assert if you
remove the " World" part of the "Hello World" message.

Alternatively adding a QTest::qWait(1000); before that line also results in a
success.

You can trigger this also by modifing the strings in line 233ff to contain the
same message.

Not really important imo as this is a constructed problem. The main issue here
is resolved for me.

I believe that this bug has been fixed. Both Andre's and Justus' test cases now
emit the same information for all user ids (see below).

If you agree that this issue is resolved, please change the status appropriately.

us@grit:~/neal/work/gpg/build/gpgme/tests/gpg$ rm -f $GNUPGHOME/tofu.db && ( gpg
--verify --with-colons --status-fd=1 /tmp/testmsg )2>/dev/null
[GNUPG:] NEWSIG
[GNUPG:] KEY_CONSIDERED A0FF4590BB6122EDEF6E3C542D727CC768697734 0
[GNUPG:] SIG_ID QfzOUKyR2prNsVx/GI/4A5q2AxU 2002-03-03 1015172412
[GNUPG:] KEY_CONSIDERED A0FF4590BB6122EDEF6E3C542D727CC768697734 0
[GNUPG:] GOODSIG 2D727CC768697734 Alfa Test (demo key) <alfa@example.net>
[GNUPG:] VALIDSIG A0FF4590BB6122EDEF6E3C542D727CC768697734 2002-03-03 1015172412
0 4 0 17 2 00 A0FF4590BB6122EDEF6E3C542D727CC768697734
[GNUPG:] KEY_CONSIDERED A0FF4590BB6122EDEF6E3C542D727CC768697734 0
[GNUPG:] TOFU_USER A0FF4590BB6122EDEF6E3C542D727CC768697734 alfa@example.net
[GNUPG:] TOFU_STATS 2 1 0 auto 1472727595 1472727595
[GNUPG:] TOFU_STATS_LONG Verified 1 message signed by "Alfa Test (demo key)
<alfa@example.net>"%0Ain the past 0~seconds.
[GNUPG:] TOFU_USER A0FF4590BB6122EDEF6E3C542D727CC768697734 alpha@example.net
[GNUPG:] TOFU_STATS 2 1 0 auto 1472727595 1472727595
[GNUPG:] TOFU_STATS_LONG Verified 1 message signed by "Alpha Test (demo key)
<alpha@example.net>"%0Ain the past 0~seconds.
[GNUPG:] TOFU_USER A0FF4590BB6122EDEF6E3C542D727CC768697734 alice%20(demo%20key)
[GNUPG:] TOFU_STATS 2 1 0 auto 1472727595 1472727595
[GNUPG:] TOFU_STATS_LONG Verified 1 message signed by "Alice (demo key)"%0Ain
the past 0~seconds.
[GNUPG:] TRUST_MARGINAL 0 tofu

us@grit:~/neal/work/gpg/build/gpgme/tests/gpg$ ../../../gpgme/tests/run-verify
/tmp/testmsg
Original file name: [none]
Signature 0

status ....: Success
summary ...:
fingerprint: A0FF4590BB6122EDEF6E3C542D727CC768697734
created ...: 1015172412
expires ...: 0
validity ..: marginal
val.reason : Success
pubkey algo: 17 (DSA)
digest algo: 2 (SHA1)
pka address: [none]
pka trust .: n/a
other flags:
primary fpr: A0FF4590BB6122EDEF6E3C542D727CC768697734
tofu addr .: alfa@example.net
  validity : 2 (little history)
  policy ..: 1 (auto)
  sigcount : 1
  firstseen: 1970-01-01 00:01:46
  lastseen : 1970-01-01 00:01:46
  desc ....: Verified 1 message signed by "Alfa Test (demo key)

<alfa@example.net>"

             in the past 1 minute, 46 seconds.
tofu addr .: alpha@example.net
  validity : 2 (little history)
  policy ..: 1 (auto)
  sigcount : 1
  firstseen: 1970-01-01 00:01:46
  lastseen : 1970-01-01 00:01:46
  desc ....: Verified 1 message signed by "Alpha Test (demo key)

<alpha@example.net>"

             in the past 1 minute, 46 seconds.
tofu addr .: [none]
  validity : 2 (little history)
  policy ..: 1 (auto)
  sigcount : 1
  firstseen: 1970-01-01 00:01:46
  lastseen : 1970-01-01 00:01:46
  desc ....: Verified 1 message signed by "Alice (demo key)"
             in the past 1 minute, 46 seconds.

Since the split format has been removed, the relevant code is gone, and I'm not
able to reproduce the issue with your test case on HEAD, I think this issue is
also gone and I'm marking it resolved. Please reopen if necessary.

Can you please tell us what version of ssh you are using (ssh -V)?

Hi Justus,

Thanks for your response. In further testing, I was able to trigger the "FAIL:
gpgtar.scm" during a make check for 2.1.13 (actually "FAIL: gpgtar.test" for
2.1.13 since it's pre-tiny-scheme). In particular, it's vanilla 2.1.13 + your
fix in 8f79c31b. So I think what may be going on is that 8f79c31b didn't
actually fully resolve that problem after all since I've now seen it occur, with
that commit included, in 2.1.13, and in 2.1.14, and in HEAD.

Tbere were two cases where a more specific error was emitted:

In one run, I saw this:

((/private/tmp/gnupg21-20160725-43964-l18ixl/gnupg-2.1.14/tools/gpgtar --gpg
/private/tmp/gnupg21-20160725-43964-l18ixl/gnupg-2.1.14/g10/gpg --gpg-args
--no-permission-warning --always-trust --tar-args --directory=. --decrypt
/tmp/gpgscm-PgAlmV/archive) failed: gpgtar: gpg: [don't know]: invalid packet
(ctb=2d)
gpgtar: gpg: [don't know]: invalid packet (ctb=2a)
gpgtar: error running
'/private/tmp/gnupg21-20160725-43964-l18ixl/gnupg-2.1.14/g10/gpg': exit status 2
) 
FAIL: gpgtar.scm

and in another run I saw this

Checking gpgtar with signature 
((/private/tmp/gnupg21-20160726-74591-maikty/gnupg-2.1.14/tools/gpgtar --gpg
/private/tmp/gnupg21-20160726-74591-maikty/gnupg-2.1.14/g10/gpg --gpg-args
--no-permission-warning --always-trust --tar-args --directory=. --decrypt
/tmp/gpgscm-0U4bUB/archive) failed: gpgtar: gpg: Fatal: zlib inflate problem:
invalid block type
gpgtar: error running
'/private/tmp/gnupg21-20160726-74591-maikty/gnupg-2.1.14/g10/gpg': exit status 2
) 
FAIL: gpgtar.scm

It's also worth noting that I've only been able to trigger the problem on
Jenkins during CI, not locally, so I don't know if the lack of TTY is relevant
or something like that.

I will do the ssh check you requested.

Thanks for letting us know. Unfortunately, we do not test on MacOS yet, but we are working
on that.

I have neither experience with debugging on MacOS, nor do I have access to such a machine.
I'm afraid you are on your own for now.

The ssh test is new, so we need to figure out why it does not work. Please do

  make -C tests/openpgp check TESTS="setup.scm ssh.scm" verbose=2

This lets us see what ssh-add prints to stderr. It might be related to the version of
OpenSSH shipped with the OS.