I suspect that the problem is the same as T2817.

Andre and I chatted about this issue offline, and I now understand what the
problem is. The TOFU_STATS status line (as documented in gnupg/doc/DETAILS) has
a "validity" field that is a number between 0 and 4 where 1 to 4 indicate how
confident we are that the binding is valid, and 0 means that the binding has an
unresolved conflict. The problem that Andre has observed is that this field is
not set to 0 if there is a conflict.

As a matter of fact, the validity field is never set to 0. This is completely
redundant as the same TOFU_STATS status line has a policy parameter, which is
"ask" if there is a conflict. Moreover, overloading this field in this way
causes a loss of information. Just because there is a conflict doesn't mean
that gpg shouldn't report the validity, or that the client can't made use of it.

Thus, in my opinion, the right thing to do is to simply use the <policy> field
to detect whether there is a conflict. Werner has suggested that this is wrong,
but I couldn't follow his logic. Thus, I'm adding him to the nosy list and I
hope he can clarify what he wants here.

Sorry for the delay in getting back to you on this issue. I think you mean they
have undefined trust (that's what I get here). Undefined trust means "not
enough information for calculation" (from trustdb.h).

Can you clarify what you mean by validity conflict?

I fully support dkg on this. If our downstream is complaining that there is a
problem, then we need to take it seriously. I respect Werner's opinion, but
disagree specifically with the idea that this is only a problem for special
users. I think it will happen to many normal users too.

@thomas: You may want to look at gpg sync, which I think makes at least some of
what you want to do easier.
https://firstlook.org/code/2016/10/12/introducing-gpg-sync-an-open-source-tool-for-organizations-that-encrypt-email/

Because it took me a while to understand what is actually going wrong, a summary
of the problem: if we get an error such as "key 517912BA66E730CA was created 78
seconds in the future", then the key's flags will be wrong (below: SCEA instead
of C) and the expiration date will not be printed.

In gpg-agent, only a single thread of execution runs at a time. So it is
entirely possible that what you are describing happens. For us to debug it, we
need a very concrete example. Please provide us with the command line(s) that
you are using to decrypt the files in parallel. Also, please list the keys. (A
small guess: you are using 16k RSA.)

FWIW, I idle on gnupg on freenode and I've helped a bunch of people over the
past two years with exactly this problem. It is not that they want to use gpg
and gpg2, but that at some point they (or some tool) ran gpg2 while they
continued to use gpg1. They then became very surprised when they used gpg2 and
it only had a subset of their keys. My advice for these users is always the
same: remove the migration file and just rerun gpg2. As far as I can tell, this
has fixed the problem in all cases.

Hi Andre,

Thanks for following up. I seem to be able to reproduce the first part of your
issue here and I'm looking in to it.

Thanks,
Neal

7a634e48b13c5d5d295b8fed9b429e1b2109a333 should fix the contention issue.
Please let me know if you are still having issues.

eec365a & 614ca00 fixed the performance issue for me here.

us@chu:~/neal/work/gpg/test (GnuPGTest)$ rm tofu.db
us@chu:~/neal/work/gpg/test (GnuPGTest)$ time gpg --no-default-keyring --keyring
/usr/share/keyrings/debian-keyring.gpg -k >/dev/null
gpg: NOTE: THIS IS A DEVELOPMENT VERSION!
gpg: It is only intended for test purposes and should NOT be
gpg: used in a production environment or with production keys!
gpg: Note: signatures using the MD5 algorithm are rejected

real 0m45.569s
user 0m34.316s
sys 0m10.872s
us@chu:~/neal/work/gpg/test (GnuPGTest)$ time gpg --no-default-keyring --keyring
/usr/share/keyrings/debian-keyring.gpg -k >/dev/null
gpg: NOTE: THIS IS A DEVELOPMENT VERSION!
gpg: It is only intended for test purposes and should NOT be
gpg: used in a production environment or with production keys!
gpg: Note: signatures using the MD5 algorithm are rejected

real 0m2.306s
user 0m2.284s
sys 0m0.020s
us@chu:~/neal/work/gpg/test (GnuPGTest)$ time gpg --no-auto-check-trustdb
--trust-model pgp --no-default-keyring --keyring
/usr/share/keyrings/debian-keyring.gpg -k >/dev/null
gpg: NOTE: THIS IS A DEVELOPMENT VERSION!
gpg: It is only intended for test purposes and should NOT be
gpg: used in a production environment or with production keys!
gpg: please do a --check-trustdb
gpg: Note: signatures using the MD5 algorithm are rejected

real 0m2.261s
user 0m2.248s
sys 0m0.012s

The first time a key is encountered, we need to do a number of checks that
require reading its keyblock. These include checking whether the key is signed
by an ultimately trusted key. So, this cost is pretty much unavoidable, but it
should be a one time thing.

That other gpg processes stall is surprising, and I will investigate this. I
went to a fair amount of trouble to make sure that that doesn't happen in practice.

That the cost is higher on subsequent runs is a bit disconcerting. I will also
investigate this.

Are the two keys that you testing ultimately trusted? If so, then their
validity is good independent of their TOFU policy.

It is a bit unfortunate that the TOFU policy doesn't show this. I will try and
fix this, but it is a bit complicated because when a key's ownertrust is changed
(or a signature is added, etc.), the tofu db is not updated.

Fixed in df5353b. Also added a test.

I believe that this bug has been fixed. Both Andre's and Justus' test cases now
emit the same information for all user ids (see below).

If you agree that this issue is resolved, please change the status appropriately.

us@grit:~/neal/work/gpg/build/gpgme/tests/gpg$ rm -f $GNUPGHOME/tofu.db && ( gpg
--verify --with-colons --status-fd=1 /tmp/testmsg )2>/dev/null
[GNUPG:] NEWSIG
[GNUPG:] KEY_CONSIDERED A0FF4590BB6122EDEF6E3C542D727CC768697734 0
[GNUPG:] SIG_ID QfzOUKyR2prNsVx/GI/4A5q2AxU 2002-03-03 1015172412
[GNUPG:] KEY_CONSIDERED A0FF4590BB6122EDEF6E3C542D727CC768697734 0
[GNUPG:] GOODSIG 2D727CC768697734 Alfa Test (demo key) <alfa@example.net>
[GNUPG:] VALIDSIG A0FF4590BB6122EDEF6E3C542D727CC768697734 2002-03-03 1015172412
0 4 0 17 2 00 A0FF4590BB6122EDEF6E3C542D727CC768697734
[GNUPG:] KEY_CONSIDERED A0FF4590BB6122EDEF6E3C542D727CC768697734 0
[GNUPG:] TOFU_USER A0FF4590BB6122EDEF6E3C542D727CC768697734 alfa@example.net
[GNUPG:] TOFU_STATS 2 1 0 auto 1472727595 1472727595
[GNUPG:] TOFU_STATS_LONG Verified 1 message signed by "Alfa Test (demo key)
<alfa@example.net>"%0Ain the past 0~seconds.
[GNUPG:] TOFU_USER A0FF4590BB6122EDEF6E3C542D727CC768697734 alpha@example.net
[GNUPG:] TOFU_STATS 2 1 0 auto 1472727595 1472727595
[GNUPG:] TOFU_STATS_LONG Verified 1 message signed by "Alpha Test (demo key)
<alpha@example.net>"%0Ain the past 0~seconds.
[GNUPG:] TOFU_USER A0FF4590BB6122EDEF6E3C542D727CC768697734 alice%20(demo%20key)
[GNUPG:] TOFU_STATS 2 1 0 auto 1472727595 1472727595
[GNUPG:] TOFU_STATS_LONG Verified 1 message signed by "Alice (demo key)"%0Ain
the past 0~seconds.
[GNUPG:] TRUST_MARGINAL 0 tofu

us@grit:~/neal/work/gpg/build/gpgme/tests/gpg$ ../../../gpgme/tests/run-verify
/tmp/testmsg
Original file name: [none]
Signature 0

status ....: Success
summary ...:
fingerprint: A0FF4590BB6122EDEF6E3C542D727CC768697734
created ...: 1015172412
expires ...: 0
validity ..: marginal
val.reason : Success
pubkey algo: 17 (DSA)
digest algo: 2 (SHA1)
pka address: [none]
pka trust .: n/a
other flags:
primary fpr: A0FF4590BB6122EDEF6E3C542D727CC768697734
tofu addr .: alfa@example.net
  validity : 2 (little history)
  policy ..: 1 (auto)
  sigcount : 1
  firstseen: 1970-01-01 00:01:46
  lastseen : 1970-01-01 00:01:46
  desc ....: Verified 1 message signed by "Alfa Test (demo key)

<alfa@example.net>"

             in the past 1 minute, 46 seconds.
tofu addr .: alpha@example.net
  validity : 2 (little history)
  policy ..: 1 (auto)
  sigcount : 1
  firstseen: 1970-01-01 00:01:46
  lastseen : 1970-01-01 00:01:46
  desc ....: Verified 1 message signed by "Alpha Test (demo key)

<alpha@example.net>"

             in the past 1 minute, 46 seconds.
tofu addr .: [none]
  validity : 2 (little history)
  policy ..: 1 (auto)
  sigcount : 1
  firstseen: 1970-01-01 00:01:46
  lastseen : 1970-01-01 00:01:46
  desc ....: Verified 1 message signed by "Alice (demo key)"
             in the past 1 minute, 46 seconds.

Since the split format has been removed, the relevant code is gone, and I'm not
able to reproduce the issue with your test case on HEAD, I think this issue is
also gone and I'm marking it resolved. Please reopen if necessary.

I don't have time to look at this immediately, but it looks related to Werner's
recent change to the tofu db code.

FWIW, I added the stricter check. Previously, we specified the header size, but
didn't check that it was respected. When discussing this with Werner, he said
that respecting the header size was important, which is why I chose to die
rather than silently change the header size.

Patch applied in dc417bf0c555a7416d0aedde6645fd1087660f92 (Dec 15, 2015)

What distribution are you using? You are probably better off using their
supplied binaries, which are hopefully tested.

Thanks for reporting this! These types of bugs are important. Thanks for
reporting it. I will take a look at it soon.

Hi Clint,

Out of curiosity, have you tried this on 2.1?

I realize this is probably very easy to reproduce, but could you nevertheless
list the commands that you used to show the bug?

Thanks!

This has been fixed (see the message from Werner today on gnupg-devel with
message-id <87bn6mr28v.fsf@wheatstone.g10code.de>)

Werner pointed out that the quick integrity check is not used due to an attack
by Mister and Zuccherato. However, this attack does not make use of any
information from the PK-ESK packet. It just uses the session key. As such, the
quick integrity check should not be done in the dek->symmetric case either.

I think it is possible to fix this issue so that we can use the quick integrity
check in the future. My post about this to the openpgp group is here:
http://mailarchive.ietf.org/arch/msg/openpgp/A_r93YIukOqzvrmd44F-Jl3dHbc .

My suggestion is a not-backwards compatible change. For messages that currently
exist, it is acceptable to do the quick integrity check if we can rate limit the
oracle (to recover the first two bytes from N blocks costs (N+1) * 2^15
decryption attempts). This is definitely safe, as Mister and Zuccerato point
out, in the interactive case. Do we have a way to reliably detect this?

Sorry, I was using --check-trustdb as a shorthand for the actual function.

Thanks for reporting this. The right solution is for --check-trustdb to ignore
legacy keys.

The reason that we encrypted the SED packet with AES256 is that is the preferred
cipher in my public key. I think that the cipher for the s2k function should be
chosen similarly.

Running from the command line with gawk and mawk, I don't get an error message.
What version of awk are you using? Does this occur when triggering this from
the command line or only when running it from smartgit?

Marking as resolved since this is available in 2.1 and we are not going to
backport this to 1.4 or 2.0. Thanks.

I assume that this patch solved the problem. Thanks for reporting!

I think this is reasonable. If you want to implement it, I'll review the
patches. Thanks.

I tend to agree with Werner: adding another pinentry program increases our
maintenance burden, but the new pinentry doesn't add any convincing features,
AFAIK. If there are some significant benefits, please add them. Otherwise, I
think I'll change this issue to wont-fix. Sorry. Nevertheless, thank you for
your contribution! I hope you'll find another way to contribute.

Does this mean that pinentry-emacs will only work when an emacs instance calls
gpg? Does pinentry-emacs need to support the case that a program other than
emacs calls gpg?

@ueno: This is reasonable. Thanks for the explanation. Do you happen to know
approximately what version started to enable these protections?

Thanks! I'm mark this as resolved.

I've pushed a slightly different version of this patch (2d1d795). Please test
not only that --edit-key detects duplicates and reorders out of place
signatures, but also that revocation certifications, self-sigs, etc. are
correctly checked. Thanks!

I've pushed this.

The branch neal/issue2236 contains an initial fix. It does two things:

• It identifies duplicate signatures (based on their message digest) and removes

duplicates.

• Instead of blindly moving signatures around, this systematically tests each

signature against its alleged component (= primary key / subkey / user id) and
if it is bad, it tries the other components in the key block and moves it if
appropriate. (If it doesn't belong to any components, then the sig is just left
where it is and GnuPG will ignore it).

I've tested this with a few keys and it seems to work well. Lucas' key just has
a lot of duplicate signatures.

Starting program: /home/us/neal/work/gpg/build/gnupg/g10/gpg2 --check-key
0x06EAA066E397832F
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
gpg: WARNING: unsafe permissions on homedir '/tmp/luca'
gpg: NOTE: THIS IS A DEVELOPMENT VERSION!
gpg: It is only intended for test purposes and should NOT be
gpg: used in a production environment or with production keys!
gpg: Ignored 852 duplicate signatures (total: 2079).
gpg: public key E397832F: timestamp: 2009-07-01 14:44:59 (1246459499)
gpg: user id: Luca Capello <luca@pca.it>
gpg: sig: class: 0x10, issuer: 109E6244, timestamp: 2013-02-05 02:24:16
(1360031056), digest: eb c3
gpg: Good signature over last major component!
gpg: sig: class: 0x13, issuer: E397832F, timestamp: 2009-07-01 14:44:59
(1246459499), digest: 93 7a
gpg: Good signature over last major component!
gpg: sig: class: 0x13, issuer: E397832F, timestamp: 2009-07-01 14:58:17
(1246460297), digest: 53 4f
gpg: Good signature over last major component!
gpg: sig: class: 0x13, issuer: E397832F, timestamp: 2010-10-10 21:44:51
(1286747091), digest: be d5
gpg: Good signature over last major component!
gpg: user id: Luca Capello <gismo@debian.org>
gpg: sig: class: 0x10, issuer: 109E6244, timestamp: 2013-02-05 02:24:16
(1360031056), digest: 4e 92
gpg: Good signature over last major component!
gpg: sig: class: 0x13, issuer: E397832F, timestamp: 2009-07-01 14:57:12
(1246460232), digest: 9c 3d
gpg: Good signature over last major component!
gpg: sig: class: 0x13, issuer: E397832F, timestamp: 2010-10-10 21:52:18
(1286747538), digest: 54 c1
gpg: Good signature over last major component!
gpg: user id: Luca Capello <luca.capello@infomaniak.ch>
gpg: sig: class: 0x13, issuer: E397832F, timestamp: 2016-01-24 14:44:42
(1453646682), digest: 79 a4
gpg: Good signature over last major component!
gpg: user id: Luca Capello <luca.capello@infomaniak.com>
gpg: sig: class: 0x13, issuer: E397832F, timestamp: 2016-01-29 22:49:59
(1454107799), digest: 43 19
gpg: Good signature over last major component!
gpg: subkey 2BB95F4B: timestamp: 2009-07-01 14:55:55 (1246460155)
gpg: sig: class: 0x18, issuer: E397832F, timestamp: 2009-07-01 14:55:55
(1246460155), digest: 4b d9
gpg: Good signature over last major component!
gpg: subkey 3BE9F36D: timestamp: 2009-07-01 15:09:03 (1246460943)
gpg: sig: class: 0x18, issuer: E397832F, timestamp: 2009-07-01 15:09:03
(1246460943), digest: c2 f9
gpg: Good signature over last major component!
gpg: Couldn't check 1216 signatures due to missing issuer keys.

Interestingly, your key contains a bad signature (the hash has been corrupted).

The reason that I haven't pushed this to master is that I need to work our how
the output should look. Also, this functionality will probably only be
available via the --edit-key menu. This patch includes an argument --check-key,
which will probably be removed.

If you have an opportunity to test this, I'd appreciate it.

I reported this to the libsecret maintainers, but it turns out that it was our
bug. Stef kindly replied a patch, which I've now applied (2f5bfa0). Looking
again at dkg's original message, he doesn't suggest that the problem is with
libsecret, but in fact correctly identified pinentry at the culprit.

What distribution are you using? What pinentry program? Can you take a look
using seahorse to make sure that your password is saved. Once it is saved, it
shouldn't be removed.

Given how trivial the fix is, I applied that.

Note: recent versions of pinentry-gtk-2 are using native widgets. If you are
using that program and not the latest version of pinentry, then please try that
first.

There is no version 2.0.22 of pinentry (the most recent version is 0.9.7). Can
you please figure out what version of pinentry you are using and which pinentry
program (there are five: pinentry-gnome3, pinentry-gtk-2, pinentry-qt,
pinentry-curses and pinentry-tty). Thanks!

The following simple patch works for me and make check still passes. I think it
makes sense to apply this patch given that this workaround is no more
complicated than an existing workaround for something similar (immediately
preceding my change). Can you please test to make sure it works for you?

D351: 776_0001-gpg-Make-ASCII-armor-decoding-more-robust-to-encodin.patch

Thanks for your contribution! A few comments based on a quick skim of the code:

Why are you using the apparently invalid email address
"madrat@users.noreply.github.com" in the headers?

The code is not formatted according to the GNU coding standards (indentation
using tabs instead of 2 spaces; some lines are longer than 80 characters). I'm
not sure how important this is since the rest of the code has a fair number of
violations.

There are not many comments.

When commenting out large blocks of code (as you do in main.cxx), please use #if
0 ... #endif rather than using /* ... */.

@werner: Do we want to add support for FLTK? If so, I'll take a closer look at
this. My main concern is that this is another thing that we have to maintain
and I'm not sure the gtk pinentry is really just a burden for weak computers.

gpg doesn't normally directly ask for a password. Instead, operations that
require a password are typically handled by gpg-agent, which is a small server
that is started on demand. (Normally, there is only a single gpg-agent per
user.) When gpg-agent needs a password, it invokes a pinentry program. The
default pinentry can be determined using `gpgconf --list-config'. This can be
overridden using the pinentry-program configuration option in gpg-agent.conf.
(If you change that file, you'll need to restart gpg-agent using something like
`gpgconf --reload gpg-agent'.)

There are several different pinentry programs: pinentry-gtk-2, pinentry-qt,
pinentry-curses and pinentry-tty. (pinentry is typically an alias that is
configured by the system's package manager.) Even if you use pinentry-gtk-2, it
will normally fall back the curses backend if there is no X display.

The issue you might be having is that pinentry might be showing up on a
different display / console.

So, I think this might just be a configuration problem. Nevertheless, I
encourage you to investigate some more and try to figure out what is going on
and report back here. Thanks!

This should be fixed in acac103. (I was able to exactly reproduce your problem
and the patch fixed it for me.) If you are able to test and it works for you,
please report back here.

Thanks!

It seems like detecting and correcting this simple manging would be
straightforward to do and relatively self contained.

I think I wasn't clear. I have two monitors, but only one X DISPLAY. This is
about the screen, not the X display, where the pinentry is shown.

Thanks for the report. Please add the stack trace here (either inline or as an
attactment) so that it does not get lost. Thanks.

Ideally, there would be a mini-game, perhaps, space invaders. As the user
plays, we automatically harvest entropy!

Why is this a reasonable assumption? This proposal changes the way that GnuPG
has been working for years and will inevitably break someone's setup. It would
be much better for the receiver to use a non-critical notation to indicate the
desired behavior.

D345: 761_0001-gpg-Fix-calc_header_length-when-LEN-is-0-and-improve.patch

Since it was so trivial to create, I've attach an alternative patch with my
proposed change. Please let me know which one to apply.

D346: 760_0001-gpg-Fix-documentation-for-calc_header_length.patch

Patch attached. Is this okay to apply?

The gnome3 problem is due to libgcr, which disables and manages the dialog.

I'll take a look at the gtk2 issue soon. Does pasting with the middle mouse
button work?