To summarize, here is the situation:

• Ideally, it would be good to modify GnuPG and Emacs EasyPG to implement status handling and input handling in better way.

Ah, I see https://dev.gnupg.org/rG2f872fa68c6576724b9dabee9fb0844266f55d0d applies cleanly. I guess can go with that, although would prefer it if on the 2.4 branch.

Is there a commit we could backport downstream to 2.4.x? We've had quite a few reports of this.

To align the default expiration time with the BSI approval and other related software we change this now to 3 years.

The problem of hang of tests/openpgp/multisig.scm is solved by rGef4f22b9d98b: gpg: Graceful exit for signature checking with --batch.
But the problem itself is not yet solved.

It is reproducible by testing tests/openpgp/multisig.scm with keyboxd enabled (it hangs), with the modification of following.

I pushed the change which keeps old status report behavior to master.
Let me test the change.

looks simpler to me.

Hmm, for the latter this:

Orthogonally, here is possible change for GnuPG, if we need to support the workaround of compress-level 0 in ~/.gnupg/gpg.conf.

OK, here is my changes which always use make-temp-file (to avoid confusion between data input and passphrase input).

I use epg.el with the change of removing the wait:

Was resolved, see T4457

GnuPG is and can't be FIPS-140-3 compliant due to the way it is implemented. We may eventually employ the new hash-and-sign API of Libgcrypt to move into this direction but that has not yet been done. However, this also requires the use of the new indicator API and the, well, a RedHat kernel.

Pushed to GnuPG master. Let us test. For my machine of Debian GNU/Linux, Wine emulation (Windows 32-bit, Windows 64-bit), make check goes all well.
After confirming the implementation, I'd like to put it into gpgrt.

Will be in 2.4.2

@werner We could make the wait conditional on (equal epg-gpg-program "gpg"), that is, only when user has GnuPG 1.x.

Well okay, then I have no workaround. However, I won't consider this a bug because BEGIN_ENCRYPTION marks the start of the actual encryption process but not when it starts to read input data.

The change rG60963d98cfd8: gpg: Detect already compressed data also when using a pipe. for T6332 introduce IOBUF_IOCTL_PEEK.

@werner I tested by switch back to GnuPG 2.4.1 (I downgraded to 2.4.0 before to temporary work around issue), adding compress-level 0 to gpg.conf file. It's not working. The problem still exist.

I have not yet experienced that although I am using Gnus with encrypted mail all the time. My guess is that this is due to the improved compressed input detection in gpg. You might be able to work around it by adding compress-level 0 to gpg.conf

Just checked 2.4.1 and looks like now everything is OK.

works, Gpg4win-4.1.1-beta295

Sorry, the comment above is my misunderstanding.

For importing key/subkey case, it doesn't matter if the smartcard is connected or not. The data in the file will be overwritten by import.

This works if the smartcard with the same private key is not connected, which it usually shouldn't be (outside of testing situations) so that's ok for me.
But I think we should inform the user what is done or isn't.
Currently we get in both cases:

gpg: Anzahl insgesamt bearbeiteter Schlüssel: 1
gpg:                             unverändert: 1
gpg:              gelesene geheime Schlüssel: 1
gpg:            geheime Schlüssel importiert: 1

Which is not very clear in any case but in case the smart card is connected it is in fact wrong, there is no secret key imported, the keystub remains.

Another miscellaneous correction for jimregexp. A condition was copy-pasted from another section without the necessary changes, resulting in incorrect logic. This seems harmless apart from inconsistent error reporting.

diff --git a/regexp/jimregexp.c b/regexp/jimregexp.c
index 1a8b8aae6..1b6e1b49c 100644
--- a/regexp/jimregexp.c
+++ b/regexp/jimregexp.c
@@ -778,7 +778,7 @@ static int regatom(regex_t *preg, int *flagp)
                                                        preg->err = REG_ERR_NULL_CHAR;
                                                        return 0;
                                                }
-                                               if (start == '\\' && *pattern == 0) {
+                                               if (end == '\\' && *pattern == 0) {
                                                        preg->err = REG_ERR_INVALID_ESCAPE;
                                                        return 0;
                                                }

Pushed the change not including OK_TO_CHANGE_ERROR_BEHAVIOR part.
Note that the modification affects main key case, too.

To minimize the impact of the change, I updated:

diff --git a/g10/import.c b/g10/import.c
index 1ed40a63c..345e8cc75 100644
--- a/g10/import.c
+++ b/g10/import.c
@@ -2955,9 +2955,23 @@ do_transfer (ctrl_t ctrl, kbnode_t keyblock, PKT_public_key *pk,
 {
   gpg_error_t err;
   struct import_stats_s subkey_stats = {0};
+  int force = 0;
+  int already_exist = agent_probe_secret_key (ctrl, pk);
+
+#ifndef OK_TO_CHANGE_ERROR_BEHAVIOR
+  if (already_exist == 1)
+    return gpg_error (GPG_ERR_EEXIST);
+#endif
+  if (already_exist == 2)
+    {
+      if (!opt.quiet)
+        log_info (_("key %s: card reference is overridden by key material\n"),
+                  keystr_from_pk (pk));
+      force = 1;
+    }

Changes may be something like:

diff --git a/g10/import.c b/g10/import.c
index 1ed40a63c..91ff0c8ec 100644
--- a/g10/import.c
+++ b/g10/import.c
@@ -2706,6 +2706,20 @@ transfer_secret_keys (ctrl_t ctrl, struct import_stats_s *stats,
           goto leave;
         }

isn't T3456 the same issue?

On gpg4win 4.1.0 (and GnuPG VSD 3.1.26) there are no longer password prompts for the subkeys when exporting (or making a backup from) secret keys.

This has been solved loooong ago.

OCB mode (i.e. packet 20) is only used if the keys announce it. Thus only after moving a (private) key from GnuPG to a non-GnuPG compatible implementation you will run into this problem. The compatibility options won't override the preference system.

gpg-agent now supports READKEY --card command which creates stub file when it's not yet available on host computer.
It was implemented by rG82cbab906a3e: agent: Add --card option for READKEY.

Thanks for the description; this is good for documentation.

That sounds like a great (and better) solution! Thank you for your attention & effort 👍🙏

See T6395 for the new feature. It will be released with 2.4.1 but it will take some time that it can actually be used because the other party needs to have an OpenPG implementation which supports this.

Please use

gpgtar -C /home/matt/data ....

instead of using an absolute name. This makes things much easier to implement in a secure way: You don't want to have absolute file names in the tarball and mapping them to relative names is not easy or even impossible in case of, say "/home/foo/x.data /home/bar/x.data". Keep in mind that gpgtar does also not handle symlinks and other special files.

I guess this is fixed with this commit for 2.2. and 2.4. Given that the report is quite old with not new infos since 2019, I'll close it.

The reason why gpg does not encrypt to multiple subkeys is that the older subkeys are viewed as deprecated. You could write a tool which does a heuristic to check when the time is reached that no more messages are encrypted to an older subkey (or are used to decrypt archived mails). At that point you can take the private part of the old subkey offline.

Debian's wiki also speaks a lot about the advantages & dream of subkeys, but also mentions the caveat:

I've read many articles mentioning the improved key handling when different devices just have different subkeys, thus allowing a semantic connection to a primary identity (instead of different "Identities" on different devices)

What do you want to achieve by using multiple encryption subkeys? Do you realize that gpg will always encrypt to one subkey (unless you explicitely specify multiple subkeys), i.e. you won't be able to decrypt on device 1 what you have encrypted for device 2 and vice-versa. Usually, this makes little sense because it seems you want to be able to decrypt anything on your main machine.

Thanks for the reply :)

Sorry, I think you have to fix the other tools. The ! suffix has virtually been supported forever and any new option to do the same complicates the code and the documentation.

Kleopatra simply copies the content of the corresponding *.key file in the private-keys-v1.d folder. If the *.key file contains a shadowed key after issuing a KEYTOCARD --force [...] command followed by a SCD LEARN --force command (note the SCD!), then gpg-agent is to blame.

Works now for gpgme. Thanks!

Understood. I appreciate the time you took to analyse the issue. Thanks.

I guess this is the first time such a key was reported. Printing diagnostics would be a bit of work because the code to compute th. expiration time is deep in gpg's guts.

The first signature is a direct key signature (class 0x1f) and this determines the expiration time. The usual case is to have the expiration time in the user id signatures. Our code does not allow to chnage the expiration time of direct key signature. This is because direct key signature are used by PGP and GnuPG only to add designated revokers. Gpg has no means to create a direct key signature like you have in your key.

Sorry, I mistakenly closed this task. I reopen it.

Could it be the case that your implementation actually used those bits to calculate a public key?

Sorry for a bit late follow up. How do you calculate a public key? RNP's crypto backend, Botan, is calculating public key without taking in account bits which should be tweaked. I.e. both tweaked and non-tweaked secret keys would produce the same public key. The same is with decryption. Could it be the case that your implementation actually used those bits to calculate a public key?

The compression check currently detects bzip2, gzip, zip, pkzip, and PDF. This also covers common document formats like odt and docx. We may add some more detection in the future. However, for large files you usually know their type and thus you better use "-z0" for already compressed data or "-z-1" if you want to force compression (may be for PDFs which often can be a shrinked to 80% or so).

So here is a redacted CLI-dump of the exact sequence I'm describing in my post. This is with untweaked keys and gpg 2.2.40 and a factory-reset yubikey.

So in case this was not clear... What I'm describing is very similar to the original description, but it is "inverted" - the untweaked key works flawlessly (import and decryption) except for keytocard. And the tweaked key can't be imported - either "Bad Secret Key" or asking for passphrase.

I am somehwat confused, my symantec system got faster. But there are some things like "Symantec Insight" which will whitelist often used files and applications, also signed files might get preferred treatment. I tried to get this slower by disabling the "Insight" and changing the "Bloodhound behavior" to agressive... So timings might not be comparable. I should probably do tests ohne without restarting my systems for a good comparison.

@onickolay Yes, I have. I have used --check-cv25519-bits and it said that it needs patching. I then did --fix-cv25519-bits and exported the key. Looking at the CV25519 private-key bytes produced by my code and by RNP, I confirmed that they did the exact same transformation.
When trying to re-import the exported key into gpg, I got the "Bad Secret Key" error again

@bigmomma Just for a quick check - did you try to use RNP's CLI command --edit-key --fix-cv25519-bits, as it's not clear from the message?

Hi! I would like to chime in on this issue as I am having some weird problems with a CV25519 sub-key and after stumbling upon this thread, I think it is related to this.
Unfortunately, I can't post the key material here, because it is my actual encryption private-key.

This can be easily tested using

Instead of using --enable-special-filenames and a separate FD the list of files is now passed to gpgtar's stdin. Similarly, we read from gpgtar's stderr instead of using a separate --logger-fd.

What about --logger-fd? Does gpgtar pass all FDs through to gpg?

Okay, I'll skip those for now.

no-tty and charset are anyway obsolete and passed only for older gpg versions. The other things should have useful defaults in gpg - in particular these defaults are taken from the same envvar as gpgme does. See send_pinentry_environment.

Another thing I have noticed when turning qt debug output on is that the qt windows platformsupport fontdatabase logs over a a timespan of over two seconds that it is adding fonts to its database.