Page MenuHome GnuPG

GnuPG 2.2 cannot import secret keys from 1.4/2.0's secring.gpg directly (if it is expired by original expiration date)
Closed, ResolvedPublic

Description

I was playing around with GnuPG 2.1 and its new storage formats when I stumbled across a strange behaviour.
As I tried to import my old (GnuPG 1.4/2.0) secret keys with

$ gpg2 --import ~/.gnupg.old/secring.gpg

I ended up with unusable (private) keys. More precisely the private keys are lacking their subkeys:

$ gpg2 --list-keys
/home/tux/.gnupg/pubring.kbx
----------------------------
pub   rsa4096/0xABCDABCDABCDABCD 2016-01-01 [C] [expired: 2017-01-01]
      12341234123412341234ABCDABCDABCDABCDABCD
uid                   [ expired] Tux Penguin <tux@example.org>
uid                   [ expired] Tux Penguin


$ gpg2 --list-secret-keys
/home/tux/.gnupg/pubring.kbx
----------------------------
sec#  rsa4096/0xABCDABCDABCDABCD 2016-01-01 [C] [expired: 2017-01-01]
      12341234123412341234ABCDABCDABCDABCDABCD
uid                   [ expired] Tux Penguin <tux@example.org>
uid                   [ expired] Tux Penguin

No subkeys listed at all. (The keys are marked as expired due to missing self-signatures which seem to be stored along with the subkeys.)

This only happens to keys where the primary secret subkey is missing (sec#).
Subkeys of keys which have a primary secret key are imported correctly by the above statement.

My question is:
Is this intended behaviour and is it documented anywhere?
If yes, then printing a warning when importing secring.gpg might be a good idea, e.g.

Warning you are importing from secring.gpg. This might not import all keys correctly.
Consider using gpg --export-secret-keys > seckeys.gpg && gpg --import seckeys.gpg instead.

Thanks in advance.

Simple workaround for people how stumble upon this and want a quick solution:
Export the secret keys into a file and import them afterwards

$ gpg2 --homedir ~/.gnupg.old --export-secret-keys > ~/tmp/seckeys.gpg
$ gpg2 --homedir ~/.gnupg     --import ~/tmp/seckeys.gpg

The shorter

$ gpg2 --homedir ~/.gnupg.old --export-secret-keys | gpg2 --homedir ~/.gnupg --import

should also work but gave me an unusable terminal due to two instances of pinentry/gpg-agent fighting over one terminal.

Event Timeline

gniibe added a subscriber: gniibe.

Thank you for reporting. Sorry, I couldn't understand some part of your report. Perhaps, due to some terminology.

There are four things: primary key public, subkey public, primary key private, and subkey private.

In GnuPG 2.1, public part is in .gnupg/pubring.kbx (both for primary key and subkey) and private part is in .gnupg/private-keys-v1.d/ (also both for primary and subkey).
In GnuPG 1.4/2.0, public part is in .gnupg/pubring.gpg (both for primary key and subkey) and private part is in .gnupg/secring.gpg (both for primary and subkey).

To figure out the situation, It is easier for me to see the output of GnuPG 1.4/2.0 for the key: gpg -k and gpg -K.

The difference between exported key and .gnupg.secring.gpg is that the latter has information for signature validation cache.

Thank you for reporting. Sorry, I couldn't understand some part of your report. Perhaps, due to some terminology.

There are four things: primary key public, subkey public, primary key private, and subkey private.

Maybe I was a bit unclear here, so let me rephrase it.

I tried to convert my old keys from GnuPG 1.4/2.0 to GnuPG 2.1. The initial situation (Gpg 2.0 output) is

# This is GnuPG version 2.0
$ gpg2 --list-keys
/home/tux/.gnupg/pubring.gpg
pub   4096R/0xABCDABCDABCDABCD 2016-01-01 [expires: 2018-01-01]
      Key fingerprint = 1234 1234 1234 1234 1234  ABCD ABCD ABCD ABCD ABCD
uid                 [ultimate] Tux Penguin
uid                 [ultimate] Tux Penguin <tux@example.org>
sub   4096R/0x1122334455667788 2016-01-09 [expires: 2018-01-01]
sub   4096R/0xAABBCCDDEEFF1122 2016-01-09 [expires: 2018-01-01]

$ gpg2 --list-secret-keys
/home/tux/.gnupg/secring.gpg
-------------------------------------
sec#  4096R/0xABCDABCDABCDABCD 2016-01-01 [expires: 2017-01-01]
      Key fingerprint = 1234 1234 1234 1234 1234  ABCD ABCD ABCD ABCD ABCD
uid                            Tux Penguin
uid                            Tux Penguin <tux@example.org>
ssb   4096R/0x1122334455667788 2016-01-01
ssb   4096R/0xAABBCCDDEEFF1122 2016-01-01

Some notes about the output:

  • I've set the options keyid-format 0xlong and with-fingerprint.
  • There is one primary key and two subkeys.
  • The primary secret key is not contained in the keyring (hash in sec#). The two private subkeys are contained in the keyring.
  • The key was created 2016-01-01 and its first expiration date was 2017-01-01. The current expiration date is 2018-01-01.
  • The expiration date of the primary secret key is in the past but GnuPG still prints expires. This seems to be normal behaviour and is, I think, not related to the issue here and it also does not prevent GnuPG from using the secret subkeys for signing/encrypting. I'm pretty sure this is related to the missing primary secret key.

Next, I moved ~/.gnupg to ~/.gnupg_v2.0 and tried to import the old secret and public keys:

$ mv ~/.gnupg ~/.gnupg_v2.0
# This is GnuPG 2.1
$ gpg2 --import ~/.gnupg_v2.0/secring.gpg

The import left me with unusable keys as described in the issue report above.

To figure out the situation, It is easier for me to see the output of GnuPG 1.4/2.0 for the key: gpg -k and gpg -K.

Listing the keys with GnuPG 2.1 yields the output given in the initial report: the two subkeys are missing (public and private subkeys).
A listing for GnuPG 1.4/2.0 is not applicable to the new imported keys since their format (pubring.kbx) is GnuPG 2.1 only.

The question:
Why does GnuPG 2.1 fail to import subkeys directly from @secring.gpg@ if the primary secret key is missing? I tested importing another GnuPG key from secring.gpg without missing primary secret key and GnuPG 2.1 imported it correctly.

gniibe triaged this task as Normal priority.
gniibe added a project: gnupg (gpg21).

Thanks for your explanation. Now, I got it.

Let me explain what's going on. I describe from viewpoint of developers. (Not intended to justify/advocate current behavior. )

In your .gnupg/secring.gpg, the primary key is expired.

When you update the expiration, I think that something was going wrong and only public part of keys was updated. This may be a possible bug of GnuPG 1.4/2.0 --- (*1)

In GnuPG 1.4/2.0, secret part of keys (primary and subkey) are recorded in .gnupg/secring.gpg in OpenPGP format, just like for public part of keys.

In GnuPG 2.1, secret part of keys are under control of gpg-agent in libgcrypt's format (of SEXP). Public part of keys are recorded in .gnupg/pubring.kbx, and .gnupg/pubring.gpg in OpenPGP format is also supported.

When --export-secret-keys, the both information (from pubring.gpg and from secring.gpg) are merged into the output; Thus, the expiration information in the output is the one in pubring.gpg.

When importing .gnupg/secring.gpg directly, GnuPG only looks at the information of the record, and I think that it is handled as an expired key. It would be useful to import even if it is expired and even if it will be recorded as an expired key, as is. This may be a possible bug of GnuPG 2.1. --- (*2)

I think that from user's point of view, we have two posiible bugs of (*1) and (*2).
In this ticket, I will handle (*2) for now.

Thanks, gniibe, for the quick reply.

Since it might be related to the expiration date, let me add some more information.

On my computer which stores the private primary key the expiration date is correct for both private and public primary key.
The initial transportation of the public primary key, public subkeys and private subkeys from the computer to the laptop was done with

# This was all done with GnuPG 2.0
computer$ gpg2 --armour --export-secret-subkeys <keyid> > secsubkeys.gpg
computer$ gpg2 --armour --export <keyid> > pubkey.gpg

laptop$ gpg2 --import pubkey.gpg
laptop$ gpg2 --import secsubkeys.gpg

To update the expiration date, I updated it on computer and exported the public key:

computer$ gpg2 --edit-key <keyid> # update expiration date
computer$ gpg2 --armour --export <keyid> > pubkey.gpg
laptop$ gpg2 --import pubkey.gpg

Apart from the strange output with [expires: 2017-01-01] (not that it says expires not expired although the date is in the past), I have no problems and generate valid signatures.

In your .gnupg/secring.gpg, the primary key is expired.
When you update the expiration, I think that something was going wrong and only public part of keys was updated. This may be a possible bug of GnuPG 1.4/2.0 --- (*1)

Here you surprised me. I though the expiration is derived from self-signatures which are signatures of the public key and thus stored in pubkey.gpg.
I didn't know that secring.gpg also stores self-signatures. You never stop learning.

I conclude from your description that I should in addition to the public key also export/import the private subkeys when I renew the expiration date.
Is that correct?
While we are at it: How should I "transport" the updated expiration date in GnuPG 2.1: Is exporting the public key (--export) enough or should I export public key and private subkeys (--export & --export-secret-subkeys)?

If the inconsistent expiration dates of my public and private keys can be explained by my doing above, then there seems to be no bug in GnuPG 1.4/2.0; apart for the fact that pubring.gpg and secring.gpg duplicate information which might be out of sync but that is---I think---fixed in GnuPG 2.1. ;-)
However, it might be useful to also update the dates in secring.gpg if they are update in pubring.gpg. I'll be happy to file a new bug report for this if you deem it worth the effort.

When importing .gnupg/secring.gpg directly, GnuPG only looks at the information of the record, and I think that it is handled as an expired key. It would be useful to import even if it is expired and even if it will be recorded as an expired key, as is. This may be a possible bug of GnuPG 2.1. --- (*2)

As you describe it, this seems to be definitely a bug. Either GnuPG 2.1 does not import expired keys, then no key (neither primary nor subkey) should be imported.
Or GnuPG does import expired keys, then also the subkeys should be imported.

While trying to identify the cause of this problem, I found that the import doesn't success with expired key.

I think that you got an error message when you import the key:

key XXXX: failed to re-lookup public key

Didn't you?

One way to import expired key is using --faked-system-time option like "--fake-system-time 20161601T000000".

Hi gniibe

Thanks for coming back to this issue.

While trying to identify the cause of this problem, I found that the import doesn't success with expired key.

I think that you got an error message when you import the key:

key XXXX: failed to re-lookup public key

Didn't you?

I'm afraid that isn't the case. I redid the import here is the complete output (from a fresh reimport, not the reimport of the original post):

$ gpg2 --version
gpg (GnuPG) 2.1.20
libgcrypt 1.8.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /home/tux/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

$ gpg2 --import ~/.gnupg.old/secring.gpg
[pinentry pops up asking for the passphrase]
gpg: key ABCDABCDABCDABCD: public key "Tux Penguin <tux@example.org>" imported
gpg: To migrate 'secring.gpg', with each smartcard, run: gpg --card-status
gpg: key ABCDABCDABCDABCD: secret key imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg:       secret keys read: 1
gpg:   secret keys imported: 1

$ gpg2 --list-keys
/home/tux/.gnupg/pubring.kbx
----------------------------
pub   rsa4096/0xABCDABCDABCDABCD 2016-01-01 [C] [expired: 2017-01-01]
      12341234123412341234ABCDABCDABCDABCDABCD
uid                   [ expired] Tux Penguin <tux@example.org>
uid                   [ expired] Tux Penguin

$ gpg2 --list-secret-keys
/home/tux/.gnupg/pubring.kbx
----------------------------
sec#  rsa4096/0xABCDABCDABCDABCD 2016-01-01 [C] [expired: 2017-01-01]
      12341234123412341234ABCDABCDABCDABCDABCD
uid                   [ expired] Tux Penguin <tux@example.org>
uid                   [ expired] Tux Penguin

Reconstructing the timeline I can also say for sure that the key was not expired.
Furthermore, I have another key "B" in my keyring which contains its primary secret key and has the same expiration dates as the one without the primary secrete key. This key "B" was imported without any problem. That's part of the reason why I'm pretty sure that the lacking primary private key has something to do with this odd behaviour.

Some more information:
The key has two self signatures (in the GnuPG 2.0 storage)

  • 2016-01-01 -- 2016-06-01
  • 2017-01-01 -- 2018-01-01

However, imported is only the first self-signature (the one which was made on key-creation).

I'd like to create somehow a reproducible scenario/script for you. Sadly I don't have GnuPG 2.0 any more. Is there a way to force GnuPG 2.1. to use the legacy key storage format?
If not, is it safe to clone the source repository, making an in-source build of the GnuPG 2.0 branch and running the generated gpg2 executable in place (i.e. without installing it on the system)?

werner edited projects, added gnupg (gpg22); removed gnupg (gpg21).

What I use to force the old keyring format is to export a public key to a file and rename that to pubring.gpg. And of course delete the pubring.kbx.

What I use to force the old keyring format is to export a public key to a file and rename that to pubring.gpg. And of course delete the pubring.kbx.

But this does not create a secring.gpg, does it?

Anyway, I created a test case for my problem (see attached script). There I use gpg1 to convert the keys back into the old secring.gpg format.
Before you use it, you might need to adjust the variables GPG1 and GPG2, which must point to a GPG1 and GPG2 executable, respectively.
Also be warned that the script creates a lot of output during key creation and setup.
Lastly, in case you are doing this in the future: After 1.1.2018 you need to add another expiration date extension in the for-loop for expiretime in ${TIME2} ${TIME3} ${TIME4}; do

For me the end result looked like:

Result
======

/tmp/tmp.gzrK1g8vp9/homedir_gpg1/pubring.gpg
--------------------------------------------
pub   rsa1024 2017-01-01 [C] [expires: 2018-01-09]
      09B61C21CB9F63934A1D5ED7983650E340542A85
uid           [ unknown] Tux Penguin
sub   rsa1024 2017-01-01 [S] [expires: 2018-01-09]
sub   rsa1024 2017-01-01 [E] [expires: 2018-01-09]

/tmp/tmp.gzrK1g8vp9/homedir_dest/pubring.kbx
--------------------------------------------
pub   rsa1024 2017-01-01 [C] [expired: 2017-04-11]
      09B61C21CB9F63934A1D5ED7983650E340542A85
uid           [ expired] Tux Penguin

gpg: starting migration from earlier GnuPG versions
gpg: porting secret keys from '/tmp/tmp.gzrK1g8vp9/homedir_gpg1/secring.gpg' to gpg-agent
gpg: To migrate 'secring.gpg', with each smartcard, run: gpg --card-status
gpg: key 983650E340542A85: secret key imported
gpg: migration succeeded
/tmp/tmp.gzrK1g8vp9/homedir_gpg1/pubring.gpg
--------------------------------------------
sec#  rsa1024 2017-01-01 [C] [expires: 2018-01-09]
      09B61C21CB9F63934A1D5ED7983650E340542A85
uid           [ unknown] Tux Penguin
ssb   rsa1024 2017-01-01 [S] [expires: 2018-01-09]
ssb   rsa1024 2017-01-01 [E] [expires: 2018-01-09]

/tmp/tmp.gzrK1g8vp9/homedir_dest/pubring.kbx
--------------------------------------------
sec#  rsa1024 2017-01-01 [C] [expired: 2017-04-11]
      09B61C21CB9F63934A1D5ED7983650E340542A85
uid           [ expired] Tux Penguin

BASEDIR=/tmp/tmp.gzrK1g8vp9

Thanks for the script.
I confirmed that secring.gpg is not updated when importing key with updated expiration date, by GPG1.
So, for GPG2, it is expired key.

gniibe renamed this task from GnuPG 2.1 cannot import secret keys with missing primary key to GnuPG 2.1 cannot import secret keys from GnuPG secring.gpg directly (if it is expired by original expiration date).Apr 13 2018, 6:57 AM

I changed the title to express the problem.

gniibe renamed this task from GnuPG 2.1 cannot import secret keys from GnuPG secring.gpg directly (if it is expired by original expiration date) to GnuPG 2.2 cannot import secret keys from 1.4/2.0's secring.gpg directly (if it is expired by original expiration date).Apr 13 2018, 6:58 AM

Do we have an update on this? I seem to be having similar issue. Cannot import a key from my old gpg 1.x version to gpg 2.x version

GnuPG itself does that in in gnupg/g10/migrate.c. We need to fixed this.

The problem here is that we we did not sync the pubring with the secring in old version (2.1 removed the secring concept also due to that syncing problem). Now if we migrate (or plainly import a secring.gpg), gpg does not see any updated self-signatures and this is the reason why outdated self-signatures are used.

Solution is to merge self-signatures from the pubring during migration. For direct import we can/t do anything. As a workaround it should work to import the public key again:

gpg-old --export  USERID | gpg-new --import

Not tested, though

Ahh there we go... I just did a --recv-keys to update from the keyserver and that picked up the extended expiration date.

werner lowered the priority of this task from High to Normal.Jul 5 2018, 5:07 PM

We have a workaround thus lowering the priority.

werner edited projects, added gnupg, workaround; removed gnupg (gpg23).