Page MenuHome GnuPG

Mutt PGP Error: "Could not decrypt PGP message" & "Could not copy message" on Ubuntu machine but works on macOS machine
Closed, ResolvedPublic

Description

Configuration files are identical on both machines since they are shared from a cloud drive.

Messages sent as a test to myself from the Ubuntu machine 'can' be decrypted and read on the macOS machine in mutt with no problems. Also, both machines sending to themselves work.

The problem is decrypting the messages on the Ubuntu machine sent from the macOS.

The Ubuntu machine has a GnugPG version which is a little older (2 years) than the macOS machine but both are above version 2.1 which saw some changes from previous versions.

I'm not using gpgme or its gpg-agent in the configuration files since I had issues with it a) defaulting to s/mime instead of PGP, b) not accepting self-signed X509 certs.

Pure PGP config

MUTT ERROR MSG

On selecting a message in mutt's message index, I get the following:

Invoking PGP ...
Could not decrypt PGP message
Could not copy message

QUESTION

  1. Is this a GnuPG backward compatibility issue?
  2. Can this be resolved and how?

Version Details

macOS:
$ gpg --version
gpg (GnuPG) 2.3.4
libgcrypt 1.10.0
Copyright (C) 2021 Free Software Foundation, Inc.
License GNU GPL-3.0-or-later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /Users/tonybarganski/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
AEAD: EAX, OCB
Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

---
Mutt 2.2.1 (2022-02-19)
Copyright (C) 1996-2022 Michael R. Elkins and others.

Compile options:
-DOMAIN
+DEBUG
-HOMESPOOL  +USE_SETGID  +USE_DOTLOCK  +DL_STANDALONE  +USE_FCNTL  -USE_FLOCK
+USE_POP  +USE_IMAP  +USE_SMTP  
+USE_SSL_OPENSSL  -USE_SSL_GNUTLS  +USE_SASL  -USE_GSASL  +USE_GSS  +HAVE_GETADDRINFO
+HAVE_REGCOMP  -USE_GNU_REGEX  
+HAVE_COLOR  +HAVE_START_COLOR  +HAVE_TYPEAHEAD  +HAVE_BKGDSET
+HAVE_CURS_SET  +HAVE_META  +HAVE_RESIZETERM  +HAVE_FUTIMENS  
+CRYPT_BACKEND_CLASSIC_PGP  +CRYPT_BACKEND_CLASSIC_SMIME  +CRYPT_BACKEND_GPGME
-EXACT_ADDRESS  -SUN_ATTACHMENT  
+ENABLE_NLS  -LOCALES_HACK  +HAVE_WC_FUNCS  +HAVE_LANGINFO_CODESET  +HAVE_LANGINFO_YESEXPR
+HAVE_ICONV  -ICONV_NONTRANS  -HAVE_LIBIDN  -HAVE_LIBIDN2  +HAVE_GETSID  +USE_HCACHE  
+USE_SIDEBAR  -USE_COMPRESSED  -USE_INOTIFY  
-ISPELL

Ubuntu20.04
gpg (GnuPG) 2.2.19
libgcrypt 1.8.5
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /home/tony/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

---
Mutt 1.13.2 (2019-12-18)
Copyright (C) 1996-2016 Michael R. Elkins and others.
Mutt comes with ABSOLUTELY NO WARRANTY; for details type `mutt -vv'.
Mutt is free software, and you are welcome to redistribute it
under certain conditions; type `mutt -vv' for details.

System: Linux 5.13.0-35-generic (x86_64)
ncurses: ncurses 6.2.20200212 (compiled with 6.2)
libidn: 1.33 (compiled with 1.33)
hcache backend: tokyocabinet 1.4.48

Compiler:
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/9/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:hsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.3.0-17ubuntu1~20.04' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable- languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --program-prefix=x86_64-linux- gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-     nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-  vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --      disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-           targets=nvptx-none=/build/gcc-9-HskZEa/gcc-9-9.3.0/debian/tmp-nvptx/usr,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-     gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)

Event Timeline

werner added a subscriber: werner.

Sorry, without detailed output of gpg we can't help you here. This is definitely not a GnuPG bug because too many people are using mutt and gnupg. You should also "set crypt_use_gpgme" -it works far better.

TonyBarganski raised the priority of this task from Low to Normal.Mar 22 2022, 11:02 PM

@werner

Hi.

Please refer to the open Mutt Bug issue 401 below regarding the troubleshooting we've performed which seems to suggest there *might* be something a skew on the gpg binaries.

A summary
Kevin J. McCarthy @kevin8t8 · 5 hours ago:
*"However, if that still doesn't do the trick then I agree this is most likely a gpg issue. Of course, Mutt could have a bug somewhere, but for the most part it is handing off decryption and encryption to gpg on both sides. And the fact that it is erroring out in all 3 cases: 1) gpgme, 2) classic-mode, and 3) manual invocation from the command line points to something off with gpg itself."*

Mutt Bug Issue

$ gpg -vv -d macos.msg


gpg: armor: BEGIN PGP MESSAGE
# off=0 ctb=84 tag=1 hlen=2 plen=94
:pubkey enc packet: version 3, algo 18, keyid 54E817B2B727BD94
        data: [263 bits]
        data: [392 bits]
gpg: public key is 54E817B2B727BD94
gpg: using subkey 54E817B2B727BD94 instead of primary key 02100B80AD7D48D0
gpg: public key encrypted data: good DEK
# off=96 ctb=84 tag=1 hlen=2 plen=94
:pubkey enc packet: version 3, algo 18, keyid 3D95ADAB42BDCFD1
        data: [263 bits]
        data: [392 bits]
gpg: public key is 3D95ADAB42BDCFD1
# off=192 ctb=d4 tag=20 hlen=3 plen=453 new-ctb
:unknown packet: type 20, length 453
dump: 01 09 02 10 f0 01 ea 9e  3e 2e 59 0c b2 56 79 24  f9 4e 31 ec 0f be e1 75
  24: 46 db b5 ef 20 01 92 65  fe 28 d3 74 cb 21 09 b5  97 28 47 ff 80 e1 67 25
  48: 51 5f ad a3 a5 92 2f a1  8d db c2 b6 a6 aa f4 8e  2b b1 ad 00 e2 a6 11 82
  72: da c8 d2 b4 c1 16 0b 82  4a 00 a9 1e b6 0a 1f 88  2f c4 71 d5 3e b4 28 09
  96: 0e 2f 25 88 a1 18 e5 ca  fd aa 7f 47 08 13 10 60  2c 91 75 a5 78 6d 8e d9
 120: 34 da 1c ed 7c 71 7a 89  59 ee b9 81 76 2a c6 5c  59 69 5f 16 58 16 42 4b
 144: 87 c8 6b cc b4 ed 53 ba  40 a8 91 cd 37 7c a4 bd  b8 ba ec 95 6f 54 07 8b
 168: a5 af d4 d4 ee 64 c6 f7  da 2e 44 90 72 07 34 84  ed 62 fe 80 fd a6 11 14
 192: d9 b2 8f 28 7c 02 3d 61  f2 41 43 48 da c2 33 96  80 4b 96 1c 64 17 45 81
 216: ad b2 a6 36 0d ae 42 e1  17 ca b7 d0 04 e8 a6 55  30 57 08 ae ca a0 b1 da
 240: 9c 99 14 77 57 b2 cb 34  34 d1 4b 13 da 49 49 cc  70 34 82 20 39 57 6a b8
 264: 77 4d f5 99 3a b3 81 3b  d8 eb 94 6c 4f 82 f6 1c  41 82 6d b6 ec f2 dc 44
 288: 56 e7 7a 98 64 07 eb ca  86 d9 8a 60 96 f5 6d d2  2c 19 9f 61 ca b7 2e 4e
 312: c3 09 fc e7 a1 e5 1f df  e0 c0 b1 9f 3d 15 36 73  3f 3a 96 34 8d 14 ed c6
 336: 8b b5 0d 03 85 98 47 fd  e6 d2 fb 0e 24 57 29 aa  7b ac 31 a9 6f 4e 6f 4e
 360: f7 dd 1b 5a 74 c5 12 39  1b b1 02 6c 42 c8 36 30  85 87 03 75 13 17 3d 46
 384: 55 12 58 2c b6 e0 e0 05  b6 c7 92 a0 a4 db ce ca  67 09 9d 1f a1 8a e5 7f
 408: a1 23 e1 2d 67 01 ca 01  6d fa 7d 80 1b 46 11 ec  58 b8 b5 1a 34 65 e8 32
 432: a8 62 1f 3c 27 8c 3a 07  03 65 ad ac 34 8c 07 d1  ac 60 11 61 be
TonyBarganski raised the priority of this task from Normal to Unbreak Now!.Mar 24 2022, 10:32 AM

Since decryption is broken, I'm raising the priority level of this ticket.
It would be wonderful if someone can take a more detailed look into this problem. :)

werner lowered the priority of this task from Unbreak Now! to Normal.Mar 24 2022, 11:53 PM

Packet 20 is the new AEAD packet which GnuPG 2.3 can generate and does generate if all recipients have new keys generated with such a versions. However, the version of gpg you use now does not support AEAD and thus fails.

  • You created a key with a new version of gpg (2.3.x) and thus listing AEAD as its preferences
  • You sent a mail to that key using gnupg 2.3 and AEAD was used
  • You used GnuPG 2.2 to decrypt this message encrypted to your own AEAD capable key. However that particular 2.2 has not even read-only support for AEAD.

The general rule for downgrading to an older versions is to adjust the algorithm preferences for keys generated with the newer version.

Thanks @werner

  1. So, firstly, can we get an error message that states something to that effect AND can also be displayed by Mutt?
  1. Secondly, are you saying that I should simply recreate the keys with the 'lowest' gpg package used?

Not sure what to do with the keys that are on the openpg.org keyservers but I'll investigate if I can update those.

werner claimed this task.
  • No we can't because current GnuPG 2.2 versions are able to decrypt such AEAD data.
  • No. Do not use a lower minor version on one of your systems or change the preferences of your new keys if you need to use outdated versions.

@werner

Hi Werner
.
Firstly, let me say how much I appreciate the work you and others do at OpenPG.org! Really.

I'm re-openning this ticket as I do not have a clear solution for this issue. Its not workable.

  1. I checked the gpg man pages and there are almost a dozen options in reference to AEAD and non of the switches seem to switch it off for new key generation.
  1. As things stand right now, someone with a Public key created on gpg version 2.3 on a macOS cannot privately communicate with someone using a Linux server, news group or Linux Desktop.

I haven't tested this on Microsoft's WSL but I imagine it will encounter the same problems since it usually installed a copy of Ubuntu 20.04. WSL incidentally is so common with the devs that I've worked with over the past few years. It would be great to get this working.

  1. There is inadequate documentation for this issue and throughout actually. I can possible help with that.

Can we have a dialog as to what the missing parts are and how this can be achieved in the interest of facilitating new adoption of this otherwise solid technology that underpins one of our fundamental human rights - freedom of speech?

Tony

Use a gpg 2.3 version:

$ gpg --edit-key YOUR_KEY
[...]
gpg> pref 
[ unknown] (1). aead-eax@test.gnupg.org
     S9 S8 S7 S2 A1 H10 H9 H8 H11 H2 Z2 Z3 Z1 [mdc] [aead] [no-ks-modify]
gpg> setpref  S9 S8 S7 S2 H10 H9 H8 H11 H2 Z2 Z3 Z1
Set preference list to:
     Cipher: AES256, AES192, AES, 3DES
     AEAD: 
     Digest: SHA512, SHA384, SHA256, SHA224, SHA1
     Compression: ZLIB, BZIP2, ZIP, Uncompressed
     Features: MDC, Keyserver no-modify
Really update the preferences? (y/N) y
[...]
gpg> save
  1. As things stand right now, someone with a Public key created on gpg version 2.3 on a macOS cannot privately communicate with someone using a Linux server, news group or Linux Desktop.

I'm pretty sure that this claim is false. Above Werner wrote "Packet 20 is the new AEAD packet which GnuPG 2.3 can generate and does generate if all recipients have new keys generated with such a versions."

Your problem is that you created a key with gpg 2.3 (or a newer version of gpg 2.2) and then copied this key to another machine with a very old version of gpg 2.2.

Despite Werner's comment "Do not use a lower minor version on one of your systems or change the preferences of your new keys if you need to use outdated versions.", I'd go for the pragmatic workaround to "recreate the keys with the 'lowest' gpg package used".

@ikloecker thanks for your reply.

RE: "I'd go for the pragmatic workaround to 'recreate the keys with the 'lowest' gpg package used'"

  1. If I create entirely NEW keys, how do I update my old keys on the 'keys.openpgp.org' keyserver?
  1. Can we get some documentation in PLAIN ENGLISH on this topic?

I found @werner language very PGP specific and I couldn't understand it, and I'm a techie.

  1. This suggestion/situation to me says that backward compatibility in PGP is broken!

Happy to be proven otherwise.

Thanks in advance.

Backward compatibility means that newer versions work with data created with older versions of a program. What you are asking for is forward compatibility, i.e. you want older versions of a program to work with data created with newer versions of a program. In the extreme that would mean that gpg must not use modern encryption algorithms because old versions of gpg cannot deal with them. It should be obvious that this doesn't make any sense.

To get compatibility with older versions of gpg you have to tell the newer gpg that it should create data (e.g. keys or encrypted data) that is compatible with older versions of gpg. That's what Werner is trying to tell you with "change the preferences of your new keys if you need to use outdated versions".

Regarding

If I create entirely NEW keys, how do I update my old keys on the 'keys.openpgp.org' keyserver?

one possibility is to revoke the old keys and then upload the revoked keys to the keyserver. This way you communicate to the world that the old keys shouldn't be used anymore.

@ikloecker Thanks for the clarification (appreciated).

  • So (how) do we do that? How do I create keys that can be read by gpg 2.2?

Ubuntu 20.04 package management only supports this version. Its not feasible to recompile gpg on Ubuntu20.04.

  • Is there some documentation you can point me to which explains this and what to do in such a situation?

Create the keys with gpg 2.2. I'm not aware of such documentation apart from the manual page of GnuPG. And, as I tried to explain, this situation isn't really different from any other software. If you create a document with the newest version of LibreOffice then you cannot expect it to look exactly the same with an older version of LibreOffice. It's your responsibility not to use new features of the new LibreOffice if you still need to use an older version on another machine.

Hi @werner
I had missed your earlier post quoted below on using setperf.

I've gone through the gpg man page but cannot see what those 1char + {1-2}digit preference codes refer to.

Where can I get a list of the correct option codes that I need to set please to turn off [aead]?

Tony

Use a gpg 2.3 version:

$ gpg --edit-key YOUR_KEY
[...]
gpg> pref 
[ unknown] (1). aead-eax@test.gnupg.org
     S9 S8 S7 S2 A1 H10 H9 H8 H11 H2 Z2 Z3 Z1 [mdc] [aead] [no-ks-modify]
gpg> setpref  S9 S8 S7 S2 H10 H9 H8 H11 H2 Z2 Z3 Z1
Set preference list to:
     Cipher: AES256, AES192, AES, 3DES
     AEAD: 
     Digest: SHA512, SHA384, SHA256, SHA224, SHA1
     Compression: ZLIB, BZIP2, ZIP, Uncompressed
     Features: MDC, Keyserver no-modify
Really update the preferences? (y/N) y
[...]
gpg> save

S9, etc. are short-hand IDs, for the cipher algorithms, digest algorithms, etc. Use showpref instead of pref to get the preference list in human-readable form (AES256, SHA512, etc.) instead of in expert form (cryptic IDs).

You can simply use the algorithm names instead of the IDs with setpref, e.g.

gpg> setpref AES256 AES192 AES 3DES SHA512 SHA384 SHA256 SHA224 SHA1 ZLIB BZIP2 ZIP

@werner
The setpref S9 S8 S7 S2 H10 H9 H8 H11 H2 Z2 Z3 Z1 worked!

But there is no documentation that I can find to tell me which setting took off the AEAD preference from my key.

This isn't very workable. I think there needs a ticket or two:

a) Create a description for the meaning of the unknown packet: type 20 error message and a link to documentation which discusses the options and how to get a working solution, i.e. change the key preferences to remove, in this case, AEAD capabilities from the key

b) Add documentation on how to change the preferences of a key and where to look up the available option codes - better still, have human readable options for AEAD

c) A writeup on the fact that this might happen and how to update the keyservers with one's updated keys

In fact, decent 2.2 versions (>=2.2.21) have the ability to decrypt AEAD packets - this has been implemented exactly for the case that some things get wrong at the user site. But we can't change old versions - we are not the Sirius Computer Corporation. I close this ticket because we can can't do anything if you are not able/willing to update to the latest version of the respective branch. Sorry.