Page MenuHome GnuPG
Create Task
Maniphest T7106

Trailing newline trouble in clearsigned message generation and verification
Open, LowPublic
Actions

Description

gpg --clearsign does not sign the actual input message if the message ends in a newline. Rather, it signs the message without the final trailing newline. gpg --verify's --output option does not emit the actual signed data, if the data does not end in a newline. Rather, it emits the signed data plus a trailing newline.

Clearsigned messages and whitespace have quite a bit of legacy trickiness, as noted in T1419 and T1692.

In particular, RFC 4880 observes that whitespace at the end of newlines will be stripped, and that:

An implementation SHOULD add a line break after the cleartext, but
MAY omit it if the cleartext ends with a line break. This is for
visual clarity.

Presumably this suggestion is meant to add a trailing line break to the message before signing, as it will lend more visual clarity to the final clearsigned message.

And, of course, whitespace at the end of a line in a clearsigned message is all stripped:

Also, any trailing whitespace -- spaces (0x20) and tabs (0x09) -- at
the end of any line is removed when the cleartext signature is
generated.

The spec is also clear that the final newline before the BEGIN PGP SIGNATURE is *not* part of the message being signed:

The line ending (i.e., the <CR><LF>) before the '-----BEGIN PGP
SIGNATURE-----' line that terminates the signed text is not
considered part of the signed text.

However, gpg --clearsign seems to *strip* the trailing newline from input before signing. By contrast, gpg --extmode --detach-sign does not do that.

Stripping the trailing newline is not the same as trimming the whitespace at the end of a line, or adding a line break if no such
linebreak exists.

Instead, the message signed with gpg --clearsign is actually the input message with one *fewer* newline, as long as the original message ended in a newline.

Here is a demonstration of this (mis)behavior, where six deterministic signatures are made in the same second, and then compared to each other. We sign each of three messages (with 0, 1, or 2 trailing newlines) using both --textmode --detach-sign and --clearsign. Then we compare the results:

$ for newlines in 0 1 2; do python3 -c 'print("test" + "\n"*'$newlines', end="")' > $newlines.txt; done
$ for nl in 0 1 2; do gpg -u $PGPID --clearsign < $nl.txt > $nl.signed & gpg -u $PGPID --detach-sign --textmode --armor < $nl.txt > $nl.sig & done; wait
[1] 1165590
[2] 1165591
[3] 1165592
[4] 1165593
[5] 1165594
[6] 1165601
[1]   Done                    gpg -u $PGPID --clearsign < $nl.txt > $nl.signed
[2]   Done                    gpg -u $PGPID --detach-sign --textmode --armor < $nl.txt > $nl.sig
[3]   Done                    gpg -u $PGPID --clearsign < $nl.txt > $nl.signed
[4]   Done                    gpg -u $PGPID --detach-sign --textmode --armor < $nl.txt > $nl.sig
[5]-  Done                    gpg -u $PGPID --clearsign < $nl.txt > $nl.signed
[6]+  Done                    gpg -u $PGPID --detach-sign --textmode --armor < $nl.txt > $nl.sig
$ diff -u 0.sig 0.signed 
--- 0.sig	2024-04-26 10:25:39.238425091 -0400
+++ 0.signed	2024-04-26 10:25:39.374431359 -0400
@@ -1,3 +1,7 @@
+-----BEGIN PGP SIGNED MESSAGE-----
+Hash: SHA256
+
+test
 -----BEGIN PGP SIGNATURE-----
 
 iHUEARYIAB0WIQR0vATEPYYIS+hnLAZ3LRYeNc1LgQUCZiu5YwAKCRB3LRYeNc1L
$ diff -u 1.sig 1.signed 
--- 1.sig	2024-04-26 10:25:39.226424538 -0400
+++ 1.signed	2024-04-26 10:25:39.234424907 -0400
@@ -1,7 +1,11 @@
+-----BEGIN PGP SIGNED MESSAGE-----
+Hash: SHA256
+
+test
 -----BEGIN PGP SIGNATURE-----
 
 iHUEARYIAB0WIQR0vATEPYYIS+hnLAZ3LRYeNc1LgQUCZiu5YwAKCRB3LRYeNc1L
-gZJPAP9TioWz+1YpILMV6EDwGT5aiyWKU96IZdOedgkCvWU8oAEAvfUs8d44otco
-OJ2QKD0jxQoDCzZ3fc1Vr4rLTvGN7QA=
-=d3IB
+gT7ZAQCerr6oFMzanfgLEfH5phZ6Rpxb7e6GRi2XZTCGfsKItQEA9BTnJ4Jzjq00
+URFjZryC0V6SR0YFFWaBDN6mI/yTHQY=
+=cpUs
 -----END PGP SIGNATURE-----
$ diff -u 0.signed 1.signed 
$ diff -u 0.sig 1.signed 
--- 0.sig	2024-04-26 10:25:39.238425091 -0400
+++ 1.signed	2024-04-26 10:25:39.234424907 -0400
@@ -1,3 +1,7 @@
+-----BEGIN PGP SIGNED MESSAGE-----
+Hash: SHA256
+
+test
 -----BEGIN PGP SIGNATURE-----
 
 iHUEARYIAB0WIQR0vATEPYYIS+hnLAZ3LRYeNc1LgQUCZiu5YwAKCRB3LRYeNc1L
$ diff -u 1.sig 2.signed 
--- 1.sig	2024-04-26 10:25:39.226424538 -0400
+++ 2.signed	2024-04-26 10:25:39.222424353 -0400
@@ -1,3 +1,8 @@
+-----BEGIN PGP SIGNED MESSAGE-----
+Hash: SHA256
+
+test
+
 -----BEGIN PGP SIGNATURE-----
 
 iHUEARYIAB0WIQR0vATEPYYIS+hnLAZ3LRYeNc1LgQUCZiu5YwAKCRB3LRYeNc1L
$ gpg --version
gpg (GnuPG) 2.2.40
libgcrypt 1.10.3
Copyright (C) 2022 g10 Code GmbH
License GNU GPL-3.0-or-later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /home/dkg/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2
$

As you can see from above, clearsigning either of test\n and test results in the same signature, which is the same signature that is made via a --textmode --detach-sign over test (with no trailing newline).

Each --clearsigned message's signature corresponds to the signature made a --textmode --detach-sign of a message with one fewer trailing newline.

I suspect this has never been noticed because gpg --verify's --output option always emits a trailing newline, adding it if the signed data did *not* have a trailing newline:

$ for x in 0 1 2; do gpg --verify --output $x.output < $x.signed ; done
gpg: Signature made Fri 26 Apr 2024 10:25:39 AM EDT
gpg:                using EDDSA key 74BC04C43D86084BE8672C06772D161E35CD4B81
gpg: Good signature from "Daniel Kahn Gillmor" [ultimate]
gpg: Signature made Fri 26 Apr 2024 10:25:39 AM EDT
gpg:                using EDDSA key 74BC04C43D86084BE8672C06772D161E35CD4B81
gpg: Good signature from "Daniel Kahn Gillmor" [ultimate]
gpg: Signature made Fri 26 Apr 2024 10:25:39 AM EDT
gpg:                using EDDSA key 74BC04C43D86084BE8672C06772D161E35CD4B81
gpg: Good signature from "Daniel Kahn Gillmor" [ultimate]
$ for x in 0 1 2; do hd < $x.output; done
00000000  74 65 73 74 0a                                    |test.|
00000005
00000000  74 65 73 74 0a                                    |test.|
00000005
00000000  74 65 73 74 0a 0a                                 |test..|
00000006
$

If i'm understanding RFC 4880 correctly, gpg --clearsign could become compliant by either:

  • not stripping the trailing newline before signing, or
  • not stripping the trailing newline before signing, and *adding* a trailing newline if none exists

gpg --verify's --output option should stop emitting a trailing newline when none exists, so that it accurately represents the signed data.

Details

Version
2.2.40

Related Objects

Event Timeline

dkg created this task.Fri, Apr 26, 5:00 PM
• werner triaged this task as Low priority.Fri, Apr 26, 5:09 PM
• werner added a subscriber: • werner.

This has been implemented and tested to be compatible with PGP - a looong time ago. iirc this was discussed around 1999 but might be only by private mail between the PGP hackers and me. Thus any change now might break PGP - which is still widely used (although mostly for encryption).

dkg added a comment.EditedFri, Apr 26, 6:06 PM

I understand the desire for stable behavior, and i agree that a change here might affect verification of existing signatures (and might mean producing signatures that will be misinterpreted by older versions).

If this is expected OpenPGP behavior, it was incorrect in RFC 2440 (released in 1998), and if you discussed the details after RFC 2440 was released, those details remained the same in RFC 4880 (2007). The same text appears in draft-ietf-openpgp-crypto-refresh, and in draft-koch-librepgp.

When the actual data signed does not include the trailing newline, the material produced by --verify --output is simply wrong, as demonstrated by the comparison of the detached signature with the clearsig.

The documentation says:

The option --output may be used to write out the actual signed data, but there are other pitfalls with this format as well. It is suggested to avoid cleartext signatures in favor of detached signatures

I fully agree with preferring detached signatures, fwiw, but i'm not sure what "other pitfalls with this format" is supposed to cover (e.g. does it refer to §7.3 of the crypto-refresh where at least some of these limitations are documented?). If it means in part "GnuPG doesn't implement the problematic cleartext signing framework according to two decades of specifications", that seems fair too, i guess.

Maybe at some point in the future, the behavior can be brought in line with the specifications using one of the --compatibility modes offered by GnuPG.