Trailing newline trouble in clearsigned message generation and verification
Closed, WontfixPublic
Actions

Assigned To

Authored By

	dkg
	Apr 26 2024, 5:00 PM

Description

gpg --clearsign does not sign the actual input message if the message ends in a newline. Rather, it signs the message without the final trailing newline. gpg --verify's --output option does not emit the actual signed data, if the data does not end in a newline. Rather, it emits the signed data plus a trailing newline.

Clearsigned messages and whitespace have quite a bit of legacy trickiness, as noted in T1419 and T1692.

In particular, RFC 4880 observes that whitespace at the end of newlines will be stripped, and that:

An implementation SHOULD add a line break after the cleartext, but
MAY omit it if the cleartext ends with a line break. This is for
visual clarity.

Presumably this suggestion is meant to add a trailing line break to the message before signing, as it will lend more visual clarity to the final clearsigned message.

And, of course, whitespace at the end of a line in a clearsigned message is all stripped:

Also, any trailing whitespace -- spaces (0x20) and tabs (0x09) -- at
the end of any line is removed when the cleartext signature is
generated.

The spec is also clear that the final newline before the BEGIN PGP SIGNATURE is *not* part of the message being signed:

The line ending (i.e., the <CR><LF>) before the '-----BEGIN PGP
SIGNATURE-----' line that terminates the signed text is not
considered part of the signed text.

However, gpg --clearsign seems to *strip* the trailing newline from input before signing. By contrast, gpg --textmode --detach-sign does not do that.

Stripping the trailing newline is not the same as trimming the whitespace at the end of a line, or adding a line break if no such
linebreak exists.

Instead, the message signed with gpg --clearsign is actually the input message with one *fewer* newline, as long as the original message ended in a newline.

Here is a demonstration of this (mis)behavior, where six deterministic signatures are made in the same second, and then compared to each other. We sign each of three messages (with 0, 1, or 2 trailing newlines) using both --textmode --detach-sign and --clearsign. Then we compare the results:

$ for newlines in 0 1 2; do python3 -c 'print("test" + "\n"*'$newlines', end="")' > $newlines.txt; done
$ for nl in 0 1 2; do gpg -u $PGPID --clearsign < $nl.txt > $nl.signed & gpg -u $PGPID --detach-sign --textmode --armor < $nl.txt > $nl.sig & done; wait
[1] 1165590
[2] 1165591
[3] 1165592
[4] 1165593
[5] 1165594
[6] 1165601
[1]   Done                    gpg -u $PGPID --clearsign < $nl.txt > $nl.signed
[2]   Done                    gpg -u $PGPID --detach-sign --textmode --armor < $nl.txt > $nl.sig
[3]   Done                    gpg -u $PGPID --clearsign < $nl.txt > $nl.signed
[4]   Done                    gpg -u $PGPID --detach-sign --textmode --armor < $nl.txt > $nl.sig
[5]-  Done                    gpg -u $PGPID --clearsign < $nl.txt > $nl.signed
[6]+  Done                    gpg -u $PGPID --detach-sign --textmode --armor < $nl.txt > $nl.sig
$ diff -u 0.sig 0.signed 
--- 0.sig	2024-04-26 10:25:39.238425091 -0400
+++ 0.signed	2024-04-26 10:25:39.374431359 -0400
@@ -1,3 +1,7 @@
+-----BEGIN PGP SIGNED MESSAGE-----
+Hash: SHA256
+
+test
 -----BEGIN PGP SIGNATURE-----
 
 iHUEARYIAB0WIQR0vATEPYYIS+hnLAZ3LRYeNc1LgQUCZiu5YwAKCRB3LRYeNc1L
$ diff -u 1.sig 1.signed 
--- 1.sig	2024-04-26 10:25:39.226424538 -0400
+++ 1.signed	2024-04-26 10:25:39.234424907 -0400
@@ -1,7 +1,11 @@
+-----BEGIN PGP SIGNED MESSAGE-----
+Hash: SHA256
+
+test
 -----BEGIN PGP SIGNATURE-----
 
 iHUEARYIAB0WIQR0vATEPYYIS+hnLAZ3LRYeNc1LgQUCZiu5YwAKCRB3LRYeNc1L
-gZJPAP9TioWz+1YpILMV6EDwGT5aiyWKU96IZdOedgkCvWU8oAEAvfUs8d44otco
-OJ2QKD0jxQoDCzZ3fc1Vr4rLTvGN7QA=
-=d3IB
+gT7ZAQCerr6oFMzanfgLEfH5phZ6Rpxb7e6GRi2XZTCGfsKItQEA9BTnJ4Jzjq00
+URFjZryC0V6SR0YFFWaBDN6mI/yTHQY=
+=cpUs
 -----END PGP SIGNATURE-----
$ diff -u 0.signed 1.signed 
$ diff -u 0.sig 1.signed 
--- 0.sig	2024-04-26 10:25:39.238425091 -0400
+++ 1.signed	2024-04-26 10:25:39.234424907 -0400
@@ -1,3 +1,7 @@
+-----BEGIN PGP SIGNED MESSAGE-----
+Hash: SHA256
+
+test
 -----BEGIN PGP SIGNATURE-----
 
 iHUEARYIAB0WIQR0vATEPYYIS+hnLAZ3LRYeNc1LgQUCZiu5YwAKCRB3LRYeNc1L
$ diff -u 1.sig 2.signed 
--- 1.sig	2024-04-26 10:25:39.226424538 -0400
+++ 2.signed	2024-04-26 10:25:39.222424353 -0400
@@ -1,3 +1,8 @@
+-----BEGIN PGP SIGNED MESSAGE-----
+Hash: SHA256
+
+test
+
 -----BEGIN PGP SIGNATURE-----
 
 iHUEARYIAB0WIQR0vATEPYYIS+hnLAZ3LRYeNc1LgQUCZiu5YwAKCRB3LRYeNc1L
$ gpg --version
gpg (GnuPG) 2.2.40
libgcrypt 1.10.3
Copyright (C) 2022 g10 Code GmbH
License GNU GPL-3.0-or-later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /home/dkg/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2
$

As you can see from above, clearsigning either of test\n and test results in the same signature, which is the same signature that is made via a --textmode --detach-sign over test (with no trailing newline).

Each --clearsigned message's signature corresponds to the signature made a --textmode --detach-sign of a message with one fewer trailing newline.

I suspect this has never been noticed because gpg --verify's --output option always emits a trailing newline, adding it if the signed data did *not* have a trailing newline:

$ for x in 0 1 2; do gpg --verify --output $x.output < $x.signed ; done
gpg: Signature made Fri 26 Apr 2024 10:25:39 AM EDT
gpg:                using EDDSA key 74BC04C43D86084BE8672C06772D161E35CD4B81
gpg: Good signature from "Daniel Kahn Gillmor" [ultimate]
gpg: Signature made Fri 26 Apr 2024 10:25:39 AM EDT
gpg:                using EDDSA key 74BC04C43D86084BE8672C06772D161E35CD4B81
gpg: Good signature from "Daniel Kahn Gillmor" [ultimate]
gpg: Signature made Fri 26 Apr 2024 10:25:39 AM EDT
gpg:                using EDDSA key 74BC04C43D86084BE8672C06772D161E35CD4B81
gpg: Good signature from "Daniel Kahn Gillmor" [ultimate]
$ for x in 0 1 2; do hd < $x.output; done
00000000  74 65 73 74 0a                                    |test.|
00000005
00000000  74 65 73 74 0a                                    |test.|
00000005
00000000  74 65 73 74 0a 0a                                 |test..|
00000006
$

If i'm understanding RFC 4880 correctly, gpg --clearsign could become compliant by either:

not stripping the trailing newline before signing, or
not stripping the trailing newline before signing, and *adding* a trailing newline if none exists

gpg --verify's --output option should stop emitting a trailing newline when none exists, so that it accurately represents the signed data.

Details

Version: 2.2.40

Related Objects

Mentioned Here: T1419: pipe-through of clearsigned documents adds an extra newline
T1692: Validation of clear signed signatures broken for appended space(s)

Event Timeline

dkg created this task.Apr 26 2024, 5:00 PM

This has been implemented and tested to be compatible with PGP - a looong time ago. iirc this was discussed around 1999 but might be only by private mail between the PGP hackers and me. Thus any change now might break PGP - which is still widely used (although mostly for encryption).

I understand the desire for stable behavior, and i agree that a change here might affect verification of existing signatures (and might mean producing signatures that will be misinterpreted by older versions).

If this is expected OpenPGP behavior, it was incorrect in RFC 2440 (released in 1998), and if you discussed the details after RFC 2440 was released, those details remained the same in RFC 4880 (2007). The same text appears in draft-ietf-openpgp-crypto-refresh, and in draft-koch-librepgp.

When the actual data signed does not include the trailing newline, the material produced by --verify --output is simply wrong, as demonstrated by the comparison of the detached signature with the clearsig.

The documentation says:

The option --output may be used to write out the actual signed data, but there are other pitfalls with this format as well. It is suggested to avoid cleartext signatures in favor of detached signatures

I fully agree with preferring detached signatures, fwiw, but i'm not sure what "other pitfalls with this format" is supposed to cover (e.g. does it refer to §7.3 of the crypto-refresh where at least some of these limitations are documented?). If it means in part "GnuPG doesn't implement the problematic cleartext signing framework according to two decades of specifications", that seems fair too, i guess.

Maybe at some point in the future, the behavior can be brought in line with the specifications using one of the --compatibility modes offered by GnuPG.

dkg updated the task description. (Show Details)Nov 26 2024, 11:24 PM

Just a note that i've tested this and --clearsign appears to be problematic for 2.4.7 as well as 2.2.40.

The patch below fixes the master branch to be compliant with the standards for CSF message generation and verification.

It also tightens the test suite to avoid a future regression.

0001-gpg-Fix-newlines-in-Cleartext-Signature-Framework-CS.patch6 KBDownload

My comment from a year ago still holds true; you may want to fix your testing framework and re-openig this bug iff you can show that there will be no regression with PGP 7 and later.

In T7106#185462, @werner wrote:

This has been implemented and tested to be compatible with PGP - a looong time ago. iirc this was discussed around 1999 but might be only by private mail between the PGP hackers and me. Thus any change now might break PGP - which is still widely used (although mostly for encryption).

I tested cleartext signing with the historical PGP 6.5.8 on two messages:

echo -n "test" > msg1
echo "test" > msg2

and signed both like this:

pgp -sta +clearsig=on msgX

The resulting cleartext signatures align with what dkg describes as the specified and desired behavior:

$ cat msg1.asc
-----BEGIN PGP SIGNED MESSAGE-----

test
-----BEGIN PGP SIGNATURE-----
Version: PGP 6.5.8

iQCVAwUBaA9QQwQX1gyCdUtfAQGelAP+J4KR8l64yhTHnjD+Eh+I8V7M9H9RGWUU
0Hsdkr10AqMJQk8v/ujgVbzpUsRDrT44GCtEEYuF4CW7aGBl6j02zg9Nan1N47j7
NeHJYNeS0nVJR8hlBGhOe1IMfJ1S5c64upZkTs2sZH5D99u7TsckXkacvgHMmaKH
CYSmhfFeVJU=
=B5Zd
-----END PGP SIGNATURE-----

$ cat msg2.asc
-----BEGIN PGP SIGNED MESSAGE-----

test

-----BEGIN PGP SIGNATURE-----
Version: PGP 6.5.8

iQCVAwUBaA9QRQQX1gyCdUtfAQEr/QP/cA/MGAjB2GG/z6j1GPwIVeDyoRGhJX+Z
WhXF59kFMotdqxoOhjUEg1yN0H5poLY4yJz+SkO4BKkmAE+koQI1+L3HAA/CZPW4
7eMMtNCAmSPchPke1VYPzWozdNGPym5KKZB+t19wPvplwFBLpY48iY/Do+0ZGrzo
kfHYYPj3w/c=
=5ldb
-----END PGP SIGNATURE-----

In conclusion, I don't see how GnuPG's behavior could serve interoperability with the non-free PGP product.

(Fwiw, the public key to verify the signatures above is

... obviously this is just a test key for the purpose of this ticket)

heiko reopened this task as Open.Mon, Apr 28, 12:05 PM

This is just one build of PGP and you would need to test all versions on Windows, macOS and Unix. You also need to test against all versions of GnuPG since 1998 (when we started with interop tests). We won't change this in GnuPG and risk regression. If you have a problem with that go and add a fix to your tool - name it bug compatibility or whatever. And please do not re-open this bug.

Err, I don't see why I would "need to test" anything further.

You suggested that GnuPG's off-spec behavior might be following the behavior of the historical non-free PGP software.

I brought further evidence that GnuPG's behavior matches no other known implementation, including an original Linux build of the historical reference implementation.
Note that the version i tested (PGP 6.5.8) matches the year that you quoted above (1999), so it's not just "one build".
It is a build from exactly the era in which you recalled you had tested interop.

Now maybe you're right, and some obscure build of the non-free PGP software, say an Amiga build from 2002, actually matches what GnuPG does.
If you have the time to find that example, I'd love to hear about it. However, even if such an example exists, that should probably be considered a bug in that build, then.

In the meantime, I assume that this is a bug in GnuPG that has no justification, and that's ok. It's your project.

heiko changed the task status from Resolved to Wontfix.Mon, Apr 28, 3:08 PM

Trailing newline trouble in clearsigned message generation and verificationClosed, WontfixPublicActions

Description

Details

Related Objects

Event Timeline

Trailing newline trouble in clearsigned message generation and verification
Closed, WontfixPublic
Actions