Change Details

`gpg --clearsign` does not sign the actual input message if the message ends in a newline. Rather, it signs the message without the final trailing newline. `gpg --verify`'s `--output` option does not emit the actual signed data, if the data does not end in a newline. Rather, it emits the signed data plus a trailing newline. Clearsigned messages and whitespace have quite a bit of legacy trickiness, as noted in T1419 and T1692. In particular, RFC 4880 observes that whitespace at the end of newlines will be stripped, and that: > An implementation SHOULD add a line break after the cleartext, but > MAY omit it if the cleartext ends with a line break. This is for > visual clarity. Presumably this suggestion is meant to add a trailing line break to the message before signing, as it will lend more visual clarity to the final clearsigned message. And, of course, whitespace at the end of a line in a clearsigned message is all stripped: > Also, any trailing whitespace -- spaces (0x20) and tabs (0x09) -- at > the end of any line is removed when the cleartext signature is > generated. The spec is also clear that the final newline before the `BEGIN PGP SIGNATURE` is *not* part of the message being signed: > The line ending (i.e., the <CR><LF>) before the '-----BEGIN PGP > SIGNATURE-----' line that terminates the signed text is not > considered part of the signed text. However, `gpg --clearsign` seems to *strip* the trailing newline from input before signing. By contrast, `gpg --extmode --detach-sign` does not do that. Stripping the trailing newline is not the same as trimming the whitespace at the end of a line, or adding a line break if no such linebreak exists. Instead, the message signed with `gpg --clearsign` is actually the input message with one *fewer* newline, as long as the original message ended in a newline. Here is a demonstration of this (mis)behavior, where six deterministic signatures are made in the same second, and then compared to each other. We sign each of three messages (with 0, 1, or 2 trailing newlines) using both `--textmode --detach-sign` and `--clearsign`. Then we compare the results: ``` $ for newlines in 0 1 2; do python3 -c 'print("test" + "\n"*'$newlines', end="")' > $newlines.txt; done $ for nl in 0 1 2; do gpg -u $PGPID --clearsign < $nl.txt > $nl.signed & gpg -u $PGPID --detach-sign --textmode --armor < $nl.txt > $nl.sig & done; wait [1] 1165590 [2] 1165591 [3] 1165592 [4] 1165593 [5] 1165594 [6] 1165601 [1] Done gpg -u $PGPID --clearsign < $nl.txt > $nl.signed [2] Done gpg -u $PGPID --detach-sign --textmode --armor < $nl.txt > $nl.sig [3] Done gpg -u $PGPID --clearsign < $nl.txt > $nl.signed [4] Done gpg -u $PGPID --detach-sign --textmode --armor < $nl.txt > $nl.sig [5]- Done gpg -u $PGPID --clearsign < $nl.txt > $nl.signed [6]+ Done gpg -u $PGPID --detach-sign --textmode --armor < $nl.txt > $nl.sig $ diff -u 0.sig 0.signed --- 0.sig 2024-04-26 10:25:39.238425091 -0400 +++ 0.signed 2024-04-26 10:25:39.374431359 -0400 @@ -1,3 +1,7 @@ +-----BEGIN PGP SIGNED MESSAGE----- +Hash: SHA256 + +test -----BEGIN PGP SIGNATURE----- iHUEARYIAB0WIQR0vATEPYYIS+hnLAZ3LRYeNc1LgQUCZiu5YwAKCRB3LRYeNc1L $ diff -u 1.sig 1.signed --- 1.sig 2024-04-26 10:25:39.226424538 -0400 +++ 1.signed 2024-04-26 10:25:39.234424907 -0400 @@ -1,7 +1,11 @@ +-----BEGIN PGP SIGNED MESSAGE----- +Hash: SHA256 + +test -----BEGIN PGP SIGNATURE----- iHUEARYIAB0WIQR0vATEPYYIS+hnLAZ3LRYeNc1LgQUCZiu5YwAKCRB3LRYeNc1L -gZJPAP9TioWz+1YpILMV6EDwGT5aiyWKU96IZdOedgkCvWU8oAEAvfUs8d44otco -OJ2QKD0jxQoDCzZ3fc1Vr4rLTvGN7QA= -=d3IB +gT7ZAQCerr6oFMzanfgLEfH5phZ6Rpxb7e6GRi2XZTCGfsKItQEA9BTnJ4Jzjq00 +URFjZryC0V6SR0YFFWaBDN6mI/yTHQY= +=cpUs -----END PGP SIGNATURE----- $ diff -u 0.signed 1.signed $ diff -u 0.sig 1.signed --- 0.sig 2024-04-26 10:25:39.238425091 -0400 +++ 1.signed 2024-04-26 10:25:39.234424907 -0400 @@ -1,3 +1,7 @@ +-----BEGIN PGP SIGNED MESSAGE----- +Hash: SHA256 + +test -----BEGIN PGP SIGNATURE----- iHUEARYIAB0WIQR0vATEPYYIS+hnLAZ3LRYeNc1LgQUCZiu5YwAKCRB3LRYeNc1L $ diff -u 1.sig 2.signed --- 1.sig 2024-04-26 10:25:39.226424538 -0400 +++ 2.signed 2024-04-26 10:25:39.222424353 -0400 @@ -1,3 +1,8 @@ +-----BEGIN PGP SIGNED MESSAGE----- +Hash: SHA256 + +test + -----BEGIN PGP SIGNATURE----- iHUEARYIAB0WIQR0vATEPYYIS+hnLAZ3LRYeNc1LgQUCZiu5YwAKCRB3LRYeNc1L $ gpg --version gpg (GnuPG) 2.2.40 libgcrypt 1.10.3 Copyright (C) 2022 g10 Code GmbH License GNU GPL-3.0-or-later <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Home: /home/dkg/.gnupg Supported algorithms: Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH, CAMELLIA128, CAMELLIA192, CAMELLIA256 Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224 Compression: Uncompressed, ZIP, ZLIB, BZIP2 $ ``` As you can see from above, clearsigning either of `test\n` and `test` results in the same signature, which is the same signature that is made via a `--textmode --detach-sign` over `test` (with no trailing newline). Each `--clearsign`ed message's signature corresponds to the signature made a `--textmode --detach-sign` of a message with one fewer trailing newline. I suspect this has never been noticed because `gpg --verify`'s `--output` option always emits a trailing newline, adding it if the signed data did *not* have a trailing newline: ``` $ for x in 0 1 2; do gpg --verify --output $x.output < $x.signed ; done gpg: Signature made Fri 26 Apr 2024 10:25:39 AM EDT gpg: using EDDSA key 74BC04C43D86084BE8672C06772D161E35CD4B81 gpg: Good signature from "Daniel Kahn Gillmor" [ultimate] gpg: Signature made Fri 26 Apr 2024 10:25:39 AM EDT gpg: using EDDSA key 74BC04C43D86084BE8672C06772D161E35CD4B81 gpg: Good signature from "Daniel Kahn Gillmor" [ultimate] gpg: Signature made Fri 26 Apr 2024 10:25:39 AM EDT gpg: using EDDSA key 74BC04C43D86084BE8672C06772D161E35CD4B81 gpg: Good signature from "Daniel Kahn Gillmor" [ultimate] $ for x in 0 1 2; do hd < $x.output; done 00000000 74 65 73 74 0a |test.| 00000005 00000000 74 65 73 74 0a |test.| 00000005 00000000 74 65 73 74 0a 0a |test..| 00000006 $ ``` If i'm understanding RFC 4880 correctly, `gpg --clearsign` could become compliant by either: - not stripping the trailing newline before signing, or - not stripping the trailing newline before signing, and *adding* a trailing newline if none exists `gpg --verify`'s `--output` option should stop emitting a trailing newline when none exists, so that it accurately represents the signed data.

`gpg --clearsign` does not sign the actual input message if the message ends in a newline. Rather, it signs the message without the final trailing newline. `gpg --verify`'s `--output` option does not emit the actual signed data, if the data does not end in a newline. Rather, it emits the signed data plus a trailing newline. Clearsigned messages and whitespace have quite a bit of legacy trickiness, as noted in T1419 and T1692. In particular, RFC 4880 observes that whitespace at the end of newlines will be stripped, and that: > An implementation SHOULD add a line break after the cleartext, but > MAY omit it if the cleartext ends with a line break. This is for > visual clarity. Presumably this suggestion is meant to add a trailing line break to the message before signing, as it will lend more visual clarity to the final clearsigned message. And, of course, whitespace at the end of a line in a clearsigned message is all stripped: > Also, any trailing whitespace -- spaces (0x20) and tabs (0x09) -- at > the end of any line is removed when the cleartext signature is > generated. The spec is also clear that the final newline before the `BEGIN PGP SIGNATURE` is *not* part of the message being signed: > The line ending (i.e., the <CR><LF>) before the '-----BEGIN PGP > SIGNATURE-----' line that terminates the signed text is not > considered part of the signed text. However, `gpg --clearsign` seems to *strip* the trailing newline from input before signing. By contrast, `gpg --textmode --detach-sign` does not do that. Stripping the trailing newline is not the same as trimming the whitespace at the end of a line, or adding a line break if no such linebreak exists. Instead, the message signed with `gpg --clearsign` is actually the input message with one *fewer* newline, as long as the original message ended in a newline. Here is a demonstration of this (mis)behavior, where six deterministic signatures are made in the same second, and then compared to each other. We sign each of three messages (with 0, 1, or 2 trailing newlines) using both `--textmode --detach-sign` and `--clearsign`. Then we compare the results: ``` $ for newlines in 0 1 2; do python3 -c 'print("test" + "\n"*'$newlines', end="")' > $newlines.txt; done $ for nl in 0 1 2; do gpg -u $PGPID --clearsign < $nl.txt > $nl.signed & gpg -u $PGPID --detach-sign --textmode --armor < $nl.txt > $nl.sig & done; wait [1] 1165590 [2] 1165591 [3] 1165592 [4] 1165593 [5] 1165594 [6] 1165601 [1] Done gpg -u $PGPID --clearsign < $nl.txt > $nl.signed [2] Done gpg -u $PGPID --detach-sign --textmode --armor < $nl.txt > $nl.sig [3] Done gpg -u $PGPID --clearsign < $nl.txt > $nl.signed [4] Done gpg -u $PGPID --detach-sign --textmode --armor < $nl.txt > $nl.sig [5]- Done gpg -u $PGPID --clearsign < $nl.txt > $nl.signed [6]+ Done gpg -u $PGPID --detach-sign --textmode --armor < $nl.txt > $nl.sig $ diff -u 0.sig 0.signed --- 0.sig 2024-04-26 10:25:39.238425091 -0400 +++ 0.signed 2024-04-26 10:25:39.374431359 -0400 @@ -1,3 +1,7 @@ +-----BEGIN PGP SIGNED MESSAGE----- +Hash: SHA256 + +test -----BEGIN PGP SIGNATURE----- iHUEARYIAB0WIQR0vATEPYYIS+hnLAZ3LRYeNc1LgQUCZiu5YwAKCRB3LRYeNc1L $ diff -u 1.sig 1.signed --- 1.sig 2024-04-26 10:25:39.226424538 -0400 +++ 1.signed 2024-04-26 10:25:39.234424907 -0400 @@ -1,7 +1,11 @@ +-----BEGIN PGP SIGNED MESSAGE----- +Hash: SHA256 + +test -----BEGIN PGP SIGNATURE----- iHUEARYIAB0WIQR0vATEPYYIS+hnLAZ3LRYeNc1LgQUCZiu5YwAKCRB3LRYeNc1L -gZJPAP9TioWz+1YpILMV6EDwGT5aiyWKU96IZdOedgkCvWU8oAEAvfUs8d44otco -OJ2QKD0jxQoDCzZ3fc1Vr4rLTvGN7QA= -=d3IB +gT7ZAQCerr6oFMzanfgLEfH5phZ6Rpxb7e6GRi2XZTCGfsKItQEA9BTnJ4Jzjq00 +URFjZryC0V6SR0YFFWaBDN6mI/yTHQY= +=cpUs -----END PGP SIGNATURE----- $ diff -u 0.signed 1.signed $ diff -u 0.sig 1.signed --- 0.sig 2024-04-26 10:25:39.238425091 -0400 +++ 1.signed 2024-04-26 10:25:39.234424907 -0400 @@ -1,3 +1,7 @@ +-----BEGIN PGP SIGNED MESSAGE----- +Hash: SHA256 + +test -----BEGIN PGP SIGNATURE----- iHUEARYIAB0WIQR0vATEPYYIS+hnLAZ3LRYeNc1LgQUCZiu5YwAKCRB3LRYeNc1L $ diff -u 1.sig 2.signed --- 1.sig 2024-04-26 10:25:39.226424538 -0400 +++ 2.signed 2024-04-26 10:25:39.222424353 -0400 @@ -1,3 +1,8 @@ +-----BEGIN PGP SIGNED MESSAGE----- +Hash: SHA256 + +test + -----BEGIN PGP SIGNATURE----- iHUEARYIAB0WIQR0vATEPYYIS+hnLAZ3LRYeNc1LgQUCZiu5YwAKCRB3LRYeNc1L $ gpg --version gpg (GnuPG) 2.2.40 libgcrypt 1.10.3 Copyright (C) 2022 g10 Code GmbH License GNU GPL-3.0-or-later <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Home: /home/dkg/.gnupg Supported algorithms: Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH, CAMELLIA128, CAMELLIA192, CAMELLIA256 Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224 Compression: Uncompressed, ZIP, ZLIB, BZIP2 $ ``` As you can see from above, clearsigning either of `test\n` and `test` results in the same signature, which is the same signature that is made via a `--textmode --detach-sign` over `test` (with no trailing newline). Each `--clearsign`ed message's signature corresponds to the signature made a `--textmode --detach-sign` of a message with one fewer trailing newline. I suspect this has never been noticed because `gpg --verify`'s `--output` option always emits a trailing newline, adding it if the signed data did *not* have a trailing newline: ``` $ for x in 0 1 2; do gpg --verify --output $x.output < $x.signed ; done gpg: Signature made Fri 26 Apr 2024 10:25:39 AM EDT gpg: using EDDSA key 74BC04C43D86084BE8672C06772D161E35CD4B81 gpg: Good signature from "Daniel Kahn Gillmor" [ultimate] gpg: Signature made Fri 26 Apr 2024 10:25:39 AM EDT gpg: using EDDSA key 74BC04C43D86084BE8672C06772D161E35CD4B81 gpg: Good signature from "Daniel Kahn Gillmor" [ultimate] gpg: Signature made Fri 26 Apr 2024 10:25:39 AM EDT gpg: using EDDSA key 74BC04C43D86084BE8672C06772D161E35CD4B81 gpg: Good signature from "Daniel Kahn Gillmor" [ultimate] $ for x in 0 1 2; do hd < $x.output; done 00000000 74 65 73 74 0a |test.| 00000005 00000000 74 65 73 74 0a |test.| 00000005 00000000 74 65 73 74 0a 0a |test..| 00000006 $ ``` If i'm understanding RFC 4880 correctly, `gpg --clearsign` could become compliant by either: - not stripping the trailing newline before signing, or - not stripping the trailing newline before signing, and *adding* a trailing newline if none exists `gpg --verify`'s `--output` option should stop emitting a trailing newline when none exists, so that it accurately represents the signed data.