Reported on gcrypt-devel mailinglist by Bhabani Sankar Das:
libgcrypt's sntrup761 decapsulation returns a different rejection key
from the specification since a constant-time refactor in 2025. I found
no attack (reasoning below), so I am reporting it here as a normal bug
with a one-line fix.Summary
On a rejected ciphertext, sntrup761_dec() returns
HashSession(0, Encode(r') XOR rho, C) instead of the specification's
HashSession(0, rho, C) (NTRU Prime round 3, Decap). The accept path is
unaffected.
- File: cipher/sntrup761.c, sntrup761_dec(), lines 1038-1041 (master 53742c7e, 1.12.4)
- Cause: commit 4e963e91 (2025-09-17), "sntrup761: use const-time helpers for memory comparison and cond move", and its 1.11 branch cherry-pick 7ff58b28; posted as [PATCH 06/10], https://lists.gnupg.org/pipermail/gcrypt-devel/2025-September/005961.html
- Affected: 1.11.3, 1.12.0 to 1.12.4, master
- Correct: 1.11.0 to 1.11.2
How to reproduce
The program below (sntrup761_reject_key.c) uses only the public
gcry_kem_* API.
It decapsulates 8 rejected ciphertexts (one bit flipped in the
confirmation hash) and compares each key with the specification
formula computed from rho in the secret key:cc sntrup761_reject_key.c -lgcrypt -o sntrup761_reject_key ./sntrup761_reject_key 1.11.2: rejected ciphertexts matching the specification: 8/8 -> MATCH 1.12.4: rejected ciphertexts matching the specification: 0/8 -> DIFFIndependently, liboqs decapsulating the same secret key and ciphertext
bytes gives the same accept key but a different reject key (0/8), while
OpenSSH, leancrypto and liboqs all match the specification (8/8).Root cause
The original SUPERCOP line was
for (i = 0; i < Inputs_bytes; ++i) r_enc[i] ^= mask & (r_enc[i] ^ rho[i]);
which on reject sets r_enc = rho. Commit 4e963e91 replaced it with
for (i = 0; i < Inputs_bytes; ++i) tmp[i] = r_enc[i] ^ rho[i]; _gcry_ct_memmov_cond (r_enc, tmp, Inputs_bytes, mask & 1);
which on reject sets r_enc = r_enc ^ rho: the second XOR is lost.
Neither tests/t-kem.c nor tests/t-sntrup761.c checks a rejection key,
so the change was not caught.Impact
I found no attack. The full input ciphertext C is still hashed, so
distinct rejected ciphertexts give distinct keys, and the key stays
unpredictable without rho. The effect is interoperability: libgcrypt's
rejection keys differ from every other implementation, and any
reject-path test vector fails. libssh with the gcrypt backend uses this
code for sntrup761x25519-sha512 (src/sntrup761.c).Suggested fix
Move rho itself, as the original code did:
- for (i = 0; i < Inputs_bytes; ++i) - tmp[i] = r_enc[i] ^ rho[i]; - _gcry_ct_memmov_cond (r_enc, tmp, Inputs_bytes, mask & 1); + _gcry_ct_memmov_cond (r_enc, rho, Inputs_bytes, mask & 1);
(and drop the now unused tmp and i). With this change, 1.12.4 matches
the specification 8/8 and tests/t-kem and t-sntrup761 still pass.AI tools assisted the analysis; every result above was reproduced and
checked by hand.If you fix it, please credit Bhabani Sankar Das
(https://github.com/007bsd).
-------------------------- sntrup761_reject_key.c --------------------------
/* sntrup761 implicit-rejection key check for libgcrypt (public gcry_kem_* API).
*
* Decapsulates a rejected ciphertext (one bit flipped in an honest one) and
* compares the result with the NTRU Prime round 3 rejection key,
* K = Hash_prefix(0, Hash_prefix(3, rho) || C),
* Hash_prefix(b, x) = SHA-512(b || x)[0..31],
* where rho is the 191-byte value stored in the secret key
* (sk = 2*Small_bytes || pk || rho || cache, as in SUPERCOP) and C is the
* full input ciphertext.
*
* Build: cc sntrup761_reject_key.c -lgcrypt -o sntrup761_reject_key
* Output "MATCH" = conformant, "DIFF" = rejection key differs from the spec.
*/
#include <gcrypt.h>
#include <stdio.h>
#include <string.h>
#define SMALL_BYTES 191
#define CACHE_BYTES 32
static void
hash_prefix (unsigned char out[32], unsigned char b,
const unsigned char *x, size_t xlen)
{
unsigned char full[64];
gcry_md_hd_t h;
gcry_md_open (&h, GCRY_MD_SHA512, 0);
gcry_md_putc (h, b);
gcry_md_write (h, x, xlen);
memcpy (full, gcry_md_read (h, GCRY_MD_SHA512), 64);
gcry_md_close (h);
memcpy (out, full, 32);
}
int
main (void)
{
static unsigned char pk[GCRY_KEM_SNTRUP761_PUBKEY_LEN];
static unsigned char sk[GCRY_KEM_SNTRUP761_SECKEY_LEN];
static unsigned char ct[GCRY_KEM_SNTRUP761_CIPHER_LEN];
static unsigned char buf[32 + GCRY_KEM_SNTRUP761_CIPHER_LEN];
unsigned char ss[32], k_rej[32], inner[32], spec[32];
const unsigned char *rho;
int bit, ok = 0, n = 8;
if (!gcry_check_version (NULL))
return 2;
gcry_control (GCRYCTL_INITIALIZATION_FINISHED, 0);
printf ("libgcrypt %s\n", gcry_check_version (NULL));
if (gcry_kem_keypair (GCRY_KEM_SNTRUP761, pk, sizeof pk, sk, sizeof sk)
|| gcry_kem_encap (GCRY_KEM_SNTRUP761, pk, sizeof pk, ct, sizeof ct,
ss, sizeof ss, NULL, 0))
return 2;
rho = sk + sizeof sk - CACHE_BYTES - SMALL_BYTES;
hash_prefix (inner, 3, rho, SMALL_BYTES);
for (bit = 0; bit < n; bit++)
{
/* Flip one bit in the trailing confirmation hash: rejected, and it
still decrypts to the same r'. */
ct[sizeof ct - 1 - bit / 8] ^= (unsigned char)(1 << (bit % 8));
if (gcry_kem_decap (GCRY_KEM_SNTRUP761, sk, sizeof sk, ct, sizeof ct,
k_rej, sizeof k_rej, NULL, 0))
return 2;
memcpy (buf, inner, 32);
memcpy (buf + 32, ct, sizeof ct);
hash_prefix (spec, 0, buf, sizeof buf);
ok += !memcmp (k_rej, spec, 32);
ct[sizeof ct - 1 - bit / 8] ^= (unsigned char)(1 << (bit % 8));
}
printf ("rejected ciphertexts matching the specification: %d/%d -> %s\n",
ok, n, ok == n ? "MATCH" : "DIFF");
return ok == n ? 0 : 1;
}