Fix incorrect counter overflow handling for GCM
ffdc6f3623a0
Actions

Description

Fix incorrect counter overflow handling for GCM

* cipher/cipher-gcm.c (gcm_ctr_encrypt): New function to handle
32-bit CTR increment for GCM.
(_gcry_cipher_gcm_encrypt, _gcry_cipher_gcm_decrypt): Do not use
generic CTR implementation directly, use gcm_ctr_encrypt instead.
* tests/basic.c (_check_gcm_cipher): Add test-vectors for 32-bit
CTR overflow.
(check_gcm_cipher): Add 'split input to 15 bytes and 17 bytes'
test-runs.

I believe we have found what seems to be a bug in counter overflow
handling in AES-GCM in libgcrypt's implementation. This leads to
incorrect results when using a non-12-byte IV and decrypting payloads
encrypted with other AES-GCM implementations, such as OpenSSL.

According to the NIST Special Publication 800-38D "Recommendation for
Block Cipher Modes of Operation: Galois/Counter Mode (GCM) and GMAC",
section 7.1, algorithm 4, step 3 [NIST38D], the counter increment is
defined as inc_32. Section 6.2 of the same document defines the
incrementing function inc_s for positive integers s as follows:

the function increments the right-most s bits of the string, regarded

as the binary representation of an integer, modulo 2^s; the remaining,

left-most len(X) - s bits remain unchanged

(X is the complete counter value in this case)

This problem does not occur when using a 12-byte IV, because AES-GCM has
a special case for the inital counter value with 12-byte IVs:

If len(IV)=96, then J_0 = IV 0^31 1

i.e., one would have to encrypt (UINT_MAX - 1) * blocksize of data to
hit an overflow. However, for non-12-byte IVs, the initial counter value
is the output of a hash function, which makes hitting an overflow much
more likely.

In practice, we have found that using

iv = 9e 79 18 8c ff 09 56 1e c9 90 99 cc 6d 5d f6 d3
key = 26 56 e5 73 76 03 c6 95 0d 22 07 31 5d 32 5c 6b a5 54 5f 40 23 98 60 f6 f7 06 6f 7a 4f c2 ca 40

will reliably trigger an overflow when encrypting 10 MiB of data. It
seems that this is caused by re-using the AES-CTR implementation for
incrementing the counter.