This requires re-evaluation of Libgcrypt to match the current FIPS specs.

If I understand correctly, to conform FIPS, we need to ensure Key/IV pair uniqueness (See "Implementation Guidance for FIPS 140-3", Annex C. "C.H Key/IV Pair Uniqueness Requirements from SP 800-38D").
Use of the API to set IV by any value may be considered bad.

Right. The AES-GCM was not allowed in FIPS mode until recently and I think now it is acceptable only for certain protocols (TLS, SSH), which guarantee that the IV is handled "correctly". As mentioned by gniibe, the requirements is that one should not be able to set IV to any specific value. The IV should be incremented automatically inside of the library (with some mask length + some generator configuration), somehow similarly as it is done with openssl, which would probably requite a new API in libgcrypt.

@pmgdeb Do you have some particular use case for AES-GCM in FIPS mode?

FWIW: Unfortunately everyone is moving to GCM, even Outlook. While GnuPG was evaluated by the German BSI we had discussions about this and their evaluators were wary about GCM due to its brittleness thus our use of OCB was very welcomed. OTOH, another approved product meanwhile comes with GCM for S/MIME and thus it seems thatGCM is accepted.

gniibe: Can you please check what openssl does exactly. The problem is that we currently have no permanent state for Libgcrypt (i.e. something stored on disk per user or even better global)

I have couple of references from libssh:

With OpenSSL, the IV is set to fixed value at first:

https://gitlab.com/libssh/libssh-mirror/-/blob/master/src/libcrypto.c#L544

And then it is incremented with another operation for each packet:

https://gitlab.com/libssh/libssh-mirror/-/blob/master/src/libcrypto.c#L684

The libgcrypt version of the same looks like this:

https://gitlab.com/libssh/libssh-mirror/-/blob/master/src/libgcrypt.c#L444

yeah .. there is no context that could hold this IV accross different packets at this moment, but my understanding of the FIPS is that it requires this context.

(OpenSSL for FIPS support is a bit tricky, which is described in README-FIPS.md in their distribution. It offers OpenSSL FIPS provider as shared library fips.so.)

In OpenSSL, for lower level API, there is GCYPTO_gcm_setiv.
And for higher level API (EVP), it seems that it is correctly handled about IV constraint for FIPS conformance.

Specifically, in openssl/crypto/evp/e_aes.c, it has following lines:

#ifdef FIPS_MODULE
/*
 * See SP800-38D (GCM) Section 8 "Uniqueness requirement on IVS and keys"

and

#ifdef FIPS_MODULE
    /*
     * FIPS requires generation of AES-GCM IV's inside the FIPS module.
     * The IV can still be set externally (the security policy will state that
     * this is not FIPS compliant). There are some applications
     * where setting the IV externally is the only option available.
     */

This means that:
(1) With OpenSSL, using GCM with arbitrary IV is possible by lower level API, and it's up to an application (say, use case for SSH and TLS).
(2) For fips.so, GCM (with unique IV) is supported, and it (somehow) ensures the uniqueness of IV by internal generation.

In my opinion, it would be good for libgcrypt
(1) to extend the behavior:

• allow use of GCM with internally generated IV

(2) to extend/add an API:

• a flag to specify internal generation of IV
• or... a flag to specify random IV at first use and incremented by each call

(3) (possibly) to make sure that only internally generated IV is used in FIPS mode

Experimental branches:
https://dev.gnupg.org/source/libgcrypt/history/t4873/
https://dev.gnupg.org/source/ntbtls/history/t4873/

Thank you for your work on the proposal. I have two comments:

• Do we have some test vector, which can be used in the testsute to test the new API?
• We need to mention the new API in the documentation.

Otherwise it looks good. I did not get to test it with libssh implementation though.

I pushed the change with documentation.

Test vector is needed, yes.

I realized that some AEAD cipher (including GCM) allows arbitrary length for IV.
But it's not good for the API of setup_geniv and geniv.

For practical use cases (like TLS/SSH), I think that limiting length to BLOCK_LEN makes sense.