It seems that the internal API is not enough.
Now, we have _gcry_md_hash_buffer function with the new FIPS service indicator.
It's used for public key crypto, too.
The compliance for hash function is a part of public key crypto, but not all.

A change for gcry_md_hash_* functions are pushed by rC3478caac62c7: fips,md: Implement new FIPS service indicator for gcry_md_hash_*..
It doesn't have tests with FIPS service indicator yet.

New external API is by GCRYCTL_FIPS_SERVICE_INDICATOR and/or the new macro gcry_get_fips_service_indicator.
This change is pushed by rCf51f4e98930e: fips: Introduce GCRYCTL_FIPS_SERVICE_INDICATOR and the macro.

New internal API is introduced with T7340 by the commit rCe1cf31232825: fips: Introduce an internal API for FIPS service indicator.

Change is pushed by rCe1cf31232825: fips: Introduce an internal API for FIPS service indicator.

I created a branch: https://dev.gnupg.org/source/libgcrypt/history/gniibe%252Ft7340/

Autoconf archive has AX_TLS: https://www.gnu.org/software/autoconf-archive/ax_tls.html
Also, AX_GCC_VAR_ATTRIBUTE(tls_model) could be used: https://www.gnu.org/software/autoconf-archive/ax_gcc_var_attribute.html

I think so. We did not submit a modules for recertification with these changes, but we do not plan this in close future so you can consider it completed.

Can we close this?

Applied the change. I write the ChangeLog entry by commit message.

The fix is in 1.10.3.

Fix is in 1.10.3.

Add the check of digest algorithm for EdDSA in: rCd15fe6aac10b: cipher:ecc:fips: Only allow defined digest algo for EdDSA.

No, there are use cases in GnuPG, where we specify the hash algo for signing, and our own tests/benchmark.c.

For the first issue, I added a check in: rCf65c30d470f5: cipher:ecc:fips: Reject use of SHAKE when it's ECDSA with RFC6979.

From the FIPS 186-5 there are some limitations to use the SHAKE in FIPS Mode that we will have to reflect:

Pushed a change in master.

I found the case of X.509, which also uses fixed length output for RSA-PSS and ECDSA: https://www.rfc-editor.org/rfc/rfc8692.html

Thank you for having a look into that! The proposed patch looks good. Should we have this change also in master?

Here is a possible change (... to master, assuming it's good to support use case of RFC 8702):

diff --git a/cipher/keccak.c b/cipher/keccak.c
index 22c40302..76e08cb5 100644
--- a/cipher/keccak.c
+++ b/cipher/keccak.c
@@ -1630,8 +1630,8 @@ const gcry_md_spec_t _gcry_digest_spec_sha3_512 =
 const gcry_md_spec_t _gcry_digest_spec_shake128 =
   {
     GCRY_MD_SHAKE128, {0, 1},
-    "SHAKE128", shake128_asn, DIM (shake128_asn), oid_spec_shake128, 0,
-    shake128_init, keccak_write, keccak_final, NULL, keccak_extract,
+    "SHAKE128", shake128_asn, DIM (shake128_asn), oid_spec_shake128, 32,
+    shake128_init, keccak_write, keccak_final, keccak_read, keccak_extract,
     _gcry_shake128_hash_buffers,
     sizeof (KECCAK_CONTEXT),
     run_selftests
@@ -1639,8 +1639,8 @@ const gcry_md_spec_t _gcry_digest_spec_shake128 =
 const gcry_md_spec_t _gcry_digest_spec_shake256 =
   {
     GCRY_MD_SHAKE256, {0, 1},
-    "SHAKE256", shake256_asn, DIM (shake256_asn), oid_spec_shake256, 0,
-    shake256_init, keccak_write, keccak_final, NULL, keccak_extract,
+    "SHAKE256", shake256_asn, DIM (shake256_asn), oid_spec_shake256, 64,
+    shake256_init, keccak_write, keccak_final, keccak_read, keccak_extract,
     _gcry_shake256_hash_buffers,
     sizeof (KECCAK_CONTEXT),
     run_selftests

Reading RFC 8702, I realized that it defines the hash size in the use of CMS as: SHAKE128 : 32-byte SHAKE256 : 64-byte.

Applied rC8cdd0d353e19: cipher:pubkey: Check digest size which should not be zero. for 1.10.

I found this use case: RFC 8702
"Use of the SHAKE One-Way Hash Functions in the Cryptographic Message Syntax (CMS)": https://www.rfc-editor.org/rfc/rfc8702.html

Another possibility for digest&sign API: it is possible to determine the length of required hash function by the underlining field Fp of the curve in use. Then, use this length instead. It's better than to (try to) get the length by _gcry_md_get_algo_dlen (for SHAKE, it's undefined).