The changelog in https://src.fedoraproject.org/rpms/libgcrypt/c/402a3b5f2eed746bea996c5743c99bec9bbc3487?branch=master

disable non-approved FIPS hashes in the enforced FIPS mode

But it had been disabled (that is, returning an error by GPG_ERR_DIGEST_ALGO) in the enforced FIPS mode... so, I wonder about what was the original intention.

Note that even after rCce1cbe16992a: Disable non-allowed algorithms in FIPS mode, gcry_md_open won't return an error with disabled algo.

I am not sure. MD5 is still important for some applications, say CRAM-MD5. IIRC, back in 2008 we dis-allowed RMD160 and added separate RMD160 code directly to gpg to fulfill FIPS requirements.

We plan this for 1.10 but it may also go into one of the next 1.9.x releases

The patch references the following bug:

https://bugzilla.redhat.com/show_bug.cgi?id=1659495

It claims MD4 was working in all the FIPS modes in RHEL8 (while not working in older RHEL versions). This was because of the condition

From my understanding, this patch is trying to make the behavior of FIPS disabled algorithms more consistent in FIPS enforced mode using the flags.fips of the md structure.

Applied by rC0f118c2dfb8e: cipher: Do not use of non-approved digests in FIPS mode.

I went a bit back to the history to figure out what is the enforced and soft fips mode as it was initially not completely clear to me. For the record, I used the following bug from 9 years ago:

https://bugzilla.redhat.com/show_bug.cgi?id=808520

The FIPS mode in libgcrypt was "soft" one by default, mostly because TLS at that time highly depended on MD5. But applications could use the GCRYCTL_SET_ENFORCED_FIPS_FLAG to enforce stricter checking, namely disable MD5.

The questions now are whether this differentiation is still needed (this was created more than 10 years back) or how to sensibly map current libgcrypt behavior to todays FIPS requirements.

I will add this to a list of questions to Stephan, but insights from @werner would be also helpful.

(can't access that bug with my account)

I'd say that MD5 and MD5+SHA1 are not anymore in use and thus this soft mode could be dropped. However, now we have the same situation with SHA-1. I am not sure how many FIPS relevant sites still require SHA-1. One use of SHA-1 is of course fingerprinting and this is even required in FIPS mode. However, with the new signing API this should not be a problem, right?

For use of SHA-1:

with the new signing API this should not be a problem, right?

It is true that: if all applications use new signing API with standard hash function, (such uses of) libgcrypt will be compliant.
It would be possible for an application to use libgcrypt with old API with non-standard hash function.

Perhaps, we should modify old API signing function to return an error when FIPS is enabled to guarantee no non-compliant use.

Right. The clarification is that SHA1 itself (for non-security and non-signature use) is still allowed in FIPS mode. But it is not allowed to be used as part of signature schemes of the new API in FIPS mode. The old API, which allows raw signatures without digests, should just fail in FIPS mode too. And the FIPS-compatible gnupg should use the new API too (it would be good to think about this when putting it together).

Getting back to the "soft"/"enforced" FIPS mode, after 0f118c2dfb8e1236893c30a9b86e7e231c8e5758, the only use of _gcry_inactivate_fips_mode() is in _gcry_set_allocation_handler(), which should not be probably allowed anymore in FIPS mode (or is it commonly used, do you know?). Also the _gcry_enforced_fips_mode() function does not look like having much uses anymore so as we are heading for the next major release, we should probably consider throwing this concepts away, deprecating/removing the API.

From Stephan I got the following response to the allocation handler use case

Now, the only other use is when we set custom allocation functions
(which should be probably completely prohibited in fips mode):

https://github.com/gpg/libgcrypt/blob/master/src/global.c#L869

This is of no concern, i.e. you can allow it provided that the zeroization is
not left to the caller-provided handlers.

Werner clarified that the free caller-provided functions are responsible for the zeroization as we do not know the size of freed memory to zeroize.

For MD5:

• soft FIPS mode: that mode violates your global service indicator idea - MD5

is non-approved and must be marked as such by the service indicator. Thus if
you want a global indicator, you must disable MD5 [in FIPS mode]

I had in my mind something like this:

Today, we have three modes of FIPS:

• "soft" fips mode, which is by default and is very permissive (allowing MD5, custom allocators allowing to disable secure memory ...)
• "forced" fips mode, which is something that is as closest to what we would like to have by default in FIPS mode
• "inactive" fips mode, which is basically drop-out from the "soft" fips mode

From my understanding, there is no ground for the above differentiation in FIPS 140-3 (not even in the older versions I saw). So I propose to drop these to simplify FIPS status.

Let me move this ticket as DONE (now Testing status), as the subject was solved (MD5 and soft/forced/inactive things).

I'll open another ticket to track hash and its restriction when FIPS enabled.

I created T5665: libgcrypt : Restrict message digest use for FIPS 140-3.

Please keep this ticket for MD5 things.
I'm going to change the title to express its content.