During testing of RNP's OpenSSL backend got to the case when GnuPG randomly generated invalid RSA signatures.
Further debugging showed that P and Q didn't meet the rule P < Q (yielding invalid U).
RNP worked with such keys, and my guess is libgcrypt utilizes U in calculations, while OpenSSL doesn't.
While it's okay to behave incorrectly on an invalid secret key, confusing things here are:
- import of invalid secret key without any warning
- randomly erroring signing: it may happen on 2nd sign operation or succeed for 100 calls.
Sample secret key and some helper scripts attached.