libgcrypt does not correctly pad RSASSA-PKCS1-v1_5 signatures that use SHA-3, because it does not have the OIDs for SHA-3 set.
To reproduce, run the attached reproducer with sha3-224, sha3-256, sha3-384, or sha3-512 as argument (the same problem occurs with shake128 or shake256, but the reproducer cannot show that).
- gcc -std=c99 -Wall -Werror -pedantic -o test test.c -lgcrypt
- ./test sha3-256 | sed 1d | sed -E 's/01(ff)*00//' | xxd -ps -r | openssl asn1parse -inform DER
Notes:
- sed 1d removes the first line of output, which is the signature in hex
- sed -E 's/01(ff)*00//' removes the PKCS#1 padding
- xxd -ps -r converts the hex output to binary
- openssl asn1parse -inform DER attempts to parse the resulting ASN.1
- As an alternative to openssl asn1parse, other asn1parsers such as der2ascii can also be used