Here is the use case :
Given an identity "some-key" with a RSA master key of keyid 0x0xBC3979CD0985291D and size 2048 bits
And its encryption subkey of keyid 0x0xBD31C81AB1654B76
When i proceed to GPG encryption of the file "testdgr.txt" with recipient "some-key"
Then i sometimes get pubkey encrypted session key packet having an MPI indicating a 2040 bits data size packet
And the effective RSA encrypted session key packet size is 255 bytes length in the GPG file
Looking at the RFC 4880, just before RSA encryption of the session key, a PKCS#1 V1.5 padding is applied , which should produce a result of length k (k being the RSA key modulus size here).
Following the use case above, when i proceed to file GPG decryption, i observe that the encrypted session key packet submitted for RSA decryption is effectively of 255 bytes length.
This decryption operation always fails when the RSA keys are protected through a Hardware Security Module, and decryption is delegated to this HSM through PKCS#11 API.
Actually, it fails because the submitted packet length is lower thant the RSA key modulus size.
A full sampleis attached with :
- testdgr.txt : plain file,
- testdgr.2040bits.gpg : GPG encrypted file,
- list-packets.res : result of "gpg --list-packets" command,
- testdgr.2040bits.gpg.analysis.txt : manual analysis of the public key encrypted session key packet,
- gen_2040bitlen_pubkeyencpacket.sh : a bash script to generate a 2040 bits data packet,
- .gnupg : GPG home files used fo this use case (pubring, secring, ...)
Reproduced with GPG 1.4.12, 2.3.0