In the first case, gpg emits a CANCELED_BY_USER status. This makes gpgme abort the operation. We may have to wait/watch for BEGIN_DECRYPTION / END_DECRYPTION.

Status output of gpg:

[GNUPG:] ENC_TO 547ECA3FEAB73FC2 18 0
[GNUPG:] NEED_PASSPHRASE_SYM 9 3 2
[GNUPG:] WARNING server_version_mismatch 0 server 'gpg-agent' is older than us (2.4.5 < 2.4.6-beta102)
[GNUPG:] PINENTRY_LAUNCHED 455567 qt 1.3.1 /dev/pts/16 xterm-256color :0 20600/1000/5 1000/1000 0

Cancel pinentry

[GNUPG:] CANCELED_BY_USER
[GNUPG:] KEY_CONSIDERED 3A8536D46F57779C49F0CF542C0444CB59852D29 0
[GNUPG:] KEY_CONSIDERED 5F4C389C9265ABDE08B21CEA41FAC215ADE1CB46 0
[GNUPG:] KEY_CONSIDERED CC2750526E7D9CA70F599CD3F612794A4EB653DD 0
[GNUPG:] KEY_CONSIDERED 8C8AFD6AEF9F30C4F6D80CF6089F4ED80F384779 0
[GNUPG:] KEYEXPIRED 1667038797
[GNUPG:] KEY_CONSIDERED 8DF42AB4B60250D675F3416B02BC64C000116E2C 0
[GNUPG:] KEYEXPIRED 1667038967
[GNUPG:] KEY_CONSIDERED E30272C0D092381922B914A7A5B048367DBFF9B2 0
[GNUPG:] KEYEXPIRED 1671200444
[GNUPG:] KEY_CONSIDERED CEEF61A3A792C6CAA40914B36E10BD5CA0946C1B 0
[GNUPG:] KEYEXPIRED 1671620400
[GNUPG:] KEY_CONSIDERED 377EAE5E59B976010A7D2EDC0E41874D72E1A302 0
[GNUPG:] KEY_CONSIDERED 3A8536D46F57779C49F0CF542C0444CB59852D29 0
[GNUPG:] PINENTRY_LAUNCHED 455594 qt 1.3.1 /dev/pts/16 xterm-256color :0 20600/1000/5 1000/1000 0
[GNUPG:] KEY_CONSIDERED 3A8536D46F57779C49F0CF542C0444CB59852D29 0
[GNUPG:] DECRYPTION_KEY 4DE4678D1CB12ECD4E9D8B6E547ECA3FEAB73FC2 3A8536D46F57779C49F0CF542C0444CB59852D29 u
[GNUPG:] BEGIN_DECRYPTION
[GNUPG:] DECRYPTION_INFO 0 9 2
[GNUPG:] PLAINTEXT 62 1630576688 
[GNUPG:] PLAINTEXT_LENGTH 31
[GNUPG:] DECRYPTION_OKAY
[GNUPG:] GOODMDC
[GNUPG:] END_DECRYPTION

In the second case, gpg emits a FAILURE gpg-exit 33554433 status at the end. I think this makes gpgme consider the operation failed. I think this is a bug in gpg because gpg does not emit a FAILURE status if a wrong symmetric passphrase is entered.

Status output of gpg:

[GNUPG:] ENC_TO 547ECA3FEAB73FC2 18 0
[GNUPG:] NEED_PASSPHRASE_SYM 9 3 2
[GNUPG:] WARNING server_version_mismatch 0 server 'gpg-agent' is older than us (2.4.5 < 2.4.6-beta102)
[GNUPG:] PINENTRY_LAUNCHED 455875 qt 1.3.1 /dev/pts/16 xterm-256color :0 20600/1000/5 1000/1000 0

Enter empty passphrase

[GNUPG:] MISSING_PASSPHRASE
gpg: gcry_kdf_derive failed: Invalid data[GNUPG:] MISSING_PASSPHRASE

^interesting log output

[GNUPG:] KEY_CONSIDERED 3A8536D46F57779C49F0CF542C0444CB59852D29 0
[GNUPG:] KEY_CONSIDERED 5F4C389C9265ABDE08B21CEA41FAC215ADE1CB46 0
[GNUPG:] KEY_CONSIDERED CC2750526E7D9CA70F599CD3F612794A4EB653DD 0
[GNUPG:] KEY_CONSIDERED 8C8AFD6AEF9F30C4F6D80CF6089F4ED80F384779 0
[GNUPG:] KEYEXPIRED 1667038797
[GNUPG:] KEY_CONSIDERED 8DF42AB4B60250D675F3416B02BC64C000116E2C 0
[GNUPG:] KEYEXPIRED 1667038967
[GNUPG:] KEY_CONSIDERED E30272C0D092381922B914A7A5B048367DBFF9B2 0
[GNUPG:] KEYEXPIRED 1671200444
[GNUPG:] KEY_CONSIDERED CEEF61A3A792C6CAA40914B36E10BD5CA0946C1B 0
[GNUPG:] KEYEXPIRED 1671620400
[GNUPG:] KEY_CONSIDERED 377EAE5E59B976010A7D2EDC0E41874D72E1A302 0
[GNUPG:] KEY_CONSIDERED 3A8536D46F57779C49F0CF542C0444CB59852D29 0
[GNUPG:] PINENTRY_LAUNCHED 455901 qt 1.3.1 /dev/pts/16 xterm-256color :0 20600/1000/5 1000/1000 0
[GNUPG:] KEY_CONSIDERED 3A8536D46F57779C49F0CF542C0444CB59852D29 0
[GNUPG:] DECRYPTION_KEY 4DE4678D1CB12ECD4E9D8B6E547ECA3FEAB73FC2 3A8536D46F57779C49F0CF542C0444CB59852D29 u
[GNUPG:] BEGIN_DECRYPTION
[GNUPG:] DECRYPTION_INFO 0 9 2
[GNUPG:] PLAINTEXT 62 1630576688 
[GNUPG:] PLAINTEXT_LENGTH 31
[GNUPG:] DECRYPTION_OKAY
[GNUPG:] GOODMDC
[GNUPG:] END_DECRYPTION
[GNUPG:] FAILURE gpg-exit 33554433

For the second case, I think that gcry_kdf_defive should not be called with pw="". The result of FAILURE gpg-exit 33554433 comes from the log_error after failure of gcry_kdf_derive.

For gpg, something like following is needed:

diff --git a/g10/passphrase.c b/g10/passphrase.c
index c5ec8eae4..82160e3b0 100644
--- a/g10/passphrase.c
+++ b/g10/passphrase.c
@@ -384,8 +384,16 @@ passphrase_to_dek (int cipher_algo, STRING2KEY *s2k,
      get_last_passphrase(). */
   dek = xmalloc_secure_clear ( sizeof *dek );
   dek->algo = cipher_algo;
-  if ( (!pw || !*pw) && create)
-    dek->keylen = 0;
+  if (!pw || !*pw)
+    {
+      if (create)
+        dek->keylen = 0;
+      else
+        {
+          xfree (dek);
+          return NULL;
+        }
+    }
   else
     {
       gpg_error_t err;

When checking this out with gpg4win-Beta-64 I can reproduce case 1 (and of course 3) but not case 2:

After returning the empty symmetric password and giving the correct password for the private key the decryption is successful.

This would leave only case 1 as open bug.

I can still reproduce case 2 with gnupg 2.4. I have to check how my local setup differs from gpg4win-Beta-64.

gnupg 2.2 behaves completely different. gnupg 2.2 first asks for the password of the OpenPGP key. If I enter an empty password and then enter the correct symmetric password then decryption is successful in Kleopatra.

Gpg4win-5.0.1 still shows case 1. (just reproduced.)

It got reported again: https://forum.gnupg.org/t/gpg4win-decryption-of-file-encrypted-with-both-symmetric-and-key-only-offers-decryption-with-symmetric-passphrase-in-contrast-when-using-gpg-line-command-im-prompted-for-both-possibilities/7143/2

I'd propose applying the patch of mine above to gpg, and letting us suggest users to input empty pass phrase to skip (instead of cancelling).
This could be a minimum change (only gpg). Or else, gpgme needs to be changed to ignore CANCEL status and to handle complex cases; I think that it's not easy change.

The semantics of cancel button with pinentry is ambiguous in this particular scenario if it means cancelling entire process or only for specific method of encryption or key.

The scenario could be more complicated when multiple keys are specified for public key encryption or --hidden-recipient is used.

Note that decryption process by gpg implementations behave differently (among versions) for the packets with PKT_PUBKEY_ENC, PKT_SYMKEY_ENC, and PKT_ENCRYPTED_*.

I was too optimistic. GPGME is required the following change, too:

diff --git a/src/passphrase.c b/src/passphrase.c
index 140cd03a..d07afa91 100644
--- a/src/passphrase.c
+++ b/src/passphrase.c
@@ -114,6 +114,11 @@ _gpgme_passphrase_status_handler (void *priv, gpgme_status_code_t code,
     case GPGME_STATUS_CANCELED_BY_USER:
       return gpg_error (GPG_ERR_CANCELED);
 
+    case GPGME_STATUS_PINENTRY_LAUNCHED:
+      /* Another pinentry is invoked, reset the passphrase status.  */
+      opd->no_passphrase = opd->bad_passphrase = 0;
+      break;
+
     case GPGME_STATUS_EOF:
       if (opd->no_passphrase || opd->bad_passphrase)
 	return gpg_error (GPG_ERR_BAD_PASSPHRASE);

It is not (easily) possible to check for available keys first, before asking for a passphrase? (Like it is with gpg 2.2.)

@ebo, I try to answer your question. As I wrote, the behavior of gpg implementations are different (for a reason). I'm not sure if you suggest the change of behavior of gpg 2.4.

Let me explain.

Actually, gpg 2.4 does check available public keys (to select best one for a user). But when it is an encrypted file by both methods (symmetric and public key), it asks a user for passphrase to decrypt symmetric encrypted first.

gpg 2.2 does: when it sees PKT_PUBKEY_ENC it asks a user to try decrypting the session key. when it sees PKT_SYMKEY_ENC it asks a user to try decrypting the encrypted session key by passphrase. When one of tries successes, it use the result (the session key) to decrypt PKT_ENCRYPTED_* packet. When there are multiple PKT_PUBKEY_ENC and PKT_SYMKEY_ENC, gpg 2.2 handles sequentially.

gpg 2.4 and later does: when it sees PKT_PUBKEY_ENC it collects the packet (encrypted session key and public key info) into a list. when it sees PKT_SYMKEY_ENC it asks a user to try decrypting the encrypted session key by passphrase. When it sees PKT_ENCRYPTED_*, it used the session key (if any) to decrypt _or_ use the list of encrypted session key and public key info to get the session key. (In this way, most relevant public key can be selected.) gpg 2.4 does this way, because having multiple secret keys are common (among some community).

In case of PKT_SYMKEY_ENC comes after PKT_PUBKEY_ENC, with gpg 2.2, it asks a user for public key encryption first, then tries symmetric encryption.

With gpg 2.4, it asks a user for symmetric encryption firstly, then public key encryption (the order of packets doesn't matter).

I pushed my patch for gpg, since it does not break anything, just allow empty passphrase input (to skip).
I also pushed my patch for gpgme. I believe that it's correct.

Hi @gniibe,
thanks for making progress on the issue.

I pushed my patch for gpg, since it does not break anything, just allow empty passphrase input (to skip).

The users that reported the issue commented on your remark because it seem to address a different problem, just to let you know:
https://forum.gnupg.org/t/gpg4win-decryption-of-file-encrypted-with-both-symmetric-and-key-only-offers-decryption-with-symmetric-passphrase-in-contrast-when-using-gpg-line-command-im-prompted-for-both-possibilities/7143/5

Best Regards,
Bernhard

why gpg 2.4? Don't you mean 2.6? I'll add the proper 2.6 tag for avoiding confusion

I guess the behavior changed with gpg 2.4, i.e. "With gpg 2.4 (or later), ..."

@bernhard Thank you for the link.

There are multiple things observed/discussed. So far, I haven't changed the issue of cancel handling at all. Perhaps, we will open a new ticket for cancel handling. The semantics is not (yet) consistent (not well-defined) among pinenty-implementations/gpg/gpgme/Kleo.

As of now, I focus on how we can provide a consistent way to a user (both of gpg and of Kleo), when decrypting a file by two methods of symmetric + pubkey.

With the changes of mine, it is possible for a user (both of gpg and Kleo) to enter empty passphrase to skip symmetric method (which always requires non-empty passphrase). This way could be one of solutions to the particular problem.