I changed that to a feature but I agree that the subkey selection mechanism
should take smartcards into account.

It would be surpising that suddendly a different subkey will be used for signing
if a smartcard is not available. Right, most users with several subkeys are
experts and know what they are going but nevertheless this is a change in behaviour.

D297: 785_sign-fix.patch

Attached a patch to call agent_probe_secret_key() during finish_lookup().

This partially solves the problem by not trying to use subkeys that have no
secret key present. This does not unexpectedly change the existing behaviour
because GnuPG will currently return an error if the automatically selected
secret key is not present.

It does not solve the issue of having multiple potential signing subkeys on
different smartcards, because these are always considered to be present (if the
subkey has been associated with a smartcard).

Updated patch to check that the requested key usage is SIG before checking for a
secret key.

D296: 793_sign-fix.patch

Any progress on this?

A reproducer, even without smartcards (please ensure that GNUPGHOME is
explicitly set):

ARGS="--pinentry-mode loopback --passphrase abc123"
ARGS="$ARGS --batch --with-colons --with-keygrip --status-fd 3"

gpg $ARGS --quick-gen test@example.org rsa cert 3>genkey.status
FPR=$(awk '/KEY_CREATED/{ print $4 }' < genkey.status)
gpg $ARGS --quick-add-key 0x$FPR rsa sign 3>addkey-1.status
sleep 5
gpg $ARGS --quick-add-key 0x$FPR rsa sign 3>addkey-2.status
GRIP=$(gpg $ARGS --list-keys | grep ^grp: | cut -f10 -d: | tail -n1)
mv $GNUPGHOME/private-keys-v1.d/$GRIP.,bak
gpg-connect-agent killagent /bye
echo test | gpg $ARGS -u "$FPR" --clear-sign

I've tested Simon's patch against 2.1.18, and i think it's the right thing. I
posted it to the mailing list in git-format-patch form here:

https://lists.gnupg.org/pipermail/gnupg-devel/2017-February/032547.html

The branch dkg/T1967 contains a fix for this. Please review!

Sorry, I just noticed this ticket now.

For smartcard, I push a different change to prefer available card key when --local-user is not specified in T1983: gpg2 prefers missing secret key to available key on card. This is at least related.

For the patch, I don't think it works if the intention is checking card key is available or not.
agent_probe_secret_key uses HAVEKEY command which only checks if the key is under control of gpg-agent or not, not about it's on-line or not.

Yes, I know it's not perfect but when the secret key is unknown to gpg-agent then it shouldn't attempt to use it.

This change was compatible with "it would be surpising that suddendly a different subkey will be used for signing" because it changes the behaviour from a guaranteed failure to a success when the key is actually present.

In T1983: gpg2 prefers missing secret key to available key on card, I applied another approach: rGfbb2259d22e6: g10: Fix default-key selection for signing, possibly by card.
Please test.

@gniibe: I've tested 2.1.22 (from Debian experimental) and, while gpg --sign works, other programs (eg: git tag -s) still prompt to insert the card of the first signing subkey, despite the card with the second signing subkey being present.
Is that expected?

I have a single primary key, with 2 sets of subkeys, each on its own device, and I'd like to seamlessy alternate between the two on the same laptop/desktop. Is this use case supposed to be covered by your patch?
I do not have a default key defined in gpg.conf (or gitconfig).

Thanks!

I also have to add that, if this really has been resolved, it only covers up the case if the missing subkey(s) is/are on the smartcard(s), it does not solve the problem when none of the missing signing subkeys are in smartcards (as in, all on different computers). And it's clear that for version 2.1.22, it fails to get the available subkey on the disk for this case.

• Using the SUBKEYFINGERPRINT! only works for some situations: only gpg and git are the ones I know that support that while it fails for Enigmail and Claws Mail (at least the last time I checked).

• Using default-key SUBKEYFINGERPRINT! on the gpg.conf doesn't work if you have multiple master keys unless you want the user to switch between gpg.conf each time.

• Deleting the public parts of the missing secret subkeys works but it means having to do so each time you use --refresh-keys.

The above three points worked back on GnuPG 1.4.x and GnuPG 2.0.x when the secret keyring existed and GnuPG could detect the existing secret subkeys, so all the workarounds explained above were not needed.

This test has been done on a testing secret key so, if there is any leaked secret, I don't mind it's compromised status. This can be reproduced by making a secret key with --quick-gen-key, then add two signing subkeys and delete/rename the .key file of the newest subkey:

PS C:\Users\MyUser> gpg -vvvvv --version
gpg (GnuPG) 2.1.22
libgcrypt 1.8.0
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: C:/Users/MyUser/AppData/Roaming/gnupg
Supported algorithms:
Pubkey: RSA (1), ELG (16), DSA (17), ECDH (18), ECDSA (19), EDDSA (22)
Cipher: IDEA (S1), 3DES (S2), CAST5 (S3), BLOWFISH (S4), AES (S7),
        AES192 (S8), AES256 (S9), TWOFISH (S10), CAMELLIA128 (S11),
        CAMELLIA192 (S12), CAMELLIA256 (S13)
Hash: SHA1 (H2), RIPEMD160 (H3), SHA256 (H8), SHA384 (H9), SHA512 (H10),
      SHA224 (H11)
Compression: Uncompressed (Z0), ZIP (Z1), ZLIB (Z2), BZIP2 (Z3)

PS C:\Users\MyUser> gpg --with-keygrip --list-secret-keys 40682A92196A5A7CB44FB85583AE0E9BEE18888E
sec   rsa2048 2017-08-07 [SC] [expires: 2019-08-07]
      40682A92196A5A7CB44FB85583AE0E9BEE18888E
      Keygrip = 1E7D58F8C5CF01811A62D56E12ED146DED569B2D
uid           [ultimate] TESTING
ssb   rsa2048 2017-08-07 [S]
      EBDA7C0209795BB9474F3772919CC4A474AA3AE7
      Keygrip = BC5A1E6D9FB0C35AFCEE689632D7F01F222C1E6B
ssb#  rsa2048 2017-08-07 [S]
      564826FB47D9DD7C69FF0E3B24D3127E280FD970
      Keygrip = E416166DD22E7D438885042ACFC6301954B4B0CE

PS C:\Users\MyUser> echo "TEST" | gpg --debug 1024 --armor --local-user 40682A92196A5A7CB44FB85583AE0E9BEE18888E --sign
gpg: reading options from 'C:/Users/MyUser/AppData/Roaming/gnupg/gpg.conf'
gpg: enabled debug flags: ipc
gpg: DBG: chan_0x00000214 <- OK Pleased to meet you
gpg: DBG: connection to agent established
gpg: DBG: chan_0x00000214 -> RESET
gpg: DBG: chan_0x00000214 <- OK
gpg: DBG: chan_0x00000214 -> GETINFO version
gpg: DBG: chan_0x00000214 <- D 2.1.22
gpg: DBG: chan_0x00000214 <- OK
gpg: DBG: chan_0x00000214 -> OPTION allow-pinentry-notify
gpg: DBG: chan_0x00000214 <- OK
gpg: DBG: chan_0x00000214 -> OPTION agent-awareness=2.1.0
gpg: DBG: chan_0x00000214 <- OK
gpg: DBG: chan_0x00000214 -> HAVEKEY 1E7D58F8C5CF01811A62D56E12ED146DED569B2D BC5A1E6D9FB0C35AFCEE689632D7F01F222C1E6B E416166DD22E7D438885042ACFC6301954B4B0CE
gpg: DBG: chan_0x00000214 <- OK
gpg: DBG: chan_0x00000214 -> KEYINFO E416166DD22E7D438885042ACFC6301954B4B0CE
gpg: DBG: chan_0x00000214 <- ERR 67108891 Not found <GPG Agent>
gpg: DBG: chan_0x00000214 -> RESET
gpg: DBG: chan_0x00000214 <- OK
gpg: DBG: chan_0x00000214 -> SIGKEY E416166DD22E7D438885042ACFC6301954B4B0CE
gpg: DBG: chan_0x00000214 <- OK
gpg: DBG: chan_0x00000214 -> SETKEYDESC Please+enter+the+passphrase+to+unlock+the+OpenPGP+secret+key:%0A%22TESTING%22%0A2048-bit+RSA+key,+ID+24D3127E280FD970,%0Acreated+2017-08-07+(main+key+ID+83AE0E9BEE18888E).%0A
gpg: DBG: chan_0x00000214 <- OK
gpg: DBG: chan_0x00000214 -> SETHASH 10 57358CF7C82AF54FE9E6B998B8D6E552B64DE3CC0FB4B9BD72315B8F9E68C81E91C88C9DE74CF7E9613C1AAD55EAD606A610593A296CD13FEEB31D0F614B9609
gpg: DBG: chan_0x00000214 <- OK
gpg: DBG: chan_0x00000214 -> PKSIGN
gpg: DBG: chan_0x00000214 <- ERR 67108881 No secret key <GPG Agent>
gpg: signing failed: No secret key
-----BEGIN PGP MESSAGE-----

gpg: signing failed: No secret key
PS C:\Users\MyUser>

My change only addressed the use case with smartcard. So, I removed [TESTING] tag.

@bluca , I realized that git invokes gpg with -u option. If not, it works well.

I have updated D297: 785_sign-fix.patch patch to minimize the impact only to secret key lookup.

@gniibe yes, I can reproduce the problem using -u.
But why does picking a UID force the usage of the first known subkey? Is that expected behaviour? Is there a relationship between UIDs and subkeys?

I've tried the latest patch you linked to on top of 2.2.1 but still the same problem.

This is the bottom half of gpg --card-status:

General key info..: sub  rsa4096/0x286BF7EFCD77241E 2015-03-28 Luca Boccassi <luca.boccassi@gmail.com>
sec#  rsa4096/0xA81CEA22BC8C7E2E  created: 2015-01-29  expires: never     
ssb>  rsa2048/0x4B29668050785162  created: 2017-02-10  expires: 2019-02-10
                                  card-no: 0006 04877670
ssb>  rsa2048/0x47D16387F9BAE40A  created: 2017-02-10  expires: 2019-02-10
                                  card-no: 0006 04877670
ssb>  rsa2048/0x9BA24C9FA0E4ED15  created: 2017-02-10  expires: 2019-02-10
                                  card-no: 0006 04877670
ssb>  rsa4096/0xF37A672371EB78F8  created: 2015-01-29  expires: 2019-02-10
                                  card-no: 0005 000029E7
ssb>  rsa4096/0xE90A51B21A5956D1  created: 2015-03-28  expires: 2019-02-10
                                  card-no: 0005 000029E7
ssb>  rsa4096/0x286BF7EFCD77241E  created: 2015-03-28  expires: 2019-02-10
                                  card-no: 0005 000029E7

Bottom 3 subkeys are used when calling gpg withou -u, top 3 are required when using -u as you pointed out.

@bluca I created a ticket for smartcard, so that this ticket can focus on the issue of available keys on host. If anything, please add comment to T3416: gpg should select available signing key on card (even with -u option).

Could we please merge it to the stable branch (2.2.3 does not have this patch yet) or it is not tested enough? Existing subkey sellection strategy doesn't play well with mail signing and affects GPGTools/GPGMail users as well as any other users with multiple signing subkeys. Thanks!

@gniibe just checking – any news for 2.2 support? Should I reopen this bug or report a new one against 2.2?