Thanks for testing 2.1 and for reporting the results.
Good to know that it works now.

I have good news : gpg 2.1 rocks !
Problem solved and here is the solution :

As Sijie said, the "smartcard compatible" pageant was loading the SIG key and
the AUTH key.

Unfortunately, under gpg 2.0.x, when you export a public key and use gpg2ssh,
the output is the ssh key for the SIG key (and not the auth).

So when using gpg-agent, it was waiting for putty to request the AUTH key and
not the SIG key (as it should !). The "smartcard enabled" pageant was sending
the SIG key so it was working with it.

Now for the good part : with gpg 2.1, we can now natively use --export-ssh-key,
and this command export the AUTH key, so in the end, it works :)

Thank you everyone for the help, and I hope it can helps other people too !

Can we close this bug please ?

Regards

For history purpose, and trying to maximize information, I have been asked to post some part of the discussion I have
on the mailing list about this problem. Here it is :

I tried older version (of gpg4win) (which, at the time, worked for people with the
same setup as myself), but I can try new version too of course.

That is helpful, because development right now is concentrating more
on Gpg4win 3 with the new GnuPG 2.1 (to become 2.2) and this is where
gpg-agent and pinentry is handled slightly differently. So making sure that
it works with the new version is better for the future.

Ok, I installed gpg4win 3.0.0 BETA 128.
The problem stay the same, no pin is asked.

In the mean time, I tried this tool : http://smartcard-auth.de/ssh-en.html
It replace the pageant.exe that ships with putty. And it works. When I
log on the server with putty, I got asked for the PIN. So I think this
is not a problem with the smartcard or with keys. It seems that it's
only that gpg-agent doesn't trigger the pinentry.

I tried witht gpg-agent on another computer (fresh install) running Windows 7 x64, and
with another smartcard, same problem : no pinentry asked.

Yes gpg-agent is started before, I can see it in the process list (and even the scdaemon process).

In fact, pageant can't be started at the same time as gpg-agent (I suppose it share the same mutex because it
says "pageant is already running" when I try to start pageant while gpg-agent is already running).

Did you start gpg-agent before putty or pageant?

I took a look at the source code and now understand what is going on here.
The code indicates: One or more secret keys (primary or sub) were found.
But the UI message suggests that the secret key of the current (primary) key was
found, hence my confusion.

Here are some ideas:

1. EASY: Update the message to indicate it is generic and not specific to the key

being edited.

OR

2. HARDER: Improve the logic so the message is specific to the key being edited.

Thoughts?

Here you go:

My master key is offline and I have subkeys on a Yubikey. As expected, I see sec# when listing keys when using the
online system:

gpg -K
sec# 4096R/2FFA7695 2016-02-01 [expires: 2020-01-31]
uid NAME <EMAIL@ADDRESS.COM>
ssb> 2048R/EA7CCF1B 2016-02-01
ssb> 2048R/1E8DA9B9 2016-02-01
ssb> 2048R/5BA60C24 2016-02-01
However, when I go into edit mode, gpg indicates that the "Secret is available":

gpg --edit-key 2FFA7695
gpg (GnuPG) 1.4.19; Copyright (C) 2015 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Secret key is available.

pub 4096R/2FFA7695 created: 2016-02-01 expires: 2020-01-31 usage: C

trust: ultimate      validity: ultimate

sub 2048R/EA7CCF1B created: 2016-02-01 expires: 2018-01-31 usage: S
sub 2048R/1E8DA9B9 created: 2016-02-01 expires: 2018-01-31 usage: E
sub 2048R/5BA60C24 created: 2016-02-01 expires: 2018-01-31 usage: A
[ultimate] (1). NAME <EMAIL@ADDRESS.COM>
[ultimate] (2) [jpeg image of size 1234]

Tested with several recent versions of GnuPG. Am I misunderstanding this message?

Please describe the error _here_ and do not link to an external page.

I believe I have also seen this issue (or something very similar) on my Windows
7 64bit machine. I am running gpg 2.1.11. I hope this isn't redundant, but it
seems that I need to restart scdaemon anytime I unplug/replug my yubikey or
suspend/resume my computer.

Sometimes it doesn't recover even after restarting scdaemon. In those cases, I
am able to fix it by stopping scdaemon, removing the yubikey, starting scdaemon,
and finally reinserting the yubikey.

Bug system broke the link URL. Here is a shorter one:
http://security.stackexchange.com/q/115230/16036

I've tested it with pubring now too and it works.
Justus mentioned in jabber that he noticed some more errors after this patch in
the scheme tests. I've not tried them.

Okay, so I can backport this to 2.0 ?

Tested this with keybox and it appears to be working. When running a keylist
while importing the import holds for a bit and continues after the keylist.
Not tested this with keyring yet.

This is caused by gpg inability of merging the secret keys. We can't fix that
in 1.4 or 2.0. 2.1 does not have this problem anymore.

We wont fix that because it has been fixed in 2.1 and backporting make no sense
given that an easy workaround is available.

I have pushed chnages to master to fix this problem. One drawback is that
during an import another process "gpg -k" may rarely see no keys at all. A full
fix would either require that we lock the keyrings during all read-only
operations, which would severely hit on the performance of all common
operations, or change the whole system to use a new key access daemon.

If this works the changes need to be backported to 2.0.

No Kleopatra does not open the pubring. Let's leave kleopatra out of this.

This bug is about multiple GnuPG processes that conflict with each other. See
msg7466 for an example.

Do you mean Kleo opens the pubring file itself?

Commit e70f7a5 fixes this for 2.1.
Should be backported.
Thanks.

Renamed the issue to my current understanding of this problem. Locking on
Windows does not work properly.

Yesterday I had a failed Keygeneration while GpgOL's certificate selection
dialog in Kleopatra was open. Tried again and it worked. I did not get Debug
output but the pattern suggests to me that the Certificate selection dialog
looked for changes in the pubring while generating a key and the locking broke
again.

This problem is rising in my priority of Windows Issues as it causes random
failures. There is also a load of similar reports on various channels to be
found through google https://www.google.at/search?q=pubring.bak

Thank you again.

It is likely that the token itself doesn't work well after wakeup from sleep
mode. In this case, all that we can do is re-inserting the token manually.

I'm not sure how PC/SC service handles USB reset after wakeup.

Sorry to say, but mapping the error to "no reader" doesn't help. The first
reset event doesn't get handled. Later it trys to remove the reader but it's
not getting correctly resetted/reinserted again.

I've attached the debug log again

Thank you for further testing.
I think that current code doesn't handle the case when card goes inactive/reset
while reader keeps working. Current code only goes to the reset sequence for a
card again when it detects reader failure. So, although the concept is
different, I think mapping PSCS_W_CARD_RESET to SW_HOST_NO_READER (for now) will
work. Given the situation we don't yet support multiple cards, this workaround
would be OK for a while.

Nope. Neither mapping the "reset card" event to SW_HOST_CARD_INACTIVE or
SW_HOST_NO_CARD helps. It seems that somewhere in the code the return code
SW error codes are not being handled correctly and the card doesn't get
resetted.

I've attached a small log where you can see that pcsc returns the error
reason "reset card" which then gets remapped to "Card reset required" (was
general error before). I also can see that the error is getting mapped to
GPG_ERR_CARD_RESET (because of the error message "Card reset required")
leaving the daemon around with no working card and reporting general errors
again (0x100b).

Additional Info: This bug only happens when you put your computer/laptop
into sleep mode while the smartcard/reader (yubikey) is plugged in. If I
remove the reader before putting it to sleep and attaching it after getting
out of the sleep mode, the scdaemon works fine.

D338: 740_0002-scd-map-card-reset-event.patch

Maybe it's more appropriate to map the PSCS_W_CARD_RESET event to the
SW_HOST_CARD_INACTIVE error code which later gets mapped to GPG_ERR_CARD_RESET
error code.

I've attached the patch file. It would make sense to backport this mapping as
well. Right now it's not yet tested.

I found another problem with the smartcard service under windows. Putting
the system into sleep mode and waking it up again creates an 0x80100068
error code (aka PCSC_W_RESET_CARD).

I'll test if it helps to map the RESET_CARD event to the same REMOVE_CARD
event to get the card reactivated after sleep mode.

Logfile:
2015-12-21 22:16:57 scdaemon[10040] DBG: send apdu: c=00 i=CA p1=00 p2=C4
lc=-1 le=256 em=0
2015-12-21 22:16:57 scdaemon[10040] DBG: PCSC_data: 00 CA 00 C4 00
2015-12-21 22:16:57 scdaemon[10040] pcsc_transmit failed: reset card
(0x80100068)
2015-12-21 22:16:57 scdaemon[10040] apdu_send_simple(0) failed: general
error

I think that this was fixed in:
http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=commit;h=d1a97585c5e73fbc7d4cf90e38f76ffc5aea305f

It will be in 2.1.11 and 2.0.30.

I've checked that 2.1.10 still has the problem. So back to you.

You can ping me directly if you need any debug logs or so.

Thanks for helping keep track of all these issues.

Yes this only fixes the problem that has already been fixed in the last Gpg4win
Versions. So that this will be fixed in future gnupg-2.1 versions.

Still to help us better seperate the problems I would like to close this as for
me this bug was about "Wrong encoding in a localized version".

• the more critical "passphrase with non ASCII characters" problem (as reported

only here, see T1691 (andreaerdna on Aug 19 2014, 02:36 AM / Roundup)); does this bug need a
dedicated new Issue to be addressed and solved?

I actually overlooked this in this issue. Can you please open another issue for
that. And add me to the Nosy.

• the "utf-8 encoding of encrypted filenames" / "strange behaviour of --utf8-

strings, --no-utf8-strings and --charset options" (as reported in Issue 1409 ad
probably similar to Gpgtar Issue 1624 / Gpa Issue 2185)

If this problem was still existing with gpg4win this is still a problem.

• the "charset weirdness searching keyserver for some non-ASCII user IDs under

non-UTF-8 locales" (as reported in Issue 1514).

This appears not to be windows specific. Also I think this works except for
cases where the Key in question is problematic. If I search on windows for
emanuel@intevation.de I get the correct Umlauts shown. Might be a Problem though
for characters that are unrepresentable in the 8 Bit codepage.

It sounds great!

So this patch, as the previous one, solves the "incorrect display of GPG 2
output translated into another language" (as reported here and previously also
in Issue 1373 and Issue 1674).

Does this patch solve also the "incorrect display of filenames with non ASCII
characters" (as reported here and previously also in Issue 1409)?

By the way, as I understand, this patch doesn't fix:

• the more critical "passphrase with non ASCII characters" problem (as reported

only here, see T1691 (andreaerdna on Aug 19 2014, 02:36 AM / Roundup)); does this bug need a
dedicated new Issue to be addressed and solved?

• the "utf-8 encoding of encrypted filenames" / "strange behaviour of --utf8-

strings, --no-utf8-strings and --charset options" (as reported in Issue 1409 ad
probably similar to Gpgtar Issue 1624 / Gpa Issue 2185)

• the "charset weirdness searching keyserver for some non-ASCII user IDs under

non-UTF-8 locales" (as reported in Issue 1514).

After some more discussion and testing in the development jabber channel werner
agreed to include this patch. Pushed to libgpg-error with 823e858. So this will
hopefully be part of the first gnupg modern release that will include localization.

D210: 737_0001-Fix-windows-8bit-encoding-conversion.patch

Updated Patch against libgpg-error where this code now lives.

Please apply this patch or something similiar.

The problem I can see is that with this code in libgpg-error now GUI
applications may use it which want to get "GUI Native".

Probably better to introduce a new function "wchar_to_console" ? And use it from
GnuPG. Does GPA use that conversion function?

Might be a good time for this now where gnupg master already depends on new
symbols in libgpg-error.

Thank you for your testing.

Your change is pushed with my comment:
http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=commit;h=d1a97585c5e73fbc7d4cf90e38f76ffc5aea305f

I'll backport this to GnuPG 2.0.

Here's the logfile with all the errors (guru debug level) vanilla 2.1.10

After some time spending fighting with the build tools of gnupg (cross compile
for windows under debian) I managed to build the installer with my patched
file.

Most important: The most common error thrown is the 0x8010001e
(E_SERVICE_STOPPED) This is the important one. The other error 0x8010001d
(E_NO_SERVICE) is only thrown in the transition from ok to stopped. So only
sometimes.

This was my process:

git clone git://git.gnupg.org/gnupg.git
cd gnupg
git checkout tags/gnupg-2.1.10
./autogen.sh
cat ../0001-scd-Fix-removal-of-unplugged-usb-readers.patch | patch -p1
sed -i -e 's/^SELFCHECK=1/SELFCHECK=0/' build-aux/speedo.mk
make -f build-aux/speedo.mk w32-installer

I've created new logfiles (vanilla 2.1.10 und patched 2.1.10) to show the
difference and confirm that it'S actually working now :-)

I'm okay with signing off the commit. I can test this for Windows 8.1 or 10,
my only problem is that I'm not able to compile gpg for windows right now. Or
are there instructions somewhere on how to achieve this?

Thank you again.

I think that Windows 8 (and later) changed the PC/SC service. The service is
only available when smartcard is there, and after the removal, it returns
PCSC_E_NO_SERVICE error. This is not expected for current code.

I'm applying your patch with my comment like above. Do you agree to put the
line in the commit log?:

   Signed-off-by: Daniel Hoffend <dh@dotlan.net>

I don't have Windows 8 machine. So, I leave this issue as testing.

No, I just installed version 2.1.10 (which included your mentioned fix). But the
error still applies.

In my case the smartcard reader never gets closed, cause the error thrown by the
pcsc/scd gets only mapped to a general_error which does not result in
removing/closing the reader interface.

I've the feeling that we've to take a closer look at the errors thrown (at least
those 2 in my patch). Maybe there're even more possible events.

If you like I can upload the debug log of scdaemon 2.1.10 ... (if that helps).

Somehow I don't have any issues when running linux, this bug applies to windows
only atm. Maybe it's just that windows is throwing different errors or events
compared to linux.

Thank you for the bug report with log.

It could be related to the bug which was just fixed:
http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=commitdiff;h=f42c50dbf00c2e6298ca6830cbe6d36805fa54a3

I'm backporting this to 2.0.x.

Some more infos:

https://www.openkeychain.org/faq/#importing-your-own-key-from-gnupg-fails
says that this is a problem for a number of people.

Werner told me that porting the fix back would mean to basically
migrate 2.0 to 2.1, which is useless because 2.1 is already 2.1.

Another possibility would be to change --export to mix public keys (certs)
with secret keys. This would create other problems and thus is not
adviable for a stable version.

So I think this is "won't fix" because it (technically) does not make
sense to fix in 1.4 or 2.0. Solutions: Use 2.1 or wait for 2.2.
As importing implementation: Be tolerant for this problem man use the cert
information if you can.

After looking at the gnupg 2.0 branch I would say the patch could be applied
to the 2.0 and 2.1 branch to fix the issue in both branches stable/modern
since both version are affected (tested with 2.1.9 and 2.0.29 from gpg2win)

I just double checked the code from 2.1 - it looks really okay.
I need to look at the 2.x branch, though.

Test data from: http://keyserver.borgnet.us/dump/sks-dump-0000.pgp.bz2

In one console window:
mkdir c:\test-issue2135
set GNUPGHOME=c:\test-issue2135
gpg2 --import c:\users\aheinecke\Desktop\sks-dump-0000.pgp

in another:
set GNUPGHOME=c:\test-issue2135
gpg2 -k

Triggers this: (And the error messages also look wrong)

gpg: waiting for lock c:/test-issue2135/pubring.gpg.lock...
gpg: renaming c:/test-issue2135/pubring.gpg' to c:/test-issue2135/pubring.bak'
failed: Permission
denied
gpg: error writing keyring `c:/test-issue2135/pubring.gpg': Permission denied
gpg: key CBB511F4: public key "[User ID not found]" imported
gpg: error reading `c:\\Users\\aheinecke\\Desktop\\sks-dump-0000.pgp':
Permission denied
gpg: import from `c:\\Users\\aheinecke\\Desktop\\sks-dump-0000.pgp' failed:
Permission denied
gpg: Total number processed: 278
gpg: w/o user IDs: 14
gpg: imported: 265 (RSA: 82)
gpg: renaming c:/test-issue2135/pubring.gpg' to c:/test-issue2135/pubring.bak'
failed: Permission
denied
gpg: failed to rebuild keyring cache: Permission denied
gpg: no ultimately trusted keys found

In this case I'm pretty sure that it does not. I check that I can come up with a
testcase that does not involve kleo.

Is Kleopatra messing around with files in ~/.gnupg in any way? IIRC, Kleo
sometimes bypasses gpgme. For example does it open pubring.gpg ?

@guilhem, @dkg I've cc'd you on this since you seem to be interested in this code.

I've just updated the branch with a few small bug fixes. Most importantly, I
fixed the memory problem by limiting the read-ahead cache to 20 MB. @guilhem:
I'd be interested to hear whether this fixes the problem that you observed.

@bernhard Sorry for not getting back to you sooner. If you checkout neal/kdb,
you'll get the latest code for the kdb format. Set --homedir or GNUPGHOME
appropriately, import your keyring and then try some operations:

$ mkdir /tmp/gnupg-kbx

$ gpg2 --export | time gpg2 --homedir /tmp/gnupg-kbx --no-default-keyring

--keyring gnupg-kdb:pubring.kdb --import /tmp/keys 2>/dev/null

This makes 6.5 seconds on my box using the kdb format and just under 2 min using
kdb. See
https://lists.gnupg.org/pipermail/gnupg-devel/2015-November/030525.html for some
example benchmarks.

I'm particularly interested to hear whether this fixes the major performance
problem that you're experiencing.

Thanks.

As I understand the problem, a key appeared in multiple keyrings and this was
causing confusion. I don't think there is a bug here so I'm marking this issue
as resolved.

I've disabled the automatic keylisting while an import job is running in
Kleopatra as this is a good idea anyway.

Still this should be fixed although we might want to give it a try with 2.1
instead as it is no longer a hard issue for gpg4win with the workarond in kleo
in place.

The import with 2.0.29 is also very slow on Windows. Over two minutes to import
650 keys while the same import with 2.1.9 on GNU/Linux only takes 20seconds.

@werner
Running with --no-sig-cache took 30 Minutes.
gpg2 --delete-key 52D717F3
time LANG=C gpg2 -v --no-sig-cache --recv-keys 52D717F3
real 29m38.897s

While time LANG=C gpg2 -v --recv-keys 52D717F3 took 2 minutes.
Debian gnupg2 Version: 2.0.26-6 i386.

@neal:
Thanks for working on this, if you think it may may sense to test this
with real data, can you point to the steps required to do this?
(I guess building gpg-2.1-from your git branch, ...)

@All,
any idea what the change between 2.0.25-99intevation2 on Wheezy
and 2.0.26-6 on Jessie could be that would cause this problem?
(Or is it just a few small certs or trust settings more that will cause
this one magnitude higher load)

I've implemented a new db format. It's still incomplete and experimental, but
it's available from the neal/next branch. Importing
/usr/share/keyrings/debian-keyring.gpg, which contains 751 keys is much faster
using this format:

$ rm pubring.kdb; time gpg2 --no-default-keyring --primary-keyring
gnupg-kdb:pubring.kdb --import debian-keyring.gpg >/dev/null
gpg: Total number processed: 751
gpg: imported: 751

real 0m7.729s
user 0m5.404s
sys 0m0.332s
$ rm pubring.kdx; time gpg2 --no-default-keyring --primary-keyring
gnupg-kbx:pubring.kdx --import debian-keyring.gpg >/dev/null
gpg: Total number processed: 751
gpg: imported: 751
gpg: public key of ultimately trusted key 2183839A not found
gpg: public key of ultimately trusted key BC15C85A not found
gpg: public key of ultimately trusted key EE37CF96 not found

real 1m52.560s
user 0m6.268s
sys 0m31.604s

Running --check-trustdb is almost an order of magnitude faster:

$ time gpg2 --no-default-keyring --primary-keyring gnupg-kdb:pubring.kdb
--check-trustdb
real 0m0.158s
user 0m0.004s
sys 0m0.004s
$ time gpg2 --no-default-keyring --primary-keyring gnupg-kbx:pubring.kbx
--check-trustdb
real 0m0.975s
user 0m0.012s
sys 0m0.032s

Doing a sequential read is a bit slower:

$ time gpg2 --no-default-keyring --primary-keyring gnupg-kdb:pubring.kdb -k |
grep ^pub | wc -l
751

real 0m2.515s
user 0m2.432s
sys 0m0.088s
$ time gpg2 --no-default-keyring --primary-keyring gnupg-kbx:pubring.kdx -k |
grep ^pub | wc -l
gpg: NOTE: THIS IS A DEVELOPMENT VERSION!
gpg: It is only intended for test purposes and should NOT be
gpg: used in a production environment or with production keys!
751

real 0m1.245s
user 0m1.168s
sys 0m0.076s

This is because the interface for doing a full scan of the DB is unsuitable. If
we decide to use the new format, it shouldn't be hard to improve this.

I'd be interested in any feedback and perhaps some more measurements in real
conditions.

Thanks,

Neal