To reproduce this issue I started Kleopatra with an empty GNUPGHOME and imported 10 S/MIME certs at once (which spawns a gpgsm process each) with enabled logging.

It is not all actions that are blocked then but all which need to write in the pubring. So this might not be related to keylisting blocking in Kleopatra.

The processes hang while waiting for the pubring.kbx.lock as they write in the debug output.

One process is looping endlessly in gnupg_rename_file trying to rename the pubring. This fails with a sharing violation. I assume that this process is the one that currently holds the lock.

But according to procexp the file "pubring.kbx" is open in a different gpgsm process PID 12400 but that process is also writing "waiting for lock" so this looks like there is at least one place where the pubring is open without having the lock.

The last lines that the process currently holding wrote in the log:

2019-05-14 10:37:59 gpgsm[12400] DBG: chan_0x00000114 <- # Home: C:\Users\aheinecke\AppData\Roaming\gnupg
2019-05-14 10:37:59 gpgsm[12400] DBG: chan_0x00000114 <- # Config: [none]
2019-05-14 10:37:59 gpgsm[12400] DBG: chan_0x00000114 <- OK Dirmngr 2.2.15 at your service
2019-05-14 10:37:59 gpgsm[12400] DBG: connection to the dirmngr established
2019-05-14 10:37:59 gpgsm[12400] DBG: chan_0x00000114 -> GETINFO version
2019-05-14 10:37:59 gpgsm[12400] DBG: chan_0x00000114 <- D 2.2.15
2019-05-14 10:37:59 gpgsm[12400] DBG: chan_0x00000114 <- OK
2019-05-14 10:37:59 gpgsm[12400] DBG: chan_0x00000114 -> OPTION audit-events=1
2019-05-14 10:37:59 gpgsm[12400] DBG: chan_0x00000114 <- OK
2019-05-14 10:37:59 gpgsm[12400] DBG: chan_0x00000114 -> LOOKUP --cache-only /CN=GlobalSign,O=GlobalSign,OU=GlobalSign%20Root%20CA%20-%20R3
2019-05-14 10:37:59 gpgsm[12400] DBG: chan_00000114 <- [ 44 20 30 82 03 5f 30 82 02 47 a0 03 02 01 02 02 ...(895 byte(s) skipped) ]
2019-05-14 10:37:59 gpgsm[12400] DBG: chan_0x00000114 <- END

After killing that 12400 the process that was looping in gnupg_rename continues and finishes. Then I'm asked to mark a root CA as trusted by PID 2412. After confirmation that process waits on the lock while 2656 hangs in gnupg_rename. But according to procexp again 2412 holds the pubring.kbx open already.

 2019-05-14 11:23:31 gpgsm[2412] DBG: chan_0x000000f4 <- OK
2019-05-14 11:23:31 gpgsm[2412] Das Wurzelzertifikat wurde nun als vertrauenswrdig markiert
2019-05-14 11:23:31 gpgsm[2656] Warte bis auf die Datei 'C:\Users\aheinecke\AppData\Roaming\gnupg\pubring.kbx' zugegriffen werden kann ...
2019-05-14 11:23:32 gpgsm[2412] waiting for lock C:\Users\aheinecke\AppData\Roaming\gnupg\pubring.kbx.lock...

That was obvious. rG6fc5df1e10129f3171d80cf731f310b9e8d97c26 fixes this.

@rjh reported a problem with keyboxd from the current 2.3 beta on the ML. This is also a locking problem and _might_ be related to this bug.

Well, this is a pure Windows bug. It easily shows up when running dozens of gpgsm processes each importing a different certificate (e.g. using Kleopatra's current importer, which spawns one process per cert). The only possible fix is to close all files before starting a long running operation *and* before locking the files.

Sure, we could do this. Shouldn't make the ImportCertificatesCommand much more complex than it already is.

With 100 concurrently running gpgsm processes they all try to get the lock for the keyring. And they need to do this several times and often also for the same certificate (fetched from an external resource to complete the chain). Not good. It might be easier to bypass the gpgsm and run gpgsm directly instead of adding a feature to gpgsm to directly import from many files.

@werner Do I understand correctly that by "It might be easier to bypass the gpgsm and run gpgsm directly" you mean using gpgsm in server mode? Or what do you mean with "bypass gpgsm and run gpgsm" (which seems contradictory).

I meant bypass the gpgme engine and call gpgsm directly. Maybe using gpgme's spawn engine. But I am not sure whether this is really a good idea. If we can find a way to pass multiple filenames to gpgsm --server that would be better. But requires updates to gpgsm.

I really don't want to bypass gpgme and then parse the import results and all other status output of gpgsm ourselves. I'll go for Andre's suggestion and serialize imports of multiple files.

In T4505#166390, @ikloecker wrote:

I really don't want to bypass gpgme and then parse the import results and all other status output of gpgsm ourselves. I'll go for Andre's suggestion and serialize imports of multiple files.

I have an Idea. Can't we read all data into memory in Kleopatra (for Certificates this should be ok) and then give this to GPGME as a single data object. So that only one process imports multiple files?

In T4505#166463, @aheinecke wrote:

I have an Idea. Can't we read all data into memory in Kleopatra (for Certificates this should be ok) and then give this to GPGME as a single data object. So that only one process imports multiple files?

We would lose the chance to inform the user about problematic files or about which file they entered the wrong password for. Moreover, we'd lose any chance to give the user a hint which file pinentry wants the password for. (Currently, pinentry also doesn't display a reference to a filename, so it wouldn't become worse than it is already. And it's pretty bad.) Together, it would become pretty much impossible to import multiple files that are protected with different passwords.

I think processing one file after the other is the better approach even if starting 1000 gpgsm processes (one after the other) is slower than starting a single gpgsm process. But let's be realistic. How often does somebody import more than 1 file with certificates, let alone 1000 files with certificates?

@ikloecker You are right, I only thought of public key import. Then lets serialize this. Might even make for a nicer Progressbar if we count the outstanding files.