I would love to defer those encoding problems to gpgme. The output of gpgme should always be UTF-8 encoded, so that higher API doesn't need to bother about encoding. Obviously, this would mean that we shouldn't bypass gpgme by running gpg commands with QProcess.

It's a pity that gpg's --display-charset "should not be used on Windows" (the man page). Because I wanted to propose to force gpg to output UTF-8 on Windows, so that we can be sure that we are dealing with UTF-8. But I guess there's a reason why that's not done on Windows.

stringFromGpgOutput is used by

• DumpCertificateCommand
• DumpCrlCacheCommand
• ImportCrlCommand
• GnuPGProcessCommand which is the base for
  • ClearCrlCacheCommand
  • ExportOpenPGPCertsToServerCommand
  • ExportPaperKeyCommand
  • GenRevokeCommand
  • ImportPaperKeyCommand
  • LearnCardKeysCommand
  • RefreshOpenPGPCertsCommand
  • RefreshX509CertsCommand

gpg always used UTF-8 the option to enable this is the default for more than a decade.

Oh well, in case you mean the diagnostic output things are different. gpg uses the code page as returned by GetConsoleOutputCP becuase we are a console program. Qt might assume that GetACP is used (as for GUis and thus we get a wrong codepage. You may try to run "chcp 65001" before starrting kleopatra or figure out how to convince Qt to set the codepage to 65001 (utf-8)

I created a self-signed X.509 test certficate with subject CN=日本国,C=JP (Nihon-koku/Nippon-koku; formal name of Japan).

Import the certificate and click on More Details in the Certificate Details window to reproduce the problem. For me the subject is displayed as

Subject: CN=µùѵ£¼Õø¢,C=JP

Test certificate:

In Windows PowerShell, I get the following output:

The active codepage is 65001.

Copying the output and pasting it in Editor (notepad?), it is displayed correctly. So maybe PowerShell just lacks a font to display Kanji characters.

Debug output in stringFromGpgOutput gives me

[1352] org.kde.pim.libkleo: stringFromGpgOutput GetConsoleOutputCP() returned 0
[1352] org.kde.pim.libkleo: stringFromGpgOutput GetOEMCP() returned 850
[1352] org.kde.pim.libkleo: stringFromGpgOutput GetACP() returned 1252

Kleopatra uses the first one that's not 0, i.e. 850, which is obviously a bad choice. The latin-*-variant 1252 won't help either.

stringFromGpgOutput actually received the following 8-bit data:

[1352] org.kde.pim.libkleo: stringFromGpgOutput ba: "      Subject: CN=\xE6\x97\xA5\xE6\x9C\xAC\xE5\x9B\xBD,C=JP"

This looks very much like UTF-8 (i.e. codepage 65001).

Since GetConsoleOutputCP() returns 0 for Kleopatra we probably need to ask gpg what codepage to use.

Should be fixed. Tested as described in the above comment T5960#165845.

Note that this requires a version of gpgconf that supports the --show-codepages option.

Here is an other instance with broken encoding (on 3.1.27.0-beta44):

In T5960#168905, @ebo wrote:

Here is an other instance with broken encoding (on 3.1.27.0-beta44):

Technically, this is a completely different problem because in this case the text is not read from stdout/stderr of a different process but it is returned by GnuPG's own gettext implementation (on Windows). We may have to call gettext_use_utf8(1) somewhere. Otherwise, GnuPG's gettext seems to convert the texts to local encoding, but Kleopatra always assumes UTF-8. This should affect all texts containing non-ASCII.

@werner Can you confirm that we have to call gettext_use_utf8(1) in Kleopatra to make GnuPG's gettext return UTF-8 on Windows?

Found this one in Gpg4win-4.2.0-beta384:

Found in GnuPG-VS-Desktop-3.2.0-beta178, broken ü and no linebreaks:

Setting this to high after consultation with @werner, these issues should be fixed in next release.

In T5960#175302, @ebo wrote:

Found in GnuPG-VS-Desktop-3.2.0-beta178, broken ü and no linebreaks:

Setting this to high after consultation with @werner, these issues should be fixed in next release.

This is export to server. This one seems to come from a direct call to gpg instead of using gpgme. Very bad.

The debug output of libkleo is missing. Please add org.kde.pim.libkleo.debug=true to your qtlogging.ini.

With regard to the ü-error in the error message on keytocard somebody else has to check what's going wrong because I'm convinced that we are doing everything right.

In gpgme (src/version.c) we call

gettext_use_utf8 (1);

which is supposed to force libgpg-error's gettext to return UTF-8 encoded messages.

And in libkleo Formatting::errorAsString which is used for the error message on keytocard we do

#if defined(Q_OS_WIN) && GPGMEPP_ERROR_ASSTRING_RETURNS_UTF8_ON_WINDOWS
    return QString::fromUtf8(error.asString());
#else
    return QString::fromLocal8Bit(error.asString());
#endif

I'm blind to seeing where this goes wrong.

Edit: The above changes have landed in the code in April 2023. It should be double-checked that Gpg4win-4.2.0-beta384 didn't use older code. Best check again with the latest build.

The previous two changes will hopefully fix the error output of export to server. I'm not entirely sure about the missing line breaks.

Hopefully, we will soon have a Windows CI to catch such errors.

In T5960#175318, @ikloecker wrote:

Also, please retest T5960#172860 with the latest build.

The ü there is still broken in Gpg4win-4.2.1-beta31 build today with up to date libkleo (but 3 days old gpgme)

[6064] org.kde.pim.kleopatra: KeyToCardCommand::doStart()
[6064] org.kde.pim.kleopatra: KeyToCardCommand::Private::start()
[6064] org.kde.pim.kleopatra: ReaderStatus::getCard() - Found card with serial number "D2760001240100000006154932980000" and app "openpgp"
[6064] org.kde.pim.kleopatra: KeyToCardCommand::Private::startKeyToOpenPGPCard()
[6064] org.kde.pim.kleopatra: ReaderStatus::getCard() - Found card with serial number "D2760001240100000006154932980000" and app "openpgp"
[6064] org.kde.pim.kleopatra: ReaderStatusThread[2nd]: new iteration command= "KEYTOCARD --force 8B8713E028E9DC05A4FEA1B70A21B293C2081819 D2760001240100000006154932980000 OPENPGP.2 20230308T111545"  ; nullSlot= false
[6064] org.kde.pim.libkleo: sendCommand "SCD SWITCHCARD D2760001240100000006154932980000"
[6064] org.kde.pim.libkleo: sendStatusLinesCommand "SCD SWITCHCARD D2760001240100000006154932980000" : got ( status( "SERIALNO" ) = "D2760001240100000006154932980000" 
[6064]  )
[6064] org.kde.pim.libkleo: sendCommand "SCD SWITCHAPP openpgp"
[6064] org.kde.pim.libkleo: sendStatusLinesCommand "SCD SWITCHAPP openpgp" : got ( status( "SERIALNO" ) = "D2760001240100000006154932980000 openpgp piv" 
[6064]  )
[6064] org.kde.pim.libkleo: sendCommand "KEYTOCARD --force 8B8713E028E9DC05A4FEA1B70A21B293C2081819 D2760001240100000006154932980000 OPENPGP.2 20230308T111545"
[6064] org.kde.pim.libkleo: sendCommand "KEYTOCARD --force 8B8713E028E9DC05A4FEA1B70A21B293C2081819 D2760001240100000006154932980000 OPENPGP.2 20230308T111545" failed: Ung?ltiger Wert (code: 55, source: SCD)

Then gettext_use_utf8 doesn't seem to be working as intended.

In T5960#175407, @ebo wrote:

The result of your changes is a bit unexpected for me, maybe something is missing in the testversion Gpg4win-4.2.1-beta31?

[6064] org.kde.pim.libkleo: gpgConfGetConsoleOutputCodePage returns 65001
[6064] org.kde.pim.libkleo: stringFromGpgOutput trying to decode "" using codepage 65001

Looks like the output of the command is empty. Maybe a timing problem?

[6064] kf.i18n.kuit: "Markup error in message {<__kuit_internal_top__><para>Beim Export der OpenPGP-Zertifikate ist ein Fehler ...}: Unerwartet ''.. Last tag parsed: message. Complete message follows:\n<__kuit_internal_top__><para>Beim Export der OpenPGP-Zertifikate ist ein Fehler aufgetreten.</para> <para>Die Ausgabe von <command>C:\\Program Files (x86)\\GnuPG\\bin\\gpg.exe</command> lautet: <message>\u0000</message></para></__kuit_internal_top__>"

Looks like we decode the string including the terminating null character.

This looks to me like the typical thing where you also do not get the "Diagnostics" button viewed. Because log-file is set in the various .conf files and the stderr does not get back to GPGME / Kleopatra so we have nothing to show there because the actual error ended up in a log file.

ok, with the new testversion VS-Desktop-3.2.0.0-beta214 and without Debug redirection (mea culpa), the info window looks like this:

[3376] org.kde.pim.libkleo: gpgConfGetConsoleOutputCodePage starting "C:\\Program Files (x86)\\GnuPG VS-Desktop\\bin\\..\\GnuPG\\bin\\gpgconf.exe" --show-codepages
[3376] org.kde.pim.libkleo: gpgConfGetConsoleOutputCodePage ConsoleOutputCP is 0 - use ACP
[3376] org.kde.pim.libkleo: gpgConfGetConsoleOutputCodePage returns 0

I tried to reproduce this with a locally built gpg4win-4.2.1-beta56. Using localhost as keyserver I got (after a looooooong timeout)

with debug output

[6472] org.kde.pim.libkleo: gpgConfGetConsoleOutputCodePage starting "C:\\Program Files (x86)\\GnuPG\\bin\\gpgconf.exe" --show-codepages
[6472] org.kde.pim.libkleo: gpgConfGetConsoleOutputCodePage returns 65001
[...]
[6472] org.kde.pim.kleopatra: "C:\\Program Files (x86)\\GnuPG\\bin\\gpg.exe --send-keys 40C8670A86AC2B2C35AC6E1B25579E2EBB089E1B gestartet ..."
[...]
[6472] org.kde.pim.libkleo: stringFromGpgOutput trying to decode "gpg: sende Schl\xC3\xBCssel 25579E2EBB089E1B auf hkps://localhost" using codepage 65001
[6472] org.kde.pim.kleopatra: "gpg: sende Schlüssel 25579E2EBB089E1B auf hkps://localhost"
[6472] org.kde.pim.libkleo: stringFromGpgOutput trying to decode "gpg: Senden an Schl\xC3\xBCsselserver fehlgeschlagen: Unknown error\r\ngpg: Senden an Schl\xC3\xBCsselserver fehlgeschlagen: Unknown error" using codepage 65001
[6472] org.kde.pim.kleopatra: "gpg: Senden an Schlüsselserver fehlgeschlagen: Unknown error\ngpg: Senden an Schlüsselserver fehlgeschlagen: Unknown error"
[6472] org.kde.pim.libkleo: stringFromGpgOutput trying to decode "gpg: sende Schl\xC3\xBCssel 25579E2EBB089E1B auf hkps://localhost\r\ngpg: Senden an Schl\xC3\xBCsselserver fehlgeschlagen: Unknown error\r\ngpg: Senden an Schl\xC3\xBCsselserver fehlgeschlagen: Unknown error\r\n" using codepage 65001
[6472] org.kde.pim.libkleo: stringFromGpgOutput trying to decode "gpg: sende Schl\xC3\xBCssel 25579E2EBB089E1B auf hkps://localhost\r\ngpg: Senden an Schl\xC3\xBCsselserver fehlgeschlagen: Unknown error\r\ngpg: Senden an Schl\xC3\xBCsselserver fehlgeschlagen: Unknown error\r\n" using codepage 65001

The remaining problem with output that's read directly from a GnuPG process seems to be that sometimes gpgconf --show-codepages prints 0 as ConsoleOutputCodePage or even as ACP (as in T5960#175580).

If a valid ConsoleOutputCodePage is printed by gpgconf as in my previous comment, then the decoding works as intended.

I have no idea why gpgconf sometimes fails to print valid code pages. Maybe it depends on the way Kleopatra is started. I started it by double clicking the application icon on the desktop and by typing kleopatra in the Search field.

With regard to invalid characters in error messages as in T5960#172860:
We are using the identical function everywhere to convert the error codes to text. If an invalid character is displayed if copying a key to a smart card failed, then invalid characters should also be displayed for example if one tries to decrypt something in the notepad without having the necessary secret key. I get "Kein geheimer Schlüssel" in this case, i.e. no invalid character.

In the VSD Testversion:

C:\Users\g10code.WIN-TEST3>gpgconf --show-codepages
Ungültige Option "--show-codepages"

So this seems to be an issue of the gpg version in VSD

gpgconf --show-codepages ist just a debugging aid. We use the code pages only for output to the console. The problem we see here is about log messages and they are always send as utf8 to stderr or the pipe used instead - without any translation.

From our always used init_common_subsystems:

#ifdef HAVE_W32_SYSTEM
  /* We want gettext to always output UTF-8 and we put the console in
   * utf-8 mode.  */
  gettext_use_utf8 (1);
  if (!SetConsoleCP (CP_UTF8) || !SetConsoleOutputCP (CP_UTF8))

See also T4365 and rGb912f07cdf (gnupg-2.2)

With the latest change the output read directly from gpg should now be displayed correctly.

Error messages were already displayed correctly for me on Windows (T5960#175623).

This looks nice! One down...

But "Zeitüberschreitung" is still broken. (See pictures above.)
For retest of "ungültiger Wert" I would need a newer gpg4win Testversion.

with VS-Desktop-3.1.90.258-Beta the ü is still broken for "Zeitüberschreitung" in the error window which occurs in the context of keytocard:

[5780] org.kde.pim.libkleo: sendCommand "KEYTOCARD --force 1A63F7F35AB5F31A7BE34555DE616C0A2E48F32E D276000124010304000500009D400000 OPENPGP.1 20230308T111343" failed: Zeit?berschreitung (code: 62, source: Pinentry)

In the context of e.g. signing there is another kind of window, where the "ü" is displayed correctly.

I tried to reproduce this but I didn't succeed. I used "Create New OpenPGP Cert" for my attempts.

First try with unpatched error dialog. Note the slashes surrouding the error text.

Next try with added translation context "@info" which makes xi18nc output HTML. Note: Now the error text is printed italic. But the error message is not translated (because of the added translation context).

Next try with patched translations adding the message with context. Done to rule out (encoding of) translation playing a role in the issue.

Next try with the exact error message the encoding issue occurred with (but still with OpenPGP certificate creation).

Final try with the exact same code for the message box.

I have no idea why it should make a difference whether the error message is shown after OpenPGP certificate creation or after copying a key to a smart card. Everything should run in the main thread. I'll add some more debug output for further investigation.

I have no idea why it should make a difference whether the error message is shown after OpenPGP certificate creation or after copying a key to a smart card. Everything should run in the main thread. I'll add some more debug output for further investigation.

The one goes through GPGME and the other is talking directly to the agent so there is a big difference.

Sure. But in both cases they get back an error code and then, back in safe Kleopatra territory, they convert this error code into a string. I don't see how this error code to string conversion should have anything to do with the way the commands triggered the operation. Both conversions should happen in the main thread, so that the state of libgpg-error should be identical. The logging will show if in the still failing case libgpg-error is not switched to UTF-8 for localized texts.