Page MenuHome GnuPG

Broken umlauts in the new Windows Terminal
Open, NormalPublic

Description

The recent unicode changes has broken umlauts with PowerShell Core and the new Windows Terminal app at least. For example, Jäätelö comes through as J├ñ├ñtel├╢ when it is passed through the PowerShell pipeline.

Here's a short repro:

  1. Initialize:
gnupgtest> $env:GNUPGHOME=$pwd
gnupgtest> gpg --list-keys
gpg: keybox 'C:\\devel\\gnupgtest\\pubring.kbx' created
gpg: C:\\devel\\gnupgtest\\trustdb.gpg: trustdb created
  1. Generate a key:
gnupgtest> gpg --generate-key
gpg (GnuPG) 2.3.4; Copyright (C) 2021 g10 Code GmbH
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Note: Use "gpg --full-generate-key" for a full featured key generation dialog.

GnuPG needs to construct a user ID to identify your key.

Real name: Jäätelö
Email address: jäätelö@example.com
You are using the 'CP437' character set.
You selected this USER-ID:
    "Jäätelö <jäätelö@example.com>"

Change (N)ame, (E)mail, or (O)kay/(Q)uit? O
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: AllowSetForegroundWindow(10208) failed: Access is denied.
gpg: AllowSetForegroundWindow(11560) failed: Access is denied.
gpg: AllowSetForegroundWindow(8160) failed: Access is denied.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: directory 'C:\\devel\\gnupgtest\\openpgp-revocs.d' created
gpg: revocation certificate stored as 'C:\\devel\\gnupgtest\\openpgp-revocs.d\\C64EF3CDED8BBFB01ADC6856F0183F7C64A93C5B.rev'
public and secret key created and signed.

pub   ed25519 2022-01-07 [SC] [expires: 2024-01-07]
      C64EF3CDED8BBFB01ADC6856F0183F7C64A93C5B
uid                      Jäätelö <jäätelö@example.com>
sub   cv25519 2022-01-07 [E] [expires: 2024-01-07]
  1. Check the key (all good so far?):
gnupgtest> gpg --list-keys
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at 2024-01-07
C:\devel\gnupgtest\pubring.kbx
------------------------------
pub   ed25519 2022-01-07 [SC] [expires: 2024-01-07]
      C64EF3CDED8BBFB01ADC6856F0183F7C64A93C5B
uid           [ultimate] Jäätelö <jäätelö@example.com>
sub   cv25519 2022-01-07 [E] [expires: 2024-01-07]
  1. Try passing some output through the PowerShell pipeline (notice the umlauts):
gnupgtest> gpg --list-keys | select-string uid

uid           [ultimate] Jäätelö <jäätelö@example.com>
  1. Versions, codepages, etc:
gnupgtest> chcp
Active code page: 65001
gnupgtest> [System.Text.Encoding]::Default.EncodingName
Unicode (UTF-8)
gnupgtest> gpgconf --show-versions
* GnuPG 2.3.4 (f74c65fd9)
MingW32
Windows 10.0 build 22000

* Libgcrypt 1.9.4 (05422ca2)
version:1.9.4:10904:1.43:12b00:
cc:80300:gcc:8.3-win32 20190406:
ciphers:arcfour:blowfish:cast5:des:aes:twofish:serpent:rfc2268:seed:camellia:idea:salsa20:gost28147:chacha20:sm4:
pubkeys:dsa:elgamal:rsa:ecc:
digests:crc:gostr3411-94::md4:md5:rmd160:sha1:sha256:sha512:sha3:tiger:whirlpool:stribog:blake2:sm3:
rnd-mod:w32:
cpu-arch:x86:
mpi-asm:i386/mpih-add1.S:i386/mpih-sub1.S:i386/mpih-mul1.S:i386/mpih-mul2.S:i386/mpih-mul3.S:i386/mpih-lshift.S:i386/mpih-rshift.S:
hwflist:intel-cpu:intel-fast-shld:intel-bmi2:intel-ssse3:intel-sse4.1:intel-pclmul:intel-aesni:intel-rdrand:intel-avx:intel-avx2:intel-fast-vpgather:intel-rdtsc:
fips-mode:n:n:
rng-type:standard:1:2010000:1:
compliance:::

* GpgRT 1.43 (d7fb048)

* Libassuan 2.5.5 (f8cfb56)

* KSBA 1.6.0 (6b3573a)

* NTBTLS 0.2.0 (6a4ffcc)
gnupgtest> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.2.1
PSEdition                      Core
GitCommitId                    7.2.1
OS                             Microsoft Windows 10.0.22000
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

It's surprisingly hard to print Windows 11 version from command line, but I'm on Windows 11 Enterprise 21H2 22000.376 with Windows Terminal 1.11.3471.0. I installed gpg from the Simple installer for the current GnuPG link, not sure if it counts as gpg4win.

I've also set the Windows Terminal app as the default terminal application for Windows.

There's one interesting line in the key generation phase:

You are using the 'CP437' character set.

I thought I was using utf-8?

Details

Version
2.3.4

Event Timeline

We use GetConsoleOutputCP but fallback to GetACP if the former fails. For some reasons one of the functions seems to return 437.

Can you please try calling gpg with the option

--charset utf-8

That seems to (mostly) work partially fix PowerShell pipeline output at least:

gnupgtest2> gpg --list-keys
gpg: keybox 'C:\\devel\\gnupgtest2\\pubring.kbx' created
gpg: C:\\devel\\gnupgtest2\\trustdb.gpg: trustdb created
gnupgtest2> gpg --generate-key --charset utf-8
gpg (GnuPG) 2.3.4; Copyright (C) 2021 g10 Code GmbH
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Note: Use "gpg --full-generate-key" for a full featured key generation dialog.

GnuPG needs to construct a user ID to identify your key.

Real name: Jäätelö
Email address: jäätelö@example.com
You are using the 'utf-8' character set.
You selected this USER-ID:
    "Jäätelö <jäätelö@example.com>"

Change (N)ame, (E)mail, or (O)kay/(Q)uit? O
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: AllowSetForegroundWindow(3568) failed: Access is denied.
gpg: AllowSetForegroundWindow(32284) failed: Access is denied.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: directory 'C:\\devel\\gnupgtest2\\openpgp-revocs.d' created
gpg: revocation certificate stored as 'C:\\devel\\gnupgtest2\\openpgp-revocs.d\\3F0AEB7A8BFA6F65523641EBA258D07FFE60E51C.rev'
public and secret key created and signed.

pub   ed25519 2022-01-10 [SC] [expires: 2024-01-10]
      3F0AEB7A8BFA6F65523641EBA258D07FFE60E51C
uid                      Jäätelö <jäätelö@example.com>
sub   cv25519 2022-01-10 [E] [expires: 2024-01-10]

Some tests:

gnupgtest2> gpg --list-keys
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at 2024-01-10
C:\devel\gnupgtest2\pubring.kbx
--------------------------------
pub   ed25519 2022-01-10 [SC] [expires: 2024-01-10]
      3F0AEB7A8BFA6F65523641EBA258D07FFE60E51C
uid           [ultimate] J��tel� <j��tel�@example.com>
sub   cv25519 2022-01-10 [E] [expires: 2024-01-10]


gnupgtest2> gpg --list-keys --charset utf-8
C:\devel\gnupgtest2\pubring.kbx
--------------------------------
pub   ed25519 2022-01-10 [SC] [expires: 2024-01-10]
      3F0AEB7A8BFA6F65523641EBA258D07FFE60E51C
uid           [ultimate] Jäätelö <jäätelö@example.com>
sub   cv25519 2022-01-10 [E] [expires: 2024-01-10]

gnupgtest2> gpg --list-keys | select-string uid

uid           [ultimate] Jäätelö <jäätelö@example.com>

gnupgtest2> gpg --list-keys --charset utf-8 | select-string uid

uid           [ultimate] Jäätelö <jäätelö@example.com>
werner renamed this task from Broken umlauts to Broken umlauts in the new Windows Terminal.Thu, Jan 20, 8:23 PM
werner triaged this task as Normal priority.