fix gpgme/gpgsm pipe server session with use_descriptor_passing (was: mutt + gpgme problems with some Outlook S/MIME emails)
Open, NormalPublic

Description

A work colleague sends me S/MIME signed messages which I read with mutt +
gpgme/gpgsm. Most of them I can read[*], but for some I get

[-- Error: decryption failed: Invalid value passed to IPC --]

as the body, nothing else.

Some more information in the hope it helps debugging: When looking at the
headers, I see

Content-Type: application/pkcs7-mime; smime-type=signed-data;
        name="smime.p7m"
Content-Disposition: attachment; filename="smime.p7m"
Content-Transfer-Encoding: base64
MIME-Version: 1.0

and then a base64 encoded body.

When I manually decode the body, I get weird characters at the start before the
real test, but perhaps that's the p7m file format:

00000000  30 80 06 09 2a 86 48 86  f7 0d 01 07 02 a0 80 30  |0...*.H........0|
00000010  80 02 01 01 31 0b 30 09  06 05 2b 0e 03 02 1a 05  |....1.0...+.....|
00000020  00 30 80 06 09 2a 86 48  86 f7 0d 01 07 01 a0 80  |.0...*.H........|
00000030  24 80 04 82 01 2d 43 6f  6e 74 65 6e 74 2d 54 79  |$....-Content-Ty|
00000040  70 65 3a 20 6d 75 6c 74  69 70 61 72 74 2f 6d 69  |pe: multipart/mi|
00000050  78 65 64 3b 0d 0a 09 62  6f 75 6e 64 61 72 79 3d  |xed;...boundary=|
00000060  22 2d 2d 2d 2d 3d 5f 4e  65 78 74 50 61 72 74 5f  |"----=_NextPart_|
00000070  30 30 30 5f 30 32 37 39  5f 30 31 44 31 36 44 34  |000_0279_01D16D4|
00000080  38 2e 34 36 42 35 41 46  38 30 22 0d 0a 58 2d 4d  |8.46B5AF80"..X-M|
  • Format for those is
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
        micalg=SHA1; boundary="----=_NextPart_000_030D_01D16D6E.123456789"

I originally reported this as a mutt bug report:

https://dev.mutt.org/trac/ticket/3808

but a recent comment there suggests it might be a problem in gpgme.

Details

Version
1.8.0
wiz set Version to 1.6.0.Jan 11 2017, 2:27 PM
wiz added projects: Bug Report, gpgme.
wiz added a subscriber: wiz.
werner added a subscriber: werner.Mar 21 2017, 10:02 AM

To debug this you need to run mutt like this:

GPGME_DEBUG=9:gpgme.trace: mutt

The trace file will be pretty verbose but contains everything GPGME sees from
the engine.

wiz added a comment.Mar 21 2017, 12:19 PM

wiz added a comment.Mar 21 2017, 12:19 PM

I've run mutt as suggested on the troublesome email; the resulting log is attached.

Thanks. So gpgsm is started but terminates before it can actually receive a
command from GPGME. Can you please add

log-file /bar/bar/gpgsm.log
debug 1024
verbose

to ~/.gnupg/gpgsm.conf and run the test again. Please post the log again.

werner claimed this task.Mar 22 2017, 8:38 AM
wiz added a comment.Mar 22 2017, 10:31 AM

Right now it seems to be complaining about an expired key.
(Could be that this is a different error than originally reported because it's
been some time...)

wiz added a comment.Mar 22 2017, 10:31 AM

The log indeed does not match the former gpgme2.trace.

I would try to switch to gpgme 1.8.0 so see whether you can still reproduce the
problem.

wiz added a comment.Apr 14 2017, 10:31 AM

I've updated to gpgme-1.8.0 and tried again.
I have one mail that I'll call "bad" which gives me in mutt:

Content:

[-- Error: decryption failed: End of file --]

Status line:

S/MIME signature could NOT be verified.

And a second one I'll call "good" which gives me:

Content:

[-- Begin signature information --]
Error: verification failed: End of file
[-- End signature information --]

[-- The following data is signed --]

[-- Attachment #1 --]
[-- Type: multipart/alternative, Encoding: 7bit, Size: 60K --]
(actual mail body follows here)
...
[-- End of signed data --]

Status line:

S/MIME signature could NOT be verified.

I'll attach the traces and gpgsm logs.

wiz updated the task description. (Show Details)Apr 14 2017, 10:31 AM
wiz changed Version from 1.6.0 to 1.8.0.
wiz updated the task description. (Show Details)

Hi, I found a workaround for neomutt (see https://github.com/neomutt/neomutt/pull/662).

The basic problem is that gpgme treats a (gpgsm) protocol error as IO error and terminates the session; (neo)mutt first tries to decrypt the mail, and then tries to only verify it - but using the same gpgme context, which is already gone.

This is because libgpgme treats gpgsm "protocol" errors as IO errors; see status_handler() in engine-gpgsm.c:758: it returns the error as function result. "operational errors" should be returned through the io_cb_data struct (passed by reference) instead. The assuan-engine in engine-assuan.c:588 demonstrates the correct usage. IO errors lead to session termination through wait-private.c:119 and gpgsm_cancel() in engine-gpgsm.c:L196.

wiz added a comment.Jul 11 2017, 3:43 PM

I've since tried neomutt-20170707 which includes stbuehler's patch, but I see the same error cases as before.

I'm sorry; given the original error message

[-- Error: decryption failed: Invalid value passed to IPC --]

I thought it was the same problem I was having.

I just had a look at good.trace and it seems gpgsm --server exits instantly (chan_7 <- [eof]). The path seems to be correct though (/usr/pkg/bin/gpgsm), and /usr/pkg/bin/gpgsm --version reads (the first 79 bytes):

gpgsm (GnuPG) 2.1.18
libgcrypt 1.7.6
libksba 1.3.5
Copyright (C) 2017 Free Soft

The --version call passes the full path in argv[0] (but the full path is always passed as first argument to execv, so it shouldn't make a difference).

Sadly it seems there is no error message from gpgsm, and also the exit code isn't shown. Maybe you could try running gpgsm --server manually; it should greet you with OK GNU Privacy Guard's S/M server * ready. An strace log might provide more insight why gpgsm --server fails.

marcus added a subscriber: marcus.Aug 16 2017, 4:50 PM

This is probably broken since Werner enabled descriptor passing by default in 5090f6f24. The analysis in https://dev.gnupg.org/T2919#99901 is correct, but it's not enough to put the operational error in the right place. Also, the calls to _gpgme_wait_one have to be replaced by _gpgme_wait_one_ext. The change overall will be somewhat destabilizing.

marcus renamed this task from mutt + gpgme problems with some Outlook S/MIME emails to fix gpgme/gpgsm pipe server session with use_descriptor_passing (was: mutt + gpgme problems with some Outlook S/MIME emails).Aug 17 2017, 3:06 PM
wiz added a comment.Aug 29 2017, 12:59 PM

I just had a look at good.trace and it seems gpgsm --server exits instantly (chan_7 <- [eof]). The path seems to be correct though (/usr/pkg/bin/gpgsm), and /usr/pkg/bin/gpgsm --version reads (the first 79 bytes):

gpgsm (GnuPG) 2.1.18
libgcrypt 1.7.6
libksba 1.3.5
Copyright (C) 2017 Free Soft

The --version call passes the full path in argv[0] (but the full path is always passed as first argument to execv, so it shouldn't make a difference).

Sadly it seems there is no error message from gpgsm, and also the exit code isn't shown. Maybe you could try running gpgsm --server manually; it should greet you with OK GNU Privacy Guard's S/M server * ready. An strace log might provide more insight why gpgsm --server fails.

"good.trace" is a trace of a case where I like the result (i.e., the mail is displayed). "bad.trace" should be more interesting.

I think bad.trace is very similar in the errors (chan_9 instead of chan_7); the difference is probably that the "bad mail" is not using a detached signature (possibly even encrypted), so mutt cannot find the body without actually decoding the message through gpgsm; the "good mail " is using a detached signature, and the body is the first part of a multi-part message which mutt can decode itself; it still can't verify the signature.

wiz added a comment.Aug 29 2017, 11:01 PM

I think bad.trace is very similar in the errors (chan_9 instead of chan_7); the difference is probably that the "bad mail" is not using a detached signature (possibly even encrypted), so mutt cannot find the body without actually decoding the message through gpgsm; the "good mail " is using a detached signature, and the body is the first part of a multi-part message which mutt can decode itself; it still can't verify the signature.

Thanks for the explanation. gpgsm --server run on its own just sits there, so I'll wait for the descriptor passing fix...

This seems to be closely related to T4257 for which I have a fix under test. The problem is that we pass the fd used by the caller to create the data object to gpgsm and close that very fd. The descriptor passing involves an implicit dup so closing is in theory okay but we should not close an fd which has been set (w/o dup) by the caller.

The simplest fix for T4257 is to always use the extra pipe which we use for data objects which don't have an fd associated.