No. I won't maintain all that stuff again.

Yes I thought to use GnuTLS here.

The depedencies I see [1] are:
gmp -> No further depedencies
libgnurx -> No further dependencies
nettle -> depenendcy to gmp

Apart from that gettext and zlip which we already have.
So it should not be that hard to package. I really would like to get rid of it,
too but until then..

Would you accept a patch against gnupg to include GnuTLS 3.x in the Windows
installer?

1: https://github.com/mxe/mxe/blob/master/src/gnutls.mk

yes there are no remaining problems that I can see here.

Thanks -> resolved.

aheinecke: Did you had a chance to test this with 2.1.7 or master?

We can't use gnutls because it depends on too many extra libraries duplicating
functionality we already have in gnupg. ntbtls will be the way forward.
Unfortunately too many other tasks delayed its development.

yeah no, With the gnupg-w32 installer becoming part of gpgwin we really need
support for hkps in that installer. Yeah gnutls sucks but thats what we have.

I'll prepare a patch.

Pushed as 5e1a844. Thanks.

Ok now I found kbxutil and learned about ephemeral certificates (Yep reading
helps) ;-)

After the first import kbxutil lists the Root certificate three times.
Twice with ephemeral flags, once without. So gpgsm -k shows it only once. But
kbxutil --find-dups already lists those duplicates.

fpr=11:B9:1B:31:EE:09:E0:84:4D:25:4E:58:7A:65:CE:51:84:F3:6B:70 recno=5 7 8
fpr=98:2D:D4:1D:BE:91:EE:72:B3:B8:43:33:F2:21:F7:74:64:39:08:7E recno=2 4 6

Now after the verify gpgsm takes the first of those certificates and unsets the
ephemeral flag as it was used as part of a complete trustchain. (sm/certchain.c:

If the first certificate was ephemeral we now have two certificates that are not
ephemeral but are the same and gpgsm -k shows both.

My solution is to check in keydb_store_cert for ephemeral certificates and
instead of inserting those again without the ephemeral flag to remove the
ephemeral flag of the existing certificate.

It's still unclear to me though why there were three certificates (Two ephemeral
and one normal) I would have expected one ephemeral and one normal certificate.

Patch attached.

D287: 648_0001-sm-Fix-cert-storage-for-ephemeral-certs.patch

I've tested this again and again the problem was no longer visible.

So I ran the following script for some time:

    export GNUPGHOME=$(mktemp -d)
    echo 11B91B31EE09E0844D254E587A65CE5184F36B70 S > $GNUPGHOME/trustlist.txt
    echo disable-crl-checks > $GNUPGHOME/gpgsm.conf
    gpgsm --import aheinecke.der > /dev/null 2>&1
    gpgsm --verify testsig > /dev/null 2>&1
    if [ $(gpgsm -k | grep 0x84F36B70 | wc -l) = "2" ]; then
        echo bug >> bugs
        echo bug
    else
        echo nobug >> nobugs
        echo nobug
    fi
    rm -r "$GNUPGHOME"

This resulted in 85 "bug" and 31 "nobug" entries. Entries were also always in a
row. Like 10 "nobug" followed by 30 "bug" situations and then again 5 "nobug".

Probably related to system I/O.

Werner do you need me to provide more information here or can you reproduce this?

The last fix was wrong and the reaper thread closed a reader object which was
still used by another thread. Fixed with commit c971983:

I assumed that the log_fd also has a reader object but that reader
object is used for stdout and needs to be closed by the consumer.

The real bug with the non-released ldap_wrapper control objects was
that when looping over distribution points we did not closed the used
reader object before the next iteration.  Now, the test case had more
than one DP and thus we lost one reader object.

amd64
libgcrypt 1.6.2
libksba 1.3.4-beta1

Btw. If I roll back your commit the crashes no longer happen.

As an additional note. From checking why dirmngr takes so long in my setup I
know that I have several certificates in my keyring where the CRL is not
available. Maybe thats part of the problem.

x86 or amd64 ?
libgcrypt version?
libksba version?

In valgrind it did not crash. The keylisting exited normally. But showed several
errors.

Attached is the valigrind log.

Can you start dirmngr under valgrind?

gpgconf --kill dirmngr
valgrind --log-file=vg.log dirmngr --daemon --homedir /my/gnupg/home/dir

I've compiled current master and it works for the testcase. But when I start
kleopatra and it runs the keylist/verify dirmngr now crashes.

Can be triggered with gpgme: run-keylist --validate --cms

It crashes at different points but it never gets through all my certificates.

An example of the debug output that is collected before it crashes (differs
between crashes):

2015-06-16 19:09:15 dirmngr[9303.1] no CRL available for issuer id
18F071EAAC08885C9434A7DE1DB3AFC30F27DD32
2015-06-16 19:09:15 dirmngr[9303.1] DBG: chan_1 -> INQUIRE SENDCERT
2015-06-16 19:09:15 dirmngr[9303.1] DBG: chan_1 <- [ 44 20 30 82 04 7d 30 82 03
65 a0 03 02 01 02 02 ...(982 byte(s) skipped) ]
2015-06-16 19:09:15 dirmngr[9303.1] DBG: chan_1 <- [ 44 20 14 34 6d f5 07 c2 04
86 4a ba a1 71 50 b0 ...(187 byte(s) skipped) ]
2015-06-16 19:09:15 dirmngr[9303.1] DBG: chan_1 <- END
2015-06-16 19:09:15 dirmngr[9303.1] checking distribution points
2015-06-16 19:09:15 dirmngr[9303.1] no distribution point - trying issuer name
2015-06-16 19:09:15 dirmngr[9303.1] fetching CRL from default location
2015-06-16 19:09:15 dirmngr[9303.1] ldap wrapper 10199 started (reader
0x00007f6a580337a0)
2015-06-16 19:09:15 dirmngr[9303.0] ldap wrapper 10198 ready: exitcode=1
2015-06-16 19:09:15 dirmngr[9303.0] ldap worker stati:
2015-06-16 19:09:15 dirmngr[9303.0] c=0x00007f6a58033740 pid=10199/10199
rdr=0x00007f6a580337a0 ctrl=0x00007f6a580008c0/1 la=1434474555 rdy=0
2015-06-16 19:09:15 dirmngr[9303.0] c=0x00007f6a58022520 pid=-1/10198
rdr=0x0000000000000000 ctrl=0x0000000000000000/0 la=1434474554 rdy=1
2015-06-16 19:09:15 dirmngr[9303.0] dirmngr_ldap[10199]: processing url 'ldap://'
2015-06-16 19:09:15 dirmngr[9303.0] dirmngr_ldap[10199]: host
'directory.verisign.com'
2015-06-16 19:09:15 dirmngr[9303.0] dirmngr_ldap[10199]: port 389
2015-06-16 19:09:15 dirmngr[9303.0] dirmngr_ldap[10198]: processing url 'ldap://'
2015-06-16 19:09:15 dirmngr[9303.0] dirmngr_ldap[10198]: host
'directory.verisign.com'
2015-06-16 19:09:15 dirmngr[9303.0] dirmngr_ldap[10198]: port 389
2015-06-16 19:09:15 dirmngr[9303.0] dirmngr_ldap[10199]: DN
'1.2.840.113549.1.9.1=#4865696E65636B656E40676D61696C2E636F6D,CN=Common
Name,ST=Some-State,C=DE'
2015-06-16 19:09:15 dirmngr[9303.0] dirmngr_ldap[10199]: filter
'objectClass=*'
2015-06-16 19:09:15 dirmngr[9303.0] dirmngr_ldap[10199]: attr
'certificateRevocationList'
2015-06-16 19:09:15 dirmngr[9303.0] dirmngr_ldap[10198]: DN
'1.2.840.113549.1.9.1=#4865696E65636B656E40676D61696C2E636F6D,CN=Common
Name,ST=Some-State,C=DE'
2015-06-16 19:09:15 dirmngr[9303.0] dirmngr_ldap[10198]: filter
'objectClass=*'
2015-06-16 19:09:15 dirmngr[9303.0] dirmngr_ldap[10198]: attr
'certificateRevocationList'
2015-06-16 19:09:15 dirmngr[9303.0] dirmngr_ldap[10198]: searching 'ldap://'
failed: No such object
2015-06-16 19:09:15 dirmngr[9303.0] ldap worker stati:
2015-06-16 19:09:15 dirmngr[9303.0] c=0x00007f6a58033740 pid=10199/10199
rdr=0x00007f6a580337a0 ctrl=0x00007f6a580008c0/1 la=1434474555 rdy=0
2015-06-16 19:09:15 dirmngr[9303.0] c=0x00007f6a58022520 pid=-1/10198
rdr=0x0000000000000000 ctrl=0x0000000000000000/0 la=1434474555 rdy=1
2015-06-16 19:09:15 dirmngr[9303.0] dirmngr_ldap[10199]: searching 'ldap://'
failed: No such object
2015-06-16 19:09:15 dirmngr[9303.0] ldap wrapper 10199 ready: exitcode=1
2015-06-16 19:09:15 dirmngr[9303.0] ldap worker stati:
2015-06-16 19:09:15 dirmngr[9303.0] c=0x00007f6a58033740 pid=-1/10199
rdr=0x0000000000000000 ctrl=0x00007f6a580008c0/1 la=1434474555 rdy=1

Backtrace for this log (also differs):

#0 0x00007f6a69109cc9 in GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f6a6910d0d8 in GI_abort () at abort.c:89
#2 0x00007f6a69146394 in __libc_message (do_abort=do_abort@entry=1,

fmt=fmt@entry=0x7f6a69254b28 "*** Error in `%s': %s: 0x%s ***\n") at

../sysdeps/posix/libc_fatal.c:175
#3 0x00007f6a6915266e in malloc_printerr (ptr=<optimized out>,
str=0x7f6a69254cf0 "double free or corruption (fasttop)", action=1)

at malloc.c:4996

#4 _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3840
#5 0x00007f6a69d4fd2d in ?? () from /opt/gnupg/lib/libgcrypt.so.20
#6 0x0000000000428802 in ldap_wrapper (ctrl=ctrl@entry=0x7f6a580008c0,
reader=reader@entry=0x7f6a64c05cd0,

argv=argv@entry=0x7f6a64c05a80) at ldap-wrapper.c:772

#7 0x000000000042218c in run_ldap_wrapper (ctrl=ctrl@entry=0x7f6a580008c0,
multi_mode=multi_mode@entry=0, proxy=0x0,

host=<optimized out>, port=<optimized out>, user=<optimized out>, pass=0x0, 
dn=dn@entry=0x7f6a58001d50

"1.2.840.113549.1.9.1=#4865696E65636B656E40676D61696C2E636F6D,CN=Common
Name,ST=Some-State,C=DE",

filter=filter@entry=0x44dcc1 "objectClass=*", attr=attr@entry=0x44b50a

"certificateRevocationList", url=url@entry=0x0,

reader=reader@entry=0x7f6a64c05cd0, ignore_timeout=0) at ldap.c:191

#8 0x00000000004228ea in attr_fetch_ldap (ctrl=0x7f6a580008c0,

dn=0x7f6a58001d50

"1.2.840.113549.1.9.1=#4865696E65636B656E40676D61696C2E636F6D,CN=Common
Name,ST=Some-State,C=DE",

attr=attr@entry=0x44b50a "certificateRevocationList",

reader=reader@entry=0x7f6a64c05cd0) at ldap.c:287
#9 0x0000000000414aed in crl_fetch_default (ctrl=ctrl@entry=0x7f6a580008c0,

issuer=issuer@entry=0x7f6a58001d50

"1.2.840.113549.1.9.1=#4865696E65636B656E40676D61696C2E636F6D,CN=Common
Name,ST=Some-State,C=DE", reader=reader@entry=0x7f6a64c05cd0) at crlfetch.c:319
#10 0x000000000041439d in crl_cache_reload_crl (ctrl=ctrl@entry=0x7f6a580008c0,
cert=0x7f6a58002740) at crlcache.c:2554
#11 0x000000000040e1d5 in inquire_cert_and_load_crl (ctx=0x7f6a58000950) at
server.c:589
#12 cmd_isvalid (ctx=0x7f6a58000950, line=<optimized out>) at server.c:901
#13 0x00007f6a6a23e96a in ?? () from /opt/gnupg/lib/libassuan.so.0
#14 0x00007f6a6a23ed49 in assuan_process () from /opt/gnupg/lib/libassuan.so.0
#15 0x000000000040edc7 in start_command_handler (fd=fd@entry=1) at server.c:2243
#16 0x000000000040ada5 in start_connection_thread (arg=arg@entry=0x1) at
dirmngr.c:1937
#17 0x00007f6a69908dbc in thread_start (startup_arg=<optimized out>) at npth.c:265
#18 0x00007f6a696f1182 in start_thread (arg=0x7f6a64c06700) at pthread_create.c:312
#19 0x00007f6a691cd47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

A different backtrace:

#0 0x00007fe9e9805cc9 in GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fe9e98090d8 in GI_abort () at abort.c:89
#2 0x00007fe9e9842394 in __libc_message (do_abort=do_abort@entry=1,

fmt=fmt@entry=0x7fe9e9950b28 "*** Error in `%s': %s: 0x%s ***\n") at

../sysdeps/posix/libc_fatal.c:175
#3 0x00007fe9e984dac2 in malloc_printerr (ptr=<optimized out>,
str=0x7fe9e994cbfc "corrupted double-linked list", action=1)

at malloc.c:4996

#4 malloc_consolidate (av=av@entry=0x7fe9d8000020) at malloc.c:4165
#5 0x00007fe9e984edf8 in _int_malloc (av=0x7fe9d8000020, bytes=1025) at
malloc.c:3423
#6 0x00007fe9e98517b0 in GI_libc_malloc (bytes=1025) at malloc.c:2891
#7 0x00007fe9ea44ad11 in ?? () from /opt/gnupg/lib/libgcrypt.so.20
#8 0x00007fe9ea44bc19 in ?? () from /opt/gnupg/lib/libgcrypt.so.20
#9 0x00007fe9ea93b284 in init_membuf (maxlen=0, initiallen=<optimized out>,
mb=0x7fe9e53018e0, ctx=0x7fe9d8000950)

at assuan-inquire.c:64

#10 assuan_inquire (ctx=ctx@entry=0x7fe9d8000950, keyword=keyword@entry=0x44774b
"SENDCERT",

r_buffer=r_buffer@entry=0x7fe9e5301d50,

r_length=r_length@entry=0x7fe9e5301d60, maxlen=maxlen@entry=0) at
assuan-inquire.c:169
#11 0x000000000040dfca in inquire_cert_and_load_crl (ctx=0x7fe9d8000950) at
server.c:567
#12 cmd_isvalid (ctx=0x7fe9d8000950, line=<optimized out>) at server.c:901
#13 0x00007fe9ea93a96a in dispatch_command (ctx=0x7fe9d8000950, line=<optimized
out>, linelen=<optimized out>)

at assuan-handler.c:675

#14 0x00007fe9ea93ad49 in process_request (ctx=0x7fe9d8000950) at
assuan-handler.c:871
#15 assuan_process (ctx=0x7fe9d8000950) at assuan-handler.c:894
#16 0x000000000040edc7 in start_command_handler (fd=fd@entry=6) at server.c:2243
#17 0x000000000040ada5 in start_connection_thread (arg=arg@entry=0x6) at
dirmngr.c:1937
#18 0x00007fe9ea004dbc in thread_start (startup_arg=<optimized out>) at npth.c:265
#19 0x00007fe9e9ded182 in start_thread (arg=0x7fe9e5302700) at pthread_create.c:312
#20 0x00007fe9e98c947d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Weel, your patch defeated the code to properly log all output from the ldap
helper and only then cleanup the wrappers. For the actual fix please see T1978.

Fixed with commit 685b782.

Duplicate of T1978

Well, there was the ldap-reaper thread still running and due to a bug in the log
output handing code it was not able to remove its context structures and thus it
kept on spinning.

Fixed with commit 685b782.

Fixed with commit 255dadd.

By killing I meant sending SIGTERM (15) through the kill command.

But
"gpgconf --kill dirmngr" also does not kill the dirmngr. Is this problem not
reproducible for you?

kill -9 kills it of course.

How do you kill dirmngr? Using "gpgconf --kill dirmngr" or by sending a signal

• which one?

Just to point this problem out again (still exists with current master of
course). The CRL checks during a normal start of kleopatra on my keyring leave
55 dirmngr zombies.

This problem is not really bad for me as I am using the attached Patch. Still
after 3 months I'd appreciate a reaction / review.

Can you let me know when you can take a look at this or was my assignment wrong
here? (If so please change it)

This is a pretty major bug imho that would leave our application servers
(without manual intervention) if we would deploy 2.1 in our company. As such it
is blocking our adoption of 2.1.

I would appreciate some kind of reaction / confirmation on this issue.

Yes, auto-detection in dirmngr-client would be great, thanks!

You need to use --pem:

  dirmngr-client -v --pem ~/tmp/google.pem

There is no auto-detection in dirmngr-client. If you think this is useful
please change Priority to "feature " and adjust the title.

During deployment of gnupg 2.1.3 this bug was still noticed and I can still
reproduce it with git master. -> back to chatting

I must have messed up the test in T1921 (aheinecke on Apr 08 2015, 04:36 PM / Roundup). Probably by using a different
sysconfig dir for that test. Apologies for that.

That is not a bug but due to non-supported certificate policy constraints.

If you want to ignore them as a workaround you may modify the function
unknown_criticals which you find in
gnupg/dirmngr/validate.c and gnupg/sm/validate.c. Add to the
"known" array the strings "2.5.29.36" and "2.5.29.54".

Here is error in dirmngr:

2015-04-22 09:23:41 dirmngr[3108.0] critical certificate extension 2.5.29.36 is
not supported
2015-04-22 09:23:41 dirmngr[3108.0] critical certificate extension 2.5.29.54 is
not supported
2015-04-22 09:23:41 dirmngr[3108.0] error checking validity of CRL issuer
certificate: Unsupported certificate
2015-04-22 09:23:41 dirmngr[3108.0] crl_parse_insert failed: Unsupported certificate
2015-04-22 09:23:41 dirmngr[3108.0] crl_cache_insert via DP failed: Unsupported
certificate
2015-04-22 09:23:41 dirmngr[3108.0] command 'ISVALID' failed: Unsupported
certificate
2015-04-22 09:23:41 dirmngr[3108.0] DBG: chan_0 -> ERR 167772263 Unsupported
certificate <Dirmngr>
2015-04-22 09:23:41 dirmngr[3108.0] DBG: chan_0 <- [eof]

I pushed a few commits which should solve that bug. This is the strategy:

1. gpg --keyserver-option http-proxy=HOST:PORT overrides all other
2. dirmngr --http-proxy=HOST:PORT overrides
3. envvar http_proxy but only if dirmngr's --honor-http-proxy is set

HOST:PORT
http://HOST:PORT
socks4://HOST:PORT

are valid ways to specify a proxy. I plan to add socks5h as well.

I would also like to see this.

Maybe --refresh-keys without arguments for "the entire keyring" should also ask
for a confirmation "This will leak your entire keyring to the keyserver and
possibly an attacker. Do you really want to do this? (y/N)", or "--yes".

...for PGP keyservers.

This is quite obvious in the code:

  err = http_open (&http,
                   post_cb? HTTP_REQ_POST : HTTP_REQ_GET,
                   request,
                   httphost,
                   /* fixme: AUTH */ NULL,
                   httpflags,
                   /* fixme: proxy*/ NULL,
                   session,
                   NULL,
                   /*FIXME curl->srvtag*/NULL);

thanks for opening this bug.

The original reporter was on 2.1.0.

It looks like I can confirm this on 2.1.3.

I can confirm that this is resolved in 2.1.3:

0 dkg@alice:~$ gpgconf --launch dirmngr
gpgconf: error running '/usr/bin/gpg-connect-agent': exit status 1
gpgconf: error running '/usr/bin/gpg-connect-agent --dirmngr NOP': General error
1 dkg@alice:~$

the error message isn't particularly helpful at directing the user to the source
of the problem (a bad dirmngr.conf), but at least it does return a non-zero
error code.

I would like to see this happen. It would be great if dirmngr could make
parcimonie obsolete, for example.

(should this be "category: dirmngr" instead of just adding it as a topic?)

This was fixed by:

http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=commit;h=6619ead2cfd2abcb95b66dc70622fdeef624fb8a

using the test described in T1921 (aheinecke on Mar 10 2015, 06:13 PM / Roundup) there are no longer duplicated certificates
shown after the verify command.

Thanks!

I can't reproduce this problem neither in our company setup nor on a vanilla debian.

I've placed the .der files in the correct directories
/var/lib/dirmngr/extra-certs and /etc/dirmngr/trusted-certs

gpgsm --import aheinecke.der

Dirmngr output shows that the LOOKUP Issuer and Intermediate -Cert are not not
found in the dirmngr cache and they are not imported into the keyring.

This is probably another bug that hid this issue in the past.