Seems it was a kernel / USB bug

In 3ef0938cfd8637e9801369f142eb8dd564f2ca61 --allow-loopback-pinentry became the default.

Fixed in 2.1.21.

Pushed: rG116cfd60779f: g10: invalidate the fd cache for keyring.

Cool. Thanks for your work here. Where would I apply this patch, or should I just wait until you guys have it fixed?

Thanks a lot!

Thank you for additional info.
gpg --recv-keys can fail when we have network problem or dirmngr doesn't work well.
I think that the failure of your original report is that it goes something wrong when it merge keys into existing keys.
It helps me if you have the pubring.gpg BEFORE you invoked "pacman-key --refresh-keys".

I went through and was receiving keys individually just to see if it would work, and all of them work, except the:

Odd. I used the pubring.gpg you uploaded.
Refresh-keys successfully retrieve keys like:

That is the one I uploaded...

Thanks. But it's wrong keyring, I suppose. What we need is not your own public keyring, but the public keyring which pacman uses.
IIUC, please upload the one in /etc/pacman.d/gnupg.

I tried what you listed above and it worked, just like you said. I have uploaded my public keyring to look at. But other users are having this problem as well. Thanks.

Hi Neal, you are right about the entropy. I tought it was gpg but I think it's
because my system is too minimal to produce enough entropy. I finally decided to
generate my keys in another machine and transfer them to my minimal
installation. Now it works perfectly, with pinentry 0.9.5.

Thanks for your help,

Regards,

Felix

I can't reproduce this. I'm using pinentry 0.9.5 and GnuPG from git. When I
generate a key, it talks nearly 3 minutes for GnuPG to gather the required
amount of entropy, but it eventually returns. Attaching to gpg-agent using gdb,
it appears that gpg-agent is "suck" in the generate key function:

  #9  0x00007f13a08da9ce in ?? () from /lib/x86_64-linux-gnu/libgcrypt.so.20
  (gdb) 
  #10 0x00007f13a08ca2db in gcry_pk_genkey ()
     from /lib/x86_64-linux-gnu/libgcrypt.so.20
  (gdb) 
  #11 0x000000000041f51f in agent_genkey (ctrl=0x1b69e80, cache_nonce=0x0, 
      keyparam=0x7f1398001e70 "(genkey(rsa(nbits 4:1024)))", keyparamlen=27, 
      no_protection=0, override_passphrase=0x0, preset=0, outbuf=0x7f139fccfdb0)
      at ../../../gnupg/agent/genkey.c:479
  479	  rc = gcry_pk_genkey (&s_key, s_keyparam );

So, I seriously doubt that this is a problem with pinentry. And also I doubt
that it is a problem with GnuPG. Most likely, you need to wait for the system
to generate more entropy.

If you think gpg or gpg-agent is really hung, it would be nice if you could use
gdb to attach and then get a backtrace and post that here.

Thanks!

Neal

Here is the content of my gpg-agent.conf:
debug-pinentry
log-file /home/fxleblanc/gpg-errors.txt
pinentry-program /usr/bin/pinentry-curses

Here is the content of my log file:
gpg: reading options from '/home/fxleblanc/.gnupg/gpg.conf'
gpg (GnuPG) 2.1.6; Copyright (C) 2015 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

gpg: enabled debug flags: packet mpi crypto filter iobuf memory cache memstat
trust hashing cardio ipc clock lookup extprog

gpg: signal Interrupt caught ... exiting

I interrupted the program because it wasn't doing anything after I entered my
passphrase.

--debug-pinentry is an option for the gpg-agent. Thus put the line

debug-pinentry

into gpg-agent.conf and make sure that there is also a log-file option.

I compiled pinentry version 0.9.5 and tried to regenerate my keys. The good news
is that the curses window appeared and I could enter my passphrase. The bad news
is that after I entered the passphrase(with the repetition), the program
freezed, not returning any prompt and not giving any sign of life(I checked with
top to be sure and nothing).

Also, when I tried to use the --debug-pinentry, gpg didn't recognize it as a
valid argument. I use gpg 2.1.6.

Please update to pinentry 0.95 and try again. You may also use the gpg-agent
option --debug-pinentry which shows the communication between gpg-agent and
pinentry.

pinentry-curses 0.9.1

Which pinentry version are you using?

How would the suggested method of programmatically using gpg be?

I'm maintaining a service that uses gpg as a streaming encryption/decryption
backend and we need to provide the passphrase for the keys somehow in a
convenient manner.

Priming the agent is not optimal too because it would force me to restart the
agent every time i add new keys.

Maybe give me the possibility to provide new passphrases to the agent via the
agent socket?

In another message (<874mnnlqxn.fsf@alice.fifthhorseman.net>) DKG notes that we
shouldn't allow loopback mode or preseeding to prevent passphrase-guessing attacks.

As DKG noted on the mailing list, we --batch shouldn't imply
--pinentry-mode=loop. He provides the example of a graphical tool that fills in
many of the fields when generating a key, but should not have to worry about
securely managing the password.

This suggests that if a password is somehow provided on the command line, we
should prime (i.e., preset) gpg agent so that it doesn't request a password.

This seems to work with gnupg-2.1.4. Thanks!

Can you please try with the latest version (2.1.4 will be released tomorrow)

I checked the code and the behavior. It is confirmed that the default of
gpg-agent disables loopback-pinentry mode and user needs to enable it.

Now, we need some fixes/improvements:
(1) gpg should automatically work with gpg-agent with the option of --passphrase
(-file, -fd).
In GnuPG 2.1.x, the secret keys are under control of gpg-agent and gpg frontend
should pass the passphrase to gpg-agent in some way.
When --passphrase (-file, -fd) option is supplied, gpg frontend could use
gpg-agent feature of either loopback-pinentry mode _OR_ preset_passphrase .
The latter requires specific key identification, so, loopback-pinentry mode
would be the solution for general.
(2) Both of loopback-pinentry mode and preset_passphrase are disabled as
default. We need to fix this default of gpg-agent _AND_ we need to fix gpg
frontend error handling of this case of disabled feature of gpg-agent. Well, I
don't know the reason this features need to be disabled...
(3) When it is gpg frontend which invokes gpg-agent, it would be natural to
enable loopback-pinentry (or preset_passphrase). But, there will be existing
gpg-agent even with --batch option. I don't know how it should work in this case.

It seems that the gpg-agent needs to be started with --allow-loopback-pinentry
for this to work.
Because I let gpg autostart the daemon for me, this does not get passed to
gpg-agent and therefore does not work when setting --pinentry-mode=loopback in gpg.

I don't know what is to do here.
Should gpg with --pinentry-mode=loopback autostart the gpg-agent with
--allow-loopback-pinentry ?
Or should I just add some documentation to the manpages to describe what is
necessary for --pinentry-mode=loopback and --passphrase-file to work?

It seems that your gpg-agent doesn't support loopback mode.
Either, your gpg-agent is from 2.0 or the socket is hijacked by gnome-keyring.
For the latter, please see http://wiki.gnupg.org/GnomeKeyring

When I try the following under gnupg 2.1.3 with arch linux:

$ gpg --homedir <gpg-dir> --batch --pinentry-mode=loopback --passphrase-file
<passfile> --decrypt myfile.gpg

I get the following error:

gpg: setting pinentry mode 'loopback' failed: Not supported
...
gpg: decryption failed: No secret key

Is the gnupg version of arch just missing some compile-time flag to support
--passphrase-file without manual pinentry? If this is the case, I could report
this back to the arch maintainer to get it fixed downstream.
Or is there still some work to be done on gnupg?

In GnuPG 2.1.x, secret key is under control of gpg-agent. You can use
--pinentry-mode=loopback.
But, I think that --batch should imply --pinentry-mode=loopback.