Fixed in 1.7 with gpgme_op_keysign.

I pushed Ueno's patches for gpgme. In particular
dee56820cabde60c43c9bf8281b8d411cb2ad644

I agree that this is a practical problem.

I think it would be the right thing.

I'm developing Schleuder, the OpenPGP-featuring mailing list manager.
If I'm receiving an empty list of public keys from GPGME I currently don't know
if there are no keys, or if the keyring couldn't be read. Thus I can't properly
decide what to do: try to fetch keys? I would run into the same problem when
trying to import them. Return an error message? Which one?

This has led repeatedly to confusion e.g. when people imported a key into a
schleuder-list's keyring in the shell as root, which results in changed
ownership of the keyring-files by gpg. Next Schleuder couldn't read the keyring
anymore and maybe refused operation because it couldn't verify any incoming
email any more — instead of giving a helpful error message that points to the
cause: lacking filesystem permissions.

Currently my only chance is to manually check the permissions of all files that
might be involved in an operation. That is working around a bug, in my eyes.

You mean you want something like EACCES instead of an empty listing? I am not
sure whether this is the right thing to do. Can please you describe your use case?

Oops; forgot to add the fix to 1.7.0

Do you have an idea when you would try to fix this? Within the next weeks or
rather months?

Right. Putting armor into gpg.conf is a very bad idea. Don't assume that. If
users want to shoot into their own foot, let them do so.

I see. So, how can I reliably know whether the output of my program is going to
be ascii or binary (to append either .asc or .gpg)? I should just assume that
'armor' will not be in gpg.conf?

If a user forces all output of gpg to be armored, it is not gpgme's task to
revert this.

Putting armor into gpg.conf is as bad an idea as putting encrypt into gpg.conf.

gpgme_get_armor() returns gpgme's current state.

Thanks for testing.

I've made new container and can't repeat the bug. gpgme
components got updated in Fedora.

Added support for the newly added size-hint to gpgmepp and kleopatra. Works
nicely, although progress could update a bit more often for my taste but its soo
much better then the old "no progress at all" that I don't want to complain ;-)

Thanks!

I do not agree, but let me see what we can. A new --input-size-hint might be an
option.

--set-filesize is used for an entirely different purpose - That it is also used
in progress is only related tothat other purpose (pre-generated OpenPGP packets
as input)

But is there any problem using it this way? I didn't see one.

If you know the file size in advance, you can pass the entire file to gpgme and
there is no need to use a pipeline.

GpgME++ provides an interface for a DataProvider class. While this is of course
inherited from Marc I find this kinda nice to use in code. If you implement the
DataProvider interface you can use that as input for GpgME.
QGpgME then provides a DataProvider for QIODevice based classes. E.g. A QFile, a
QBuffer etc. that way you can easily pass a QFile or QByteArray or a QString to
GpgME and get it encrypted. Very convenient API.

If you want a pipeline/stremaing, gpg won't
assume anything about the input file size. telling it that size is an ugly hack.

But e.g. I have a 1GB Mail I wish to decrypt, that is stored in some internal
format I know the size, but you are sugessting that I should cut out the
encrypted / signed part, save it to files and then pass the files to gpgme?

Mean while I added a Units arg to the PROGRESS status line, so that we can do
further tweaking in gpgme and won't need to mess with gnupg. Maybe we can
eventually find a solution which affects only gpgme.

Yes I've seen that and I think it might be useful but it does not solve the
problem that gnupg is not statusing the total for callback / piped operations.

If you really want a percent indication, why not using a file watcher in Kleo
and stat(2) the file.

Yes, that is what I meant by:

msg8719:

I could work around the problem in
Kleopatra by just assuming for files > 1MiB the progress is always scaled and
live with a slight jump after MiB.
And then calculate progress based on the Input size (as total) Kleopatra knows.

But that would be a workaround for "GpgME does not let me do what I want to do"
Maybe we need an "gpgme_add_engine_cmd_line_args" hack interface in GpgME to
give callers the flexibility to do add arbitrary arguments to gpgme engine
calls. This could probably reduce the "GpgME is a hindrance" perception.

Maybe we should phone about this.

Done in master.

version.texi is such an annoyance. I'll take care of it and will replace it
with a different annoying thing.

--set-filesize is used for an entirely different purpose - That it is also used
in progress is only related tothat other purpose (pre-generated OpenPGP packets
as input)

If you know the file size in advance, you can pass the entire file to gpgme and
there is no need to use a pipeline. If you want a pipeline/stremaing, gpg won't
assume anything about the input file size. telling it that size is an ugly hack.

Mean while I added a Units arg to the PROGRESS status line, so that we can do
further tweaking in gpgme and won't need to mess with gnupg. Maybe we can
eventually find a solution which affects only gpgme.

If you really want a percent indication, why not using a file watcher in Kleo
and stat(2) the file.

PNGs are noe rejected.

All not good.

To be honest I'm a bit pigheaded here. I could work around the problem in
Kleopatra by just assuming for files > 1MiB the progress is always scaled and
live with a slight jump after MiB.
And then calculate progress based on the Input size (as total) Kleopatra knows.

The Problem for me is that QGpgME will never emit current + total progress
because it always provides Data through callbacks. And GpgME++ also is pretty
much designed for this in the Dataprovider interface. I dislike maintaining half
working / weird behaving code so I looked into possible ways to fix that.

What I did then was to take a look at gnupg's progress code and saw that total
is modified by --set-filesize. So I thought "awesome there is a mechanism to
provide gnupg with the total filesize even if callbacks are used" and did that.
I still think that this is great, and a good solution (no changes to gnupg
required etc.).

You try to do something which does not make sense. would have exact numbers
they do not tell you anything valid. It might be that
large parts of the file are compressed into just a few bytes and thus your
progressbar makes a huge leap at one time and later it gets slow again despite
that these are only a few 100 MiB (compared to the 10 GiB or zeroes).

I'm not trying to have a 100% reliable progress or a second exact estimate of
when a job is finished. But I want to show some general information "Ok the task
is 90% done, just stay tuned a bit longer"

This is User Interface basics. If you have a long running task (and crypo tasks
can easily run into minutes / hours) show _some_ progress indication. Due to the
pecularities / bugs of the API Kleopatra just shows "I'm working". This is very
bad User Interface and I would like to fix that. And Ideally my fix for this
would be where the Problem happens and not a workaround for the problem in the
user interface.

Fixed with commit b5e16b0

All not good. You try to do something which does not make sense. Even if you
would have exact numbers they do not tell you anything valid. It might be that
large parts of the file are compressed into just a few bytes and thus your
progressbar makes a huge leap at one time and later it gets slow again despite
that these are only a few 100 MiB (compared to the 10 GiB or zeroes).

Always returning KiB would work for me as a compromise I don't know otherwise
that the switch from Bytes to KiB happened because I have no total.

Makes the code for QGpgME / GpgMEpp users more complicated though as they need a
mapping of progress to input file size. With the --set-filesize patch It would
be nicer as I could just handle this generically in GpgMEpp if an input is
seekable It would provide GnuPG with the size information and afterwards we have
progress where current and total could be used for relative progress calculation:

if (dp->isSupported(DataProvider::Seek)) {

off_t size = seek(0, SEEK_END);
seek(0, SEEK_SET);
gpgme_data_set_file_size(d->data, static_cast<unsigned long long>(size));

}

What do you think of changing GnuPG's PROGRESS interface to always return KiB
and cap that value before it overflows? We would also cap in GPGME in case
gpgme is a 32 bit application and gnupg is 64 bit (or Windows).

I changed this ussie to a feature request.

Debian's codesearch shows that gpgme_op_assuan_transact is only used by gpa and
a configure test in kdelibpim for its own copy of gpgme. In gpa it is harmless
to enable this. The only effect is that a status line callback will see a
status keyword "#" and status callbacks should always ignore unknown status lines.

Let's enable it by default.

This was already mentioned in T2360 so let's not clutter the tracker.
Resolved as duplicate.

Duplicate of T2360

Fixed in 135185b7.

Ok, there are no significant patches on top of pygpgme. Note that pygpgme is not really
maintained, and that we neither develop nor support pygpgme. But seeing that dnf is important to
Fedora, let's figure this out.

It would be nice if you could try to reproduce the problem without pygpgme though, just to make a
more minimal test case. I see the exception is thrown during some import. This is how I strace
gnupg to see what ioctls it is issuing:

% strace -eioctl g10/gpg --import ../tests/openpgp/samplekeys/ecc-sample-1-pub.asc
gpg: NOTE: THIS IS A DEVELOPMENT VERSION!
gpg: It is only intended for test purposes and should NOT be
gpg: used in a production environment or with production keys!
gpg: key 0BA52DF0BAA59D9C: public key "ec_dsa_dh_256 <openpgp@brainhub.org>" imported

• SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=26716, si_uid=1000, si_status=0,

si_utime=0, si_stime=0} ---
ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon
echo ...}) = 0
ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B38400 opost isig icanon
echo ...}) = 0
gpg: Total number processed: 1
gpg: imported: 1
+++ exited with 0 +++

Note that if you try to strace your gpgme-based application, you need to pass '-f' to strace to
follow forks.

I have grepped through gpgme and gnupg, and it looks like gnupg is only doing ioctls to terminals,
so maybe your container setup is doing something funny to terminals. But let's see what the strace
shows.

Here is the info about Fedora patches
https://www.rpmfind.net/linux/RPM/fedora/secondary/devel/rawhide/src/p/pygpgme-0.3-15.fc24.src.html

On Wed, Jul 27, 2016 at 1:24 PM, Justus Winter via BTS
<gnupg@bugs.g10code.com> wrote:

I see that you are using pygpgme, is that correct?If so, which version, and are
there significant patches applied in the Fedora package? And can you please tell
me what version of libgpgme you are using?

Thanks for the report.

I see that you are using pygpgme, is that correct? If so, which version, and are
there significant patches applied in the Fedora package? And can you please tell
me what version of libgpgme you are using?

Let's try to figure out which ioctl fails. Could you try to strace this process?

While the detection works now to distinguish between PGP and S/MIME data it
might be more robust if it would do some more sanity checking on the packet.

E.g. PNG Graphics are detected as PGP Signatures because they start with 0x89

But this is not super neccessary as for the use case of file extension support
valid data will be detected correctly.

This is a GnuPG problem:

teythoon@europa ~/repos/g10/gpgme/obj/tests (git)-[master] % rm $GNUPGHOME/tofu.db && ( gpg2 --verify --with-
colons --status-fd=1 /tmp/testmsg )2>/dev/null | grep TOFU_STATS
[GNUPG:] TOFU_STATS 1 0 0 auto
[GNUPG:] TOFU_STATS_LONG Verified 0 messages signed by "Alfa Test (demo key) <alfa@example.net>".
[GNUPG:] TOFU_STATS 1 0 0 none
[GNUPG:] TOFU_STATS 1 0 0 none
teythoon@europa ~/repos/g10/gpgme/obj/tests (git)-[master] % rm $GNUPGHOME/tofu.db && ( gpg2 --verify --with-
colons --status-fd=1 /tmp/testmsg && gpg2 --verify --with-colons --status-fd=1 /tmp/testmsg && sleep 1 && gpg2 -
-verify --with-colons --status-fd=1 /tmp/testmsg )2>/dev/null | grep TOFU_STATS
[GNUPG:] TOFU_STATS 1 0 0 auto
[GNUPG:] TOFU_STATS_LONG Verified 0 messages signed by "Alfa Test (demo key) <alfa@example.net>".
[GNUPG:] TOFU_STATS 1 0 0 none
[GNUPG:] TOFU_STATS 1 0 0 none
[GNUPG:] TOFU_STATS 1 0 0 none
[GNUPG:] TOFU_STATS 1 0 0 none
[GNUPG:] TOFU_STATS 1 0 0 none
[GNUPG:] TOFU_STATS 2 1 0 auto 1 1
[GNUPG:] TOFU_STATS_LONG Verified 1 message signed by "Alfa Test (demo key) <alfa@example.net>"%0Ain the past
1~second.
[GNUPG:] TOFU_STATS 2 1 0 auto 1 1
[GNUPG:] TOFU_STATS_LONG Verified 1 message signed by "Alpha Test (demo key) <alpha@example.net>"%0Ain the past
1~second.
[GNUPG:] TOFU_STATS 2 1 0 auto 1 1
[GNUPG:] TOFU_STATS_LONG Verified 1 message signed by "Alice (demo key)"%0Ain the past 1~second.

I agree, these are problems we should address, they might be the symptoms of a
race somewhere. I bet they are in GnuPG though.

D369: 860_0001-Core-Add-file_size-support-to-gpgme_data.patch

Since Kleopatra is using data callbacks the total is always 0 so I can't use the
way to calculate percent.

Previously kleopatra used the filesize as total value. This does not work if
total is always 0 and the progress switches based on the current file size. E.g
for a large file the prgress decreases after 1024*1024 bytes have been processed.

I could probably add some weird "if gnupg > 2.1.14 and the file size is >
1024*1024 and the progress is < 1024*1024 expect it to be bytes and otherwise
expect it to be kilobytes." But this is not nice to use API.

My attached patch solves this by giving data callback users the opportunity to
provide GnuPG with the information how much input size it can expect. This makes
total / current workable from the start and everything is fine.

But as we jabbered about you do not like this patch :'-(

Problem not resolved for me as I think the weird handling currently imposed by
GnuPG is definitely not "Easy"

I think that the charset header in the armor is not a good idea. In fact gpg
does not consider it at all. The armor headers are not protected and thus they
should not not chnage the semantics of the encrypted message. There is also no
way to keep this information after removing the armor or to re-create the header
from a binary message.

I consider a new flag for the Literal Data Packet to indicate theat the content
is a MIME message to be better. Standard MIME methods can then be used to
describe the content. Right now we only have an 'u' flag to indicate UTF-8
encoding (which to some interpretation of OpenPGP is anyway the default).
An 'm' flag would make it explicit that the content is MIME encoded and there
would be no more need to derive that info from the context.

I also created a set of examples messages. They are in
gnupg/tests/openpgp/samplemsgs/

interested in this, too. Especially since you always tell me assuan is so easy
to use because it's all in gpgme :-)

Fix for the difference in detection of armored vs. binary detached signatures
was trivial so I've pushed it with rev. 570bf2a

Looks good to me know. I'll start using it in Kleopatra and we will see what
breaks :-)

Have not tested different S/MIME messages yet.

Thanks.

I've created some examples to test it. They are all done with alfa@example.com
test key. Found an issue through that.

-ba (detached ascii armored signature) is detected as PGP-Signed and not as
PGP-Signature.

examples/plain.txt.asc: PGP-signed

A discussion about KMail handling PGP/Inline encodings [1] also makes me wonder
if data_identfiy should also return the output charset of text messages if it is
provided in the Armor Header. Afaik there is currently no API in gpgme to check
this and semantically It would make sense to me to parse this in identify, too.
But this is more of a question wether or not you think it makes sense to do this
directly. I'm not strongly opinionated about this.

1: https://bugs.kde.org/show_bug.cgi?id=248058

Done with commit cf37a57.
Note that only the first 2k are inspected.

Binary PGP messages are now detected. Not well tested but the run-verify test
tool can help to check the feature.

I need to implement a partial OpenPGP parser to decide what a certain PGP
Message is about and also bot all binary OpenPGP files. Please have some patience.