Sry I accidentally posted an incomplete message with T2812 (aheinecke on Oct 31 2016, 05:08 PM / Roundup) (I used itsalltext
and postet a wrong version).

I wanted to write:

On the command line it's looking good. The second keylist is also down to 5
seconds on Windows.

But used from gpgme it still takes about a minute. If you add --with-colons and
slow down system calls by using strace you can also see this on GNU/Linux:

~> time strace gpg2 --no-default-keyring \

--keyring /usr/share/keyrings/debian-keyring.gpg \
--no-auto-check-trustdb --trust-model pgp \
--with-colons -k >/dev/null 2>&1

2.26s user 0.40s system 102% cpu 2.601 total
~> time strace gpg2 --no-default-keyring \

--keyring /usr/share/keyrings/debian-keyring.gpg \
--no-auto-check-trustdb --trust-model tofu \
--with-colons -k >/dev/null 2>&1

21.43s user 24.47s system 108% cpu 42.451 total

On Windows it's:

PS C:\Users\aheinecke> Measure-Command -Expression { gpg --no-auto-check-trustdb
--trust-model tofu --list-keys > $null
}
TotalSeconds : 7.0945596

PS C:\Users\aheinecke> Measure-Command -Expression { gpg --no-auto-check-trustdb
--with-colons --trust-model tofu --lis
t-keys > $null }
TotalSeconds : 56.0914993

PS C:\Users\aheinecke> Measure-Command -Expression { gpg --no-auto-check-trustdb
--with-colons --trust-model pgp --list
-keys > $null }
TotalSeconds : 1.4855689

I'm also still seeing decryption blocked on Windows while a keylist
--with-colons runs.

I wonder if we should generally check out performance of reading the keyring on
Windows
mabye we could genrally improve it so that it's better cached by Windows.

No both have unknown trust.

Duplicate of T2341

Thanks for your report,

This was already fixed in T2341

Which is currently not yet released. I'm marking this issue here as released
with superseder (duplicate) to keep the tracker clean.

GpgOL is built with DEP and and ASLR now. Need to enable this for GpgEX and some
other parts of Gpg4win, too. So not yet fully resolved but I keep it in mind.

Fixed with: 5579c4b4f

The code was overcomplicated as it was based on a bad assumption about Outlook
which I never questioned myself. We now properly encrypt in the send event so no
need for ticklish threads / callbacks.

D387: 897_keylist-from-data.diff

This is resolved. we have WITH_TOFU_INFO now.

Under GNU/Linux you can compare the strace output to see that there is a problem
even if it's quick because it is cached:

~> time strace gpg2 --no-auto-check-trustdb --trust-model pgp -k 2>&1 |wc -l
33383
strace gpg2 --no-auto-check-trustdb --trust-model pgp -k 2>&1 1.04s user 0.45s
system 104% cpu 1.433 total
wc -l 0.02s user 0.16s system 12% cpu 1.433 total

~> time strace gpg2 --no-auto-check-trustdb --trust-model tofu -k 2>&1 |wc -l
558528
strace gpg2 --no-auto-check-trustdb --trust-model tofu -k 2>&1 9.60s user 8.47s
system 106% cpu 17.022 total
wc -l 0.60s user 2.34s system 17% cpu 17.022 total

This is with my normal pubring that contains 790 public keys.

I was unaware that the released version does not require it.

In that case it's no bug imo. because otherwise we would also need to work with
older autotools versions etc.

btw. reason for this report is a setup of WKS where you require most recent
modern gnupg on long time distro running servers.

What I meant by "KArchive" is that we already have all that nice archiving code
in Kleopatra already: https://api.kde.org/frameworks/karchive/html/index.html
To work with standard formats like tar / zip / 7zip etc.

This would get us the included platform abstraction through Qt for stuff like
filenames etc. and we wouldn't have to maintain our own implementations for
these archive formats.

https://www.gnu.org/software/tar/manual/html_section/tar_68.html gives a good
overview imo.

So yes raising the file name length limit could be problematic with
compatibility and we might have to change more in our implementation to create
formats of a different spec.

From the discussion in the forum it looks like the error was silently discarded
when used in Kleopatra. We need error handling in that case. So I think this is
an Urgent bug as silent discard of archive contents can lead to data loss. So
for me this part is an urgent bug. Actually handling longer filenames is another
issue.

As a sidenote:
Kleopatra already links KArchive for svgz handling so it already contains a good
API for ZIP file creation. I'd like to add that to Kleopatra and make it default
so that the default is not our own error prone tar implementation. (Other tar
implementations also are problematic for windows). In that case we could also
drop the extraction as zip file support is native in the windows file explorer.

And as suggested in the forum entry we should probably also document how to add
7zip support to kleopatra or check for this at runtime and add some 7zip archive
options if it is available.
This should be doable by editing libkleopatrarc but I'd have to check the syntax
myself in the code as its not documented afaik.

Thanks for clarifying this.

I've update the comment in the test accordingly. This issue is resolved for me.

I'm using latest master and I still can't export a secret key without passphrase.

And Justus also has not closed this bug or wrote that he commited something
more. So I think the 2.1.13 announcement was mistaken and this problem still
exists. (Or am I missing some option / need a different pinentry mode?)

Jochen: I'd rather you (manually) patch the dirmngr tarball included in
gpg4win-2 and create a testinstaller and try that one out.

I found the Problem in this issue and tested that the attached patch solves the
problem, yes It would have worked on GNU/Linux as the "b" has no effect there.
Finding out since when the problem existed appears moot to me and you would have
to check in dirmngr's SVN and likely always existed.

But maybe there are additional problems (as this is imo a very exotic feature)
so it would probably make sense to test it again on Windows before preparing the
next stable Gpg4win release.

Thanks, this works now as expected.

While enabling the checks for signcount in gpgme/lang/qt/tests/t-tofuinfo.cpp
I've noticed though that if I sign and verify the same plaintext twice
immediately after another the signcount is not incremented correctly.

In line 266 of that test. The call to signAndVerify leads to an Assert if you
remove the " World" part of the "Hello World" message.

Alternatively adding a QTest::qWait(1000); before that line also results in a
success.

You can trigger this also by modifing the strings in line 233ff to contain the
same message.

Not really important imo as this is a constructed problem. The main issue here
is resolved for me.

D381: 870_0001-dirmngr-Open-CRL-s-in-binary-mode.patch

Woops didn't want to submit the last message as I had already looked into it myself.

This was reproducible using libksba's t-crl-parse with our root ca's clr but not
with an example file lying next to it.

Turned out that t-crl-parse opened the file in text mode. Conversion errors then
caused an invalid (too large read). When switching to binary mode it worked as
expected.

Dirmngr used the same. I've tested that crl parsing worked with the attached patch.

Now I get:
dirmngr[780]: error fetching certificate by subject: Configuration error
dirmngr[780]: crl_parse_insert failed: Missing certificate

But I think that is a different error as I get the same one when trying to
import the CRL on an empty homedir and parsing works now.

2.1.11 is not in the latest beta. Should be 2.1.13.

For testing / reporting it is also better to download the latest version from
gnupg.org
https://gnupg.org/download/index.html

Fixed for the next version with 037a5a7ed

This could be a nasty one. The crash occurs after the data structure of the mail
was unloaded in outlook and GpgOL already completely detached it's event
handlers from the object and frees up the memory. GpgOL is not executing any
code when the crash occurs. That outlook blames GpgOL is likely because it jumps
into an invalid memory region that was allocated for GpgOL but is no longer
valid. This shouldn't happen though as we have already successfully unregistered
all our callbacks.

So I currently think that somehow when using send from explorer outlook through
some side effect / bug does a callback into GpgOL's event handling code which
was already destroyed. I'll try to confirm that theory on monday by keeping the
event handlers around after the unload event occurred.

Also does not appear that we are the first ones with that problem:
https://forum.kaspersky.com/index.php?showtopic=225375

:-/

Added support for the newly added size-hint to gpgmepp and kleopatra. Works
nicely, although progress could update a bit more often for my taste but its soo
much better then the old "no progress at all" that I don't want to complain ;-)

Thanks!

Thanks for the report. I am able to reproduce the problem.

Looking into it.

--set-filesize is used for an entirely different purpose - That it is also used
in progress is only related tothat other purpose (pre-generated OpenPGP packets
as input)

But is there any problem using it this way? I didn't see one.

If you know the file size in advance, you can pass the entire file to gpgme and
there is no need to use a pipeline.

GpgME++ provides an interface for a DataProvider class. While this is of course
inherited from Marc I find this kinda nice to use in code. If you implement the
DataProvider interface you can use that as input for GpgME.
QGpgME then provides a DataProvider for QIODevice based classes. E.g. A QFile, a
QBuffer etc. that way you can easily pass a QFile or QByteArray or a QString to
GpgME and get it encrypted. Very convenient API.

If you want a pipeline/stremaing, gpg won't
assume anything about the input file size. telling it that size is an ugly hack.

But e.g. I have a 1GB Mail I wish to decrypt, that is stored in some internal
format I know the size, but you are sugessting that I should cut out the
encrypted / signed part, save it to files and then pass the files to gpgme?

Mean while I added a Units arg to the PROGRESS status line, so that we can do
further tweaking in gpgme and won't need to mess with gnupg. Maybe we can
eventually find a solution which affects only gpgme.

Yes I've seen that and I think it might be useful but it does not solve the
problem that gnupg is not statusing the total for callback / piped operations.

If you really want a percent indication, why not using a file watcher in Kleo
and stat(2) the file.

Yes, that is what I meant by:

msg8719:

I could work around the problem in
Kleopatra by just assuming for files > 1MiB the progress is always scaled and
live with a slight jump after MiB.
And then calculate progress based on the Input size (as total) Kleopatra knows.

But that would be a workaround for "GpgME does not let me do what I want to do"
Maybe we need an "gpgme_add_engine_cmd_line_args" hack interface in GpgME to
give callers the flexibility to do add arbitrary arguments to gpgme engine
calls. This could probably reduce the "GpgME is a hindrance" perception.

Maybe we should phone about this.

All not good.

To be honest I'm a bit pigheaded here. I could work around the problem in
Kleopatra by just assuming for files > 1MiB the progress is always scaled and
live with a slight jump after MiB.
And then calculate progress based on the Input size (as total) Kleopatra knows.

The Problem for me is that QGpgME will never emit current + total progress
because it always provides Data through callbacks. And GpgME++ also is pretty
much designed for this in the Dataprovider interface. I dislike maintaining half
working / weird behaving code so I looked into possible ways to fix that.

What I did then was to take a look at gnupg's progress code and saw that total
is modified by --set-filesize. So I thought "awesome there is a mechanism to
provide gnupg with the total filesize even if callbacks are used" and did that.
I still think that this is great, and a good solution (no changes to gnupg
required etc.).

You try to do something which does not make sense. would have exact numbers
they do not tell you anything valid. It might be that
large parts of the file are compressed into just a few bytes and thus your
progressbar makes a huge leap at one time and later it gets slow again despite
that these are only a few 100 MiB (compared to the 10 GiB or zeroes).

I'm not trying to have a 100% reliable progress or a second exact estimate of
when a job is finished. But I want to show some general information "Ok the task
is 90% done, just stay tuned a bit longer"

This is User Interface basics. If you have a long running task (and crypo tasks
can easily run into minutes / hours) show _some_ progress indication. Due to the
pecularities / bugs of the API Kleopatra just shows "I'm working". This is very
bad User Interface and I would like to fix that. And Ideally my fix for this
would be where the Problem happens and not a workaround for the problem in the
user interface.

Always returning KiB would work for me as a compromise I don't know otherwise
that the switch from Bytes to KiB happened because I have no total.

Makes the code for QGpgME / GpgMEpp users more complicated though as they need a
mapping of progress to input file size. With the --set-filesize patch It would
be nicer as I could just handle this generically in GpgMEpp if an input is
seekable It would provide GnuPG with the size information and afterwards we have
progress where current and total could be used for relative progress calculation:

if (dp->isSupported(DataProvider::Seek)) {

off_t size = seek(0, SEEK_END);
seek(0, SEEK_SET);
gpgme_data_set_file_size(d->data, static_cast<unsigned long long>(size));

}

This was already mentioned in T2360 so let's not clutter the tracker.
Resolved as duplicate.

Duplicate of T2360

To piggyback something on this issue.

To quote T2359 (aheinecke on May 17 2016, 11:59 AM / Roundup):

e.g. an API to check which key: gpg -er aheinecke@intevation.de

I did not have groups on the radar for this. If a recipient is a group then
gnupg would use multiple keys in this command.

I think locate-keys would be a great mechanism to support this easily in MUAs.
When we change it that for a given mailbox only the single most valid Key is
returned we could also have the semantic that if then multiple Keys are returned
we have a group.

This is fixed now. Was a problem with an incorrect port from KTemporaryFile to
QTemporaryFile for qt5

While the detection works now to distinguish between PGP and S/MIME data it
might be more robust if it would do some more sanity checking on the packet.

E.g. PNG Graphics are detected as PGP Signatures because they start with 0x89

But this is not super neccessary as for the use case of file extension support
valid data will be detected correctly.

My problems are resolved. I have not encountered a problem since your last
fixes. Although I sometimes have to reenter pin so I think the errors still
occur occassionally but gnupg recovers.

Thanks.

D369: 860_0001-Core-Add-file_size-support-to-gpgme_data.patch

Since Kleopatra is using data callbacks the total is always 0 so I can't use the
way to calculate percent.

Previously kleopatra used the filesize as total value. This does not work if
total is always 0 and the progress switches based on the current file size. E.g
for a large file the prgress decreases after 1024*1024 bytes have been processed.

I could probably add some weird "if gnupg > 2.1.14 and the file size is >
1024*1024 and the progress is < 1024*1024 expect it to be bytes and otherwise
expect it to be kilobytes." But this is not nice to use API.

My attached patch solves this by giving data callback users the opportunity to
provide GnuPG with the information how much input size it can expect. This makes
total / current workable from the start and everything is fine.

But as we jabbered about you do not like this patch :'-(

Problem not resolved for me as I think the weird handling currently imposed by
GnuPG is definitely not "Easy"

interested in this, too. Especially since you always tell me assuan is so easy
to use because it's all in gpgme :-)

Hi,

There was a problem in Gpg4win-2.3.1 that GpgOL and GpgEX might be unable to
start Kleopatra or GPA. This problem has been fixed with 2.3.2

Thanks for your report. Please check if 2.3.2 does not solve your problem and
let us know if it still persists.

Regards,
Andre

Gpg4win 2.3.1 and 2.3.2 included 64 bit versions of gpgol.

Hi,
Thanks for your report. Could you please retest with 2.3.2 we've fixed an issue
where GpgOL had problems communicating with Kleopatra that could cause your
behavior although I'm not 100% sure as I can't reproduce your problem.

Thanks,
Andre

Thanks for your report. Sorry I missed this for 2.3.2.
I'll look into it for the next version.