Page MenuHome GnuPG
Create Task
Maniphest T6363

Add progress status output to gpgtar
Closed, ResolvedPublic
Actions

Description

gpgtar does not yet output progess indication. If can be partly done using

gpgtar --gpg-args --enable-progress-filer --status-fd N ....

but that is limited because gpg does not known the total size of data. However, gpgtar knows this because it first scans all files. The proposal is to emit progress status lines at the gpgtar site with information on the number of files processed and the their size. Frontends may then pick whatever they like.

Revisions and Commits

rG GnuPG
rGed9a420a221a gpgtar: Emit progress status lines in create mode.
rGf84264e8acf7 gpgtar: Emit progress status lines in create mode.

Related Objects

Event Timeline

• werner triaged this task as Normal priority.Jan 30 2023, 2:59 PM
• werner created this task.
• werner added a commit: rGf84264e8acf7: gpgtar: Emit progress status lines in create mode..Jan 30 2023, 3:28 PM
• werner added a comment.Jan 30 2023, 3:47 PM

How with --status-fd passed to gpgtar we will get these progress lines:

[GNUPG:] PROGRESS gpgtar c 5000 0
[GNUPG:] PROGRESS gpgtar c 10000 0
[...]
[GNUPG:] PROGRESS gpgtar c 50000 0
[GNUPG:] PROGRESS gpgtar c 0 50223
[GNUPG:] PROGRESS gpgtar s 0 6543 MiB
[GNUPG:] PROGRESS gpgtar c 100 50223
[...]
[GNUPG:] PROGRESS gpgtar s 6300 6543 MiB
[GNUPG:] PROGRESS gpgtar s 6400 6543 MiB
[GNUPG:] PROGRESS gpgtar c 44800 50223
[...]
[GNUPG:] PROGRESS gpgtar c 50210 50223
[GNUPG:] PROGRESS gpgtar s 6543 6543 MiB

The first lines with zero as last value are emitted during the initial scan and thus before starting to write. Followed by this are two lines with the expected numbers (50223 files with a total size of 6543 MiB). The writing takes place and the current values are updated. As of now in increments of 100 for the file count and in 100MiB for the total size. The last two lines give the actual written numbers. The file count is a bit less because a few files were not suitable for archiving.

When also enabling the progress filter of gpg, the status line from gpg are mixed in like here:

[GNUPG:] PROGRESS gpgtar s 100 6543 MiB
[GNUPG:] PROGRESS stdin ? 108224 0 KiB
[GNUPG:] PROGRESS stdin ? 127488 0 KiB

Due to compression and the tar format overhead gpg still does not known the final size of the file. gpg uses the name of the input file as first word; this will always be "stdin" here. Thus grepping for gpgatr and 'c' or 's' is sufficient to distinguish the two outputs.

• werner added a project: gpgme.Jan 30 2023, 4:01 PM
• werner moved this task from Backlog to WiP on the gnupg24 board.

I guess we need some gpgme support as well.

How to handle the extraction progress depends on how we can pass the size of the filename etc. Also gpgme related.

• werner moved this task from WiP to gnupg-2.4.1 on the gnupg24 board.Jan 31 2023, 12:35 PM
• werner edited projects, added gnupg24 (gnupg-2.4.1); removed gnupg24.
• werner moved this task from Backlog to WiP on the gnupg22 board.Mar 15 2023, 11:29 AM
• werner added a commit: rGed9a420a221a: gpgtar: Emit progress status lines in create mode..Mar 15 2023, 12:06 PM
• werner changed the task status from Open to Testing.Apr 5 2023, 12:11 PM
• werner moved this task from WiP to QA on the gnupg22 board.
• ebo closed this task as Resolved.Apr 6 2023, 9:04 AM
• ebo claimed this task.
• ebo moved this task from QA to gnupg-2.2.42 on the gnupg22 board.
• ebo edited projects, added gnupg22 (gnupg-2.2.42); removed gnupg22.
• werner mentioned this in T6454: Release GnuPG 2.4.1.Apr 28 2023, 2:45 PM
aheinecke added a subscriber: aheinecke.Jun 9 2023, 4:42 PM

btw. this does not work on the decrypting side because kleopatra there apparently does not provide a size hint. So it goes:

[4888] org.kde.pim.kleopatra: Task: "Entschlüsselung: test1-archive.tar.gpg ..." has no total progress set.
[4888] org.kde.pim.kleopatra: Collection Progress: 1 total: 1000
[4888] org.kde.pim.kleopatra: Collection Progress: 219 total: 1000
[4888] org.kde.pim.kleopatra: Collection Progress: 429 total: 1000
[4888] org.kde.pim.kleopatra: Collection Progress: 732 total: 1000
[4888] org.kde.pim.kleopatra: Collection Progress: 1000 total: 1000
[4888] org.kde.pim.kleopatra: Collection Progress: 1000 total: 1000
[4888] org.kde.pim.kleopatra: Collection Progress: 1000 total: 1000
[4888] org.kde.pim.kleopatra: Collection Progress: 1000 total: 1000
[4888] org.kde.pim.kleopatra: Collection Progress: 1000 total: 1000
[4888] org.kde.pim.kleopatra: Collection Progress: 1000 total: 1000

But I won't reopen that here as this would be a kleo issue to pass the size hint afaik. And T6530 would solve that anyway.

ikloecker added a subscriber: ikloecker.Jun 9 2023, 5:59 PM

For me this does work also when decrypting:

org.kde.pim.kleopatra: Task:  "Decrypting: src.tar.gpg..."  has no total progress set. 
org.kde.pim.kleopatra: Kleo::Crypto::DecryptVerifyTask(0x24b2550) setProgress 0 1032480
org.kde.pim.kleopatra: Kleo::Crypto::DecryptVerifyTask(0x24b2550) setProgress 167991 1032480
org.kde.pim.kleopatra: Collection Progress:  162  total:  1000
org.kde.pim.kleopatra: Kleo::Crypto::DecryptVerifyTask(0x24b2550) setProgress 315831 1032480
org.kde.pim.kleopatra: Collection Progress:  305  total:  1000
org.kde.pim.kleopatra: Kleo::Crypto::DecryptVerifyTask(0x24b2550) setProgress 538364 1032480
org.kde.pim.kleopatra: Collection Progress:  521  total:  1000
org.kde.pim.kleopatra: Kleo::Crypto::DecryptVerifyTask(0x24b2550) setProgress 734711 1032480
org.kde.pim.kleopatra: Collection Progress:  711  total:  1000
org.kde.pim.kleopatra: Kleo::Crypto::DecryptVerifyTask(0x24b2550) setProgress 972424 1032480
org.kde.pim.kleopatra: Collection Progress:  941  total:  1000
org.kde.pim.kleopatra: Kleo::Crypto::DecryptVerifyTask(0x24b2550) setProgress 1032480 1032480
org.kde.pim.kleopatra: Collection Progress:  1000  total:  1000

The additional log lines are only local, but I also see a progressing progress bar. And gpgme calls gpgtar with

2023-06-09 17:46:34 gpgme[29033.71d9]     _gpgme_io_spawn: check: argv[ 0] = gpgtar
[...]
2023-06-09 17:46:34 gpgme[29033.71d9]     _gpgme_io_spawn: check: argv[21] = --gpg-args
2023-06-09 17:46:34 gpgme[29033.71d9]     _gpgme_io_spawn: check: argv[22] = --input-size-hint=1057260135
aheinecke added a comment.Jun 9 2023, 8:31 PM

Mh, let me check where the size hint should come from and why it might be missing in my windows test. That is indeed strange then.

aheinecke added a comment.Jun 12 2023, 1:27 PM

Probably some issue with large files / integer overflow. I am testing on Windows with 32 bit.

For a 5gb file I get:

2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: enter: path=0x033ba808 path=c:\\GnuPG\\bin\\gpgtar.exe
2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: check: path=0x033ba808 argv[ 0] = gpgtar.exe
2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: check: path=0x033ba808 argv[ 1] = --batch
2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: check: path=0x033ba808 argv[ 2] = --status-fd
2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: check: path=0x033ba808 argv[ 3] = 1
2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: check: path=0x033ba808 argv[ 4] = --gpg-args
2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: check: path=0x033ba808 argv[ 5] = --no-tty
2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: check: path=0x033ba808 argv[ 6] = --gpg-args
2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: check: path=0x033ba808 argv[ 7] = --charset=utf8
2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: check: path=0x033ba808 argv[ 8] = --gpg-args
2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: check: path=0x033ba808 argv[ 9] = --enable-progress-filter
2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: check: path=0x033ba808 argv[10] = --gpg-args
2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: check: path=0x033ba808 argv[11] = --exit-on-status-write-err
2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: check: path=0x033ba808 argv[12] = --gpg-args
2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: check: path=0x033ba808 argv[13] = --ttyname=/dev/tty
2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: check: path=0x033ba808 argv[14] = --decrypt
2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: check: path=0x033ba808 argv[15] = --directory
2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: check: path=0x033ba808 argv[16] = C:/Users/Andre Heinecke/Ap
2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: check: path=0x033ba808 argv[17] = --
2023-06-12 13:15:20 gpgme[3640.c50]     _gpgme_io_spawn: check: path=0x033ba808 tmp_name = C:\\Users\\ANDREH~1\\AppDa
2023-06-12 13:15:20 gpgme[3640.c50]       gpgme:AllowSetForegroundWindow: call: called for pid 6636; result=0

While for a 50mb file I see:

pawn: enter: path=0x06f73be8 path=c:\\GnuPG\\bin\\gpgtar.exe
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[ 0] = gpgtar.exe
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[ 1] = --batch
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[ 2] = --status-fd
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[ 3] = 1
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[ 4] = --gpg-args
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[ 5] = --no-tty
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[ 6] = --gpg-args
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[ 7] = --charset=utf8
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[ 8] = --gpg-args
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[ 9] = --enable-progress-filter
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[10] = --gpg-args
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[11] = --exit-on-status-write-error
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[12] = --gpg-args
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[13] = --ttyname=/dev/tty
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[14] = --decrypt
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[15] = --directory
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[16] = C:/Users/Andre Heinecke/AppData/Local/Temp/kleopatra-quIYDw
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[17] = --gpg-args
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[18] = --input-size-hint=49404929
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 argv[19] = --
2023-06-12 13:25:27 gpgme[6492.11a8]     _gpgme_io_spawn: check: path=0x06f73be8 tmp_name = C:\\Users\\ANDREH~1\\AppData\\Local\\Temp\\gpgme-1RWrOf

Both files were created the same way with Kleopatra and live in the same folder.

ikloecker added a comment.Jun 12 2023, 3:00 PM

Yeah, probably a Windows/MinGW 32-bit problem. GpgME::Data does

off_t size = seek(0, SEEK_END);
seek(0, SEEK_SET);
std::string sizestr = std::to_string(size);
// Ignore errors as this is optional
gpgme_data_set_flag(d->data, "size-hint", sizestr.c_str());

gpgme_data_set_flag then converts the string again to a gpgme_off_t value. Later add_input_size_hint (engine-gpg.c) adds the value to the command line if it's not zero.

aheinecke added a comment.Jun 12 2023, 3:12 PM

Yeah, its the ugly off_t again. I am just testing how this works with single files above that threshold we worked quite a bit on this back in the days https://dev.gnupg.org/T2368

aheinecke added a comment.Jun 12 2023, 3:13 PM

On 64 bit linux this works btw. so I think it comes down to the difference between 32 bit off_t and 64 bit off_t

aheinecke added a comment.Jun 12 2023, 3:18 PM

Yeah no progress for files larger then 32 bit o.O... But this used to work 😭

aheinecke reopened this task as Open.Jun 12 2023, 3:44 PM
aheinecke claimed this task.
aheinecke added a subscriber: • ebo.

I'm reopening this. Its probably not a regression but I was sure that we had progress for large files fixed in the past.

thesamesam added a subscriber: thesamesam.Sep 1 2023, 5:30 AM
aheinecke changed the task status from Open to Testing.Oct 19 2023, 1:39 PM

I think this was fixed with the fix for https://dev.gnupg.org/T6534

• ebo closed this task as Resolved.Oct 19 2023, 2:21 PM
• ebo moved this task from Backlog to done on the gpgme board.

yes, fixed

• werner moved this task from done to gpgme 1.23.x on the gpgme board.Oct 25 2023, 10:16 AM
• werner edited projects, added gpgme (gpgme 1.23.x); removed gpgme.