gpgtar does not yet output progess indication. If can be partly done using
gpgtar --gpg-args --enable-progress-filer --status-fd N ....
but that is limited because gpg does not known the total size of data. However, gpgtar knows this because it first scans all files. The proposal is to emit progress status lines at the gpgtar site with information on the number of files processed and the their size. Frontends may then pick whatever they like.
| rG GnuPG | |||
| rGed9a420a221a gpgtar: Emit progress status lines in create mode. | |||
| rGf84264e8acf7 gpgtar: Emit progress status lines in create mode. | |||
How with --status-fd passed to gpgtar we will get these progress lines:
[GNUPG:] PROGRESS gpgtar c 5000 0 [GNUPG:] PROGRESS gpgtar c 10000 0 [...] [GNUPG:] PROGRESS gpgtar c 50000 0 [GNUPG:] PROGRESS gpgtar c 0 50223 [GNUPG:] PROGRESS gpgtar s 0 6543 MiB [GNUPG:] PROGRESS gpgtar c 100 50223 [...] [GNUPG:] PROGRESS gpgtar s 6300 6543 MiB [GNUPG:] PROGRESS gpgtar s 6400 6543 MiB [GNUPG:] PROGRESS gpgtar c 44800 50223 [...] [GNUPG:] PROGRESS gpgtar c 50210 50223 [GNUPG:] PROGRESS gpgtar s 6543 6543 MiB
The first lines with zero as last value are emitted during the initial scan and thus before starting to write. Followed by this are two lines with the expected numbers (50223 files with a total size of 6543 MiB). The writing takes place and the current values are updated. As of now in increments of 100 for the file count and in 100MiB for the total size. The last two lines give the actual written numbers. The file count is a bit less because a few files were not suitable for archiving.
When also enabling the progress filter of gpg, the status line from gpg are mixed in like here:
[GNUPG:] PROGRESS gpgtar s 100 6543 MiB [GNUPG:] PROGRESS stdin ? 108224 0 KiB [GNUPG:] PROGRESS stdin ? 127488 0 KiB
Due to compression and the tar format overhead gpg still does not known the final size of the file. gpg uses the name of the input file as first word; this will always be "stdin" here. Thus grepping for gpgatr and 'c' or 's' is sufficient to distinguish the two outputs.
I guess we need some gpgme support as well.
How to handle the extraction progress depends on how we can pass the size of the filename etc. Also gpgme related.
btw. this does not work on the decrypting side because kleopatra there apparently does not provide a size hint. So it goes:
[4888] org.kde.pim.kleopatra: Task: "Entschlüsselung: test1-archive.tar.gpg ..." has no total progress set.
[4888] org.kde.pim.kleopatra: Collection Progress: 1 total: 1000
[4888] org.kde.pim.kleopatra: Collection Progress: 219 total: 1000
[4888] org.kde.pim.kleopatra: Collection Progress: 429 total: 1000
[4888] org.kde.pim.kleopatra: Collection Progress: 732 total: 1000
[4888] org.kde.pim.kleopatra: Collection Progress: 1000 total: 1000
[4888] org.kde.pim.kleopatra: Collection Progress: 1000 total: 1000
[4888] org.kde.pim.kleopatra: Collection Progress: 1000 total: 1000
[4888] org.kde.pim.kleopatra: Collection Progress: 1000 total: 1000
[4888] org.kde.pim.kleopatra: Collection Progress: 1000 total: 1000
[4888] org.kde.pim.kleopatra: Collection Progress: 1000 total: 1000
But I won't reopen that here as this would be a kleo issue to pass the size hint afaik. And T6530 would solve that anyway.
For me this does work also when decrypting:
org.kde.pim.kleopatra: Task: "Decrypting: src.tar.gpg..." has no total progress set. org.kde.pim.kleopatra: Kleo::Crypto::DecryptVerifyTask(0x24b2550) setProgress 0 1032480 org.kde.pim.kleopatra: Kleo::Crypto::DecryptVerifyTask(0x24b2550) setProgress 167991 1032480 org.kde.pim.kleopatra: Collection Progress: 162 total: 1000 org.kde.pim.kleopatra: Kleo::Crypto::DecryptVerifyTask(0x24b2550) setProgress 315831 1032480 org.kde.pim.kleopatra: Collection Progress: 305 total: 1000 org.kde.pim.kleopatra: Kleo::Crypto::DecryptVerifyTask(0x24b2550) setProgress 538364 1032480 org.kde.pim.kleopatra: Collection Progress: 521 total: 1000 org.kde.pim.kleopatra: Kleo::Crypto::DecryptVerifyTask(0x24b2550) setProgress 734711 1032480 org.kde.pim.kleopatra: Collection Progress: 711 total: 1000 org.kde.pim.kleopatra: Kleo::Crypto::DecryptVerifyTask(0x24b2550) setProgress 972424 1032480 org.kde.pim.kleopatra: Collection Progress: 941 total: 1000 org.kde.pim.kleopatra: Kleo::Crypto::DecryptVerifyTask(0x24b2550) setProgress 1032480 1032480 org.kde.pim.kleopatra: Collection Progress: 1000 total: 1000
The additional log lines are only local, but I also see a progressing progress bar. And gpgme calls gpgtar with
2023-06-09 17:46:34 gpgme[29033.71d9] _gpgme_io_spawn: check: argv[ 0] = gpgtar [...] 2023-06-09 17:46:34 gpgme[29033.71d9] _gpgme_io_spawn: check: argv[21] = --gpg-args 2023-06-09 17:46:34 gpgme[29033.71d9] _gpgme_io_spawn: check: argv[22] = --input-size-hint=1057260135
Mh, let me check where the size hint should come from and why it might be missing in my windows test. That is indeed strange then.
Probably some issue with large files / integer overflow. I am testing on Windows with 32 bit.
For a 5gb file I get:
2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: enter: path=0x033ba808 path=c:\\GnuPG\\bin\\gpgtar.exe 2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: check: path=0x033ba808 argv[ 0] = gpgtar.exe 2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: check: path=0x033ba808 argv[ 1] = --batch 2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: check: path=0x033ba808 argv[ 2] = --status-fd 2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: check: path=0x033ba808 argv[ 3] = 1 2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: check: path=0x033ba808 argv[ 4] = --gpg-args 2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: check: path=0x033ba808 argv[ 5] = --no-tty 2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: check: path=0x033ba808 argv[ 6] = --gpg-args 2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: check: path=0x033ba808 argv[ 7] = --charset=utf8 2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: check: path=0x033ba808 argv[ 8] = --gpg-args 2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: check: path=0x033ba808 argv[ 9] = --enable-progress-filter 2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: check: path=0x033ba808 argv[10] = --gpg-args 2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: check: path=0x033ba808 argv[11] = --exit-on-status-write-err 2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: check: path=0x033ba808 argv[12] = --gpg-args 2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: check: path=0x033ba808 argv[13] = --ttyname=/dev/tty 2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: check: path=0x033ba808 argv[14] = --decrypt 2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: check: path=0x033ba808 argv[15] = --directory 2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: check: path=0x033ba808 argv[16] = C:/Users/Andre Heinecke/Ap 2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: check: path=0x033ba808 argv[17] = -- 2023-06-12 13:15:20 gpgme[3640.c50] _gpgme_io_spawn: check: path=0x033ba808 tmp_name = C:\\Users\\ANDREH~1\\AppDa 2023-06-12 13:15:20 gpgme[3640.c50] gpgme:AllowSetForegroundWindow: call: called for pid 6636; result=0
While for a 50mb file I see:
pawn: enter: path=0x06f73be8 path=c:\\GnuPG\\bin\\gpgtar.exe 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[ 0] = gpgtar.exe 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[ 1] = --batch 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[ 2] = --status-fd 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[ 3] = 1 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[ 4] = --gpg-args 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[ 5] = --no-tty 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[ 6] = --gpg-args 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[ 7] = --charset=utf8 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[ 8] = --gpg-args 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[ 9] = --enable-progress-filter 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[10] = --gpg-args 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[11] = --exit-on-status-write-error 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[12] = --gpg-args 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[13] = --ttyname=/dev/tty 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[14] = --decrypt 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[15] = --directory 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[16] = C:/Users/Andre Heinecke/AppData/Local/Temp/kleopatra-quIYDw 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[17] = --gpg-args 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[18] = --input-size-hint=49404929 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 argv[19] = -- 2023-06-12 13:25:27 gpgme[6492.11a8] _gpgme_io_spawn: check: path=0x06f73be8 tmp_name = C:\\Users\\ANDREH~1\\AppData\\Local\\Temp\\gpgme-1RWrOf
Both files were created the same way with Kleopatra and live in the same folder.
Yeah, probably a Windows/MinGW 32-bit problem. GpgME::Data does
off_t size = seek(0, SEEK_END); seek(0, SEEK_SET); std::string sizestr = std::to_string(size); // Ignore errors as this is optional gpgme_data_set_flag(d->data, "size-hint", sizestr.c_str());
gpgme_data_set_flag then converts the string again to a gpgme_off_t value. Later add_input_size_hint (engine-gpg.c) adds the value to the command line if it's not zero.
Yeah, its the ugly off_t again. I am just testing how this works with single files above that threshold we worked quite a bit on this back in the days https://dev.gnupg.org/T2368
On 64 bit linux this works btw. so I think it comes down to the difference between 32 bit off_t and 64 bit off_t
I'm reopening this. Its probably not a regression but I was sure that we had progress for large files fixed in the past.