Hello.
I´ve installed Gpg4win 3.1.5 on a Windows 10 Home laptop. I created a directory with 10 files inside it, .pdf, .txt and .wmv. When I signed and cyphered the directory through Kleopatra menu, everything seems to go OK. I even received the integrity OK message form Kleopatra. BUT, when i reversed the process, verify and desencrypt, data on the directory is corrupted. I lost files and even the video cannot be reproduced.
The same error on 3.1.4 and 3.1.2.
Please help!
| rKLEOPATRA Kleopatra | |||
| rKLEOPATRA18447fd4ff21 Fix handle closing in windowsprocessdevice | |||
| rKLEOPATRA0a985a08297f Use windows pipe io device for gpgtar | |||
| rKLEOPATRA20d29e409f7d Add output process error handling | |||
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Resolved | • aheinecke | T4332 Gpg4win, data corruption | ||
| Resolved | • aheinecke | T4264 Gpg4win 3.1.6 | ||
| Resolved | • werner | T4289 GnuPG 2.1.12 release |
Have you used a very old version (< 3 ) to create the directory? I know you said that you are currently using Gpg4win-3.1.5 but maybe you have used an older version to create the archive.
In the past we had a problem that Kleopatra would not see the errors from the archive program but that was fixed some time ago. The Archive problem had problems with non ASCII filenames but nowadays it only has problems with full unicode filenames and those errors are shown in Kleopatra.
Can you say more about "corruption" are files missing or are the file contents bad?
If you use gpg --decrypt "the archive" on the command line, does it show any output indicating problems?
On the archive you should then be able to extract the files using: gpgtar --skip-crypto --extract "the archive"
I assign this normal priority for now until we understand if it is a general bug or a specific problem.
Hello.
Have you used a very old version (< 3 ) to create the directory? I know you said that you are currently using Gpg4win-3.1.5 but maybe you have used an older version to create the archive.
The cyphered directory is a Windows 10 directory.
In the past we had a problem that Kleopatra would not see the errors from the archive program but that was fixed some time ago. The Archive problem had problems with non ASCII filenames but nowadays it only has problems with full unicode filenames and those errors are shown in Kleopatra.
An example of directory name and files inside it:
Can you say more about "corruption" are files missing or are the file contents bad?
Yes. For example, the result of the processing (cyphering and decyphering) of the same directory as above, but with errors. 7 files deleted.
In other cases, no deleted files but the video was corrupted, impossible to see it.
Even sometimes the process goes ok.
If you use gpg --decrypt "the archive" on the command line, does it show any output indicating problems?
Command line tools worked perfect for encryption and decription, BUT I used tar to make the package before encryption.
No error at command line execution.
IN Kleopara, even when my files were corrupted, no error is showed, everything seems to go ok.
I think the problem could be gpgtar.
On the archive you should then be able to extract the files using: gpgtar --skip-crypto --extract "the archive"
I didn´t probe it, I´will and let you know about the results.
I assign this normal priority for now until we understand if it is a general bug or a specific problem.
OK, thanks.
El 22 ene. 2019, a las 06:20, aheinecke (Andre Heinecke) <noreply@dev.gnupg.org> escribió:
aheinecke triaged this task as "Normal" priority.
aheinecke added a comment.Have you used a very old version (< 3 ) to create the directory? I know you said that you are currently using Gpg4win-3.1.5 but maybe you have used an older version to create the archive.
In the past we had a problem that Kleopatra would not see the errors from the archive program but that was fixed some time ago. The Archive problem had problems with non ASCII filenames but nowadays it only has problems with full unicode filenames and those errors are shown in Kleopatra.Can you say more about "corruption" are files missing or are the file contents bad?
If you use gpg --decrypt "the archive" on the command line, does it show any output indicating problems?
On the archive you should then be able to extract the files using: gpgtar --skip-crypto --extract "the archive"
I assign this normal priority for now until we understand if it is a general bug or a specific problem.
TASK DETAIL
https://dev.gnupg.org/T4332 https://dev.gnupg.org/T4332
EMAIL PREFERENCES
https://dev.gnupg.org/settings/panel/emailpreferences/ https://dev.gnupg.org/settings/panel/emailpreferences/
To: aheineckeCc: aheinecke, jmanuel, Rafixmod, ccharabaruk, gp_ast
This is an automated email from the GnuPG development hub. If you have registered in the past at https://bugs.gnupg.org/ https://bugs.gnupg.org/ your account was migrated automatically. You can visit https://dev.gnupg.org/ https://dev.gnupg.org/ to set a new password and update your email preferences.
I almost forget. If I zip the directory and the encypher the .zip file through Kleopatra, everything goes ok.
El 22 ene. 2019, a las 07:02, Juan Manuel Mosso <jmmosso@gmail.com> escribió:
Hello.
Have you used a very old version (< 3 ) to create the directory? I know you said that you are currently using Gpg4win-3.1.5 but maybe you have used an older version to create the archive.
The cyphered directory is a Windows 10 directory.
In the past we had a problem that Kleopatra would not see the errors from the archive program but that was fixed some time ago. The Archive problem had problems with non ASCII filenames but nowadays it only has problems with full unicode filenames and those errors are shown in Kleopatra.
An example of directory name and files inside it:
<Captura de pantalla 2019-01-22 a la(s) 06.48.05.png>
Can you say more about "corruption" are files missing or are the file contents bad?
Yes. For example, the result of the processing (cyphering and decyphering) of the same directory as above, but with errors. 7 files deleted.
<Captura de pantalla 2019-01-22 a la(s) 06.50.26.png>
In other cases, no deleted files but the video was corrupted, impossible to see it.
Even sometimes the process goes ok.
If you use gpg --decrypt "the archive" on the command line, does it show any output indicating problems?
Command line tools worked perfect for encryption and decription, BUT I used tar to make the package before encryption.
No error at command line execution.
IN Kleopara, even when my files were corrupted, no error is showed, everything seems to go ok.
I think the problem could be gpgtar.
On the archive you should then be able to extract the files using: gpgtar --skip-crypto --extract "the archive"
I didn´t probe it, I´will and let you know about the results.
I assign this normal priority for now until we understand if it is a general bug or a specific problem.
OK, thanks.
El 22 ene. 2019, a las 06:20, aheinecke (Andre Heinecke) <noreply@dev.gnupg.org <mailto:noreply@dev.gnupg.org>> escribió:
aheinecke triaged this task as "Normal" priority.
aheinecke added a comment.Have you used a very old version (< 3 ) to create the directory? I know you said that you are currently using Gpg4win-3.1.5 but maybe you have used an older version to create the archive.
In the past we had a problem that Kleopatra would not see the errors from the archive program but that was fixed some time ago. The Archive problem had problems with non ASCII filenames but nowadays it only has problems with full unicode filenames and those errors are shown in Kleopatra.Can you say more about "corruption" are files missing or are the file contents bad?
If you use gpg --decrypt "the archive" on the command line, does it show any output indicating problems?
On the archive you should then be able to extract the files using: gpgtar --skip-crypto --extract "the archive"
I assign this normal priority for now until we understand if it is a general bug or a specific problem.
TASK DETAIL
https://dev.gnupg.org/T4332 https://dev.gnupg.org/T4332
EMAIL PREFERENCES
https://dev.gnupg.org/settings/panel/emailpreferences/ https://dev.gnupg.org/settings/panel/emailpreferences/
To: aheineckeCc: aheinecke, jmanuel, Rafixmod, ccharabaruk, gp_ast
This is an automated email from the GnuPG development hub. If you have registered in the past at https://bugs.gnupg.org/ https://bugs.gnupg.org/ your account was migrated automatically. You can visit https://dev.gnupg.org/ https://dev.gnupg.org/ to set a new password and update your email preferences.
Hi, jmanuel, I agree with you.
I did a similar test: Encripting without signing a directory with several files with open pgp (using gpgme). It seems that it encripts ok (for the size of resulting file), but when I decript it only few files are shown. A lot are missing!
Hi jmrexach,
Can I help with something?
El 22 ene. 2019, a las 14:37, jmrexach (Josep M. Rexach) <noreply@dev.gnupg.org> escribió:
jmrexach added a comment.
Hi, jmanuel, I agree with you.
I did a similar test: Encripting without signing a directory with several files with open pgp (using gpgme). It seems that it encripts ok (for the size of resulting file), but when I decript it only few files are shown. A lot are missing!
TASK DETAIL
https://dev.gnupg.org/T4332 https://dev.gnupg.org/T4332
EMAIL PREFERENCES
https://dev.gnupg.org/settings/panel/emailpreferences/ https://dev.gnupg.org/settings/panel/emailpreferences/
To: jmrexachCc: jmrexach, aheinecke, jmanuel, Rafixmod, ccharabaruk, gp_ast
This is an automated email from the GnuPG development hub. If you have registered in the past at https://bugs.gnupg.org/ https://bugs.gnupg.org/ your account was migrated automatically. You can visit https://dev.gnupg.org/ https://dev.gnupg.org/ to set a new password and update your email preferences.
No, thks. It's just a test to confirm this rare behaviour. I did the same test several times with several and diferent results. But I think that a data corruption occurs sometimes when encripting a folder with several and large files. Just aconfirmation of the bug for the developpers.
Thks jmanuel.
Thanks for the confirmation and additional info. In that case I give this high priority as I agree with the potential for data loss.
@jmrexach you write:
I did a similar test: Encripting without signing a directory with several files with open pgp (using gpgme). It seems that it encripts ok (for the size of resulting file), but when I decript it only few files are shown. A lot are missing!
What do you mean by "using gpgme" do you mean you have used Kleopatra or some other gpgme based application?
I was able to reproduce this. I tried it three times with a very large folder and it worked fine. The fourth try though created a corrupted archive and Kleopatra did not show an error either creating or extracting this archive!
Using gpgme means selecting the folder and use secondary button of the mouse (I think that this is the name of the integration tool with the desktop, isn't it?). Also fails using Kleopatra menu.
I tried several times with the same folder and I obtained diferent results. If I zip the folder and encrypt a unique large file seems to work fine. It seems to fail in pack-encrypt phase. Decoding several times the same encrypted file gives the same wrong result. Decrypt using command line also gives a wrong result. I didn't try to encrypt-tar from command line.
I think that this is the name of the integration tool with the desktop, isn't it?
That is GpgEX ;-) No problem, I just wanted to avoid confusion on my side.
It seems to fail in pack-encrypt
I agree. The tar folder looks corrupted. I have not yet understood under which circumstances this happens.
Sorry for the mistake!
It seems related to the load of the cpu. I'm using win7 32 bit in a netbook. When cpu is heavy loaded corruption is worse. I don't know if it's really relevant, only seems!
Kleopatra now shows an error in this case when extracting. So now we only need to fix that this happens at all.
I've uploaded a damaged archive which was created by gpgtar here: http://heinecke.or.at/T4332/damaged-archive.tar.gz
It's larger then necessary but this is what I had at hand.
I've patched gpgtar to dump out the first header record where a problem occurs:
This is what it reads:
00000000: 3020 525d 0d65 6e64 6f62 6a0d 3232 3630 0 R].endobj.2260 00000010: 2030 206f 626a 0d5b 3735 3020 3020 3020 0 obj.[750 0 0 00000020: 3020 3020 3020 3020 3020 3020 3020 3020 0 0 0 0 0 0 0 0 00000030: 3020 3020 3020 3020 3020 3020 3020 3020 0 0 0 0 0 0 0 0 00000040: 3237 3820 3237 3820 3020 3020 3020 3020 278 278 0 0 0 0 00000050: 3020 3020 3333 3320 3333 3320 3020 3020 0 0 333 333 0 0 00000060: 3237 3820 3333 3320 3237 3820 3237 3820 278 333 278 278 00000070: 3535 3620 3535 3620 3535 3620 3535 3620 556 556 556 556 00000080: 3535 3620 3535 3620 3535 3620 3535 3620 556 556 556 556 00000090: 3535 3620 3535 3620 3237 3820 3020 3020 556 556 278 0 0 000000a0: 3020 3020 3535 3620 3020 3636 3720 3636 0 0 556 0 667 66 000000b0: 3720 3020 3732 3220 3636 3720 3631 3120 7 0 722 667 611 000000c0: 3737 3820 3732 3220 3237 3820 3530 3020 778 722 278 500 000000d0: 3636 3720 3535 3620 3020 3732 3220 3020 667 556 0 722 0 000000e0: 3636 3720 3020 3732 3220 3636 3720 3020 667 0 722 667 0 000000f0: 3732 3220 3636 3720 3934 3420 3020 3020 722 667 944 0 0 00000100: 3631 3120 3020 3020 3020 3020 3020 3020 611 0 0 0 0 0 0 00000110: 3535 3620 3535 3620 3530 3020 3535 3620 556 556 500 556 00000120: 3535 3620 3237 3820 3535 3620 3535 3620 556 278 556 556 00000130: 3232 3220 3232 3220 3530 3020 3232 3220 222 222 500 222 00000140: 3833 3320 3535 3620 3535 3620 3535 3620 833 556 556 556 00000150: 3020 3333 3320 3530 3020 3237 3820 3535 0 333 500 278 55 00000160: 3620 3530 3020 3732 3220 3020 3530 3020 6 500 722 0 500 00000170: 3530 3020 3020 3020 3020 3020 3020 3020 500 0 0 0 0 0 0 00000180: 3020 3020 3020 3020 3020 3020 3020 3020 0 0 0 0 0 0 0 0 00000190: 3020 3020 3020 3020 3020 3020 3020 3020 0 0 0 0 0 0 0 0 000001a0: 3020 3020 3020 3020 3020 3020 3020 3020 0 0 0 0 0 0 0 0 000001b0: 3020 3020 3020 3020 3020 3020 3020 3020 0 0 0 0 0 0 0 0 000001c0: 3020 3020 3020 3020 3020 3020 3535 3620 0 0 0 0 0 0 556 000001d0: 3020 3020 3020 3020 3020 3020 3020 3020 0 0 0 0 0 0 0 0 000001e0: 3020 3020 3020 3020 3020 3020 3020 3020 0 0 0 0 0 0 0 0 000001f0: 3020 3020 3020 3020 3020 3020 3020 3020 0 0 0 0 0 0 0 0
This is obviously not a header. (The offset of this data in the tar is 010f3400)
The header for this file was:
00ed6c00: 6f6b 756c 6172 2d74 6573 7464 6174 656e okular-testdaten 00ed6c10: 2f62 7567 3237 3137 3238 5f65 6469 745f /bug271728_edit_ 00ed6c20: 6e6f 5f72 6573 7472 6963 745f 6e6f 5f65 no_restrict_no_e 00ed6c30: 7874 656e 6465 6564 2e70 6466 0000 0000 xtendeed.pdf.... 00ed6c40: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6c50: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6c60: 0000 0000 3030 3030 3634 3000 3030 3030 ....0000640.0000 00ed6c70: 3030 3000 3030 3030 3030 3000 3030 3031 000.0000000.0001 00ed6c80: 3033 3432 3134 3300 3133 3237 3330 3537 0342143.13273057 00ed6c90: 3232 3100 3032 3332 3134 0020 3000 0000 221.023214. 0... 00ed6ca0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6cb0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6cc0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6cd0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6ce0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6cf0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6d00: 0075 7374 6172 0030 3072 6f6f 7400 0000 .ustar.00root... 00ed6d10: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6d20: 0000 0000 0000 0000 0072 6f6f 7400 0000 .........root... 00ed6d30: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6d40: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6d50: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6d60: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6d70: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6d80: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6d90: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6da0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6db0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6dc0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6dd0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6de0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6df0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 00ed6e00: 2550 4446 2d31 2e36 0d25 e2e3 cfd3 0d0a %PDF-1.6.%......
The size here is correct. The file is indeed 2212963 (0x21C463) bytes long.
So data should end at:
0x21C463 + 0xED6E00 = 0x10f3263 followed by random data until
hex((2212963 % 512) + 0x10f3263) = 0x10f3400
The read for the next header record seems correct based on the size:
(2212963 + 512-1) / 512 = 4323
hex(4323*512) = 0x21c600
0x21c600 + 0xed6e00 = 0x10f3400 (which was the read from before)
But the file actually ends at: 010f4063
And the actual next header record is at: 010f4200
I do not see how this is possible in the code.
I'll now take a look at the difference between 0xED6E00 - 010f4063 and the actual file. Maybe this will show something.
The difference is between: 0x01035400 and 0x01034600 where 7 blocks of zero bytes are in the broken archive which are not present in the original file.
And this is also the difference between the actual end of file and the calculated end of file. 0x10f3263 +0xe00 = 0x10f4063
Further testing leads me to believe that this is probably a Kleopatra / QGpgME / Qt issue. I can pretty reliably reproduce this when using Kleopatra but never have I gotten this with gpgtar only, and I tested it a lot of times.
After further debugging it showed that it had to be an issue in how we use QProcess. So I've rewritten the way we call gpgtar on Windows and replaced it with a simple anonymous pipe solution. I've tested more then ten times with various directories that the data corruption no longer occurs.
The performance is slightly better, but we still use GPGME so it's not as good as if we would pipe it directly. But not using GPGME was not really an option because otherwise I would have had to implement a full blown "how to call gpg" with error handling etc. for Kleopatra and I really did not want that.
I've also verified that the error handling of Input Errors still works (and improved it a bit).
Just for history if I ever need it again here is a patch I wrote debugging QIODeviceDataprovider. I have not commited it for fear of regressions and apparently the code is correct for Linux and that it did not work as expected on Windows had other reasons.
I've started some documentation how to repair a broken archive under: https://wiki.gnupg.org/TroubleShooting#Restoring_corrupted_Archives_created_by_Kleopatra
Thanks people, thanks aheinecke, for your support.
Cheers.
Juan Mosso.
El 27 mar. 2019, a las 09:55, aheinecke (Andre Heinecke) <noreply@dev.gnupg.org> escribió:
aheinecke closed subtask T4264: Gpg4win 3.1.6 as "Resolved".
TASK DETAIL
https://dev.gnupg.org/T4332 https://dev.gnupg.org/T4332
EMAIL PREFERENCES
https://dev.gnupg.org/settings/panel/emailpreferences/ https://dev.gnupg.org/settings/panel/emailpreferences/
To: aheineckeCc: jmrexach, aheinecke, jmanuel, Rafixmod, ccharabaruk, gp_ast
This is an automated email from the GnuPG development hub. If you have registered in the past at https://bugs.gnupg.org/ https://bugs.gnupg.org/ your account was migrated automatically. You can visit https://dev.gnupg.org/ https://dev.gnupg.org/ to set a new password and update your email preferences.