Gpg4win, data corruption
Closed, ResolvedPublic

Description

Hello.

I´ve installed Gpg4win 3.1.5 on a Windows 10 Home laptop. I created a directory with 10 files inside it, .pdf, .txt and .wmv. When I signed and cyphered the directory through Kleopatra menu, everything seems to go OK. I even received the integrity OK message form Kleopatra. BUT, when i reversed the process, verify and desencrypt, data on the directory is corrupted. I lost files and even the video cannot be reproduced.

The same error on 3.1.4 and 3.1.2.

Please help!

jmanuel created this task.Jan 21 2019, 5:21 PM
aheinecke triaged this task as Normal priority.Jan 22 2019, 10:20 AM
aheinecke added a subscriber: aheinecke.

Have you used a very old version (< 3 ) to create the directory? I know you said that you are currently using Gpg4win-3.1.5 but maybe you have used an older version to create the archive.
In the past we had a problem that Kleopatra would not see the errors from the archive program but that was fixed some time ago. The Archive problem had problems with non ASCII filenames but nowadays it only has problems with full unicode filenames and those errors are shown in Kleopatra.

Can you say more about "corruption" are files missing or are the file contents bad?

If you use gpg --decrypt "the archive" on the command line, does it show any output indicating problems?

On the archive you should then be able to extract the files using: gpgtar --skip-crypto --extract "the archive"

I assign this normal priority for now until we understand if it is a general bug or a specific problem.

Hello.

Have you used a very old version (< 3 ) to create the directory? I know you said that you are currently using Gpg4win-3.1.5 but maybe you have used an older version to create the archive.

The cyphered directory is a Windows 10 directory.

In the past we had a problem that Kleopatra would not see the errors from the archive program but that was fixed some time ago. The Archive problem had problems with non ASCII filenames but nowadays it only has problems with full unicode filenames and those errors are shown in Kleopatra.

An example of directory name and files inside it:

Can you say more about "corruption" are files missing or are the file contents bad?

Yes. For example, the result of the processing (cyphering and decyphering) of the same directory as above, but with errors. 7 files deleted.

In other cases, no deleted files but the video was corrupted, impossible to see it.

Even sometimes the process goes ok.

If you use gpg --decrypt "the archive" on the command line, does it show any output indicating problems?

Command line tools worked perfect for encryption and decription, BUT I used tar to make the package before encryption.

No error at command line execution.

IN Kleopara, even when my files were corrupted, no error is showed, everything seems to go ok.

I think the problem could be gpgtar.

On the archive you should then be able to extract the files using: gpgtar --skip-crypto --extract "the archive"

I didn´t probe it, I´will and let you know about the results.

I assign this normal priority for now until we understand if it is a general bug or a specific problem.

OK, thanks.

El 22 ene. 2019, a las 06:20, aheinecke (Andre Heinecke) <noreply@dev.gnupg.org> escribió:

aheinecke triaged this task as "Normal" priority.
aheinecke added a comment.

Have you used a very old version (< 3 ) to create the directory? I know you said that you are currently using Gpg4win-3.1.5 but maybe you have used an older version to create the archive.
In the past we had a problem that Kleopatra would not see the errors from the archive program but that was fixed some time ago. The Archive problem had problems with non ASCII filenames but nowadays it only has problems with full unicode filenames and those errors are shown in Kleopatra.

Can you say more about "corruption" are files missing or are the file contents bad?

If you use gpg --decrypt "the archive" on the command line, does it show any output indicating problems?

On the archive you should then be able to extract the files using: gpgtar --skip-crypto --extract "the archive"

I assign this normal priority for now until we understand if it is a general bug or a specific problem.

TASK DETAIL
https://dev.gnupg.org/T4332 https://dev.gnupg.org/T4332
EMAIL PREFERENCES
https://dev.gnupg.org/settings/panel/emailpreferences/ https://dev.gnupg.org/settings/panel/emailpreferences/
To: aheinecke
Cc: aheinecke, jmanuel, Rafixmod, ccharabaruk, gp_ast

I almost forget. If I zip the directory and the encypher the .zip file through Kleopatra, everything goes ok.

El 22 ene. 2019, a las 07:02, Juan Manuel Mosso <jmmosso@gmail.com> escribió:

Hello.

Have you used a very old version (< 3 ) to create the directory? I know you said that you are currently using Gpg4win-3.1.5 but maybe you have used an older version to create the archive.

The cyphered directory is a Windows 10 directory.

In the past we had a problem that Kleopatra would not see the errors from the archive program but that was fixed some time ago. The Archive problem had problems with non ASCII filenames but nowadays it only has problems with full unicode filenames and those errors are shown in Kleopatra.

An example of directory name and files inside it:

<Captura de pantalla 2019-01-22 a la(s) 06.48.05.png>

Can you say more about "corruption" are files missing or are the file contents bad?

Yes. For example, the result of the processing (cyphering and decyphering) of the same directory as above, but with errors. 7 files deleted.

<Captura de pantalla 2019-01-22 a la(s) 06.50.26.png>

In other cases, no deleted files but the video was corrupted, impossible to see it.

Even sometimes the process goes ok.

If you use gpg --decrypt "the archive" on the command line, does it show any output indicating problems?

Command line tools worked perfect for encryption and decription, BUT I used tar to make the package before encryption.

No error at command line execution.

IN Kleopara, even when my files were corrupted, no error is showed, everything seems to go ok.

I think the problem could be gpgtar.

On the archive you should then be able to extract the files using: gpgtar --skip-crypto --extract "the archive"

I didn´t probe it, I´will and let you know about the results.

I assign this normal priority for now until we understand if it is a general bug or a specific problem.

OK, thanks.

El 22 ene. 2019, a las 06:20, aheinecke (Andre Heinecke) <noreply@dev.gnupg.org <mailto:noreply@dev.gnupg.org>> escribió:

aheinecke triaged this task as "Normal" priority.
aheinecke added a comment.

Have you used a very old version (< 3 ) to create the directory? I know you said that you are currently using Gpg4win-3.1.5 but maybe you have used an older version to create the archive.
In the past we had a problem that Kleopatra would not see the errors from the archive program but that was fixed some time ago. The Archive problem had problems with non ASCII filenames but nowadays it only has problems with full unicode filenames and those errors are shown in Kleopatra.

Can you say more about "corruption" are files missing or are the file contents bad?

If you use gpg --decrypt "the archive" on the command line, does it show any output indicating problems?

On the archive you should then be able to extract the files using: gpgtar --skip-crypto --extract "the archive"

I assign this normal priority for now until we understand if it is a general bug or a specific problem.

TASK DETAIL
https://dev.gnupg.org/T4332 https://dev.gnupg.org/T4332
EMAIL PREFERENCES
https://dev.gnupg.org/settings/panel/emailpreferences/ https://dev.gnupg.org/settings/panel/emailpreferences/
To: aheinecke
Cc: aheinecke, jmanuel, Rafixmod, ccharabaruk, gp_ast

Hi, jmanuel, I agree with you.

I did a similar test: Encripting without signing a directory with several files with open pgp (using gpgme). It seems that it encripts ok (for the size of resulting file), but when I decript it only few files are shown. A lot are missing!

Hi jmrexach,

Can I help with something?

El 22 ene. 2019, a las 14:37, jmrexach (Josep M. Rexach) <noreply@dev.gnupg.org> escribió:

jmrexach added a comment.

Hi, jmanuel, I agree with you.

I did a similar test: Encripting without signing a directory with several files with open pgp (using gpgme). It seems that it encripts ok (for the size of resulting file), but when I decript it only few files are shown. A lot are missing!

TASK DETAIL
https://dev.gnupg.org/T4332 https://dev.gnupg.org/T4332
EMAIL PREFERENCES
https://dev.gnupg.org/settings/panel/emailpreferences/ https://dev.gnupg.org/settings/panel/emailpreferences/
To: jmrexach
Cc: jmrexach, aheinecke, jmanuel, Rafixmod, ccharabaruk, gp_ast

No, thks. It's just a test to confirm this rare behaviour. I did the same test several times with several and diferent results. But I think that a data corruption occurs sometimes when encripting a folder with several and large files. Just aconfirmation of the bug for the developpers.

Thks jmanuel.

aheinecke raised the priority of this task from Normal to High.Jan 24 2019, 9:20 AM
aheinecke claimed this task.

Thanks for the confirmation and additional info. In that case I give this high priority as I agree with the potential for data loss.

@jmrexach you write:

I did a similar test: Encripting without signing a directory with several files with open pgp (using gpgme). It seems that it encripts ok (for the size of resulting file), but when I decript it only few files are shown. A lot are missing!

What do you mean by "using gpgme" do you mean you have used Kleopatra or some other gpgme based application?

I was able to reproduce this. I tried it three times with a very large folder and it worked fine. The fourth try though created a corrupted archive and Kleopatra did not show an error either creating or extracting this archive!

Using gpgme means selecting the folder and use secondary button of the mouse (I think that this is the name of the integration tool with the desktop, isn't it?). Also fails using Kleopatra menu.
I tried several times with the same folder and I obtained diferent results. If I zip the folder and encrypt a unique large file seems to work fine. It seems to fail in pack-encrypt phase. Decoding several times the same encrypted file gives the same wrong result. Decrypt using command line also gives a wrong result. I didn't try to encrypt-tar from command line.

I think that this is the name of the integration tool with the desktop, isn't it?

That is GpgEX ;-) No problem, I just wanted to avoid confusion on my side.

It seems to fail in pack-encrypt

I agree. The tar folder looks corrupted. I have not yet understood under which circumstances this happens.

Sorry for the mistake!
It seems related to the load of the cpu. I'm using win7 32 bit in a netbook. When cpu is heavy loaded corruption is worse. I don't know if it's really relevant, only seems!

Kleopatra now shows an error in this case when extracting. So now we only need to fix that this happens at all.

I've uploaded a damaged archive which was created by gpgtar here: http://heinecke.or.at/T4332/damaged-archive.tar.gz
It's larger then necessary but this is what I had at hand.

I've patched gpgtar to dump out the first header record where a problem occurs:

This is what it reads:

00000000: 3020 525d 0d65 6e64 6f62 6a0d 3232 3630  0 R].endobj.2260
00000010: 2030 206f 626a 0d5b 3735 3020 3020 3020   0 obj.[750 0 0 
00000020: 3020 3020 3020 3020 3020 3020 3020 3020  0 0 0 0 0 0 0 0 
00000030: 3020 3020 3020 3020 3020 3020 3020 3020  0 0 0 0 0 0 0 0 
00000040: 3237 3820 3237 3820 3020 3020 3020 3020  278 278 0 0 0 0 
00000050: 3020 3020 3333 3320 3333 3320 3020 3020  0 0 333 333 0 0 
00000060: 3237 3820 3333 3320 3237 3820 3237 3820  278 333 278 278 
00000070: 3535 3620 3535 3620 3535 3620 3535 3620  556 556 556 556 
00000080: 3535 3620 3535 3620 3535 3620 3535 3620  556 556 556 556 
00000090: 3535 3620 3535 3620 3237 3820 3020 3020  556 556 278 0 0 
000000a0: 3020 3020 3535 3620 3020 3636 3720 3636  0 0 556 0 667 66
000000b0: 3720 3020 3732 3220 3636 3720 3631 3120  7 0 722 667 611 
000000c0: 3737 3820 3732 3220 3237 3820 3530 3020  778 722 278 500 
000000d0: 3636 3720 3535 3620 3020 3732 3220 3020  667 556 0 722 0 
000000e0: 3636 3720 3020 3732 3220 3636 3720 3020  667 0 722 667 0 
000000f0: 3732 3220 3636 3720 3934 3420 3020 3020  722 667 944 0 0 
00000100: 3631 3120 3020 3020 3020 3020 3020 3020  611 0 0 0 0 0 0 
00000110: 3535 3620 3535 3620 3530 3020 3535 3620  556 556 500 556 
00000120: 3535 3620 3237 3820 3535 3620 3535 3620  556 278 556 556 
00000130: 3232 3220 3232 3220 3530 3020 3232 3220  222 222 500 222 
00000140: 3833 3320 3535 3620 3535 3620 3535 3620  833 556 556 556 
00000150: 3020 3333 3320 3530 3020 3237 3820 3535  0 333 500 278 55
00000160: 3620 3530 3020 3732 3220 3020 3530 3020  6 500 722 0 500 
00000170: 3530 3020 3020 3020 3020 3020 3020 3020  500 0 0 0 0 0 0 
00000180: 3020 3020 3020 3020 3020 3020 3020 3020  0 0 0 0 0 0 0 0 
00000190: 3020 3020 3020 3020 3020 3020 3020 3020  0 0 0 0 0 0 0 0 
000001a0: 3020 3020 3020 3020 3020 3020 3020 3020  0 0 0 0 0 0 0 0 
000001b0: 3020 3020 3020 3020 3020 3020 3020 3020  0 0 0 0 0 0 0 0 
000001c0: 3020 3020 3020 3020 3020 3020 3535 3620  0 0 0 0 0 0 556 
000001d0: 3020 3020 3020 3020 3020 3020 3020 3020  0 0 0 0 0 0 0 0 
000001e0: 3020 3020 3020 3020 3020 3020 3020 3020  0 0 0 0 0 0 0 0 
000001f0: 3020 3020 3020 3020 3020 3020 3020 3020  0 0 0 0 0 0 0 0

This is obviously not a header. (The offset of this data in the tar is 010f3400)

The header for this file was:

00ed6c00: 6f6b 756c 6172 2d74 6573 7464 6174 656e  okular-testdaten
00ed6c10: 2f62 7567 3237 3137 3238 5f65 6469 745f  /bug271728_edit_
00ed6c20: 6e6f 5f72 6573 7472 6963 745f 6e6f 5f65  no_restrict_no_e
00ed6c30: 7874 656e 6465 6564 2e70 6466 0000 0000  xtendeed.pdf....
00ed6c40: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6c50: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6c60: 0000 0000 3030 3030 3634 3000 3030 3030  ....0000640.0000
00ed6c70: 3030 3000 3030 3030 3030 3000 3030 3031  000.0000000.0001
00ed6c80: 3033 3432 3134 3300 3133 3237 3330 3537  0342143.13273057
00ed6c90: 3232 3100 3032 3332 3134 0020 3000 0000  221.023214. 0...
00ed6ca0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6cb0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6cc0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6cd0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6ce0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6cf0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6d00: 0075 7374 6172 0030 3072 6f6f 7400 0000  .ustar.00root...
00ed6d10: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6d20: 0000 0000 0000 0000 0072 6f6f 7400 0000  .........root...
00ed6d30: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6d40: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6d50: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6d60: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6d70: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6d80: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6d90: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6da0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6db0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6dc0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6dd0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6de0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6df0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00ed6e00: 2550 4446 2d31 2e36 0d25 e2e3 cfd3 0d0a  %PDF-1.6.%......

The size here is correct. The file is indeed 2212963 (0x21C463) bytes long.
So data should end at:
0x21C463 + 0xED6E00 = 0x10f3263 followed by random data until
hex((2212963 % 512) + 0x10f3263) = 0x10f3400

The read for the next header record seems correct based on the size:

(2212963 + 512-1) / 512 = 4323
hex(4323*512) = 0x21c600
0x21c600 + 0xed6e00 = 0x10f3400 (which was the read from before)

But the file actually ends at: 010f4063
And the actual next header record is at: 010f4200

I do not see how this is possible in the code.
I'll now take a look at the difference between 0xED6E00 - 010f4063 and the actual file. Maybe this will show something.

The difference is between: 0x01035400 and 0x01034600 where 7 blocks of zero bytes are in the broken archive which are not present in the original file.

And this is also the difference between the actual end of file and the calculated end of file. 0x10f3263 +0xe00 = 0x10f4063

Further testing leads me to believe that this is probably a Kleopatra / QGpgME / Qt issue. I can pretty reliably reproduce this when using Kleopatra but never have I gotten this with gpgtar only, and I tested it a lot of times.

aheinecke changed the task status from Open to Testing.Mar 15 2019, 8:38 AM

After further debugging it showed that it had to be an issue in how we use QProcess. So I've rewritten the way we call gpgtar on Windows and replaced it with a simple anonymous pipe solution. I've tested more then ten times with various directories that the data corruption no longer occurs.
The performance is slightly better, but we still use GPGME so it's not as good as if we would pipe it directly. But not using GPGME was not really an option because otherwise I would have had to implement a full blown "how to call gpg" with error handling etc. for Kleopatra and I really did not want that.

I've also verified that the error handling of Input Errors still works (and improved it a bit).

Just for history if I ever need it again here is a patch I wrote debugging QIODeviceDataprovider. I have not commited it for fear of regressions and apparently the code is correct for Linux and that it did not work as expected on Windows had other reasons.

aheinecke closed this task as Resolved.Mar 27 2019, 1:52 PM

3.1.6 is released.

Thanks people, thanks aheinecke, for your support.

Cheers.

Juan Mosso.

El 27 mar. 2019, a las 09:55, aheinecke (Andre Heinecke) <noreply@dev.gnupg.org> escribió:

aheinecke closed subtask T4264: Gpg4win 3.1.6 as "Resolved".

TASK DETAIL
https://dev.gnupg.org/T4332 https://dev.gnupg.org/T4332
EMAIL PREFERENCES
https://dev.gnupg.org/settings/panel/emailpreferences/ https://dev.gnupg.org/settings/panel/emailpreferences/
To: aheinecke
Cc: jmrexach, aheinecke, jmanuel, Rafixmod, ccharabaruk, gp_ast