Braintrust, (graphics file gurus):
I have a handful of family pictures that have gotten corrupted somehow. A majority have the thumbnails in place showing the initial exif header is in good shape, but the scan section may be hosed. All pictures were taken from the same kodak camera, so a hex edit replacement of the jpg header may be an option. I don't know, I've just been googling for ideas. I suspect the corruption occurred in the copy from the flash drive to the hard-disk. The flash drive is long long overwritten.
I'm looking for any and all ideas. Different apps, different potential mods/recompiles to libjpeg, anything?? I've tried the different apps, gimp, kview, convert, etc.. convert gives the most descriptive error:
family_pictures/2008/12> convert 2008-12-24-095905.jpg -resize 95% ~/tmp/2008-12-24-095905.jpg convert: Corrupt JPEG data: 3896 extraneous bytes before marker 0x51 `2008-12-24-095905.jpg' @ warning/jpeg.c/JPEGWarningHandler/325. convert: Unsupported marker type 0x51 2008-12-24-095905.jpg @ error/jpeg.c/JPEGErrorHandler/297. convert: missing an image filename `/home/david/tmp/2008-12-24-095905.jpg' @ error/convert.c/ConvertImageCommand/3015.
From the error I can tell that there is a messed up jpeg marker, but I haven't a clue to what markers are legit and what aren't. (obviously it doesn't like 0x51...)
What say the gurus? Anyone every solved something like this before?
On 08/18/2011 01:04 AM, David C. Rankin wrote:
Braintrust, (graphics file gurus):
I have a handful of family pictures that have gotten corrupted somehow. A majority have the thumbnails in place showing the initial exif header is in good shape, but the scan section may be hosed. All pictures were taken from the same kodak camera, so a hex edit replacement of the jpg header may be an option. I don't know, I've just been googling for ideas. I suspect the corruption occurred in the copy from the flash drive to the hard-disk. The flash drive is long long overwritten.
I'm looking for any and all ideas. Different apps, different potential mods/recompiles to libjpeg, anything?? I've tried the different apps, gimp, kview, convert, etc.. convert gives the most descriptive error:
family_pictures/2008/12> convert 2008-12-24-095905.jpg -resize 95% ~/tmp/2008-12-24-095905.jpg convert: Corrupt JPEG data: 3896 extraneous bytes before marker 0x51 `2008-12-24-095905.jpg' @ warning/jpeg.c/JPEGWarningHandler/325. convert: Unsupported marker type 0x51 2008-12-24-095905.jpg @ error/jpeg.c/JPEGErrorHandler/297. convert: missing an image filename `/home/david/tmp/2008-12-24-095905.jpg' @ error/convert.c/ConvertImageCommand/3015.
From the error I can tell that there is a messed up jpeg marker, but I haven't a clue to what markers are legit and what aren't. (obviously it doesn't like 0x51...)
What say the gurus? Anyone every solved something like this before?
have you tried photorec http://www.cgsecurity.org/wiki/PhotoRec
On 08/17/2011 06:25 PM, Togan Muftuoglu wrote:
On 08/18/2011 01:04 AM, David C. Rankin wrote:
<snip>
From the error I can tell that there is a messed up jpeg marker, but I haven't a clue to what markers are legit and what aren't. (obviously it doesn't like 0x51...)
What say the gurus? Anyone every solved something like this before?
have you tried photorec http://www.cgsecurity.org/wiki/PhotoRec
Thanks Togan,
I'll give it a go. On first look, it looks like it is more of a deleted file recovery system as opposed to a graphic file repair utility. From the description, it will look for specific jpeg markers in order to find a deleted jpg:
<quote> For example, PhotoRec identifies a JPEG file when a block begins with:
0xff,0xd8,0xff,0xe0 0xff,0xd8,0xff,0xe1 or 0xff,0xd8,0xff,0xfe </quote>
I think the 'files' are actually OK, I just think I have either corrupted jpg headers or possible the actual scan data.
What I really need is a tool that can examine the different parts of a jpg file and see if it can detect/repair inconsistencies between the header/compression information and the scan (data) section of the image.
All ideas or welcome! I'll post a summary of what I find as I get through this process.
On Wednesday, August 17, 2011 07:17:36 PM David C. Rankin wrote:
On 08/17/2011 06:25 PM, Togan Muftuoglu wrote:
On 08/18/2011 01:04 AM, David C. Rankin wrote:
<snip>
From the error I can tell that there is a messed up jpeg marker, but I
haven't a clue to what markers are legit and what aren't. (obviously it doesn't like 0x51...)
What say the gurus? Anyone every solved something like this before?
have you tried photorec http://www.cgsecurity.org/wiki/PhotoRec
Thanks Togan,
I'll give it a go. On first look, it looks like it is more of a deleted file recovery system as opposed to a graphic file repair utility. From the description, it will look for specific jpeg markers in order to find a deleted jpg:
<quote> For example, PhotoRec identifies a JPEG file when a block begins with:
0xff,0xd8,0xff,0xe0 0xff,0xd8,0xff,0xe1 or 0xff,0xd8,0xff,0xfe</quote>
I think the 'files' are actually OK, I just think I have either corrupted jpg headers or possible the actual scan data.
What I really need is a tool that can examine the different parts of a jpg file and see if it can detect/repair inconsistencies between the header/compression information and the scan (data) section of the image.
All ideas or welcome! I'll post a summary of what I find as I get through this process.
Well, this is not a Linux tool but saved some jpg images for me some time ago.
http://www.datarescue.com/photorescue/index.htm
and
http://www.impulseadventure.com/photo/jpeg-snoop.html
Regards,
On 17/08/11 19:04, David C. Rankin wrote:
What say the gurus? Anyone every solved something like this before?
Did you tried jpegoptim ?
1) backup 2) run jpegoptim --strip-all <photo.jpg>
Does it barf ?
Cheers.
On 08/17/2011 10:04 PM, Cristian Rodríguez wrote:
On 17/08/11 19:04, David C. Rankin wrote:
What say the gurus? Anyone every solved something like this before?
Did you tried jpegoptim ?
- backup
- run jpegoptim --strip-all<photo.jpg>
Does it barf ?
Cheers.
Yup :(
12:07 alchemy:~/tmp> jpegoptim --strip-all --verbose 2008-12-24-095905.jpg 2008-12-24-095905.jpg (Corrupt JPEG data: 3896 extraneous bytes before marker 0x51) (Unsupported marker type 0x51) [ERROR]
Thanks for the thought though!
On 08/17/2011 06:04 PM, David C. Rankin wrote:
Braintrust, (graphics file gurus):
I have a handful of family pictures that have gotten corrupted somehow. A majority have the thumbnails in place showing the initial exif header is in good shape, but the scan section may be hosed. All pictures were taken from the same kodak camera, so a hex edit replacement of the jpg header may be an option. I don't know, I've just been googling for ideas. I suspect the corruption occurred in the copy from the flash drive to the hard-disk. The flash drive is long long overwritten.
Just wanted to drop a note and say thanks for your help and suggestions. The files are actually corrupt jpeg files. I don't know why I didn't do this before, but I finally just opened the files in a hex-editor to take a cursory look at the content and file structure. (probably because I didn't think I would be able to read it anyway :) They contain garbage for the data stream. There is all kinds of junk in the files. e.g.
0006:9460 80 00 20 00 58 00 00 00 1b 00 00 00 00 00 00 00 .. .X........... 0006:9470 1a 00 00 00 00 00 00 00 53 00 56 00 47 00 46 00 ........S.V.G.F. 0006:9480 45 00 43 00 6f 00 6e 00 76 00 6f 00 6c 00 76 00 E.C.o.n.v.o.l.v. 0006:9490 65 00 4d 00 61 00 74 00 72 00 69 00 78 00 45 00 e.M.a.t.r.i.x.E. 0006:94a0 6c 00 65 00 6d 00 65 00 6e 00 74 00 07 7f 00 00 l.e.m.e.n.t..... 0006:94b0 18 95 ce f8 07 7f 00 00 00 01 25 00 20 00 00 00 ..��......%. ... 0006:94c0 78 1f 9f 01 08 7f 00 00 0e 00 00 00 00 00 00 00 x............... 0006:94d0 00 00 00 00 39 00 32 00 80 03 2c 00 18 00 00 00 ....9.2...,..... 0006:94e0 00 00 00 00 00 00 00 00 28 16 c3 04 00 00 00 00 ........(.�..... 0006:94f0 80 03 00 00 18 00 00 00 00 00 00 00 00 00 00 00 ................ 0006:9500 80 06 59 03 00 00 00 00 ac 06 00 00 60 00 00 00 ..Y.....�...`...
Parts of the file are fine, but other parts make no sense at all. It appears that the actual SD memory card itself was hosed. That is the only way I can think of that I could actually copy the file and have it end up in a proper 'abc.jpg' filename without generating a copy error, but still have garbage inside. Something like a cross-linked file on the card causing the garbage in the files. Leave it to the DOS filesystem on the card to mess things up :)
Oh, well, the downside to digital photography... I guess this is the equivalent to shooting on an exposed roll of film...
On another note, for those faced with figuring out what is wrong with a jpeg, the combination of 'jhead' and the different parts of 'convert' give you the best information. Both great tools for getting image data.
Le 19/09/2011 20:10, David C. Rankin a écrit :
error, but still have garbage inside. Something like a cross-linked file on the card causing the garbage in the files. Leave it to the DOS filesystem on the card to mess things up :)
when you write (or read) a flash card, a large part of the operation is buffered
I sometime (too often) notice that one can umount the reader *before* all the reading/writing operations are completely done. Look at the blinking diode if you have one, it keep blinking
so one have to *wait* (30s, one minutes) before unplugging
few weeks ago I copied VirtualBox images from my computer to copy them to a computer 16 km away
when I tryed to copy, the files where present but unreadable. This special usb key do not have led :-(
I beg that the copy program first create the files, then, copy the data
jdd
On 09/19/2011 01:16 PM, jdd wrote:
Le 19/09/2011 20:10, David C. Rankin a écrit :
error, but still have garbage inside. Something like a cross-linked file on the card causing the garbage in the files. Leave it to the DOS filesystem on the card to mess things up :)
when you write (or read) a flash card, a large part of the operation is buffered
I sometime (too often) notice that one can umount the reader *before* all the reading/writing operations are completely done. Look at the blinking diode if you have one, it keep blinking
so one have to *wait* (30s, one minutes) before unplugging
few weeks ago I copied VirtualBox images from my computer to copy them to a computer 16 km away
when I tryed to copy, the files where present but unreadable. This special usb key do not have led :-(
I beg that the copy program first create the files, then, copy the data
jdd
Here, the data was written to the SD card by the camera, then the SD card was put into a memory card "reader" on my laptop. The files were then copied from the card to the hard drive using a script that basically did:
PICFILES=$(find /media/disk -type f -iname '*.jpg') cp -ur $PICFILES ~/img/photo
It worked well. Then I began experiencing the corruption issues.
I don't see how the command line copy operation could have not waited for the buffer to write to disk before copying the next image from the SD card??
Even after the copy operation was finished, I would then rsync the files to a local server before removing the SD card from the reader. This would have allowed at least 1-5 minutes before card removal. I would have expected an rsync error if it was trying to transfer a file that still had some type of buffer open?? (I could be wrong there.. I just don't know)
Anybody see how the multi-copy of files from the SD card via the script snippets above could have induced the corruption? To me, (at least as of now) the culprit still looks like a bad filesystem on the SD card. The card was reformatted shortly after this corruption issue, but given the corruption, I just went back to a digikam download direct from the camera rather than a copy from the SD card itself. I haven't had a problem since. Perhaps another test is in order.
Thoughts? Guidance?
On 9/19/2011 2:50 PM, David C. Rankin wrote:
On 09/19/2011 01:16 PM, jdd wrote:
Le 19/09/2011 20:10, David C. Rankin a écrit :
error, but still have garbage inside. Something like a cross-linked file on the card causing the garbage in the files. Leave it to the DOS filesystem on the card to mess things up :)
when you write (or read) a flash card, a large part of the operation is buffered
I sometime (too often) notice that one can umount the reader *before* all the reading/writing operations are completely done. Look at the blinking diode if you have one, it keep blinking
so one have to *wait* (30s, one minutes) before unplugging
few weeks ago I copied VirtualBox images from my computer to copy them to a computer 16 km away
when I tryed to copy, the files where present but unreadable. This special usb key do not have led :-(
I beg that the copy program first create the files, then, copy the data
jdd
Here, the data was written to the SD card by the camera, then the SD card was put into a memory card "reader" on my laptop. The files were then copied from the card to the hard drive using a script that basically did:
PICFILES=$(find /media/disk -type f -iname '*.jpg') cp -ur $PICFILES ~/img/photo
It worked well. Then I began experiencing the corruption issues.
I don't see how the command line copy operation could have not waited for the buffer to write to disk before copying the next image from the SD card??
Even after the copy operation was finished, I would then rsync the files to a local server before removing the SD card from the reader. This would have allowed at least 1-5 minutes before card removal. I would have expected an rsync error if it was trying to transfer a file that still had some type of buffer open?? (I could be wrong there.. I just don't know)
Anybody see how the multi-copy of files from the SD card via the script snippets above could have induced the corruption? To me, (at least as of now) the culprit still looks like a bad filesystem on the SD card. The card was reformatted shortly after this corruption issue, but given the corruption, I just went back to a digikam download direct from the camera rather than a copy from the SD card itself. I haven't had a problem since. Perhaps another test is in order.
Thoughts? Guidance?
If you unmounted before you removed, then you're fine. If you didn't unmount, then no amount of waiting is really garanteed, although usually dbflush will have flushed everything every 30 seconds.
But no there are no timing related problems or fixes. If it's mounted, the kernel handles all flow control (serialization/blocking/waiting), and when you unmount, when the umount command returns then umount and the kernel have promised everyone that they are all done and the sd card itself has acknowledged the last write as completed, and the sd card does not have a ram buffer cache in it that will lose data when you unplug the card.
That sample "garbage" looked tantalizingly image related. SVGFEConvolveMatrix googles up as an image manipulation function, which as far as I can tell, shouldn;t exist actually inside an image, but certainly exists in image manipulation software.
Have you tried running an image recovery util that scans a raw drive or drive image and recovers files by recognizing headers and other known image data structures in the raw data, not, or at least not exclusively, by looking at the filesystem. They recover several kinds of files but don't pretend to know their names. You end up with a bunch of numbered files. You have to open them to decide what they are and rename or discard them.
The point would be this kind of util would not let the filesystem metadat tell it where files start and stop. It would read every byte of the disk or disk image from zero to end, trying to recognize jpg and other data structures and it would output 1.jpg, 2.png, etc... based only on it's own parsing of the raw data, not by consulting the FAT etc.
I did this to recover a lot of deleted images from my girlfriends iphone after it crashed and the only fix was to factory reset, which involved a format. I did the factory reset to get the phone alive again then immediately jailbroke it and captured a dd image over the network and was able to recover thousands of images and other common files from the 8G image. Most were junk of course, browser cache and images and sound clips that were parts of apps, but also were many of the camera images. Of course some stuff was lost for good since the new OS install and the jailbreaking process did overwrite some of the drive just to get to the point where it was even possible to get a dd image.
Of course I don't remember the name of that app. It's not on this machine. I think it was a stand-alone app, not requiring install, and it's probably at home on an external drive along with the dd image and the recovered files.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Monday, 2011-09-19 at 15:56 -0400, Brian K. White wrote:
Of course I don't remember the name of that app. It's not on this machine. I think it was a stand-alone app, not requiring install, and it's probably at home on an external drive along with the dd image and the recovered files.
PhotoRec? http://www.cgsecurity.org/wiki/PhotoRec
- -- Cheers, Carlos E. R. (from 11.4 x86_64 "Celadon" at Telcontar)
On 09/19/2011 02:56 PM, Brian K. White wrote:
That sample "garbage" looked tantalizingly image related. SVGFEConvolveMatrix googles up as an image manipulation function, which as far as I can tell, shouldn;t exist actually inside an image, but certainly exists in image manipulation software.
I saw the SVG info and did kind of scratch my head. Looks like part of an inkscape file somehow got cross-linked. But, alas, it is just garbage... Other junk interesting stuff was:
0001:f380 50 08 00 00 00 00 00 00 44 00 00 00 00 00 00 00 P.......D....... 0001:f390 f0 2a aa 02 08 7f 00 00 72 00 6e 00 61 00 6c 00 �*�.....r.n.a.l. 0001:f3a0 20 00 46 00 6f 00 72 00 77 00 61 00 72 00 64 00 .F.o.r.w.a.r.d. 0001:f3b0 20 00 48 00 69 00 73 00 74 00 6f 00 72 00 79 00 .H.i.s.t.o.r.y. 0001:f3c0 00 00 00 00 00 00 00 00 a1 9a 00 00 00 00 00 00 ........�.......
0002:2000 3d 00 22 00 32 00 30 00 30 00 39 00 2d 00 30 00 =.".2.0.0.9.-.0. 0002:2010 31 00 2d 00 30 00 39 00 54 00 30 00 37 00 3a 00 1.-.0.9.T.0.7.:. 0002:2020 32 00 35 00 3a 00 32 00 39 00 5a 00 22 00 2f 00 2.5.:.2.9.Z."./. 0002:2030 3e 00 0a 00 20 00 20 00 3c 00 74 00 79 00 70 00 >... . .<.t.y.p. 0002:2040 65 00 64 00 5f 00 68 00 69 00 73 00 74 00 6f 00 e.d._.h.i.s.t.o. 0002:2050 72 00 79 00 5f 00 69 00 74 00 65 00 6d 00 20 00 r.y._.i.t.e.m. . 0002:2060 63 00 6f 00 6e 00 74 00 65 00 6e 00 74 00 3d 00 c.o.n.t.e.n.t.=. 0002:2070 22 00 68 00 74 00 74 00 70 00 3a 00 2f 00 2f 00 ".h.t.t.p.:././. 0002:2080 77 00 77 00 77 00 2e 00 77 00 33 00 73 00 63 00 w.w.w...w.3.s.c. 0002:2090 68 00 6f 00 6f 00 6c 00 73 00 2e 00 63 00 6f 00 h.o.o.l.s...c.o. 0002:20a0 6d 00 2f 00 73 00 69 00 74 00 65 00 6d 00 61 00 m./.s.i.t.e.m.a. 0002:20b0 70 00 2e 00 61 00 73 00 70 00 22 00 20 00 74 00 p...a.s.p.". .t. 0002:20c0 79 00 70 00 65 00 3d 00 22 00 74 00 65 00 78 00 y.p.e.=.".t.e.x. 0002:20d0 74 00 22 00 0a 00 20 00 20 00 20 00 20 00 20 00 t."... . . . . . 0002:20e0 20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00 . . . . . . . .
0002:2480 20 00 20 00 6c 00 61 00 73 00 74 00 5f 00 74 00 . .l.a.s.t._.t. 0002:2490 79 00 70 00 65 00 64 00 3d 00 22 00 32 00 30 00 y.p.e.d.=.".2.0. 0002:24a0 30 00 39 00 2d 00 30 00 31 00 2d 00 30 00 38 00 0.9.-.0.1.-.0.8. 0002:24b0 54 00 32 00 32 00 3a 00 31 00 32 00 3a 00 33 00 T.2.2.:.1.2.:.3. 0002:24c0 30 00 5a 00 22 00 2f 00 3e 00 0a 00 20 00 20 00 0.Z."./.>... . . 0002:24d0 3c 00 74 00 79 00 70 00 65 00 64 00 5f 00 68 00 <.t.y.p.e.d._.h. 0002:24e0 69 00 73 00 74 00 6f 00 72 00 79 00 5f 00 69 00 i.s.t.o.r.y._.i. 0002:24f0 74 00 65 00 6d 00 20 00 63 00 6f 00 6e 00 74 00 t.e.m. .c.o.n.t. 0002:2500 65 00 6e 00 74 00 3d 00 22 00 68 00 74 00 74 00 e.n.t.=.".h.t.t. 0002:2510 70 00 3a 00 2f 00 2f 00 6c 00 6f 00 63 00 61 00 p.:././.l.o.c.a. 0002:2520 6c 00 68 00 6f 00 73 00 74 00 2f 00 6c 00 69 00 l.h.o.s.t./.l.i. 0002:2530 6e 00 75 00 78 00 2f 00 6f 00 70 00 65 00 6e 00 n.u.x./.o.p.e.n. 0002:2540 53 00 75 00 53 00 45 00 2d 00 73 00 65 00 72 00 S.u.S.E.-.s.e.r. 0002:2550 76 00 65 00 72 00 2e 00 70 00 68 00 70 00 22 00 v.e.r...p.h.p.". 0002:2560 0a 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00 .. . . . . . . .
0002:4700 61 00 72 00 67 00 69 00 6e 00 00 00 62 00 6f 00 a.r.g.i.n...b.o. 0002:4710 74 00 74 00 6f 00 6d 00 20 00 6d 00 61 00 72 00 t.t.o.m. .m.a.r. 0002:4720 67 00 69 00 6e 00 00 00 31 00 30 00 30 00 00 00 g.i.n...1.0.0... 0002:4730 f0 00 00 00 00 00 00 00 54 00 00 00 00 00 00 00 �.......T.......
0002:b400 20 45 78 70 69 72 79 3d 31 38 30 30 30 0a 44 6f Expiry=18000.Do 0002:b410 63 73 20 4d 6f 64 69 66 69 63 61 74 69 6f 6e 3d cs Modification= 0002:b420 31 0a 46 69 67 73 20 4d 6f 64 69 66 69 63 61 74 1.Figs Modificat 0002:b430 69 6f 6e 3d 32 0a 4f 74 68 65 72 20 4d 6f 64 69 ion=2.Other Modi 0002:b440 66 69 63 61 74 69 6f 6e 3d 32 0a 45 6d 70 74 79 fication=2.Empty 0002:b450 20 4f 6e 20 45 78 69 74 3d 31 0a 0a 5b 50 65 72 On Exit=1..[Per 0002:b460 66 6f 72 6d 61 6e 63 65 5d 0a 4d 61 78 20 43 6f formance].Max Co 0002:b470 6e 6e 65 63 74 69 6f 6e 73 20 54 6f 74 61 6c 3d nnections Total= 0002:b480 36 34 0a 4d 61 78 20 43 6f 6e 6e 65 63 74 69 6f 64.Max Connectio 0002:b490 6e 73 20 53 65 72 76 65 72 3d 31 36 0a 0a 5b 46 ns Server=16..[F 0002:b4a0 69 6c 65 20 53 65 6c 65 63 74 6f 72 5d 0a 53 68 ile Selector].Sh 0002:b4b0 6f 77 20 44 65 74 61 69 6c 73 3d 30 0a 53 68 6f ow Details=0.Sho 0002:b4c0 77 20 48 69 64 64 65 6e 20 46 69 6c 65 73 3d 30 w Hidden Files=0 0002:b4d0 0a 0a 5b 43 6f 6c 75 6d 6e 73 5d 0a 49 6e 70 75 ..[Columns].Inpu 0002:b4e0 74 5f 74 72 65 65 76 69 65 77 3d 30 2c 20 32 34 t_treeview=0, 24 0002:b4f0 37 34 36 2c 20 31 2c 20 31 2c 20 32 34 35 30 32 746, 1, 1, 24502
0002:f050 43 00 68 00 6f 00 6f 00 73 00 65 00 20 00 68 00 C.h.o.o.s.e. .h. 0002:f060 65 00 6c 00 70 00 65 00 72 00 20 00 61 00 70 00 e.l.p.e.r. .a.p. 0002:f070 70 00 6c 00 69 00 63 00 61 00 74 00 69 00 6f 00 p.l.i.c.a.t.i.o. 0002:f080 6e 00 73 00 20 00 66 00 6f 00 72 00 20 00 6f 00 n.s. .f.o.r. .o. 0002:f090 74 00 68 00 65 00 72 00 20 00 70 00 72 00 6f 00 t.h.e.r. .p.r.o. 0002:f0a0 74 00 6f 00 63 00 6f 00 6c 00 73 00 00 00 00 00 t.o.c.o.l.s..... 0002:f0b0 20 06 00 00 00 00 00 00 34 00 00 00 00 00 00 00 .......4.......
I don't know what to make of any of it other than the whole filesystem was suspect :(
* David C. Rankin drankinatty@suddenlinkmail.com [09-19-11 17:46]:
On 09/19/2011 02:56 PM, Brian K. White wrote:
That sample "garbage" looked tantalizingly image related. SVGFEConvolveMatrix googles up as an image manipulation function, which as far as I can tell, shouldn;t exist actually inside an image, but certainly exists in image manipulation software.
I saw the SVG info and did kind of scratch my head. Looks like part of an inkscape file somehow got cross-linked. But, alas, it is just garbage... Other junk interesting stuff was:
0001:f380 50 08 00 00 00 00 00 00 44 00 00 00 00 00 00 00 P.......D....... 0001:f390 f0 2a aa 02 08 7f 00 00 72 00 6e 00 61 00 6c 00 �*�.....r.n.a.l. 0001:f3a0 20 00 46 00 6f 00 72 00 77 00 61 00 72 00 64 00 .F.o.r.w.a.r.d. 0001:f3b0 20 00 48 00 69 00 73 00 74 00 6f 00 72 00 79 00 .H.i.s.t.o.r.y.
It is quite possible the FAT on the flash drive was scrambled. After all, FAT is quite susceptible.
On 09/19/2011 04:49 PM, Patrick Shanahan wrote:
It is quite possible the FAT on the flash drive was scrambled. After all, FAT is quite susceptible.
I'd bet money on it. I also noticed it started after video was shot by the camera. I suspect the card couldn't keep up with the video data rate and got cooked. After the video was recorded, about 6 out of 10 files exhibited this corrupting. The ones that were saved to the good part of the card where (I guess) the FAT was intact, all was good.
This was a good quality card as well for the time (supposedly...) It occurred on a SanDisk Extreme III card. I'm going to run a few more tests with in on 11.4 and see if I can reproduce it. I'll let you know if I come up with anything interesting.
What is the best way to format one of these disks? In camera? mkfs.vfat?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2011-09-19 23:59, David C. Rankin wrote:
What is the best way to format one of these disks? In camera? mkfs.vfat?
In camera, IMO, so that the camera is happy :-)
- -- Cheers / Saludos,
Carlos E. R. (from 11.4 x86_64 "Celadon" at Telcontar)
* Carlos E. R. robin.listas@telefonica.net [09-19-11 18:55]:
On 2011-09-19 23:59, David C. Rankin wrote:
What is the best way to format one of these disks? In camera? mkfs.vfat?
In camera, IMO, so that the camera is happy :-)
I agree.
On Mon, 19 Sep 2011 16:59:02 -0500, "David C. Rankin" drankinatty@suddenlinkmail.com wrote:
What is the best way to format one of these disks? In camera? mkfs.vfat?
FAT is fat, no matter who creates it. So it shouldn't matter if you format it under Linux or in a camera.
Philipp
On Tuesday 20 Sep 2011, Philipp Thomas wrote:
On Mon, 19 Sep 2011 16:59:02 -0500, "David C. Rankin"
drankinatty@suddenlinkmail.com wrote:
What is the best way to format one of these disks? In camera? mkfs.vfat?
FAT is fat, no matter who creates it. So it shouldn't matter if you format it under Linux or in a camera.
So it may be, but my experience is that it DOES matter in some cases ... and swapping the memory card from one camera from phone to another CAN cause problems too ... Hidden files, proprietry "extensions" or "optimisations" (in the format or software) ... there are many things which might cause problems. If in doubt - format it with the device it'll be used in.
Dylan
Le 20/09/2011 12:00, Dylan a écrit :
If in doubt - format it with the device it'll be used in.
cameras don't only format, but create special file structure, proprietary software...
jdd
On Tue, 20 Sep 2011 12:04:54 +0200, jdd jdd@dodin.org wrote:
cameras don't only format, but create special file structure, proprietary software...
But my experience is that the special file/directory structure is created on the first access to the card. That's why you can simply plug in SD/CF cards as they are already formated and use them. I only had to format them when I wanted to partition them and/or use a fs other than FAT (for those cards/flash drives that wheren't used in my cameras).
Philipp
Philipp Thomas wrote:
On Mon, 19 Sep 2011 16:59:02 -0500, "David C. Rankin" drankinatty@suddenlinkmail.com wrote:
What is the best way to format one of these disks? In camera? mkfs.vfat?
FAT is fat, no matter who creates it. So it shouldn't matter if you format it under Linux or in a camera.
In theory there is no difference between theory and practice :)
In practice the FAT implementations in cameras (and other devices) differ and sometimes they do get confused. One common case I have experienced is previously deleted images reappearing.
I believe the safest way is to let the camera do all interaction with the card and only read data via USB (or whatever i/f). Sure, its a cargo cult but I haven't experienced any more problems since I adopted this mode of use. Of course if the card starts to give trouble, I'll use other tools as well to try to recover data.
Cheers, Dave
Le 20/09/2011 12:02, Dave Howorth a écrit :
the card and only read data via USB (or whatever i/f). Sure, its a cargo cult
yes, mainly because cameras have different usb connections, when most computer have a sd card reader
jdd
On Tue, 20 Sep 2011 11:02:42 +0100, Dave Howorth dhoworth@mrc-lmb.cam.ac.uk wrote:
One common case I have experienced is previously deleted images reappearing.
My experience shows that only happens when previously used cards are quick formated. Then only the basic structures are zeroed and rewritten but most data is left in place. Then it can indeed happen that deleted images reappear.
A though format that completely zeros all data areas should result in the same FS no matter who did the formating.
Philipp
On 9/20/2011 6:39 PM, Philipp Thomas wrote:
On Tue, 20 Sep 2011 11:02:42 +0100, Dave Howorth dhoworth@mrc-lmb.cam.ac.uk wrote:
One common case I have experienced is previously deleted images reappearing.
My experience shows that only happens when previously used cards are quick formated. Then only the basic structures are zeroed and rewritten but most data is left in place. Then it can indeed happen that deleted images reappear.
A though format that completely zeros all data areas should result in the same FS no matter who did the formating.
Wrong again. True only in theory.
On Tue, 20 Sep 2011 19:51:10 -0400, "Brian K. White" brian@aljex.com wrote:
Wrong again. True only in theory.
How can my experience be wrong? And that isn't theory! All SD cards I bought, both normal and micro, where factory formatted and reformatting them and my CF cards under both Linux and Windows didn't give *any* problems. So please give real examples instead of just stating the above.
Philipp
On 9/20/2011 10:10 PM, Philipp Thomas wrote:
On Tue, 20 Sep 2011 19:51:10 -0400, "Brian K. White"brian@aljex.com wrote:
Wrong again. True only in theory.
How can my experience be wrong? And that isn't theory! All SD cards I bought, both normal and micro, where factory formatted and reformatting them and my CF cards under both Linux and Windows didn't give *any* problems. So please give real examples instead of just stating the above.
Philipp
In my experience, Chinese people never die. I mean, I've never seen it happen once.
Why should I do a lot of work exhaustively documenting your ignorance? Feel free to read up on FAT, or don't. I don't care how educated any one person is (IE you), but I do care that false info is stated and goes uncontested, which is a form of assent.
Yes in theory any FAT implimentation that adheres to the spec (oh, wait, what "the" spec, there are multiple specs? gee...) should always work in perfect harmony with any other FAT implementation that also adheres to the same spec, you know, as long as it's a good spec to begin with.
That theory applies, and fails, in practically every thing any human has ever done. There is nothing special about FAT that makes it exempt such that it requires an actual dead baby in your lap to prove that it can fail some times.
But gee if you haven't used enough flash devices to have encountered a few oddball compatibility quirks directly over the years, just do something incredibly advanced like pull up the wikipedia entry for FAT, and right there alone, just as a starter, you should see that between the various versions of fat that exist or have existed, and the various configurable values and the various subsets of those that were actually ever common or ever generated by any Microsoft program, there are many many possible permutations that would all fall within the rules. Any of those might not be liked by some poorly written firmware that only bothered to account for the common case.
Also, this discussion isn't even limited to FAT. The real question was "format on my pc or let the camera do it" well "format on my pc" could just as easily generate NTFS or something else unless you specifically see to it.
My most resent actual issue happened to be with a camera that couldn't handle cards over 2G total size. Except it could, sorta. The docs for the camera simply said 2G of course since that's the biggest size where everything "just works". But if the card is larger, up to 4G but so long as it wasn't an SDHC card, The camera could actually use up to 2G of space just fine, _as long as there was only 2G or less free space on the card_. So, fill up the card with pics or tunes or a movie, using your pc, get the available space down to 2G or less, and then the camera could both take pics and videos normally, but it could even play the movie/tunes/pictures from anywhere in the whole 4G too.
but FAT is FAT is FAT right?
...and we haven't even touched on partitioning. Whenever something is common, such as say, how practically all thumb drives are shipped preformatted with a MSDOS partition table and a single fdisk partition #1 that takes up the whole space, pretty soon countless devices and software grows to make broken assumptions that that which is common is actually the only thing that even exists or is possible. I have no idea what that camera or indeed any camera would do if you partitioned an sd card into more than one partition. probably it would just ignore everything but partition 1, or it might simply say the card is invalid and reject it entirely.
There are no end of boneheaded things that firmware writers do. Like how some motherboard bios's will not even _try_ to boot a device or partition unless it sees what IT thinks is a standard DOS MBR, even if the partitioning is normal, and there partition is flagged bootable, and there is a valid bootloader there that just doesn't happen to be one of the standard DOS MBR's, it skips it. Or others that make utterly arbitrary decisions based on the size of the device. USB drive smaller than 128M then assume it's a floppy and reject it if it's not FAT12 with a fat12 mbr... and such things that are supposed to work in theory, but don't in reality, are endless.
If you want tosay "in my experience" well that's fine but stress that part a bit more and do not make any actual claims or statements that extend beyond that.
Even within the limited scope of just fat16 and fat32, there are gobs of variables there, and, despite your incredulity, firmware writers and hardware manufacturers really do code for the common case many times and it's altogether easy to be within spec yet outside that devices expectations, just by having an unusual block size or total size or volume name or permissions or partitioning or boot record etc...
Suppose the user unwittingly had their sd slot configured for performance instead of safety (write caching), suppose they had compression turned on? Suppose they had an uncommon locale/codepage? I had one camera that would crash simply if an unexpected file were on the sd card at all. Sure a smart firmware writer would not attempt to scan any files that the camera itself didn't write. In _theory_ that should never have been a problem... Another one choked on jpeg images that were valid but merely had different specs than the few types the camera itself ever produced.
Yes, _usually_ it's not a problem. By definition it can't really be exactly a common problem because any camera that is too delicate about the formatting either quickly becomes an extinct camera or quickly gets fixed with an update.
But usually is only usually.
On Wed, 21 Sep 2011 02:33:29 -0400, "Brian K. White" brian@aljex.com wrote:
On 9/20/2011 10:10 PM, Philipp Thomas wrote:
On Tue, 20 Sep 2011 19:51:10 -0400, "Brian K. White"brian@aljex.com
Feel free to read up on FAT, or don't.
I do know what FAT is, I even repaired FAT drives with a disk editor, so don't call me ignorant or unknowledgeable. Believe me, FAT is among the simplest file systems ever created.
But gee if you haven't used enough flash devices to have encountered a few oddball compatibility quirks directly over the years, just do something incredibly advanced like pull up the wikipedia entry for FAT, and right there alone, just as a starter, you should see that between the various versions of fat that exist or have existed, and the various configurable values and the various subsets of those that were actually ever common or ever generated by any Microsoft program, there are many many possible permutations that would all fall within the rules.
I know, but then pray tell me why most people can simply plug their SD card into their camera and it works without having been formated?
could just as easily generate NTFS
Not really. The difference between FAT and NTFS is as big as yours to a stoneage man, i.e. lightyears apart.
The docs for the camera simply said 2G of course since that's the biggest size where everything "just works".
[...]
but FAT is FAT is FAT right?
What you describe has more to do with the SD card spec than with FAT, but there are of cause at least fat16, fat32 and now that abomination Exfat.
And yes, in my day-to-day job I get to know a fair bit of broken BIOS code and devices, specially cheap (vor varying definitions of cheap :) consumer devices also have their quirks.
Philipp
Le 21/09/2011 04:10, Philipp Thomas a écrit :
On Tue, 20 Sep 2011 19:51:10 -0400, "Brian K. White"brian@aljex.com wrote:
Wrong again. True only in theory.
How can my experience be wrong?
it's only one's experience...
don't forget there are at least 3 fat version (12, 16 & 32)
I had sometime problems, specially mixing card from different cameras. It's so easy to format with the camera that have to use it, why search for problems?
software is never completely reliable. Too many devices with each his own one and his bugs
jdd
On Wed, 21 Sep 2011 08:58:58 +0200, jdd jdd@dodin.org wrote:
don't forget there are at least 3 fat version (12, 16 & 32)
Yes, but that doesn't contradict what I wrote. As far as you format the drive according to what variant the camera expects it shouldn't matter who did the formatting.
Philipp
On 9/21/2011 3:35 PM, Philipp Thomas wrote:
On Wed, 21 Sep 2011 08:58:58 +0200, jddjdd@dodin.org wrote:
don't forget there are at least 3 fat version (12, 16& 32)
Yes, but that doesn't contradict what I wrote. As far as you format the drive according to what variant the camera expects it shouldn't matter who did the formatting.
Philipp
Since when does "should" mean anything in computers? Or anything else technical?
You are really missing the point of the original recommendation I think.
When someone actually asks for advise for something like this at all, they have by doing so, advertised that they by definition would prefer, out of all the theoretical possible ways to do something, the simplest method that is the most likely to always work with the least requirement for them to be responsible for doing something the right way every time. The fewest and simplest rules to follow. The fewest opportunities for a possible mistake or unforeseen issue.
That is absolutely not "Eh just use whatever, it should usually work fine as long as everyone involved in the various different development teams for camera and the desktop software all did what they were supposed to." Yes that's true, but it doesn't come close to comparing to "Let the camera do it. That way it is either guaranteed to work perfectly, or you can safely know either the camera or the card is defective and return it for a new one or another model."
Unlikelihood is often a broken and wrong way to think in computer/tech/science areas. What matters is possible vs not-possible. Likelihood only matters in comparison to some other likelihood to judge which of two or more options presents the least or most likelihood. Then it's still a clear either/or choice, maximal vs not-maximal, if you can't have possible vs not-possible.
If you format using a pc, it's possible to have a problem. It's also possible for the problem to be your fault.
If you format using the device that will use the card most, then it is least "likely" for it to go wrong, but more importantly, it's not-possible for it to be your fault. So if it craps out, at least you have already done the biggest debug step already, it can't possibly be something wrong with your pc or software or your skill or awake-that-day-ness in using them.
There is no such thing as perfection or absolute certainty of course, but there is very clearly a difference in robustness of approaches. There is a maximize-chances vs don't-bother-to-maximize-chances. However pretty-good the less-robust way is, it's still the less-robust way. No getting around that math.
True, for this card formatting question it usually doesn't matter and true sometimes you want to move the card around between several devices and so there is no clear way to pick which one should receive the ideal treatment. Deciding that is a case by case basis. But the question asked was a simple one with a simple set of parameters to consider and a clear more-certain and a clear less-certain.
But you went a little further than that even. You didn't merely recommend "don't worry about it". That's actually fine, because really, it usually is, I don't bother to worry about it myself just for the record and my own recommendation normally would be don't worry about it. But the OP asked specifically if there was a, however small, chance of a difference between the camera vs the pc, and the answer to _that_ question is absolutely yes there is. But you went a tad further and simply stated an untrue fact that there is actually no difference and that there is no such thing even possible as a problem that would be affected one way or the other by the formatting device/method.
Well you're flat wrong about that. Yes I know my example wasn't the result of a difference in the fat formatting itself. Irrelevant on more than one level. It was still a perfect example and the honestly most recent hard personal example I happened to remember having myself first hand. There was nothing wrong with the pc or the sd card. The fat formatting was perfectly within specs and normal, and yet the camera had a problem with it, but if the camera were to format the same card, it would create a 2G filesystem, leaving 2G of unused space, (and unusable by anything else either unless the partition was shrunk and another partition created to use it). The camera would work perfectly with the formatting IT created, and it would NOT work perfectly with the formatting a pc created, even by a quite competent user who knows enough to specify fat16 or fat32, avoid unusual (== unexpected) block sizes, etc.
It not only serves as a _general_ example that things that are supposed to work don't always, as if that should ever need explaining or proving, It actually is an example of _exactly_ what the OP was wondering about. His question was higher level and encompassing more possible types of or reasons for problems than the single detail of possible FAT formats. He just asked if it made any difference to let the camera format the card. For the camera I was tallking about (which happens to be Samsung NV3 from a few years ago), it absolutely did exactly that. Whether it was because of a weird partial size-limit that only affected some routines and not others in the camera, and not specifically a low level fat structure issue, is irrelevant. All that matters is that, without having to know anything, the rule "let the camera format the cameras card" would ensure that camera "just worked", and not following that rule could result in a camera/card combo that doesn't.
Stop trying to defend the indefensible.
On Wed, 21 Sep 2011 17:47:11 -0400, "Brian K. White" brian@aljex.com wrote:
Unlikelihood is often a broken and wrong way to think in computer/tech/science areas. What matters is possible vs not-possible.
Ah, statistics mean something to you? Possibilities are used in quite a few other areas then you mention. But I won't discuss in this style so I'm off.
Philipp
On 9/20/2011 3:52 AM, Philipp Thomas wrote:
On Mon, 19 Sep 2011 16:59:02 -0500, "David C. Rankin" drankinatty@suddenlinkmail.com wrote:
What is the best way to format one of these disks? In camera? mkfs.vfat?
FAT is fat, no matter who creates it. So it shouldn't matter if you format it under Linux or in a camera.
Wrong.
On 09/19/2011 04:59 PM, David C. Rankin wrote: <snip>
I'm going to run a few more tests with in on 11.4 and see if I can reproduce it. I'll let you know if I come up with anything interesting.
What is the best way to format one of these disks? In camera? mkfs.vfat?
As a follow-up, I dusted off the old script I used to copy the files from the SD card mounted in the card reader and tested on 2 separate cameras. (one containing the original SD card where the corruption occurred). After copying 69 photos directly through the card reader, all were fine, none were corrupt.
After revisiting the issue, I am more convinced than ever it was a FAT problem on the SD card that caused the original corruption. A new format of the SD card seems to have corrected the problem. It is much nicer to have the flexibility of transferring the photos via a script rather than using a photo-management app to do it for you.
I'll continue to keep a close eye on the issue, but regardless of the exact nature of the original cause - it's working like a champ now...
On 9/20/2011 4:14 PM, David C. Rankin wrote:
On 09/19/2011 04:59 PM, David C. Rankin wrote:
<snip>
I'm going to run a few more tests with in on 11.4 and see if I can reproduce it. I'll let you know if I come up with anything interesting.
What is the best way to format one of these disks? In camera? mkfs.vfat?
As a follow-up, I dusted off the old script I used to copy the files from the SD card mounted in the card reader and tested on 2 separate cameras. (one containing the original SD card where the corruption occurred). After copying 69 photos directly through the card reader, all were fine, none were corrupt.
After revisiting the issue, I am more convinced than ever it was a FAT problem on the SD card that caused the original corruption. A new format of the SD card seems to have corrected the problem. It is much nicer to have the flexibility of transferring the photos via a script rather than using a photo-management app to do it for you.
I'll continue to keep a close eye on the issue, but regardless of the exact nature of the original cause - it's working like a champ now...
Doh... so, the original SD card has since been re-written? And all you have are files that were transferred while the corrupt fat was in effect? you don't have a raw dd image of the sd card before it was re-written?
If you had either the original sd card or a raw dd image of it, then PhotoRec or similar tool might have recovered a lot of images because it would scan the raw data without consulting the fat. But that path is not available if the original raw data is gone and all you have are files that were created according to the old probably corrupt fat.
Then again, if all you did was format the sd card and haven't yet written a lot onto it, maybe PhotoRec can still recover something from the card.
On 09/20/2011 03:38 PM, Brian K. White wrote:
Doh... so, the original SD card has since been re-written? And all you have are files that were transferred while the corrupt fat was in effect? you don't have a raw dd image of the sd card before it was re-written?
If you had either the original sd card or a raw dd image of it, then PhotoRec or similar tool might have recovered a lot of images because it would scan the raw data without consulting the fat. But that path is not available if the original raw data is gone and all you have are files that were created according to the old probably corrupt fat.
Then again, if all you did was format the sd card and haven't yet written a lot onto it, maybe PhotoRec can still recover something from the card.
You got it :) The corruption wasn't discovered until months after the corruption occurred and the original card re-written to many times. The primary reason it went undiscovered is that the thumbnails in the jpeg headers were fine so when you looked at the directory in konqueror, win explorer, gallery2, etc.. all the photos appeared fine. It wasn't until you tried to open the actual data stream of the jpeg/jfif file that the corruption became apparent. Oh well -- live and learn and ... always check the actual image for corruption before deleting the original :)
My backups are about 4 levels deep -- the only problem is ... if the 1st one is corrupt, "your screwed all the way down...."
Le 19/09/2011 20:50, David C. Rankin a écrit :
I don't see how the command line copy operation could have not waited for the buffer to write to disk before copying the next image from the SD card??
no idea, I use Dolphin
Anybody see how the multi-copy of files from the SD card via the script snippets above could have induced the corruption? To me, (at
I once lost a hole bunch of jpeg files by an not really understood backup script. Ended with original files *and* backup files of size zero. I suspected a link loop, but found the problem too late to fix it :-(
least as of now) the culprit still looks like a bad filesystem on the SD card. The card was reformatted shortly after this corruption issue, but given the corruption, I just went back to a digikam download direct from the camera rather than a copy from the SD card itself. I haven't had a problem since. Perhaps another test is in order.
do't think this will make any good. If the file system is bad, it's bad everywhere
computer are sometime like men... unreliable
jdd
-
Brian K. White -
Carlos E. R. -
Cristian Rodríguez -
Dave Howorth -
David C. Rankin -
Dylan -
jdd -
Patrick Shanahan -
Philipp Thomas -
Ricardo Chung -
Togan Muftuoglu