Dealing with Damaged RAW Files

First published on PhotographyLife

You may find this article to be useful in a practical way, not just as an isolated case of raw data damage. Often, just a casual look into raw data provides arguments allowing one to persuade technical support that there is a problem with your camera body that needs to be addressed.

Figure 0. Normal and damaged panorama frames

The case started with this post at DPReview:

"Hi all, Last weekend I took a 360 deg panorama and on processing the files discovered two frames had the partial magenta coloring. Both affected frames appeared identical and were taken 3 seconds apart. I have included the preceding frame as a reference.

The sun would have been at the top of the left hand edge on the affected frame.

The 5D3 has done 35k frames and I was using a 24 105L lens - 2 yrs old and manual exposure. I have not had this problem before and fired off another 73 frames without incident. Exif intact

Any comments. Thanks Allan" ¹

And here are the JPEGs (no processing of any kind):

Figure 1. Embedded JPEG of the normal shot

Figure 2. Embedded JPEG of the damaged shot

"I should have checked all the frames before I left the site!

There does seem to be a problem with the green channel but with the tools I have I can't get my head around it. Looks like I will have to contact Canon." ²

"I contacted Canon (New Zealand) and the tech said 'Clear all camera settings' and see if the problem still occurs. He wasn't interested in looking at the RAW file. We shall see." ³

Later Allan kindly provided three consequent shots, two of which are damaged and one isn't; and gave us permission to share them:

Normal shot

Damaged shot

Another damaged shot

To start solving this mystery, we need to first establish that a faulty memory card did not cause this problem.

We opened the damaged RAW shot in FastRawViewer, and switching between RAW and embedded JPEG  (by pressing "J" on the keyboard), we saw that they both look similar, which proves that we can exonerate the memory card and recording interface from all charges - otherwise the embedded JPEG and RAW would look different.

Figure 3. FastRawViewer. Damaged shot, RAW, 22% view

Figure 4. FastRawViewer. Damaged shot. Embedded JPEG, 22% view

Now, if we take a close look at the bottom of the shot we see a narrow strip of, for lack of a better term, garbage at the very bottom, both in the raw and in the embedded JPEG.

Let's zoom to 100% and see that odd band:

Figure 5. FastRawViewer. Bottom part of the Damaged shot. RAW, 100% view,

Figure 6. FastRawViewer. Bottom part of the damaged shot. Embedded JPEG, 100% view,

The garbage at the bottom is another clue to the problem - as a result of this problem, the start of the image buffer seems shifted "to the right" by a number that should closely correspond to the number of garbage pixels at the bottom, and the read operation goes past the image, into the memory block which contains some other information or even arbitrary bytes; hence the garbage. The shift also affected the Bayer pattern, so the supposedly red, green, and blue pixels are now not in the expected order, and consequently they were recorded in the wrong channels in the raw data.

The undamaged shot, however, looks quite normal - both the RAW and the embedded JPEG; no odd bands at the bottom, too.

Figure 7. FastRawViewer. Normal shot. RAW, 22% view

Figure 8. FastRawViewer. Normal shot. Embedded JPEG, 22% view

Figure 9. FastRawViewer. Bottom part of the normal shot. RAW, 100% view

Figure 10. FastRawViewer. Bottom part of the normal shot. Embedded JPEG, 100% view,

So we can deduce that there was some sort of in-camera synchronization glitch. But what sort of the glitch was there? And could we somehow repair it, if the file must be salvaged?

To answer it, let's open a damaged RAW in RawDigger (we have RawDigger on a Shift-R shortcut in FastRawViewer), and look at the histogram of the portion of the frame that should not contain clipping. We placed the selection on a cloud (shown as a grey rectangle positioned close to the right edge of the shot and overlaying the magenta-looking sky):

Figure 11. RawDigger. Damaged shot. The RAW histogram of the sampled area

So we see that according to the histogram, the Red and Blue channels are nearly equal, while the Green channels (both G and G2) are significantly different. This is not right. We know that the G and G2 channels are always very similar, while on the skies (except for sunrise / sunset) or the water reflecting the bluish sky, the Red channel is nearly always the weakest and the Blue channel has a peak in-between the Green and Red channel peaks, which is obviously not the case here. Based on that, it is apparent that the green pixels were recorded in the Red and Blue channels in the RAW file, while the red and blue pixels were recorded in the Green channels. It is the channel sequence that is mixed up here; the "colors" of pixels and the "colors" of channels do not match.

Suppose you do not have a good image to compare to, but (out of curiosity, or for the purpose of repairing the raw file) you still need to attribute channels as they are recorded in raw to the real red and blue (as you've seen, detecting green channels is rather trivial, based on their similarity).

To find out which of the Green channels in raw contains red pixels and which contains blue ones, let's have a look at the per-channel view, keeping in mind that the "true" Red channel typically has lower values for this type of light and subject than "true" Blue, and more contrast and details in the sky are also in the "true" Red channel (Dan Margulis in his books offers a lot of information on recognizing which channel is which).

Figure 12. RawDigger. The damaged shot. RAW, Channel G, which is actually the blue (B) channel

Figure 13. The damaged shot. RAW, Channel G2, which is actually the red (R) channel

From these images, and taking the above-mentioned into account (more details and contrast on the sky is in red), it is obvious that the G channel contains blue pixels, while the G2 contains red pixels.

We can also see it from the comparison with the correctly recorded shot.

Figure 14. Histograms of a sampled area on the sky

As you can see, on the normal shot (the left one) both Green channels are vertically aligned, the Red channel is the weakest, the Blue channel is in-between Green and Red channels. That is how it should be. As to the damaged shot (to the right), see above.

If this shot is very important to you, you can try to salvage it. One can export the channels and put them in the correct order in Photoshop, but a better way would be to recompile a decoder in a raw converter with a different Bayer pattern just for this case to rescue the shot (if you can't do it yourself, your friendly neighborhood open source raw converter developer should be able to help).

However, in the long run, this camera will be unpredictable and should be replaced or fixed.

In either case, you now have irrefutable proof to show tech support, and they will have to deal with your issue instead of making you try and hang up the phone in frustration.

¹  http://www.dpreview.com/forums/post/56331303

²  http://www.dpreview.com/forums/post/56337033

³  http://www.dpreview.com/forums/post/56339983