Help needed...

[otumay]

It seems the file were corrupted somehow. Is it the same in both location, HDD and DVD? Do you have any other backups?

Thank you for your interest Manzur. The Raid backup system is at my office and the DVD backups are at home. I always backup to two different locations (now to the Cloud, instead of DVDs) before I process the raw files. I have no idea how this mishap happened, it seems to me that they were corrupted prior to the backups. These files date from 2013 and I had processed them back then. I just wanted to try again this year and thus became aware of the problem.
Kind regards,
Osman

 

[Manzur Fahim]

Thank you for your interest Manzur. The Raid backup system is at my office and the DVD backups are at home. I always backup to two different locations (now to the Cloud, instead of DVDs) before I process the raw files. I have no idea how this mishap happened, it seems to me that they were corrupted prior to the backups. These files date from 2013 and I had processed them back then. I just wanted to try again this year and thus became aware of the problem.
Kind regards,
Osman

Sorry about the corruption Osman. I had this happened to me once. One of the files were corrupted and showing me pink and grey lines in the photos. That day I realized, file will get corrupted one way or another unless I protect them somehow. So I started archiving the files that were from long ago, and started archiving them using WinRAR.

WinRAR has a volume system, where you can split the archive into any size files you want. It also has 256-bit AES encryption. It can create recovery volumes, which is kind of a RAID5 for the files, and it can include recovery records too.

For example, I want to archive a 20GB folder in 20 x 1GB rar files, and I want it password protected and encrypted, to have 2 recovery volumes and 10% recovery record. I can do that with WinRAR. It will create 20 x 1GB files, and 2 x 1GB recovery volumes.

Now let's say one of the file got corrupted. I can just open that file and try to repair it. It will use that 10% recovery record and try to fix it. If the error changed the file more than 10%, then it won't be able to, but files usually gets corrupted down to a single bit or byte and that makes the whole file unreadable, so I think 10% is plenty. WinRAR will, in that case, can repair the file and my archive is secure.

And if I lose one or two of the 20 rar files, and if I try to extract files, winrar will see that two files are missing, and it will rebuild those two files from parity data from those recovery files.

This way, so far, I haven't had any corruption issues. I had a few corruption with winrar files, but they were able to repair and fix themselves.

Mentioned it here just in case you want to have these kind of protections within your archives.

 

[otumay]

Sorry about the corruption Osman. I had this happened to me once. One of the files were corrupted and showing me pink and grey lines in the photos. That day I realized, file will get corrupted one way or another unless I protect them somehow. So I started archiving the files that were from long ago, and started archiving them using WinRAR.

WinRAR has a volume system, where you can split the archive into any size files you want. It also has 256-bit AES encryption. It can create recovery volumes, which is kind of a RAID5 for the files, and it can include recovery records too.

For example, I want to archive a 20GB folder in 20 x 1GB rar files, and I want it password protected and encrypted, to have 2 recovery volumes and 10% recovery record. I can do that with WinRAR. It will create 20 x 1GB files, and 2 x 1GB recovery volumes.

Now let's say one of the file got corrupted. I can just open that file and try to repair it. It will use that 10% recovery record and try to fix it. If the error changed the file more than 10%, then it won't be able to, but files usually gets corrupted down to a single bit or byte and that makes the whole file unreadable, so I think 10% is plenty. WinRAR will, in that case, can repair the file and my archive is secure.

And if I lose one or two of the 20 rar files, and if I try to extract files, winrar will see that two files are missing, and it will rebuild those two files from parity data from those recovery files.

This way, so far, I haven't had any corruption issues. I had a few corruption with winrar files, but they were able to repair and fix themselves.

Mentioned it here just in case you want to have these kind of protections within your archives.

Thank you again, Manzur. I went to WinRaR website and discovered that this software was mainly for Windows. I did find a site to download the Mac version, the download was not successful. I shall look into this in due time.

 

[Serge]

My archives exist in a RAID system....

Thanks for the kind reply Osman!

In my opinion, the reason for data corruption was the raid controller.
If the raid controller is not supported by a separate small battery and there is no overall Smart-UPS for the server / NAS / computer, then during power failures the data that is in the raid controller memory will be randomly dumped to the raid array (disks) and the existing data on the disks will be corrupted.

Anyway, being able to save at least the jpeg's is a good thing. And it is always better to keep the archives in 2 copies. At least on a flash drive or external USB drive.

 

[Manzur Fahim]

Thank you again, Manzur. I went to WinRaR website and discovered that this software was mainly for Windows. I did find a site to download the Mac version, the download was not successful. I shall look into this in due time.

You are most welcome, and you should be able to download it from their site:

WinRAR archiver, a powerful tool to process RAR and ZIP files

WinRAR is a Windows data compression tool that focuses on the RAR and ZIP data compression formats for all Windows users. Supports RAR, ZIP, CAB, ARJ, LZH, TAR, GZip, UUE, ISO, BZIP2, Z and 7-Zip

www.rarlab.com

 

[otumay]

Thanks for the kind reply Osman!

In my opinion, the reason for data corruption was the raid controller.
If the raid controller is not supported by a separate small battery and there is no overall Smart-UPS for the server / NAS / computer, then during power failures the data that is in the raid controller memory will be randomly dumped to the raid array (disks) and the existing data on the disks will be corrupted.

Anyway, being able to save at least the jpeg's is a good thing. And it is always better to keep the archives in 2 copies. At least on a flash drive or external USB drive.

What I learned from this mishap is that I should back up my files BEFORE doing anything else, and always from the memory card, as the second backup from the Raid system may already be corrupted.

 

[otumay]

You are most welcome, and you should be able to download it from their site:

WinRAR archiver, a powerful tool to process RAR and ZIP files

WinRAR is a Windows data compression tool that focuses on the RAR and ZIP data compression formats for all Windows users. Supports RAR, ZIP, CAB, ARJ, LZH, TAR, GZip, UUE, ISO, BZIP2, Z and 7-Zip

www.rarlab.com

Thank you, but the Mac-version is a command-line version, which is very difficult for me.

 

[amadeus21]

Hello to the Sigma forum! I came over as DPR-orphan ;-) And will tell more about me later.

I've tried a lot of softwares on Mac Studio M1 in order to open your x3f-files. But no success. This is very miserable, I do know very well, if pictures are gone...

The JPG-part can obviously be rescued via the software "GraphicConverter" (Mac).
That's more than nothing.

Best wishes and goog luck!
Johannes

 

[otumay]

Hello to the Sigma forum! I came over as DPR-orphan ;-) And will tell more about me later.

I've tried a lot of softwares on Mac Studio M1 in order to open your x3f-files. But no success. This is very miserable, I do know very well, if pictures are gone...

The JPG-part can obviously be rescued via the software "GraphicConverter" (Mac).
That's more than nothing.

Best wishes and goog luck!
Johannes

View attachment 964

Thank you Johannes! I shall try that software as well. Its interface looks more friendly to me as a Mac user.

 

[amadeus21]

I think, you will use GraphicConver only to save the pictures as (16bit ?) .tiff, and the further processing do in PSD or other software you are familiar with.

GraphicConverter is really the "Swiss Army Knife" for opening picture files. It doesn't ask big about the filename extension, but just rolls up its sleeves and starts works. The software is really worth the money.