This is why to test your backups!
Attention DBAs: listen carefully to this tale of woe…
At the Alaska Dept. of Revenue recently, a “technician” committed not one, but two, serious errors: first, he mistakenly deleted over 600,000 customer records (which were scanned paper applications from state residents); second, he reformatted the backup drive!
I’ll wait for the gasps and nausea to subside among you DBAs reading this.
But wait, it gets better. The department had a third line of defense: the data was also on tape. Lo and behold — you guessed it — the backup tapes were unreadable. Did I mention that the data was worth $38 billion (not million, billion)? The phrase “criminally incompetent” comes to mind.
Believe it or not, there was a happy ending to this sad story. The “old school” backup was still available — 300 boxes containing the original paper applications. These were re-scanned and the database was reconstructed. It took 70 people six weeks to do this, to the tune of $200,000 in overtime.
The morals of the story are obvious: 1) have a backup and recovery plan that includes worst-case scenarios, 2) test your backups regularly and 3) check the references of “technicians” that you hire!
Cheers, Tim
Posted: March 20th, 2007 under Oracle database administration.
this no way is worth 38 billion $, if it cost $200,00 to recreate the data from the applications. The most it would have cost is mailing each and every one of the Alaska residents the forms and ask them to refill them. Therefore, several million $ is the true cost.
Comment by Zachi — March 23, 2007 @ 1:41 pm
As the value of data was in billions, the concerned company is also reponsible, as the management should have taken precaution for having more backups at different location and media. Instead of spending $200,000 in recovery, lesser investment in better technologies for storage and recovery would have saved them from this problem.
Comment by Sukaina — March 25, 2007 @ 12:42 am
Zachi: You’re correct. It seems that the value of the data itself is unclear in the original article. However, what I believe that the article meant was that the content or meaning of the data — i.e., the money being disbursed to Alaska residents — was worth $38 billion.
Comment by Tim DiChiara — March 26, 2007 @ 10:48 pm