Been there, done that, stinkin T-shirts

I have a lot of sympathy for what Danny has been through as soon as he said 'database.'

Earlier this year, one of our remote systems had a database problem that could not be repaired. We could get some reports but one that reported on the health of that system stopped working. So I scheduled a repair action to wipe and restore only to discover the database backup would not restore. There was no helping it. So we had to restart from day zero.

Today I have another system, #132, whose database backup won't restore. It appears to be polling, recording data, and reporting but an accumulation of soft errors has left it appearing to work but not trustable. Worse, there is an unrelated issue that requires upgrade to the next version. Upgrade requires a wipe and reload but we know the backup won't restore.

Fortunately, I have a parallel system, #58, that we wiped and restarted in August when the restore from #132 failed. Since then I have moved production over to the parallel system. Over the coming Christmas and New Years holidays, I will wipe-and-load the older, #132, and try to restore from #58 backup. If the restore works, great. If not, we'll run them in parallel starting in January 2015.

The true test of a database is restoration of the backup to a fully operational system. If the restore does not work, a database may look OK but it is just waiting for a Murphy moment . . . and the joy of wearing the same T-shirt for a couple of days.

Bob Wilson

ps. For two years, I have requested enough disk storage to be able to collect and quality check the databases. But it wasn't until noon today that I finally got a 'left-over' system with enough disk space to do quality tests. Starting in 2015 I'll be able to do database quality checks before the Murphy moment.