The Daily WTF: Curious Perversions in Information Technology

The Backup Pipeline

by Charles Robinson in Feature Articles on 2016-01-18

Dick was the main man in charge of his homegrown facilities-management system known as Q-Max. His brainchild was utilized across a large office complex to enter and track building maintenance tasks. Whenever a sink was broken, toilet clogged, or a foul-smelling science experiment had to be exterminated from a fridge, Q-Max was there to track and route the incident.

While that was its original intent, Dick wound up selling the office managers on expanding its role to include reporting that would help them make budgetary decisions. It helped them determine which maintenance guys deserved a raise and which buildings needed renovations the most. So despite the maintenance department despising Q-Max because it made them do stuff, management loved every bit of it. Little did they know how precarious the system was on the backend.

Dick focused on keeping this amazing application running, and let the company's "Disaster Recovery Team"- one guy named Pete- worry about the worst case scenarios. They kept a regular offsite backup of its data so that in case of something catastrophic like a meteor strike, Q-Max would not go the way of the dinosaurs. Their backup system was quite simple, yet had a necessary workaround step - Every night around midnight an Oracle backup was created on the database server. From there, they copied to an offsite tape backup. Due to infrastructure limitations- bandwidth and reliability, mostly- this mirroring process had to run to get the backup to a location the tape robot could get its iron clutches on.