It’s been years since I last attended school, but unless things have changed dramatically, tests are still basically the same as they were back in the day. You know the subject of the test, you can study the material in advance, but you’re not privy to the questions until the appointed hour the test begins. There are many types of tests including ‘open book’ tests (where you can bring reference materials), but all have the same surprise factor: you don’t know the answers in advance. If the examiner supplied the answers in advance it wouldn’t qualify as a test.
Now consider that the Disaster Recovery Preparedness Council’s 2015 annual report claims that only 33% of survey participants test their DR Plan – and 65% of those who do, fail their own test.
That would be a dismal accounting – if only it were true. It’s quite likely that the situation is far worse. I’m not accusing the Disaster Recovery Preparedness Council of perjury. I’m simply observing that a) statistically, voluntary responses to surveys give higher positive results than randomly polled survey responses, and b) most organizations that test their DR plans fail to do so using the test definition discussed above.
Before anyone shows up at my home or workplace to protest, let me explain.
I have had the pleasure of working with many organizations, of varying sizes, industries and nationalities as a Business Continuity planner and consultant. I’ve observed or participated in many DR tests. I have personally witnessed the planning that goes into many of those DR tests. Weeks of preparation; writing of recovery test plans; drafting of multiple, detailed scripts of the test activities for each system and application to be recovered. Sound familiar?
The test participants are given the ‘test plan’ (equivalent to the ‘questions’ on the test) in advance, and provided sufficient time to write detailed scripts of what they must do to “pass” the test. Yet – according to the DRPC – only 35% of them manage to pass.
Admittedly, Disaster Recovery tests are not the same as Business Continuity exercises; they require a level of logistics planning that even the most complex BC exercise doesn’t require. You can temporarily disrupt a Business Process to exercise it – if it fails, just go back to work. Try the same with a ‘live’ test of an IT application and you risk devastation if you fail. To protect day-to-day operations, most Disaster Recovery Plans are tested in parallel – recovered in an alternate environment to avoid disrupting ongoing business operations.
Certainly that takes planning. IT infrastructure relies on name and address conventions. Type the wrong IP address or the wrong domain name and either nothing happens or chaos reigns. Using a test infrastructure requires an entire set of devices and device names that have no chance of interfering with day-to-day operations.
With that advance planning, what explains the failure rate? Sixty-five percent sounds bad. But consider the other side of that equation. If only 33% test their plans – and 65% of those fail, then only 11.5% of organizations successfully test their IT Disaster Recovery Plans each year. Eleven and one-half percent!
Every participant in a DR test has advance notice of the test, of the scenario, of the circumstances and sufficient time to update their plans. Yet barely one-third are successful (of the one third that actually tests).
If you work for once of the remaining 88.5% of businesses whose DR plans either aren’t tested, or which routinely fail their tests, you might be a tad concerned. If you are responsible for DR testing at your organization, you might want to rethink your testing methodology and your recovery strategy. One or both are flawed.
