How To Recover When The System Fails

Careful planning and transparent communications bolster the beginning and end of disaster planning and business continuity.

 
 

"Be prepared" is the Boy Scout motto, and it’s not such bad advice for credit unions, too. Disaster planning begins with preparedness, and that means thinking about what might spring up to take down a credit union’s operations.

A good first step is making a list of what could possibly go wrong, advises Stephen Adwell, vice president of compliance and security at core processor EPL in Birmingham, AL.

"It doesn’t have to be a natural disaster," Adwell says. "It could be a fire in the building, pipes bursting, somebody digging up a data or phone line. Don’t just focus on the big things. Focus on the most likely things."

EPL serves approximately 80 credit unions, most of them using the core processor’s data center as their host. Credit unions that outsource like that need to be particularly aware of the level of preparedness going on at their service bureau.

"What we do for business continuity and disaster recovery capabilities for our own operation also helps them," Adwell says. "That’s why we always invite several of our credit unions to participate here when we test the system."

You should intentionally fail various components of your system and you should test for things you haven’t done before.

Steve_Adwell_EPL
Stephen Adwell is vice president of compliance and security for EPL in Birmingham, AL.
Kirk_Drake_headshot__high_res
Kirk Drake is president/CEO of Ongoing Operations in Hagerstown, MD.
MikeLawson_2015pic
Mike Lawson is principal at DML Communications in San Diego, CA.

Know The Limits

Even credit unions that run their systems in-house still depend on outside providers as backup sites in case things fail, and when it comes to testing, one industry insider says, they shouldn’t be afraid to do it aggressively.

"Don’t fall into that trap of not testing for fear of failing," says Kirk Drake, president/CEO of Ongoing Operations, a provider of data backup, failover testing and planning, and other disaster recovery and business continuity (DR/BC) services to approximately 120 credit unions directly and 1,000 altogether through other vendors. "You should intentionally fail various components of your system and you should test for things you haven’t done before. That’s what will help your ability to respond evolve."

Drake — whose company brings back online one or two credit unions a month after server failures and other technology fails — also decries seeking perfection in testing as "a fallacy and a waste of time."

"It’s far more important to have discipline around the process," he adds.

He likens a credit union’s technology infrastructure to a living organism that constantly changes and needs to be tested regularly, just as people should keep up with their health through preventative care such as checking blood pressure and cholesterol levels regularly.

Those human tests speak to risk factors, and assessing risk is another core component of effective DR/BC thinking. Drake advises credit unions to document heavily while assessing and prioritizing risks and to thoroughly analyze the business impact of each threat. Then, spend the needed time and money to offset that risk.

Perhaps the biggest risk of all is coming between members and their money. In that regard, the rapid growth of cloud storage and other technologies — and shrinking reliance on physical backup in the form of tape and disks — have made a dramatic difference in the ability to resume business after natural and man-made disasters. 

"The biggest change in the past few years has been in what is technically possible," Drake says. "The bottleneck used to be how fast you could access and recover the data. Now the biggest chokepoint is often the credit union’s own legacy relationship with the Internet."

Tell your members and the media how you’re going to solve the problem and prevent it from happening again.

Communication Challenges

Whether it's going to take a few hours or a few days to resume operations, someone needs to say something. And someone needs to be in charge. According to EPL’s Adwell, team play matters greatly in times of crisis.

A well-documented plan identifies the people with the most intricate knowledge of processes and infrastructure as well as their roles in business recovery. It also details how to contact everyone.

"Make sure there are a couple of hard copies in the right hands," Adwell says.  "Don’t just leave the document on a server or in the cloud."

The documented plan also should identify who holds the responsibility to speak to members and the public — and, if necessary, the media — on behalf of the credit union in the wake of an outage.

And regardless of whether it’s a data breach, denial of service attack, or a plain old fire or natural disaster, some rules do apply.

"The important thing is to have a plan in place to address whatever happens," advises Mike Lawson of DML Communications in San Diego. "Be professional, and be transparent. Your members and the media will appreciate that."

Also, it’s vital the credit union takes responsibility for its part in the problem, however big or small.

"Own up to it and apologize," says the longtime industry communications consultant and producer of CUbroadcast. "Tell your members and the media how you’re going to solve the problem and prevent it from happening again."

Don’t hide or say "no comment." According to Lawson, silence spurs rumor and innuendo, which can create a firestorm of falsities.

"It appears you're hiding something," he says. "Be proactive and professional with your swift response. Silence equals mistrust."

Have a statement ready to go and a spokesperson ready to respond. Appoint one or two people at most — ideally the CEO and a subject matter expert — to communicate on behalf of the credit union. And although transparency is important, don’t report every little change in the situation, Lawson advises.

"That can make it look like you’re not on top of things," he says.

Finally, if the spokesperson doesn’t know information offhand or can’t answer a question accurately, it’s OK to tell members and the media the credit union will have find out and deliver the answer ASAP. However, it must follow up on that promise.

People appreciate transparency, humbleness, and action, Lawson says. And honesty trumps all.

"It might be challenging and a bit painful in the beginning," he says, "But it will bode well for your credit union in the long run."

7 Steps To Talk Your Way Out Of A Crisis

Mike Lawson spent several years in marketing communications with Symitar before launching DML Communications in 2002. Since then he has helped 45 different organizations with their media relations and marketing and is about to produce his 500th CUbroadcast interview.

Here Lawson outlines steps to manage communications during a crisis.

Step 1: Brainstorm all possible non-natural disasters. What are the scenarios? What are the solutions? What is the message? Who are the staff experts to address each scenario?

Step 2: Designate a spokesperson (most likely the CEO) and a credit union staff expert who is well versed in the specific situation to speak about the crisis at hand.

Step 3: Craft an honest, transparent message for members and the media. "If there's any ounce of tap-dancing around the issue, the media and members will sense it and your credibility is blown," Lawson says.

Step 4: Distribute the message in a timely manner. "It's your chance to control the message, its tempo, and its direction," Lawson says. "Be swift in your timing."

Step 5: Gauge the response of members and the media. Be prepared to address their concerns as quickly as possible.

Step 6: Update members and the media on significant changes in the situation. Again, be proactive and deliver reports swiftly.

Step 7: Keep a plan in a notebook that's easily accessible when one of these scenarios arises. Review the different scenarios each quarter or every six months, update accordingly, and don't forget to add new scenarios.

 

 

 

 

Nov. 30, 2015


Comments

 
 
 
  • The 7 steps listed are great tips, not just for credit unions, but for any business following a public breakdown. I have much more respect for businesses which quickly (and in a detailed manner) disclose the issues faced, even if they were to blame. Raise the bar even more when they then update me as details emerge, and a "how we will prevent it from occurring again" plan is released. How you handle the "crisis" publicly is far more important than if it was "your fault". We all make mistakes; it's the learning afterwards that matters.
    Joe Winn
     
     
     
  • Good point, Joe, about taking responsibility. I should have included that in the piece. Thanks for your commentary.
    Marc Rapport