What the Hawaii Missile Scare Can Teach Incident Response Teams
By Seth Jaffe.
Heads finally rolled over at Hawaii’s Emergency Management Agency. What can the incident response community take away from this latest real-life example? Procedures, Rules, and Communication Protocols, which are the underlying principles of a modern incident response program. I’ve written about all three in prior incident response posts, but let’s apply them to the missile scare.
We should consider procedures first, in view of a couple of takeaways from the Fox News article. Apparently, the employee who sent out the incorrect warning message “’froze’ and ‘seemed confused.” I’ve been there—in fact, several times during training at NASA. “Deer-in-the-headlights” is a right of passage, and good instructors purposefully attempt to box in a trainee for this type of experience.
How do you get operators moving again? Two ways. This first is a cultural issue. Operators need to have experienced this phenomenon before, to understand that mistakes do happen, and that one is expected to focus on the present workarounds, rather than fester on the past mistake. Only training and an above-board culture can accomplish this mindset. The second motivation comes from solid procedures. I’ve written about deer-in-the-headlights moments before. Comprehensive and mature procedures combat this phenomenon by providing clarity of direction in a time of need. Pull out the procedure and start at step one. Now you are moving again. When the step is complete, the operator moves to the next step and, surprisingly, the shock wears off and the operator finds herself back in the game. Most incident response teams do not have on-site backup team members, so getting this operator on her feet is fundamentally crucial to the mission.
We also learned from the article that the language of the alert message to the operator strayed from typical scripts, and included the phrase “THIS IS NOT A DRILL.” That raises the issue of Rules, about which I’ve also written. Rules capture decisions made by steering committees charged with dictating policy, and by response teams learning lessons from training exercises and real-life events. Rules are isolated, numbered documents that clarify “shalts” and “shall nots.” For example, thou shall not use the term “THIS IS NOT A DRILL” during live drills.
Finally, the FCC cited a miscommunication during a shift change as contributing to the mistake. That’s the kind of thing that can cripple a response team. If your incident response team does not employ periodic communication exercises, focusing on uniform vernacular, concise comm tactics that combat tautology, and methods to ensure the correct parties are involved, then you may want to consider budgeting some time for them. Effective communication can make or break an incident response team. More on that soon. Until then, keep your finger off the red missile button.
Seth Jaffe is the head of the Incident Response Division at LEO Cyber Security. He spent nearly 14 years as a NASA flight controller in Mission Control, where he was certified on the Space Shuttle and the International Space Station. As a controller, evaluator, and instructor, Seth trained candidates to react to time-sensitive emergency situations and to effectively communicate in the Mission Control environment. He took part in over 100 simulations and logged over 3000 hours flying the ISS, experience he draws upon in his incident response practice.
Seth is our official rocket scientist in residence. Hailing from NASA’s Mission Control Center, Seth brings a unique perspective to incident response, applying aspects of one of the world’s preeminent emergency operations platforms to cyber response. In addition to twenty-plus years’ of technical experience, Seth was previously a member of the data protection task force at a large law firm, and served as the lead Legal team member of an incident response team at a major U.S. airline. Seth is a certified business continuity professional, and he holds a juris doctorate, which is why he also wears the General Counsel hat at LEO.