Benefit of Cross-Training Incident Response Team Members

By Seth Jaffe.

WARNING – Fire onboard the Space Shuttle. That’s the scenario that former Mission Operations Director Paul Hill leads off with in describing a great exchange between NASA flight controllers that demonstrates the importance of effective cross-training.[1] Paul is filling the role of Flight Director during a Shuttle simulation (during training simulations, NASA mixes certified flight controllers with trainees to give the trainees the benefit of working with seasoned veterans). Plenty of systems have already failed, from navigation systems, to propulsion systems, to electrical systems, but they’ve managed to perform a necessary burn in order to put the Shuttle on a trajectory to rendezvous with the MIR space station when a smoke detector rings in the avionics bay. Paul and the environmental officer are working the problem when the propulsion officer pops her head up with concern about rendezvousing with a fire onboard. Normally, an onboard fire would terminate a rendezvous, except that, in this case, there is not a fire onboard. Both the environmental officer and Paul know that a smoke detector warning is just the first sign of fire, but it needs to be confirmed. In this scenario, no other smoke detectors have rung, nor does the crew smell smoke. This is likely a sensor failure. The propulsion officer trainee, while educated enough to perform simulations at her console position, probably has not come up to speed on some of the other systems yet. Fortunately, the Flight Director is there to cut short what could be a lengthy and wasteful interaction, and keep the mission on track.

This same scenario occurs more often than not in both incident response table top exercises as well as real events. Various team members spend precious time chasing a non-event because they are not trained in each other’s subjects, and leadership in the form of a strong Director is not present to shut it down. In a prior article, I discussed the importance of a dedicated Director office staffed with personnel trained on each of the core discipline subjects. The same holds true for the core team members themselves.

Why? Because of the “Go See the Doctor” Doctrine.

Imagine your family physician, midway through your annual check-up, suddenly remarking “You need surgery. Be here tomorrow at 8:00 am and don’t eat anything,” and then walking out of the room. That might be a bit scary, and raise a few questions. What type of surgery, for what, and what are the risks? Why can’t I eat, and for how long before? Do I need to bring someone with me to take me home? Will I be admitted to the hospital? This may seem extreme as an analogy, but it happens all the time with trainees in Mission Control. They become so well versed in their respective areas of expertise that they lose the ability to communicate effectively with other disciplines. One of the ways to combat this problem is cross-training, such that each discipline is familiar with the level of knowledge of every other discipline. Each has its own vernacular, each its own procedures and protocols. Cross-training pulls the vernacular into a common language and helps each discipline forecast the actions of its counterparts.

Evaluators could always identify the “newbies” in a simulation. They were the ones calling the Propulsion Officer during an engine failure or the Controls Officer during a loss of attitude control. That person is busy, and does not have the bandwidth to answer low priority questions in the face of a critical failure. A good Flight Director will insulate that person from non-important requests, position the other team members into a supporting posture, pause the timeline as needed, consider the next worst failures, and put in motion recovery actions.

A good incident response team will introduce each discipline member to the capabilities, concerns, procedures, and directives of the other disciplines. Cross-training better prepares them to communicate with each discipline efficiently, concisely, and effectively.

[1] See Paul Sean Hill, “Leadership from the Mission Control Room to the Boardroom.”

Seth is our official rocket scientist in residence. Hailing from NASA’s Mission Control Center, Seth brings a unique perspective to incident response, applying aspects of one of the world’s preeminent emergency operations platforms to cyber response. In addition to twenty-plus years’ of technical experience, Seth was previously a member of the data protection task force at a large law firm, and served as the lead Legal team member of an incident response team at a major U.S. airline. Seth is a certified business continuity professional, and he holds a juris doctorate, which is why he also wears the General Counsel hat at LEO.


Leave a Comment