A bespoke safety-improvement process that collects and analyzes pilot performance data has expanded American Airlines’ insight on flight crew performance, augmenting existing safety programs by focusing on an adherence to standard procedures instead of spotlighting deviations.

Launched in 2018, American’s Learning and Improvement Team (LIT) is part of the airline’s embracing of Safety-II—a concept that turns the accepted risk-assessment process on its head. Rather than analyze what went wrong, Safety-II focuses on understanding why things almost always go right and looks to adjust factors that influence the process, from training to broader cultural norms.

• Project observes pilots and conducts surveys, interviews
• Focus is on analyzing routine operations
• Initiative embraces Safety-II concept

“From a Safety-II perspective, the purpose of safety management is to ensure that as much as possible goes right, in the sense that everyday work achieves its stated purposes,” wrote human factors and system safety expert Erik Hollnagel in a 2013 paper credited for helping introduce the concept. “This cannot be done by responding alone, since that will only correct what has happened. Safety management must instead be proactive, so that adjustments are made before something happens and therefore affect how it happens or even prevent something from happening.”

Like many airlines, American has several well-developed and effective data-driven safety initiatives, including Line Operations Safety Assessments (LOSA), a Flight Operations Quality Assurance flight data monitoring program and an Aviation Safety Action Program. But the airline determined that maximizing its safety management system (SMS) and minimizing risk requires looking beyond these reactive, Safety-I-centric efforts.

Inspired by Hollnagel’s research, American started LIT and developed a program. The team modified his baseline model of four potential outcomes for desired, or “resilient,” behavior to fit the airline’s flight operations environment. American set up a dedicated team of pilots and other safety executives to develop the program. It christened its model LPAC, an acronym for its four desired general categories of pilot proficiency—learn, plan, adapt and coordinate—that form the model’s core.

The team settled on three sources for generating LPAC data: line-pilot observations, short surveys and more extensive one-on-one interviews, or “shop talks.” American uses an iterative approach with each process, developing initial frameworks and changing them as lessons are learned. While each step plays a key role in generating important data for LPAC, the line observation process has gotten particular attention from the LIT team due to its complexity.

“We decided early on we needed a way to quantify or maybe to limit the amount of data that was captured,” said Bogomir Glavan, American first officer and LIT member, during a recent presentation at the Flight Safety Foundation (FSF) International Air Safety Summit. “On a [2-hr.] flight, there’s so much taking place to potentially collect. We had to draw a line in the sand. We needed something tangible that we could quantify for observers, and also eventually, the end-user training and our line pilots.”

The team settled on about 30 proficiencies—the term it picked to describe behaviors or competencies—that are part of American’s standard operating procedures (SOP) for its pilots, Glavan said. It mapped each proficiency to one of the four LPAC behaviors and developed a master codebook, or a customized language, that observers and pilots alike use to help ensure consistency. Each proficiency has an associated code and examples of resilient behavior linked to it. For example, wake turbulence encounters and aircraft malfunctions fall under “address unanticipated new pressure,” which is one of the “adapt” proficiencies.

“Essentially, we’re looking at [a set of] different proficiencies, which are tangible ways for us to go out and measure the crew’s performance,” Glavan said. The team updates the proficiencies and associated language as warranted.

By mid-2020, American had data from about 100 line-flight observations. Combined with qualitative input from surveys and interviews, the dataset began to reveal some trends. Among them, captains consistently exhibit more resilient behavior. In other words, they demonstrate more of American’s specified SOP proficiencies than first officers. Similarly, a crew member acting as the pilot flying (PF) will exhibit more resilient behavior than when acting as the pilot monitoring (PM).

“As you can imagine, if you combine those two, the first officer as a pilot monitoring is not a good role as far as resilient performance from what we’ve seen so far,” Glavan said.

The team had some theories about the proficiency gaps but sought more substantive data. Last fall, it developed a voluntary survey for pilots. The survey was short but, like the proficiency codebook, it was carefully constructed. LIT sought input from NASA’s Human Performance Contributions to Safety team on survey construction and question phrasing.

Among the findings, captains were five times more likely to favor a PM role, while first officers were only 1.7 times more likely.

“This ‘desired’ crew pairing differs substantially from our observed flight deck resilient behavior captured by our proficiencies, which shows that [captains] as PF and [first officers] as PM exhibit the most resilient behavior during actual line flying,” a LIT-produced white paper on the program states.

“The knowledge captured is being used to develop more vigorous and comprehensive leadership and mentoring training for American Airlines pilots,” the white paper continues. “This enhanced training is in direct support of the ultimate goal of LIT, which remains to encourage and stimulate a better learning culture in [American Airlines’] flight decks, across the entire spectrum of operations.”

Despite taking an 11-month precautionary pause during the COVID-19 pandemic, American has completed about 140 LIT observations and plans to reach 200 by year-end. Even as the program is helping American modify certain types of training, such as captain leadership courses, it continues to evolve. After relying on LOSA observers at the outset, it shifted to dedicated LIT observers and, earlier this year, introduced specialized training for them. The combination of the training and the refined codebook has improved the consistency of data collected during observations, the team says.

Among LIT’s goals is to share its process with other American work groups as part of a broader goal of creating a more beneficial SMS.

“We think maybe we’re starting to scratch the surface of how you measure culture at a commercial airline,” Glavan said. “We hope to expand it to other departments. There’s no reason it has to stay within flight operations.”

The approach aligns with International Civil Aviation Organization SMS standards and best practices and follows an FSF call to embrace Safety-II.

“Data collection needs to expand from a focus on hazardous events to analysis of routine operational data,” the FSF says in its Learning From All Operations white paper issued in July. “Learning from everyday operational data and events can enhance safety management that is often based on a small subset of performance information, which may introduce avoidable but unrecognized consequences into the aviation system.”