Starliner Gives Boeing A Hard Lesson In How Not To Verify Software

Boeing’s CST-100 Starliner, a reusable capsule designed to ferry astronauts and fare-paying passengers into low Earth orbit, may look like previous generations of spacecraft, but appearances can be deceiving. 

“We are no longer building hardware in which we install a modicum of enabling software,” says Patricia Sanders, who chairs the long-standing NASA Aerospace Safety Advisory Panel (ASAP). “We are actually building software systems, which we wrap up in enabling hardware, yet we have not matured to where we are uniformly applying rigorous systems engineering principles to the design of that software.” 

• 1 million lines of code to be reverified
• NASA launches safety assessment of Boeing

Case in point: The Starliner, which returned from an abbreviated flight test rife with potential software problems. One error manifested shortly after launch when the capsule missed an orbit insertion burn to reach the International Space Station (ISS) because its timer was set 11 hr. ahead of the actual mission elapsed time. 

The problem was compounded by some type of interference—still under investigation—that prevented flight controllers from communicating with the Starliner via NASA’s Tracking and Data Relay Satellites. By the time communications were restored, the Starliner had expended so much fuel firing steering thrusters that managers decided to pass on the ISS docking and salvage what they could of the flight demonstration, including a successful deorbit, reentry and parachute landing in New Mexico. 

But the Starliner’s troubles were more serious than had been suspected. During its Feb. 6 meeting, the safety panel disclosed a second software error that was discovered as Boeing engineers pored over lines of code prior to the Starliner’s deorbit on Dec. 22. The error misconfigured a thruster, which could have caused the service module to bump into the crew capsule following separation after the deorbit burn. 

If the anomaly had not been corrected, “it would have led to erroneous thruster firing and uncontrolled motion during service module separation from deorbit, with the potential for catastrophic spacecraft failure,” says ASAP member Paul Hill, a former NASA flight director. 

The independent review team assessing the Starliner’s Orbital Flight Test (OFT) is not expected to complete its work until the end of February, but NASA and Boeing said on Feb. 7 that the panel had found multiple areas where the software verification process failed.

“The process broke down in many areas,” says NASA’s spaceflight chief Douglas Loverro. “We don’t know how many software errors we have. We don’t know if we have just two or we have many hundred.” The fault is not entirely Boeing’s, Loverro adds. “Our NASA oversight was insufficient—that’s obvious . . . and good learning for us.”

NASA also launched an organizational safety assessment into Boeing in an attempt to root out potential systemic issues. The agency last year conducted a similar review of SpaceX, which is developing a second crew transportation system. SpaceX is on track for a crewed flight test of its Dragon capsule this spring. 

“We need as a nation—and we have an obligation—to have multiple providers that can get us access to low Earth orbit,” NASA Administrator Jim Bridenstine told reporters on Feb. 7. “That’s part of what the Commercial Crew program is all about. We are working very hard to understand what the [Starliner] anomalies are, remedy those and move forward.”

NASA and Boeing agree it is too early to say if a second uncrewed Starliner test flight will be required prior to flying astronauts, a mission that had been targeted for this year. Nor would Boeing comment about how long it would take to reverify the Starliner’s entire flight software, about 1 million lines of code. 

“On the OFT mission, we successfully exercised about 66% of the scripts that we have, but we’re not taking that at face value,” says John Mulholland, Boeing Starliner program manager. “We are going to have our team go back and fully verify the correct implementation of all the software, in partnership with NASA.” 

For now, NASA is counting on SpaceX to continue the program’s momentum—particularly with ISS staffing ramping down as NASA’s contract for crew ferry flights aboard Russian Soyuz capsules comes to an end. Currently, the last U.S.-purchased seat is reserved for veteran NASA astronaut Chris Cassidy, who is scheduled to launch in April.