Starliner Gives Boeing A Hard Lesson In How Not To Verify Software

test flight of reusable capsule
The Starliner was hoisted atop a United Launch Alliance Atlas V in preparation for its orbital flight test.
Credit: Cory Huston/NASA

Boeing’s CST-100 Starliner, a reusable capsule designed to ferry astronauts and fare-paying passengers into low Earth orbit, may look like previous generations of spacecraft, but appearances can be deceiving. 

“We are no longer building hardware in which we install a modicum of enabling software,” says Patricia Sanders, who chairs the long-standing NASA Aerospace Safety Advisory Panel (ASAP). “We are actually building software systems, which we wrap up in enabling hardware, yet we have not matured to where we are uniformly applying rigorous systems engineering principles to the design of that software.” 

  • 1 million lines of code to be reverified
  • NASA launches safety assessment of Boeing

Case in point: The Starliner, which returned from an abbreviated flight test rife with potential software problems. One error manifested shortly after launch when the capsule missed an orbit insertion burn to reach the International Space Station (ISS) because its timer was set 11 hr. ahead of the actual mission elapsed time. 

The problem was compounded by some type of interference—still under investigation—that prevented flight controllers from communicating with the Starliner via NASA’s Tracking and Data Relay Satellites. By the time communications were restored, the Starliner had expended so much fuel firing steering thrusters that managers decided to pass on the ISS docking and salvage what they could of the flight demonstration, including a successful deorbit, reentry and parachute landing in New Mexico. 

But the Starliner’s troubles were more serious than had been suspected. During its Feb. 6 meeting, the safety panel disclosed a second software error that was discovered as Boeing engineers pored over lines of code prior to the Starliner’s deorbit on Dec. 22. The error misconfigured a thruster, which could have caused the service module to bump into the crew capsule following separation after the deorbit burn. 

If the anomaly had not been corrected, “it would have led to erroneous thruster firing and uncontrolled motion during service module separation from deorbit, with the potential for catastrophic spacecraft failure,” says ASAP member Paul Hill, a former NASA flight director. 

The independent review team assessing the Starliner’s Orbital Flight Test (OFT) is not expected to complete its work until the end of February, but NASA and Boeing said on Feb. 7 that the panel had found multiple areas where the software verification process failed.

“The process broke down in many areas,” says NASA’s spaceflight chief Douglas Loverro. “We don’t know how many software errors we have. We don’t know if we have just two or we have many hundred.” The fault is not entirely Boeing’s, Loverro adds. “Our NASA oversight was insufficient—that’s obvious . . . and good learning for us.”

NASA also launched an organizational safety assessment into Boeing in an attempt to root out potential systemic issues. The agency last year conducted a similar review of SpaceX, which is developing a second crew transportation system. SpaceX is on track for a crewed flight test of its Dragon capsule this spring. 

“We need as a nation—and we have an obligation—to have multiple providers that can get us access to low Earth orbit,” NASA Administrator Jim Bridenstine told reporters on Feb. 7. “That’s part of what the Commercial Crew program is all about. We are working very hard to understand what the [Starliner] anomalies are, remedy those and move forward.”

NASA and Boeing agree it is too early to say if a second uncrewed Starliner test flight will be required prior to flying astronauts, a mission that had been targeted for this year. Nor would Boeing comment about how long it would take to reverify the Starliner’s entire flight software, about 1 million lines of code. 

“On the OFT mission, we successfully exercised about 66% of the scripts that we have, but we’re not taking that at face value,” says John Mulholland, Boeing Starliner program manager. “We are going to have our team go back and fully verify the correct implementation of all the software, in partnership with NASA.” 

For now, NASA is counting on SpaceX to continue the program’s momentum—particularly with ISS staffing ramping down as NASA’s contract for crew ferry flights aboard Russian Soyuz capsules comes to an end. Currently, the last U.S.-purchased seat is reserved for veteran NASA astronaut Chris Cassidy, who is scheduled to launch in April. 

Irene Klotz

Irene Klotz is Senior Space Editor for Aviation Week, based in Cape Canaveral. Before joining Aviation Week in 2017, Irene spent 25 years as a wire service reporter covering human and robotic spaceflight, commercial space, astronomy, science and technology for Reuters and United Press International.

Comments

5 Comments
I think we all know that things don't always go right the first time. SpaceX had their capsule issue, and the software in the Starliner needs to be fixed.
This article seems like the Starliner software team worked completely ignorant of what was done with Apollo, the Shuttle or the ISS. I had the privilege of being loosely associated with the IBM FSD team that provided the Shuttle software in the 1980s (and had some AP101 core memory software that I had written fly on missions until they were replaced with AP101S) and there are statements in this article that NEVER would have come out of that shop: in fact, they're cringe-worth.

The software development process used by the Shuttle team was not rocket science - it was basically Waterfall that followed the CMMI Level 5 verification approaches. NASA has a good paper explaining the complexity of the shuttle systems and software development process that was followed (although not the culture and quality of the people involved) here: ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20110014946.pdf

The quote at the start of the article really demonstrates a very high level of ignorance regarding the level of sophistication and quality that the software developed for the Shuttle and previous space platforms had.
Who wrote the software? Was it Boeing-Seattle or subcontracted to India?
I don't see how a second uncrewed Starliner test flight wouldn't absolutely be required to ensure integrity of the spacecraft. If it's not required, there is a deep problem somewhere.
Things can go wrong, indeed, but it is pityfull that Boeing cannot do better than this. SpaceX moves very fast and their problems with the abort system may have shown the price for this, but they always correct their problems very quickly, and Dragon seems on the path for a manned flight soon after the successful in flight abort test. Still, should I choose, I guess I would still prefer a Sojuz seat. NASA seems to have deep problems with its manned program: lack of oversight and failed schedules. Is the reason simply that they try to accomplish too much under an insufficient budget? Or that they operate much, much less effectively than does SpaceX. Jens Hoeg, Herlev, Denmark