While it is no secret that virtually all major U.S. defense acquisition programs are over budget and behind schedule, it is not clear who is to blame or, more importantly, what should be done about it.

The problem is particularly acute in aerospace, where chronic delays to high-profile, complex programs such as the F-35 Joint Strike Fighter are largely blamed on the test community. In response, evaluation and development specialists in the industry are studying new initiatives that are designed to make them part of the solution, rather than the problem.

“There is a belief [in government] that development, test and evaluation and operational, test and evaluation communities are culpable for this,” says U.S. Air Force Lt. Gen. (ret.) George Muellner, former president of Boeing Integrated Defense Systems Advanced Systems as well as the American Institute of Aeronautics and Astronautics. “They believe the testers are the enemy—that's the fairly prevalent view in Washington and at Wright Patterson AFB, [Ohio].”

Citing a 2011 Government Accountability Office (GAO) report that identified $402 billion in budget over-runs and schedule slippages of up to 22 months for the largest acquisition programs, Muellner says that “most major weapons systems development cost overruns are in excess of 30 percent, and because of that, several major defense acquisition programs fail.”

At the Society of Experimental Test Pilots symposium in Anaheim, Calif., in September, Muellner said the common thread to many of the issues is the increasing complexity and integration of systems. “Software flight testing has become enormous,” he said. “There's clearly a belief in the minds of the GAO and those covering other oversight issues that we are not doing a good job of adequately dealing with [it].”

But he questions how much of this is the fault of evaluators. “Is it because we test poorly or the software is written poorly? Do we, as testers, drive these cost and schedule overruns—or do we get saddled with that?” he asked.

Part of the answer lies in the findings of a team put together by Undersecretary of Defense for Acquisition Frank Kendall to delve into the true issues involving test and evaluation (T&E). It found that, as a percentage of program costs, development testing (DT) accounted on average for 8-10%, while operational test (OT) accounted for 1.5%. Although these numbers remained relatively consistent, the expenditure on DT/OT was also found to have occurred late in each program, when the margin was depleted. “So it is a constant part of a significantly increased number,” Muellner notes.

Other findings included the fact that T&E costs are significantly underestimated, particularly because the complexity of new systems has not been well-understood. Software verification processes for such systems also drives unplanned software releases, leading to unanticipated test sorties and program delays. Moreover, today's acquisition streamlining is being affected by cutbacks in the military-industrial complex. Researchers looking for input into the F-22 pilot's oxygen system problem, for example, “went to look for the air medical people at Brooks, but it been turned into an industrial park,” says Muellner, referring to Brooks AFB, the former site of the aerospace medical school in San Antonio, that was closed in 2011.

The GAO report concluded that testers had not in fact played a significant part in the endemic problems. However, it did uncover major issues, including weak alignment of the requirement, development and test communities. “All three treat it as a serial process and, as a result, all the trades that need to occur don't get done. It's a key part of the problem,” he says. The report also noted serious flaws in systems engineering and the obvious point that without improvements in this overarching discipline, problems with inadequate software will persist. In particular, it also found poor alignment between acquisition and testing strategies.

“So, we are not the guilty folks. But on the other hand, we're not that clean,” Muellner says. “We can be part of the solution, but we need to change the way we approach the overall acquisition programs.” Key recommendations to improve the role of T&E include greater up-front involvement in the grass roots of all new programs, long before design—let alone assembly—is begun. “We must be involved in program planning to insure requirements are testable and that adequate resources are provided for realistic test-planning factors,” says Muellner.

The recommendation is already gaining traction in some parts of the industry. Boeing, for example, is implementing changes executed by the leader of its test organization, Dennis O'Donoghue, that will involve T&E from the earliest phases of all new product development programs.

“We need to work the software piece very hard,” Muellner adds. “The Scientific Advisory Board will be pursuing a study to address this issue. As systems become more complex and integrated, developers and testers must identify and implement new approaches to software development and testing to reduce cost and schedule impact.”