Review and Analysis of GAO Reports on Major Weapon Systems

Abstract

DoD major weapon systems historically have been over budget, behind schedule, and typically under performing in terms of suitability. The Secretary of Defense proposed a fiscal year (FY) 2010 budget that ended or curtailed all or part of at least a half dozen major defense acquisition programs that were over cost, behind schedule, or no longer suited to meet the warfighters current needs. The Government Accountability Office (GAO) publishes a major DoD weapon systems report every March. The GAO spends a considerable amount of time evaluating several dozen major weapon system programs and provides a two-page analysis of the largest defense programs. This research project examines eight major DOD weapon systems using the yearly published GAO reports from 2003-2011. The purpose of this research is to understand what risk management was being performed on those programs, how it was implemented, when it was implemented, and how effective it was. The desired outcome is to make recommendations on improvements to the risk management process recommended by the Risk Management Guide for DoD Acquisition (DoD, 2006) More specifically, this research paper addresses risk mitigation activities that are considered best practices and that are documented in the yearly GAO Assessments of Selected Weapon Programs reports. Changes in cost, schedule, and performance have been tracked over time, which provides a unique perspective to understanding program execution. It is not possible to examine every aspect of eight programs over a nine-year period. This project has focused on some key systems engineering principles that are risk-mitigation activities. The activities studied are acquisition strategy, including acquisition phases, milestones, technical reviews, and major decision reviews. Also evaluated were technical maturity, design maturity, earned value, production maturity, and software design. An effective program acquisition strategy reduces program risks. Breaking the development into phases lowers the chances of designing or fielding a system that is immature. The milestone reviews are used to evaluate the system and to hold back programs that aren't mature enough to proceed to the next phase. Technical reviews are used to measure the progress of the program development and are event based to ensure the timely completion of the development effort. Developmental and operational tests are used to verify system performance before large sums of money are spent on production units. Technology Readiness Levels (TRLs) are a measure used by the Department of Defense to assess the maturity of an evolving technology prior to incorporating that technology into a system. Unfortunately, this research found that they are not a good indicator of program performance. Six of the eight programs studied identified their technologies as mature even though the programs experienced significant delays and cost overruns after the technologies had been identified as mature. In the GAO reports, design maturity was based on the number of released drawings. Some of the values are questionable because they are exact multiples of 10%. In many cases, the percent of drawings decreased after obtaining 98% or higher in a previous year. This suggests that much redesign had taken place, that the measurement of the original value was poor, and/or that the original drawing was released before the design had stabilized. Earned value management (EVM) is a project management technique for measuring project performance and progress in an objective manner. Earned value was required on all of the programs in the GAO reports because of their size. However, only a few programs mentioned EVM in the GAO reports (only 22 times in nine years). Even when mentioned, the data were often vague. In some cases, the EVM system was identified as broken. Therefore, EVM was not an effective risk management tool. Using Statistical Process Control (SPC) tools is a best practice for managing quality in a manufacturing environment. It is a data-driven method for monitoring and controlling the manufacturing processes that affect product quality. The GAO reports focused on the use of SPC, which was seldom used by contractors. Therefore, in most cases the risks of manufacturing problems could not be addressed. Production readiness reviews are also a best practice and should be used to address production risks. The production readiness review assesses the maturity of the design for going into production and can be an early indicator of future manufacturing problems. The GAO reports often neglected the role of software development in an acquisition program. Design maturity is not just the number of drawings released. That is only a measure of the physical hardware design. Software is usually very important and risky on large weapon programs. Any company that claims to be CMMI Level 2 or higher will be using metrics to manage their software projects. So, theoretically, performance metrics should exist and should be reported to manage the risk of software development problems.