Tradespace Exploration for better Verification Strategies

Abstract

This research, led by Dr. Alejandro Salado at Virginia Tech, addressed the definition of verification strategies in large-scale systems. Verification activities provide the evidence of contractual fulfillment. Thus, the importance of adequately defining verification activities in any acquisition program is unquestionable. Its significance extends beyond contracting though. The biggest portion of the development financial budget is spent in executing verification activities and verification activities are the main vehicle in discovering knowledge about the system, which is key to reduce development risk. In current practice, the definition of verification strategies is driven by industry standards and subject matter expert assessment. This approach leads to four major risks. First, there is a high uncertainty associated to the optimality of the selected verification strategy in terms of mitigated risk with respect to verification cost. Second, there is a lack of a quantitative risk measurement associated to chosen verification strategy, which jeopardizes any mindful effort to execute informed trade-off's regarding execution of verification activities. Third, there is a high risk associated to the verification coverage of the selected verification strategy, which threats the successful integration of components and the successful operation of the system. Fourth, there is a lack of alignment between stakeholder objectives and verification strategy, which leads to suboptimal decisions regarding the execution of verification activities.

In order to cope with these challenges, this research project addressed the main question of whether tradespace exploration can support the definition of more valuable verification strategies than current practice. In particular, the research had the following objectives: (1) To develop a metric for measuring the value of a verification activity, which will be achieved by elaborating a mathematical framework that computes the value of a verification activity as a function of the knowledge it discovers, and (2) To conduct a comparative analysis between tradespace exploration and a benchmark, which will be achieved by developing a tradespace exploration tool for verification strategies and use it to assess the value of verification strategies in actual, existing systems. The research employed a combination of theoretical, mathematical elaborations and practical algorithm development and implementation. The hypotheses were tested on an Air Force Institute of Technology notional satellite.

By fulfilling the research objectives, this research is anticipated to significantly improve the value and cost of verification strategies. Furthermore, the direct public benefit of this research is anticipated to be higher early safety and efficacy of commercial products and public services. Finally, while we consider an application for the Air Force as a test case, we anticipate that the methodologies and insights provided in this work can be applicable to a broad range of systems that are subjected to limited verification: other defense systems, space systems, aeronautics, automotive systems, manufacturing systems, electronic products, civil infrastructure, public health systems, or transportation systems. The results of this research have been disseminated through professional conferences and scientific journals.