Certifying Tools for Test Reduction in Open Architecture

Abstract

In this paper, we describe a method for evaluating tools that can be used to guide decisions about how much retesting is needed and to check conditions under which testing of unmodified components can be reduced or avoided. The approach uses a combination of dependency analysis applied to source code and automated testing applied to executable component implementations. Dependability of such tools is a key concern in this context, which our ongoing research addresses. We also discuss other applications of software dependency analysis, such as risk-based testing, and discuss applications of dependency analysis to improve acquisition processes in the context of open architectures (OA). The Navy's OA framework is intended to promote reuse, improve system flexibility, and reduce costs. In this paper, we apply open architecture principles to reduce testing effort and costs in cases where the requirements and code for a subsystem have not been changed, but the component will be used together with new or modified components that may include a new version of the operating system. This situation is common in the Navy due to technology advancement upgrades and accounts for a substantial fraction of the testing cost. Applying traditional U.S. Navy weapon and combat system test and evaluation (T&E) practices, which currently include manual retesting after each system modification, to future OA systems will nullify many of the benefits that OA brings to the table, such as system scalability, rapid configuration changes, and effective component reuse. Combining (1) Naval Postgraduate School (NPS) research on dependency analysis focused on determining when it is safe not to retest a component with (2) automated software testing should enable these benefits and keep resource requirements at feasible levels.