Digital Engineering Enhanced T&E of Learning-Based Systems

Abstract

The design of test and evaluation (T&E) programs requires new thinking for learning-based systems enabled by AI. A critical question is how much information is needed about the training data, the algorithm, and the resulting performance for testers to adequately test a system. The answer to these questions will inform acquisition of data/model rights for learning‐based systems. The principal objective of this research is understand how increasing government access to the models and learning‐agents (AI algorithms) used in system design might decrease the need and expense of testing and increase confidence in results. The principal hypotheses investigated in this incubator project are that the number of samples needed to test AI/ML models to an acceptable degree of assurance ca be reduced if we have access to the models themselves (in mathematics or software), reduced still further if we also have access to the algorithms and data used to train the models, and reduced further yet if we also have access to systems models and other artifacts of the digital engineering process. Therefore, the cost of acquisition can be reduced if T&E programs are based on the optimal balance between the cost of acquiring the technical data/algorithm rights of AI/ML systems, and the cost of testing those systems. This research establishes theory and methods for exploring how T&E requirements can and should change as a function of the test team knowledge of the technical specifications of learn based systems (LBS).