Decision-Based Metrics for Test and Evaluation Experiments

Abstract

We develop a new decision-based metric for determining sample sizes in Test and Evaluation experiments. Traditional confidence intervals for the mean can be used, and we present sequential confidence interval procedures as a way to derive efficient intervals. We discuss decision rules for analyzing the observed output and how to choose confidence interval methods for calibrating these decision rules. The metric presented can help determine if a fast decision on the quality of the system can be made or if many more tests are needed to ensure an accurate estimate of performance relative to a desired standard.