This collection includes the full proceedings of each annual acquisition research symposium, individual proceedings papers, presentations and NPS graduate student research posters since 2004. https://dair.nps.edu/handle/123456789/15 2026-06-05T04:03:25Z 2026-06-05T04:03:25Z Alexandra Adams, Protima Banerjee Theresa Cauble, Noah Pape Agam Singh, Eric Toa https://dair.nps.edu/handle/123456789/5497 2026-05-18T16:51:56Z 2026-04-30T00:00:00Z

Title: HemingwAI: The Confidence Also Rises Authors: Alexandra Adams, Protima Banerjee; Theresa Cauble, Noah Pape; Agam Singh, Eric Toa Abstract: As the Department of War (DoW) increasingly adopts Large Language Models (LLMs) to accelerate mission-critical functions, trust in model outputs becomes essential. While LLMs offer significant capability gains, their susceptibility to hallucinations presents an unacceptable risk in high-consequence environments. This paper introduces HemingwAI, an LLM-agnostic, modular trust framework designed to detect hallucinations and quantify hallucination risks. HemingwAI evaluates factual accuracy of LLM responses alongside response completeness, relevance, and subjectivity, producing actionable risk signals to support informed decision-making. The HemingwAI framework is designed to integrate seamlessly into existing DoW workflows and to support deployment in secure and air-gapped environments. HemingwAI was evaluated using open-domain hallucination benchmarks (HaluBench); benchmark results show measurable improvement in hallucination detection rates above the baseline. More importantly, HemingwAI’s operational relevance was evaluated through an internal Subject Matter Expert (SME) guided pilot. ASRC Federal domain subject matter experts confirmed strong alignment between HemingwAI outputs and human evaluation, demonstrating that the tool can effectively reduce analyst review burden. The work presented in this paper positions HemingwAI as a foundational capability for trusted, mission-ready AI adoption across DoW programs. Description: 23rd Annual Acquisition Research Symposium and Innovation Summit

2026-04-30T00:00:00Z Christina Joseph Symantha “Sam” Loflin https://dair.nps.edu/handle/123456789/5438 2025-05-13T22:05:15Z 2025-05-13T00:00:00Z

Title: The Secret Sauce of Program Management Is the Best Defense to Mitigate Contract Risk Authors: Christina Joseph; Symantha “Sam” Loflin Abstract: "The author has written this paper to defend and strengthen the use of government initiatives, industry, and academia risk mitigation measures that prevent divergence from successful Program Management (PM) with the framework of Contract Management (CM), and the integration of Earned Value Management (EVM), and Agile methodologies and practices. On December 14, 2016, Public Law No: 114-264, the “Program Management Improvement Accountability Act [PMIAA]” was signed into law (H.R. 114-637S.1550, 2016). This law was enacted to improve program and project management practices within the federal government by requiring agencies to conduct [document] annual portfolio reviews of “high risk” programs that the Government Accountability Office (GAO) identified. Additionally, the PMIAA establishing a Program Management Improvement Officer (PMIO), who will “assess the quality and effectiveness of program management” (2016). These measures will highlight the possibilities of future performance growth, increased demand, and technological advancements in the Defense Industrial Base (DIB) (DoD, 2022). Additionally, improvement in workforce acquisition career paths and skill levels (H.R. 114-637S.1550, 2016). At the onset, the “delivery of performance [will be] at the speed of relevance” (Mattis, 2018, p. 10). Effective and efficient PM requires a solid foundation of knowledge and the framework of CM, with the integration of EVM, and Agile methodologies and practices. These disciplines will provide the capabilities required to maximize innovation, mitigate contract risk, and develop the workforce that supports the proper stewardship of taxpayer dollars. In February 2022, the National Defense Industrial Association reported that U.S. national security interests are at risk given the declining health of the DoD’s supply chain, surge readiness, and production capacity. It is essential to communicate and collaborate with all stakeholders to develop and grow the DIB and engage the workforce with the right people, processes, and tools at the right time. " Description: SYM Paper

2025-05-13T00:00:00Z Marissa McLaren https://dair.nps.edu/handle/123456789/5437 2025-05-13T22:02:24Z 2025-05-13T00:00:00Z

Title: Identifying Downtime Drivers Using SIMLOX Simulations to Rapidly Develop Solutions Improving System and Mission Readiness Authors: Marissa McLaren Abstract: "In this paper we present a simulation model that directly relates the system results to the individual parts and resources, otherwise known as downtime drivers. Understanding operational downtime drivers and whether they are item- or subsystem-specific is crucial to making strategic decisions for missions and operations. Identifying the drivers in the modeling phase allows for increased preparation and problem-solving to improve mission requirements. While working to solve the issues created by downtime drivers, industry and defense can work together to determine a reasonable solution to overcome the impact an item or subsystem can have on the overall system. This case study describes a scenario where industry and defense have been able to identify downtime drivers for a complicated system and develop a set of reasonable alternatives to address these issues. OPUS10 identifies the initial spares purchase optimization for a given availability requirement. We can then utilize that recommendation in SIMLOX, a Monte Carlo–based simulation tool. Simulation results are often used to identify bottlenecks within the supply chain, spares, and support organization. With the recent software updates to SIMLOX, we can identify downtime drivers. Stakeholders can identify which subsystem(s) or items are causing a system to be down. " Description: SYM Paper

2025-05-13T00:00:00Z Ryan Bell Ryan Longshore Raymond Madachy https://dair.nps.edu/handle/123456789/5436 2025-05-13T21:59:47Z 2025-05-13T00:00:00Z

Title: Exploring Visual Question Answering Capabilities of Multi-Modal Large Language Models with Model Based Systems Engineering Models Authors: Ryan Bell; Ryan Longshore; Raymond Madachy Abstract: "The continued advancement of large language models (LLMs) has unlocked new opportunities for systems engineering particularly in the field of visual question answering (VQA). Multi-modal LLMs are capable of processing both textual and graphical inputs, allowing them to interpret the graphical elements of model-based systems engineering (MBSE) models alongside accompanying textual descriptions. This paper explores the capabilities of multi-modal LLMs in understanding and interpreting Systems Modeling Language (SysML) v1 block definition diagrams (BDDs). BDDs are visual diagrams that formally capture a system’s structural elements, properties, relationships, and multiplicities. We evaluate both proprietary and open-source multi-modal LLMs using a curated dataset of SysML BDDs and associated multiple-choice question set designed to assess LLM performance at the first two levels of Bloom’s Taxonomy, Remember and Understand. We also analyzed the effect of model size on accuracy. The results provide insights into which current LLMs are able to natively interpret SysML BDD syntax which informs future research aimed at enhancing systems modeling processes with AI agents." Description: SYM Paper

2025-05-13T00:00:00Z