HemingwAI: The Confidence Also Rises

Abstract

As the Department of War (DoW) increasingly adopts Large Language Models (LLMs) to accelerate mission-critical functions, trust in model outputs becomes essential. While LLMs offer significant capability gains, their susceptibility to hallucinations presents an unacceptable risk in high-consequence environments. This paper introduces HemingwAI, an LLM-agnostic, modular trust framework designed to detect hallucinations and quantify hallucination risks. HemingwAI evaluates factual accuracy of LLM responses alongside response completeness, relevance, and subjectivity, producing actionable risk signals to support informed decision-making. The HemingwAI framework is designed to integrate seamlessly into existing DoW workflows and to support deployment in secure and air-gapped environments. HemingwAI was evaluated using open-domain hallucination benchmarks (HaluBench); benchmark results show measurable improvement in hallucination detection rates above the baseline. More importantly, HemingwAI’s operational relevance was evaluated through an internal Subject Matter Expert (SME) guided pilot. ASRC Federal domain subject matter experts confirmed strong alignment between HemingwAI outputs and human evaluation, demonstrating that the tool can effectively reduce analyst review burden. The work presented in this paper positions HemingwAI as a foundational capability for trusted, mission-ready AI adoption across DoW programs.