Please use this identifier to cite or link to this item:
https://dair.nps.edu/handle/123456789/5034
Title: | Assessing DoD Confidence and Bias in AI/LLM Authored Evaluation Factors |
Authors: | Steven Hedgepeth, Ryan Mark Tagatac |
Keywords: | Artificial Intelligence Source Selection FAR DFARS ChatGPT |
Issue Date: | 27-Dec-2023 |
Publisher: | Acquisition Research Program |
Citation: | Published--Unlimited Distribution |
Series/Report no.: | Acquisition Management;NPS-AM-24-015 |
Abstract: | Artificial intelligence (AI)/Large Language Models (LLMs) have shown promise in various tasks, but their use in authoring source selection evaluation factors in the Department of Defense (DOD) is not well studied. Understanding the effectiveness of AI-authored evaluation factors is crucial for reliable decision-making. The integration of LLM technology in the DOD aligns with the rise of AI. This exploratory analysis investigated DOD acquisition professionals’ confidence in and bias toward AI-authored evaluation factors. Surveys at George Mason University (GMU) and Naval Postgraduate School presented professionals with requirements documentation and human or AI-generated evaluation factors. Due to statistically significant differences between the surveys, only the GMU data was relied on. Statistical and qualitative analyses evaluated variations in confidence ratings across different participant groupings and authorship disclosure. Results reveal reduced confidence and slight algorithm aversion to AI-authored factors versus human-authored, especially among older professionals. Despite limitations including sampling constraints, notable discrepancies emerge in perceptions of AI versus human outputs. Recommendations include the development of an AI guide to aid responsible use of AI in acquisitions. Further research with larger, varied samples and various AI tools is needed. This initial work advances AI integration policy discussions and public trust in defense acquisitions. |
Description: | Acquisition Management / Graduate Student Research |
URI: | https://dair.nps.edu/handle/123456789/5034 |
Appears in Collections: | NPS Graduate Student Theses & Reports |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
NPS-AM-24-015.pdf | Student Thesis | 32.76 MB | Adobe PDF | View/Open |
Student Poster.pdf | Student Poster | 662.85 kB | Adobe PDF | View/Open |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.