Data labeling and AI evaluation firm Scale AI has announced a landmark partnership with the U.S. Department of Defense’s Chief Digital and Artificial Intelligence Office (CDAO) to develop a national standard for testing and evaluating Large Language Models (LLMs). The collaboration aims to create a government-owned, comprehensive framework to assess the performance, safety, and reliability of AI models for military and defense applications.
The initiative, part of the CDAO’s Task Force Lima, will leverage a customized version of Scale AI’s commercial evaluation platform. This will allow the DoD to rigorously test various commercial and open-source LLMs against criteria specifically designed for defense scenarios, moving beyond standard academic benchmarks. The goal is to create a standardized “pass/fail” system to determine if an AI model is suitable for deployment in sensitive operations.
Dr. Matthew Johnson, the CDAO’s lead for LLM evaluation, emphasized the need for a model-agnostic framework. “This is about creating a government-owned test and evaluation capability that allows us to understand which model is the right tool for a specific job,” Johnson stated. The framework will assess models on metrics including accuracy, fairness, cybersecurity, and resilience against adversarial attacks.
This partnership marks a significant step toward the operationalization of generative AI within the U.S. government. By establishing a trusted, in-house evaluation standard, the DoD can more confidently and securely integrate powerful AI technologies. It addresses critical national security concerns by ensuring that AI systems are thoroughly vetted before being used in contexts where reliability and security are paramount. The resulting standards could influence how other government agencies and allied nations approach AI adoption and regulation in the future.
English 
Deutsch