Home / Enhancing AI for Homeland Security

Overview

MAST for Evaluating Generative AI in Worker-Automation Team Tasks (MEGAWATT)

The Multisource AI Scorecard Table (MAST) for Evaluating Generative AI in Worker-Automation Team Tasks (MEGAWATT) project aims to leverage MAST, a tool rooted in Intelligence Community Directive (ICD) 203, to evaluate and enhance the performance of large language models (LLMs) like OpenAI’s ChatGPT. Focusing on intelligence and analysis (I&A) tasks crucial for homeland security, MEGAWATT explores the suitability of ChatGPT’s outputs, investigates the potential of few-shot learning to enhance its performance, and evaluates whether its dialogic modality fosters more trustworthy interactions compared to other AI platforms.

Solution 

Utilizing the MAST framework, MEGAWATT assesses ChatGPT’s responses to open-ended prompts across various I&A tasks, including text summarization, recognizing key details, and creating timelines from large amounts of text. The project investigates the feasibility of a method called few-shot learning, aiming to improve AI systems’ relevance and accuracy. This approach could make these AI systems more reliable and useful for decision-making in homeland security.

Impact 

MEGAWATT’s findings contribute to understanding the performance and appropriate use of generative AI systems like ChatGPT in critical intelligence tasks. By evaluating ChatGPT’s adherence to MAST criteria and exploring the efficacy of few-shot learning, the project seeks to enhance the trustworthiness and utility of AI-enabled decision support systems. Furthermore, MEGAWATT examines whether ChatGPT’s dialogic interaction mode enhances user trust and engagement, a feature not explicitly captured in the MAST criteria. The project’s outcomes aim to inform the adoption of AI technologies within the homeland security enterprise, promoting effective decision-making and risk management.

Research Leadership Team 

Principal Investigator:  Erin Chiou, Arizona State University
Co-PI: Mickey Mancenido, Arizona State University 

Homeland security risk sciences

Present

Discover more projects

The CAOE is committed to developing innovative tools and techniques to safeguard our homeland from potential threats and vulnerabilities.