電腦科學與資訊工程科 Computer Science & Information Engineering
190020 Taiwan
AI 資安分析師:基於 RAG 的全自動化威脅情資生成與評估 之實證研究 AI Cybersecurity Analyst: An Empirical Study of RAG-based Fully Automated Threat Intelligence Generation and Evaluation
Abstract Amidst increasingly severe cybersecurity threats, this research investigates the feasibility of deploying localized Large Language Models (LLMs) on consumer-grade hardware, aiming to create an ""AI Cybersecurity Analyst"" utilizing Retrieval-Augmented Generation (RAG). We successfully implemented an automated RAG-based threat intelligence generation system, constructing a knowledge base from over 14,000 documented attack incidents. To overcome the inherent bottlenecks of small and medium-sized LLMs (SMLs) in cross-lingual tasks, we experimented with a ""Two-Stage Prompt Chain."" This methodology decomposes complex tasks (i.e., intelligence extraction followed by answer synthesis), achieving a stable improvement in faithfulness and relevance metrics to over 90%.
For scientific validation, we established an ""AI-evaluates-AI"" automated evaluation framework, which leverages LLMs to serve as both question generators and evaluators. This framework analyzes performance based on metrics including hit rate (retrieval accuracy), faithfulness, and relevance. Our experiments identified the combination of Mistral 7B (for generation) and BAAI/bge-large-en-v1.5 (for retrieval) as optimal.
Furthermore, the data validates an ""inverted U-shaped curve"" in RAG performance, confirming the trade-off between ""information gain"" and ""context noise penalty."" This finding demonstrates that excessive contextual information negatively impacts faithfulness. This study not only delivers an affordable and efficient prototype but also establishes a fully automated RAG construction and evaluation framework, providing empirical evidence for the viability of localized AI deployment.