How Safe Are Agentic Frameworks? Examining Privacy and Security Risks

This study evaluates the security and privacy risks of ten agentic frameworks using three benchmarks. The selected agentic frameworks are: 1) AutoGPT, 2) AutoGen, 3) CrewAI, 4) SuperAGI, 5) Swarm, 6) OpenAI Agents Python, 7) Agent Zero, 8) Semantic Kernel, 9) Pydantic AI, and 10) Google ADK. We evaluated the security and privacy behavior of these frameworks using three benchmarks: 1) Agent Security Bench (ASB), 2) CONFAIDE, and 3) DecodingTrust.

Dataset Usage Instructions

This dataset contains two main files: an Excel workbook and a compressed JSON archive. The Excel workbook documents the selection and evaluation workflow used in the study, including how agentic frameworks and benchmarks were identified, filtered, and finalized. It also contains the summarized benchmark results reported in the paper. The JSON archive contains the raw output files generated during the benchmark experiments across the selected agentic frameworks.

1. Excel Workbook

The Excel workbook provides a structured overview of the framework selection, benchmark selection, and final results. It contains the following sheets:

Search StringThis sheet contains the search strings used to identify candidate agentic frameworks from GitHub.

Initial Frameworks SelectionThis sheet lists the initially identified agentic frameworks before applying the final selection criteria. It includes the broader pool of frameworks considered during the early screening stage.

Finalized_FrameworksThis sheet contains the final set of 10 agentic frameworks selected for evaluation. These frameworks were chosen based on the defined inclusion and exclusion criteria.

Benchmark SelectionThis sheet documents the benchmark selection process. It includes the initially identified security and privacy benchmarks and shows how the final three benchmarks were selected for the study.

Benchmarks_ResultsThis sheet contains the summarized experimental results for the selected frameworks across the three benchmarks: Agent Security Bench (ASB), CONFAIDE, and DecodingTrust.

Expiremental Detail

This sheet provides the experimental details required for replication, including the Python version, commit hashes, and experiment dates for each evaluated agentic framework. This information is intended to support future replication and validation of the reported results.

Framework Mechanisim

The sheet contains the source-code analysis of 10 agentic frameworks. We analyzed each framework’s source code with respect to three architectural mechanisms: prompt wrapping, memory management, and tool calling or handling. The main purpose was to understand why the framework results differ across the benchmarks.

2. JSON Results Archive

The compressed JSON archive contains the raw experimental outputs generated during benchmark execution. It is organized into three main folders, each corresponding to one benchmark used in the study:

ASBThis folder contains the raw security evaluation results from Agent Security Bench. It includes the outputs for all 10 evaluated agentic frameworks across different ASB attack categories.

CONFAIDEThis folder contains the raw privacy evaluation results from the CONFAIDE benchmark. It includes framework-level outputs across the CONFAIDE privacy tiers.

DecodingTrustThis folder contains the raw privacy evaluation results from the DecodingTrust benchmark. It includes outputs for the privacy context and few-shot settings, including one-shot, two-shot, and five-shot evaluations.

Each benchmark folder contains results for the same 10 agentic frameworks, enabling direct comparison across security and privacy settings. The JSON files can be used to verify the summarized results in the Excel workbook, reproduce the analysis, or conduct further framework-level investigation.