sop-bench

# SOP-Bench: Complex Industrial SOPs for Evaluating LLM Agents [](https://creativecommons.org/licenses/by-nc/4.0/) [](https://www.python.org/downloads/) 📄 **Paper:** [SOP-Bench: Complex Industrial SOPs for Evaluating LLM Agents](https://arxiv.org/abs/2506.08119) 🏭 **Human Expert-Authored SOPs** · 🤖 **Human-AI Collaborative Framework** · 📊 **Executable Interfaces** · 🔧 **Two Agent Architectures** · 📈 **11 Frontier Models Evaluated** ## Dataset Description - **Repository:** [amazon-science/SOP-Bench](https://github.com/amazon-science/SOP-Bench) - **Paper:** SOP-Bench: Complex Industrial SOPs for Evaluating LLM Agents (KDD 2026 Datasets and Benchmarks Track) - **Point of Contact:** [GitHub Issues](https://github.com/amazon-science/SOP-Bench/issues) - **License:** CC-BY-NC-4.0 ## Dataset Summary **SOP-Bench** is a comprehensive benchmark for evaluating LLM-based agents on complex, multi-step Standard Operating Procedures (SOPs) that are fundamental to industrial automation. Built from 2,000+ tasks across 12 industrial domains (healthcare, logistics, finance, content moderation, etc.), SOP-Bench addresses the gap between existing benchmarks and real-world procedural complexity. Standard Operating Procedures (SOPs) are the backbone of industrial operations—from content moderation to healthcare intake to supply chain logistics. These multi-step procedures require: - **Sequential reasoning** across 10-50+ decision points - **Tool orchestration** to gather information from multiple systems - **Implicit knowledge** that humans learn but rarely document - **Ambiguity handling** when procedures don't cover edge cases ## Key Findings Early findings from evaluation of 11 frontier models: - **Function-Calling agents**: ~64% average task success rate - **ReAct agents**: ~55% average task success rate - **High execution rates (95%+)** indicate failures are reasoning-based, not technical - **Open-source models** (DeepSeek-R1, Llama 3.3) approach proprietary performance - **Architecture-model co-design matters**: Newer reasoning models can degrade ReAct performance without targeted prompt engineering ## Dataset Structure ### Configurations (Subsets) Each configuration represents a distinct industrial domain benchmark: | Configuration | Domain | Description | Tasks | Complexity | |---|---|---|---|---| | `aircraft_inspection` | Aviation | Pre-flight safety inspections following aviation procedures | 150 | 9/10 | | `content_flagging` | Content Moderation | Bot detection, trust scoring, and violation assessment | 226 | 9/10 | | `customer_service` | Support | Diagnose and resolve customer service issues using system diagnostics | 208 | 8/10 | | `dangerous_goods` | Supply Chain | Classify dangerous goods using safety data sheets and scoring systems | 327 | 7/10 | | `email_intent` | Retail | Categorize seller support emails and route appropriately | 122 | 7/10 | | `know_your_business` | Finance | Verify business entities for compliance and risk assessment | 122 | 9/10 | | `order_fulfillment` | Supply Chain | Order processing and fulfillment workflows | — | — | | `patient_intake` | Healthcare | Register new patients with insurance and medical history validation | 90 | 7/10 | | `referral_abuse_detection_v1` | Fraud Detection | Detect referral program abuse patterns (v1) | — | — | | `referral_abuse_detection_v2` | Fraud Detection | Detect referral program abuse patterns (v2) | — | — | | `traffic_spoofing_detection` | Cybersecurity | Detect spoofed traffic patterns | — | — | | `video_annotation` | Autonomous Driving | Detect and annotate objects in driving videos | 168 | 10/10 | | `video_classification` | Media | Classify and moderate user-generated video content | 198 | 9/10 | | `warehouse_package_inspection` | Logistics | Inspect packages for damage and compliance | 200 | 9/10 | ### Data Files Per Configuration Each benchmark configuration contains: | File | Description | |---|---| | `test_set_with_outputs.csv` | Test cases with ground truth outputs | | `test_set_without_outputs.csv` | Test cases without outputs (for blind evaluation) | | `sop.txt` | The Standard Operating Procedure document | | `tools.py` | Python tool implementations available to the agent | | `toolspecs.json` | Tool specifications in JSON format | | `metadata.json` | Input/output column definitions | Some configurations also include `data.csv` (full dataset), `croissant.json`, `LICENSE.md`, and `disclaimer.txt`. ### Data Fields Each configuration has different input/output columns defined in its `metadata.json`. For example: **customer_service:** - Input columns: `account_id`, `service_area_code`, `service_type`, `subscribed_bandwidth` - Output columns: `final_resolution_status` **aircraft_inspection:** - Input columns: `aircraft_id`, `tail_number`, `maintenance_record_id`, `expected_departure_time`, `component_serial_number`, ... - Output columns: `aircraft_ready`, `mechanical_inspection_result`, `electrical_inspection_result`, ... ## Usage ### Loading with 🤗 Datasets ```python from datasets import load_dataset # Load a specific benchmark configuration dataset = load_dataset("amazon/sop-bench", "customer_service") # Access the test split test_data = dataset["test"] print(test_data[0]) # Load another benchmark dangerous_goods = load_dataset("amazon/sop-bench", "dangerous_goods") ``` ### Loading Supporting Files The SOP documents, tools, and metadata are also available in the repository: ```python from huggingface_hub import hf_hub_download # Download the SOP document for a benchmark sop_path = hf_hub_download( repo_id="amazon/sop-bench", filename="data/customer_service/sop.txt", repo_type="dataset" ) # Download tool specifications toolspecs_path = hf_hub_download( repo_id="amazon/sop-bench", filename="data/customer_service/toolspecs.json", repo_type="dataset" ) ``` ### Full Evaluation Framework For running evaluations with the full agent framework, use the GitHub repository: ```bash git clone https://github.com/amazon-science/SOP-Bench.git cd SOP-Bench pip install -e . # List available benchmarks ./sop-bench list # Evaluate on a single task ./sop-bench evaluate content_flagging --agent function_calling --max-tasks 1 ``` ## Example: What Does an SOP Task Look Like? Here's a simplified example from the **Dangerous Goods** benchmark: **SOP Instruction (excerpt):** > *"1. Retrieve the Safety Data Sheet for the product. 2. Check if the product contains any Class 3 flammable liquids. 3. If flash point < 23°C, classify as Packing Group I..."* **Task Input:** ```json { "product_id": "CHEM-2847", "shipment_type": "air_freight" } ``` **Expected Agent Behavior:** 1. Call `get_safety_data_sheet(product_id="CHEM-2847")` 2. Call `check_hazard_class(sds_id="SDS-2847")` 3. Call `get_flash_point(sds_id="SDS-2847")` 4. Apply classification logic from SOP 5. Return: `{"classification": "Class 3", "packing_group": "II"}` **Ground Truth:** `packing_group: II` The agent must correctly orchestrate tools AND apply the SOP's decision logic. ## Evaluation Metrics - **Task Success Rate (TSR)**: Percentage of tasks where the agent made the correct decision - **Execution Completion Rate (ECR)**: Percentage of tasks that completed without errors - **Conditional Task Success Rate (C-TSR)**: Of the tasks that completed execution, how many were accurate? - **Tool Accuracy**: Percentage of tool calls that were correct **Note**: TSR = ECR × C-TSR ## Agent Types | Agent | Description | Best For | |---|---|---| | `function_calling` | Native Bedrock Converse API | Structured tool use | | `react` | Custom ReAct loop (recommended) | All model families | ## Adding Your Own Benchmarks SOP-Bench is extensible. Create new benchmarks with: ``` benchmarks/data/your_benchmark/ ├── sop.txt # Natural language procedure ├── tools.py # Tool implementations ├── toolspecs.json # Tool schemas for LLM ├── data.csv # Test cases with ground truth └── metadata.json # Configuration ``` See the [Adding Benchmarks Guide](https://github.com/amazon-science/SOP-Bench/blob/main/docs/ADDING_BENCHMARKS.md) for details. ## Dataset Creation ### Curation Rationale SOP-Bench was created to address the gap between existing LLM benchmarks and real-world procedural complexity. Existing benchmarks focus on isolated tasks, while industrial SOPs require multi-step reasoning, tool orchestration, and handling of ambiguous edge cases. ### Source Data All SOPs are **human expert-authored** based on real industrial procedures. Test cases were created through a **human-AI collaborative framework** to ensure coverage of edge cases and realistic complexity. ### Annotations Ground truth outputs were generated by human experts following the SOPs with access to the mock tools, ensuring correctness of the expected agent behavior. ## Considerations for Using the Data ### Social Impact This benchmark enables evaluation and improvement of LLM agents for industrial automation tasks. Improved agent performance could reduce human error in safety-critical procedures (aviation, healthcare) while also potentially displacing human workers in routine procedural tasks. ### Known Limitations - Mock tools return deterministic outputs; real-world tools may have latency, errors, or non-deterministic behavior - SOPs are simplified versions of real industrial procedures - Performance on SOP-Bench may not directly translate to production deployment readiness ## Citation If you use SOP-Bench in your research, please cite: ```bibtex @inproceedings{sopbench2026, title={SOP-Bench: Complex Industrial SOPs for Evaluating LLM Agents}, author={Nandi, Subhrangshu and Datta, Arghya and Vichare, Nikhil and Nama, Rohith and Patel, Udita and Bhattacharya, Indranil and Asija, Shivam and Gupta, Arushi and Carenini, Giuseppe and Xu, Jing and Ray, Shayan and Raja, Huzefa and Chan, Aaron and Carbone, Francesco and Fei, Esther Xu and Du, Gaoyuan and Akhtar, Zuhaib and Grover, Prince and Bhaduri, Sreyoshi and Chen, Weian and Zhang, Wei and Xiong, Ming}, booktitle={KDD}, year={2026} } ``` ## License This dataset is licensed under [Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)](https://creativecommons.org/licenses/by-nc/4.0/). ## Contributing We welcome contributions! See [CONTRIBUTING.md](https://github.com/amazon-science/SOP-Bench/blob/main/CONTRIBUTING.md) for guidelines. - 🐛 [Report bugs](https://github.com/amazon-science/SOP-Bench/issues) - 💡 [Request features](https://github.com/amazon-science/SOP-Bench/issues) - 📖 [Improve docs](https://github.com/amazon-science/SOP-Bench/pulls) - 🔬 [Add benchmarks](https://github.com/amazon-science/SOP-Bench/blob/main/docs/ADDING_BENCHMARKS.md)