Swarm-Steward Benchmark Data: Feature RAG, Swarm Scalability, and LLM Model Evaluation Artifacts

Abstract Swarm-Steward is a platform-agnostic system for natural-language swarm control that enables a non-expert operator to coordinate many drones by creating and commanding groups, bridging high-level intent to reliable, low-level execution. The system uses a hierarchical LLM-based multi-agent design where planning and context gathering are separated from actuation: a tool-less Coordinator decomposes each request into staged sub-tasks and delegates them to specialised agents — a Spatial agent and a History agent for non-actuating context gathering (map relations and telemetry), and a Swarm agent as the sole authority permitted to modify world state by dispatching deterministic, schema-constrained commands through a safety gate (geofencing, altitude and separation limits), with optional operator preview before execution. To keep grounding scalable, Swarm-Steward applies dual retrieval-augmented generation over both map features (Feature RAG) and telemetry variables (State RAG), injecting only relevant candidates at each step. The system supports both MQTT and ROS 2/DDS communication backends and has been validated in simulation as well as on real DJI Mini 4 Pro drones via WildBridge. This dataset accompanies the paper "Swarm-Steward: Scalable and Reliable Natural-Language Coordination of Autonomous Aerial and Ground Robots", submitted to the International Conference on Unmanned Aircraft Systems (ICUAS 2026). It provides the experimental evidence for the three benchmarks reported in the paper (§V): Feature RAG retrieval quality (§V-A), swarm scalability from 5 to 500 drones (§V-B), and a repeated 15-model mid-size LLM benchmark (§V-C), together with demonstration images from simulation-to-real validation with DJI drones (§V-D). Contents The deposit contains 5 structured CSV files, 1 raw JSON trace, 3 interactive HTML reports, 14 publication-quality figures (PDF), and 4 demonstration images. Benchmark A — Feature RAG Retrieval Quality (§V-A) Evaluates the Feature RAG module that grounds free-form geographic references by embedding map-feature descriptions and retrieving the top-k candidates most similar to the user query. Four embedding models — two cloud (OpenAI text-embedding-3-small, 1,536-d; Google gemini-embedding-001, 3,072-d) and two local (BAAI/bge-m3, 1,024-d; intfloat/e5-large-v2, 1,024-d) — are benchmarked across three operational scenarios augmented with in-domain synthetic noise (same-category near-duplicates) up to 10,000 features. Metrics: Hit Rate at k=10 (HR@10) and Mean Reciprocal Rank (MRR), computed over four query difficulty levels (exact name, partial name, category, descriptive) with three repetitions per condition. Principal finding: exact-name, partial-name, and category queries saturate at or near 100% HR@10 across all models, so descriptive queries are the primary discriminator. gemini-embedding-001 achieves the best aggregate score at the 10,000-feature pool (about 97% HR@10 and 0.86 MRR), while the two local models remain close in quality with substantially lower embedding latency. Search latency stays low at all pool sizes, and only a fixed top-k set enters prompts, so scaling the map does not inflate LLM context. Data files:- rag_results.csv — Aggregate metrics per experimental condition (model × scenario × pool size × query type × repetition)- rag_queries.csv — Individual query-level results with retrieved feature names and latencies Figures:- rag_scalability.pdf — HR@10 vs. pool size, one line per model- rag_model_heatmap.pdf — Model × query-type heatmap at the 10,000-feature pool- rag_per_scenario.pdf — Per-scenario breakdown- rag_noise_comparison.pdf — Generic vs. in-domain noise robustness comparison Benchmark B — Swarm and Group Scalability (§V-B) Evaluates how the multi-agent pipeline scales with fleet size across two orders of magnitude: 5, 10, 25, 50, 100, 200, and 500 simulated drones. Each configuration is executed 4 times, yielding 28 sessions and 280 prompts total. Each session executes a fixed sequence of 10 natural-language prompts covering group takeoff and movement, dynamic grouping, feature-referenced traversals and orbits, area coverage, telemetry-conditioned actions through the History agent and State RAG, nearest-feature inspection through the Spatial agent, and coordinated return-to-home. The MQTT-based simulator backend is used to isolate LLM orchestration performance from network and DDS transport variability. The LLM backbone is Llama-3.3-70B, hosted via the Groq API. The orchestration pipeline, Docker containers, and MQTT simulator ran on a desktop PC with an AMD Ryzen 9 5950X, 32 GB RAM, and an NVIDIA RTX 4090 GPU, under Ubuntu 22.04. Principal finding: total token consumption remains approximately constant with respect to fleet size at about 678k ± 15% tokens per session, confirming O(1) cost complexity enabled by group-level abstraction. The Swarm agent reasons over groups rather than individual drones, so the main scaling effect of increasing fleet size is a larger state payload rather than a proportional growth in LLM context. Mean LLM pipeline latency is about 5.1 s across all sizes, rising gradually from 4.4 s at N=5 to 7.9 s at N=500. 260 of 280 prompts succeed (92.9% overall). All configurations up to N=50 remain at or above 97.5% success, and the degradation appears mainly at N ≥ 100. Failures are concentrated in workload-sensitive prompts: multi-group partitioning can exceed the Swarm agent’s single-pass tool-call budget, identifying the southernmost drone becomes harder as more vehicles share similar latitude values, and some prompts at N=500 exceed the 131,072-token context window. These failures do not reflect incorrect Coordinator planning; they arise downstream from execution-layer limits under larger payloads and group counts. Data files:- sessions.csv — Session-level summaries (total tokens, success rate, latency, payload size)- prompts.csv — Per-prompt measurements (latency breakdown, token counts, action outcomes)- agent_executions.csv — Per-agent-per-prompt breakdowns (LLM calls, tool invocations, token attribution)- drone_scalability_mqtt.json — Raw benchmark trace with full LLM conversations, tool calls, and timing data for all 28 sessions Figures:- fig1_tokens_vs_size.pdf — Token scaling with fleet size- fig2_state_payload.pdf — State payload growth vs. fleet size- fig3_llm_latency.pdf — LLM pipeline latency distribution- fig4_success_heatmap.pdf — Prompt × fleet-size success rate heatmap- fig5_tool_usage.pdf — Tool invocation frequency by agent type- fig6_agent_breakdown.pdf — Token distribution by agent type- fig8_session_time_actions.pdf — Session wall-clock time vs. actions- fig9_llm_tool_calls.pdf — LLM tool calls per prompt- fig10a_success.pdf — Summary success panel- fig10b_latency.pdf — Summary latency panel- fig10c_tokens.pdf — Summary token panel- fig10d_payload.pdf — Summary payload panel Benchmark C — Model Benchmark on a Mid-Size Scenario (§V-C) Benchmarks 15 LLM models on a representative mid-size mission by replaying the same 10 snapshots derived from the 50-drone copenhill setup used in the swarm-scalability study. The full batch is repeated three times under identical prompts, world states, and evaluation rules, yielding 30 replay instances per model. Each replay evaluates both the Coordinator plan against the recorded reference plan and the downstream sub-agent executions against the recorded tool calls. Three complementary metrics are reported: coordinator planning success, aggregate agent execution success, and mean end-to-end replay latency. Models include cloud frontier models, open-weight models served via Groq, and local GPU deployments. Principal finding: the strongest cloud models achieve near-perfect reliability, but they do not dominate on efficiency: several high-capability models reach essentially the same execution success while taking substantially more time than the fastest Groq-hosted and lightweight OpenAI models. The main weakness appears more often in planning than in downstream tool execution: agent execution success is perfect for most models, whereas coordinator planning success drops more often for smaller and local models. Among the fastest high-performing models, Llama-3.3-70B served via Groq provides the best latency–reliability trade-off, while the repeated runs also highlight the gap between local compact models and stronger dense models in planning quality. Data files:- batch_results_20260424_122359.html — Interactive report for the first full 15-model × 10-snapshot benchmark run- batch_results_20260424_125526.html — Interactive report for the second repeated run under the same benchmark configuration- batch_results_20260424_132450.html — Interactive report for the third repeated run under the same benchmark configuration Each HTML report is self-contained and can be opened in any browser without a server. Together, the three reports provide per-prompt drill-down, execution traces, latency, token usage, and cost for the repeated mid-size benchmark discussed in the paper. Demonstration Images Images from the simulation-to-real validation (§V-D), in which a seven-step operator script was executed through the same natural-language interface used in simulation on five DJI Mini 4 Pro drones at the SDU cricket field, via the WildBridge ROS 2 adapter. - GUI_example.png — Swarm-Steward web interface showing the operator chat, live map, and telemetry panels during a multi-drone session- real-scenario - formation.png — Real drone formation flight during field validation- real-scenario - take off.jpg — Coordinated takeoff of a 5× DJI Mini 4 Pro swarm- DJI Pro 4.jpeg — DJI Mini 4 Pro drone used for real-world tests File Formats - CSV: UTF-8, comma-delimited, double-quote escaping. Multi-value fields use semicolons (;) as separators.- JSON: UTF-8 encoded raw benchmark output with full execution traces.- HTML: Self-contained interactive reports (open in any browser, no server required).- PDF: Publication-quality figures.- Timestamps: ISO 8601 with timezone.