遇见数据集

Free local LLMs fail rule-grounded fee computation: 0/840 exact on RuleArena airline

收藏
Zenodo2026-07-04 更新2026-08-13 收录
官方服务:

资源简介:

Three free, locally-runnable language models (mistral 7B, llama3.1:8b, qwen2.5:7b, via Ollama, temperature 0) were given the complete rule set and asked to compute total travel cost (ticket + baggage fees) for 280 tasks each from the RuleArena airline domain (three complexity levels), 840 runs in total. Exact match against the deterministic ground truth: 1/840 (0.12%). Dissection of the single hit showed a numeric collision, not correct rule application: the model mis-applied the oversize rule (0 vs ground-truth 30), but the max operation in the fee structure collapsed the error to the correct total. Real rule-application success: 0/840. Under an alternative total-cost convention (excluding ticket price): 3/840. Per model per level, the exact-match rate ranged 0.0-1.0% against a pre-registered threshold of 20%. All cells were >=98% parseable (2/840 format failures, both truncations), so the result cannot be attributed to format failure. The error grows with the rule set: median absolute deviation from ground truth grows monotonically with complexity level across all three models (level 0: 436-880 USD; level 2: 1200-2410 USD). The models are also not consistent with their own arithmetic (a ground-truth-independent check of stated total vs sum of stated sub-fees plus ticket): mistral 17/280, llama3.1:8b 0/278, qwen2.5:7b 3/280. Method: structured output with no parser language model; deterministic value comparison against the ground truth with no language model in the verification chain; full pre-registration with thresholds, numerators, denominators and aggregation rules bound before any measurement data existed. Contribution: the model class (free, local 7-8B models runnable offline/air-gapped, relevant for public sector without cloud access), full pre-registration, deterministic verification, and the ground-truth-independent internal-consistency finding. Rights holder: EcoDeco AS. License: CC BY 4.0.

提供机构:
Zenodo
创建时间:
2026-07-04
二维码
社区交流群
二维码
科研交流群
商业服务