skillsbench-leaderboard

# SkillsBench Leaderboard and Evidence Archive This repository stores public SkillsBench submissions, raw BenchFlow trial artifacts, trajectory evidence, audit reports, and the release-aligned official leaderboard exports. Official benchmark definition: [benchflow/skillsbench](https://huggingface.co/datasets/benchflow/skillsbench) Latest public benchmark release: [SkillsBench v1.1](https://github.com/benchflow-ai/skillsbench/releases/tag/v1.1) Latest source commit: `27738384b1df694ea2ae466e416f476e94d8fab9` ## Current Official Release The latest public results are under: ```text analysis/skillsbench/v1.1/official-selected/ leaderboard/skillsbench/v1.1/official.csv leaderboard/skillsbench/v1.1/official.json ``` The current official export is the 2026-06-16 v1.1 board with 24 model-harness configurations. It combines reviewed public main results with PR #14 and PR #15 evidence, normalized to the SkillsBench v1.1 87-task roster. `v1.1` is aligned to the public 87-task release. Earlier candidate analysis contained one extra task, `taxonomy-tree-merge`; those trials are retained in `excluded_from_v1_1_release.jsonl` and excluded from official scores. ## Layout ```text submissions/skillsbench/v1.1/ # public v1.1 trial artifacts, including migrated historical evidence analysis/skillsbench/v1.1/official-selected/ # reviewed release-aligned selected runs and audits leaderboard/skillsbench/v1.1/official.csv # compact leaderboard table leaderboard/skillsbench/v1.1/official.json # machine-readable leaderboard table schemas/ # submission/result schemas ``` All retained submission artifacts, including older historical evidence, are stored under the unified `v1.1` namespace. Do not add new `v0.1` submission paths. ## Submission Contract New submissions should go under: ```text submissions/skillsbench/<version>/<agent-mode>__<model>/<run-ts>/<task>__<trial-id>/ ``` Required files per trial: - `config.json` - `result.json` - `timing.json` - `trajectory/acp_trajectory.jsonl` - `trajectory/llm_trajectory.jsonl` when provider-level LLM traces are available Required metadata per `<agent-mode>__<model>` directory: - `metadata.yaml`, following `schemas/submission.schema.json` ## Scoring Official scores are computed from reviewed non-timeout trials only: - `with_skills_score = 100 * mean(reward | skill_mode = with-skill)` - `without_skills_score = 100 * mean(reward | skill_mode = no-skill)` - `skill_lift = with_skills_score - without_skills_score` Incomplete cells remain visible with `coverage_status=partial`. ## Hugging Face Benchmark Integration The Hugging Face framework key is tracked in [huggingface/huggingface.js#2139](https://github.com/huggingface/huggingface.js/pull/2139). Once `evaluation_framework: benchflow` is deployed, model repositories can reference `benchflow/skillsbench` from `.eval_results/skillsbench.yaml`, with `source.url` pointing back to this leaderboard dataset.