ClinSeek-Evaluation-Results

# Results Directory All evaluation outputs, reorganized by **benchmark × run mode × model**. ## Layout ``` results/ ├── ehr_bench/ # text-only EHR-Bench (1800 rows, 45 tasks) │ ├── agentic/ # multi-turn tool-calling via deploy_agent.py │ │ ├── smoke20/<model>/ │ │ ├── full1800/<model>/ ← rollout outputs (results.jsonl, run.log, per-region shards) │ │ └── scored/ ← evaluate_results.py output (WIP slot) │ └── oneshot/ # single Bedrock invoke via deploy_reasoning_model.py │ ├── smoke20/<model>/ │ └── full1800/<model>/ │ ├── agent_ehr_bench/ # AgentEHR-Bench (600 rows, 6 MIMIC tasks) ← symlinks into DeepMed-eval │ └── agentic/ │ └── subset600/<model>/ ← symlinks to /fsx-shared/juncheng/DeepMed-eval/openresearcher_ehr/subsets_600_* │ ├── mm_bench/ # multimodal EHR-Bench (2703 rows, 6 tasks) │ ├── agentic/ # multi-turn via deploy_agent_mm.py (+ image MCP, + EHR MCP) │ │ ├── full2703/<model>/ ← rollout outputs (Opus=real dir, others=symlinks into _multi_eval_raw/) │ │ ├── full2703/_multi_eval_raw/ ← raw 5-model fan-out run (multi_eval/full5_*) │ │ └── scored/<model>/ ← scorer_mm.py --vocab output │ └── oneshot/ # single Bedrock invoke via deploy_reasoning_model_mm.py │ ├── smoke20/<model>/ │ ├── full2703/<model>/ │ └── scored/ ← full2703 scored (model=<model>), smoke20 scored (prefix smoke20_*) │ └── _archive/ # superseded / half-finished / early-pilot runs kept for audit ``` ## Model slug convention All leaf directories use snake-cased slugs: | friendly name | slug | |---|---| | Claude Opus 4.6 | `claude_opus_4_6` | | Claude Sonnet 4.6 | `claude_sonnet_4_6` | | Qwen3-VL-235B | `qwen3_vl_235b` | | Qwen3-235B (text-only) | `qwen3_235b` | | Kimi K2.5 | `kimi_k2_5` | | GLM-4.7 | `glm_4_7` | | MiniMax M2.5 | `minimax_m2_5` | | gpt-oss-120b | `gpt_oss_120b` | ## Canonical "where are the numbers" - **EHR-Bench agentic full**: `ehr_bench/agentic/full1800/<model>/results.jsonl` - **EHR-Bench one-shot full**: `ehr_bench/oneshot/full1800/<model>/results.jsonl` - **AgentEHR-Bench agentic**: `agent_ehr_bench/agentic/subset600/<model>/results.jsonl` - **MM agentic full**: `mm_bench/agentic/full2703/<model>/` (Opus: `results.jsonl`; others: `merged_unique.jsonl`) - **MM agentic scored (vocab)**: `mm_bench/agentic/scored/<model>/summary_vocab.{json,md}` - **MM one-shot full**: `mm_bench/oneshot/full2703/<model>/results.jsonl` - **MM one-shot scored (vocab)**: `mm_bench/oneshot/scored/<model>/summary.{json,md}` ## Notes on the MM agentic symlinks The five non-Anthropic MM agentic runs were produced by `run_multi_model_eval.py` on 2026-04-21, which uses a different folder layout (`<run>/scored_vocab/<model>/` with hyphenated model names). For browsability we symlink underscore-slugged peer directories pointing into `_multi_eval_raw/`: ``` mm_bench/agentic/full2703/claude_sonnet_4_6 -> _multi_eval_raw/full5_20260421T091517Z/claude-sonnet-4-6 mm_bench/agentic/scored/claude_sonnet_4_6 -> ../full2703/_multi_eval_raw/full5_20260421T091517Z/scored_vocab/claude-sonnet-4-6 ``` The Opus MM agentic (`mm_bench/agentic/full2703/claude_opus_4_6`) is a real directory because it was produced by a separate Opus-solo run on 2026-04-20. ## Archive `_archive/` contains: - Half-completed runs (`ehr_bench_opus46_1800_pre_ssl_retry`, `ehr_bench_opus46_retry693_ssl_fix`) - Duplicate/salvage-patched re-smokes (`ehr_bench_smoke20_salvage_patched_*`) - Early pilot MM runs (`mm_agentic_smoke5_early`, `mm_agentic_mix6_early`, `mm_agentic_images_pilot_{a,b}`, etc.) - Pre-merge retry outputs (`ehr_bench_oneshot_full1800_kimi_k2_5_pre_retry`, `ehr_bench_oneshot_retry_{kimi,minimax}_*`) - Earlier non-vocab MM scored outputs (`mm_agentic_opus_scored_novocab`, `mm_agentic_prepared_scored_novocab`) Safe to delete after ~30 days if no one has needed to reference them.