FailureScope: Cross-Regime Behavioral Diagnosis of Language Model Weaknesses

FailureScope is a cross-regime behavioral-diagnosis methodology for language models, validated across single-turn benchmarks, multi-turn dialogue, and adversarial agent attacks. A single primitive — leave-one-model-out (LOMO) clustering of cross-model pass/fail patterns — recovers stable, interpretable failure structure in all three regimes. This umbrella record points to the three component datasets (linked under "Related identifiers" below) and provides the consolidated Croissant 1.0 metadata bundle for all three.

Component datasets:- Single-turn (DOI 10.5281/zenodo.20034013): 2,664 tasks x 18 models.- Multi-turn (DOI 10.5281/zenodo.20034373): 363 frontier-hard tasks x 5 frontier models x 7 depths.- Adversarial (DOI 10.5281/zenodo.20034377): 630 multi-turn agent traces x 3 attack families x 3 frontier models.

Headline results: Kendall's tau = 0.81 cluster-level diagnostic accuracy at N=50 tasks, AUC 0.88 cross-model failure prediction, and a 73-100 percentage-point gap between LLM-judge ASR and ground-truth network execution. To use the full FailureScope release, download all three component zips (DOIs above) and the Croissant bundle attached here.