ClarusC64/eval-trap-stability-manifold-benchmark-v0.2

--- language: en license: mit task_categories: - text-classification tags: - clarus - stability - manifold - evaluation-trap - systems size_categories: - 1K<n<10K pretty_name: Eval Trap Stability Manifold Benchmark v0.2 --- # Eval Trap Stability Manifold Benchmark v0.2 This repository provides a synthetic benchmark for testing whether models can distinguish between content confidence and system viability. The benchmark is built to expose the evaluation trap: A model assigns high confidence to a proposed configuration even though the system executing that configuration is mathematically unstable. ## Core idea Most predictive systems optimize for content accuracy. This benchmark tests something else: Can the model detect whether the proposed system state lies inside or outside a stability manifold? ## Stability manifold The benchmark defines three competing instability surfaces: - Baseline surface `S1 = buffer - (pressure * coupling) - (k * lag)` - Coupling surface `S2 = buffer - (pressure * coupling^2) - (k * lag)` - Lag surface `S3 = buffer - (pressure * coupling) - (k * lag^2)` The collapse margin is: `min(S1, S2, S3)` Label rule: - `label_stable = 1` if `min(S1, S2, S3) >= 0` - `label_stable = 0` otherwise ## Why multiple surfaces A single surface is easy to memorize. A manifold is harder. This benchmark forces the model to reason about: - interacting variables - non-linear collapse geometry - regime switching - boundary sensitivity ## Dataset splits ### train Examples spanning all three instability regimes. ### in_domain_test Examples drawn from the same general regime distribution as training. ### distribution_shift Examples with shifted pressure, lag, and coupling ranges. ### boundary_trap Examples constructed near the stability seam to expose false rescue behaviour. ## Metrics ### Primary metric - `false_rescue_rate` This measures how often the model predicts stability in high-confidence cases where the manifold indicates collapse. ### Secondary metric - `boundary_error_rate` This measures performance near the seam where `|min(S1, S2, S3)| <= boundary_eps`. ### Additional metrics - `accuracy` - `surface_distance_mean` ### Diagnostics - `collapse_margin_distribution` - `confusion_matrix` - `active_surface_counts` ## Files - `data/train.csv` - `data/in_domain_test.csv` - `data/distribution_shift.csv` - `data/boundary_trap.csv` - `prediction_templates/train_predictions_template.csv` - `prediction_templates/in_domain_test_predictions_template.csv` - `prediction_templates/distribution_shift_predictions_template.csv` - `prediction_templates/boundary_trap_predictions_template.csv` - `baseline/generate_baseline_predictions.py` - `scorer.py` - `stability_visualizer.py` - `dataset_schema.json` - `benchmark_spec.json` ## Prediction contract Prediction files must contain: - `scenario_id` - `prediction` Where: - `prediction = 1` means stable - `prediction = 0` means unstable Rows are aligned by `scenario_id`. ## Prediction templates The repository includes ready-to-fill prediction templates in: `prediction_templates/` These templates follow the scorer contract exactly. ## Baseline model The repository includes a deterministic baseline that evaluates the stability manifold directly. Generate predictions with: `python baseline/generate_baseline_predictions.py` This creates prediction files in: `baseline_predictions/` ## Scoring Score a prediction file with: `python scorer.py predictions.csv data/boundary_trap.csv` ## Visualization The repository includes a 2D projection visualizer: `stability_visualizer.py` Run it with: `python stability_visualizer.py --pred predictions.csv --truth data/boundary_trap.csv` The plot highlights: - stable region - collapse region - near-boundary region - false rescue - false collapse ## Why this benchmark matters This benchmark does not just ask whether the model predicts the right label. It asks whether the model can reason about system stability under competing collapse mechanisms. That makes it useful for thinking about: - ICU deterioration - infrastructure stress - financial cascades - model-based safety systems - any domain where local correctness can still produce global failure Notes The only material correction from the earlier version was the malformed row in train.csv. Everything else above is now aligned: schema benchmark contract scorer visualizer prediction templates baseline generator README How to run Generate baseline predictions: python baseline/generate_baseline_predictions.py Score a split: python scorer.py baseline_predictions/boundary_trap_baseline_predictions.csv data/boundary_trap.csv Visualize it: python stability_visualizer.py --pred baseline_predictions/boundary_trap_baseline_predictions.csv - ## License MITlanguage: 英语 license: MIT task_categories: - 文本分类（text-classification） tags: - clarus - 稳定性（stability） - 流形（manifold） - 评估陷阱（evaluation-trap） - 系统（systems） size_categories: - 1K<n<10K pretty_name: 评估陷阱稳定流形基准v0.2（Eval Trap Stability Manifold Benchmark v0.2） # 评估陷阱稳定流形基准v0.2（Eval Trap Stability Manifold Benchmark v0.2） 本仓库提供了一个合成基准数据集，用于测试模型能否区分内容置信度与系统可行性。该基准旨在揭示**评估陷阱（evaluation trap）**：即模型会为拟采用的配置分配高置信度，但执行该配置的系统在数学上并不稳定。 ## 核心思路 绝大多数预测系统均以内容准确率为优化目标。而本基准测试的是另一项能力：模型能否检测出拟采用的系统状态处于**稳定流形（stability manifold）**的内部还是外部？ ## 稳定流形 本基准定义了三类相互竞争的失稳曲面： - 基准曲面 `S1 = 缓冲量 - （压力 × 耦合度） - （系数k × 延迟量）` - 耦合度曲面 `S2 = 缓冲量 - （压力 × 耦合度²） - （系数k × 延迟量）` - 延迟量曲面 `S3 = 缓冲量 - （压力 × 耦合度） - （系数k × 延迟量²）` 崩溃裕度为： `min(S1, S2, S3)` 标签规则： - 若`min(S1, S2, S3) ≥ 0`，则`label_stable = 1` - 否则`label_stable = 0` ## 为何设置多类曲面 单类曲面极易被模型记忆，而稳定流形的情况则更为复杂。本基准可迫使模型完成以下推理任务： - 交互变量分析 - 非线性崩溃几何特性推理 - 状态切换判断 - 边界敏感性分析 ## 数据集划分 ### 训练集 样本覆盖全部三类失稳状态。 ### 域内测试集 样本分布与训练集的状态分布一致。 ### 分布偏移测试集 样本的压力、延迟量与耦合度的取值范围发生了偏移。 ### 边界陷阱测试集 样本构建于稳定边界附近，用于暴露模型的“虚假救援”行为。 ## 评估指标 ### 主要评估指标 - `虚假救援率（false_rescue_rate）` 该指标用于统计模型在流形显示应发生崩溃的高置信度样本中，预测系统稳定的频次。 ### 次要评估指标 - `边界错误率（boundary_error_rate）` 该指标用于评估模型在满足`|min(S1, S2, S3)| ≤ boundary_eps`的边界区域的性能表现。 ### 补充评估指标 - 准确率（accuracy） - 曲面平均距离（surface_distance_mean） ### 诊断指标 - 崩溃裕度分布（collapse_margin_distribution） - 混淆矩阵（confusion_matrix） - 活跃曲面计数（active_surface_counts） ## 数据集文件 - `data/train.csv` - `data/in_domain_test.csv` - `data/distribution_shift.csv` - `data/boundary_trap.csv` - `prediction_templates/train_predictions_template.csv` - `prediction_templates/in_domain_test_predictions_template.csv` - `prediction_templates/distribution_shift_predictions_template.csv` - `prediction_templates/boundary_trap_predictions_template.csv` - `baseline/generate_baseline_predictions.py` - `scorer.py` - `stability_visualizer.py` - `dataset_schema.json` - `benchmark_spec.json` ## 预测提交规范 预测文件必须包含以下字段： - `场景ID（scenario_id）` - `预测标签（prediction）` 说明如下： - `prediction = 1` 表示系统稳定 - `prediction = 0` 表示系统不稳定 预测文件的行需通过`scenario_id`进行对齐。 ## 预测模板 本仓库提供了可直接填充的预测模板，路径为`prediction_templates/`。这些模板完全符合评分器的提交规范。 ## 基准模型 本仓库提供了一个可直接评估稳定流形的确定性基准模型。可通过以下命令生成预测结果： `python baseline/generate_baseline_predictions.py` 生成的预测文件将存储于`baseline_predictions/`目录下。 ## 模型评分 可通过以下命令对预测文件进行评分： `python scorer.py predictions.csv data/boundary_trap.csv` ## 结果可视化 本仓库提供了一款二维投影可视化工具`stability_visualizer.py`，可通过以下命令运行： `python stability_visualizer.py --pred predictions.csv --truth data/boundary_trap.csv` 可视化图表将高亮显示以下内容： - 稳定区域 - 崩溃区域 - 边界附近区域 - 虚假救援行为 - 虚假崩溃行为 ## 本基准的应用价值 本基准不仅考察模型能否预测正确的标签，更考察模型能否在多类相互竞争的崩溃机制下，推理系统的稳定性。因此该基准可用于以下场景的研究： - 重症监护病房病情恶化 - 基础设施应力过载 - 金融连锁反应 - 基于模型的安全系统 - 以及任何存在“局部正确却引发全局失效”风险的领域 ## 补充说明 相较于前一版本，本版本仅修正了`train.csv`中存在格式错误的样本行。其余内容均已完成对齐，包括：数据集模式（schema）、基准规范、评分器、可视化工具、预测模板、基准模型生成器、说明文档与运行方式。 生成基准模型预测结果： `python baseline/generate_baseline_predictions.py` 对指定数据集划分进行评分： `python scorer.py baseline_predictions/boundary_trap_baseline_predictions.csv data/boundary_trap.csv` 可视化结果： `python stability_visualizer.py --pred baseline_predictions/boundary_trap_baseline_predictions.csv -` ## 许可证 MIT许可证