zjunlp/PredictBeforeExecute

# Can We Predict Before Executing Machine Learning Agents? -- Data <h4 align="center"> <a href="https://arxiv.org/abs/2601.05930" target="_blank">📄Paper</a> • <a href="https://arxiv.org/abs/2601.05930" target="_blank">🛠️Code</a> • <a href="https://huggingface.co/papers/2601.05930" target="_blank">🤗HFPaper</a> • <a href="https://drive.google.com/drive/folders/1rn3GuRcl-BrnPG2xUJYCOJB-BwGp7bp0?usp=sharing" target="_blank">📦Data & Runtime (GoogleDrive)</a> • <a href="https://x.com/zxlzr/status/2010603724931285141" target="_blank">𝕏Blog</a> • <a href="http://xhslink.com/o/8Ac0jDoHeyw" target="_blank">📕小红书</a> </h4> This project studies **Data-centric Solution Preference**—predicting which ML solution will perform better *before* executing it—by leveraging data analysis context and LLM reasoning. The repository provides curated solution corpora, task resources, agent-run outputs, and analysis artifacts to support both the main evaluation and follow-up studies. This directory is the **central data workspace** for the project. It contains the full solution corpus, experiment subsets, agent-run outputs, analysis artifacts, task resources, and a cached Docker image. --- ## Top-level contents - [solutions_all/](solutions_all/) The **full solution corpus** provided by us (all available solutions). This is the source pool from which all subsets are sampled. - [solutions_subset_50/](solutions_subset_50/) The **main-experiment subset**, capped at **50 solutions per task** (used in the paper’s primary experiments). - [solutions_subset_15/](solutions_subset_15/) The **analysis subset**, sampled from `solutions_subset_50/`, capped at **15 solutions per task** (used for downstream analysis experiments). - [agent_runs/](agent_runs/) Outputs from agent executions. Subfolders include: - [agent_runs/AIDE/](agent_runs/AIDE/) — AIDE-generated runs (task-name + UUID per run). - [agent_runs/ForeAgent/](agent_runs/ForeAgent/) — ForeAgent-generated runs (task-name + UUID per run). - For detailed per-run structure and where to find trajectories/logs, see [agent_runs/README.md](agent_runs/README.md). - [analysis_exp/](analysis_exp/) Analysis experiment artifacts (RQ1–RQ4). See its README for details. - [tasks/](tasks/) The **shared data hub** for competitions, prepared data, task descriptions, data analysis reports, and task lists. See its README for details. - [docker_images/](docker_images/) Cached Docker images used by the execution pipeline. - [docker_images/predict-before-execute.tar](docker_images/predict-before-execute.tar) — a prebuilt image archive matching the base image referenced in the Dockerfile. - [2601.05930v1.pdf](2601.05930v1.pdf) A local copy of the paper PDF. --- ## Docker image note (for the execution pipeline) The Dockerfile in prepare_bench_subset/env/Dockerfile uses: - `FROM johnsonzheng03/predict-before-execute` If pulling this base image directly is slow or unstable, you can load the cached image tarball from `docker_images/` instead. This extraction may take **a long time** depending on disk and Docker performance. Command: ``` docker load -i path/to/predict-before-execute.tar ``` --- ## Solutions directories (shared layout) The three `solutions_*` directories share the **same internal layout**. Each task folder typically looks like: ``` solutions_root/ <task_name>/ annotation/ annotations_semantic.json keywords_by_rank.json code/ solution_*.py submission_solution_*/ eval_output.json exec_output.txt submission.csv ground_truth/ groups_<task_name>_n*.json output/ output_*.txt report/ alignment_*.json grade_report_*.txt ``` Each task folder contains: - `annotation/` - `annotations_semantic.json`: per-solution semantic labels used for subset sampling and analysis. - `keywords_by_rank.json`: aggregated keyword statistics by rank. - `code/` - `solution_*.py`: the runnable solution files. - `submission_solution_*/`: execution artifacts for each solution (created after running). - `submission.csv`: the model’s predicted submission. - `exec_output.txt`: execution logs / stdout+stderr. - `eval_output.json`: grading results (if already evaluated). - `ground_truth/` - `groups_<task_name>_n*.json`: ground-truth comparison groups for evaluation. - `output/` - `output_*.txt`: optional runtime or extraction logs. - `report/` - `grade_report_*.txt`: human-readable grading reports. - `alignment_*.json`: alignment artifacts derived from reports. This is the canonical layout used by our preparation, grading, and analysis scripts. --- ## Where to find what (quick locator) - **Main experiment solutions/logs**: [solutions_subset_50/](solutions_subset_50/) - **Analysis experiment solutions/logs**: [solutions_subset_15/](solutions_subset_15/) - **Full corpus (all solutions)**: [solutions_all/](solutions_all/) - **Agent trajectories and logs**: [agent_runs/](agent_runs/) (details in [agent_runs/README.md](agent_runs/README.md)) - **Analysis experiment artifacts (RQ1–RQ4)**: [analysis_exp/](analysis_exp/) - **Task resources (competition configs, prepared data, descriptions, data analysis)**: [tasks/](tasks/)# 能否在执行机器学习智能体前进行预测？——数据集 <h4 align="center"> <a href="https://arxiv.org/abs/2601.05930" target="_blank">📄论文</a> • <a href="https://arxiv.org/abs/2601.05930" target="_blank">🛠️代码</a> • <a href="https://huggingface.co/papers/2601.05930" target="_blank">🤗HFPaper</a> • <a href="https://drive.google.com/drive/folders/1rn3GuRcl-BrnPG2xUJYCOJB-BwGp7bp0?usp=sharing" target="_blank">📦数据集与运行时（Google云端硬盘）</a> • <a href="https://x.com/zxlzr/status/2010603724931285141" target="_blank">𝕏博客</a> • <a href="http://xhslink.com/o/8Ac0jDoHeyw" target="_blank">📕小红书</a> </h4> 本研究聚焦**以数据为中心的解决方案偏好（Data-centric Solution Preference）**——即借助数据分析上下文与大语言模型（LLM）推理能力，在执行机器学习解决方案前预判其性能优劣——。本仓库提供经过精选的解决方案语料库、任务资源、智能体运行输出与分析工件，以支撑主实验评估与后续拓展研究。 本目录为本项目的**核心数据工作区**，涵盖完整解决方案语料库、实验子集、智能体运行输出、分析工件、任务资源以及缓存的Docker镜像。 --- ## 顶层目录结构 - [solutions_all/](solutions_all/) **完整解决方案语料库**，包含我们整理的全部可用解决方案，是所有实验子集的采样源池。 - [solutions_subset_50/](solutions_subset_50/) **主实验子集**，每个任务最多包含50个解决方案（为本论文核心实验所用）。 - [solutions_subset_15/](solutions_subset_15/) **分析子集**，从`solutions_subset_50/`中采样得到，每个任务最多包含15个解决方案（用于下游分析实验）。 - [agent_runs/](agent_runs/) 智能体执行的输出结果，其子目录包括： - [agent_runs/AIDE/](agent_runs/AIDE/) — 由AIDE生成的运行记录（每个运行以任务名+UUID标识）。 - [agent_runs/ForeAgent/](agent_runs/ForeAgent/) — 由ForeAgent生成的运行记录（每个运行以任务名+UUID标识）。 关于单条运行的详细结构以及轨迹/日志的获取位置，请参阅[agent_runs/README.md](agent_runs/README.md)。 - [analysis_exp/](analysis_exp/) 分析实验工件（对应研究问题RQ1至RQ4），详细信息请参阅其README文件。 - [tasks/](tasks/) **共享数据中心**，包含竞赛相关数据、预处理数据集、任务说明、数据分析报告以及任务列表，详细信息请参阅其README文件。 - [docker_images/](docker_images/) 执行流水线所用的缓存Docker镜像。 - [docker_images/predict-before-execute.tar](docker_images/predict-before-execute.tar) — 与Dockerfile中引用的基础镜像匹配的预构建镜像归档文件。 - [2601.05930v1.pdf](2601.05930v1.pdf) 论文PDF的本地副本。 --- ## Docker镜像说明（针对执行流水线） prepare_bench_subset/env/Dockerfile 中的Dockerfile使用了以下基础镜像： FROM johnsonzheng03/predict-before-execute 如果直接拉取该基础镜像速度过慢或不稳定，可改为从`docker_images/`目录加载缓存的镜像归档文件。根据磁盘与Docker性能的不同，该解压过程可能耗时较久。 加载命令： bash docker load -i path/to/predict-before-execute.tar --- ## 解决方案目录（通用布局） 三个`solutions_*`目录采用**完全一致的内部布局**，每个任务文件夹的典型结构如下： solutions_root/ <task_name>/ annotation/ annotations_semantic.json keywords_by_rank.json code/ solution_*.py submission_solution_*/ eval_output.json exec_output.txt submission.csv ground_truth/ groups_<task_name>_n*.json output/ output_*.txt report/ alignment_*.json grade_report_*.txt 每个任务文件夹包含以下内容： - `annotation/` - `annotations_semantic.json`：用于子集采样与分析的单解决方案语义标签文件。 - `keywords_by_rank.json`：按排序聚合的关键词统计文件。 - `code/` - `solution_*.py`：可运行的解决方案脚本文件。 - `submission_solution_*/`：每个解决方案的执行工件（运行后生成）。 - `submission.csv`：模型生成的预测提交文件。 - `exec_output.txt`：执行日志（标准输出与标准错误输出）。 - `eval_output.json`：评分结果（若已完成评估）。 - `ground_truth/` - `groups_<task_name>_n*.json`：用于评估的真值对比组文件。 - `output/` - `output_*.txt`：可选的运行时或提取日志文件。 - `report/` - `grade_report_*.txt`：人类可读的评分报告文件。 - `alignment_*.json`：从报告中提取的对齐工件文件。 该结构为本项目预处理、评分与分析脚本所采用的标准布局。 --- ## 快速导航：所需内容的位置 - **主实验解决方案与日志**：[solutions_subset_50/](solutions_subset_50/) - **分析实验解决方案与日志**：[solutions_subset_15/](solutions_subset_15/) - **完整语料库（全部解决方案）**：[solutions_all/](solutions_all/) - **智能体轨迹与日志**：[agent_runs/](agent_runs/)（详细信息参阅[agent_runs/README.md](agent_runs/README.md)） - **分析实验工件（RQ1至RQ4）**：[analysis_exp/](analysis_exp/) - **任务资源（竞赛配置、预处理数据集、任务说明、数据分析）**：[tasks/](tasks/)