ATLAS VRA v1 - Training Data and Code

Quick Summary

The Virtual Research Assistant is a bot (or set of bots) that help ATLAS eyeballers by ordering+prioritising the alerts in the eyeball list, removing the crappiest objects, and sending automatic triggers for transients within 100 Mpc to be followed up with the Mookodi Telescope.This is the first public release of the data and codes used to train the models that power the VRA. 

Who is this repo for and how to use it.

[D] Devs - for reproducibility and book-keeping. 

[U] Users (eyeballers) who want to understand the models and its limitation.

[S] Scientist who want to understand the method.

For each resource we flag which type of user we think will benefit by using the abbreviations [D/U/S]

Requirements

matplotlib, numpy, pandas, scikit-leanrn, joblib, atlasvras, atlasapiclient

Note: All the notebooks call a matplotlib style which is not released here (vra.mplstyle or vra_light.mplstyle). Replace this style choice in the first cell of the notebooks with your own style file or comment it out if you want to actually run the notebooks. 

[/!\]* notebooks marked with this warning sign cannot be run without access to the raw JSON files which will have you download nearly 40 GB of data (once unzipped). 

------------------------------------------------------------------------------------------------------------------------

Contents

Raw JSON data

/!\ LARGE SIZE - 40 GB unzipped /!\ Unless you have a very good reason to download this we don't recommend it. Even if you want the data for a specific (or a group of) transient that is part of this release you can get the data from the ATLAS Transient Server through the ATLAS API without having to download all of this.

[D] json_files contains the JSON files for the ``Crabby`` data 

[D] data_objects_with_decision_NEWEYEBALLLIST contains the JSON files for the extra data added to ``Duck`` 

Note if you are here for raw data - these do not represent the full range of proporties of the data we see in our complete stream. They are data for objects that made all cuts in the data processing and would have been shown to eyeballers. They are solely intended to train a model that works downstream of previous automation steps. 

----------------------------------------------------------------------------------------------------------------------

Duck and Duck 1.1

Cleans up the training data gathered between 18th August 2024 - 22nd Januray 2025, and adds it to the previous Crabby data set to make a larger,  near complete (in terms of sky coverage) data set to retrain the models with.

There are two "ducks" because the first one relates to to VRA 1.0 release which contains models trained with a sub-par training set (see Chapter 3 of the Technical Manual) ; that model was in production from 3rd February to 6th March 2025 .

Each directory has their own README.md summarising their content. Below I describe what content to reference if we focus on the current in production version of the VRA and the contents of the paper.

Overview and key figures from the paper:  ./Duck1.1

[D/U/S]  Overview.ipynb: A general overview that introduces the VRA, it's day 1 and day N models, the ranking, and looks at feature importance. Many of the plots of the paper come from here.

[D/U/S] Key_transients.ipynb: Looks at how they day 1 models perform on a number of important transients, to see how they fare against our models and chosen eyeballing policies.

[D/U] Policy_evaluation.ipynbNotebook to check how our models and chosen policies work together - how much do we auto-garbage? How much do we eyeball? Do we loose any good transients? Are these acceptable losses?

[D/U/S] Interpreting_AT2024lwd.ipynb

Clean new data ./Duck/data

Directories    *  clean_data_csv: Clean csv files containing the contextual information, detection and non detection data, and relevant``tcs_vra_scores`` data.

Notebooks   * [D] [/!\] clean_data.ipynb: Notebook used to extract the csv files in ``clean_data_csv`` from the raw JSON data 

Features ./Duck1.1/data

Directories    *  clean_data_csv: Clean csv files containing the contextual information, detection and non detection data, and relevant``tcs_vra_scores`` data.    *  features_and_labels_csv: Features and labels for the day 1 and day N models    *  figures: pictures    

Notebooks     * [D/U]  features_day1.ipynb: notebook used to extract the day 1 features    * [D/U]features_day1_train_val_split.ipynb: notebook used to extract the day 1 featuresthen split the new data set into train and validation set, balance the training set with subsamplng and finally combine these new data with the ``Crabby`` training and validation sets.     * [D/U] features_dayN.ipynb notebook used to extract the day N features.      * [D] add_detmagmedianmin5d_tocrabby.ipynb: notebook to add ``DET_mag_median_min5d`` to ``Crabby`` samples    * [D/U/S]  Summary_plots.ipynb Summary tables and plots of the data set and feature distributions. Many of the plots of the paper come from here.

 ./models

Where we store the models to be used in production - created by the ``Overview.ipynb`` notebook.

----------------------------------------------------------------------------------------------------------------------

Crabby 

Training data gathered between 27th March 2024 - 13th August 2024.These models were in production from 6th December 2024 until mid February 2025.

 ./

[D/U/S] Overview.ipynb: A general overview that introduces the VRA, it's day 1 and day N models, the ranking, and looks at feature importance.

[D/U/S] Key_transients.ipynb: Looks at how they day 1 models perform on a number of important transients, to see how they fare against our models and chosen eyeballing policies.

[D/U] Policy_evaluation.ipynbNotebook to check how our models and chosen policies work together - how much do we auto-garbage? How much do we eyeball? Do we loose any good transients? Are these acceptable losses?

[D]_Gal_candidate_policy.ipynb: The notebook used to test out a new policy to flag objects as being Galactic Candidates. This was later implemented when retraining with the ``Duck`` data set for VRA 1.0 in Februrary 2025.

[D]Policy_evaluation_new_strat.ipynb: Same as the policy evaluation notebook above with the added step of having a galactic candidate eyeball list as tested in the ``_Gal_candidate_policy.ipynb`` notebook.  

./data

Directories    * clean_data_csv: Clean csv files containing the contextual information, detection and non detection data, and relevant``tcs_vra_scores`` data.    * features_and_labels_csv: Features and labels for the day 1 and day N models    * figures: pictures     * to_reeyball``: csv files containing the data of the alerts that were flagged for re-eyeballing AND the csv containing the re-eyeballing decisions.   

Notebooks    * [D] [/!\] clean_data.ipynb: Notebook used to extract the csv files in ``clean_data_csv`` from the raw JSON data     * [D/U/S] features_day1.ipynb: notebook used to extract the day 1 features, split the train and validation set and do the subsampling of the over-represented classes (for the training set only)    * [D/U/S] features_dayN.ipynb notebook used to extract the day N features.     * [D/U/S] Summary_plots.ipynb: Summary tables and plots of the data set and feature distributions.     

./old_vs_new

[D]in\_prod_comparison.ipynb: Comparison of the models created with the `Crabby` data set compared to the `BMO` version. These are unpublished and had training data from 27th March until mid July 2024 - the main differences lay in the features used. This notebook is mostly useful to the dev team and to see how model comparisons are done when upgrading to a new VRA. 

 ./train_scoring_models/hyperparameter_tuning

[D]

This directory contains all the nitty gritty of the hyper paramter tuning phase.The details can be found in the hp_tuning_evaluation.ipynb notebook. 

If you only want to see the paramters we used for our production models see the ``Overview.ipynb`` notebook in the top directory.

./train_scoring_models/prod_models

[D]

The subdirectories ``day1`` and ``dayN`` contain the production models used in the ATLAS server (before and after re-eyeballing and retraining). These are mostly there for dev book-keeping.

Final Note

In addition to the main directories and notebooks listed here there are csv files in some of the subdirectories - those are created by the notebookes here and not described in this description or the readme to keep the documentation readable and because their names are self-explanataory (and their source cna be found in the notebooks).

----------------------------------------------------------------------------------------------

Anything Unclear?

Please get in touch if you have any questions about the contents of this data archive. I did my best to documents and explain the process but I did this in hinsight once I'd been working on the project for a year - what is clear to me may not be clear to a fresh reader. 

[e-mail]

快速概览
虚拟研究助手（Virtual Research Assistant，简称VRA）是一款智能机器人（bot）集群，可为ATLAS目视审核人员提供支持：对目视审核列表中的警报进行排序与优先级划分，剔除质量最差的天体，并为距离在100百万秒差距（Mpc）以内的暂现源自动触发观测指令，以使用穆科迪望远镜（Mookodi Telescope）开展后续观测。本数据集为支撑VRA运行的模型训练所用数据与代码的首次公开版本。

本代码库的适用人群与使用方法
[D] 开发者（Devs）：用于复现实验与文档留存。
[U] 用户（目视审核人员）：希望了解模型原理与局限性的人群。
[S] 科研人员：希望理解研究方法的人群。
本仓库对每项资源标注了[D/U/S]缩写，以指明其对应的受益用户群体。

依赖环境
matplotlib、numpy、pandas、scikit-learn、joblib、atlasvras、atlasapiclient
注意：所有Jupyter Notebook均调用了未随本仓库发布的matplotlib绘图样式（vra.mplstyle或vra_light.mplstyle）。若需实际运行Notebook，请将首个代码单元格中的样式调用替换为您自定义的样式文件，或直接注释掉该行代码。

[！] 带有该警告标记的Notebook需依赖原始JSON数据方可运行，解压后数据体量近40GB，若无特殊需求不建议下载。

目录结构
原始JSON数据
[！] 数据体量庞大：解压后约40GB。若无充足理由，我们不建议下载该部分数据。即便您需要本发布中特定暂现源或某类暂现源的数据，也可通过ATLAS API从ATLAS暂现源服务器获取，无需下载全部原始数据。
[D] json_files目录：存储用于“Crabby”数据集的JSON文件。
[D] data_objects_with_decision_NEWEYEBALLLIST目录：存储添加至“Duck”数据集的额外数据对应的JSON文件。

注意：若您仅需原始数据，需明确本数据并非完整数据流中所有天体的全部属性。本数据仅包含经过全部数据处理流程筛选、可供目视审核人员查看的天体数据，仅用于训练在前期自动化流程之后运行的下游模型。

Duck与Duck 1.1数据集
该数据集整理了2024年8月18日至2025年1月22日期间采集的训练数据，并将其与此前的“Crabby”数据集合并，构建出规模更大、天空覆盖度接近完整的数据集，用于模型的重新训练。
本仓库包含两个“Duck”数据集，其中首个版本对应VRA 1.0公开版本，其模型使用了质量欠佳的训练集训练而成（详见技术手册第3章）；该模型于2025年2月3日至3月6日期间投入生产使用。
每个子目录均配有独立的README.md文件，用于概述其包含的内容。下文将针对当前生产环境中运行的VRA版本与论文相关内容，指明所需参考的内容路径。
论文概述与核心图表：./Duck1.1

[D/U/S] Overview.ipynb：通用概述文档，介绍VRA的day 1模型与day N模型、排序机制以及特征重要性分析。论文中的多数图表均源自该Notebook。
[D/U/S] Key_transients.ipynb：分析day 1模型在若干重要暂现源上的表现，对比其与当前模型及预设目视审核策略的效果差异。
[D/U] Policy_evaluation.ipynb：用于评估模型与预设审核策略的协同效果——包括自动剔除的劣质天体占比、需人工审核的警报占比、是否遗漏优质暂现源，以及此类遗漏是否可接受。
[D/U/S] Interpreting_AT2024lwd.ipynb：用于解读暂现源AT2024lwd的分析文档。

新增清理数据：./Duck/data
子目录：
- clean_data_csv：存储清理后的CSV文件，包含上下文信息、探测与非探测数据，以及相关的`tcs_vra_scores`数据。
Notebook：
[D] [！] clean_data.ipynb：用于从原始JSON数据中提取`clean_data_csv`目录下的CSV文件。

特征数据：./Duck1.1/data
子目录：
- clean_data_csv：存储清理后的CSV文件，包含上下文信息、探测与非探测数据，以及相关的`tcs_vra_scores`数据。
- features_and_labels_csv：存储day 1模型与day N模型所用的特征与标签数据。
- figures：存储可视化图表。
Notebook：
- [D/U] features_day1.ipynb：用于提取day 1模型特征的文档。
- [D/U] features_day1_train_val_split.ipynb：用于提取day 1模型特征，将新增数据集划分为训练集与验证集，通过欠采样平衡训练集类别分布，并将新增数据与“Crabby”数据集的训练、验证集合并的文档。
- [D/U] features_dayN.ipynb：用于提取day N模型特征的文档。
- [D] add_detmagmedianmin5d_tocrabby.ipynb：用于为“Crabby”数据集样本添加`DET_mag_median_min5d`特征的文档。
- [D/U/S] Summary_plots.ipynb：用于生成数据集与特征分布的汇总表格与可视化图表的文档，论文中的多数图表均源自该Notebook。

./models目录
存储生产环境中使用的模型，这些模型由根目录下的`Overview.ipynb`生成。

Crabby数据集
该数据集的训练数据采集于2024年3月27日至8月13日期间，对应的模型于2024年12月6日至2025年2月中旬期间投入生产使用。

[D/U/S] Overview.ipynb：通用概述文档，介绍VRA的day 1模型与day N模型、排序机制以及特征重要性分析。论文中的多数图表均源自该Notebook。
[D/U/S] Key_transients.ipynb：分析day 1模型在若干重要暂现源上的表现，对比其与当前模型及预设目视审核策略的效果差异。
[D/U] Policy_evaluation.ipynb：用于评估模型与预设审核策略的协同效果——包括自动剔除的劣质天体占比、需人工审核的警报占比、是否遗漏优质暂现源，以及此类遗漏是否可接受。
[D] _Gal_candidate_policy.ipynb：用于测试标记银河候选天体的新策略的文档，该策略后于2025年2月使用“Duck”数据集重新训练VRA 1.0模型时得到应用。
[D] Policy_evaluation_new_strat.ipynb：与上述策略评估文档功能一致，新增了基于`_Gal_candidate_policy.ipynb`中测试的银河候选天体目视审核列表的步骤。

./data目录
子目录：
- clean_data_csv：存储清理后的CSV文件，包含上下文信息、探测与非探测数据，以及相关的`tcs_vra_scores`数据。
- features_and_labels_csv：存储day 1模型与day N模型所用的特征与标签数据。
- figures：存储可视化图表。
- to_reeyball：存储标记为需重新目视审核的警报数据CSV文件，以及包含重新审核决策的CSV文件。
Notebook：
- [D] [！] clean_data.ipynb：用于从原始JSON数据中提取`clean_data_csv`目录下的CSV文件。
- [D/U/S] features_day1.ipynb：用于提取day 1模型特征，划分训练集与验证集，并对训练集中占比过高的类别进行欠采样平衡的文档。
- [D/U/S] features_dayN.ipynb：用于提取day N模型特征的文档。
- [D/U/S] Summary_plots.ipynb：用于生成数据集与特征分布的汇总表格与可视化图表的文档。

./old_vs_new目录
[D] in_prod_comparison.ipynb：对比基于“Crabby”数据集训练的模型与“BMO”版本模型的差异。该模型的训练数据采集于2024年3月27日至7月中旬，二者的核心差异在于所用特征不同。该Notebook主要面向开发团队，用于展示升级VRA版本时的模型对比方法。

./train_scoring_models/hyperparameter_tuning目录
该目录包含超参数调优阶段的全部细节，相关说明可查阅hp_tuning_evaluation.ipynb文档。若仅需查看生产环境模型所用的超参数，请参阅根目录下的`Overview.ipynb`文档。

./train_scoring_models/prod_models目录
其子目录`day1`与`dayN`分别存储ATLAS服务器在重新审核与模型重训练前后使用的生产环境模型，主要用于开发团队的文档留存。

最终说明
除本说明中列出的主要目录与Notebook外，部分子目录中还包含由Notebook生成的CSV文件。为保证文档可读性，且此类文件的文件名已清晰说明其用途（其生成代码可在对应Notebook中查看），本说明与各目录的README.md未对其进行详细描述。

如有疑问？
若您对本数据档案的内容有任何疑问，请随时联系我们。我已尽力对流程进行文档编写与解释，但本说明是我参与该项目一年后补写的，对我而言清晰的内容可能对初次阅读的读者并不友好。
联系邮箱：[e-mail]