ZJTustc/SciTS

--- configs: - config_name: default data_files: - split: test path: meta_data.jsonl license: cc-by-nc-sa-4.0 task_categories: - time-series-forecasting - question-answering language: - en tags: - time series - timeseries - audio - benchmark - time series Reasoning - time series Classification - time series QA - time series Anomaly Detection - classification - anomaly detection size_categories: - 10K<n<100K pretty_name: 'SciTS: Scientific Time Series Understanding and Generation with LLMs' --- # SciTS: Scientific Time Series Understanding and Generation with LLMs This repository contains the official dataset for [**SciTS: Scientific Time Series Understanding and Generation with LLMs** (ICLR 2026)](https://openreview.net/forum?id=5YXccEP6uc). SciTS is a large-scale benchmark designed to evaluate the capabilities of large language models on complex scientific time series data. It spans 12 scientific disciplines, 43 distinct tasks, and includes 54,023 instances.  ## Dataset Structure The benchmark is organized into a main `meta_data.jsonl` file, a `process` directory for handling restricted datasets, and 38 individual dataset folders. Each folder is named using the convention: `Domain-DatasetName-Scene-Task`. ``` ├── process/ │ ├── process_ETT.py │ ├── process_iNaturalist.py │ ├── infer_template.py │ ├── eval.py │ └── requirements.txt ├── Domain-DatasetName-Scene-Task_1/ │ ├── raw_input_data/ │ └── raw_gt_data/ (for generation tasks) ├── Domain-DatasetName-Scene-Task_2/ │ └── raw_input_data/ ... ├── Domain-DatasetName-Scene-Task_38/ │ ├── raw_input_data/ │ └── raw_gt_data/ └── meta_data.jsonl ``` - **`process/`**: Contains utility scripts, including `process_ETT.py` and `process_iNaturalist.py` for processing restricted datasets which cannot be released directly due to license restrictions, `infer_template.py` as an inference template, `eval.py` for evaluation, and `requirements.txt` for dependency installation. - **`Dataset Folders`**: Each of the 38 folders contains the raw time series data for a specific dataset. `raw_input_data` holds the input signals, while `raw_gt_data` (present only for generation tasks) holds the ground truth output signals. - **`meta_data.jsonl`**: A JSON Lines file containing metadata for every instance in the benchmark. Each line corresponds to one data sample. ### Dataset Collection The 38 released datasets are listed below: | Domain | Dataset Folder Name | Task ID | | :--- | :--- | :--- | | Astronomy | `Astronomy-GWOSC_GW_Event-Gravitational_wave-Anomaly_detection+Event_localisation` | ASU01, ASG02 | | | `Astronomy-LEAVES-Light_curve-Classification` | ASU03 | | Earth Science | `Earth_Science-STEAD-Earthquake-Anomaly_detection+Event_localisation` | EAU01, EAG02 | | Bioacoustics | `Bioacoustics-Powdermill-Birds_vocalisation-Classification` | BIU01 | | | `Bioacoustics-MarmAudio-Marmoset_vocalisation-Classification` | BIU03 | | Meteorology | `Meteorology-TS_MQA-Weather-Anomaly_detection` | MEU01 | | | `Meteorology-TIMECAP-Rainfall-Anomaly_detection` | MEU02 | | | `Meteorology-MT_bench-Temperature-Forecasting` | MEG03 | | | `Meteorology-MT_bench-Temperature-MCQ` | MEU04 | | Economics | `Economics-FinMultiTime-Stock_closing_price-Forecasting` | ECG01 | | | `Economics-MT_bench-Stock_price-Forecasting` | ECG02 | | | `Economics-MT_bench-Stock-MCQ` | ECU03 | | Neuroscience | `Neuroscience-MDD-Depressive_disorder-Anomaly_detection` | NEU01 | | | `Neuroscience-TUEV-EEG_pattern-Classification` | NEU02 | | | `Neuroscience-TS_MQA-EEG_signal-Forecasting` | NEG03 | | | `Neuroscience-TS_MQA-EEG_signal-Imputation` | NEG04 | | | `Neuroscience-WBCIC_SHU-Motor_imagery-Classification` | NEU05 | | | `Neuroscience-Sleep-Sleep_staging-Classification` | NEU06 | | Energy | `Energy-NewsForecast-Electronic_load-Forecasting` | ENG01 | | | `Energy-TextETT-Sensor_signal_trend-Synthesis` | ENG03 | | | `Energy-TS_MQA-Comprehensive_electricity-Forecasting` | ENG04 | | | `Energy-TS_MQA-Comprehensive_electricity-Imputation` | ENG05 | | Physiology | `Physiology-PTB_XL-ECG_status-Classification` | PHU01 | | | `Physiology-TS_MQA-Physiological_signal-Forecasting` | PHG02 | | | `Physiology-TS_MQA-Physiological_signal-Imputation` | PHG03 | | | `Physiology-TS_MQA-ECG-Anomaly_detection` | PHU04 | | | `Physiology-TS_MQA-Gait_freezing-Anomaly_detection` | PHU05 | | | `Physiology-TS_MQA-Human_activity-Classification` | PHU06 | | Urbanism | `Urbanism-NewsForecast-Traffic_flow-Forecasting` | URG01 | | | `Urbanism-TS_MQA-Pedestrian_flow-Forecasting` | URG02 | | | `Urbanism-TS_MQA-Pedestrian_flow-Imputation` | URG03 | | | `Urbanism-TS_MQA-Traffic_flow-Anomaly_detection` | URU04 | | | `Urbanism-MetroTraffic-Traffic_volume-Forecasting` | URG05 | | Manufacturing | `Manufacturing-CWRU-Bearings_fault_location+Bearings_fault_size-Classification` | MFU01, MFU02 | | | `Manufacturing-MIMII_Due-Machine_malfunction-Anomaly_detection` | MFU03 | | Radar | `Radar-RadSeg-Coding_scheme-Classification` | RAU01 | | | `Radar-RadarCom-Modes_and_modulation-Classification` | RAU02 | | Math | `Math-Chaotic-Chaotic_system-Forecasting` | MAG01 | ## `meta_data.jsonl` Format Each line in this file is a JSON object with the following structure, providing all necessary metadata to load and use a data sample. ```json { "task_id": ["TASK_ID"], // List of task IDs associated with this sample (e.g., ["ASU03"] or ["ASU01", "ASG02"] for merged datasets) "id": "DATASET_ID", // Unique identifier of this sample within the dataset "data_type": "csv"/"npy"/"wav"/"flac", // File format of the raw time series data "input_ts":{ "num_channel": int, // Number of channels (dimensions) in the input signal "channel_detail": [], // List of channel names, empty if none "path": "raw_input_data/sample_001_input.npy", "length": int, // Length of the input time series "timestamps": [], // Auxiliary timestamp information, empty if none "fs": int // Sampling frequency in Hz }, "input_text": "INPUT_TEXT", // Textual prompt or task instruction provided as input "gt_text": "GT_TEXT", // Ground truth textual answer (for understanding tasks; empty for generation tasks) "gt_ts": { "path": "raw_gt_data/sample_001_output.npy", "length": int // Length of the ground truth time series }, "gt_result": { ... }, // Structured ground truth result; format varies by task type (see below) "meta_data": {} // Additional metadata from the original data source } ``` ### `gt_result` Field Format The structure of the `gt_result` field varies depending on the task type. This field provides the original ground truth for metric computation. **1. MCQ** ```json "gt_result": { "answer": "TEXT" // The correct textual answer } ``` **2. Synthesis, Forecasting, Imputation** ```json "gt_result": { "num_channel": int, // Number of channels (dimensions) in the ground truth signal "channel_detail": [], // List of channel names, empty if none "timestamps": [] // Auxiliary timestamp information, empty if none } ``` **3. Classification** For the `CWRU` dataset, which involves two classification sub-tasks, the category keys in class_list and gt_class are `"diameter"` and `"position"` respectively. For all other classification tasks, the category key is `"default"`. ```json "gt_result": { "class_list": { "default": ["class_A", "class_B"], // List of candidate classes for each category ... }, "gt_class": { "default": ["GT_CLASS"], // Ground truth class label for each category ... } } ``` **4. Anomaly Detection** ```json "gt_result": { "contain": Boolean // Boolean indicating if the required event is present } ``` **5. Anomaly Detection + Event Localisation** For the `GWOSC GW Event` and `STEAD` datasets, each of which includes both an `Anomaly Detection` task and an `Event Localisation` task, the gt_result field is defined in the following combined format: ```json "gt_result": { "contain": Boolean, // Boolean indicating if the required event is present "start_time": int // The event index if contain is true, else null } ``` ## Handling Restricted Datasets Due to license restrictions, the **ETT** (`ENG02`) and **iNaturalist** (`BIU02`) datasets are not directly included in this repository. To use them, the user need to download the original data and run the provided processing scripts. **Step 1: Download the Data** - **ETT**: Download `ETTh1.csv` from the official repository: [https://github.com/zhouhaoyi/ETDataset](https://github.com/zhouhaoyi/ETDataset) - **iNaturalist**: Download the `Test Recordings` from the official repository: [https://github.com/visipedia/inat_sounds/tree/main/2024](https://github.com/visipedia/inat_sounds/tree/main/2024) **Step 2: Install Dependencies** Before running the processing scripts, install the required Python packages: ```shell pip install -r process/requirements.txt ``` **Step 3: Run the Processing Script** Place the downloaded files into a local directory. Then, from the root of this repository, run the corresponding script to process the data into the standard benchmark format. - For ETT: ```shell python process/process_ETT.py --data_path /path/to/your/ETTh1.csv ``` - For iNaturalist: ```shell python process/process_iNaturalist.py --data_folder /path/to/your/iNaturalist/test ``` This will generate the `Energy-ETT-Transformer_sensor_signal-Forecasting` and `Bioacoustics-INaturalist-Animal_vocalisation-Classification` folders along with their `raw_input_data`, `raw_gt_data` subdirectories, as well as the processed test files. ## Baseline Inference and Evaluation The `process` directory also includes scripts for running inference and evaluating the results. ### Inference `process/infer_template.py`: Template code for the inference script. Implement the `initialize_model` function, then inference can be done by running: ```shell python process/infer_template.py --scits_dir /path/to/scits_dir --output_dir /path/to/output_dir ``` ### Evaluation `process/eval.py`: Evaluation script. Run: ```shell python process/eval.py evaluate --infer_dir /path/to/infer_dir ``` The evaluation results will be saved to `/path/to/infer_dir/results/`. ## Citation If you use the SciTS benchmark, please cite the paper: ```bibtex @inproceedings{ wu2026scits, title={Sci{TS}: {S}cientific Time Series Understanding and Generation with {LLM}s}, author={Wen Wu and Ziyang Zhang and Liwei Liu and Xuenan Xu and Jimin Zhuang and Ke Fan and Qitan Lv and Junlin Liu and Chen Zhang and Zheqi Yuan and Siyuan Hou and Tianyi Lin and Kai Chen and Bowen Zhou and Chao Zhang}, booktitle={The Fourteenth International Conference on Learning Representations}, year={2026}, url={https://openreview.net/forum?id=5YXccEP6uc} } ```--- 配置项： - 配置名称：default 数据文件： - 拆分集：test 路径：meta_data.jsonl 许可证：cc-by-nc-sa-4.0 任务类别： - 时间序列预测（time-series-forecasting） - 问答（question-answering） 语言： - en 标签： - 时间序列 - 时序数据 - 音频 - 基准测试 - 时间序列推理 - 时间序列分类（time series Classification） - 时间序列问答（time series QA） - 时间序列异常检测（time series Anomaly Detection） - 分类（classification） - 异常检测（anomaly detection） 样本规模： - 10K<n<100K 友好名称：'SciTS：基于大语言模型（Large Language Model, LLM）的科学时间序列理解与生成' --- # SciTS：基于大语言模型的科学时间序列理解与生成 本仓库为[**SciTS：基于大语言模型的科学时间序列理解与生成**（ICLR 2026）](https://openreview.net/forum?id=5YXccEP6uc)的官方数据集。SciTS是一款大规模基准测试集，旨在评估大语言模型在复杂科学时间序列数据上的能力。该数据集涵盖12个科学学科、43项独立任务，总计包含54023条数据样本。  ## 数据集结构 本基准测试集由主元数据文件`meta_data.jsonl`、用于处理受限数据集的`process`目录，以及38个独立数据集文件夹组成。所有文件夹均遵循`领域-数据集名称-场景-任务`的命名规范。 ├── process/ │ ├── process_ETT.py │ ├── process_iNaturalist.py │ ├── infer_template.py │ ├── eval.py │ └── requirements.txt ├── Domain-DatasetName-Scene-Task_1/ │ ├── raw_input_data/ │ └── raw_gt_data/ (for generation tasks) ├── Domain-DatasetName-Scene-Task_2/ │ └── raw_input_data/ ... ├── Domain-DatasetName-Scene-Task_38/ │ ├── raw_input_data/ │ └── raw_gt_data/ └── meta_data.jsonl - **`process/`**：包含各类实用脚本，其中`process_ETT.py`与`process_iNaturalist.py`用于处理因许可证限制无法直接发布的受限数据集；`infer_template.py`为推理模板脚本；`eval.py`用于模型评估；`requirements.txt`则列出了依赖安装所需的Python包。 - **`数据集文件夹`**：38个文件夹各自对应一个特定数据集的原始时间序列数据。其中`raw_input_data`目录存储输入信号，`raw_gt_data`目录（仅生成类任务包含）存储真实输出信号。 - **`meta_data.jsonl`**：采用JSON Lines格式的元数据文件，存储了基准测试集中所有数据样本的元信息，每一行对应一条数据样本。 ### 数据集采集 本次发布的38个数据集如下表所示： | 学科领域 | 数据集文件夹名称 | 任务ID | | :--- | :--- | :--- | | 天文学 | `Astronomy-GWOSC_GW_Event-Gravitational_wave-Anomaly_detection+Event_localisation` | ASU01, ASG02 | | | `Astronomy-LEAVES-Light_curve-Classification` | ASU03 | | 地球科学 | `Earth_Science-STEAD-Earthquake-Anomaly_detection+Event_localisation` | EAU01, EAG02 | | 生物声学 | `Bioacoustics-Powdermill-Birds_vocalisation-Classification` | BIU01 | | | `Bioacoustics-MarmAudio-Marmoset_vocalisation-Classification` | BIU03 | | 气象学 | `Meteorology-TS_MQA-Weather-Anomaly_detection` | MEU01 | | | `Meteorology-TIMECAP-Rainfall-Anomaly_detection` | MEU02 | | | `Meteorology-MT_bench-Temperature-Forecasting` | MEG03 | | | `Meteorology-MT_bench-Temperature-MCQ` | MEU04 | | 经济学 | `Economics-FinMultiTime-Stock_closing_price-Forecasting` | ECG01 | | | `Economics-MT_bench-Stock_price-Forecasting` | ECG02 | | | `Economics-MT_bench-Stock-MCQ` | ECU03 | | 神经科学 | `Neuroscience-MDD-Depressive_disorder-Anomaly_detection` | NEU01 | | | `Neuroscience-TUEV-EEG_pattern-Classification` | NEU02 | | | `Neuroscience-TS_MQA-EEG_signal-Forecasting` | NEG03 | | | `Neuroscience-TS_MQA-EEG_signal-Imputation` | NEG04 | | | `Neuroscience-WBCIC_SHU-Motor_imagery-Classification` | NEU05 | | | `Neuroscience-Sleep-Sleep_staging-Classification` | NEU06 | | 能源科学 | `Energy-NewsForecast-Electronic_load-Forecasting` | ENG01 | | | `Energy-TextETT-Sensor_signal_trend-Synthesis` | ENG03 | | | `Energy-TS_MQA-Comprehensive_electricity-Forecasting` | ENG04 | | | `Energy-TS_MQA-Comprehensive_electricity-Imputation` | ENG05 | | 生理学 | `Physiology-PTB_XL-ECG_status-Classification` | PHU01 | | | `Physiology-TS_MQA-Physiological_signal-Forecasting` | PHG02 | | | `Physiology-TS_MQA-Physiological_signal-Imputation` | PHG03 | | | `Physiology-TS_MQA-ECG-Anomaly_detection` | PHU04 | | | `Physiology-TS_MQA-Gait_freezing-Anomaly_detection` | PHU05 | | | `Physiology-TS_MQA-Human_activity-Classification` | PHU06 | | 城市学 | `Urbanism-NewsForecast-Traffic_flow-Forecasting` | URG01 | | | `Urbanism-TS_MQA-Pedestrian_flow-Forecasting` | URG02 | | | `Urbanism-TS_MQA-Pedestrian_flow-Imputation` | URG03 | | | `Urbanism-TS_MQA-Traffic_flow-Anomaly_detection` | URU04 | | | `Urbanism-MetroTraffic-Traffic_volume-Forecasting` | URG05 | | 制造业 | `Manufacturing-CWRU-Bearings_fault_location+Bearings_fault_size-Classification` | MFU01, MFU02 | | | `Manufacturing-MIMII_Due-Machine_malfunction-Anomaly_detection` | MFU03 | | 雷达 | `Radar-RadSeg-Coding_scheme-Classification` | RAU01 | | | `Radar-RadarCom-Modes_and_modulation-Classification` | RAU02 | | 数学 | `Math-Chaotic-Chaotic_system-Forecasting` | MAG01 | ## `meta_data.jsonl` 格式规范 该文件的每一行均为一个JSON对象，包含加载与使用数据样本所需的全部元信息，格式如下： json { "task_id": ["TASK_ID"], // 当前样本关联的任务ID列表（例如合并数据集可使用["ASU01", "ASG02"]，单任务数据集使用["ASU03"]） "id": "DATASET_ID", // 数据集内当前样本的唯一标识符 "data_type": "csv"/"npy"/"wav"/"flac", // 原始时间序列数据的文件格式 "input_ts":{ "num_channel": int, // 输入信号的通道（维度）数 "channel_detail": [], // 通道名称列表，无则为空数组 "path": "raw_input_data/sample_001_input.npy", "length": int, // 输入时间序列的长度 "timestamps": [], // 辅助时间戳信息，无则为空数组 "fs": int // 采样频率，单位为Hz }, "input_text": "INPUT_TEXT", // 作为输入的文本提示或任务指令 "gt_text": "GT_TEXT", // 真实文本答案（仅理解类任务包含，生成类任务留空） "gt_ts": { "path": "raw_gt_data/sample_001_output.npy", "length": int // 真实时间序列的长度 }, "gt_result": { ... }, // 结构化真实结果，格式随任务类型而异（详见下文） "meta_data": {} // 原始数据源的附加元信息 } ### `gt_result` 字段格式 `gt_result`字段的结构随任务类型有所不同，该字段提供用于指标计算的原始真实值。 **1. 多项选择题（Multiple Choice Question, MCQ）** json "gt_result": { "answer": "TEXT" // 正确的文本答案 } **2. 序列生成、预测与补全** json "gt_result": { "num_channel": int, // 真实信号的通道（维度）数 "channel_detail": [], // 通道名称列表，无则为空数组 "timestamps": [] // 辅助时间戳信息，无则为空数组 } **3. 分类任务** 针对包含两项分类子任务的`CWRU`数据集，`class_list`与`gt_class`中的类别键分别为`"diameter"`（轴承直径）与`"position"`（故障位置）；其余所有分类任务的类别键均为`"default"`。 json "gt_result": { "class_list": { "default": ["class_A", "class_B"], // 每个类别的候选类别列表 ... }, "gt_class": { "default": ["GT_CLASS"], // 每个类别的真实类别标签 ... } } **4. 异常检测任务** json "gt_result": { "contain": Boolean // 指示是否包含目标事件的布尔值 } **5. 异常检测与事件定位联合任务** 针对同时包含异常检测与事件定位任务的`GWOSC GW Event`和`STEAD`数据集，`gt_result`字段采用以下联合格式： json "gt_result": { "contain": Boolean, // 指示是否包含目标事件的布尔值 "start_time": int // 当`contain`为`true`时为事件索引，否则为`null` } ## 受限数据集处理 由于许可证限制，**ETT**（任务ID：`ENG02`）与**iNaturalist**（任务ID：`BIU02`）数据集未直接包含在本仓库中。若需使用这两类数据集，用户需自行下载原始数据并运行提供的处理脚本。 **步骤1：下载数据** - **ETT**：从官方仓库下载`ETTh1.csv`：[https://github.com/zhouhaoyi/ETDataset](https://github.com/zhouhaoyi/ETDataset) - **iNaturalist**：从官方仓库下载`Test Recordings`：[https://github.com/visipedia/inat_sounds/tree/main/2024](https://github.com/visipedia/inat_sounds/tree/main/2024) **步骤2：安装依赖** 运行处理脚本前，请先安装所需的Python包： shell pip install -r process/requirements.txt **步骤3：运行处理脚本** 将下载的文件放入本地目录，随后在本仓库根目录下运行对应脚本，将数据处理为标准基准测试集格式： - 针对ETT： shell python process/process_ETT.py --data_path /path/to/your/ETTh1.csv - 针对iNaturalist： shell python process/process_iNaturalist.py --data_folder /path/to/your/iNaturalist/test 该操作将生成`Energy-ETT-Transformer_sensor_signal-Forecasting`与`Bioacoustics-INaturalist-Animal_vocalisation-Classification`文件夹及其`raw_input_data`、`raw_gt_data`子目录，以及处理后的测试文件。 ## 基线推理与评估 `process`目录还包含用于模型推理与结果评估的脚本。 ### 推理 `process/infer_template.py`：推理脚本模板。实现`initialize_model`函数后，可通过以下命令运行推理： shell python process/infer_template.py --scits_dir /path/to/scits_dir --output_dir /path/to/output_dir ### 评估 `process/eval.py`：评估脚本。运行以下命令即可： shell python process/eval.py evaluate --infer_dir /path/to/infer_dir 评估结果将保存至`/path/to/infer_dir/results/`。 ## 引用声明 若使用SciTS基准测试集，请引用以下论文： bibtex @inproceedings{ wu2026scits, title={Sci{TS}: {S}cientific Time Series Understanding and Generation with {LLM}s}, author={Wen Wu and Ziyang Zhang and Liwei Liu and Xuenan Xu and Jimin Zhuang and Ke Fan and Qitan Lv and Junlin Liu and Chen Zhang and Zheqi Yuan and Siyuan Hou and Tianyi Lin and Kai Chen and Bowen Zhou and Chao Zhang}, booktitle={The Fourteenth International Conference on Learning Representations}, year={2026}, url={https://openreview.net/forum?id=5YXccEP6uc} }