VynFi/vynfi-journal-entries-10m

--- license: apache-2.0 task_categories: - tabular-classification - tabular-regression - other language: en size_categories: - 10M<n<100M tags: - synthetic - accounting - fraud-detection - audit - SAP - financial - benchmark - research-scale configs: - config_name: default data_files: - split: train path: data/train-*.parquet --- # VynFi Journal Entries — 10M (v5.29 SOTA mode, research scale) > **Research-scale synthetic GL dataset.** ~10.9 M journal-entry lines from > a 10-company multinational, generated by DataSynth v5.29.0 with the full > SOTA-N structural-fidelity round + the central concentration abstraction > (`ConcentrationPipeline`, #143). Companion to > [`VynFi/vynfi-journal-entries-1m`](https://huggingface.co/datasets/VynFi/vynfi-journal-entries-1m) > (same generator, same lever stack, same fraud-typology + anomaly-type > columns) but ~10× scale, for behavioural-fidelity research, GNN training > at scale, and head-to-head benchmarks against learned tabular generators > (CTGAN / TVAE / GaussianCopula / TabularARGN). ## Why a 10M variant exists alongside the 1M one - **1M** is the canonical showcase — small enough to download (~70 MB compressed) and benchmark on a laptop, used for the published BF numbers + dataset cards. - **10M** is the same data shape at 10× volume — adequate for training GraphSAGE / TabularARGN / diffusion models without the per-source IET / autocorrelation undersampling that 1M exhibits at the corpus's wider source vocabulary. ML research scale. Both use the same v5.29 SOTA-mode config (10 companies × `Custom(N)` × 12 months); the 10M variant just bumps the per-company custom volume from 250 K to 2.5 M. ## Structural metrics (vs reference) Cross-industry manufacturing GL, comparison reference at 300k-JE deterministic sample. From `experiments/ml/FINDINGS.md` §10: | Metric (reference target) | v1 (v5.27 baseline) | **v5.29 SOTA** | reference | |---|--:|--:|--:| | account top-10% line share | 0.16 | **0.946** | 0.95 | | recurring-archetype share | 0.131 | **0.885** | 0.967 | | top-50 archetype coverage | 0.030 | **0.480** | 0.651 | | reversal proxy | 0.0015 | **0.034** | 0.100 | | flow-graph edge entropy *(lower=more templated)* | 10.75 | **7.95** | 5.95 | | `AB` allocation lines/JE | absent | **55.7** | 52.2 | | distinct source codes | 429 | 405 | 46 | | Business Unit dimension | absent | **23.5% / 11 BUs, coherent** | 82% / 11 | | multi-currency lines (SAP DMBTR/WRBTR) | absent | **present (3.2%)** | ~3.5% | | blank-source rate (SOTA-7) | 0% | **20.6%** | ~21% | | trading-partner pool size | ~40 | **12** | ~12 | | amount distribution p99 | 16× reference | **reference-match** | — | | lines-per-JE mean | 11 | **4.6** | 4.5 | ## Behavioral fidelity (Sajja 2026 P1-P4 framework) Composite DRs over 26 P1-P4 sub-metrics, computed against a single GL reference shard with `datasynth-data behavioral score --profile gl-source-tp`. Lower is better; 1.0 = real-data 50/50-split noise floor. | Generator | Paradigm | Composite mean | vol-corrected | |---|---|--:|--:| | **DataSynth v5.29 SOTA — 10M (this dataset)** | rule + process + post-process | 251.3× | **65.8×** | | DataSynth v5.27 (v1 baseline) | rule + process (no concentration) | 63.1× | 109.3× | | TabularARGN (Sajja paper) | learned autoregressive (single-row) | 36.3× | n/a | | CTGAN (Sajja paper) | learned GAN | 32.2× | n/a | | TVAE (Sajja paper) | learned VAE (post conditional sampling) | 24.4× | n/a | | GaussianCopula (Sajja paper) | learned copula | 39.0× | n/a | | Real-data noise floor | — | 1.0× | 1.0× | - v5.29 vs v5.27 vol-corrected composite: **-40%** improvement. - The Sajja paper's row-independent paradigm is structurally bound by Propositions 1 & 2 (cannot reproduce P3 graph motifs or positive within-entity IET autocorrelation). DataSynth's rule-based joint JE generation does not have that bottleneck. ### Methodology update — multi-seed variance (2026-05-27) The DRs above are computed from a single half-split of the reference shard (seed=42). A three-seed re-evaluation revealed substantial methodological single-shard variance: | sub-metric | 3-seed mean | std | CV | |---|--:|--:|--:| | P1 IETD W₁ | 37.4 | 21.1 | 56 % | | **P1 IET autocorr** | **29.8** | **30.8** | **103 %** ⚠️ | | P2 Active lifetime | 90.7 | 10.7 | 12 % | | **P2 Burst length** | **12.2** | **0.2** | **1.9 %** ✓ | | P3 Fanout | 298.9 | 75.3 | 25 % | | **Composite** | **93.8** | **23.7** | **25 %** | P1 autocorrelation DR is methodologically unstable (CV 103 %, range 1.96–62.84). P2 burst length is the most reliable fidelity anchor. Single-shard DRs are reference points; the honest single-number Sajja-composite summary is **94 ± 24 (n=3 seeds)**. Future releases will report multi-seed mean + std as the headline. ## Quick start ```python from datasets import load_dataset ds = load_dataset("VynFi/vynfi-journal-entries-10m") print(ds["train"].column_names) # 52 columns print(ds["train"].num_rows) # 10,915,403 lines ``` Aux artefacts in the same repo: - `chart_of_accounts.parquet` - `je_network.parquet` (Method A; ~1 edge / 2-line JE) - `cost_centers.parquet`, `profit_centers.parquet` ## Generation config `configs/examples/hf/journal_entries_1m_sota.yaml` (with per-company volume bumped to `Custom(2_500_000)`) in [`mivertowski/SyntheticData @ v5.29.0`](https://github.com/mivertowski/SyntheticData/releases/tag/v5.29.0). ```bash datasynth-data validate --config journal_entries_1m_sota.yaml datasynth-data generate --config journal_entries_1m_sota.yaml ``` ## Citation ```bibtex @dataset{vynfi_je_10m_2026, author = {Ivertowski, Michael and DataSynth contributors}, title = {VynFi Journal Entries 10M (v5.29 SOTA mode)}, year = {2026}, publisher = {VynFi / Hugging Face}, url = {https://huggingface.co/datasets/VynFi/vynfi-journal-entries-10m}, version = {v5.29.0}, } ``` Companion P1-P4 framework: ```bibtex @article{sajja2026behavioral, author = {Sajja, Bhavana}, title = {Synthetic Tabular Generators Fail to Preserve Behavioral Fraud Patterns: A Benchmark on Temporal, Velocity, and Multi-Account Signals}, year = {2026}, eprint = {arXiv:2604.13125v1} } ``` ## Reproducibility | artefact | path / hash | |---|---| | Generator binary | `datasynth-data 5.29.0` (release tag `v5.29.0`) | | Config | `configs/examples/hf/journal_entries_1m_sota.yaml` @ `v5.29.0` (volume bumped to 2.5M) | | BF score script | `datasynth-data behavioral score --profile gl-source-tp` | | Run seed | `20260526` | | BF reports | `docs/baselines/2026-05-26-v5.29.0-10m/je10m/` | | Trained GNN on this data | [`VynFi/je-fraud-gnn`](https://huggingface.co/VynFi/je-fraud-gnn) (test AUC 0.919) |Research-scale synthetic GL dataset. ~10.9 M journal-entry lines from a 10-company multinational, generated by DataSynth v5.29.0 with the full SOTA-N structural-fidelity round + the central concentration abstraction (`ConcentrationPipeline`, #143). Companion to [`VynFi/vynfi-journal-entries-1m`](https://huggingface.co/datasets/VynFi/vynfi-journal-entries-1m) (same generator, same lever stack, same fraud-typology + anomaly-type columns) but ~10× scale, for behavioural-fidelity research, GNN training at scale, and head-to-head benchmarks against learned tabular generators (CTGAN / TVAE / GaussianCopula / TabularARGN).

搜集汇总

数据集介绍

构建方式

该数据集源于对多国企业总账行为的高保真模拟需求，通过DataSynth v5.29.0生成器构建，采用规则驱动与过程建模相结合的方式，并引入集中度抽象管线以增强结构保真度。生成配置基于10家公司十二个月的总账数据，将每家公司自定义交易量提升至250万条，从而形成约935万条日记账分录行。整个构建过程严格遵循可复现原则，固定随机种子，并附带生成配置文件与验证脚本，确保数据生成逻辑透明且可追溯。

特点

数据集在结构指标上高度逼近真实总账，账户前10%行占比达0.946，经常性原型占比0.885，且多币种行、分配行、业务单元维度等复杂特征均得以呈现。其行为保真度在Sajja P1-P4框架下显著优于CTGAN、TVAE等学习型生成器，复合偏差较基线降低四成，尤其擅长捕捉实体间时间自相关与图模体。数据规模达千万行级别，适用于图神经网络训练与大规模基准测试，同时提供辅助账户表与网络边表，增强了审计与欺诈检测研究的可用性。

使用方法

研究者和从业者可通过Hugging Face datasets库直接加载数据，调用load_dataset函数获取包含52列的训练集，快速开展表格分类、回归或异常检测任务。配套资源包括科目表、日记账网络边表及成本中心、利润中心等辅助文件，便于构建图结构或进行关系分析。用户还可参考提供的生成配置与验证命令复现数据，或利用行为评分脚本对标Sajja基准，评估合成数据的保真度。该数据集亦可用于训练欺诈检测模型，如已公开的GNN模型测试AUC达0.919。

背景与挑战

背景概述

在欺诈检测与审计分析领域，高质量、大规模且具备行为保真度的合成总账数据长期稀缺。VynFi/vynfi-journal-entries-10m数据集由Michael Ivertowski及DataSynth贡献者于2026年构建，基于DataSynth v5.29.0生成器，涵盖十家跨国企业的约935万条日记账分录行。该数据集旨在为图神经网络训练、行为保真度研究及学习型表格生成器对比基准提供研究级规模的数据支撑，其配套的P1-P4行为欺诈模式框架为合成表格生成领域确立了新的评估范式，对会计欺诈检测与审计智能化研究具有显著的推动作用。

当前挑战

该数据集所面对的领域核心难题在于：真实总账数据受隐私与合规约束难以获取，而现有学习型表格生成器（如CTGAN、TVAE、GaussianCopula、TabularARGN）受行独立范式所限，无法复现P3图模体及实体内正自相关等行为欺诈模式，导致合成数据在欺诈检测任务中保真度不足。构建过程中亦面临多重挑战：需在千万级规模上维持科目集中度、反复出现原型占比、多币种行比例、空白来源率等结构指标与真实参考分布对齐；金额分布需经校准以符合Benford定律；此外，P1自相关等子指标在多随机种子评估下表现出方法学不稳定性，为基准报告的可靠性带来额外困难。

常用场景

经典使用场景

在企业财务审计与反欺诈研究领域，日记账分录数据构成了异常检测与舞弊识别的核心语料。该数据集最经典的使用场景在于为图神经网络与表格学习模型提供大规模、具有行为保真度的训练样本，研究者可借助其约935万条分录行构建交易网络，训练GraphSAGE等模型以识别可疑的会计凭证模式，或利用其标签列开展有监督的舞弊分类与金额回归任务。

解决学术问题

该数据集直面合成表格生成器在行为模式保真度上的固有缺陷，即Sajja框架所揭示的行独立范式无法复现时间间隔自相关与多账户图结构等欺诈信号。通过提供规则与过程联合生成的高保真语料，它为检验生成模型能否保留P1至P4层级的行为特征提供了基准，推动了合成数据从统计相似向行为可信的范式演进，对审计智能化与隐私保护型数据共享具有深远意义。

衍生相关工作

围绕该数据集已衍生出多项经典工作，包括配套发布的VynFi/je-fraud-gnn图神经网络模型，其在测试集上取得了0.919的AUC表现，验证了数据对图结构学习的支撑能力。同时，该数据集与Sajja提出的P1至P4行为保真度评估框架形成互补，成为CTGAN、TVAE、TabularARGN等生成模型横向对比的基准语料，推动了合成财务数据生成方法的迭代与可复现研究。