Predicting the Validity of Set Data with a Self-supervised Masked Transformer

This repository contains the official implementation and dataset preprocessing utilities for SetMTr, a self-supervised masked Transformer model designed for validity prediction of set-structured data. SetMTr solves the problem of determining whether a given set is valid, coherent, or likely to originate from a real distribution. The model is trained using a masked reconstruction objective: a portion of a valid set is masked, the model reconstructs the missing elements, and the reconstruction quality serves as a validity score. Valid sets are reconstructed more accurately due to stronger underlying co-occurrence patterns. Repository Contents: setmtr/: Core model implementation, data loaders, loss functions, metrics, utilities, and experiment configurations. datasets/preprocess.py: Converts text-format set data into pickle files used by the model. datasets/: Contains dataset format descriptions and preprocessing scripts. datasets/original_data_process/: Scripts and intermediate files for regenerating several benchmark datasets.