MLRegTest: A benchmark for the machine learning of regular languages

MLRegTest is a benchmark for machine learning systems on sequence classification, which contains training, development, and test sets from 1,800 regular languages. MLRegTest organizes its languages according to their logical complexity (monadic second order, first order, propositional, or monomial expressions) and the kind of logical literals (string, tier-string, subsequence, or combinations thereof). The logical complexity and choice of literal provides a systematic way to understand different kinds of long-distance dependencies in regular languages, and therefore to understand the capacities of different ML systems to learn such long-distance dependencies., The languages were generated by creating finite-state acceptors and the datasets were generated by sampling from these finite-state acceptors. The scripts and software used for these processes are open source and available. For details, see https://github.com/heinz-jeffrey/subregular-learning. Details are described in the arxiv preprint \"MLRegTest: A Benchmark for the Machine Learning of Regular Languages\"., , # MLRegTest: A benchmark for the machine learning of regular languages

[https://doi.org/10.5061/dryad.dncjsxm4h](https://doi.org/10.5061/dryad.dncjsxm4h)

MLRegTest provides training and testing data for 1800 regular languages.

This repository contains three gzipped tar archives.

\> data.tar.gz (21GB)
\> languages.tar.gz (4.5MB)
\> models.tar.gz (76GB)

When uncompressed, these yield three directories, described in detail below.

\> data (43GB)
\> languages (38MB)
\> models (87GB)

# Languages

Languages are named according to the scheme `Sigma.Tau.class.k.t.i.plebby`, where `Sigma` is a two-digit alphabet size, `Tau` a two-digit number of salient symbols (the 'tier'), `class` the named subregular class, `k` the width of factors used (if applicable), `t` the threshold counted to (if applicable), and `i` a unique identifier. The table below unabbreviates the class names, and shows how many languages of each class there are.

| class | name ...