Table Union Search Benchmarks

🚨V2 UPDATE: LB-OpenData and LB-WebTable have been updated by the original authors as of May 20th, 2025. (https://github.com/RLGen/LakeBench). We've preprocessed (see below) and repackaged them accordingly. We've also added a larger truncated version of LB-OpenData (30k rows max per table). The results in our paper have been updated to reflect these changes. While there are slight variations, they do not affect the core conclusions of our study.

🐟"Something's Fishy in the Data Lake: A Critical Re-evaluation of Table Union Search Benchmarks"

Please note that these benchmarks were not created by us. We simply repackaged them into a cleaner and more unified format to ensure compatibility with our code, which is available at: https://github.com/Allaa-boutaleb/fishy-tus. You can download the benchmarks by following the procedures and links in their original repositories:

- SANTOS: https://github.com/northeastern-datalab/santos- TUS-SMALL / TUS-LARGE: https://github.com/RJMillerLab/table-union-search-benchmark- PYLON: https://github.com/superctj/pylon/tree/main- UGEN-V1 / UGEN-V2: https://github.com/northeastern-datalab/gen- LB-OPENDATA / LB-WEBTABLE: https://github.com/RLGen/LakeBench

Each of these benchmarks requires different preprocessing steps before they can be used with our code to reproduce the results.

Inside each .tar.gz file, you will find:- a folder named `datalake/` containing datalake CSV tables- a folder named `query/` containing query CSV tables- a file named `benchmark.pkl`, which represents the benchmark ground truth. The `benchmark.pkl` file is a dictionary where the key is the name of the query table and the value is a list of unionable candidate table names. Example: key: "query.csv", value: ["candidate_1.csv", "candidate_2.csv", ...]

Notes:

- TUS and TUS-LARGE already include the sampled set of queries used for evaluation, located in the `query/` folder. The original full set of queries is also available in the `all_query/` subfolder.- For LakeBench benchmarks (LB-OPENDATA / LB-WEBTABLE), additional preprocessing was done:  a) Removed any table not referenced in the ground truth  b) Removed any ground truth entry pointing to a non-existent CSV  c) Added each query table as a candidate to itself for consistency

- For LB-OpenData, a smaller version was also created, limiting tables to a maximum of 1,000 rows. This was necessary due to the high computational cost of running some methods (e.g., Starmie) on the full version.

LICENSE:

The datasets are shared under the CC BY-SA 4.0 license.

DISCLAIMER:

THIS DATA IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND. WE ARE NOT THE ORIGINAL AUTHORS OF THE BENCHMARKS, AND WE ONLY REDISTRIBUTED THEM IN A MORE ACCESSIBLE FORMAT TO FACILITATE REPRODUCIBILITY.