Table Union Search Benchmarks

This repository contains the data used to reproduce the results of the paper: "Something's Fishy in the Data Lake: A Critical Re-evaluation of Table Union Search Benchmarks"

Please note that these benchmarks were not created by us. We simply repackaged them into a cleaner and more unified format to ensure compatibility with our code, which is available at: https://github.com/Allaa-boutaleb/fishy-tus. You can download the benchmarks by following the procedures and links in their original repositories:

- SANTOS: https://github.com/northeastern-datalab/santos- TUS-SMALL / TUS-LARGE: https://github.com/RJMillerLab/table-union-search-benchmark- PYLON: https://github.com/superctj/pylon/tree/main- UGEN-V1 / UGEN-V2: https://github.com/northeastern-datalab/gen- LB-OPENDATA / LB-WEBTABLE: https://github.com/RLGen/LakeBench

Each of these benchmarks requires different preprocessing steps before they can be used with our code to reproduce the results.

Inside each .tar.gz file, you will find:- a folder named `datalake/` containing datalake CSV tables- a folder named `query/` containing query CSV tables- a file named `benchmark.pkl`, which represents the benchmark ground truth. The `benchmark.pkl` file is a dictionary where the key is the name of the query table and the value is a list of unionable candidate table names. Example: key: "query.csv", value: ["candidate_1.csv", "candidate_2.csv", ...]

Notes:

- TUS and TUS-LARGE already include the sampled set of queries used for evaluation, located in the `query/` folder. The original full set of queries is also available in the `all_query/` subfolder.- For LakeBench benchmarks (LB-OPENDATA / LB-WEBTABLE), additional preprocessing was done:  a) Removed any table not referenced in the ground truth  b) Removed any ground truth entry pointing to a non-existent CSV  c) Added each query table as a candidate to itself for consistency

- For LB-OpenData, a smaller version was also created, limiting tables to a maximum of 1,000 rows. This was necessary due to the high computational cost of running some methods (e.g., Starmie) on the full version.

LICENSE:

The datasets are shared under the CC BY-SA 4.0 license.

DISCLAIMER:

THIS DATA IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND. WE ARE NOT THE ORIGINAL AUTHORS OF THE BENCHMARKS, AND WE ONLY REDISTRIBUTED THEM IN A MORE ACCESSIBLE FORMAT TO FACILITATE REPRODUCIBILITY.