FEMP-X: A Witness-Augmented Dataset for Candidate Functionally Equivalent Java Method Pairs

Unlike datasets that only provide pair-level labels, FEMP-X adds execution-validated witnesses of inequivalence when candidate equivalence can be falsified. Each retained witness is backed by an executable JUnit test and exposes a concrete behavioral difference between two methods, such as a difference in returned values, side effects, exception behavior, or termination behavior.

What this release contains

This release contains a SQLite-based dataset artifact and accompanying documentation for analyzing candidate functionally equivalent Java method pairs.

The main release includes:

the SQLite dataset file

`README.md`

`LICENSE`

`llm_inspected_pairs.md` documenting the main witness-augmented table

The generated JUnit test cases are not distributed as separate files in this release. Instead, they are stored inside the SQLite database as source text in the `test_src` field of `llm_inspected_pairs`.

Background

FEMP-X is built on top of the original candidate pool introduced by FEMPDataset. The original candidate pool contains 13,710 Java method pairs collected as candidates for functional equivalence through automated test generation and cross-execution.

FEMP-X extends that pool with a new witness-backed data layer constructed with GPT-5.4 through Codex in an agentic workflow. For each candidate pair, the workflow attempts to generate a concrete distinguishing test, executes it immediately, and retains it only when the test actually reproduces a behavioral difference.

Original dataset

FEMP-X is built on top of FEMPDataset, which is publicly available at:

https://github.com/YoshikiHigo/FEMPDataset

The original dataset is described in the following paper:

Yoshiki Higo. "Dataset of Functionally Equivalent Java Methods and Its Application to Evaluating Clone Detection Tools." IEICE Transactions on Information and Systems, E107.D(6):751--760, 2024. DOI: 10.1587/transinf.2023EDP7268

Main data characteristics

The main witness-augmented table is `llm_inspected_pairs`.

At the time of this release, the table contains:

total rows: 13,710

`witness_found`: 4,193

`no_witness_found`: 9,517

Breakdown of `witness_found` rows by `difference_type`:

- `return_value`: 2,676

- `exception`: 798

- `no_termination`: 415

- `side_effect`: 304

Detailed distribution by human label:

- `human_equivalent` / `no_witness_found`: 1,225

- `human_equivalent` / `witness_found`: 117

- `human_inequivalent` / `no_witness_found`: 4

- `human_inequivalent` / `witness_found`: 848

- `not_manually_inspected` / `no_witness_found`: 8,288

- `not_manually_inspected` / `witness_found`: 3,228

Interpretation notes

FEMP-X is intentionally asymmetric.

A pair with `witness_found` has an execution-validated witness of inequivalence. By contrast, `no_witness_found` does not prove equivalence. It only means that the present workflow did not obtain an executable witness within the current search budget.

Accordingly:

`human_equivalent` should not be interpreted as a formal proof of equivalence

`no_witness_found` should not be treated as ground-truth equivalence

witness-backed pairs provide stronger evidence in the negative direction because they include executable distinguishing tests

Potential uses

FEMP-X is intended to support research on:

semantic clone detection

behavioral similarity analysis

false-positive analysis for clone and similarity detectors

counterexample generation

executable test generation

LLM-assisted dataset curation

hybrid human–LLM curation workflows

Because witness-backed pairs include executable distinguishing tests, FEMP-X supports analyses that go beyond binary labels and enables researchers to inspect how a candidate pair was falsified.

AI usage disclosure

This dataset was curated in part with GPT-5.4 through Codex.

The model was used in an agentic workflow to generate candidate distinguishing tests, execute them immediately, and iteratively refine the search when no behavioral difference was reproduced. Only execution-validated witness tests were retained in the released artifact.

Repository

Project repository: https://github.com/YoshikiHigo/FEMP-X

Citation

If you use this dataset in its current pre-publication stage, please cite:

the Zenodo record for the specific released version

the original FEMPD paper

If an accompanying FEMP-X paper is published later, its citation information can be added in a future revision of this description.