AffordAny

# AffordAny AffordAny is a real-image benchmark for language-conditioned 3D affordance grounding. Given a monocular RGB observation and a natural-language interaction instruction, the task is to localize the corresponding functional region on an object-centric 3D Gaussian representation. > This directory is a private release candidate. Do not make it public until the > NoDerivatives and mixed-license review in `DATA_LICENSE.md` is complete. ## Statistics | Unit | Count | |---|---:| | Valid objects | 5,334 | | Valid part instances | 10,633 | | Generated instructions | 31,899 | | Formal protocol rows | 22,767 | | Object categories | 473 | | Unique part names | 678 | The formal protocol holds out object instances, object categories, and instruction paraphrases. Category assignment uses a fixed seed of 42. Seen categories are split at the object level, while validation and test unseen-category sets contain entirely held-out categories. ## Repository layout ```text AffordAny/ ├── metadata/ │ ├── objects.parquet │ ├── parts.parquet │ ├── instructions.parquet │ ├── source_licenses.parquet │ └── quality_report.json ├── splits/*.parquet ├── data/<bucket>/<object_id>.tar ├── checksums/SHA256SUMS ├── schemas/ ├── scripts/ └── examples/ ``` The release candidate contains 5,334 object archives totaling 215.86 GiB. Each archive is independently addressable and ranges from 2.71 MiB to 137.16 MiB. Each object archive extracts to a directory named after its `object_id`: ```text <object_id>/ ├── object.json ├── source/image.png ├── source/instance_mask.png ├── recon3d/splat.ply ├── recon3d/camera.json ├── render/cameras.json ├── render/view_000.png ... view_005.png ├── label3d/part_membership_scores.npz ├── label3d/unknown_mask.npz └── instruction/part_instructions.json ``` `part_membership_scores.npz` stores the ordered `part_names`, a `[P, N]` continuous score matrix, best-part indices, maximum scores, and visibility counts. `unknown_mask.npz` stores the supervision exclusion mask and its diagnostic masks. The Gaussian count `N` matches the vertex count in `splat.ply`. ## Loading metadata ```python from datasets import load_dataset protocol = load_dataset("lzlf/AffordAny", "protocol") objects = load_dataset("lzlf/AffordAny", "objects", split="train") ``` Download the file payload from Hugging Face: ```bash hf download lzlf/AffordAny --repo-type dataset --local-dir ./AffordAny ``` Or from ModelScope: ```bash ms-hub download lzlfwow/AffordAny --repo-type dataset --local-dir ./AffordAny ``` Use `examples/load_object.py` to extract and inspect one object archive. Detailed creation and mirroring commands are in `PUBLISHING.md`. ## Dataset construction The pipeline selects interaction-relevant LVIS instances, reconstructs each retained object as 3D Gaussians, renders six canonical views, discovers functional parts, segments those parts in 2D, and lifts the masks into 3D through multi-view voting. An LLM generates three interaction-oriented English instructions for every validated part. Low-confidence, ambiguous, or invisible Gaussians are marked unknown and are excluded from supervision. ## Intended use and limitations AffordAny supports research on language-guided 3D affordance segmentation, open-vocabulary grounding, unseen-object generalization, unseen-category generalization, and instruction robustness. Labels are generated automatically and can contain reconstruction, segmentation, semantic, and language errors. Source data inherits LVIS/COCO geographic, cultural, category, and photographic biases. The benchmark predicts interaction regions; it does not establish that a physical action is safe or feasible for a robot. ## Licensing and provenance The source images use several per-image COCO/Flickr licenses. Consult `DATA_LICENSE.md` and `metadata/source_licenses.parquet`; do not infer a blanket license from the repository metadata. A removal or correction process and contact address must be added before public release. ## Citation The final paper citation must replace this release-candidate entry before publication: ```bibtex @article{affordany2026, title = {AffordAny: VLM-Guided Open-World 3D Affordance Grounding from a Monocular RGB Image}, author = {AffordAny Authors}, year = {2026} } ```