Sevleete/lehome-submission

# LeHome Challenge Submission

This image bundles a self-contained evaluation runner: an internal model-server,
the simulation assets, and a websocket bridge that drives `scripts/eval`. After
loading the image, a single `docker run` invocation evaluates the four garment
categories end-to-end and writes per-category logs to a host-mounted directory.

---

## 1. Prerequisites (organizer host)

- Linux x86_64 host with a CUDA-capable NVIDIA GPU (tested on RTX 4090, ≥24GB).
- NVIDIA driver ≥ 535 (verified with `nvidia-smi`).
- Docker (≥ 24.x).
- `nvidia-container-toolkit` installed and Docker configured to use it:
```bash
sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
# Sanity check:
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
```
- About **40 GB** free disk for the loaded image.

---

## 2. Download the image tarball

```bash
wget https://huggingface.co/datasets/Sevleete/lehome-submission/resolve/main/lehome-submission-v1.tar.gz
```

(Roughly 25–35 GB. The HuggingFace download supports resume; if interrupted,
just rerun the same `wget` command.)

---

## 3. Load the image into Docker

```bash
docker load -i lehome-submission-v1.tar.gz
docker images | grep lehome-submission # expect: lehome-submission v1 ...
```

---

## 4. Run the evaluation

A single command runs the full evaluation suite (4 categories × 10 episodes):

```bash
mkdir -p eval_results
docker run --rm --gpus all \
-v $PWD/eval_results:/workspace/eval_results \
lehome-submission:v1
```

Inside the container the entrypoint script:

1. Starts the bundled model-server on `127.0.0.1:9000` (background process).
2. Waits up to 600 s for the server to accept connections.
3. For each garment category in
`[top_long, top_short, pant_long, pant_short]`, runs

```
xvfb-run -a python -m scripts.eval \
--policy_type openpi_ws \
--policy_path ws://127.0.0.1:9000 \
--garment_type <category> \
--num_episodes 10 \
--enable_cameras --device cpu --headless
```

4. Saves per-category logs to `/workspace/eval_results/eval_<category>.log`,
plus `server.log`.

Total wall-clock time on a single RTX 4090 host is roughly **2–4 hours**.

### 4a. Override the official evaluation Assets

The official challenge `Assets/` directory shipped inside this image only
contains the public garment set (`Seen_0..9` plus `Unseen_0,1`). Organizers
holding the held-out `Unseen_2..9` (or any other private assets) can override
the bundled `Assets/` by bind-mounting a host directory:

```bash
docker run --rm --gpus all \
-v /path/to/official/Assets:/opt/lehome-challenge/Assets \
-v $PWD/eval_results:/workspace/eval_results \
lehome-submission:v1
```

The eval driver reads the per-category garment list from
`Assets/objects/Challenge_Garment/Release/<Category>/<Category>.txt`, so
whichever `.txt` is in the mounted `Assets/` controls which garments get
rolled out. No code changes inside the image are required.

### 4b. Override the per-garment episode count

`NUM_EPISODES` defaults to **10**. To run a different count (e.g. 5 for a
quick smoke test, or 20 for full statistics):

```bash
docker run --rm --gpus all \
-e NUM_EPISODES=5 \
-v $PWD/eval_results:/workspace/eval_results \
lehome-submission:v1
```

---

## 5. Collect results

After the run finishes:

```bash
ls eval_results/
# eval_top_long.log
# eval_top_short.log
# eval_pant_long.log
# eval_pant_short.log
# server.log
```

Each `eval_*.log` ends with a per-garment success-rate summary. The companion
`all_data.txt` in this submission contains the corresponding numbers we
obtained on our local machine.

---

## 6. What's inside the image

| Path | Contents |
|---|---|
| `/opt/lehome-challenge/` | Official `lehome-challenge:latest` base image (uv venv at `.venv/`, isaac sim, scripts) |
| `/opt/lehome-challenge/Assets/` | Simulation assets (scenes, garments, robots) |
| `/opt/lehome-challenge/scripts/eval_policy/openpi_ws_policy.py` | Websocket bridge policy that forwards observations to the model-server |
| `/workspace/model_server/` | Internal model-server with bundled weights (loaded automatically at container start) |
| `/workspace/start_eval.sh` | Entrypoint launched by `CMD` — see step 4 above |
| `xvfb` | Installed via apt to satisfy `pynput`'s import-time X11 requirement under headless mode |

The two Python environments (lehome simulator vs. model-server) are isolated:
the simulator uses the uv venv under `/opt/lehome-challenge/.venv`, while the
model-server uses its own pixi env under
`/workspace/model_server/.pixi/envs/default`. Communication between them is
exclusively via the local websocket on port 9000.

---

## 7. Troubleshooting

| Symptom | Likely cause | Fix |
|---|---|---|
| `could not select device driver "" with capabilities: [[gpu]]` | `--gpus` flag works only when nvidia-container-toolkit is installed and Docker daemon restarted | See "Prerequisites" |
| Container exits with `server didn't start in 600s` | GPU OOM, missing driver, or weights not found at expected path | Inspect `eval_results/server.log` for the underlying traceback |
| `pynput: this platform is not supported ... display ":0"` | The `xvfb-run` wrapper was bypassed. The bundled `start_eval.sh` already wraps every eval call in `xvfb-run -a` | Make sure you do **not** override `--entrypoint` or pass a custom command |
| Slow eval / sim hangs | `--device cpu` is required by the official challenge protocol; sim-side compute is CPU-only by design | Expected; the GPU is used by the model-server only |

---

## 8. Notes on training pipeline

- All training and dataset preprocessing was performed offline on a separate
workstation; this image only contains the inference path needed for
evaluation.
- The model-server runs the trained checkpoint at `garment_v1/final/` (path
inside the image) — no additional weight downloads are required at runtime.

---

## 9. Source code

(Optional, for organizers who want to debug failed runs.)

If a source-code link is provided alongside this submission in the Google form,
note that **the source code is for reference only** — the evaluation procedure
above does not pull anything from it. Everything needed to reproduce the
reported numbers is already inside the docker image.