ReMoT_data

## 🚀 Evaluation Code Usage ### 1. Single Dataset Evaluation Use `index_eval.py` for multi-GPU parallel evaluation of a single dataset: ```bash CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python eval_codes/index_eval.py \ --model /path/to/your/model \ --input /data1/guozeyu/DATA/CVPR2026/data_upload/agibot/test.jsonl \ --output /path/to/output/result.jsonl \ --check /path/to/checkpoint.jsonl \ --max-new-tokens 10000 \ --num-gpus 8 ``` ### 2. Batch Evaluation Use `infer_remot_bench.py` to evaluate all datasets at once: ```bash # First modify configuration parameters in the script (MODEL_PATH, CKPT_TAG, CHECK_FILE, etc.) python eval_codes/infer_remot_bench.py ``` ### 3. Result Aggregation Use `summary.py` to generate evaluation reports: ```bash # Modify configuration parameters in the script (BASE_FOLDER, MODEL_NAME, etc.) before running python eval_codes/summary.py ``` ## ⚙️ Parameter Description ### index_eval.py Parameters | Parameter | Type | Required | Description | |------|------|------|------| | `--model` | string | ✅ | HuggingFace model path or name | | `--input` | string | ✅ | Input test set JSONL file path | | `--output` | string | ✅ | Output result file path | | `--check` | string | ✅ | Checkpoint file path (for resuming from breakpoint; can use any placeholder path for first run) | | `--max-new-tokens` | int | ❌ | Maximum number of generated tokens, default 128 | | `--num-gpus` | int | ❌ | Number of GPUs, default 8 | ### infer_remot_bench.py Configuration Modify the following configurations at the beginning of the script: | Configuration | Description | |--------|------| | `MODEL_PATH` | Model path | | `CKPT_TAG` | Output filename tag | | `CHECK_FILE` | Checkpoint file path | | `BASE_OUTPUT_DIR` | Output directory | | `GPUS` | GPU list, e.g., `"0,1,2,3"` | | `MAX_NEW_TOKENS` | Maximum number of generated tokens | | `NUM_GPUS` | Number of GPUs | ### summary.py Configuration | Configuration | Description | |--------|------| | `AGIBOT_INDEX_PATH` | AGIBOT dataset index path | | `GENERALDATA_INDEX_PATH` | GeneralData dataset index path | | `BASE_FOLDER` | Evaluation results directory | | `MODEL_NAME` | Model name (for matching result files) | | `TASK_NAMES` | Task list `["agibot", "generaldata", "scannet"]` | ## 📈 Evaluation Metrics ### Metric Descriptions | Metric | Description | |------|------| | **Exact Match (Ov)** | Exact match accuracy, prediction completely matches ground truth | | **Per-Char Match (Par)** | Character-level match accuracy, calculates correct ratio at each position | | **Avg Response Words** | Average number of response words | ### Dataset-Specific Metrics #### AGIBOT - **Gripper-Move**: Gripper movement tasks (move_lr + lift_lower) - **Gripper-State**: Gripper state tasks (grab_release) - **Composite**: Composite tasks (move_complex) #### ScanNet - **Camera Ov**: Camera viewpoint exact match (average of partial matches across viewpoints) - **Camera Par**: Camera viewpoint character-level match #### GeneralData - **Rel-Pos**: Relative position tasks (mm class) - **Grounding**: Object detection tasks (om class) - **Counting**: Counting tasks (vl class) ## 🔄 Resume from Checkpoint The evaluation script supports resuming from checkpoint: 1. When evaluation is interrupted, completed results are saved in temporary files (`{output}_gpu{id}.jsonl`) 2. When rerunning, specify the previous output file using the `--check` parameter 3. The script automatically skips processed data and continues from the checkpoint ## 📝 Output Format ### Inference Results (JSONL) ```json { "id": "idx_resize_4_test_0", "gpu_id": 0, "response_word_count": 45, "model_response": "Based on image analysis...<answer>CAC</answer>", "pred_answer": "CAC", "solution": "<answer>CAC</answer>", "gt_answer": "CAC", "correct_count": 3 } ``` ### Summary Report (Excel) The generated Excel file contains two worksheets: | Worksheet | Content | |--------|------| | `Summary_Indicators` | Core metric summary for each task (Sheet1) | | `Detailed_Metrics` | Detailed classification accuracy | ## 📖 Usage Workflow - Prepare model → Download or train a vision-language model - Run batch evaluation → python infer_remot_bench.py - Generate summary report → python summary.py - View results → Open the generated Excel file