MMLongEmbed

Qwen2-VL / Qwen2.5-VL <|image_pad|> <|video_pad|> Alibaba Qwen series LLaVA-NeXT <image> <image> Llama-based VLM Phi-3-Vision <|image_1|> - Microsoft Phi-3 E5-V Llama3 template Llama3 template Embeddings specialized GME / LamRA No special tokens No special tokens Plain text+image fusion InternVideo2 - - Video understanding ColPali - - Document retrieval Installation Create conda environment: conda create -n longvlm2vec python=3.11 conda activate longvlm2vec Install PyTorch (CUDA 12.4): pip install torch==2.6.0 torchvision --index-url https://download.pytorch.org/whl/cu124 Install vLLM and dependencies: pip install vllm>=0.6.0 pip install flashinfer-python -i https://flashinfer.ai/whl/cu124/torch2.6/ Install other requirements: pip install datasets transformers pyyaml pytrec_eval numpy tqdm Quick Start 1. Prepare Dataset Configuration Create a YAML file (e.g., configs/imagenet_eval.yaml): ImageNet-1K: dataset_parser: "image_cls" dataset_name: "imagenet-1k" image_root: "/path/to/imagenet" model_backbone: "qwen2_vl" image_resolution: "high" # high: 1344x1344, mid: 672x672, low: 128x128 num_sample_per_subset: 1000 2. Run Evaluation python eval.py \ --dataset_config configs/imagenet_eval.yaml \ --data_basedir /workspace/data \ --model_name Qwen/Qwen2-VL-7B-Instruct \ --output_dir ./results \ --gpu_ids 0,1,2,3,4,5,6,7 \ --port 8006 \ --concurrency_per_gpu 1 \ --max_model_len 32768 \ --dtype bfloat16 3. Monitor Progress The framework automatically: - Displays progress bars for query/candidate encoding - Saves embeddings to {output_dir}/{dataset_name}/qry.jsonl and cand.jsonl - Computes metrics and saves to metrics.json - Supports resume from interruptions (checks existing IDs) Experiment Results ImageNet-1K Zero-Shot Classification Setup: - Model: Qwen2-VL-7B-Instruct (embedding mode) - Dataset: ImageNet-1K (1,000 query images, 1,000 candidate classes) - GPU: 8x A100 (80GB) - Embed Dim: 2048 - Normalization: L2-normalized Results: Metric @1 @5 @10 -------------------------------------------------------- Hit 0.0510 0.1260 0.1650 NDCG (linear) 0.0510 0.0896 0.1024 NDCG (exponential) 0.0510 0.0896 0.1024 Precision 0.0510 0.0252 0.0165 Recall 0.0510 0.1260 0.1650 MRR 0.0510 0.0776 0.0830 Statistics: - Total queries processed: 1,000 - Total candidates: 1,000 - Embedding shape: (1000, 2048) - Mean norm: ~1.0000 (properly normalized) Advanced Configuration vLLM Deployment Arguments --runner pooling # Task type: pooling for embeddings --max_model_len 32768 # Context window --dtype bfloat16 # Precision: float16/bfloat16/fp8 --gpu_memory_utilization 0.90 # VRAM usage limit --max_num_seqs 16 # Concurrent sequences per iteration Data Processing - Multi-resolution Support: Dynamic image resizing (high/mid/low) before encoding - Batch Processing: Automatic batching across GPUs with configurable concurrency - Caching: Deduplicated candidate generation to avoid redundant encoding Metrics Explanation - Hit@K: Binary indicator if ground truth appears in top-K predictions - NDCG@K: Normalized Discounted Cumulative Gain (supports binary or graded relevance) - Precision@K: Relevant items in top-K divided by K - Recall@K: Relevant items in top-K divided by total relevant items - MRR@K: Mean Reciprocal Rank of first relevant item within top-K Troubleshooting flashinfer ABI Errors If encountering undefined symbol errors with flashinfer: unset VLLM_USE_FLASHINFER_SAMPLER export VLLM_ATTENTION_BACKEND=FLASH_ATTN CUDA Out of Memory - Reduce --max_num_seqs (default: 16) - Lower --max_model_len - Increase --gpu_memory_utilization (if fragmentation issues) Checkpoint Resume The framework automatically detects existing embeddings in qry.jsonl/cand.jsonl and skips processed IDs. To force re-evaluation, delete these files. Citation If you use this framework in your research, please cite: @software{longvlm2vec2024, title = {LongVLM2Vec: A Scalable Framework for Multimodal Embedding Evaluation}, year = {2024}, url = {https://github.com/your-repo/longvlm2vec} } License MIT License Built with vLLM for efficient LLM serving and HuggingFace datasets for easy data management.