Harry-1234/IntentRouterTrain
收藏资源简介：
--- license: bsd-3-clause task_categories: - text-generation tags: - agent --- # MAOmni: A Self-Correcting Multi-Agent Omni-Modal Reasoning Framework For Affective and Intentional Analysis <div style="display: flex; flex-wrap: wrap; align-items: center; gap: 5px;"> <a href="https://huggingface.co/Harry-1234/MAOmni" target="_blank"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue"></a> <a href="https://huggingface.co/spaces/Harry-1234/MAOmni" target="_blank"><img src="https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-sm-dark.svg"></a> <a href="https://github.com/eeee-sys/MAOmni" target="_blank"><img src="https://img.shields.io/badge/Project-Page-brightgreen"></a> <a href="https://github.com/eeee-sys/MAOmni/blob/main/LICENSE" target="_blank"><img src="https://img.shields.io/badge/License-BSD--3--Clause-purple"></a> </div> **MAOmni** is a novel self-correcting multi-agent omni-modal framework endowed with deliberative reasoning capabilities. MAOmni decomposes the reasoning process through a dynamic cognitive workflow orchestrated by five specialized agents, a generative Retriever for global context distillation, an adaptive AKD Router Agent for dynamic reasoning routing, a GRPO Grounder for precise continuous-time spatio-temporal localization, Reasoning Agent for explicit structured logical inference, and a TTA Reviser for test-time adaptive self-correction via ephemeral LoRA tuning. ## 🔖 Model Details - **Model type:** Omni-modal Large Language Model - **License:** BSD-3-Clause ## 👀 MAOmni Overview Understanding human intentions and social interaction contexts from complex, dynamic omni-modal streams is a fundamental yet challenging problem in artificial intelligence. Existing multi-modal large language models (MLLMs) typically rely on monolithic, black-box reasoning paradigms, making them highly susceptible to cognitive overload, shortcut learning, and hallucinated predictions when processing long-duration inputs. To address these limitations, we proposes MAOmni, a novel self-correcting multi-agent omni-modal framework endowed with deliberative reasoning capabilities. MAOmni decomposes the reasoning process through a dynamic cognitive workflow orchestrated by five specialized agents, a generative ELT Retriever Agent for global context distillation, an adaptive AKD Router Agent for dynamic reasoning routing, a GRPO Grounder for precise continuous-time spatio-temporal localization, OMLT Reasoner Agent for explicit structured logical inference, and a TTA Reviser for test-time adaptive self-correction via ephemeral LoRA tuning. Extensive experiments on three challenging benchmarks demonstrate the superiority of our framework. Notably, despite its compact 7B parameter scale, MAOmni achieves state-of-the-art results, consistently outperforming leading open-source models and surpassing several proprietary systems, including GPT-4o and Gemini-2.5-Pro. <p align="center"> <img src="https://github.com/eeee-sys/MAOmni/blob/main/assets/method.png" width="100%" height="100%"> </p> #### 🌟 Contributions in MAOmni 1. We propose MAOmni, a unified omni-modal reasoning framework that pioneers the application of multi-agent collaboration in the field of affective analysis. Our framework introduces dynamic strategy selection via a planning module, enabling the model to adaptively determine whether to perform temporal grounding or direct reasoning based on input complexity. 2. We introduce GRPO Grounder and TTA Reviser. We train the video locator implemented by the autoregressive method using the GRPO algorithm and fine-tune the reasoning module during testing using the test-time adaption and REINFORCE with Baseline algorithms. This method enables our framework to have sample-level answering capabilities. 3. MAOmni achieves state-of-the-art results across three Benchmarks: IntentBench, Daily-Omni, WorldSense. Notably, our approach surpasses a host of commercial closed-source and open-source models, including GPT-4o, Gemini-2.5-Pro (think). Extensive ablations further confirm its effectiveness. ## 💻 Code Repository The code for MAOmni, including training and evaluation scripts, can be found on GitHub: [https://github.com/eeee-sys/MAOmni](https://github.com/eeee-sys/MAOmni) ## 📈 Experimental Results #### 📍 Results <p align="center"> <img src="assets/dailyomni.png" width="100%" height="100%"> </p> <p align="center"> <img src="assets/worldsense.png" width="100%" height="100%"> </p> <p align="center"> <img src="assets/intentbench.png" width="100%" height="100%"> </p> ## 🚀 Quick Start ### Install the environment 1. Clone the repository from GitHub. ```shell git clone git@github.com:eeee-sys/MAOmni.git cd MAOmni ``` 2. Initialize conda environment. ```shell conda create -n grpo_grounder python=3.11 -y conda activate grpo_grounder pip install -r src/requirements_grpo_grounder.txt ``` ```shell conda create -n maomni_main python=3.10 -y conda activate maomni_main pip install -r src/requirements_main.txt ``` ### Quick Inference Demo The script below showcases how to perform inference with MAOmni's different roles. Please refer to our [GitHub Repository](https://github.com/eeee-sys/MAOmni) for more details about this framework. ```python import torch from transformers import ( Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniThinkerForConditionalGeneration, Qwen2_5OmniProcessor, ) from peft import LoraConfig, get_peft_model, PeftModel from qwen_omni_utils import process_mm_info # ============================================================ # Main Process # ============================================================ def main(): # ---- Initialize Models ---- print(f"\n[INIT] Loading Base Model ({args.base_model_path}) on {args.main_gpu}") base_model = Qwen2_5OmniForConditionalGeneration.from_pretrained( args.base_model_path, torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2" ).to(args.main_gpu) base_processor = Qwen2_5OmniProcessor.from_pretrained(args.base_model_path) # Load Planner LoRA onto thinker submodule print(f"[INIT] Loading Planner LoRA onto base_model.thinker") base_model.thinker.load_adapter(args.planner_lora_path, adapter_name="planner") base_model.eval() print(f"[INIT] Loading HumanOmniV2 ({args.humanomni_path}) on {args.humanomni_gpu}") humanomni_model = Qwen2_5OmniThinkerForConditionalGeneration.from_pretrained( args.humanomni_path, torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2" ).to(args.humanomni_gpu) humanomni_processor = Qwen2_5OmniProcessor.from_pretrained(args.humanomni_path) lora_config = LoraConfig( r=64, lora_alpha=128, target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], lora_dropout=0.05, bias="none", task_type="CAUSAL_LM" ) humanomni_model = get_peft_model(humanomni_model, lora_config, adapter_name="initial_dummy") humanomni_model.enable_input_require_grads() humanomni_model.gradient_checkpointing_enable() print(f"[INIT] Starting Grounder process on {args.grounder_gpu}...") grounder_script = os.path.join(SCRIPT_DIR, "grounder_worker_grpo.py") grounder_env = os.environ.copy() grounder_env["CUDA_VISIBLE_DEVICES"] = args.grounder_gpu.replace("cuda:", "") grounder_proc = subprocess.Popen([ args.grounder_python, grounder_script, "--model_path", args.grounder_path, "--grpo_adapter_path", args.grpo_adapter_path, "--device", "cuda:0" ], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=None, text=True, bufsize=1, env=grounder_env) ready_line = grounder_proc.stdout.readline().strip() if not ready_line or json.loads(ready_line).get("status") != "ready": print("[ERROR] Grounder worker failed to start.") sys.exit(1) print("[INIT] All models ready!") os.makedirs(args.lora_save_dir, exist_ok=True) tmp_dir = tempfile.mkdtemp(prefix="idea3_reviser7b_") # ---- 3. Loop through dataset ---- for sample in samples_to_process: try: # ====== PLANNER STAGE ====== # a) Collector Phase (LoRA disabled) base_model.thinker.set_adapter("planner") # Ensure adapter is active before disabling base_model.thinker.disable_adapters() collector_text = stage1_collector(base_model.thinker, base_processor, video_path, query, args.main_gpu) print(f"[Collector output] {collector_text}") # b) Planner Phase (LoRA enabled) base_model.thinker.enable_adapters() (use_grounder, gnd_query), planner_raw = stage2_planner(base_model.thinker, base_processor, video_path, query, collector_text, args.main_gpu) print(f"[Planner output] {planner_raw}") print(f"[Planner] Use Grounder: {use_grounder} | query: {gnd_query}") # ====== GROUNDER STAGE ====== generation_video = video_path grounded_span = None if use_grounder: pred_spans, success = stage3_grounder(grounder_proc, video_path, gnd_query or query, duration) print(f"[Grounder output] {pred_spans}") grounded_span = pred_spans[0] trim_path = os.path.join(tmp_dir, f"trim_{dataset_id}.mp4") trim_video_ffmpeg(video_path, grounded_span[0], grounded_span[1], trim_path) generation_video = trim_path print(f"[Grounder] Grounded to {grounded_span[0]:.1f}s - {grounded_span[1]:.1f}s") # ====== HUMANOMNI & REINFORCE STAGE ====== humanomni_query = build_humanomni_query(sample) adapter_name = f"sample_{dataset_id}".replace(".", "_") humanomni_model.add_adapter(adapter_name, lora_config) humanomni_model.set_adapter(adapter_name) # Ensure adapter parameters require gradients for n, p in humanomni_model.named_parameters(): if adapter_name in n: p.requires_grad = True humanomni_model.train() trainable_params = [ p for n, p in humanomni_model.named_parameters() if p.requires_grad and adapter_name in n ] optimizer = torch.optim.AdamW(trainable_params, lr=args.lr) b = args.b0 best_score = -1 best_answer = "" best_raw_resp = "" all_history = [] early_stop = False for t in range(1, args.t_max + 1): gc.collect(); torch.cuda.empty_cache() humanomni_model.eval() inputs = get_humanomni_inputs(humanomni_processor, generation_video, humanomni_query, sample, args.humanomni_gpu) with torch.no_grad(): output_ids = humanomni_model.generate(**inputs, max_new_tokens=1024, do_sample=True, temperature=0.85) generated_sequence = output_ids[0][inputs.input_ids.size(1):] y_t_text = humanomni_processor.decode(generated_sequence, skip_special_tokens=True) print(f" [Iter {t}/{args.t_max}] Answer = {y_t_text}") base_model.thinker.disable_adapters() score_t, reviser_raw = revise_answer(base_model.thinker, base_processor, video_path, query, y_t_text, args.main_gpu) all_history.append({"iter": t, "answer": y_t_text, "score": score_t, "reviser_raw": reviser_raw}) # --- RL Update (REINFORCE) --- humanomni_model.train() optimizer.zero_grad() advantage = float(score_t - b) advantage_tensor = torch.tensor([advantage], device=args.humanomni_gpu, dtype=torch.bfloat16) outputs = humanomni_model(**forward_kwargs) nll_loss = outputs.loss final_loss = nll_loss * advantage_tensor.detach() final_loss.backward() optimizer.step() b = args.alpha * b + (1.0 - args.alpha) * score_t
--- license: bsd-3-clause 任务类别： - 文本生成 标签： - 智能体（Agent） --- # MAOmni：面向情感与意图分析的自校正多智能体全模态推理框架 <div style="display: flex; flex-wrap: wrap; align-items: center; gap: 5px;"> <a href="https://huggingface.co/Harry-1234/MAOmni" target="_blank"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue"></a> <a href="https://huggingface.co/spaces/Harry-1234/MAOmni" target="_blank"><img src="https://huggingface.co/datasets/huggingface/badges/resolve/main/open-in-hf-spaces-sm-dark.svg"></a> <a href="https://github.com/eeee-sys/MAOmni" target="_blank"><img src="https://img.shields.io/badge/Project-Page-brightgreen"></a> <a href="https://github.com/eeee-sys/MAOmni/blob/main/LICENSE" target="_blank"><img src="https://img.shields.io/badge/License-BSD--3--Clause-purple"></a> </div> **MAOmni** 是一种具备审慎推理能力的新型自校正多智能体全模态框架。MAOmni通过由五个专业智能体协同编排的动态认知工作流拆解推理过程，具体包括：用于全局上下文提炼的生成式检索器（Retriever）、用于动态推理路由的自适应AKD路由智能体、用于精准连续时空定位的GRPO定位器（Grounder）、用于显式结构化逻辑推理的推理智能体，以及通过短时LoRA（Low-Rank Adaptation）微调实现测试时自适应自校正的TTA修正器（Reviser）。 ## 🔖 模型细节 - **模型类型**：全模态大语言模型 - **许可证**：BSD-3-Clause ## 👀 MAOmni 总览 从复杂动态的全模态流中理解人类意图与社交交互语境，是人工智能领域一项基础却极具挑战的问题。现有多模态大语言模型（Multi-modal Large Language Model, MLLMs）通常依赖单一整体式黑箱推理范式，在处理长时序输入时极易出现认知过载、捷径学习与幻觉预测问题。为解决上述局限，我们提出MAOmni——一种具备审慎推理能力的新型自校正多智能体全模态框架。MAOmni通过由五个专业智能体协同编排的动态认知工作流拆解推理过程：用于全局上下文提炼的生成式ELT检索智能体、用于动态推理路由的自适应AKD路由智能体、用于精准连续时空定位的GRPO定位器、用于显式结构化逻辑推理的OMLT推理智能体，以及通过短时LoRA微调实现测试时自适应自校正的TTA修正器。我们在三项极具挑战性的基准测试上开展了大量实验，证实了本框架的优越性。值得注意的是，尽管参数量仅为紧凑的7B规模，MAOmni却取得了当前最优性能，持续超越主流开源模型，并在多项专有系统（包括GPT-4o与Gemini-2.5-Pro）之上实现性能突破。 <p align="center"> <img src="https://github.com/eeee-sys/MAOmni/blob/main/assets/method.png" width="100%" height="100%"> </p> #### 🌟 MAOmni 的创新贡献 1. 我们提出MAOmni这一统一全模态推理框架，首次将多智能体协作应用于情感分析领域。本框架通过规划模块引入动态策略选择机制，使模型能够根据输入复杂度自适应决策是否执行时序定位或直接推理。 2. 我们提出GRPO定位器与TTA修正器。我们采用GRPO算法训练自回归方法实现的视频定位器，并通过测试自适应（Test-time Adaption, TTA）与带基线的REINFORCE算法在测试阶段微调推理模块。该方法使本框架具备样本级应答能力。 3. MAOmni在三项基准测试（IntentBench、Daily-Omni、WorldSense）上均取得当前最优性能。值得注意的是，我们的方法超越了包括GPT-4o、Gemini-2.5-Pro（思考版）在内的多款商业闭源与开源模型。大量消融实验进一步验证了其有效性。 ## 💻 代码仓库 MAOmni的代码（包括训练与评估脚本）可在GitHub获取：[https://github.com/eeee-sys/MAOmni](https://github.com/eeee-sys/MAOmni) ## 📈 实验结果 #### 📍 实验结果 <p align="center"> <img src="assets/dailyomni.png" width="100%" height="100%"> </p> <p align="center"> <img src="assets/worldsense.png" width="100%" height="100%"> </p> <p align="center"> <img src="assets/intentbench.png" width="100%" height="100%"> </p> ## 🚀 快速开始 ### 安装环境 1. 从GitHub克隆仓库。 shell git clone git@github.com:eeee-sys/MAOmni.git cd MAOmni 2. 初始化Conda环境。 shell conda create -n grpo_grounder python=3.11 -y conda activate grpo_grounder pip install -r src/requirements_grpo_grounder.txt shell conda create -n maomni_main python=3.10 -y conda activate maomni_main pip install -r src/requirements_main.txt ### 快速推理演示 下述脚本展示了如何使用MAOmni的不同角色进行推理。有关本框架的更多细节，请参阅我们的[GitHub仓库](https://github.com/eeee-sys/MAOmni)。 python import torch from transformers import ( Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniThinkerForConditionalGeneration, Qwen2_5OmniProcessor, ) from peft import LoraConfig, get_peft_model, PeftModel from qwen_omni_utils import process_mm_info # ============================================================ # 主流程 # ============================================================ def main(): # ---- 初始化模型 ---- print(f" [INIT] 正在{args.main_gpu}上加载基础模型 ({args.base_model_path})") base_model = Qwen2_5OmniForConditionalGeneration.from_pretrained( args.base_model_path, torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2" ).to(args.main_gpu) base_processor = Qwen2_5OmniProcessor.from_pretrained(args.base_model_path) # 将规划器LoRA加载至thinker子模块 print(f"[INIT] 正在base_model.thinker上加载规划器LoRA") base_model.thinker.load_adapter(args.planner_lora_path, adapter_name="planner") base_model.eval() print(f"[INIT] 正在{args.humanomni_gpu}上加载HumanOmniV2 ({args.humanomni_path})") humanomni_model = Qwen2_5OmniThinkerForConditionalGeneration.from_pretrained( args.humanomni_path, torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2" ).to(args.humanomni_gpu) humanomni_processor = Qwen2_5OmniProcessor.from_pretrained(args.humanomni_path) lora_config = LoraConfig( r=64, lora_alpha=128, target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], lora_dropout=0.05, bias="none", task_type="CAUSAL_LM" ) humanomni_model = get_peft_model(humanomni_model, lora_config, adapter_name="initial_dummy") humanomni_model.enable_input_require_grads() humanomni_model.gradient_checkpointing_enable() print(f"[INIT] 正在{args.grounder_gpu}上启动定位器进程...") grounder_script = os.path.join(SCRIPT_DIR, "grounder_worker_grpo.py") grounder_env = os.environ.copy() grounder_env["CUDA_VISIBLE_DEVICES"] = args.grounder_gpu.replace("cuda:", "") grounder_proc = subprocess.Popen([ args.grounder_python, grounder_script, "--model_path", args.grounder_path, "--grpo_adapter_path", args.grpo_adapter_path, "--device", "cuda:0" ], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=None, text=True, bufsize=1, env=grounder_env) ready_line = grounder_proc.stdout.readline().strip() if not ready_line or json.loads(ready_line).get("status") != "ready": print("[ERROR] 定位器工作进程启动失败。") sys.exit(1) print("[INIT] 所有模型就绪！") os.makedirs(args.lora_save_dir, exist_ok=True) tmp_dir = tempfile.mkdtemp(prefix="idea3_reviser7b_") # ---- 3. 遍历数据集 ---- for sample in samples_to_process: try: # ====== 规划器阶段 ====== # a) 收集器阶段（禁用LoRA） base_model.thinker.set_adapter("planner") # 确保适配器处于活动状态后再禁用 base_model.thinker.disable_adapters() collector_text = stage1_collector(base_model.thinker, base_processor, video_path, query, args.main_gpu) print(f"[收集器输出] {collector_text}") # b) 规划器阶段（启用LoRA） base_model.thinker.enable_adapters() (use_grounder, gnd_query), planner_raw = stage2_planner(base_model.thinker, base_processor, video_path, query, collector_text, args.main_gpu) print(f"[规划器输出] {planner_raw}") print(f"[规划器] 是否使用定位器：{use_grounder} | 查询词：{gnd_query}") # ====== 定位器阶段 ====== generation_video = video_path grounded_span = None if use_grounder: pred_spans, success = stage3_grounder(grounder_proc, video_path, gnd_query or query, duration) print(f"[定位器输出] {pred_spans}") grounded_span = pred_spans[0] trim_path = os.path.join(tmp_dir, f"trim_{dataset_id}.mp4") trim_video_ffmpeg(video_path, grounded_span[0], grounded_span[1], trim_path) generation_video = trim_path print(f"[定位器] 已定位至{grounded_span[0]:.1f}s - {grounded_span[1]:.1f}s") # ====== HumanOmni与REINFORCE阶段 ====== humanomni_query = build_humanomni_query(sample) adapter_name = f"sample_{dataset_id}".replace(".", "_") humanomni_model.add_adapter(adapter_name, lora_config) humanomni_model.set_adapter(adapter_name) # 确保适配器参数需要梯度 for n, p in humanomni_model.named_parameters(): if adapter_name in n: p.requires_grad = True humanomni_model.train() trainable_params = [ p for n, p in humanomni_model.named_parameters() if p.requires_grad and adapter_name in n ] optimizer = torch.optim.AdamW(trainable_params, lr=args.lr) b = args.b0 best_score = -1 best_answer = "" best_raw_resp = "" all_history = [] early_stop = False for t in range(1, args.t_max + 1): gc.collect(); torch.cuda.empty_cache() humanomni_model.eval() inputs = get_humanomni_inputs(humanomni_processor, generation_video, humanomni_query, sample, args.humanomni_gpu) with torch.no_grad(): output_ids = humanomni_model.generate(**inputs, max_new_tokens=1024, do_sample=True, temperature=0.85) generated_sequence = output_ids[0][inputs.input_ids.size(1):] y_t_text = humanomni_processor.decode(generated_sequence, skip_special_tokens=True) print(f" [迭代 {t}/{args.t_max}] 应答 = {y_t_text}") base_model.thinker.disable_adapters() score_t, reviser_raw = revise_answer(base_model.thinker, base_processor, video_path, query, y_t_text, args.main_gpu) all_history.append({"iter": t, "answer": y_t_text, "score": score_t, "reviser_raw": reviser_raw}) # --- 强化学习更新（REINFORCE）--- humanomni_model.train() optimizer.zero_grad() advantage = float(score_t - b) advantage_tensor = torch.tensor([advantage], device=args.humanomni_gpu, dtype=torch.bfloat16) outputs = humanomni_model(**forward_kwargs) nll_loss = outputs.loss final_loss = nll_loss * advantage_tensor.detach() final_loss.backward() optimizer.step() b = args.alpha * b + (1.0 - args.alpha) * score_t
