SceneDial

# SceneDial: Scene-Aware Dialogue Generation with Hybrid Memory and Speaker Modeling **SceneDial** is a dialogue generation framework that incorporates visual scene context and conversational history to generate coherent, role-specific, and context-sensitive responses. It introduces a hybrid memory fusion module and speaker-aware decoding strategy for realistic multi-turn dialogue in situated environments. --- ## 🧠 Overview This repository implements the model proposed in: > **"Contextual Scene-Aware Dialogue Generation via Hybrid Memory Fusion and Role-Specific Speaker Modeling"** > Yue Li, Guangxi University for Nationalities, 2025 Scene-aware conversations (e.g., in vision-grounded agents, robotics, AR/VR) require an understanding of: - Multimodal visual features (scene objects, relationships) - Speaker-specific roles (e.g., guide vs tourist) - Coherent response generation across turns --- ## 🔧 Key Modules - **Hybrid Memory Fusion (HMF):** Fuses external scene knowledge (object detection + relations) and internal dialogue history. - **Role-Specific Speaker Modeling (RSSM):** Encodes user role (e.g., director vs navigator) to generate speaker-aware responses. - **Dual Encoder with Cross-Attention:** Leverages both scene features and memory-enhanced textual context for decoding. --- ## 📁 Project Structure ```bash.├── models/ # HMF, RSSM, dual encoder-decoder├── data/ # SceneDial / DSTC10 / SIMMC2 datasets├── scripts/ # train, eval, inference scripts├── configs/ # YAML configs for each dataset├── utils/ # metrics, preprocessing, logging├── checkpoints/ # Trained model weights└── README.md