NeuroJustice

# Multimodal Supervision Framework for Offenders with Mental Health Challenges This repository presents the implementation of **CrossFusionNet**, a novel multimodal architecture, and its associated **Supervision Harmonization Protocol (SHP)** for improving offender management, particularly for individuals with mental health challenges. The project introduces new approaches to integrate heterogeneous data sources, handle supervision noise, and provide adaptive, real-time offender profiling. ## 🔍 Overview Traditional criminal justice supervision systems often struggle with the dynamic and complex needs of mentally ill offenders due to their reliance on simplistic, single-modality data and static evaluation methods. This project addresses those limitations with: - **CrossFusionNet**: A deep neural architecture that uses modality-specific encoders and a learnable cross-modal attention mechanism to retain modality fidelity while enabling semantic alignment.- **SHP (Supervision Harmonization Protocol)**: A robust training scheme that mitigates weak supervision, noisy labels, and partial modality availability through dynamic weighting, self-distillation, and uncertainty regularization. ## 🧠 Key Features - Multimodal integration of visual, audio, and textual data.- Cross-modal attention and fusion via joint interaction tensors.- Structured loss functions for modality-specific and global consistency.- Dynamic reliability scoring and knowledge distillation between modalities.- Latent consistency enforcement using frequency attention and alignment penalties. ## 🏗️ Architecture ### CrossFusionNet - **Input**: Modality-specific streams (e.g., spatio-temporal, textual, audio).- **Encoders**: Per-modality neural encoders generate embeddings.- **Fusion**: Joint interaction tensors and attention-based global fusion.- **Output**: Distributional embeddings for downstream tasks. > *See Figure 1 on page 7 of the paper for a detailed architecture diagram.* ### Supervision Harmonization Protocol (SHP) - **Dynamic Reliability Weighting**: Adjusts contribution of each modality using entropy-based trust scores and inter-modality agreement.- **Knowledge Distillation**: Teacher-student strategy where high-confidence modalities guide weaker ones.- **Latent Consistency Enforcement**: Penalties for uncertainty and alignment errors in shared latent space. > *See Figures 2–4 on pages 9–11 for schematics of loss structure and consistency enforcement.* ## 📊 Datasets Evaluated on four multimodal benchmarks: - **VQA** – Visual Question Answering- **MSCOCO** – Image captioning and object detection- **Flickr30k** – Text-image retrieval- **AVSpeech** – Audio-visual speech recognition ## 🏆 Performance Outperforms state-of-the-art models (e.g., CLIP, ViLT, LXMERT, BLIP) across all datasets and metrics, with significant gains in: - Accuracy- F1 Score- AUC- Recall See **Tables 1–4** (pages 14–15) for full benchmark and ablation results. ## ⚙️ Implementation Details - Framework: PyTorch- Optimizer: AdamW- Training: 50 epochs, cosine annealing, early stopping, mixed-precision- Hardware: 8 × NVIDIA A100 GPUs- Data Preprocessing: Resize to 224×224, BPE tokenization, spectrogram extraction ## 📁 File Structure ```bash├── data/ # Dataset loaders and preprocessors├── models/ # CrossFusionNet and SHP modules├── training/ # Training loops and loss functions├── utils/ # Metrics, visualizations, config files├── scripts/ # Scripts for experiments└── README.md # Project overview