BanHealthAIRespRelv3: A Benchmark Dataset for Evaluating Bengali AI-Generated Health Responses Based on User-Specific Prompts

BanHealthAIRespRelv3 serves as a benchmark dataset for evaluating the relevance and safety of AI-generated health responses in Bengali, a low-resource language. Its primary goal is to address risks of medical hallucinations, irrelevant advice, and misinformation in online health information seeking (OHIS), particularly in resource-constrained settings like Bangladesh.
To ensure high quality, the dataset underwent rigorous preprocessing (text cleaning, duplicate removal, and filtering incomplete responses) and annotation by three native Bengali speakers, with reliability confirmed via Fleiss’ Kappa (substantial to almost perfect agreement) and retention of only high-confidence instances (>0.8).
BanHealthAIRespRelv3 is a ternary-class dataset comprising 15,388 instances of user-simulated health prompts and corresponding AI responses (from ChatGPT and Google Gemini), categorized as:

Highly Relevant: 5,004 instances
Partially Relevant: 5,300 instances
Not Relevant: 5,084 instances

The dataset has broad applications in multiple NLP and AI areas.
including:
Relevance and hallucination detection in health AI
Development of safer, culturally sensitive health chatbots
Trust calibration and ethical AI in digital health
Misinformation mitigation in low-resource languages
Educational and research advancements in Bengali medical NLP

BanHealthAIRespRelv3 is openly available for academic and research purposes, promoting collaboration and innovation in Bengali NLP. By establishing a robust benchmark, it aims to foster trustworthy, context-aware AI systems for health-related applications in under-resourced languages.