BhashaBench-Legal

<div align="center"> <img src="https://huggingface.co/bharatgenai/Param-1-2.9B-Instruct/resolve/main/BharatGen%20Logo%20(1).png" width="60%" alt="BharatGen" /> </div> <hr> <div align="center"> <a href="https://arxiv.org/abs/2510.25409" style="margin: 4px;"> <img alt="Paper" src="https://img.shields.io/badge/arXiv-2510.25409-b31b1b?style=flat" /> </a> <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank" style="margin: 4px;"> <img alt="License" src="https://img.shields.io/badge/License-CC--BY--4.0-blue.svg" /> </a> <a href="#" target="_blank" style="margin: 4px;"> <img alt="Blog" src="https://img.shields.io/badge/Blog-Read%20More-brightgreen?style=flat" /> </a> </div> # BhashaBench-Legal (BBL): Benchmarking AI on Indian Legal Knowledge <div style="display: flex; gap: 5px;"> <a href="https://github.com/BharatGen-IITB-TIH/BhashaBench"><img src="https://img.shields.io/badge/GITHUB-black?style=flat&logo=github&logoColor=white" alt="GitHub"></a> <a href="#"><img src="https://img.shields.io/badge/Paper-Coming%20Soon-lightgrey?style=flat" alt="ArXiv"></a> <a href="https://creativecommons.org/licenses/by/4.0/"><img src="https://img.shields.io/badge/License-CC%20BY%204.0-lightgrey.svg" alt="CC BY 4.0"></a> </div> ## Overview BhashaBench-Legal (BBL) is the comprehensive benchmark designed to rigorously evaluate AI models on Indian legal knowledge. Tailored for India's complex legal framework, constitutional structure, and diverse jurisprudential traditions, BBL draws from official judicial service exams, bar examinations, and legal education assessments to test models' ability to provide accurate, contextually relevant, and legally sound advice within the Indian legal system. ## Key Features - **Languages**: English and Hindi (with plans for more Indic languages) - **Exams**: 50+ unique legal government and institutional exams across India - **Domains**: 20+ legal and allied disciplines, spanning over 200 specialized topics - **Questions**: 24,365 rigorously validated, exam-based questions - **Difficulty Levels**: Easy (8,200), Medium (12,150), Hard (4,015) - **Question Types**: Multiple Choice, Assertion-Reasoning, Match the Column, Rearrange the Sequence, Fill in the Blanks - **Focus**: Practical, context-rich, jurisdiction-specific legal knowledge essential for Indian legal practice ## Dataset Statistics | Metric | Count | | ------------------------ | ------------------------- | | Total Questions | 24,365 | | English Questions | 17,047 | | Hindi Questions | 7,318 | | Subject Domains | 20+ | | Government Exams Covered | 50+ | ## Dataset Structure ### Test Set The test set consists of the BhashaBench-Legal (BBL) benchmark, which contains approximately 24,365 multiple-choice questions across 2 Indic languages (English and Hindi). We will add support for more Indic languages in upcoming versions. ### Subjects spanning BBL | Subject Domain | Count | |----------------------------------|-------| | Civil Litigation & Procedure | 7126 | | Constitutional & Administrative Law| 3609 | | Criminal Law & Justice | 2769 | | Corporate & Commercial Law | 2700 | | General Academic Subjects | 1756 | | Legal Theory & Jurisprudence | 1421 | | Family & Personal Law | 991 | | International & Comparative Law | 962 | | Legal Skills & Communication | 816 | | Real Estate & Property Law | 629 | | Environmental & Energy Law | 430 | | Interdisciplinary Studies | 363 | | Tax & Revenue Law | 231 | | Employment & Labour Law | 175 | | Technology & Cyber Law | 123 | | Intellectual Property Law | 91 | | Consumer & Competition Law | 75 | | Media & Entertainment Law | 54 | | Healthcare & Medical Law | 25 | | Human Rights & Social Justice | 19 | ## Usage Since this is a gated dataset, after your request for accessing the dataset is accepted, you can set your HuggingFace token: ```bash export HF_TOKEN=YOUR_TOKEN_HERE ``` To load the BBL dataset for a Language: ```python from datasets import load_dataset language = 'Hindi' # Use 'test' split for evaluation split = 'test' language_data = load_dataset("bharatgenai/BhashaBench-Legal", data_dir=language, split=split, token=True) print(language_data[0]) ``` ## Evaluation Results Summary - **25+ models evaluated**, including GPT-4o, Claude-3.5, and various specialized legal LLMs. - **Top accuracy:** - **English:** 75%+ by best models - **Hindi:** 65–70%, showing need for enhanced Indic language legal reasoning - **Strong domains:** - Constitutional Law, Legal Theory, Corporate Law (~80% accuracy) - **Weak domains:** - Technology & Cyber Law, Environmental Law, Healthcare Law (<55%) - **Challenges:** - Complex procedural questions and jurisdiction-specific cases remain challenging across models For detailed results and analysis, please refer to our [blog](#). ## Citation Please cite our benchmark if used in your work: ```bibtex @misc{devane2025bhashabenchv1comprehensivebenchmark, title={BhashaBench V1: A Comprehensive Benchmark for the Quadrant of Indic Domains}, author={Vijay Devane and Mohd Nauman and Bhargav Patel and Aniket Mahendra Wakchoure and Yogeshkumar Sant and Shyam Pawar and Viraj Thakur and Ananya Godse and Sunil Patra and Neha Maurya and Suraj Racha and Nitish Kamal Singh and Ajay Nagpal and Piyush Sawarkar and Kundeshwar Vijayrao Pundalik and Rohit Saluja and Ganesh Ramakrishnan}, year={2025}, eprint={2510.25409}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2510.25409}, } ``` ## License This dataset is released under the [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). ## Contact For any questions or feedback, please contact: - Vijay Devane ([e-mail]) - Mohd. Nauman ([e-mail]) - Bhargav Patel ([e-mail]) - Kundeshwar Pundalik ([e-mail]) ## Links - [GitHub Repository 💻](https://github.com/BharatGen-IITB-TIH/BhashaBench) - [Paper 📄](https://arxiv.org/abs/2510.25409)