weerayuth/nitibench

--- license: mit task_categories: - sentence-similarity - text-generation tags: - legal - RAG - LCLM size_categories: - 10K<n<100K configs: - config_name: default data_files: - split: ccl path: data/ccl-* - split: tax path: data/tax-* dataset_info: features: - name: question dtype: string - name: answer dtype: string - name: relevant_laws list: - name: law_name dtype: string - name: section_content dtype: string - name: section_num dtype: string - name: reference_answer dtype: string - name: reference_laws list: - name: law_name dtype: string - name: section_content dtype: string - name: section_num dtype: string splits: - name: ccl num_bytes: 18873695 num_examples: 3729 - name: tax num_bytes: 2227708 num_examples: 50 download_size: 4728201 dataset_size: 21101403 --- # 👩🏻‍⚖️ NitiBench: A Thai Legal Benchmark for RAG **[[📄 Technical Report](https://arxiv.org/pdf/2502.10868)] | [[👨‍💻 Github Repository](https://github.com/vistec-AI/nitibench/)]** This dataset provides the **test** data for evaluating LLM frameworks, such as RAG or LCLM. The benchmark consists of two datasets: - [NitiBench-CCL](#️-nitibench-ccl) - [NitiBench-Tax](#-nitibench-tax) ## 🏛️ NitiBench-CCL Derived from the [WangchanX-Legal-ThaiCCL-RAG Dataset](https://huggingface.co/datasets/airesearch/WangchanX-Legal-ThaiCCL-RAG), our version includes an additional preprocessing step in which we separate the reasoning process from the final answer. The dataset contains 35 pieces of legislation related to **C**orporate and **C**ommercial **L**aw (CCL). Information about the 35 pieces of legislation is provided in the table below: | Legislation | Legal Terminology | Training | Test | |-------------|-------------------|----------|------| | Organic Act on Counter Corruption, B.E. 2561 | organic law | ✓ | | | Civil and Commercial Code | code | ✓ | ✓ | | Revenue Code | code | ✓ | ✓ | | Accounting Act, B.E. 2543 | act | ✓ | ✓ | | Accounting Profession Act, B.E. 2547 | act | ✓ | ✓ | | Act on Disciplinary Offenses of Government Officials Performing Duties in Agencies Other than Government Agencies, B.E. 2534 | act | ✓ | | | Act on Offences of Officials Working in State Agencies or Organizations, B.E. 2502 | act | ✓ | | | Act on Offences Relating to Registered Partnerships, Limited Partnerships, Companies Limited, Associations and Foundations, B.E. 2499 | act | ✓ | ✓ | | Act on the Establishment of Government Organizations, B.E. 2496 | act | ✓ | | | Act on the Management of Shares and Stocks of Ministers, B.E. 2543 | act | ✓ | | | Act Repealing the Agricultural Futures Trading Act, B.E. 2542 B.E. 2558 | act | ✓ | | | Budget Procedure Act, B.E. 2561 | act | ✓ | | | Business Registration Act, B.E. 2499 | act | ✓ | ✓ | | Chamber of Commerce Act, B.E. 2509 | act | ✓ | ✓ | | Derivatives Act, B.E. 2546 | act | ✓ | ✓ | | Energy Conservation Promotion Act, B.E. 2535 | act | ✓ | ✓ | | Energy Industry Act, B.E. 2550 | act | ✓ | ✓ | | Financial Institutions Business Act, B.E. 2551 | act | ✓ | ✓ | | Fiscal Discipline Act, B.E. 2561 | act | ✓ | | | Foreign Business Act, B.E. 2542 | act | ✓ | ✓ | | Government Procurement and Supplies Management Act, B.E. 2560 | act | ✓ | | | National Economic and Social Development Act, B.E. 2561 | act | ✓ | | | Petroleum Income Tax Act, B.E. 2514 | act | ✓ | ✓ | | Provident Fund Act, B.E. 2530 | act | ✓ | ✓ | | Public Limited Companies Act, B.E. 2535 | act | ✓ | ✓ | | Secured Transactions Act, B.E. 2558 | act | ✓ | ✓ | | Securities and Exchange Act, B.E. 2535 | act | ✓ | ✓ | | State Enterprise Capital Act, B.E. 2542 | act | ✓ | | | State Enterprise Committee and Personnel Qualifications Standards Act, B.E. 2518 | act | ✓ | | | State Enterprise Development and Governance Act, B.E. 2562 | act | ✓ | | | State Enterprise Labor Relations Act, B.E. 2543 | act | ✓ | | | Trade Association Act, B.E. 2509 | act | ✓ | ✓ | | Trust for Transactions in Capital Market Act, B.E. 2550 | act | ✓ | ✓ | | Emergency Decree on Digital Asset Businesses, B.E. 2561 | emergency decree | ✓ | | | Emergency Decree on Special Purpose Juristic Person for Securitization, B.E. 2540 | emergency decree | ✓ | ✓ | The training split of `nitibench-ccl` can be found in the [WangchanX-Legal-ThaiCCL-RAG dataset](https://huggingface.co/datasets/airesearch/WangchanX-Legal-ThaiCCL-RAG). ### Data Format Each data point contains four columns: - `question: str` — A question relevant to the `relevant_laws`. - `answer: str` — The original answer generated by an LLM, which has been revised and edited by legal experts to include both the reasoning steps and the final answer. - `relevant_laws: List[Dict[str, str]]` — A list of relevant law name, section, and contents. - `reference_answer: str` — The answer to the question based on the `relevant_laws`, provided without the reasoning steps. - `reference_laws: List[Dict[str, str]]` - A list of referenced law mentioned in `relevant_laws` column. Formally, given the data triple \((q, T=\{p_1, p_2, \dots, p_K\}, y)\), \(q\) represents the `question`, \(T\) represents `relevant_laws`, and \(y\) represents the `answer`. ### Data Curation Using the notation described above, the data was curated as follows: 1. Queries (\(q\)) and answers (\(y\)) were manually crafted by legal experts based on a single section sampled from the legal texts of the 35 pieces of legislation. 2. For each data triple \((q, T, y)\), the manually crafted question was carefully quality-assured by a second legal expert. Thus, for the test data, there is only one positive per query (\(|T|=1\)). The diagram below shows how the test data was collected.  ## 💸 NitiBench-Tax This subset provides a question, relevant laws, and an answer for each data point. Instead of having legal experts manually craft the questions, we scraped the data from a reliable source: the [Revenue Department Website](https://rd.go.th). This subset contains Tax Ruling Cases officially provided by the Revenue Department since 2021. As a result, this subset is considerably more challenging, as it requires extensive legal reasoning both for searching for relevant documents and for generating the answer. The data collection procedure is illustrated in the figure below:  ### Data Format This split uses the same format as described in the [NitiBench-CCL split](#data-format). ## Contact For any inquiries or concerns, please reach out to us via email: [Chompakorn Chaksangchaichot](mailto:[e-mail]). ## Citation ```bibtex @inproceedings{akarajaradwong-etal-2025-nitibench, title = "{N}iti{B}ench: Benchmarking {LLM} Frameworks on {T}hai Legal Question Answering Capabilities", author = "Akarajaradwong, Pawitsapak and Pothavorn, Pirat and Chaksangchaichot, Chompakorn and Tasawong, Panuthep and Nopparatbundit, Thitiwat and Pratai, Keerakiat and Nutanong, Sarana", editor = "Christodoulopoulos, Christos and Chakraborty, Tanmoy and Rose, Carolyn and Peng, Violet", booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing", month = nov, year = "2025", address = "Suzhou, China", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2025.emnlp-main.1739/", doi = "10.18653/v1/2025.emnlp-main.1739", pages = "34292--34315", ISBN = "979-8-89176-332-6", abstract = "Large language models (LLMs) show promise in legal question answering (QA), yet Thai legal QA systems face challenges due to limited data and complex legal structures. We introduce NitiBench, a novel benchmark featuring two datasets: (1) NitiBench-CCL, covering Thai financial laws, and (2) NitiBench-Tax, containing Thailand{'}s official tax rulings. Our benchmark also consists of specialized evaluation metrics suited for Thai legal QA. We evaluate retrieval-augmented generation (RAG) and long-context LLM (LCLM) approaches across three key dimensions: (1) the benefits of domain-specific techniques like hierarchy-aware chunking and cross-referencing, (2) comparative performance of RAG components, e.g., retrievers and LLMs, and (3) the potential of long-context LLMs to replace traditional RAG systems. Our results reveal that domain-specific components slightly improve over naive methods. At the same time, existing retrieval models still struggle with complex legal queries, and long-context LLMs have limitations in consistent legal reasoning. Our study highlights current limitations in Thai legal NLP and lays a foundation for future research in this emerging domain." } @misc{akarajaradwong2025nitibenchcomprehensivestudiesllm, title={NitiBench: A Comprehensive Studies of LLM Frameworks Capabilities for Thai Legal Question Answering}, author={Pawitsapak Akarajaradwong and Pirat Pothavorn and Chompakorn Chaksangchaichot and Panuthep Tasawong and Thitiwat Nopparatbundit and Sarana Nutanong}, year={2025}, eprint={2502.10868}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2502.10868}, } ``` ## License This dataset is provided under the **MIT License**. ## Acknowledgment We sincerely appreciate the generous support from the WangchanX program sponsors—PTT, SCB, and SCBX—whose funding made this project possible. We are also grateful for the invaluable collaboration with VISTEC, which was crucial in bringing this project to fruition. <br> <div style="display: flex; align-items: center; gap: 20px;"> Sponsored by <img src="./assets/VISAI_Logo_Horizontal.png" style="height:30px;" alt="VISAI Logo"> <img src="./assets/Logo_vistec.png" style="height:30px;" alt="VISTEC Logo"> </div>许可证：MIT许可证 任务类别： - 句子相似度 - 文本生成 标签： - 法律领域 - 检索增强生成 (Retrieval-Augmented Generation, RAG) - 长上下文大语言模型 (Long Context Large Language Model, LCLM) 样本规模：10000 < 样本数 < 100000 配置项： - 配置名称：默认 数据文件： - 拆分集：ccl 路径：data/ccl-* - 拆分集：tax 路径：data/tax-* 数据集信息： 字段特征： - 字段名：question 数据类型：字符串 - 字段名：answer 数据类型：字符串 - 字段名：relevant_laws 列表类型，元素为： - 字段名：law_name 数据类型：字符串 - 字段名：section_content 数据类型：字符串 - 字段名：section_num 数据类型：字符串 - 字段名：reference_answer 数据类型：字符串 - 字段名：reference_laws 列表类型，元素为： - 字段名：law_name 数据类型：字符串 - 字段名：section_content 数据类型：字符串 - 字段名：section_num 数据类型：字符串 拆分集信息： - 拆分集名称：ccl 字节数：18873695 样本数：3729 - 拆分集名称：tax 字节数：2227708 样本数：50 下载大小：4728201 数据集总大小：21101403 --- # 👩🏻‍⚖️ NitiBench：面向检索增强生成 (Retrieval-Augmented Generation, RAG) 的泰语法律基准数据集 **[[📄 技术报告](https://arxiv.org/pdf/2502.10868)] | [[👨‍💻 GitHub 仓库](https://github.com/vistec-AI/nitibench/)]** 本数据集专为评估大语言模型 (Large Language Model, LLM) 框架（如检索增强生成 (RAG) 或长上下文大语言模型 (LCLM)）而提供测试数据。该基准包含两个子数据集： - [NitiBench-CCL](#-nitibench-ccl) - [NitiBench-Tax](#-nitibench-tax) ## 🏛️ NitiBench-CCL 本子集源自[WangchanX-Legal-ThaiCCL-RAG 数据集](https://huggingface.co/datasets/airesearch/WangchanX-Legal-ThaiCCL-RAG)，我们的版本新增了一项预处理步骤：将推理过程与最终答案分离。本数据集涵盖35部与**公司与商业法（Corporate and Commercial Law, CCL）**相关的立法文件，35部法律的相关信息如下表所示： | 法律名称 | 法律术语分类 | 训练集 | 测试集 | |-------------|-------------------|----------|------| | 《反贪污组织法，佛历2561年》（Organic Act on Counter Corruption, B.E. 2561） | 组织法 | ✓ | | | 《民商法典》（Civil and Commercial Code） | 法典 | ✓ | ✓ | | 《税收法典》（Revenue Code） | 法典 | ✓ | ✓ | | 《会计法案，佛历2543年》（Accounting Act, B.E. 2543） | 法案 | ✓ | ✓ | | 《会计职业法案，佛历2547年》（Accounting Profession Act, B.E. 2547） | 法案 | ✓ | ✓ | | 《非政府机构公职人员职务违纪行为法案，佛历2534年》（Act on Disciplinary Offenses of Government Officials Performing Duties in Agencies Other than Government Agencies, B.E. 2534） | 法案 | ✓ | | | 《国家机关或组织公职人员犯罪法案，佛历2502年》（Act on Offences of Officials Working in State Agencies or Organizations, B.E. 2502） | 法案 | ✓ | | | 《注册合伙、有限合伙、有限公司、协会与基金会相关犯罪法案，佛历2499年》（Act on Offences Relating to Registered Partnerships, Limited Partnerships, Companies Limited, Associations and Foundations, B.E. 2499） | 法案 | ✓ | ✓ | | 《政府组织设立法案，佛历2496年》（Act on the Establishment of Government Organizations, B.E. 2496） | 法案 | ✓ | | | 《部长股份与股票管理法案，佛历2543年》（Act on the Management of Shares and Stocks of Ministers, B.E. 2543） | 法案 | ✓ | | | 《废除农产品期货交易法案，佛历2542年与佛历2558年》（Act Repealing the Agricultural Futures Trading Act, B.E. 2542 B.E. 2558） | 法案 | ✓ | | | 《预算程序法案，佛历2561年》（Budget Procedure Act, B.E. 2561） | 法案 | ✓ | | | 《商业登记法案，佛历2499年》（Business Registration Act, B.E. 2499） | 法案 | ✓ | ✓ | | 《商会法案，佛历2509年》（Chamber of Commerce Act, B.E. 2509） | 法案 | ✓ | ✓ | | 《衍生品法案，佛历2546年》（Derivatives Act, B.E. 2546） | 法案 | ✓ | ✓ | | 《能源节约促进法案，佛历2535年》（Energy Conservation Promotion Act, B.E. 2535） | 法案 | ✓ | ✓ | | 《能源产业法案，佛历2550年》（Energy Industry Act, B.E. 2550） | 法案 | ✓ | ✓ | | 《金融机构业务法案，佛历2551年》（Financial Institutions Business Act, B.E. 2551） | 法案 | ✓ | ✓ | | 《财政纪律法案，佛历2561年》（Fiscal Discipline Act, B.E. 2561） | 法案 | ✓ | | | 《外国商业法案，佛历2542年》（Foreign Business Act, B.E. 2542） | 法案 | ✓ | ✓ | | 《政府采购与供应管理法案，佛历2560年》（Government Procurement and Supplies Management Act, B.E. 2560） | 法案 | ✓ | | | 《国家经济与社会发展法案，佛历2561年》（National Economic and Social Development Act, B.E. 2561） | 法案 | ✓ | | | 《石油所得税法案，佛历2514年》（Petroleum Income Tax Act, B.E. 2514） | 法案 | ✓ | ✓ | | 《公积金法案，佛历2530年》（Provident Fund Act, B.E. 2530） | 法案 | ✓ | ✓ | | 《公众有限公司法案，佛历2535年》（Public Limited Companies Act, B.E. 2535） | 法案 | ✓ | ✓ | | 《担保交易法案，佛历2558年》（Secured Transactions Act, B.E. 2558） | 法案 | ✓ | ✓ | | 《证券与交易法案，佛历2535年》（Securities and Exchange Act, B.E. 2535） | 法案 | ✓ | ✓ | | 《国有企业资本法案，佛历2542年》（State Enterprise Capital Act, B.E. 2542） | 法案 | ✓ | | | 《国有企业委员会与人员资质标准法案，佛历2518年》（State Enterprise Committee and Personnel Qualifications Standards Act, B.E. 2518） | 法案 | ✓ | | | 《国有企业发展与治理法案，佛历2562年》（State Enterprise Development and Governance Act, B.E. 2562） | 法案 | ✓ | | | 《国有企业劳动关系法案，佛历2543年》（State Enterprise Labor Relations Act, B.E. 2543） | 法案 | ✓ | | | 《行业协会法案，佛历2509年》（Trade Association Act, B.E. 2509） | 法案 | ✓ | ✓ | | 《资本市场交易信托法案，佛历2550年》（Trust for Transactions in Capital Market Act, B.E. 2550） | 法案 | ✓ | ✓ | | 《数字资产业务紧急法令，佛历2561年》（Emergency Decree on Digital Asset Businesses, B.E. 2561） | 紧急法令 | ✓ | | | 《资产证券化特殊目的法人紧急法令，佛历2540年》（Emergency Decree on Special Purpose Juristic Person for Securitization, B.E. 2540） | 紧急法令 | ✓ | ✓ | `nitibench-ccl` 的训练拆分集可在[WangchanX-Legal-ThaiCCL-RAG 数据集](https://huggingface.co/datasets/airesearch/WangchanX-Legal-ThaiCCL-RAG)中获取。 ### 数据格式 每条数据样本包含四个字段： - `question: str` — 与`relevant_laws`相关的问题 - `answer: str` — 由大语言模型生成的原始答案，经法律专家修订编辑，整合了推理步骤与最终答案 - `relevant_laws: List[Dict[str, str]]` — 相关法条列表，包含法律名称、法条编号与法条内容 - `reference_answer: str` — 基于`relevant_laws`生成的问题答案，不含推理步骤 - `reference_laws: List[Dict[str, str]]` — `relevant_laws`字段中提及的参考法条列表 正式而言，给定数据三元组 ((q, T={p_1, p_2, dots, p_K}, y))，其中(q)代表`question`，(T)代表`relevant_laws`，(y)代表`answer`。 ### 数据构建流程 采用上文所述的标记方式，本数据集的构建流程如下： 1. 法律专家基于从35部立法文件中抽取的单条法条，手动编写查询语句（(q)）与答案（(y)）。 2. 针对每个数据三元组 ((q, T, y))，由另一名法律专家对手动编写的问题进行严格的质量审核。 因此，测试集的每条查询仅对应一条正样本（(|T|=1)）。下图展示了测试集的采集流程：  ## 💸 NitiBench-Tax 本子集为每条数据样本提供问题、相关法条与答案。与NitiBench-CCL不同，本子集的问题并非由法律专家手动编写，而是从可靠来源——[泰国税务局官网](https://rd.go.th)——爬取所得。本子集包含2021年以来泰国税务局官方发布的税务裁定案例，因此难度显著更高，无论是检索相关法条还是生成答案，均需要进行复杂的法律推理。数据采集流程如下图所示：  ### 数据格式 本拆分集采用与[NitiBench-CCL 拆分集](#数据格式)相同的字段格式。 ## 联系方式 如有任何疑问或建议，请通过邮件联系：[Chompakorn Chaksangchaichot](mailto:[e-mail])。 ## 引用 bibtex @inproceedings{akarajaradwong-etal-2025-nitibench, title = "{N}iti{B}ench: Benchmarking {LLM} 框架在泰语法律问答任务中的性能", author = "Akarajaradwong, Pawitsapak and Pothavorn, Pirat and Chaksangchaichot, Chompakorn and Tasawong, Panuthep and Nopparatbundit, Thitiwat and Pratai, Keerakiat and Nutanong, Sarana", editor = "Christodoulopoulos, Christos and Chakraborty, Tanmoy and Rose, Carolyn and Peng, Violet", booktitle = "2025年自然语言处理经验方法会议论文集", month = nov, year = "2025", address = "中国苏州", publisher = "计算语言学协会", url = "https://aclanthology.org/2025.emnlp-main.1739/", doi = "10.18653/v1/2025.emnlp-main.1739", pages = "34292--34315", ISBN = "979-8-89176-332-6", abstract = "大语言模型（LLM）在法律问答（QA）任务中展现出应用潜力，但泰语法律问答系统仍面临数据稀缺与法律结构复杂的双重挑战。我们推出NitiBench，一款全新的基准数据集，包含两个子数据集：(1) NitiBench-CCL，覆盖泰语金融法律领域；(2) NitiBench-Tax，包含泰国官方税务裁定案例。本基准还配套专为泰语法律问答设计的专属评估指标。我们针对检索增强生成（RAG）与长上下文大语言模型（LCLM）两类方法，从三个核心维度进行评估：(1) 领域专属技术（如层级分块与跨法条引用）的性能增益；(2) 检索增强生成组件（如检索器与大语言模型）的对比表现；(3) 长上下文大语言模型替代传统检索增强生成系统的可行性。实验结果显示，领域专属组件较朴素方法性能仅有小幅提升。与此同时，现有检索模型仍难以处理复杂法律查询，长上下文大语言模型在一致性法律推理方面也存在局限。本研究揭示了当前泰语法律自然语言处理领域的现存局限，为该新兴领域的后续研究奠定了基础。" } @misc{akarajaradwong2025nitibenchcomprehensivestudiesllm, title={NitiBench：泰语法律问答场景下大语言模型框架性能的全面评估}, author={Pawitsapak Akarajaradwong and Pirat Pothavorn and Chompakorn Chaksangchaichot and Panuthep Tasawong and Thitiwat Nopparatbundit and Sarana Nutanong}, year={2025}, eprint={2502.10868}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2502.10868}, } ## 许可证 本数据集采用**MIT许可证**进行授权。 ## 致谢 我们衷心感谢WangchanX项目赞助商PTT、SCB与SCBX的慷慨资助，为本项目提供了必要的资金支持。同时感谢VISTEC团队的宝贵协作，为本项目的顺利完成提供了关键帮助。 <br> <div style="display: flex; align-items: center; gap: 20px;"> 赞助方 <img src="./assets/VISAI_Logo_Horizontal.png" style="height:30px;" alt="VISAI 标志"> <img src="./assets/Logo_vistec.png" style="height:30px;" alt="VISTEC 标志"> </div>