PLawBench

# PLawBench PLawBench is a rubric-based benchmark for evaluating large language models on real-world Chinese legal practice. It covers the three hierarchical levels of a practitioner's workflow: public legal consultation, practical case analysis, and legal document drafting. This repository re-packages the official release ([skylenage/PLawbench](https://github.com/skylenage/PLawbench)) into a ModelScope dataset with one config per task, so each PLawBench task can be loaded as an independent subset. ## Subsets | Config (subset) | Split | Samples | PLawBench task | | --- | --- | --- | --- | | `case_analysis` | `test` | 250 | Practical case analysis (conclusion / facts / reasoning / statutes) | | `legal_consultation` | `test` | 18 | Public legal consultation (fact-eliciting question lists, "mid" difficulty) | | `plaintiff_statement` | `test` | 6 | Legal document drafting - statement of complaint | | `defendant_statement` | `test` | 6 | Legal document drafting - statement of defense | ## Fields | Field | Type | Description | | --- | --- | --- | | `id` | string | Unique sample id, prefixed with the task name. | | `task` | string | Task / subset name. | | `judge_type` | string | Rubric grading protocol: `case_analysis`, `legal_qa`, or `document_generation`. | | `category` | string | Legal domain label (`case_analysis`) or cause-of-action tag (drafting tasks). | | `context` | string | Case background, only populated for `case_analysis`. | | `question` | string | The question, or the client's statement for consultation / drafting tasks. | | `rubrics` | string | Expert-annotated rubric. A JSON array string for `case_analysis`, plain text otherwise. | | `max_points` | int64 | Total points available for the sample. | ## Usage ```python from modelscope import MsDataset ds = MsDataset.load('evalscope/PLawBench', subset_name='case_analysis', split='test') ``` Or evaluate directly with EvalScope: ```bash evalscope eval --model <your-model> --datasets plawbench --judge-strategy llm ``` ## Citation Please cite the original PLawBench release: https://github.com/skylenage/PLawbench