JamC-QA

# Dataset Card for JamC-QA English/[Japanese](README_ja.md) ## Dataset Summary This benchmark evaluates knowledge specific to Japan through multiple-choice questions. It covers eight categories: culture, custom, regional_identity, geography, history, government, law, and healthcare. Achieving high performance requires broad and detailed understanding of Japan across these categories. ## Leaderboard ### Evaluation Metric In our evaluation, the LLM outputs the option string rather than the option label, and the following table shows the proportion of outputs that exactly match the gold option string. | Model | All | culture | custom | regional_identity | geography | history | government | law | healthcare | |:---|----|---:|---:|---:|---:|---:|---:|---:|---:| | [sarashina2-8x70b](https://huggingface.co/sbintuitions/sarashina2-8x70b) | **0.725** | 0.714 | **0.775** | **0.761** | 0.654 | **0.784** | 0.736 | 0.632 | **0.917** | | [sarashina2-70b](https://huggingface.co/sbintuitions/sarashina2-70b) | **0.725** | **0.719** | 0.745 | 0.736 | **0.673** | 0.764 | 0.764 | 0.666 | **0.917** | | [Llama-3.3-Swallow-70B-v0.4](https://huggingface.co/tokyotech-llm/Llama-3.3-Swallow-70B-v0.4) | 0.697 | 0.689 | **0.775** | 0.589 | 0.566 | 0.776 | **0.773** | **0.783** | 0.854 | | [RakutenAI-2.0-8x7B](https://huggingface.co/Rakuten/RakutenAI-2.0-8x7B) | 0.633 | 0.622 | 0.725 | 0.617 | 0.511 | 0.714 | 0.709 | 0.575 | 0.813 | | [plamo-100b](https://huggingface.co/pfnet/plamo-100b) | 0.603 | 0.602 | 0.650 | 0.637 | 0.504 | 0.682 | 0.609 | 0.515 | 0.688 | | [Mixtral-8x7B-v0.1-japanese](https://huggingface.co/abeja/Mixtral-8x7B-v0.1-japanese) | 0.593 | 0.602 | 0.670 | 0.579 | 0.493 | 0.612 | 0.736 | 0.545 | 0.667 | | [Meta-Llama-3.1-405B](https://huggingface.co/meta-llama/Llama-3.1-405B) | 0.571 | 0.558 | 0.545 | 0.484 | 0.500 | 0.679 | 0.646 | 0.629 | 0.688 | | [llm-jp-3.1-8x13b](https://huggingface.co/llm-jp/llm-jp-3-8x13b) | 0.568 | 0.595 | 0.635 | 0.582 | 0.449 | 0.589 | 0.627 | 0.502 | 0.625 | | [Nemotron-4-340B-Base](https://huggingface.co/mgoin/Nemotron-4-340B-Base-hf) | 0.567 | 0.573 | 0.615 | 0.511 | 0.467 | 0.595 | 0.727 | 0.582 | 0.667 | | [Qwen2.5-72B](https://huggingface.co/Qwen/Qwen2.5-72B) | 0.527 | 0.522 | 0.595 | 0.426 | 0.438 | 0.606 | 0.609 | 0.562 | 0.688 | ## Language Japanese ## Dataset Structure ### Data Instances An example from culture category looks as follows: ``` { "qid": "jamcqa-test-culture-00001", "category": "culture", "question": "「狂った世で気が狂うなら気は確かだ」の名言を残した映画はどれ?", "choice0": "影武者", "choice1": "羅生門", "choice2": "隠し砦の三悪人", "choice3": "乱", "answer_index": 3, "difficulty": 4, } ``` ## Data Fields - `qid (str)`: A unique identifier for each question. - `category (str)`: The category of the question. - culture, custom, regional_identity, geography, history, government, law, and healthcare - `question (str)`: The question text. - Converted from full-width to half-width characters, excluding katakana characters. - Does not contain any line breaks (`\n`). - Leading and trailing whitespace is removed. - `choice{0..3} (str)`: Four answer options (`choice0` to `choice3`). - Converted from full-width to half-width characters, excluding katakana characters. - Does not contain any line breaks (`\n`). - Leading and trailing whitespace is removed. - `answer_index (int)`: The index of the correct answer among `choice0` to `choice3` (0–3). - `difficulty (int)`: The number of models that correctly answered the example (0-9). - 0: 0 models were able to choose the correct answer. - 9: 9 models were able to choose the correct answer. - For details, please refer to section 6.2 of [\[Oka+, 26\]](https://lrec.elra.info/lrec2026-main-356). ## Data Splits - `dev`: 4 examples per category, intended for few-shot evaluation - `test`: 2,309 examples in total Number of Examples: | Category | dev | test | | --- | ---: | ---: | | culture | 4 | 640 | | custom | 4 | 200 | | regional_identity | 4 | 397 | | geography | 4 | 272 | | history | 4 | 343 | | government | 4 | 110 | | law | 4 | 299 | | healthcare | 4 | 48 | | total | 32 | 2,309 | # Licensing Information - [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/) # Usage ## Dataset Loading ```python $ python >>> import datasets >>> jamcqa = datasets.load_dataset('sbintuitions/JamC-QA', 'v1.0') >>> print(jamcqa) DatasetDict({ test: Dataset({ features: ['qid', 'category', 'question', 'choice0', 'choice1', 'choice2', 'choice3', 'answer_index', 'difficulty'], num_rows: 2309 }) dev: Dataset({ features: ['qid', 'category', 'question', 'choice0', 'choice1', 'choice2', 'choice3', 'answer_index', 'difficulty'], num_rows: 32 }) }) >>> jamcqa_test = jamcqa['test'] >>> print(jamcqa_test) Dataset({ features: ['qid', 'category', 'question', 'choice0', 'choice1', 'choice2', 'choice3', 'answer_index', 'difficulty'], num_rows: 2309 }) >>> print(jamcqa_test[0]) {'qid': 'jamcqa-test-culture-00001', 'category': 'culture', 'question': '「狂った世で気が狂うなら気は確かだ」の名言を残した映画はどれ?', 'choice0': '影武者', 'choice1': '羅生門', 'choice2': '隠し砦の三悪人', 'choice3': '乱', 'answer_index': 3, 'difficulty': 4} >>> ``` ## Evaluation with FlexEval You can easily use [FlexEval](https://github.com/sbintuitions/flexeval) (version 0.13.3 or later) to evaluate the JamC-QA score by simply replacing `commonsense_qa` with `jamcqa` in the [Quickstart](https://github.com/sbintuitions/flexeval?tab=readme-ov-file#quick-start) guide. ### Run Command ```python flexeval_lm \ --language_model HuggingFaceLM \ --language_model.model "sbintuitions/sarashina2.2-0.5b" \ --language_model.default_gen_kwargs "{ do_sample: false }" \ --eval_setup "jamcqa" \ --save_dir "results/jamcqa" ``` `--language_model.default_gen_kwargs "{ do_sample: false }"` disables sampling and performs [greedy search](https://huggingface.co/docs/transformers/generation_strategies#greedy-search). ### Output ``` ... 2025-09-03 15:48:24.633 | INFO | flexeval.core.evaluate_generation:evaluate_generation:92 - {'exact_match': 0.2368990905153746, 'finish_reason_ratio-stop': 1.0, 'avg_output_length': 6.94283239497618, 'max_output_length': 93, 'min_output_length': 2} ... ``` # Citation Information ```bibtex @inproceedings{oka-etal-2026-jamcqa, author={Oka, Teruaki and Shibata, Tomohide and Yoshida, Nao}, title={JamC-QA: A Multiple-Choice Question Answering Benchmark for Japan-Specific Knowledge}, year={2026}, month={May}, booktitle={Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026)}, pages={4536--4546}, doi={10.63317/33xwat5joobd}, } ```