japanese-mail-bench

Japanese-Mail-Bench 是一个用于评估日语商务邮件生成能力的基准测试数据集。该数据集由 Preferred Networks 内部构建，包含 500 条测试样本，所有样本均为单一 test 分割。每条样本提供一个基于场景的日语邮件撰写提示（包含发件人、收件人、目的、情境及语气约束），并配有一个由基准构建流程生成的 LLM 参考答案。数据以 JSONL 格式存储，字段包括：唯一标识符 id、样本索引 sample_index、聊天格式的提示 messages、参考邮件 gold_answer（含主题和正文）、场景描述 scenario（sender、receiver、purpose、context）、发件人-收件人关系 sender_receiver_relation（20 种不同值，如“社員 -> 顧客”）、邮件目的 email_purpose（8 种值，如“依頼”、“断り”、“感謝”等）、言语行为 speech_act（4 种：お願い、説明、質問、指示）、交互类型 interaction_type（提供/要求）、交互对象 interaction_object（物品・サービス/情報）。数据集不包含任何真实个人信息，所有电子邮件地址、URL、电话号码、邮政编码、人名和公司名称均使用占位符。该数据集适用于评估模型在指令跟随下的日语商务邮件写作能力，配套评估脚本 eval.py 使用 OpenAI 兼容的裁判模型对模型输出进行评分，唯一指标为 naturalness_and_appropriateness（自然性和适当性），按 1-5 分制（5 分为最佳）与参考答案进行比较。Japanese-Mail-Bench is a benchmark dataset for evaluating Japanese business email generation capabilities. It was internally constructed by Preferred Networks, containing 500 test samples in a single test split. Each sample provides a scenario-based prompt for writing a Japanese business email (including sender, receiver, purpose, context, and tone constraints) along with a reference answer generated by an LLM via the benchmark construction pipeline. The data is stored in JSONL format with fields: id, sample_index, messages (chat-format prompt), gold_answer (subject and body), scenario (sender, receiver, purpose, context), sender_receiver_relation (20 values, e.g., employee -> customer), email_purpose (8 values, e.g., request, refusal, thanks), speech_act (4 values: request, explanation, question, instruction), interaction_type (provide/request), interaction_object (goods&services/information). The dataset contains no real personal information; all email addresses, URLs, phone numbers, postal codes, person names, and company names are placeholders. It is suitable for evaluating instruction-following ability in Japanese business email writing, with an accompanying evaluation script eval.py using an OpenAI-compatible judge model to score outputs on a 1-5 scale (5 best) for the unique metric naturalness_and_appropriateness compared to the reference answer.