MathChat: Benchmarking Mathematical Reasoning and Instruction Following in Multi-Turn Interactions

1. follow_up.jsonl

This file contains entries that facilitate follow-up questioning. Each line consists of three keys:

question: Sourced from the GSM8k testing set.

answer: Corresponding answer from the GSM8k testing set.

followup: Includes two rounds of follow-up questions and reference answers, formatted as a conversation between a user (A:) and an assistant (B:).

2. error_correction.jsonl

This file is designed for error correction tasks. Each line consists of three keys:

question: Sourced from the GSM8k testing set.

answer: Corresponding answer from the GSM8k testing set.

error_correction: Contains a conversation between a user (A:) and an assistant (B:), which includes the original question, an incorrect answer, and the process of correcting the error.

3. error_analysis.jsonl

This file also focuses on error correction but employs a different prompt strategy. Each line consists of three keys:

question: Sourced from the GSM8k testing set.

answer: Corresponding answer from the GSM8k testing set.

error_analysis: Includes a conversation between a user (A:) and an assistant (B:), where the model is prompted to independently determine the correctness of the answer without being explicitly told.

4. p2p_generation.jsonl

This file contains entries for problem generation tasks. Each line consists of three keys:

question: Sourced from the GSM8k testing set.

answer: Corresponding answer from the GSM8k testing set.

new_problem: A new problem generated by GPT-4 to serve as a reference answer.