Lila

Lila是一个统一的数学推理基准，由四个维度的 23 个不同任务组成：(i) 数学能力，例如算术、微积分 (ii) 语言格式，例如问答、填空 (iii) 语言多样性，例如，没有语言，简单的语言 (iv) 外部知识，例如常识、物理学。该基准测试是通过收集Python程序形式的任务指令和解决方案来扩展20个数据集基准测试而构建的，从而除了正确答案之外还获得了可解释的解决方案。

Lila is a unified mathematical reasoning benchmark composed of 23 distinct tasks across four dimensions: (i) Mathematical capabilities, such as arithmetic and calculus; (ii) Language formats, such as question answering and fill-in-the-blank; (iii) Linguistic diversity, such as language-free and simple natural language; (iv) External knowledge, such as common sense and physics. This benchmark is constructed by expanding 20 existing dataset benchmarks by collecting task instructions and solutions in the form of Python programs, thereby obtaining interpretable solutions alongside correct final answers.

搜集汇总

背景与挑战

背景概述

Lila是一个数学推理的综合基准数据集，包含超过140K自然语言问题，使用Python程序和自然语言指令注释，覆盖23个不同任务。该数据集提供多个分割（如Lila-IID、Lila-OOD和Lila-Robust），旨在评估大语言模型在数学推理方面的性能，适用于文本评测和评估场景。