Unlocking LLM Insights: A Dataset for Automatic Model Card Generation

Language models (LMs) are no longer restricted to the ML community, and instruction-following LMs have led to a rise in autonomous AI agents. As the accessibility of LMs grows, it is imperative that an understanding of their capabilities, intended usage, and development cycle also improves. Model cards are a widespread practice for documenting detailed information about an ML model. To automate model card generation, we introduce a dataset of 500 question-answer pairs for 25 LMs that cover crucial aspects of the model, such as its training configurations, datasets, biases, architecture details, and training resources. We employ annotators to extract the answers from the original paper. Further, we explore the capabilities of LMs in generating model cards by answering questions. We experiment with three configurations: zero-shot generation, retrieval-augmented generation, and fine-tuning on our dataset. The fine-tuned Llama 3 model shows an improvement of 7 points over the retrieval-augmented generation setup. This indicates that our dataset can be used to train models to automatically generate model cards from paper text and reduce the human effort in the model card curation process.

大语言模型（Large Language Models，LMs）已不再局限于机器学习（Machine Learning，ML）社区，遵循指令的大语言模型推动了自主AI智能体（AI Agent）的蓬勃发展。随着大语言模型的可及性日益提升，学界与产业界对其能力、预期用途与开发周期的认知也亟需同步深化。模型卡片（Model Cards）现已成为记录机器学习模型详细信息的通用实践范式。为实现模型卡片生成的自动化，我们构建了一套涵盖25款大语言模型的数据集，其中包含500组问答对，覆盖模型训练配置、训练数据集、偏差特性、架构细节与训练资源等核心维度。我们聘请专业标注人员从原始论文中提取问答对的答案内容。此外，我们通过问答任务探究了大语言模型生成模型卡片的能力，并针对三种实验配置展开验证：零样本（Zero-shot）生成、检索增强生成以及基于本数据集的模型微调。经微调的Llama 3模型相较于检索增强生成配置，性能提升了7个百分点。这表明，本数据集可用于训练模型以自动从论文文本生成模型卡片，从而有效降低模型卡片整理流程中的人力投入。