Fsoft-AIC/MainframeBench

---
configs:
- config_name: question_answering
data_files: question-answering/qas.csv
- config_name: multiple_choice_question
data_files: multiple-choice-question/mcs.csv
- config_name: COBOL_code_summarization
data_files: COBOL-code-summarization/summary.csv

language:
- code
- en
license: mit
task_categories:
- question-answering
- summarization
- text-classification
pretty_name: MainframeBench
viewer: true
tags:
- code
- synthetic
size_categories:
- 1K<n<10K
---

## Table of Contents
- [Dataset Summary](#dataset-summary)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Data Statitics](#dataset-statistics)
- [Usage](#usage)
- [Additional Information](#additional-information)
- - [Other Resources](#other-resources)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)

## Dataset Description

- **Repository:** [FSoft-AI4Code/MainframeBench](https://huggingface.co/datasets/Fsoft-AIC/MainframeBench)
- **Paper:** [XMAiNframe: A Large Language Model for
Mainframe Modernization](https://arxiv.org/abs/2408.04660)
- **Contact:** [e-mail]
- **Website:** https://www.fpt-aicenter.com/ai-residency/

<p align="center">
<img src="./asset/XMAiNframe.png" width="560px" alt="logo">
</p>

<div align="center">

# XMAiNframe: A Large Language Model for Mainframe Modernization
</div>

## Dataset Summary
This dataset - **MainframeBench** - contains a comprehensive benchmark for assessing mainframe knowledge, including three sub-tasks: multiple-choice questions, question answering, and COBOL code summarization.

## Dataset Structure
### Data Instances for Question Answering
```
{
"id": 0,
"prompt": "As a supportive AI assistant, you've been presented with a query related to a Cobol-related topic. Please furnish a reply to the question.",
"question": "What is the future of COBOL in mainframe computing?",
"answer":"As businesses increasingly migrate away from mainframes and update their legacy applications, the future of COBOL in mainframe computing is uncertain. However, it will likely continue to be used for maintaining existing systems and for specific business needs."
}
```
### Data Fields

Data fields for question-answering task:
- **id** (string): the unique id
- **prompt** (string): sequence to instruct LLM
- **question** (string): the question related to mainframe
- **answer** (string): answer for the corresponding question

Data fields for multiple-choice question task:
- **id** (string): the unique id
- **prompt** (string): sequence to instruct LLM
- **question** (string): the question related to mainframe
- **A**, **B**, **C**, **D** (string): four option of the question
- **answer** (string): the true choice (A or B or C or D) for the corresponding question

Data fields for COBOL code summarization task:
- **id** (string): the unique id
- **prompt** (string): sequence to instruct LLM
- **source** (string): the COBOL code snippet
- **summary** (string): the summary of the given code

### Data Splits

This benchmark is split into 3 subsets, corresponding to 3 sub-tasks: Question-Answering, Multiple-Choice Questions, and COBOL Code Summarization.

## Dataset Statistics

| Type | Number of Samples|
|:------------------------------|-----------------:|
| Question Answering | 2,598 |
| Multiple-Choice Question | 1,931 |
| COBOL code summarization | 2,523 |

## Usage
You can load this dataset using datasets library: ```pip install datasets```

```python
from datasets import load_dataset

# Load each sub-set in MainframeBench
QA_set = load_dataset("Fsoft-AIC/MainframeBench", 'question_answering')
MC_set = load_dataset("Fsoft-AIC/MainframeBench", 'multiple_choice_question')
Summarization_set = load_dataset("Fsoft-AIC/MainframeBench", 'COBOL_code_summarization')
```

## Additional Information
### Other Resources:
- Github: https://github.com/FSoft-AI4Code/XMainframe
- Paper: https://arxiv.org/abs/2408.04660

### Licensing Information
MIT License

### Citation Information

```
@misc{dau2024xmainframelargelanguagemodel,
title={XMainframe: A Large Language Model for Mainframe Modernization},
author={Anh T. V. Dau and Hieu Trung Dao and Anh Tuan Nguyen and Hieu Trung Tran and Phong X. Nguyen and Nghi D. Q. Bui},
year={2024},
eprint={2408.04660},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2408.04660},
}
```

### Contributions
This dataset is developed by [FSOFT AI4Code team](https://github.com/FSoft-AI4Code).