CForge

# CForge **CForge** is a unified benchmark for evaluating C code generation, organised into four difficulty tiers and designed to jointly measure **functional correctness** and **memory safety** of generated programs. | Tier | # Problems | Test cases (mean) | I/O type | |---|---|---|---| | **CForge-Introductory** | 164 | 6.95 | call-based | | **CForge-Easy** | 500 | 3.68 | call-based | | **CForge-Medium** | 5,000 | 21.20 | mostly stdin/stdout | | **CForge-Hard** | 1,055 | 26.78 | mixed (call & stdin/stdout) | ## Loading ```python from modelscope.msdatasets import MsDataset # subset_name can be one of: introductory / easy / medium / hard ds = MsDataset.load("cannhy/CForge", subset_name="medium", split="test") print(len(ds), next(iter(ds))) ``` ## Schema All four tiers share a common header (`question_id`, `question_content`, `starter_code`, `solution`, `test_code`, `type`). The harder tiers also include extra metadata: | Field | Introductory | Easy | Medium | Hard | |---|---|---|---|---| | `question_id` | ✓ | ✓ | ✓ | ✓ | | `question_content` | ✓ | ✓ | ✓ | ✓ | | `starter_code` | ✓ | ✓ | | ✓ | | `solution` | ✓ | ✓ | ✓ | ✓ | | `test_code` | ✓ | ✓ | ✓ | | | `type` | ✓ | ✓ | ✓ | ✓ | | `input_output` | | | ✓ | ✓ | | `difficulty` | | | ✓ | ✓ | | `url` | | | ✓ | | | `platform` | | | | ✓ | | `contest_date` | | | | ✓ | | `public_test_cases` | | | | ✓ | | `config` | | | | ✓ | The `type` field is one of: - `call_based` — solution implements a specific function signature; the test harness compiles your code together with `test_code` and asserts via function calls. - `io_based` — solution is a self-contained program that reads from stdin and writes to stdout. - `call_based_ctypes` — same as `call_based` but evaluated via Python `ctypes` using the I/O specs in `input_output`. ## Evaluation The companion repository [CForge](https://github.com/cannhy/CForge) provides a sandboxed evaluator that, for every generation: 1. Compiles the code with `gcc`, 2. Runs the test cases (records `pass@k`), 3. Audits memory behaviour with `valgrind` (records the **MSC** memory-safety score and the joint **MSC-pass@k** metric). A program is considered "fully passing" only when it both passes the tests **and** is reported as memory-clean by `valgrind` — capturing the gap between "it works" and "it works safely". ## Citation ```bibtex ``` ## License Apache License 2.0.