LLM Benchmark Performance and Sentence Embeddings Dataset

Overview

This dataset accompanies research on the structural similarity of large language model (LLM) performance rankings across benchmarks, as measured through sentence-level embeddings of benchmark prompts. It contains raw performance scores for 66 frontier and open-source LLMs evaluated on four benchmarks, together with precomputed sentence embeddings produced by six embedding models under multiple chunking strategies.

All benchmark scores were collected from the HELM Capabilities leaderboard as of May 2025.

Benchmarks

Four single-turn benchmarks were selected to ensure all instances can be represented as independent text embeddings:

Benchmark
Task Type
Score Distribution

MMLU-Pro
Knowledge-intensive QA
Binary per-item accuracy; smooth, high rank separability

GPQA
Knowledge-intensive QA
Binary per-item accuracy; smooth, high rank separability

IFEval
Instruction following
Discretized; reduced granularity

Omni-MATH
Mathematical reasoning
Bounded/discretized; reduced granularity

Note: Because HELM scores may reflect differing prompting strategies across model families (e.g., zero-shot vs. few-shot), comparisons are relative to HELM-reported baselines rather than a controlled prompting protocol.

Models Evaluated

66 frontier and open-source LLMs are included, spanning families such as GPT, Claude, Gemini, Llama, Qwen3, DeepSeek, and Mistral. The full model list is provided in the accompanying paper.

Embedding Models

Embeddings were computed using six models spanning two architectural regimes:

Small encoder-based models (~512-token context window):

bge-large-en-v1.5

e5-large

GIST-small-Embedding-v0

Large decoder-based models (up to 4,096-token context window):

Llama-Embed-Nemotron-8B

Qwen3-4B

Qwen3-8B

All models use attention-mask-weighted mean pooling over final hidden states followed by L2 normalization.

Chunking Strategies

Encoder models: Length-weighted averaging over non-overlapping 510-token chunks (to avoid truncation artifacts given the 512-token context limit).

Decoder models — full sequence (NOCHUNK): Prompts encoded in their entirety (maximum observed length: 2,278.5 tokens, within the 4,096-token context window).

Decoder models — chunked (CHUNKED): Overlapping windows with stride 256, enabling controlled comparison between full-sequence and segmented representations.

File Descriptions

Aggregated Performance Data

Raw benchmark scores for all 66 LLMs.

File
Description

GPQA_aggregated.json
Per-model GPQA scores (1.2 MB)

IFEval_aggregated.json
Per-model IFEval scores (1.5 MB)

MMLU-Pro_aggregated.json
Per-model MMLU-Pro scores (2.7 MB)

Omni-MATH_aggregated.json
Per-model Omni-MATH scores (2.8 MB)

Each file contains aggregated accuracy/performance metrics per LLM as reported on the HELM Capabilities leaderboard.

Large (Decoder-based) Embeddings

Precomputed embeddings from the three decoder-based models (Llama-Embed-Nemotron-8B, Qwen3-4B, Qwen3-8B), one file per benchmark per chunking strategy. Each file is a JSONL where each line represents one benchmark instance.

File
Chunking
Size

embeddings_large_GPQA_CHUNKED.jsonl
Overlapping windows (stride 256)
109.1 MB

embeddings_large_GPQA_NOCHUNK.jsonl
Full sequence
109.1 MB

embeddings_large_IFEval_CHUNKED.jsonl
Overlapping windows (stride 256)
132.3 MB

embeddings_large_IFEval_NOCHUNK.jsonl
Full sequence
132.3 MB

embeddings_large_MMLU-Pro_CHUNKED.jsonl
Overlapping windows (stride 256)
244.6 MB

embeddings_large_MMLU-Pro_NOCHUNK.jsonl
Full sequence
244.6 MB

embeddings_large_Omni-MATH_CHUNKED.jsonl
Overlapping windows (stride 256)
243.6 MB

embeddings_large_Omni-MATH_NOCHUNK.jsonl
Full sequence
243.7 MB

Small (Encoder-based) Embeddings

Precomputed embeddings from the three encoder-based models (bge-large-en-v1.5, e5-large, GIST-small-Embedding-v0), one file per benchmark. Chunking (non-overlapping 510-token windows with length-weighted averaging) is applied uniformly to all encoder models.

File
Size

small_embeds_sp_GPQA_fixed.jsonl
24.0 MB

small_embeds_sp_IFEval_fixed.jsonl
29.2 MB

small_embeds_sp_MMLU-Pro_fixed.jsonl
53.9 MB

small_embeds_sp_Omni-MATH_fixed.jsonl
53.8 MB

JSONL Format

Each line in a .jsonl file is a JSON object with the following fields:

json

{
"instance_id": "<benchmark_instance_identifier>",
"model": "<embedding_model_name>",
"embedding": [0.012, -0.034, ...]
}

Embeddings are L2-normalized vectors. Dimensionality depends on the embedding model used.

Intended Use

This dataset is intended to support research on:

Structural similarity and alignment of LLM benchmark rankings

The relationship between benchmark prompt content and model performance ordering

Embedding-based analysis of benchmark difficulty and item clustering

Rank correlation methods (e.g., Spearman's ρ, Kendall's W) applied to LLM evaluation