Cross-Lingual Jailbreak Evaluation

Cross-Lingual Jailbreak Evaluation - Frozen Dataset and Outputs

This archive contains the frozen analysis dataset and the intermediate pipeline outputs produced for the master's thesis Multilingual Jailbreaks in Language-Specific Large Language Models (Gabriel de Jesus Coelho da Silva, advisor Carlos Becker Westphall, Universidade Federal de Santa Catarina, 2026).

The study evaluates cross-lingual jailbreak success on eight instruction-tuned, language-specific large language models, organized as four weak/strong pairs aligned in Portuguese, Italian, Swedish, and Bulgarian, plus an English-aligned reference baseline. Harmful prompts are sourced from StrongREJECT, translated into thirteen attack languages with NLLB-200, queried against each target model, back-translated to English, and scored with the official StrongREJECT evaluator. The analysis combines URIEL+ featural angular distance, BELEBELE-derived post-training specialization (IF / CONS / SPEC), tokenizer diagnostics, and a mixed-effects logistic regression with crossed prompt and language random intercepts.

What the archive contains:

dataset_frozen.parquet
Canonical frozen dataset. One row per retained or excluded prompt-model-language run, with StrongREJECT score and binary label, refusal / specificity / convincingness, URIEL+ distance, IF / CONS / SPEC, tokenizer diagnostics, translation QC, and exclusion flags.

frozen_jsonl_artifacts.zip
Aggregated JSONL views: the line-delimited equivalent of the parquet plus all StrongREJECT evaluator records concatenated across model-language shards.

belebele_predictions_jsonl.zip
Aggregated per-item BELEBELE prediction view and the per-(model, language) IF / CONS / SPEC summary table.

tables.zip
Twenty CSVs that drive every results table and figure in the thesis (ASR by model and language, Spearman correlations, GLMM coefficients, collinearity diagnostics, preregistered falsification check, coverage, and translation QC summaries).

audit.zip
XSTS LLM-judge decisions on the BLASER-flagged translation queue and the queue manifests, for each of the 13 attack languages.

translations.zip
NLLB-200 forward translations, BLASER 2.0 QE scores, English back-translations of the prompts (round-trip drift diagnostic), and per-language LLM audit decisions, for all 13 attack languages.

configs_frozen.zip
Pinned YAML configuration files (models, languages, assets, runtime) that produced this run, including upstream version identifiers and access dates for third-party assets.

model_artifacts_{MODEL_NAME}.zip
All sharded artifacts for a given model: 13 raw generations with provider metadata, 13 back-translated outputs, 13 StrongREJECT-scored shards, and the model's BELEBELE predictions.

Companion code: the evaluation harness is available at GitHub. Combined with the configuration snapshot in configs_frozen.zip, the GitHub repository plus this archive is sufficient to reproduce the analysis chapters end-to-end.

What is not redistributed: model weights, the StrongREJECT prompt set itself, BELEBELE items, NLLB-200 weights, BLASER 2.0 weights, and URIEL+ feature vectors are obtained from their original sources and remain under their respective licenses. Sabiá-3 is closed-API; the archive contains its scored outputs and request metadata only, not raw API responses.

跨语言越狱评估：冻结数据集与输出结果

本归档文件包含为硕士论文《面向特定语言大语言模型的多语言越狱攻击》（Gabriel de Jesus Coelho da Silva，指导教师Carlos Becker Westphall，圣卡塔琳娜联邦大学，2026年）所制作的冻结分析数据集与流水线中间输出结果。

本研究针对8个经过指令微调的特定语言大语言模型（Large Language Model, LLM）开展跨语言越狱攻击成功率评估，这些模型被划分为4组强弱配对，分别对应葡萄牙语、意大利语、瑞典语与保加利亚语，此外还包含一个以英语对齐的参考基准。有害提示词源自StrongREJECT数据集，通过NLLB-200工具被翻译为13种攻击语言，随后向各目标模型发起查询，再将查询结果回译为英语，并通过官方StrongREJECT评估器进行打分。本分析结合了URIEL+特征角距离、源自BELEBELE的训练后专业化指标（IF/CONS/SPEC）、分词器诊断结果，以及带有交叉提示词与语言随机截距项的混合效应逻辑回归模型。

本归档文件包含以下内容：

dataset_frozen.parquet：标准冻结数据集。每一行对应一条保留或剔除的提示词-模型-语言运行记录，包含StrongREJECT打分与二元标签、拒绝率/特异性/说服力、URIEL+距离、IF/CONS/SPEC指标、分词器诊断结果、翻译质量控制（QC）标记以及剔除标志。

frozen_jsonl_artifacts.zip：聚合JSONL格式视图：以行分隔格式存储的等价Parquet数据集，加上所有跨模型-语言分片拼接的StrongREJECT评估器记录。

belebele_predictions_jsonl.zip：聚合的单条BELEBELE预测视图，以及按（模型，语言）分组的IF/CONS/SPEC指标汇总表。

tables.zip：共计20个CSV文件，用于生成论文中所有结果表格与图表（包含按模型与语言划分的攻击成功率ASR、斯皮尔曼相关性、广义线性混合模型（Generalized Linear Mixed Model, GLMM）系数、共线性诊断、预注册证伪检验、覆盖率以及翻译质量控制汇总结果）。

audit.zip：针对13种攻击语言中每一种，包含XSTS大语言模型（LLM）评审员对BLASER标记的翻译队列及其队列清单的决策结果。

translations.zip：针对全部13种攻击语言，包含NLLB-200正向翻译结果、BLASER 2.0质量评估QE得分、提示词的英语回译结果（用于往返漂移诊断），以及按语言划分的大语言模型（LLM）评审决策结果。

configs_frozen.zip：用于生成本次实验的固定YAML配置文件（包含模型、语言、资源与运行环境配置），其中包含第三方资源的上游版本标识符与获取日期。

model_artifacts_{MODEL_NAME}.zip：对应特定模型的所有分片工件：13份带有提供商元数据的原始生成结果、13份回译输出、13份经过StrongREJECT打分的分片文件，以及该模型的BELEBELE预测结果。

配套代码：本评估工具已托管至GitHub。结合configs_frozen.zip中的配置快照，GitHub仓库与本归档文件即可完整复现论文的分析章节内容。

未重新分发的内容：模型权重、StrongREJECT提示词集本身、BELEBELE题目、NLLB-200权重、BLASER 2.0权重以及URIEL+特征向量均需从原始来源获取，且仍受各自版权协议约束。Sabiá-3为闭源API服务，本归档文件仅包含其打分结果与请求元数据，不包含原始API响应内容。