Cross-Lingual Jailbreak Evaluation

Cross-Lingual Jailbreak Evaluation - Frozen Dataset and Outputs

This archive contains the frozen analysis dataset and the intermediate pipeline outputs produced for the master's thesis Multilingual Jailbreaks in Language-Specific Large Language Models (Gabriel de Jesus Coelho da Silva, advisor Carlos Becker Westphall, Universidade Federal de Santa Catarina, 2026).

The study evaluates cross-lingual jailbreak success on eight instruction-tuned, language-specific large language models, organized as four weak/strong pairs aligned in Portuguese, Italian, Swedish, and Bulgarian, plus an English-aligned reference baseline. Harmful prompts are sourced from StrongREJECT, translated into thirteen attack languages with NLLB-200, queried against each target model, back-translated to English, and scored with the official StrongREJECT evaluator. The analysis combines URIEL+ featural angular distance, BELEBELE-derived post-training specialization (IF / CONS / SPEC), tokenizer diagnostics, and a mixed-effects logistic regression with crossed prompt and language random intercepts.

What the archive contains:

dataset_frozen.parquet
Canonical frozen dataset. One row per retained or excluded prompt-model-language run, with StrongREJECT score and binary label, refusal / specificity / convincingness, URIEL+ distance, IF / CONS / SPEC, tokenizer diagnostics, translation QC, and exclusion flags.

frozen_jsonl_artifacts.zip
Aggregated JSONL views: the line-delimited equivalent of the parquet plus all StrongREJECT evaluator records concatenated across model-language shards.

belebele_predictions_jsonl.zip
Aggregated per-item BELEBELE prediction view and the per-(model, language) IF / CONS / SPEC summary table.

tables.zip
Twenty CSVs that drive every results table and figure in the thesis (ASR by model and language, Spearman correlations, GLMM coefficients, collinearity diagnostics, preregistered falsification check, coverage, and translation QC summaries).

audit.zip
XSTS LLM-judge decisions on the BLASER-flagged translation queue and the queue manifests, for each of the 13 attack languages.

translations.zip
NLLB-200 forward translations, BLASER 2.0 QE scores, English back-translations of the prompts (round-trip drift diagnostic), and per-language LLM audit decisions, for all 13 attack languages.

configs_frozen.zip
Pinned YAML configuration files (models, languages, assets, runtime) that produced this run, including upstream version identifiers and access dates for third-party assets.

model_artifacts_{MODEL_NAME}.zip
All sharded artifacts for a given model: 13 raw generations with provider metadata, 13 back-translated outputs, 13 StrongREJECT-scored shards, and the model's BELEBELE predictions.

Companion code: the evaluation harness is available at GitHub. Combined with the configuration snapshot in configs_frozen.zip, the GitHub repository plus this archive is sufficient to reproduce the analysis chapters end-to-end.

What is not redistributed: model weights, the StrongREJECT prompt set itself, BELEBELE items, NLLB-200 weights, BLASER 2.0 weights, and URIEL+ feature vectors are obtained from their original sources and remain under their respective licenses. Sabiá-3 is closed-API; the archive contains its scored outputs and request metadata only, not raw API responses.