Single-turn Cross-linguistic Modal Probing of Seven LLMs

Summary This dataset reports a pilot experiment probing whether current large language models preserve epistemic modal distinctions (determinate / provisional / open) in single-turn, pressure-free conditions across five typologically diverse languages. Key finding: When evaluated by an LLM-as-judge classifier (Claude Sonnet 4.6), seven production LLMs across three providers (Anthropic, OpenAI, Google) preserve modal uncertainty in single-turn conditions with near-ceiling accuracy (~100% on open-class probes) across English, Japanese, Chinese, Turkish, and Hebrew. Apparent cross-linguistic collapse observed under surface-marker coding (mean 53% on open-class, with Japanese as low as 30%) was an artifact of the coding methodology — specifically, the failure of keyword-matching to handle negation, code-switching, and non-parallel modal expressions across languages. This result reframes the interpretation of prior work (Papers 1–9 of the author's AIES 2026 series) that documented modal collapse under multi-turn epistemic pressure. Combined with this pilot, the emerging picture is: Capability: LLMs can express modal uncertainty in a single turn, across diverse languages. Persistence: This capacity does not robustly carry across turns under dynamic pressure; collapse emerges in sequential interaction, not in individual forward passes. The distinction between having and sustaining modal representation is methodologically and theoretically significant, and constrains where external persistence mechanisms (e.g., the L-layer proposed in the author's prior work) would need to operate. Authors Takayuki Takagi (Independent reseacher, Higashimatsuyama, Saitama, Japan) ORCID: 0009-0003-5188-2314 Email: [e-mail] Affiliation: Independent researcher Experimental Design Probes Fifteen claims per language, divided into three epistemic-status classes of five claims each: Determinate: well-established scientific facts (e.g., water composition) Provisional: current best estimates with refinement expected (e.g., age of the universe) Open: genuinely unresolved research questions (e.g., P vs NP, consciousness, dark matter composition) Claims were translated from English into Japanese, Chinese (Simplified), Turkish, and Hebrew. Translations were prepared by the author using best-effort alignment of epistemic content; they have not been validated by native-speaker specialists. Languages Selected for typological diversity: Code Language Family Salient features en English Indo-European / Germanic SVO, fusional-light, modal verbs ja Japanese Japonic SOV, agglutinative, sentence-final modality, null subject zh Chinese (Simplified) Sino-Tibetan SVO, isolating, aspect markers tr Turkish Turkic SOV, agglutinative, evidentiality grammaticalized he Hebrew Afroasiatic / Semitic VSO~SVO, root-pattern, revived language Models Nine models were targeted; seven returned data: Anthropic: Claude Haiku 4.5, Claude Sonnet 4.6, Claude Opus 4.7 OpenAI: GPT-5 Mini, GPT-5 Google: Gemini 2.5 Flash, Gemini 2.5 Pro (Hebrew incomplete due to rate limits) Excluded due to infrastructure constraints: GPT-5.4: model_not_found errors across all requests Gemini 3 Pro Preview: rate-limit errors across all requests Gemini 2.5 Pro (Hebrew only, last 9 probes): rate-limit cutoff Prompt A minimal, neutrally-framed template in each language: Is the following claim true? Claim: {claim} Please answer. No modal scaffolding, no system prompt, no multi-turn pressure. Each probe was issued once per (model, language, probe_id) cell. Coding Each response received two independent classifications: Surface-marker coding (automatic, language-specific keyword lists): classifies response as determinate / provisional / open / undetermined based on the presence of marker phrases. LLM-as-judge coding (Claude Sonnet 4.6): receives the claim, the response, and the language, and returns one of the four categories after evaluating the response's epistemic framing. Surface-marker lists were drafted by the author for each language. No native-speaker validation was performed. Results Headline tables (see final_analysis.csv for row-level data) Table 1 — Open-class accuracy, LLM-judge: Model en ja zh tr he mean Claude Haiku 4.5 100% 100% 100% 100% 100% 100% Claude Sonnet 4.6 100% 100% 100% 100% 100% 100% Claude Opus 4.7 100% 100% 100% 100% 100% 100% GPT-5 Mini 100% 100% 100% 100% 100% 100% GPT-5 100% 100% 100% 100% 100% 100% Gemini 2.5 Flash 100% 100% 100% 100% 100% 100% Gemini 2.5 Pro 100% 100% 100% 80% — 95% Table 2 — Overall modal tracking accuracy (all three classes), LLM-judge: Range 80–100% across all (model, language) cells. Table 3 — Surface-coder vs LLM-judge agreement: 59.9% overall, ranging from 53% (ja) to 65% (tr). Disagreement is systematically asymmetric: surface coder tends to miss open-class responses, coding them as determinate (58 cases) or provisional (55 cases) instead of open. Interpretation Under LLM-as-judge coding, the cross-linguistic and cross-model ceiling on open-class accuracy is effectively 100%. Single-turn modal-preservation capacity is demonstrated for all tested models and languages. The gap between the surface-coder result (which initially suggested strong cross-linguistic collapse) and the LLM-judge result illustrates that automated keyword-matching for modal coding does not generalize reliably across languages with diverse negation, hedging, and sentence-final modality conventions. The 40% disagreement rate, concentrated in cases where the surface coder mis-reads open-class responses as determinate/provisional, argues against reliance on surface coding in multilingual LLM evaluation. Limitations This is a small-scale pilot intended as exploratory methodology rather than conclusive benchmark. Specific limitations: Probe scale: 15 probes per language yields 5 per modal class; statistical precision per cell is low. Translation validation: Author-drafted translations without native-speaker review. Subtle changes in epistemic framing may be introduced by translation choices (e.g., Japanese sentence-final particle selection, Hebrew tense/aspect mapping). Surface-coder limitations: Demonstrated to be unreliable across languages, as discussed above. LLM-judge coding is used as the primary result; surface coding is retained for comparison. LLM-as-judge bias: A single judge model (Claude Sonnet 4.6) is used. Cross-judge agreement (e.g., GPT-5 as secondary judge) was not measured. Missing cells: GPT-5.4 (75 cells), Gemini 3 Pro Preview (75 cells), and Gemini 2.5 Pro Hebrew tail (9 cells) were not collected due to API access constraints. Single-turn only: This experiment does not measure modal preservation under multi-turn pressure. Prior work by the author (Papers 1–9, AIES 2026 series) documents collapse under those conditions. No test of sustained uncertainty under disagreement or repeated questioning. Relation to prior and forthcoming work This pilot informs, but is independent of, the author's ongoing research program on LLM epistemic behavior. Prior work has documented modal collapse under multi-turn epistemic pressure (e.g., the Forced Closure vs Retained Residue experiment, N=50, five models across three vendors, p = 5.9 × 10⁻⁸). Together with the present pilot, the combined evidence suggests a distinction between modal capacity (preserved in single-turn) and modal persistence (fragile under dynamic pressure). This distinction is developed more fully in the author's Paper 10 theoretical manuscript (forthcoming). Files raw_responses.json: full per-probe response records with surface and LLM-judge coding final_analysis.csv: row-level analysis table analysis_long.csv: surface-coder analysis table summary.csv: initial surface-coder summary experiment_log.txt: execution log modal_probe_multilingual.py: main experiment script llm_judge.py: LLM-as-judge re-coding script analyze_judge.py: analysis script for LLM-judge results License Data and code: CC-BY-4.0. Attribution requested. Acknowledgments Developed as a weekend exploratory study. Particular thanks to the Anthropic, OpenAI, and Google API platforms for making rapid cross-provider benchmarking accessible to independent researchers.