Supplementary Materials for "Accurate but Not Valid: A Mixed-Methods Validity and Fair-ness Audit of a Machine-Learning Classifier Inferring Arabic Learners' First Language from Assessment Profiles": Anonymized Instruments, Participant-Level Data, Qualitative Sub-Dataset, and Recalculated Analysis

Description This record contains the supplementary materials (File S1) accompanying the research article “Accurate but Not Valid: A Mixed-Methods Validity and Fairness Audit of a Machine-Learning Classifier Inferring Arabic Learners’ First Language from Assessment Profiles,” submitted to Behavioral Sciences (MDPI). The study investigates the construct validity and fairness of machine-learning classification of first-language (L1) backgrounds among adult learners of Arabic. Specifically, it examines whether the predictive information used to classify learners reflects cross-linguistic influence or is substantially associated with language of schooling and other educational-linguistic experiences. The study employed a sequential explanatory mixed-methods design involving 65 adult learners of Arabic at Prince Sattam bin Abdulaziz University, Al-Kharj, Saudi Arabia. The quantitative classification sample comprised 64 participants from five L1 macro-groups, excluding one heritage/contact participant (P038), who was retained in the descriptive and qualitative records. The canonical class-weighted linear support vector machine, evaluated using participant-level nested cross-validation, achieved macro-F1 = 0.6883 (95% paired-bootstrap CI [0.624, 0.753]), accuracy = 0.6719, and balanced accuracy = 0.7200. Seventeen of the 21 classification errors involved direct confusions between the Atlantic-Congo and Indo-Aryan/Iranian groups, whose schooling-language distributions overlap substantially. The strong association between L1 macro-group and language of schooling (Cramér’s V = 0.890), together with the feature-block analyses and qualitative findings, supports a schooling-related proxy interpretation without establishing individual-level causality. Contents of Supplementary Materials (File S1) Appendix A — Research Instruments and Measurement Reliability Contains the adapted Language Experience and Proficiency Questionnaire (LEAP-Q); two controlled Arabic writing prompts; an Arabic read-aloud passage designed to elicit selected phonological contrasts; the complete 40-item Arabic proficiency battery comprising 20 grammar/morphosyntax and 20 vocabulary/semantics items, with response alternatives, answer key, and documented scoring specification; and the semi-structured interview guide. This appendix also reports independent writing and speaking assessment reliability based on paired evaluations of 65 participants. The two-way random-effects, single-measure, absolute-agreement intraclass correlation coefficients were ICC(2,1) = 0.9474 for writing and 0.9507 for speaking, accompanied by confidence intervals, ANOVA components, and agreement statistics. Appendix B — Anonymized Participant-Level and Qualitative Data Includes the variable dictionary and anonymized records for all 65 participants, covering L1 macro-group, language of schooling, prior Arabic exposure, institutional proficiency level, grammar and vocabulary scores, writing and speaking assessments, and the reported proficiency composite. It also contains 16 manual transfer-coding records relating to 15 participants, including selected writing and speech characteristics, and a thematic interview sub-dataset containing representative excerpts from 21 participants. Appendix C — Statistical Analyses and Computational Reproducibility Documents the classification methodology, including class-weighted linear support vector machines, within-fold preprocessing, five-fold outer and three-fold inner stratified cross-validation, and macro-F1-based hyperparameter tuning. The appendix presents the L1-by-schooling association analysis, classification performance and confusion matrix, class-level precision and recall, majority-class and deterministic schooling-lookup benchmarks, feature-block comparisons, and robustness analyses. The canonical observed-model results are distinguished from historical exploratory analyses. The 95% confidence interval is derived from 5,000 paired-prediction bootstrap resamples. Permutation testing is specified as a separate procedure requiring complete model retraining after each target-label shuffle; previously reported permutation summaries are not represented as independently reproduced results of the canonical model. Data Protection and Availability All participants are represented by sequential pseudonymous identifiers (P001–P065). Direct personal identifiers are not included in the supplementary records, and potentially identifying location references in qualitative excerpts were generalized. No participant records were generated, replaced, or imputed. Original audio recordings, complete interview transcripts, and raw learner-produced texts are not included in this deposit. Files Included The supplementary materials are supplied as a compiled PDF and their editable XeLaTeX source. The document contains Appendices A–C, including research instruments, participant-level data, statistical tables, and methodological documentation. Repository Identifiers Version-specific DOI (Version 1.0): https://doi.org/10.5281/zenodo.23806181 Concept DOI (all versions): https://doi.org/10.5281/zenodo.23806180