Anishinaabemowin AI Translation Tool Evaluation Dataset

This dataset contains comprehensive evaluation results from systematic testing of five commercially available AI translation tools for Anishinaabemowin (Ojibwe language). The research addresses critical questions about AI reliability for Indigenous language education, particularly for second-language learners in off-reserve locations with limited access to fluent speakers.

Phase 1: Comparative Tool Analysis

5 AI translation tools tested (AnythingTranslate, Musely.ai, Ojibwechat, ChatGPT Indigenous Language Supporter, Claude.ai Pro)

17 Anishinaabemowin words plus 2 complex sentences

Structured documentation with validation against authoritative print and online dictionaries

Systematic consistency testing through repeated queries to demonstrate all models have errors

Data files: 4 CSV files (2025-08-12-14 Ojibwe Translation Tests)

Phase 2: Guardrail Development

JSON-based cultural protocol guardrails for Indigenous language AI

Conversational testing with ChatGPT, Claude.ai, and Perplexity

Evaluation of bilingual response formats (Anishinaabemowin first, English in brackets), gentle correction methodologies, and community authority deference

Assessment of educational suitability for language learning contexts

Data files: AI TEST 2025-08-17 JSON File.md (guardrail specifications) + 4 conversation test files

Phase 3: Open Conversation Testing

"Cold start" conversations without JSON guardrails

Testing of Perplexity, ChatGPT Indigenous Language Bot, and Claude.ai

Mixed Anishinaabemowin-English conversation evaluation

Error pattern documentation and learning risk assessment

Data files: 4 conversation test files

KEY FINDINGS

Tool Performance:

AnythingTranslate and Musely.ai demonstrated complete unreliability with different translations each query

ChatGPT, Claude.ai, and Perplexity showed best overall performance with grammatical understanding but persistent word stem errors

Ojibwechat demonstrated some understanding but produced grammatical errors and defaults to translation

All systems exhibited errors requiring fluent speaker verification

Critical Translation Error Example: English: "stop walking" AI Translation (ChatGPT): "gego bimosen" ❌ (means "don't walk") Correct Translation: "boon bimosen" ✓ (means "stop walking")

This demonstrates English-based reasoning (negation vs. cessation) that poses significant learning risks.

JSON Guardrail Results:

Successfully enforced culturally appropriate behaviors (bilingual formats, gentle correction methods)

Underlying accuracy issues persist despite behavioral constraints

Demonstrates potential for Indigenous pedagogy integration with critical limitations

Risk Assessment:

Highest risk: Beginner learners without verification access (cannot identify errors)

Moderate risk: Intermediate learners with occasional fluent speaker access

Lowest risk: Advanced learners with regular community speaker access

Critical concern for off-reserve second-language learners (one-third of Indigenous language speakers per 2021 Census)

RESEARCH CONTEXT

This research responds to the 4% decline in Indigenous language speakers from 1991-2021 (Statistics Canada) and the increasing proportion of second-language learners. With Indigenous communities seeking technological tools to support language revitalization, this dataset provides critical evidence for evaluating AI tool safety and developing community-controlled alternatives.

The data supports development of culturally-grounded AI evaluation frameworks—specifically contributing to the proposed "Anishinaabe Turing Test"—that measure AI systems against Indigenous knowledge protocols rather than European cognitive patterns.

DATASET CONTENTS

Phase 1 Data Files:

2025-08-12-14 Ojibwe Translation Tests [4 CSV files] - Comparative tool analysis results with 17 words and 2 complex sentences

Phase 2 Data Files:

AI TEST 2025-08-17 JSON File.md - Cultural protocol guardrail specifications in JSON format

AI TEST 2025-08-17 JSON [4 MD files] - JSON guardrail conversation testing results (Claude, ChatGPT, Perplexity)

Phase 3 Data Files:

AI TEST [4 MD files] - Open conversation testing results without guardrails

Supporting Materials:

methodology.md - Detailed testing protocols and validation procedures

README.md - Complete documentation and usage guidelines

LICENSE - CC BY 4.0 license text

METHODOLOGY

Testing employed standardized protocols including:

Consistent word/phrase lists across all tools (17 words + 2 complex sentences)

Multiple researcher cross-validation

Authoritative dictionary verification against Anishinaabemowin print and online sources

Systematic documentation for reproducibility

Initial consistency tracking to demonstrate all models have errors

Iterative testing with multiple queries to assess reliability

Cultural Protocols: Research conducted with respect for Indigenous knowledge systems and in support of language revitalization efforts. Testing focused exclusively on publicly available AI tools to assess community safety. No traditional knowledge was used or shared with the AI systems. Researchers worked exclusively with language data the LLMs already possessed, ensuring protection of cultural knowledge.

IMPLICATIONS

For Researchers:

Framework for evaluating AI tools for Indigenous languages

Methodology for culturally-grounded AI assessment

Evidence of systematic errors in commercial translation systems

For Educators:

Evidence-based assessment of tool safety for classroom use

Documentation of error patterns for post-use correction protocols

Verification protocols requiring fluent speaker oversight

For Indigenous Communities:

Data to inform AI tool adoption decisions before integration into language programs

Support for community-controlled AI development initiatives

Validation of data sovereignty principles in language technology

Protection against potential harm from unreliable AI systems

EDUCATIONAL IMPACT

AI systems pose high risk for isolated learners who may internalize errors as correct language patterns. Current tools require intensive community oversight and systematic verification protocols. Without fluent speaker access, beginners cannot distinguish between correct and incorrect outputs, leading to potential long-term language acquisition problems.

TECHNICAL SPECIFICATIONS

Data Formats: CSV (comma-separated values), Markdown (.md) Character Encoding: UTF-8 Testing Period: August 12-17, 2025 Number of AI Systems Tested: 5 translation tools + 3 conversational LLMs Research Context: PhD research, York University Digital Media program Funding Support: IBET (Indigenous and Black Engineering and Technology), Abundant Intelligences Tkaronto Pod

CITATION

When using this dataset, please cite:

McConnell, A., & Ly, J. (2025). Anishinaabemowin AI Translation Tool Evaluation Dataset (Version 1.0.0) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.17274098

Associated Publication: McConnell, A., & Ly, J. (2025). "Anishinaabemowin Aabajichigan Gaawiin Nibwaakaasii (Efficacy of AI for Ojibwe Language Education)." Connected Minds 2025, Toronto, ON, October 6-8, 2025.

BibTeX: @dataset{mcconnell_anishinaabemowin_2025, author = {McConnell, Andrew and Ly, Jasmine}, title = {Anishinaabemowin AI Translation Tool Evaluation Dataset}, year = {2025}, publisher = {Zenodo}, version = {1.0.0}, doi = {10.5281/zenodo.17274098}, url = {https://doi.org/10.5281/zenodo.17274098} }

KEYWORDS

Indigenous languages, Anishinaabemowin, Ojibwe language, artificial intelligence, machine translation, language revitalization, language education, AI evaluation, Indigenous data sovereignty, cultural appropriateness, language learning, second language acquisition, Indigenous pedagogy, AI ethics, language technology, ChatGPT, Claude, Perplexity, translation tools, JSON guardrails, Indigenous futurities, off-reserve language learning

LICENSE

Creative Commons Attribution 4.0 International (CC BY 4.0) https://creativecommons.org/licenses/by/4.0/

ACKNOWLEDGMENTS

This work is supported by IBET (Indigenous and Black Engineering and Technology) and Abundant Intelligences, Tkaronto Pod. Research conducted as part of PhD studies in Digital Media at York University.

GITHUB REPOSITORY

Complete dataset and documentation available at: https://github.com/giigdo/AI_Bot_Tests_Q3_2025