MAGE framework testing of ChatGPT4

Evaluation of vulnerabilities of critical thinking skills and values of ChatGPT4 as of October 2023. The MAGE framework to evaluate six cognitive skills - justify, evaluate, reflect, analyse, interpret and explain. There were eight values of inquiry used - clarity, accuracy, precision, depth, breadth, relevance and significance and coherence. Each document contains one cognitive skill and four appropriate values of inquiry. Four values per cognitive skill were seen as the maximum number that could be effectively evaluated, as well as being likely to be the maximum number of values that would be assessed in a given task. Each task corresponds to an undergraduate university assessment question, with the precise details altered to maintain the integrity of the original exam (unless publicly available on the UQ eSpace). Due to the limitations of ChatGPT4 and the number of tokens provided, only the “minimally adapted” and “prompt engineered” tests were performed. It is unknown if these results will be replicable as the technology continues to update.

截至2023年10月，针对ChatGPT4的批判性思维技能与探究价值观的脆弱性展开评估。本研究采用MAGE框架，对六项认知技能——证成（justify）、评估（evaluate）、反思（reflect）、分析（analyse）、阐释（interpret）与解释（explain）——开展评测。本次实验共设置八项探究价值观：清晰性、准确性、精确性、深刻性、广博性、关联性、重要性与融贯性。每份评估材料包含一项认知技能与四项适配的探究价值观。研究团队认定，单类认知技能搭配四项探究价值观，是可实现有效评估的最大数量，同时也契合特定任务中常规评估的价值观上限。每项任务均对应大学本科阶段的考核试题，为保留原始考试的命题完整性，对试题细节进行了调整（若原始试题已在UQ eSpace（昆士兰大学电子空间）公开，则不作修改）。鉴于ChatGPT4的模型局限性与上下文Token（Token）长度限制，本次仅开展了"minimally adapted"与"prompt engineered"两类测试。随着该技术持续迭代更新，目前尚无法确定本次实验结果是否具备可复现性。