Confusion matrix for the POS trigrams.

Public comments are an important opinion for civic when the government establishes rules. However, recent AI can easily generate large quantities of disinformation, including fake public comments. We attempted to distinguish between human public comments and ChatGPT-generated public comments (including ChatGPT emulated that of humans) using Japanese stylometric analysis. Study 1 conducted multidimensional scaling (MDS) to compare 500 texts of five classes: Human public comments, GPT-3.5 and GPT-4 generated public comments only by presenting the titles of human public comments (i.e., zero-shot learning, GPTzero), GPT-3.5 and GPT-4 emulated by presenting sentences of human public comments and instructing to emulate that (i.e., one-shot learning, GPTone). The MDS results showed that the Japanese stylometric features of the public comments were completely different from those of the GPTzero-generated texts. Moreover, GPTone-generated public comments were closer to those of humans than those generated by GPTzero. In Study 2, the performance levels of the random forest (RF) classifier for distinguishing three classes (human, GPTzero, and GPTone texts). RF classifiers showed the best precision for the human public comments of approximately 90%, and the best precision for the fake public comments generated by GPT (GPTzero and GPTone) was 99.5% by focusing on integrated next writing style features: phrase patterns, parts-of-speech (POS) bigram and trigram, and function words. Therefore, the current study concluded that we could discriminate between GPT-generated fake public comments and those written by humans at the present time.

政府制定公共政策时，公众意见是关键的民意参考。然而，当前人工智能（AI）可轻松生成海量虚假信息，其中便包含伪造的公众意见文本。本研究采用日语文体计量分析（stylometric analysis），旨在区分人类撰写的公众意见与ChatGPT生成的公众意见（含ChatGPT模仿人类口吻生成的内容）。研究1采用多维标度法（multidimensional scaling, MDS），对5个类别的500篇文本展开对比分析，具体类别包括：人类撰写的公众意见、仅基于人类公众意见标题生成的GPT-3.5与GPT-4文本（即零样本学习，GPTzero）、基于人类公众意见语句并要求模仿生成的GPT-3.5与GPT-4模仿式文本（即单样本学习，GPTone）。多维标度法结果显示，真实公众意见的日语文体计量特征与GPTzero生成文本的特征完全迥异；此外，GPTone生成的公众意见相比GPTzero生成的文本，更贴近人类撰写的内容。在研究2中，本研究针对区分三类文本（人类撰写文本、GPTzero文本与GPTone文本）的随机森林（random forest, RF）分类器性能展开评估。通过聚焦整合的写作风格特征：短语模式、词性（parts-of-speech, POS）二元组与三元组以及功能词，随机森林分类器对人类公众意见的分类精确率最高可达约90%，对GPT生成的伪造公众意见（GPTzero与GPTone）的分类精确率最高为99.5%。因此，本研究得出结论：当前可有效区分GPT生成的伪造公众意见与人类撰写的真实公众意见。