Detecting AI-Generated Disinformation and Astroturfing: A 2024 Study on Bot Detection Methods

# Precision check: A critical look at the reliability of AI detection tools

Karen Paullet, Robert Morris University, [paullet@rmu.edu](mailto:paullet@rmu.edu) Jamie Pinchot, Robert Morris University, [pinchot@rmu.edu](mailto:pinchot@rmu.edu) Evan Kinney, Robert Morris University, [kinney@rmu.edu](mailto:kinney@rmu.edu) Tyler Stewart, Robert Morris University, [stewartty@rmu.edu](mailto:stewartty@rmu.edu)

# Abstract

Debates concerning the creativity, ethics, and the changing nature of learning have undoubtedly been triggered by the expanding application of students using generative AI in academia to complete their assignments. These tools raise questions regarding academic integrity. Differentiating between human created assignments and AI written material has become a challenge for educators and institutions. An experiment was conducted to test the efficacy and validity of AI detecting tools. Three tools were tested, GPTZero, Quillbot, and Polygraf AI, using 100 samples that were written by a human, completely written using AI or a combination of both human and AI written material to determine the tools capacity to accurately identify the content’s source.

Keywords: artificial intelligence, GPTZero, ChatGPT, QuillBot, Polygraph AI, large language models

# Introduction

The rapid advancements and use of artificial intelligence (AI) in academia is a critical topic that has received a great deal of attention in the field of higher education (Rafiq et al., 2025; Cotton et al., 2023). While advancements in AI have the potential to create a more engaging learning experience for students by providing personalized feedback and various types of support (Fuchs, 2023), they have also raised concerns regarding academic honesty and plagiarism (Cotton et al., 2023).

One of the most prevalent AI tools in use is ChatGPT, an AI-driven chatbot that was developed by OpenAI, an AI research company. ChatGPT was designed to generate human-like text in a conversational style, and it was introduced to the public in November of 2022. In the name ChatGPT, Chat refers to the tool’s classification as a chatbot, and GPT stands for Generative Pre-trained Transformer. The name refers to the purpose of the tool, with “generative” referring to the ability to generate text and “pre-trained” referring to the content that is used to train the tool’s algorithm so that it can continually improve text-generation capabilities as it absorbs additional data (Lieberman, 2024). ChatGPT is a large language model (LLM) that is trained on a vast amount of textual data and uses natural language processing (NLP) to have humanlike conversations with users, answering questions or generating content based on human prompts (Fuchs, 2023).

ChatGPT can perform a wide range of writing and language tasks, including text generation, summarization, paraphrasing, grammar editing, question answering, and language translation (Uyen & An, 2025; Cotton et al., 2023). It can generate research papers, ess

... (truncated, 48 KB total)