AI Detection Software Use in Schools Faces Scrutiny Over Accuracy

Use of AI Detection Software in Educational Settings Raises Concerns

The implementation of artificial intelligence (AI) detection software in educational environments has led to instances of students being accused of using AI in their assignments. These tools, designed to identify AI-generated text, have faced scrutiny regarding their accuracy and impact on students.

Ailsa Ostovitz, a junior at Eleanor Roosevelt High School in Maryland, was accused on three separate assignments of using AI. In one instance, a writing assignment received a 30.76% probability score from an AI detection program. Ostovitz disputed the accusation, stating she had not used AI. Her teacher reportedly did not respond to her initial query and docked her grade.

Ostovitz's mother, Stephanie Rizk, subsequently met with the teacher. Prince George's County Public Schools (PGCPS), the district involved, stated that Ostovitz's teacher used the AI detection tool independently. PGCPS does not fund such software and advises educators against relying on it due to documented inaccuracies and inconsistencies. Following the meeting, the teacher reportedly no longer believed Ostovitz used AI.

To prevent future issues, Ostovitz now pre-screens her assignments using multiple AI detection tools and revises sentences flagged as potentially AI-generated, which adds approximately 30 minutes to her work per assignment.

A nationally representative poll by the Center for Democracy and Technology indicated that over 40% of 6th- to 12th-grade teachers utilized AI detection tools during the previous school year. This widespread adoption occurs despite multiple research studies highlighting the tools' variable reliability.

Mike Perkins, a researcher in academic integrity and AI at British University Vietnam, identified issues with several popular AI detectors, including Turnitin, GPTZero, and Copyleaks. His research showed these tools could misclassify both human-written and AI-generated text, with accuracy decreasing when AI-generated text was modified to appear more human.

Despite reliability concerns, some school districts across the U.S. have invested in AI detection software.

• Broward County Public Schools, Florida: The district has a three-year contract worth over $550,000 with Turnitin, which introduced an AI detection feature in 2023. Turnitin generates a percentage indicating the likelihood of AI generation but notes scores below 20% are less reliable. The company advises against using its AI writing detection as the sole basis for adverse actions against students due to potential inaccuracies. Broward district leadership stated the tool facilitates conversation and feedback rather than being used for grading. The district also uses it to assist with authentication requirements for academic programs like International Baccalaureate (IB), though IB does not mandate such software for authentication.

• Shaker Heights City School District, Ohio: This district spends approximately $5,600 annually for GPTZero licenses for 27 teachers. Language and literature teacher John Grady utilizes GPTZero as an initial indicator. If an assignment receives an AI likelihood score exceeding 50%, Grady conducts further investigation, including reviewing revision histories. He reported that approximately 75% of students admit to AI use when confronted under these circumstances. Edward Tian, co-founder and CEO of GPTZero, characterized the tool as a "tool in the toolkit" for educators, not a definitive judgment instrument, and recommended further examination for scores above 50%.

Concerns have been raised regarding potential biases in AI detection software. Zi Shi, a non-native English speaker and student at Shaker Heights, stated his writing style, characterized by word repetition due to limited vocabulary, has been flagged by GPTZero. His teacher suggested that using grammar-checking software like Grammarly might have contributed to the flagging. Research on AI detection tools' bias against non-native English speakers is currently limited and yields contradictory results.

Carrie Cofer, an English teacher in the Cleveland Metropolitan School District, reported that a chapter from her Ph.D. dissertation was flagged as 89-91% AI-written by GPTZero during an experiment. Her district currently does not purchase AI detection software, and Cofer advocates against its use, citing inefficiency and the likelihood of students finding workarounds, such as using "AI humanizer" programs or pre-screening their work with detectors.

Educators like Cofer suggest adapting teaching and assessment methods as a more effective response to the rise of AI in education.