The hidden cost of AI detection false positives

The public conversation about AI detection in schools is almost entirely about the problem of students who cheat and go undetected. The other side of the equation gets far less attention: the students who are flagged when they shouldn't be, and what happens to them when they are.

What a false positive actually means

In statistical terms, a false positive is a case where the test returns a positive result but the underlying condition is absent - in this context, a student who is flagged as having used AI but who didn't. This can happen for several reasons: the student writes in formal, careful language; they are a non-native English speaker; their writing style happens to share statistical characteristics with AI-generated text.

In practical terms, a false positive can mean an accusation of academic misconduct. Depending on institutional policy, this might involve a formal interview, a mark reduction, a permanent record, or in severe cases suspension or expulsion. For international students whose visa status is tied to academic standing, even an unresolved misconduct allegation can have serious consequences. These are not minor inconveniences.

Who gets flagged most often

Research into AI detector accuracy consistently identifies the same groups as most likely to receive false positives. Non-native English speakers - including international students, EAL students, and heritage language speakers writing in their non-dominant language - are flagged at significantly higher rates than native speakers. Students who write carefully and formally, adopting academic conventions, produce text patterns that detectors associate with AI generation.

Students with certain cognitive profiles may also be affected. Those who write with unusual regularity or predictability - including some students with autism or other conditions that affect writing style - may produce text that reads as statistically AI-like without any AI involvement. The detector has no mechanism to account for this variation in human writing.

The research and its implications

A 2023 Stanford study tested seven commonly used AI detectors against essays written by non-native English speakers and found these tools flagged non-native writing as AI-generated at rates reaching 61%. The same texts written by native speakers were flagged far less often. Subsequent studies have replicated the finding across different tools, different language backgrounds, and different proficiency levels.

This isn't a fringe concern. In many UK schools and universities, international and EAL students form a substantial proportion of the student body. A detection method that systematically flags this group at disproportionate rates isn't a neutral tool applied equitably - it's a tool that creates discriminatory outcomes, regardless of intent. Several higher education institutions in the UK and US have suspended or discontinued text-based AI detectors as a direct response to this evidence.

The legal and institutional risk

In the UK, the Equality Act 2010 places a public sector equality duty on schools and universities requiring them not only to avoid active discrimination but to consider actively whether their policies and processes produce equitable outcomes across protected characteristics. A detection tool that produces disparate outcomes for international students - who may be protected under the characteristic of race - creates exactly the kind of indirect discrimination the duty is designed to prevent.

Beyond formal legal exposure, there are reputational and relational risks. Schools that have publicly accused students of AI misconduct and later found the evidence was weak have faced significant reputational damage. The practical risk of a high-profile false accusation is often greater than the risk of a missed detection.

Designing for fairness

The most effective way to reduce false positive risk is to change the detection method. Text-based analysis produces ESL false positives because it evaluates writing style, and writing style correlates with language background. Process-based detection doesn't evaluate writing style at all - it evaluates writing behaviour, which doesn't correlate with language background in the same way.

The second most effective intervention is procedural: treating any detection signal as the beginning of an enquiry, not the end of one. No automated tool - text-based or process-based - should be the sole basis for a formal misconduct finding. A conversation with the student that addresses their specific process, combined with documented evidence, is substantially fairer and more reliable than any tool used alone.