False Positive Rates in AI Detection Tools

A writer who discloses AI assistance honestly is more likely to be flagged for misconduct than one who runs a full AI draft through evasion software. That finding, from a University of Notre Dame study (AILS '26), is the clearest evidence that false positive rates in AI detection are not noise around an otherwise sound instrument. They are the predictable output of a mechanism that cannot tell statistical writing style apart from machine-generated text. Most institutions treat detectors as reliable enough when used carefully, directing staff to weigh a high score as one factor among others. That assumption depends on flagging tracking AI authorship, and the research says it tracks something else: surface linguistic features that have nothing to do with who, or what, wrote the sentence.

The Notre Dame findings make the scale of the failure concrete. Light AI-editing, the kind of touch-up that guideline-compliant AI assistance would produce, gets flagged at elevated rates. Unmodified peer-reviewed abstracts published between 2023 and 2025, written entirely by human researchers, still get flagged at meaningful rates, with non-STEM fields showing significantly higher rates than STEM fields. A detector that cannot clear human-written, peer-reviewed prose cannot be trusted to clear a student's lightly-edited paragraph either, which collapses the policy built on top of these two results. Worse, the same study found that honest AI-editing carries higher sanction risk than evasion-assisted rewriting. A detector built to catch rule violations ends up punishing the student who follows the rules and missing the one who doesn't. That inversion is the subject of everything that follows.

How KGW watermarking and token-statistics detectors produce their scores

Detectors do not read for meaning. They measure statistical properties of token distributions, and any piece of writing that produces similar token-level patterns will score similarly, regardless of whether a person or a model put it on the page. That single fact explains almost everything else in this article.

The dominant watermarking method, known as the KGW scheme, works by splitting a language model's vocabulary into a green list and a red list before text generation begins, then biasing the model's token selection toward the green list. Detection works by computing a z-score: the ratio of green-list tokens actually found in a passage, measured against what a preset partition would predict, checked against a threshold. The scheme has a documented structural weakness. When the actual green-token ratio in a piece of text drifts from the partition ratio the detector expects, z-score detection with a fixed threshold can fail outright, so a genuinely watermarked passage may register under one hash key and vanish under another.

Classifier-based detectors work differently but land on the same problem. Rather than checking for a planted watermark, they are trained to recognize the surface patterns typical of model output: high lexical density, formal connective structure, vocabulary drawn heavily from the Academic Word List. Those features also define polished academic prose, whoever wrote it. The Notre Dame study isolates two specific features driving elevated scores on human-written text: long-token density and Academic Word List density. Neither one marks AI authorship. Both mark formal register. If a detector's core signal is built from the vocabulary and sentence structure of careful, formal writing, then careful, formal writing will trip that signal, and no amount of threshold tuning changes what the detector is actually counting.

Why formal writing style reliably elevates detection scores

Academic and professional writing shares enough surface features with AI-generated text, dense vocabulary, structured syntax, a narrow set of connective phrases, that token-statistics detectors cannot reliably tell them apart. Formal writing style elevates detection scores at the center of detector performance, not just at its edges.

The Notre Dame study quantifies the problem directly: non-STEM peer-reviewed abstracts, written by credentialed researchers and already through peer review, with no AI involvement of any kind, score a 14.9% false positive rate under Pangram. Non-STEM rates come in significantly above STEM rates, and the study attributes that gap to the heavier concentration of Academic Word List vocabulary that characterizes humanities and social-science writing. A literature scholar's prose simply contains more of the vocabulary a classifier has learned to associate with machine output than a chemist's does, independent of either one touching a language model.

None of this points to bad engineering. The features these tools are built to detect are the same features that mark high-quality, carefully edited formal prose. There is no threshold that separates them, because the signal a detector is hunting for and the confound that trips it false sit in the exact same feature space. Tightening the threshold doesn't isolate authorship from register; it just moves the line within a space where both live together.

How professional editing affects human text detection

If formal style alone inflates detection scores, the next question follows naturally: what happens when a human draft gets professionally edited? Editing improves clarity and polish, and in doing so, it pushes a manuscript's surface features closer to the patterns many detectors were trained on. The editing itself doesn't make the writing any more AI-generated. It makes the writing more closely resemble what the detector is scanning for.

A study out of Chung-Ang University and Wordvice tested this directly, examining document pairs from a professional academic editing service spanning 2018 to 2025, original manuscripts and their edited counterparts, with content, topic, and authorship held constant throughout. The only variable that changed was linguistic revision. Detector scores moved anyway, and not consistently: some detectors pushed scores higher after the same edit that pushed other detectors' scores lower. That divergence confirms something important beyond the direction of any single score. Detector outputs respond to linguistic style independent of who wrote the text, and different detectors are not measuring the same underlying thing even when fed identical passages.

The practical consequence runs backward from what anyone would design on purpose. A writer who submits a grant proposal, a journal manuscript, or a piece of polished business content after professional editing is structurally more exposed to false flagging than a writer who submits a rougher, unedited draft. The incentive to polish writing, which every institution claims to want, sits in direct tension with the incentive to avoid a false accusation of AI use.

Why non-native English writing carries a disproportionate false-positive burden

The false-positive burden described so far does not land evenly. Non-native English writers face systematically higher false-positive rates than native speakers, and AI use has nothing to do with it. Their writing tends toward lower lexical diversity, simpler syntax, and more repetition, the same surface patterns classifiers have learned to associate with machine-generated text.

The most cited finding on this question comes from Liang et al., published in 2023 in Cell Press Patterns: a majority of TOEFL essays written by non-native English speakers were falsely flagged as AI-generated across seven separate detectors. The Chung-Ang University study places that finding inside the same token-feature framework developed for editing effects: lower lexical diversity and increased repetition in non-native prose resemble the statistical fingerprints detectors use to flag model output, whether or not a model ever touched the text.

Notre Dame's study adds a further layer specific to institutional policy. Non-native writers who use AI legitimately for translation, clarity, or academic polish, without handing over authorship of the ideas themselves, face elevated detector scores for doing what guidelines permit. Writers who use evasion software after generating a full AI draft face near-zero detection rates for doing what guidelines forbid. Non-native writers are more likely to be falsely flagged on writing that involved no AI at all, and more likely to be flagged again if they use AI assistance the way their institution's rules say they should. Meanwhile, the population most likely to be evading detection outright is precisely the population the detector fails to catch. This pattern is not confined to classrooms. Any professional setting where non-native English speakers produce formal documents, contracts, regulatory filings, business reports, runs the same detectors against the same disadvantage.

Multi-model pipelines and statistical signal dispersal

Everything up to this point describes human writing caught by mistake. A separate and more active problem concerns content that passes through more than one model. When a document is produced or assembled using outputs from multiple models, the statistical watermark any single model would have left behind gets diluted or canceled, and a single-model detector's score on that content stops meaning anything.

The mechanism is documented at the training level. When a student model is trained on outputs from two teacher models using the KGW scheme with complementary, opposing keys, watermark detection confidence drops sharply, because the two keys cancel each other's statistical signal. Enterprise AI writing workflows in 2025 and 2026 increasingly rely on multi-model routing, directing each part of a task, research synthesis, tone, structural revision, to whichever model handles it best through decision trees and dynamic routers. A single published article may pass through several different models before it reaches you. The premise a single-model watermark detector depends on, that one model generated the whole text, no longer holds for content built this way.

A detector is not measuring AI authorship in that case; it is measuring the statistical residue of a pipeline it was never designed to read. A low score on such content is a measurement of noise, not evidence of partial AI use.

Some organizations have begun treating multi-model content generation as an architectural fix for the structural fragility built into single-model watermarking. If you disperse token choices across several language models, each with its own green-list partition, you get a mixed token-distribution pattern that never concentrates into the single signature a KGW-style detector is built to find. Lettershred's architecture works on this principle: parallel model kernels, line-by-line judge selection among candidate outputs, and token dispersal across providers, which together produce content whose z-score falls below the KGW detection threshold by construction rather than through editing performed after the text exists. Lettershred's published benchmarks show the drop quantitatively, from an elevated z-score down into negative territory, with most tokens in a given piece rewritten through cross-provider dispersal. That distinction matters for the argument this article is making: a tool that swaps vocabulary or reshuffles sentence structure is still fighting a detector's signal inside the same feature space that produces false positives on formal human writing. A true multi-model pipeline, where each model contributes to the final token selection, reduces the statistical concentration a detector depends on without forcing the writing into a less formal, less precise register. The signature never forms, which is a different proposition than editing it away once it has.

Why threshold calibration does not solve the base-rate problem

The standard institutional fix for false-positive complaints is to tighten the threshold: require a higher score before treating a flag as actionable. That response sounds reasonable and fails for a reason that has nothing to do with how well any particular detector is built. When the base rate of actual AI use in a population is low, the mathematics of conditional probability guarantee that even a highly accurate detector will generate more false positives than true positives among everything it flags.

Consider a population where AI use is roughly five percent of submissions, a plausible figure for many academic or professional settings. Even a detector with strong accuracy and a low false positive rate in isolation will still produce a substantial share of false positives among its flagged cases, because the pool of honest submissions is so much larger than the pool of actual violations that even a small error rate applied to the larger pool outweighs the true positives drawn from the smaller one. The lower the base rate, the worse that ratio gets, independent of how good the detector's underlying statistics are.

Jisc's National Centre for AI, in a 2025 update, documented a decisive shift in sentiment away from automated AI detection in UK education, driven partly by what scale does to even a modest error rate. An institution running hundreds of thousands of assessments a year turns a "low" false positive rate into an operationally unmanageable caseload, and a source of real harm to the students caught in it. The objection that this is an open-source problem rather than an enterprise one does not hold up either. A University of Chicago Booth working paper (2025-116) found that RoBERTa, the open-source classifier baseline still used as a detection backbone in resource-constrained institutions, performs close to random guessing and misclassifies a large share of human text as AI-generated, making it unfit for any high-stakes decision regardless of how it is calibrated.

Vendor-reported accuracy compounds the problem. Vendor benchmarks typically test against clean, unedited AI output under controlled conditions. Independent benchmarks test against the population a detector actually encounters in the field, edited drafts, paraphrased passages, non-native English, and those benchmarks have found false positive rates ranging from 14 to 28%. Calibration can shift where a threshold sits inside that range. It cannot change the base rate of AI use in the population being screened, and it cannot separate the feature confound described earlier from genuine authorship signal. Responsible deployment requires modeling the base rate explicitly, not simply picking a stricter cutoff and calling the problem solved.

What a detection score can and cannot prove

A detection score measures statistical proximity to patterns typical of model output. It does not measure AI authorship, and the distance between those two things is wide enough that no score, by itself, should stand as proof of misconduct or compliance.

Notre Dame's researchers state this directly: detector scores should not serve as standalone misconduct evidence. That conclusion now has legal weight behind it. In Newby v. Adelphi University, decided by the New York Supreme Court in Nassau County in January 2026, a court annulled an academic-integrity finding that had rested on a perfect Turnitin AI score. The court's reasoning turned on process: the university had failed to follow its own procedures and had ignored the student's contradictory results from other detectors run on the same text. A single tool's number, treated as a verdict rather than a data point, did not survive judicial review.

What a detection score can do is narrower and still useful. It can serve as a triage signal that prompts a human to look closer at a document. It can serve as one input among several in a broader evidentiary picture, weighed alongside a writer's drafts, revision history, and context. Jisc's framing places it inside a wider approach that includes how assessments are designed in the first place and the judgment of the person reviewing the work, never as the final word on its own. Read that way, a score is a question raised, not an answer delivered; every mechanism described above, the KGW partition, the formal-register confound, editing effects, non-native writing patterns, multi-model dispersal, base-rate math, explains why that distinction has to hold.