An AI detection tool is software that reads a piece of writing and gives a guess about whether a person or an AI wrote it. Schools, publishers, and employers use these tools because chatbots now write fluent text in seconds. The tools sound simple: paste text, get a score. In practice, the score is a statistical guess, and several universities and researchers say it is not strong enough to prove anything about one person’s work. This guide explains how detectors work, what research and schools say about their accuracy, who is most at risk of a false flag, and how to use a detector without hurting anyone. Vendors often claim very high accuracy, so we treat those numbers as claims, not facts, and we point out where the sources are dated or limited.
What an AI Detection Tool Actually Does
Most detectors are trained on large piles of text that are labeled as human-written or machine-written. The tool learns patterns that tend to show up more often in one group than the other. When you paste in a new passage, it compares your text to those patterns and returns a probability, such as “80 percent likely AI.”
That score does not show where the text came from. It shows how much the writing looks like the examples the tool learned from. Clear, simple, and well-organized writing can look “machine-like” to a tool, even when a person wrote it. And AI text that has been edited by a person can look “human-like.” So the score is a hint, not a fact.
Why the Problem Is Hard
AI models are built to write the kind of text people write. As models improve, the gap between human and machine text shrinks. A tool trained on last year’s models may struggle with this year’s models. Short texts are harder still, because there is less pattern to measure. Lists, code, and very formal writing can also confuse a detector.
This is why many experts say detection is a race that detectors cannot easily win. A new model appears, the detectors catch up, and then another model or a simple editing trick makes the work harder again.
What a University Teaching Page Says
The Brandeis University AI task force publishes a page on the limits of detection tools. It points to a December 2023 study in the International Journal for Educational Integrity by Weber-Wulff and colleagues, which tested 12 public tools and two commercial systems, Turnitin and PlagiarismCheck. According to the page, the study found the tools were neither accurate nor reliable, and that they leaned toward labeling text as human-written. It adds that changing or disguising the text made detection worse. You can read it on the Brandeis page about limits of AI detection tools.
The same page notes that Turnitin has claimed a false positive rate of about 1 percent, while testing by a Washington Post columnist found a higher rate and reported at least one innocent student flagged. It also cites a Common Sense survey, reported by Education Week in September 2024, in which about 10 percent of teens said their work had been wrongly labeled as AI-written. Those are secondhand summaries of other work, and the page is a university resource rather than a new study, so use them as pointers to the original sources.
Another Faculty Guide
A faculty resource from Chemeketa Community College says detectors produce both false positives, which flag human writing as AI, and false negatives, which miss AI writing. It describes one study in behavioral health writing in which a free detector flagged more than 27 percent of real academic text as AI, and a larger study of 14 tools that called them neither accurate nor reliable, with the best tool right only about half the time. It also says paraphrasing, translation apps, and small human edits can get AI text past detectors.
The page gives no dates for those studies in its body text, so we cannot say how current they are. It also says OpenAI shut down its own detection tool because of poor accuracy, but gives no date for that either. Treat the specific numbers as claims to check, while the general warning matches what other sources say.
A University That Turned the Tool Off
In June 2023, the University of Pittsburgh’s teaching center announced that it did not endorse any generative AI detection tools and had disabled Turnitin’s AI detection feature. Its reasons were the risk of false positives, the lack of clear information about error rates from some vendors, and the harm to trust and motivation if students are wrongly accused. The center noted that vendors often claim 98 to 99 percent accuracy, while outside testing found higher false positive rates. You can read the notice on the University of Pittsburgh’s page about AI detection.
That notice is more than three years old now, and detectors have changed since then. It still matters, because it shows a large university choosing caution over convenience, and it shows why a single score should never decide a person’s fate.
Who Is Most at Risk of a False Flag
Several sources raise the concern that some writers are flagged more often than others. The Brandeis page says detection tools can be biased against non-native English speakers, though it does not cite a specific study for that claim. It does cite the Education Week report that about 20 percent of Black teens in the survey said their work was falsely identified as AI-written, compared with 7 percent of white teens and 10 percent of Latino teens. The Chemeketa page says English language learners and neurodivergent students may be especially vulnerable.
These are survey results and general warnings, not proof of how every tool behaves. But the pattern is a strong reason for teachers and managers to be careful. A writer who uses simple sentences, learned English from textbooks, or writes in a very structured way can look “predictable” to a tool.
People Are Not Good Detectors Either
You might think a skilled reader can simply tell. The Brandeis page cites a 2024 survey of 135 faculty and students by Kumar and Mindzak, in which participants correctly identified AI-written text just over 24 percent of the time. That is a small survey, and it does not mean teachers cannot recognize a student’s voice over time. It does show that gut feelings are unreliable, too. The best evidence usually comes from knowing a person’s earlier work, talking with them, and looking at drafts.
Vendor Claims Versus Independent Tests
Almost every detector website advertises accuracy figures above 98 percent. These numbers usually come from the company’s own tests, on datasets the company chose. They may not match how the tool behaves on your students’ essays, your freelancers’ articles, or your company’s emails.
When you read an accuracy claim, ask a few questions. What kind of text was tested? How long were the samples? Which AI models were used? Were the texts edited or paraphrased? What was the false positive rate, not just the overall accuracy? Who ran the test? A good vendor answers these clearly. A weak one repeats a big number.
How to Use a Detector Fairly
If you decide to use a detector at all, treat it as one small clue. Never punish or fire someone based on a score alone. Ask for the draft history, notes, and sources. Have a calm conversation with the writer, and let them explain how they worked. Compare the writing with samples you know are theirs.
Be clear about your rules ahead of time. If AI help is allowed for brainstorming but not for final text, say so in writing. Students and employees cannot follow a policy they have not seen. Tell people which tools you may use and how scores will be handled.
Designing Work That Needs Less Policing
The Chemeketa page suggests designing assignments that are harder to outsource, such as in-class writing, oral presentations, personal reflections, and original research. Teachers can also ask for outlines and drafts at set steps, so the process is visible. Employers can set clear standards for review, sourcing, and disclosure.
When people understand why a task matters and how AI may be used, many will follow the rules without being watched. Detectors then become a rarely used last check, not the main way to keep trust.
Protecting Your Own Work From a Wrong Flag
If you are a student or a writer, you can protect yourself with simple habits. Write in a tool that saves version history, such as a word processor that tracks edits. Keep notes, outlines, and early drafts. Save links to the sources you used. If you used an AI tool for a permitted purpose, keep a note of how.
If you are accused, stay calm. Ask which tool was used and what score it gave. Share your drafts and history. Explain your process. Ask for a human review, and ask where to find the school or company policy. If the matter is serious, ask for help from an advisor, a union, or an ombudsperson.
Privacy When You Paste Text Into a Checker
Many online checkers take your text and send it to a company’s servers. Before you paste a student essay, a client report, or a private draft, read the privacy policy. Check whether the text is stored, shared, or used to train a model. Do not paste confidential or personal material into a tool you have not checked. For simple habits that help, read this guide on staying private online.
Also be careful with free tools you find through ads. Some are built to collect data, and a fake site can carry malware. Use sites you can identify, with a named company, clear terms, and a way to contact them.
Choosing a Tool If You Still Need One
If your school or company requires a detector, compare a few options before choosing. Look for a clear statement of the false positive rate, published testing, support for the languages you work in, and sentence-level highlighting, so a human can see why text was flagged. Check how the vendor handles your data and what you pay. Run a test with text you know is human-written, including writing by non-native speakers, and see what happens.
Remember that a tool that is good today may be weaker after a new AI model arrives. Plan to review your choice at least once a year.
What We Could Not Verify
We did not run any detector ourselves, and we did not read the original Weber-Wulff study, the Washington Post tests, or the Education Week report. We relied on how university pages summarize them. The Chemeketa page gave no dates for its studies. The Pittsburgh notice is from 2023, and Turnitin and other tools have been updated since then. We cannot say how any specific tool performs today.
Conclusion
An AI detection tool gives a probability, not proof. University pages and a published study reviewed by those pages say that detectors can miss AI text, flag human writing, and be fooled by simple edits, and survey data suggests some groups of students may be flagged more often. Vendors often advertise very high accuracy, but those numbers are usually claims from their own tests.
The fair way to use a detector is as one small clue among many. Look at drafts, talk with the writer, set clear rules, and design work that shows the process. Protect your own privacy when you use online checkers, and keep your drafts so you can show how you wrote.
Frequently Asked Questions
1. What is an AI detection tool?
It is software that estimates whether text was written by a person or an AI, usually by giving a probability score.
2. Are AI detectors accurate?
Sources we read say they are not reliable enough to prove anything alone. They can miss AI text and flag human writing.
3. Can a detector wrongly accuse me?
Yes. False positives are well documented, and some surveys suggest certain groups are flagged more often.
4. What should I do if my work is flagged?
Stay calm, ask which tool and score were used, share drafts and notes, and ask for a human review.
5. Should schools use detectors?
Some have turned them off. If used, a score should be one clue, never the only reason for a penalty.
