Overview & How AI Detectors Work

AI detection tools are designed to help an individual identify AI produced writing. There are many tools out there, even some embedded into products we already have, like turnitin. For UNO faculty, the turnitin tool has been intentionally disabled (consistent with many other universities like Vanderbilt) and we do not recommend using any AI detectors to check student work. If you choose to use them, we recommend you proceed with caution and do not use them as the end-all-be-all in proving academic dishonesty. In order to understand our stance on AI detectors, it is important for us to understand how they work and where their shortcomings are. On this page, you'll learn more how detection tools work, potential issues with detection tools and what to do instead.

AI detection tools in themselves are artificial intelligence. They are trained to identify specific patterns and attributes associated with AI writing including word choice and repetitive sentence structure, among other items. When they analyze written work, most detection tools give a score in relation to a certain percentage, for example 73% human-generated means that there is a 73% chance it was written by a human. One may ask, "well, what does this mean?" and rightfully so. Our acceptance levels of AI writing are different, and there is a lack of clarity when it comes to what is an acceptable percentage in AI detection.

While these tools are great in theory, they tend to have poor execution. It is very possible that with any detection tool, you can get a false positive or a false negative where the detector incorrectly identifies the writing.

Side by side comparison of 10 AI detector tools from forbes article

The (Un)reliability of Detection Tools

Unfortunately, detection tools are not a reliable way to find out if your students are using AI on their assignments. In the image to the right, you'll see a table listing the accuracy and rating of different AI tools on a 5-point scale. This image comes from the Forbes Article, The 10 best AI Content Detector Tools, where they discuss the pros and cons of each detection tool, giving you more insight into the potential issues with them.

So how reliable are detection tools? When looking at the image to the right, you can see that 60% of the tools mentioned in the table to the right received a 70% or less on their accuracy of identifying AI writing work, with one as low as 38%. Only ONE detection tool scored a 90%; so while Undetectable has a 90% accuracy rating, that still means that for every 9 you identify accurately, you're misidentifying 1 student's work as AI written. There is no detection tool with 100% accuracy in identifying AI writing.

Misidentifying student work as AI written can have serious consequences for students and instructors (more on that below).

The Bias of Detection Tools

While detection tools have been more accurate with individuals where English is their first language, it has been grossly misidentifying the writing of non-native English writers as AI-generated. According to research by Stanford University, AI detectors misclassified over 60% of TOEFL (Test of English as a Foreign Language) essays written by non-native English Students.

"According to the study, all seven AI detectors unanimously identified 18 of the 91 TOEFL student essays (19%) as AI-generated and a remarkable 89 of the 91 TOEFL essays (97%) were flagged by at least one of the detectors"

These numbers are staggering and bring up valid concerns about AI detectors and their reliability. By falsely accusing students, specifically vulnerable and foreign exchange students, we could harm their educational futures, many of who have strict regulations and rules they have to follow to continue their education here.

Trying out the Undetectable AI Detector

Since it was ranked #1 on Forbes list of AI detectors, I wanted to try Undetectable AI out for myself and see how good it is. My first impression when going to the website, was that this detection tool is more geared towards students and people using AI rather than educators to check to see if students are using AI on assignments; this was evident by the ability to either check for AI and to humanize your writing and even more evident from the wording used at the top of the page:

Rated #1 Best AI Detector by Forbes! Use our AI detector tool to check if your text will be detected as AI-written content. Our AI text humanizer enhances your content to match the quality of human writing. Clicking “humanize” removes AI detection and ensures your text will bypass AI detectors.

What I Did & Results

To test it out, I took an actual student's writing I was skeptical of and pasted it into the detector. I asked it to check for AI and was surprised to see that it actually told me the likelihood of AI being detected by 8 different detection tools: GPTZero, OPENAI, Writer, CrossPlag, Copyleaks, Sapling, Content at scale, and ZeroGPT. Ultimately it alerted me that six of them were flagged as 0% human-generated, two of them were flagged as 50% human-generated, and zero being flagged as 100% human-generated.

undetectable's results using their AI detector

If I took the detection results as fact, I would accuse this student of using AI to write their assignment and give them a zero on it, but how accurate are these results and do I have solid proof that this student used AI to write their work? I set out to explore my options.

Comparing Undetectable with other AI detector tools

I wanted to see how accurate the results were compared to other AI Detection tools. I chose to use one that Undetectable AI showed me the likelihood of, Writer AI detector. In the 3-minute video below, I show the results of both Undetectable AI content detector and compare it to the results from Writer AI. At the end of this video, you'll see the lack of consistency of AI detectors and their inaccuracy.

It is important to note that a portion of Undetectable AI's business, and how they make money, is through the "humanize my writing" portion of their website. Could this company be falsely identifying work as AI written to increase sales and have users pay for the humanizing services? ABSOLUTELY. We would be naive to think otherwise.

Can Students Trick AI detectors

The answer is yes. As you saw in the undetectable tool above, students can use different tools to humanize the AI writing to "beat the system." In addition, AI detectors can easily be tricked by paraphrasing or switching out a couple words; so although it may sound odd to you, it isn't detected by the detection tool. There are many tools out there, like quillbot, that can help students easily change their writing to "trick" the AI detector.

Let me share an example:

In my online Introduction to Sociology course, I have students complete weekly mini-reflections over the topic of the week: this mini reflection has two questions: 1) what are you understanding best about the topic? Explain in your own words, and 2) what is confusing for you? What questions do you have over the topic?

This assignment as a way to make sure they are understanding content and an easy way for them to ask questions when they don't understand something. The topic for this specific assignment was socialization. Here is the student submission:

Learning more about society has broadened my perspective, unraveling the intricate layers that shape human interactions. It's like discovering a hidden complexity beneath the surface, challenging preconceptions, and fostering a deeper understanding of the diverse forces that influence our collective existence. People's behaviors and mannerisms in society are shaped as a result of so many other factors like upbringing and cultural differences.

When reading this, I wholeheartedly believed this was written by AI; it 1) doesn't answer what is asked of them and 2) uses a lot of big words that are unnecessary in this informal assignment. But what would a detector show? I tried using Undetectable AI on two separate occasions with this prompt; ironically, I got two different responses

Attempt 1: Passes All AI detectors as 100% human-generated

Attempt 2: Passes 6/8 as 100% human-generated, with 2 being flagged at 50% human generated.

What to do Instead

In some regards, you're fighting a losing battle when it comes to AI detection. With students easily being able to "cheat a detector" and AI detectors not having the ability to provide us with solid proof that student writing is or is not AI generated, there isn't concrete evidence to back up your claims against student work. The risks of using AI tools, simply outweighs the reward. If even one student is falsely accused of using AI, it can have severe repercussions for the student, and for the instructor; increased headaches and stress trying to prove something that may be "unprovable" with tools and with the university).

As an individual, you'll need to decide what your stance on AI is and create a policy that works for you. Depending on the assignment, it is highly likely that they will not complete the assignment accordingly to the criteria asked of them (as seen in the example above).

So what should you do instead?

Compare this student submission to previous student writing.
Grade the assignment according to the rubric and criteria, focusing more on if they completed the task correctly.
Have a conversation with the student; use this as a learning opportunity for the student.
Evaluate your assessment and consider changes/implementing AI into it.

Just as Generative AI bots are advancing and changing, so are detection tools. It is very possible that in the future, detection tools will be more reliable and a beneficial way to check for academic dishonesty, but for now incorporating different AI-resistant or AI-inclusive assessments and constantly conversing with students is the most effective method for encouraging academic integrity.