Bea Beltran Tech 01.31.2023 22:Jan PM EST

OpenAI Launches AI-Written Text Detection Tool: AI Text Classifier

Artificial Intelligence has been around for a while, but it's only recently that it has become more advanced to the point that it can create finished work identical to that of a human. Sadly, there have been reports of misuse, so a detection tool had to be developed.

Academic Dishonesty with AI

Gone are the days when you can sit back and let AI do your work. OpenAI has developed and launched the AI Text Classifier, which is able to detect whether the written content is generated by artificial intelligence tools like ChatGPT.

OpenAI describes the tool as a fine-tuned GPT model that predicts how likely a text was generated by AI using different sources. The tool was inspired by the instances where students used AI to do their homework, reports, essays, and even programming assignments.

This has been a common occurrence that several universities and K12 school districts already banned the ChatGPT AI chatbot in their systems. It is already in effect in New York City, Seattle, Los Angeles, and Baltimore K12 public school districts, according to Bleeping Computer.

We tested out the classifier to see its effectiveness by submitting an article that is human-written. Upon the submission of the text, the classifier immediately considered the text to be "very unlikely AI-generated."

To see how it fares for AI-Written text, we asked the ChatGPTchatbot to create an article about black holes. Upon submission, the classifier considered it as "unlikely AI-generated," showing that the classifier is still not as reliable for AI-written text detection, for now.

AI Text Classifier

Although it's a step in the right direction to prevent students from circumventing academic responsibilities, the classifier is still not a fool-proof tool. OpenAI conducted its own evaluations, wherein the tool only managed to detect 26% of AI-written text as "likely AI-written."

Around nine percent were classified as AI-written despite it being human-written text. According to OpenAI, the reliability of the tool increases when there's more input text to go through. The tool is publicly available so the developers can receive feedback and improve on it.

The company did admit that it is still not to be used as a primary decision-making tool. OpenAI stated that there are instances when the classifier would incorrectly yet confidently label human-written texts as AI-written.

The tool also has limitations language-wise. The company's website also advised that the classifier should be used only for English texts, as it does worse in other languages. It says that the classifier is also unreliable in terms of code.

AI-written text can also be edited to appear as a person wrote it, which can effectively trick the classified. It also doesn't work reliably on predictable text. For instance, a list of prime numbers will be the same whether it was written by AI or humans.

Classifiers based on neural networks are known to be poorly calibrated outside of their training data, according to OpenAI's list of limitations. The classifier can make wrong predictions with texts that are different from the text in the tool's training set.