Chasing shadows: Is AI text detection a critical need or a fool's errand?

Before I start on today's article, let me give you a quick reminder of my last one. Have you entered your observation into the Globe At Night application or web site?

The current campaign ends in just 4 days, on April 9

. For me, we've had cloudy skies all week, so I plan to make an observation this weekend. It looks like skies should be clear tonight and tomorrow.

Now, on to today's topic:

---

Summary

There is an interesting article in this month's Communications of the ACM, The Science of Detecting LLM-Generated Text. The article is a fascinating review of the state of the art with LLM text detection, but I can't help but wonder if the whole idea that we can ever detect AI generated text reliably is really just an exercise in tilting at windmills?

In the end, maybe the problem comes down to the simple observation that people lie sometimes. In all of human history, it has been difficult to determine when people are being honest or lying. I'm not sure if technology can change that.

Although my tone here may sound skeptical about the future of reliable AI text detection, I definitely recommend reading the article and watching the video that was posted with it:

Note that for brevity in this article, I will use the acronyms "AI" and "LLM" interchangeably. In the real world, that's not exactly right.

Introduction

When I was in elementary school, the calculator was just seeing widespread consumer adoption. The calculator of the day could only do addition, subtraction, multiplication and division, but it was already a threat to the education system. As far as I remember, there was absolutely no controversy at the time. Children should memorize their multiplication tables (up to 12x12), and arithmetic should be done with pencil and paper - even long division. Using a calculator (or a slide rule) was a form of cheating.

As I got older, these ideas started to be controversial. Why should children do tasks that a calculator could do for them? And how could the teacher even know? If a child brings their arithmetic homework in with the correct answers, how could the teacher know whether those answers were produced from the child's mind or by a calculator? Maybe a child's time would be better spent learning other things that couldn't be automated with a calculator?

By the time that @cmp2020 was in elementary school, the controversy was over and the calculator had won. I guess the kids at some schools might still memorize the multiplication tables, but the calculator is now their companion throughout the entire education experience (many of them also don't know how to tell time on an analog clock, but that's a topic for a different day. 😉)

In some ways, I'm starting to feel that the path to detecting AI generated text might follow the same course. Maybe what matters isn't whether the text was composed by a person or by a machine. Maybe what matters is the message that it conveys. Does it convey interesting, accurate, and useful information? Maybe the real "intelligence" isn't in writing the words, but in choosing the topics and deciding which words, phrases, and paragraphs to include or exclude.

Against that backdrop, here's some of what I learned from the article followed by some additional discussion.

AI detection is desired in order to minimize risk and fulfill AI's potential

In the article, Ruixiang (Ryan) Tang, Yu-Neng (Allen) Chuang, and Xia (Ben) Hu argue that the ability to detect LLM generated text is a critical need because AI can only reach its full potential if detection tools are able to minimize the risks from things like, "phishing, disinformation, and academic dishonesty". To date, the authors suggest that AI's potential has been hampered because of these risks.

Accordingly, the authors describe their goal for the article, as follows:

We aim to unleash the potential of powerful LLMs by providing fundamental concepts, algorithms, and case studies for detecting LLM-generated text.

An overview of detection types

The authors provide a taxonomy of AI detection techniques that starts with two branches at the top. These are black box and white box. A black box technique is one where the detector only has access to the output from an LLM. A white box technique is one where the detector is able to affect the text that the LLM produces.

The tools that Steem curators are probably familiar with would be "black box". These would operate by checking the frequencies of words, phrases, and sentences in an article and comparing those frequencies against texts that are generated by humans and also against texts that are generated by LLMs. Based on how close the comparisons are, the detector gives back some level of confidence about whether the text was created by man or machine.

On the other hand, white box techniques operate by intentionally creating the text in a way that can be recognized by a detector. Simplistically, this can be thought of as putting a coded message into the text that can only be read by the detector, and says, "this document was created by an AI."

You can click through to the article to learn about each of these techniques in more detail.

In order to determine the effectiveness of these tools, researchers use a variety of benchmarking datasets which have been collected and published. These include the following:

Challenges to success

One of the key benchmarking challenges arises from the speed of development in the AI sector. Despite the ongoing work on benchmarking, detection researchers are challenged to keep up with the pace of LLM advancement, and there are no comprehensive benchmarks datasets that can keep up with the wide array of LLMs that have been created.

Another challenge stems from adversarial attacks. An example comes from Can AI-Generated Text be Reliably Detected? by Sajith V Sadasivan and colleagues. As described, this technique reduced white box detector accuracy from 97% to 80% and black box detection from 100% to 80%. In this attack, LLM output from one model is sent to a different LLM model for paraphrasing prior to being delivered to the detector.

The authors also raise other concerns. For black box detectors, the authors point out that the tools are vulnerable to bias problems in the training data. For white box detectors, the authors observe that there is a general tradeoff between the quality of the text and the watermark's efficacy. Further, white box models are susceptible to being reverse engineered by a determined prompt engineer.

Additionally, the authors suggest that the research into assessing confidence levels is sparse. We would like to be able to say that a text with a 95% confidence score of AI generation is more likely to have been created by an LLM than another text with a 50% score. Unfortunately, the research into these confidence scores is thin. Further, the methods for determining accuracy are subject to false positives and false negatives, and they may be biased against certain populations, such as non-native users of the language.

Finally, the explosion of open source LLMs makes the task of detection research all the more difficult. The authors even suggest that the explosive growth of Open Source LLMs will eventually render black box techniques ineffective.

Discussion

In all, this video and article gave a fascinating overview into the state of the art on detection of AI-generated text. The authors definitely argue for a popular position when they say that the ability to detect AI-generated text is critical.

Certainly, I agree with them that we need to minimize risks like "phishing, disinformation, and academic dishonesty". But these problems are not new with the arrival of AI. I am starting to believe that reliably detecting AI generated text will eventually be equivalent to asking whether an arithmetic solution was "created" by a calculator.

LLMs are getting better and better, and it's trivial to imagine future adversarial attacks (instead of using one LLM to paraphrase a text, use two, or three, and so on; interweave sentences from two separate LLMs; intentionally insert typos/spelling errors, etc...) In the end, I think maybe we're just going to have to understand that there's a human owner of the text, and that person is ultimately responsible for what was said - regardless of whether an LLM was used as an intermediary. Then, the risks of "phishing, disinformation, and academic dishonesty" would be addressed by ethics and laws, not by technology - as they have always been.

In closing, as a firm believer in the maxim that any problem can be made worse by adding government, the last seconds of the video illuminate my biggest concern with the technologies:

However, the challenge is that how could we design these stealthy and robust watermarks into the LLM generated text, and how could we force all these big companies to add these watermarks into their products? So that will be a very challenging question. So definitely we need to collaborate with governments to have policies and laws to further regulate the use of these powerful generative AIs.

---

Thank you for your time and attention.

As a general rule, I up-vote comments that demonstrate "proof of reading".

---

Steve Palmer is an IT professional with three decades of professional experience in data communications and information systems. He holds a bachelor's degree in mathematics, a master's degree in computer science, and a master's degree in information systems and technology management. He has been awarded 3 US patents.

---

^{Pixabay license, source}

Reminder

Visit the /promoted page and #burnsteem25 to support the inflation-fighters who are helping to enable decentralized regulation of Steem token supply growth.