Last month, an artificial intelligence bot (AI) that deals with the technical support of cursor, a promising tool for computer programmers, alerted several clients about a change in the company’s policy. He affirmed that they were no longer allowed to use cursor in more than one computer.
In Internet forums, customers expressed their discomfort. Some canceled their cursor accounts. And some were even more angry when they realized what had happened: the bot of AI had announced a change of policy that did not exist.
“We do not have that policy. Of course they can use cursor on several machines,” wrote the general and co -founder director of the company, Michael Tuelll, in a Reddit publication. “Unfortunately, this is an incorrect response of a first -line support support.”
More than two years after the arrival of the chatgpt, technology companies, office workers and consumers in general use AI bots for an increasingly wide range of tasks. But there is still no way to ensure that these systems produce precise information.
The newest and most powerful technologies – the so -called reasoning systems of companies such as OpenAi, Google and the emerging Chinese company Depseek – are generating more mistakes, no less. While their mathematical skills have improved considerably, their management has become more unstable. It is not entirely clear why.
Current AI bots are based on complex mathematical systems that learn their skills analyzing huge amounts of digital data. They do not decide, nor can they decide, what is true or what is a lie. Sometimes, they invent, a phenomenon that some IA researchers call hallucinations. In a test, the hallucination rates of the most recent AI systems reached 79 percent.
These systems use mathematical probabilities to guess the best response, not a strict set of rules defined by human engineers. That is why they make a certain number of mistakes. “Despite our best efforts, they will always hallucinate,” said Amr Awadallah, who was a Google executive and is now a general director of Vectara, an emerging company that creates AI tools for companies. “That will never disappear.”
For several years, this phenomenon has raised concern about the reliability of these systems. Although they are useful in some situations, such as writing end -of -course works, summarizing office documents and generating computer code, their errors can cause problems.
Artificial intelligence bots linked to search engines such as Google and Bing sometimes generate search results whose results are so wrong that they laugh. If you ask you about a good marathon on the west coast, you could suggest a career in Philadelphia. If they tell you the number of Illinois households, they may quote a source that does not include that information.
These hallucinations may not be a big problem for many people, but it is for those who use technology with judicial documents, medical information or confidential business data.
“You spend a lot of time trying to find out which answers are based on facts and which are not,” said Pratik Verma, co -founder and general director of Okahu, a company that helps companies overcome the problem of hallucinations. “Do not deal with these errors properly in essence eliminates the value of AI systems, which are supposed to automate tasks for you.”
Cursor and Tuelll did not respond to requests for comments.
For more than two years, companies such as Openai and Google constantly improved their AI systems and reduced the frequency of these errors. But with the use of new reasoning systems, errors are increasing. According to its own tests, the most recent version of the Alfai Openai systems in greater proportion than the previous system of the company.
The company discovered that O3, its most powerful system, hallucinated 33 percent of the times when executing its Personqa reference test, which consists of answering questions about public characters. This is more than double hallucinations than the previous OpenAI reasoning system, called O1. The new O4-mini hallucinated in an even greater percentage: 48 percent.
When executing another test called simpleqa, which asks more general questions, the hallucination rates of O3 and O4-mini were 51 and 79 percent. The previous system, O1, hallucinated 44 percent of the time.
In a document in which the evidence is detailed, OpenAi states that more research is needed to understand the cause of these results. Since AI systems learn from more data that people can understand, technologists strive to determine why they behave in the way they do.
Tests carried out by independent companies and researchers indicate that hallucination rates are also increasing in business reasoning models such as Google and Deepseek.
Since the end of 2023, Awadallah’s company, Vectara, has monitored the frequency with which chatbots deviate from the truth. The company asks these systems to perform a simple and easily verifiable task: summarize certain news articles. Even so, chatbots invent information persistently.
Vectara’s original research estimated that, in this situation, chatbots invented information at least 3 percent of the time and sometimes up to 27 percent.
In the year and a half after then, companies such as Openai and Google reduced those figures to 1 or 2 percent. Others, such as the emerging company of San Francisco Anthropic, approached 4 percent. But those hallucination percentages in this test have increased with reasoning systems. The Deepseek reasoning system, R1, hallucinated 14.3 percent of the time. OPENAI O3 rose to 6.8 percent.
(The New York Times sued Openai and his partner, Microsoft, for violating copyright in relation to the news content related to AI systems. OpenAi and Microsoft deny those accusations).
For years, companies like OpenAi were based on a simple concept: the more Internet data they introduced into their AI systems, the better they would work. But they only used the entire internet English text, which meant that they needed a new way to improve their chatbots.
Subscriber content
So these companies depend more than one technique that scientists call learning by reinforcement. With this process, a system can learn behaviors through essay and error. It works well in certain areas, such as mathematics and computer programming. But it is falling short in other areas.
“Based on how these systems are trained, they will begin to focus on a task and to forget about others,” said Laura Pérez-Beltrachini, a researcher at the University of Edinburgh that is part of a team that closely examines the problem of hallucinations.