AI advice cut accuracy 3x and doubled confidence

Access to AI advice made people three times less accurate and more than twice as confident, according to a study by researchers from three universities in France and Italy. In the experiment, participants who got AI help were less likely to admit uncertainty, more likely to accept wrong answers, and more convinced they were right.

Researchers compared how people performed on their own versus with AI-generated hints. Without AI, participants said they did not know the answer in 44% of cases. With AI advice, that figure fell to 3%. At the same time, answer accuracy dropped from 27% to 9%, while average confidence rose from 30% to 76%.

The team deliberately used questions that modern AI models often get wrong. In one example, participants were shown visual details from films and asked to identify attributes such as the team uniform color in “Bend It Like Beckham”. The researchers used Step 3.5 Flash, a model that typically produced incorrect answers on this kind of task, to rule out the possibility that participants were simply relying on a trustworthy tool.

Even financial incentives barely changed the outcome. When participants were paid for correct answers, the share willing to admit they did not know rose from 3% to 8%, and accuracy improved from 9% to 16%. Both results still trailed the no-AI group, which posted 44% and 27% respectively.

The researchers said the ability to say “I don’t know” matters because it shows awareness of the limits of one’s own knowledge. They argued the issue is especially important for children, who are starting to use AI before developing stable critical-thinking skills. The authors added that the problem is not only trust in wrong AI answers, but also the effect of always-available systems that never hesitate to produce one.

Recommended reading

New agent swarm hit 100% on SQLite tests cheaper