AI advice cut accuracy and doubled confidence

Access to AI advice made people far less likely to admit uncertainty, and much more likely to be wrong with confidence. Researchers from three French and Italian universities found that willingness to say “I don’t know” fell from 44% to 3% when participants could consult an AI model. Over the same shift, accuracy dropped from 27% to 9%, while confidence climbed from 30% to 76%.

“People became much worse, the accuracy was only one third, but they were twice as confident.”

The study was led by Capraro, with Chiara Marcoccia of École Normale Supérieure and Walter Quattrociocchi of Sapienza University of Rome. The team intentionally used questions that AI models typically get wrong, including visual details from films such as the color of a team’s uniform in Bend It Like Beckham.

They used Step 3.5 Flash, a model that was usually wrong on those questions, specifically to rule out a simple explanation that people were sensibly delegating to a reliable tool. In some cases, participants who would have answered correctly on their own asked the AI and switched to a wrong answer.

Recommended reading

AI advice cut accuracy 3x and doubled confidence

Offering monetary incentives improved results only slightly. Willingness to admit ignorance rose from 3% to 8%, and accuracy increased from 9% to 16% — still far below the no-AI baselines of 44% and 27%.

Wharton researchers earlier this year described a similar effect as “cognitive surrender,” finding that people accepted incorrect AI answers 80% of the time while reporting higher confidence than those working without AI. This new study pushes the point further: the problem is not only misplaced trust in wrong answers, but that simply having AI available appears to suppress the habit of recognizing what you do not know.

“For humans, the capacity to say 'I don’t know' is very important because it represents the recognition of the limits of our own knowledge.”

Capraro said he is especially worried about children, who are growing up with these systems before developing critical thinking skills. The concern comes as Google’s AI search overhaul replaces links with confident AI-generated summaries, and Common Sense Media this week called that design an “unacceptable risk” for students.

The pattern, as described in the report, is straightforward: AI products are built to answer, not to say “I don’t know.” The people using them may be learning the same behavior.