AI Advice Cut Accuracy and Doubled Confidence

Access to AI advice made people dramatically less willing to admit they did not know an answer, according to researchers from universities in France and Italy. The same study found that participants became less accurate and more confident when they could consult a chatbot, even when the bot was wrong.

Valerio Capraro, an associate professor at the University of Milano-Bicocca, said the work focused on a basic but important human response: recognizing the limits of one’s own knowledge.

“For humans, the capacity to say, 'I don’t know,' is very important because it represents the recognition of the limits of our own knowledge. But now with AI, we can get an easy answer to virtually every question, so we wondered whether this would interfere with human capacity to say, 'I don’t know,' to suspend judgment.”

Capraro and co-authors Chiara Marcoccia of École Normale Supérieure and Walter Quattrociocchi of Sapienza University of Rome tested how people responded to questions about visual details in films, including the color of a team’s uniform in Bend It Like Beckham and the vehicle Monica drives in Like a Cat on a Highway.

Recommended reading

AI advice cut accuracy and doubled confidence

They deliberately used Step 3.5 Flash because, as the paper explains, it was usually wrong on these questions. The idea was to rule out the possibility that participants were simply delegating sensibly to a reliable tool. The researchers also checked newer models — GPT-5.5, Claude Sonnet 4.6, and Gemini 3.5 Flash — which still missed the vehicle question but often got other details right.

The results were stark:

• Without AI advice, 44 percent of participants said they did not know the answer.
• With AI advice, that fell to 3 percent.
• Without AI advice, 27 percent answered correctly.
• With AI advice, only 9 percent answered correctly.
• Confidence rose from 30 percent in the baseline group to 76 percent with AI help.

“So basically people became much worse – the accuracy was only one third – but they were twice as confident.”

Adding monetary incentives improved performance somewhat, but not enough to recover the baseline. Willingness to suspend judgment rose from 3 percent to 8 percent, and accuracy increased from 9 percent to 16 percent. Both remained below the no-AI baseline of 44 percent and 27 percent respectively.

The paper’s title sums up the finding: “AI advice suppresses people’s willingness to say 'I don’t know', even when the advice is wrong and accuracy is incentivized.” Capraro said he believes the problem should be addressed through AI literacy and education policy, especially for children growing up with these systems.