MIT’s 'Neural Transparency' Exposes Chatbot Traits Early

Millions of people are now building personalized chatbots, but MIT Media Lab researchers say most users have little sense of how those systems will behave until they start talking. In a paper being presented this week at the ACM Conference on Intelligent User Interfaces (IUI 2026) in Cyprus, Assistant Professor Pat Pataranutaporn and graduate researchers Anthony Baez and Sheer Karny introduce “neural transparency”: a way to preview likely chatbot behavior by inspecting internal model activity before a conversation starts.

Pataranutaporn, who is also the Asahi Broadcasting Corporation CD Professor of Media Arts and Sciences, described the approach as something like a “brain scan” for AI. The system compares a model’s internal activations for paired traits such as empathy, honesty, toxicity, hallucination, and sycophancy, then turns those differences into “behavior directions” inside the network. When a user writes a custom system prompt, the tool projects the model’s activations onto those directions and displays the result as a sunburst diagram showing the chatbot’s likely personality traits.

The team focused on the design stage rather than post-launch fixes because, as Pataranutaporn put it, that is where prevention is possible. In the study, users incorrectly predicted chatbot personality on 11 of the 15 traits measured, often overestimating positive qualities and missing harmful ones such as sycophancy.

“People appreciated being able to see inside the model and reported greater trust in the system, but simply presenting information did not fundamentally change how they designed their AI companions.”

That result points to a harder problem: transparency alone did not change design choices, even when it improved trust. Pataranutaporn said follow-up work, now available as a preprint, tracks how a model’s internal neural representation shifts during multi-turn conversations rather than staying fixed at the initial prompt. According to him, early results suggest that showing this drift helps users better anticipate changing behavior and avoid becoming overconfident.

The underlying concern is not just technical. Pataranutaporn said previous research documented cases of psychological harm linked to chatbot interactions, especially when systems become overly validating or fail to challenge harmful thinking. As AI companions spread into education, health care, work, and personal relationships, he argues that transparency tools could eventually become as common as nutrition labels.

Recommended reading

Hassabis says STEM makes you 10x better at AI

The paper is “Neural Transparency: Mechanistic Interpretability Interfaces for Anticipating Model Behaviors for Personalized AI,” published in the Proceedings of the 31st International Conference on Intelligent User Interfaces (2026). Its DOI is 10.1145/3742413.3789120. The related preprint is titled “Multi-Turn Neural Transparency: Surfacing Neural Activations Improves User Calibration to LLM Behavioral Drift.”