Bottom line

• Deep learning’s core tools have partial but limited transfer to Theory of Mind (ToM) in artificial systems. Grounded language models and multimodal methods can infer others’ mental states in constrained settings and with targeted training (e.g., reinforcement learning or inverse planning), sometimes approaching adult performance on narrow higher-order ToM items. However, consistent adult-like competence on canonical false-belief tasks, deceptive/intent-inference in novel situations, and realistic social interaction remains elusive, and evaluation benchmarks are often contested for design and robustness. Thus, many V&L-era success factors contribute pieces to ToM, but not a complete solution on their own. LLMs have been reported to achieve adult human performance on higher-order theory-of-mind tasks in some evaluations^[2] whereas other work finds broad failures on bespoke false-belief tasks that are considered gold standards in developmental ToM^[7] with critiques that common ToM benchmarks are not built to test adaptation to new situations^[3] and earlier work concluded ToM performance in LLMs does not generalize beyond a few standard contexts^[5] position papers argue most ToM benchmarks are inadequate for LLM-scale testing^[8].

What deep models currently achieve on ToM

LLM evaluations show uneven progress along ToM dimensions. Some studies claim adult-like performance on higher-order reasoning (e.g., recursive belief about others’ beliefs and goals) in carefully curated tasks. A study reports that LLMs can perform tasks requiring higher-order ToM at levels approaching adult human accuracy^[2]. In contrast, controlled examinations of false-belief—widely used as a ToM litmus test—demonstrate frequent failures, suggesting gaps relative to human children and highlighting sensitivity to specific task designs. An evaluation of 11 LLMs on 40 bespoke false-belief tasks (a standard in developmental testing) found limited success^[7]. Cross-species comparison work further argues that most ToM benchmarks fail to test adaptation, producing inflated or unrepresentative performance readouts. A recent position/empirical critique argues that most ToM benchmarks are broken because they cannot test how LLMs adapt to new problems^[3] and shows that on opponent-preference modeling tasks, LLMs’ ToM often does not generalize beyond a few standard contexts^[6] in line with earlier conclusions^[5].

Recent broad comparisons emphasize both strengths and weaknesses. A Nature study compares performance across a comprehensive battery of ToM abilities in humans and multiple LLMs, underscoring where models and people diverge^[4]. Collectively, these findings imply that while scaling and supervision improve certain linguistic forms of ToM, models still struggle with counterfactual mental-state reasoning, unobserved beliefs, and tasks requiring adaptation.

Mechanistically-informed reviews converge on ToM being multi-procedural: it uses theory-laden inferences about latent beliefs, norms, and goals beyond literal textual entailment. Comparative analyses across abilities in humans and LLMs map different ToM competencies^[4] and show that opponent preference inference (a canonical ToM task) remains challenging^[10] surveys note both behavioral and representational ToM in LLMs and raise safety concerns as models improve^[1].

What from vision-and-language carries over to ToM?

Several ingredients used in V&L have positive portability:

1. Multimodal grounding and alignment. When visual and textual cues are co-integrated, models can infer goals, intents, and belief states from grounded scenes. A comprehensive multimodal video QA benchmark (ToM-VideoQA) evaluates ToM across seven abilities derived from the ATOMS taxonomy, explicitly tying vision-language understanding to mental-state inference^[11] and Video Theory of Mind (VToM) architectures show that integrating visual and textual modalities can enhance ToM inference on videos^[19] with evidence that LLMs and LMMs still lack robust ToM in general, while targeted methods can begin to help^[13]. This mirrors V&L progress where cross-modal alignment improved compositional reasoning and commonsense—suggesting that richer perceptual grounding can support latent-state inference about agents.

2. Interactive agent environments and reinforcement learning. Directed feedback loops and reward for planning/agent modeling can induce ToM-like behavior in smaller LLMs. Reinforcement learning has been used to “unlock” ToM reasoning in small LLMs (0.5B–7B), implying that training dynamics and feedback can make latent-state attribution more reliable^[24]. This line echoes V&L outcomes where RL/feedback improved compositional generalization and grounded instruction-following. Related game-theoretic approaches formalize maintaining beliefs over other agents’ models during action selection. In Hanabi, deep RL agents maintain beliefs over physical states and models of other agents using Bayesian updates to act cooperatively^[25].

3. Inverse planning/intent inference and social priors. Methods explicitly modeling intentional agents improve belief/disbelief reasoning by inverting observed actions under plausible utilities and beliefs. The MMToM-QA benchmark introduces Bayesian Inverse Planning (BIP) accelerated by LMs (BIP-ALM) to better evaluate and model ToM in multimodal settings^[14]. This leverages tools akin to trajectory prediction and goal inference that have shown success in V&L for animacy and affordances.

4. Benchmarks with task taxonomy and stress tests. New ToM benchmarks borrow from V&L evaluation practices (e.g., video QA, scene understanding) to test multiple ToM abilities, enabling targeted training and fairer comparisons. A multimodal ToM benchmark enumerates distinct ToM abilities (e.g., belief, desire, deception), anchoring progress tracking^[11] MuMA-ToM frames ToM as inferring goals and beliefs from video and text in embodied multi-agent interactions^[18].

Where ToM still breaks down relative to V&L successes

The same approaches that worked for perception-and-language do not yet deliver adult-level ToM robustness:

1. False-belief/hidden-state reasoning remains fragile. False-belief tests systematically expose LLM weaknesses, with limited accuracy across diverse prompts^[7]. Critiques also argue that many benchmarks are not constructed to test adaptation from seen to unseen situations, inflating performance and masking generalization limits^[3]. Even broad higher-order ToM claims are made on specialized test suites and may not generalize to adversarial or real-world social variability. Adult-like performance on higher-order ToM is reported in curated settings^[2] but the Nature comparison shows divergences across ToM abilities relative to humans^[4].

2. Generalization and external adaptation gaps. Models often rely on statistical patterns and shallow world knowledge rather than simulating changing beliefs. Earlier evidence concluded that LLM ToM does not generalize beyond a few standard contexts^[5] and opponent-preference modeling remains brittle despite increased parameter count^[6]. Benchmarks repeatedly show poor transfer to novel agents, objects, or contexts. Position analyses stress that current ToM benchmarks fail to measure necessary adaptation^[8].

3. Social-interactive complexity and hidden goals. Real ToM requires decoding unobserved beliefs, norms, and cooperation, which remain hard in video-based QA and multi-agent domains. LLMs and LMMs still lack robust ToM capacity on multimodal ToM tasks^[13] despite methods like Bayesian inverse planning showing promise. BIP-ALM improves evaluation and modeling but models still underperform ideal ToM^[14]. The need to reason about multiple agents’ goals and beliefs in dynamic scenes pushes beyond current multimodal models’ capability to integrate long-term latent states. MuMA-ToM evaluates multi-modal ToM across embodied multi-agent interactions, revealing current shortfall^[16] and related benchmarks likewise report limited ToM performance in current models^{[20, 17]}.

4. Benchmark brittleness and safety risks. As models get better at social language, new risks surface. Surveys of ToM in LLMs emphasize that behavioral and representational ToM advances bring safety concerns, as models may mimic harmful social reasoning^[1]. This argues for stress-testing ToM under distribution shift and adversarial settings, which current benchmarks often lack.

Evidence for emerging cross-over: multimodal ToM progress and mechanisms

Despite broad limits, there are concrete signals of transfer from V&L-era methods:

• Integrating visual and textual modalities helps models infer goals and beliefs from scenes. A Video Theory of Mind architecture that fuses video and language yields improved ToM inference on multimodal tasks^[19]. A comprehensive ToM-VideoQA benchmark operationalizes specific belief/desire/deception abilities for tracking progress^[11].

• Training via reinforcement and Bayesian inference about agents can unlock latent-state reasoning. RL-based training unlocked ToM in 0.5B–7B LLMs, indicating that feedback and planning loops can compensate for model size^[24]. Bayesian agent modeling in Hanabi formalizes belief updates over others’ mental states and improves cooperative play, suggesting transferable mechanisms: maintain belief distributions over latent states, and invert observed behavior under utilities and beliefs.

• Newer benchmarks bridge the gap by testing multiple abilities and multimodal inputs, yielding more realistic progress vectors. A multimodal ToM benchmark frames ToM as essential for socially intelligent multimodal agents^[12], and methods like Bayesian inverse planning accelerate learning. BIP-ALM demonstrates that inverse planning plus LM priors improves ToM inference on video-based tasks^[14].

These results are consistent with the broader V&L story: compositional grounding, planning under uncertainty, and interactional feedback improved reasoning in perception-language, and similar levers are now moving ToM forward, albeit in constrained, often video-grounded, multi-agent settings. Multimodal fusion for ToM mirrors cross-modal alignment that boosted V&L reasoning^[19] and RL-driven induction of ToM complements iterative feedback loops that improved compositional generalization^[24].

Comparative perspective: what does and doesn’t transfer from V&L

Direct transfer is strong for tasks that can be made concrete and verifiable: when goals are visible or inferable from actions and scene affordances, models can approximate belief and desire. Video-based ToM architectures and datasets show this concrete path^[19]. Transfer is weaker for abstract or adversarial ToM, where mental states are hidden, counterfactual, or socially contingent. False-belief tasks expose failures in these abstract cases^[7]. Deep learning excels when it can (a) ground in sensory channels, (b) get feedback in simulation or the real world, and (c) operate under structured latent-variable models; ToM benefits when benchmarks provide measurable subskills and training loops target belief updates. Taxonomy-driven multimodal ToM benchmarks give these subskills^[11] and RL/inverse planning give targeted learning loops^[24].

Outlook and practical guidance

• Success factors that likely carry over: multimodal grounding; action-grounded simulation and feedback loops; explicit models of agents as planners (inverse planning, Bayesian belief updates); targeted training on subskills (goal inference, belief revision, cooperation) via RL or reward; and benchmarking across multiple ToM abilities rather than single-shot accuracy. Comparative evaluations highlight subskill granularity important for progress^[4] and multimodal ToM architectures show gains when vision and language are jointly modeled^[19] with RL improving ToM even in small models^[24].

• Known limitations to expect: brittleness on hidden/unobserved beliefs and counterfactual reasoning; poor adaptation to out-of-distribution social contexts; and benchmark design gaps that can overestimate ToM. False-belief failures remain common^[7] and benchmark critiques emphasize adaptation testing^[3] with evidence of non-generalization beyond narrow contexts^[5].

• Research directions: build embodied, interactive ToM environments with verifiable latent beliefs; scale inverse-planning methods and selective RL for belief updates; develop multimodal benchmarks that stress adaptation and deception; and audit for safety/harmful social reasoning. RL-unlocked ToM in small LLMs suggests scalable training can help^[24] but comprehensive multimodal ToM benchmarks are needed to measure belief/desire/deception reliably^[11] and surveys warn of safety risks that warrant proactive auditing^[1].

Answering the question

Do deep learning’s V&L successes carry over to modeling others’ mental states? Partially yes—with clear gains when models are grounded in multimodal scenes and trained with feedback loops and inverse-planning priors—sometimes achieving adult-like performance on narrow higher-order ToM items and markedly improving belief/goal inference in video-grounded tasks. Adult-level performance on higher-order ToM has been observed in curated tasks^[2] and VToM-style models show that fusing vision and language improves ToM inference^[19] with RL improving ToM even in small LLMs^[24]. However, consistent adult-like competence on false-belief reasoning, deceptive or hidden-state inference, and real-world social interaction has not been achieved; benchmarking remains contentious and often underestimates out-of-distribution failure. False-belief tasks show persistent weaknesses^[7] and critiques argue that current ToM benchmarks fail to test adaptation^[3] with evidence of non-generalization beyond narrow contexts^[5]. Thus, V&L-era tools provide necessary but not sufficient ingredients; ToM progress will likely require stronger social priors, embodied interaction, and rigorous, adaptive evaluation to move beyond today’s partial successes. New multimodal ToM benchmarks operationalize this progress^[11] and inverse-planning methods begin to address latent belief inference^[14] but safety and generalization concerns remain central design constraints^[1].