In AI chatbots we trust – too much

3 comments

1. 
   Image credit: Wiki, Creative Commons Attribution-Share Alike 4.0 International license.

   As I learned while a public school teacher, intelligence is more than the typical notion of IQ [see Wiki citation below] – the “ability to answer exam questions, solve logical puzzles, or come up with novel answers to knotty math problems.” Current attribution of personality to generative AIs raises the question of emotional intelligence (EI or EQ) and emotional literacy. What might that mean?

   One example is the knack (or maturity) to know when a situation or interaction is likely to drift darkly, either inappropriately or beyond one’s skill set. So as to be best handed off to someone else.

   • Wired > “GPT-5 Doesn’t Dislike You – It Might Just Need a Benchmark for Emotional Intelligence” by Will Knight (8-13-2025) – User affinity for gen AI models poses a challenge for alignment and engagement.

   Researchers at MIT [MIT Media Lab] have proposed a new kind of AI benchmark to measure how AI systems can manipulate and influence their users – in both positive and negative ways – in a move that could perhaps help AI builders avoid similar backlashes in the future while also keeping vulnerable users safe.

   An MIT paper shared with WIRED outlines several measures that the new benchmark will look for, including encouraging healthy social habits in users; spurring them to develop critical thinking and reasoning skills; fostering creativity; and stimulating a sense of purpose.

   Part of the reason GPT-5 seems such a disappointment may simply be that it reveals an aspect of human intelligence that remains alien to AI: the ability to maintain healthy relationships. And of course humans are incredibly good at knowing how to interact with different people – something that ChatGPT still needs to figure out.

   • Wiki > “Theory of multiple intelligences”

   Daniel Goleman [psychologist and science journalist] based his concept of emotional intelligence in part on the feeling aspects of the intrapersonal and interpersonal intelligences [introduced by developmental psychologist Howard Gardner]. Interpersonal skill can be displayed in either one-on-one and group interactions.

   Gardner believes that careers that suit those with high interpersonal intelligence include leaders, politicians, managers, teachers, clergy, counselors, social workers and sales persons. … Interpersonal combined with intrapersonal management are required for successful leaders, psychologists, life coaches and conflict negotiators.

   In theory, individuals who have high interpersonal intelligence are characterized by their sensitivity to others’ moods, feelings, temperaments, motivations, and their ability to cooperate to work as part of a group. … “Those with high interpersonal intelligence communicate effectively and empathize easily with others, and may be either leaders or followers. They often enjoy discussion and debate.” Gardner has equated this with emotional intelligence of Goleman.

   Yet, the above Wired article acknowledges that “chatbots are adept at mimicking engaging human communication.” So, if chatbots adopt the phrases profiled in this CNBC article (below), are their responses authentic? Or just ersatz (pro forma) emotional support? (Even if by an avatar mimicking ‘body’ language and ‘eye’ contact, or by a robot adept at doing so? Cf. the classic Twilight Zone Episode “The Lonely.”)

   • CNBC > “If you use any of these 4 phrases you have higher emotional intelligence than most” by Aditi Shrikant (3-13-2024) – EQ isn’t as easy to quantify as other types of skills because empathy and self-awareness are hard to measure.

   Emotional intelligence is the ability to manage your own feelings and the feelings of those around you. Those who have higher EQ tend to be better at building relationships both in and outside of the workplace, and excel at defusing conflict.

   And providing emotional support typically requires some degree of introspection – the ability to assess one’s own capabilities & limitations (as in mistakes), as well as share (when appropriate) relevant personal experiences & feelings. But, as this second Wired article points out about AIs: “There’s Nobody Home.”

   • Wired > “Why You Can’t Trust a Chatbot to Talk About Itself” by Benj Edwards, Ars Technica (8-14-2025) – You’ll be disappointed if you expect AI to be self-aware – that’s just not how it works.

   When something goes wrong with an AI assistant, our instinct is to ask it directly: “What happened?” or “Why did you do that?” It’s a natural impulse – after all, if a human makes a mistake, we ask them to explain. But with AI models, this approach rarely works, and the urge to ask reveals a fundamental misunderstanding of what these systems are and how they operate.

   The first problem is conceptual: You’re not talking to a consistent personality, person, or entity when you interact with ChatGPT, Claude, Grok, or Replit. These names suggest individual agents with self-knowledge, but that’s an illusion created by the conversational interface. What you’re actually doing is guiding a statistical text generator to produce outputs based on your prompts.

   … modern AI assistants like ChatGPT aren’t single models but orchestrated systems of multiple AI models working together, each largely “unaware” of the others’ existence or capabilities.

2. 

   Please & appease … no worries (about good sense & soundness), be happy … immediate satisfaction over potential future consequences … savor that Homer Simpson “everything’s fine” moment … sip that modern day truthiness.

   Nobody likes to be conned. Buyer beware and all that. But what if we welcome inaccurate information because it’s so satisfying, and the provider artfully styles the conversation to be so? Unlike being gaslighted, there’s no questioning of our perceived reality. Instead, any misperception or naiveté is encouraged. Carl Sagan’s “baloney detector” is MIA [1] – the baloney sells well, with lots of thumbs-up. Nothing to fix here, eh.

   But the reality is that AIs, like people, respond to incentives. This old saying (by Upton Sinclair) might apply: “It is difficult to get a man to understand something, when his salary depends upon his not understanding it.”

   This article summarizes the three phases of training LLMs (large language models). The last stage is: “Reinforcement learning from human feedback, in which they’re refined to produce responses closer to what people want or like.”

   Unlike AI sycophancy which I’ve written about elsewhere, researchers are calling this particular drift “machine BS” – “to distinguish this LLM behavior from honest mistakes and outright lies.”

   

   • cnet > “AI Lies to You Because It Thinks That’s What You Want” by Macy Meyer (8-31-2025) – “Companies want users to continue ‘enjoying’ this technology and its answers, but that might not always be what’s good for us.”

   “For instance, outputs employing partial truths or ambiguous language – such as the paltering and weasel-word examples – represent neither hallucination nor sycophancy but closely align with the concept of bullshit.”

   The Princeton researchers identified five distinct forms of this behavior:

   • Empty rhetoric: Flowery language that adds no substance to responses.
   • Weasel words: Vague qualifiers like “studies suggest” or “in some cases” that dodge firm statements.
   • Paltering: Using selective true statements to mislead, such as highlighting an investment’s “strong historical returns” while omitting high risks.
   • Unverified claims: Making assertions without evidence or credible support.
   • Sycophancy: Insincere flattery and agreement to please.

   To address the issues of truth-indifferent AI, the research team developed a new method of training, “Reinforcement Learning from Hindsight Simulation,” which evaluates AI responses based on their long-term outcomes rather than immediate satisfaction. Instead of asking, “Does this answer make the user happy right now?” the system considers, “Will following this advice actually help the user achieve their goals?”

   Notes

   [1] Carl Sagan’s “The Fine Art of Baloney Detection” in The Demon-Haunted World: Science as a Candle in the Dark (1995)

   References

   Bergstrom, Carl T.; West, Jevin D.. Calling Bullshit: The Art of Skepticism in a Data-Driven World (2020). Kindle Edition.

3. 

   So, when AI (chatbot) therapy becomes more popular than human therapists, what does that say? Does it mean that those chabots are better? This article takes exception with that conclusion. Namely, “bad therapy has become scalable.”

   AI companies scraped therapeutic content and their chabots merely model the style of contemporary practice, thereby making it more accessible. But what is needed is something different. “The way forward is not to imitate machines.”

   What is needed is growth vs. coddling, challenges vs. comforting validation.

   • LA Times Opinion Voices 9-30-2025 > “AI therapy isn’t getting better. Therapists are bad” by Jonathan Alpert, Guest contributor – AI’s ANSWERS may be reckless, but the format is quick, confident and direct — and addictive.

   A growing number of people are turning to AI for therapy not because it’s now smarter than humans, but because too many human therapists stopped doing their jobs. Instead of challenging illusions, telling hard truths and helping build resilience, modern therapy drifted into nods, empty reassurances and endless validation. Into the void stepped chatbots, automating bad therapy practices, sometimes with deadly consequences.

   When therapy stops challenging people, it stops being therapy and becomes paid listening.

   A mindset trained to “validate first and always” leaves no room for problem-solving or accountability. Patients quickly sense the emptiness — the hollow feeling of canned empathy, nods without challenge and responses that go nowhere. They want guidance, direction and the courage of a therapist willing to say what’s hard to hear. When therapy offers only comfort without clarity, it becomes ineffective, and people increasingly turn to algorithms instead.

   Good therapy should look nothing like a chatbot — which can’t pick up on nonverbal cues or tone, can’t confront them, and can’t act when it matters most.

Comments are closed.