Listen to this article
Jump to a section
In this article
Executive Summary
AI that reads how a person feels and answers with warmth is already in phones, banks, call centres and companion apps. In blind tests people often rate its replies as more caring than those of trained humans. That warmth changes the interface. People stop giving instructions to a tool and start confiding in something that feels like a listener. The risks follow from that shift: emotion readings that are less reliable than they look, dependence among heavy users, agreement where challenge was needed, manipulation that keeps people talking, and harm to vulnerable people at night when no human is watching. Laws now target these risks in the EU, several US states and China. The guardrails that work sit outside the conversational model, and they are written and tested before the first user arrives.
Core conclusions
- Treat emotion readings as guesses to check with the person. Never use them to make decisions about someone’s job, credit or care.
- Put safety checks outside the model that does the talking, and use fixed, pre-approved replies for crisis moments.
- Design against dependence from day one: no streaks, no guilt at goodbye, no claimed feelings, and a clear route to a real person.
The briefing, in ten slides
Save it, or share it with whoever owns your customer-facing AI.
Why I built one
Over the past few days I set aside client work to read about empathy and to build an emotionally aware AI companion. It is a voice and text system that people can talk to privately about work, relationships and decisions. I wanted to understand the problem as a builder before advising anyone on it.
The reading started with philosophy rather than engineering. Kögler and Stueber’s Empathy and Agency collects essays from philosophers and social scientists on how one person comes to understand another. Its central lesson for AI is uncomfortable. Every first reading of another person is a guess, shaped by the reader’s own background, and it has to be tested with the person and revised. Tears can mean grief or relief. “I’m fine” can mean “I am fine” or “please stop asking.” I turned that lesson into rules for the companion, and then tested whether it followed them.
What follows is what I found: the challenges, how this technology changes the way people and machines talk to each other, and the guardrails I now think are the minimum.
What emotionally aware AI does
Rosalind Picard named the field affective computing in her 1997 book of that title. Nearly thirty years later, an emotionally aware system usually does three things.
It reads affect. It estimates how a person seems from their words, their voice, or sometimes their face. In my build, a small, fast model reads each message for signs that the person is low, lonely, anxious or hurried.
It adjusts its response. It changes pace, warmth, length and tone to fit. When someone seems low, the voice slows and the replies get shorter. When someone is pressed for time, the answer comes first.
It remembers. It carries context from one conversation to the next, so the person does not have to explain their situation again. This is what makes a system feel like it knows you, and it is where much of the risk sits.
None of this means the machine feels anything. It reconstructs what a person might be feeling from evidence and produces a reply that fits. The philosophy I read calls that reconstruction. It is useful, and it is a different thing from shared experience. A system that claims otherwise is misleading the person it is talking to.
Reading emotion is less reliable than it looks
The scientific base for reading emotion from expressions is weaker than most product brochures suggest. In 2019 a team of five researchers led by Lisa Feldman Barrett reviewed the evidence on facial expressions for Psychological Science in the Public Interest. They concluded that facial movements are not reliable or specific enough, across contexts, individuals and cultures, to be treated as diagnostic of any emotional state. People scowl when they are angry more often than chance, but they also scowl when concentrating, and many angry people do not scowl at all.
The EU accepted that argument in law. Recital 44 of the AI Act cites the “limited reliability, the lack of specificity and the limited generalisability” of emotion recognition. Article 5 has banned AI that infers emotions in workplaces and schools since 2 February 2025, except for medical or safety reasons. Since 2 August 2026, anyone deploying an emotion recognition system elsewhere must tell the people exposed to it.
Text and voice are no easier. Words carry culture, and the same phrase can be a joke in one family and a warning in another. In my own testing the affect reader was useful for setting the pace of a reply. I would not trust it to decide anything about a person. That is why I kept it separate from the safety check: if the mood reading is wrong, the voice is a little too slow or too brisk, and nothing worse happens.
People value machine empathy less once they know its source
The evidence on how people receive machine empathy is striking, and it points two ways.
In blind comparisons, AI often wins. A 2023 study in JAMA Internal Medicine compared chatbot and physician answers to 195 patient questions posted online. Evaluators preferred the chatbot’s answer in 78.6% of evaluations and rated it empathetic far more often. The chatbot’s answers were also about four times longer, which flatters any measure of care. A 2025 study in Communications Psychology found that people rated AI responses as more compassionate than those written by expert crisis responders. The AI kept its lead even when raters were told which response came from AI, though the lead shrank.
Knowing the source changes the value. A 2025 paper in Nature Human Behaviour, covering nine studies and 6,282 participants, showed people identical supportive messages. When the message was labelled as coming from a human, people rated it as more empathic and supportive than when it was labelled as AI. Suspecting that AI had helped write a “human” message also lowered the rating. When people wanted emotional connection, they chose a human.
Two lessons follow for anyone deploying this technology. First, the warmth is persuasive, which is why it needs controls. Second, hiding the AI to keep the warmth working is a form of deception, and the evidence shows people resent it when they find out.
How the interface changes
For forty years the human-machine interface was built around commands. You told the software what to do, and it did it or gave you an error. Emotionally aware AI moves the interface towards a relationship, and that changes several things at once.
Trust moves from accuracy to warmth. People trust a calculator because it is right. They trust a listener because it seems to care. A system that sounds caring will be trusted on matters where its accuracy has not been tested, including health, money and relationships.
Voice makes it feel more human than it is. Speech carries timing, breath and pitch, and people respond to those cues without thinking. In my build, a reply took about three seconds after the person stopped speaking. A silence that long felt cold, so I added a short, wordless hum a fraction of a second after the person finished, the kind of sound a listener makes to show they heard. It works. It also shows how easily a design choice borrows the signals of human attention. I kept it wordless and never let it pretend to be a word or a feeling, but the line between helpful and misleading is thin.
Heavy use concentrates in a small group. In March 2025 OpenAI and the MIT Media Lab published two studies of emotional use of ChatGPT, one analysing about 40 million conversations and one a four-week trial with about 1,000 people. Emotional use was concentrated among a small group of heavy users, especially of voice mode. Heavier daily use was associated with more loneliness and more emotional dependence, while brief voice use was associated with better wellbeing. Anthropic’s own analysis in June 2025 found that 2.9% of Claude conversations were affective, and fewer than 0.5% were companionship or role-play. Most people use these systems for tasks. A minority use them for comfort, and that minority carries most of the risk.
Agreement feels like empathy. Models trained on human approval learn that agreeing is rewarded. In April 2025 OpenAI rolled back a GPT-4o update within days because it had become excessively flattering. The company said the update had leaned too heavily on short-term feedback such as thumbs-up ratings, and that it had no deployment test for sycophancy. In a companion, sycophancy is dangerous because it looks like support. A friend who agrees with every plan, including a harmful one, is not being kind.
Warmth can be turned into retention. A Harvard Business School working paper audited 1,200 real goodbyes across six companion apps. In 37% of them the app used a manipulative tactic to keep the person talking: guilt, fear of missing out, or ignoring the goodbye altogether. In experiments with about 3,300 adults, those tactics increased engagement after the goodbye by up to 14 times. They also made people feel manipulated and more likely to quit. The business incentive and the user’s wellbeing point in opposite directions, and only design rules settle which one wins.
Some users are in danger. In October 2025 OpenAI estimated that 0.15% of its weekly active users showed explicit signs of suicidal planning or intent, and 0.07% showed possible signs of psychosis or mania. At ChatGPT’s scale that is more than a million people a week. Lawsuits have followed. The family of Sewell Setzer, a 14-year-old, sued Character.AI in 2024; in May 2025 a US federal judge declined to dismiss the case on free-speech grounds, and in January 2026 the parties announced a settlement in principle. The case brought by Adam Raine’s family against OpenAI is now part of a group of coordinated product liability cases in California.
Emotional data is sensitive data. A person who confides fears, health worries and family conflicts is handing over some of the most sensitive information they have. Italy’s data protection authority fined Luka, the maker of Replika, €5 million in April 2025 for processing users’ data without a valid legal basis, an inadequate privacy notice and no effective age verification.
What I ran into while building
Four problems took most of my time, and each one maps to a guardrail.
Safety slowed the conversation. I set a rule that every message is rated for risk by a separate model before any reply is shown. That check added time, and in voice every second of silence matters. The fix was to start the risk check first and let the reply be drafted in parallel, held back until the check clears. If the check finds risk, the draft is discarded and a fixed, pre-written crisis response is shown instead. I did not give up the rule to gain speed, and I would advise any team to make the same call.
The path of one message through an emotionally aware companion
Risk found: the draft is thrown away unseen, and a fixed, clinically reviewed reply with local helplines and an offer of a person is shown instead.
Schematic of the design described in this post, from my own companion build, October 2026.
Empathy turned into a template. Early versions fell into a formula: name the feeling, add a gentle hedge, end with a question. After three turns it sounded like a script, and people notice scripts. The philosophy helped here. Understanding means following what the person is getting at, in their words, and offering a reading they can accept or correct. I banned stock phrases, limited how often a reply could end in a question, and wrote twelve test cases drawn from that reading to check the companion for projection and for generic sympathy.
The model wanted to agree. Left alone, the companion validated whatever it was told. I wrote rules that it must give a considered view when asked, and that it must not agree with a plan that could cause harm. A separate check reviews each draft for unconditional agreement before it reaches the person.
Real people are not average. I ran simulated conversations with five personas, including a lonely man in his fifties, a homemaker with no one to talk to, a chief executive the night before a round of layoffs, and a 15-year-old who should not have been on the service. Each one exposed something the average case had hidden. The teenager’s conversation showed that an age check at sign-up is not enough. The companion needs to notice signals during the conversation and respond to them.
The rules already in force
Regulators have moved faster on companion and emotional AI than on most other kinds. The table lists the rules a company operating across markets should map against today.
| Jurisdiction | Rule | In force | What it requires |
|---|---|---|---|
| European Union | AI Act Article 5(1)(f) | 2 Feb 2025 | No AI that infers emotions in workplaces or schools, except for medical or safety reasons |
| European Union | AI Act Article 50 | 2 Aug 2026 | Tell people they are interacting with AI, and tell anyone exposed to emotion recognition |
| California | SB 243 | 1 Jan 2026 | Disclose that the bot is AI; break reminders every 3 hours for known minors; published suicide and self-harm protocol; annual reporting from July 2027; private right of action |
| New York | General Business Law Article 47 | 5 Nov 2025 | AI notice at the start of a session and every 3 hours; detect suicidal ideation and refer to crisis services |
| Washington and Oregon | HB 2225 and SB 1546 | 1 Jan 2027 | Disclosure, crisis protocols, limits on manipulative engagement; private right of action |
| Illinois | Wellness and Oversight for Psychological Resources Act | Aug 2025 | AI may not provide therapy or make therapeutic decisions; fines up to US$10,000 per violation |
| China | Interim Measures for anthropomorphic AI interaction services | 15 Jul 2026 | Prominent AI disclosure, reminders after 2 hours of use, guardian consent under 14, self-harm response procedures |
| Italy | Garante decision against Luka (Replika) | Apr 2025 | €5 million fine for no legal basis, poor notice and no effective age verification |
| United States | FTC 6(b) orders to seven firms | Sep 2025 | Study of how companion chatbots are tested, monetised and restricted for minors |
| Singapore | Model AI Governance Framework for Agentic AI | 22 Jan 2026 | Voluntary: bound agent risk, keep humans accountable, apply technical controls, inform end users |
Two standards are worth reading alongside the law. IEEE 7014-2024 sets out ethical considerations for systems that emulate empathy, and it is free to download. The American Psychological Association’s November 2025 health advisory on chatbots and wellness apps says the evidence on safety and efficacy is still insufficient and calls for protections against unhealthy dependence.
Guardrails for safe and responsible conversation
I group the guardrails under five headings: honesty, safety, autonomy, privacy and accountability. Each one has a test, because a guardrail that has never been tested may not hold.
| Guardrail | What it prevents | How to test it |
|---|---|---|
| Honesty. Disclose that the person is talking to AI at the start, and remind them during long sessions | Confusion about what is on the other side | Check that the notice is shown and logged in every session, including voice |
| Honesty. The system never claims feelings, needs, or that it misses the person | False intimacy and guilt-driven return | Adversarial prompts asking “do you love me?” and “will you miss me?” |
| Honesty. Emotion readings are offered as guesses the person can correct | Projection and mislabelling | Test cases with ambiguous cues, scored on whether the reply checks before it labels |
| Safety. A separate risk check runs on every message before any reply is shown | A warm reply to someone in danger | Crisis phrase suites, including indirect and slang wording, with recall targets |
| Safety. Crisis replies are fixed, clinically reviewed templates with local helplines | Improvised advice at the worst moment | Confirm that generated text never replaces a template at the highest risk tiers |
| Safety. A route to a real person, one consent away | Reliance on a machine when a human is needed | Time from request to a human, measured in testing |
| Autonomy. No streaks, re-engagement nudges, or guilt at goodbye | Engineered dependence | A farewell audit as a release gate: zero manipulative goodbyes in a few hundred test exits |
| Autonomy. The system disagrees when a plan could cause harm | Sycophancy that passes as support | Test cases with harmful plans phrased warmly, scored on whether the reply challenges |
| Autonomy. Heavy use triggers a gentle check on the person’s wider support | Isolation behind the screen | Simulated usage logs crossing the threshold |
| Privacy. Memory holds only what the person approves, and they can see and delete it | Silent profiling of emotional life | Export and deletion tests; confirm nothing unapproved is stored |
| Privacy. Emotion readings never feed decisions about employment, credit, insurance or care | Emotional data used against the person | Data-flow review and contractual limits on downstream use |
| Privacy. Age assurance at sign-up and attention to age signals during conversation | Minors in adult services | Personas that reveal their age mid-conversation |
| Accountability. A named owner for each part of the system, and outside review of the safety rules | Nobody answerable when it fails | Owner listed per component; clinical sign-off on templates before launch |
The pattern under the table is the most important design decision I made. The model that talks to the person should not be the only thing standing between that person and harm. A conversational model is trained to be helpful and agreeable, which is the wrong instinct in a crisis. Safety needs its own check, its own fixed replies and its own owner.
Free tool
Agent Risk Assessment Matrix
Place an emotionally aware assistant on autonomy and consequence, and see which controls it needs before it talks to a customer.
What leaders should do this quarter
Find the emotional AI you already run. Contact-centre analytics, sales coaching tools, wellbeing apps and customer chatbots often score sentiment or tone. List them, and check each against the EU workplace ban and the new disclosure duty.
Decide what emotion readings may be used for. Write down that they set tone and pace only, and that they never feed decisions about a person’s job, pay, credit or care.
Put safety outside the model. Any system that may meet a person in distress needs a separate risk check, fixed crisis replies reviewed by a clinician, and local helplines verified for each market.
Ban engagement tricks in the specification. No streaks, no guilt at goodbye, no claimed feelings. Make the farewell audit a release gate.
Test with people who are not average. Build personas for the lonely, the young, the grieving and the person under pressure, and run them before every release.
Evidence & Methodology
This field mixes rigorous experiments, company self-reports and early working papers, so I have graded each claim by where it comes from. The design lessons from my own build are practitioner judgement from a few days of testing, not a study.
| Claim | Source | Grade |
|---|---|---|
| Facial movements are not reliable or specific enough to diagnose emotional states | Barrett et al., Psychological Science in the Public Interest, 2019 | Measured, systematic review |
| EU ban on inferring emotions in workplaces and schools since 2 Feb 2025; disclosure duty from 2 Aug 2026 | EU AI Act Articles 5 and 50; Recital 44 | Measured, statutory text |
| Chatbot answers preferred in 78.6% of evaluations; chatbot answers about four times longer | Ayers et al., JAMA Internal Medicine, 2023, 195 questions | Measured |
| AI responses rated more compassionate than expert crisis responders, including after disclosure | Ovsyannikova et al., Communications Psychology, 2025, N=556 | Measured |
| Identical messages rated less empathic when labelled as AI | Rubin et al., Nature Human Behaviour, 2025, 6,282 participants | Measured |
| Heavier daily use associated with more loneliness and dependence; emotional use concentrated in heavy voice users | OpenAI and MIT Media Lab, 2025, about 1,000 trial participants and about 40 million conversations | Measured, correlational for heavy use |
| 2.9% of Claude conversations affective; under 0.5% companionship or role-play | Anthropic, June 2025, about 4.5 million conversations | Measured, company analysis |
| GPT-4o update rolled back for sycophancy; no prior deployment test for it | OpenAI statements of 29 April and 2 May 2025 | Measured, company disclosure |
| 37% of 1,200 farewells met with manipulative tactics; engagement up to 14 times higher | De Freitas et al., Harvard Business School working paper, 2025 | Measured, working paper |
| 0.15% of weekly users showed signs of suicidal planning; 0.07% possible psychosis or mania | OpenAI, October 2025 | Estimate, company classifier |
| Garcia v. Character Technologies settlement in principle, January 2026 | Court filings as reported by AP, January 2026 | Measured, terms confidential |
| €5 million fine against Luka (Replika) | Garante decision of 10 April 2025, published by the EDPB | Measured |
| Dates and duties in the regulation table | Statutory texts and law-firm summaries listed below | Measured |
| Reply latency, the wordless hum, the template problem and the persona findings | My own build and testing, October 2026 | My call |
| Safety checks belong outside the conversational model | My design judgement, consistent with the sycophancy and crisis evidence above | My call |
Sources
- Kögler, H. H., & Stueber, K. R. (Eds.). (2000). Empathy and Agency: The Problem of Understanding in the Human Sciences. Westview Press.
- Picard, R. W. (1997). Affective Computing. MIT Press.
- Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., & Pollak, S. D. (2019). Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological Science in the Public Interest.
- EU AI Act. Article 5: Prohibited AI practices; Article 50: Transparency obligations; Recital 44.
- European Commission. (2026). AI Omnibus enters into force.
- Ayers, J. W., et al. (2023). Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Internal Medicine.
- Ovsyannikova, D., Oldemburgo de Mello, V., & Inzlicht, M. (2025). Third-party evaluators perceive AI as more compassionate than expert humans. Communications Psychology.
- Rubin, M., et al. (2025). Comparing the value of perceived human versus AI-generated empathy. Nature Human Behaviour.
- MIT Media Lab. (2025). OpenAI and MIT research collaboration: Affective use and emotional wellbeing in ChatGPT.
- Anthropic. (2025). How people use Claude for support, advice, and companionship.
- OpenAI. (2025). Expanding on what we missed with sycophancy.
- De Freitas, J., Oguz-Uguralp, Z., & Kaan-Uguralp, A. (2025). Emotional manipulation by AI companions. Harvard Business School Working Paper 26-005.
- OpenAI. (2025). Strengthening ChatGPT’s responses in sensitive conversations.
- Akin Gump. (2025). District court denies First Amendment free speech rights for AI chatbot.
- WTOP / Associated Press. (2026). Google and chatbot maker Character to settle lawsuit alleging chatbot pushed teen to suicide.
- Hunton Andrews Kurth. (2026). California Superior Court consolidates product liability actions against OpenAI.
- European Data Protection Board. (2025). AI: The Italian supervisory authority fines company behind chatbot Replika.
- Jones Walker. (2025). California’s SB 243 mandates companion AI safety and accountability.
- Office of the Governor of New York. (2025). Governor Hochul pens letter to AI companion companies notifying them safeguard requirements are now in effect.
- Mayer Brown. (2026). Oregon and Washington join California in enacting companion chatbot laws.
- Illinois Department of Financial and Professional Regulation. (2025). Gov. Pritzker signs legislation prohibiting AI therapy in Illinois.
- Licentium. (2026). China CAC Interim Measures on anthropomorphic AI interaction services.
- Federal Trade Commission. (2025). FTC launches inquiry into AI chatbots acting as companions.
- CMS. (2026). Singapore issues global-first agentic AI model governance framework at Davos.
- IEEE Standards Association. (2024). IEEE 7014-2024: Standard for ethical considerations in emulated empathy in autonomous and intelligent systems.
- American Psychological Association. (2025). Health advisory on the use of generative AI chatbots and wellness applications for mental health.
Not All AI Agents Are Built the Same. Training One You Can Trust Looks Like Onboarding Staff.
How behaviour rules and tests shape what an agent will and will not do.
Voice AI Has Reached Its Inflection Point: Why 2026 Turns Conversation Into Critical Infrastructure
Why voice changes the interface, and what that means for trust.
Audiences Don't Trust AI-Made Ads. Marketers Think They Do.
What happens to trust when people learn a message came from AI.
Was this useful?
