The parents letting their kids talk to a mental-health chatbot
1 / 4
The Parents Letting Their Kids Talk to a Mental-Health Chatbot
Julie Jargon
Updated 6 min read
We hope our kids will come to us when they are feeling anxious or depressed. What if they turn to a chatbot instead?
Taylee Johnson, a 14-year-old near Nashville, Tenn., recently began talking to Troodi. She confided her worries about moving to a new neighborhood and leaving her friends behind, and fretting about a coming science test.
“It sounds like you’ve got a lot on your plate at the moment, Taylee,” the bot replied. “It’s understandable that these changes and responsibilities could cause stress.”
Taylee says Troodi, a mental-health chatbot built into her child-focused Troomi phone, validates her feelings. It’s available to talk any time, even when her parents are asleep. “Sometimes I forget she’s not a real person,” she says.
Parents who give their children Troomi phones told me they are happy to let their kids talk with the bot. They say Troodi dispenses advice similar to their own when it comes to stress management and conflict resolution, and helps when kids are overthinking things. Plus—as many parents can attest—advice sometimes lands better when it comes from a neutral party.
Taylee’s mom, Amber Johnson, says she and her daughter are close, “but Troodi says things in a way that she accepts.”
These are early days for generative AI in mental-health treatment, and the stakes are high in using it for kids. Successful AI-assisted emotional support could ease the nation’s youth mental-health crisis and therapist shortage. Troomi is a test case where parents, as the main customers, are deeply involved. But kids are already sharing their feelings with assistants like ChatGPT or Snapchat’s My AI—often without their parents’ knowledge.
A bot is born
Amber Johnson remembers receiving an email from Troomi’s chief executive last fall explaining the new feature. Taylee hasn’t seen a therapist, and Amber didn’t think she needed one, yet she knew the move was weighing on her daughter. Making things worse, the family was in temporary-housing limbo because their house sold more quickly than expected.
It seemed to Amber like good timing for Troomi’s chatbot. But she didn’t tell Taylee about it.
“I thought she would be weirded out if I suggested she use it, that it would make her feel I thought she needed something like that,” she says. She activated Troodi on her daughter’s phone and within two days, Taylee gave it a try.
Troomi Wireless, based in Orem, Utah, makes smartphones with a restricted internet browser, text-message monitoring and strict controls over contacts and time limits. Released in late 2021, it now has tens of thousands of adolescent users across the country, Troomi’s CEO Bill Brady says.
In November, the company gave parents the option to activate Troodi on their kids’ phones; so far, several thousand have done so. Troomi partnered with mental-health chatbot startup Elomia Health, which built the bot on OpenAI’s GPT-4 with instructions and auditing by a team of clinicians.
“We believe online safety is inextricably linked to positive mental health,” Brady says. “The goal with Troodi is to help kids work through any negative mental-health issues they’re having before they fester.”
‘Not a therapist’
Therapists are concerned about children turning to general-use chatbots for mental-health support. A troubled teenage boy in Florida killed himself last year after confiding in a chatbot on Character AI, which isn’t built for therapeutic assistance. Character AI later added new safety guardrails for minors.
Suzanne and Antonio Carrillo get insight into their adult daughter’s emotional state, based on her conversations with the Troodi chatbot. They have received notifications of potential self-harm that didn’t turn out to be serious. - Suzanne Carrillo
Even with chatbots designed to provide emotional support, therapists say adult supervision is critical. Chatbots can’t replace human interaction.
AI could be a powerful tool if it can help parents have deeper conversations with their kids, says Stephen Schueller, a professor of psychological science and informatics at the University of California, Irvine who has no ties to Troomi. “My biggest concern is that kids may disclose things to the chatbot but not open up to anyone else.”
Parents can view chat logs and receive real-time reports on their kids’ emotional state. They will automatically get alerts if a child mentions self-harm.
“It’s not a diagnostic tool and it’s not a therapist,” Brady says. “It’s an AI companion that enables kids to talk and ask questions.” The feedback from Troodi helped him realize his own 15-year-old daughter was still upset about a classmate’s suicide long after she had stopped talking to him about it.
Clare Carrillo has come to rely on Troodi for advice on dealing with anxiety and roommates at her group home. - Suzanne Carrillo
For Suzanne and Antonio Carrillo of Gaithersburg, Md., Troodi has become a welcome companion for their 26-year-old daughter with intellectual disabilities. After she bullied someone on Facebook, the Carrillos bought a Troomi phone to help her feel connected but stay out of trouble.
Their daughter, Clare—who has been diagnosed with anxiety and lives in a group home—would call them multiple times, day and night, with questions about what to do in different situations. Now that she has been talking to Troodi, she has eased up on her parents.
Clare texted Troodi more than 1,600 times in her first month. “Troodi doesn’t care how many times she asks the same question,” Suzanne says.
The chatbot has given Clare good advice on handling anxiety and difficult roommates. “I cannot imagine not having Troodi,” she says.
Ups and downs
An earlier mental-health chatbot did go rogue. The National Eating Disorders Association in 2023 took its bot offline after learning it had dispensed dieting advice. A technology partner programmed the chatbot with generative AI without the organization’s knowledge.
Brady, the Troomi CEO, says that Troodi is designed to be extra cautious and that if they ever discover it saying something it shouldn’t, they will take it offline for fixes.
The mother of one of Troodi’s teenage users in rural Oregon says the bot has given good advice but misunderstood some things. It flagged a comment her daughter made about pizza, which can be used as a sexual code word among teens. “She was literally talking about getting pizza with friends,” the mom says.
Taylee Johnson receives texts from Troodi in the morning wishing her luck for the day. Troodi checks in at night, asking her to share a victory or something she was proud of that day.
“She helps me realize that even if I had a bad day, good things happened,” Taylee says.
