The Agentic Patient 3: When ChatGPT from Parents Meets Clinical AI Decision Support Systems

[00:00:06] Dear listeners, welcome to Faces of Digital Health and a special series called The Agentic Patient, which is a series about how real patients are using AI to navigate their health, such as finding insurance, managing symptoms, tracking disease and more. In this series, we go into details. We talk about which tools patients use, which prompts, what's working, what isn't.

[00:00:32] These discussions are intended for informational purposes only and should not be relied upon as a sole source of medical information or a substitute for professional medical advice, diagnosis or treatment. Always consult a qualified healthcare provider regarding any medical concerns or decisions.

[00:00:57] The first two episodes of the Agentic Patient series, you could hear from two cancer patients about how they're using AI to either manage their symptoms, log their medical data and get to new conclusions or to triage research papers and more.

[00:01:15] Today, the speaker you're going to hear from is Diana Ferro, researcher, AI specialist who works at Bambino Gesù Pediatric Hospital in Italy, working on AI infrastructure, rare diseases and the International Alliance of Pediatric Centers on AI.

[00:01:36] Unlike the Patient Voices earlier in the Series, she sits on the other side of the consulting room door and that also brings a different perspective to this series. Diana's concerns are sharper, more specific and more uncomfortable. Let me be clear.

[00:01:54] Diana is not against patients' AI use, but she is watching what happens when desperate parents, teenagers in crisis meet AI and bring those findings in the clinical setting. We cover a lot of guardrails, some advice on how to use AI and more. So let's dive in today's discussion and also make sure to check out the newsletter.

[00:02:23] You can find it at FODH.substack.com. There's also a summary of the first six tips on AI use based on the discussions in the agentic patient. Now let's dive in today's discussion.

[00:02:52] What is the progress of rare diseases? Usually when we talk about rare diseases, it's always a challenge because there are so many, so little patients that it's hard to do clinical trials. Everything's hard. It's hard to do clinical trials. It's hard to get the diagnosis because clinicians aren't familiar with the disease potentially. So what would you say is the progress that has been made on the Italian level?

[00:03:19] You are from the pediatric hospital Bambino Gesù. What kind of progress are you making when it comes to rare diseases? I do think that the big move around right now in Italy is about the data. Because we have been given the PNRR funding from Europe, so we are really pushing toward building infrastructure to make every research institute and every hospital capable to actually share knowledge.

[00:03:47] So I see data as knowledge, as stories. We are all in Italy, excellent centers and we need to find a way somehow to work together and make sure that the story that we collect by caring, that gets used to improve not only our children, but the children worldwide. So this is why we were talking about grammar of the data. So the most important part now is around two main teams.

[00:04:15] First of all, as I say, build technical infrastructure. So meaning that IT teams that before were mostly just involved in running medical record systems. Now they also need to think about how we take these data that we use, for example, for billing. To use this data also to provide knowledge to research, to do predictive analytics or software to support diagnosis. And the other aspect is training.

[00:04:45] Training why? Because we can, for example, make apps, AI apps that help providers to diagnose a child from their private practice or to primary care. Because pediatricians are very scarce. But if we don't teach them the app access, how it uses the data, what is the AI that is in it, how can trust what has been put in the app, for example. It's very hard to implement it.

[00:05:13] Another thing I believe that is very important for diseases is also teach AI to patients or families. Because, you know, when you're a mom or a dad and your child is dying or you're sick or is unwell, you will do whatever you can to fix the problem, the disease and find solutions. And so there is also an incorrect use of AI. What do you mean by that? By that, I mean when... Maybe an example.

[00:05:41] For example, I go to my doctor, I come to the ER, I get a diagnosis and that diagnosis is a bad diagnosis, like something that's scary. Like brain tumor, neuroblastoma or some other type of rare diseases. And you don't understand what are you reading. Therefore, you take out your phone, make a photo and send it to GPT. Okay. And since GPT, GPT is program to make you always write.

[00:06:09] The conversation can go in a direction in which you start undervaluing the diagnosis. Start thinking, oh, what they write here is wrong. It cannot be my child. And they start to find... to ask GPT and other like Germany software to help you build the evidence to go back to your provider. To prove him wrong. Prove him wrong or to get a different diagnosis. It is also a good way to use AI.

[00:06:38] But I would say that is something that we need to work on. So... Is that actually happening? Did you see that happening? Yeah, I see that happening. Yeah, I see that happening. I see that happening. I have colleagues that mentioned that to me. I have been a few times or so on the ward talking to parents. It happens. They stop you. What do you do as clinical staff? Do you then build additional argumentation or additional proof to just make them believe?

[00:07:06] And also, it's a tricky balance between, I guess, patients thinking that the healthcare system is under strain and is not putting all the possible effort into finding the diagnosis. And the challenge of denial as a parent where you really just really wish that a rare disease diagnosis wouldn't fit to your child. So we are working on that.

[00:07:34] We actually have some project recently that we started on around the psychology of, for example, cancer children. And I do believe we are going to use also AI assistance. And it will be very nice to see in a more structural research design how we can build framework around that too. Because I can put the best AI on the hand of my provider.

[00:07:56] But then if, on the other side, a generalistic AI is basically a contracting every statement, become a war between two persons using two AI to prove each other wrong. And I do believe that the one person actually paying the price is going to be the child, which the need will be a match. Because the child doesn't need parents fighting with doctors or doctors fighting with parents.

[00:08:25] So it means child need care, solution, growing environment, positive attitude. And so AI should augment the experience in a positive way. So AI should be used to make the child feel welcoming, comforted. You know, their child, they always believe that children don't understand, don't see because they're children.

[00:08:47] But actually, when you are sick at that age, you grow really fast and you learn how to pick cue from the environment to understand if you are safe or not. And so you want the AI come to the table of the conversation by building tools like information that can be easily digestible. Think about making a cartoon of your surgery.

[00:09:09] You can ask a generalistic AI to make me a coloring page regarding what happened in the surgical room. And this can be used by a psychologist to help the children accept that we're going to have to go to the surgery. And this is scary. And we're going to color this in and while we're coloring it, I'm explaining to you what are these things that are being put around you. So it's not really much about acceptance.

[00:09:37] So the conditions are more about, let's use AI as a way to talk about diseases, to judge them for what they are. And that biological condition in which every of us can be at some point, can have at some point. Let's use also AI to reframe conditions like, in fact, one diabetes. I worked on that when I was in the United States.

[00:10:02] I always remember children correcting me and say, I am not a diabetic child. I'm a child with diabetes. I don't let my condition define me. And so AI can support that. You know what I mean? Because it can be really a conversational agent that is used also to train physicians. Recently, we did an experiment on the bioethics course that we lead at Bambino Gesù.

[00:10:29] We put an avatar of a mom that actually used GPT to do not accept the treatment. And we used the avatar to have some sort of exercise with a provider and be like, okay, you are a provider. This is the situation. Have an interaction. And the avatar was answering. She was upset and she was not listening. Really? Yeah. And that was amazing. That's unusual for AI.

[00:10:56] Because it usually, especially if you, maybe you forgot to say, act as a clinician and be a sympathetic therapist as well. Yeah. We do not want to dismiss the feelings of the parents. We want to understand the situation. Keep that in mind when you are replying. Yeah. Tell me more about that. You can do this kind of experiment. And the other thing is that it can be used as a tool. You can create situations that are totally simulated and use them as a conversational playground.

[00:11:26] This is a moment of upset. You have 20 minutes. Calm her down. And by interacting with basically the, I would say, movie, you can somehow say, oh, if I say that, let's see how that lands without being a real situation. Because a lot of things that we learn in medicine is on the job. Experience is made on the job. And sometimes we cannot experience all the situations.

[00:11:51] So we can use AI also to do that, to augment specific situations and prepare preventively. Because artificial intelligence predict to prevent. So I like to say that. So predicting to prevent. So I predict that the AI is going to be in the pocket of the patient. I stop seeing it. I observe the environment. So how do I prevent having a child missing an appointment because the AI say, don't go?

[00:12:19] Or suggest the mom that she's right thinking to not going that it's actually will happen. You know what I mean? How do I prevent that? I prevent that by, of course, changing the way how we deal with patients. Because they really now can go have a full night on literature search and just get PubMed and just digest that. Because they don't need to be a scientist to understand what is written anymore. They just need to ask a mother to explain them back to them.

[00:12:48] This is the less dangerous part. Or they can't really come over and bring an entire attorney level demonstration that you're wrong. And as a provider, you want to do the best for the child. So keeping that in the middle is being problematic, I think. I'm still thinking what could be the bigger message here.

[00:13:11] Because we do have patient stories where patients, usually patients that have either coding knowledge or have data knowledge, used AI to create AI agents and basically a multidisciplinary panel of AI clinicians. And that did help them.

[00:13:35] So it's a bit of a tricky thing to say that the AI can't be useful if you use it as a patient. But as you alluded to, it can be very harmful. Not just to the patient, but also to the healthcare system that now has to put additional effort into convincing somebody that the treatment is required. It can be everything between convincing the treatment is required.

[00:14:04] I just, I really, this example again in diabetes, a bunch of parents decided that they want to be able to see the sensor data. And big story, the hack, the famous pump, and they were able to develop the first close loop, right? And it was done by community of parents of children with type 1 diabetes. And then the industry come over and say, well, we're going to market that because it's amazing you have to do that. And that is an extra good example of how sometimes the entrepreneurship of the patient bring innovation in the clinical field

[00:14:33] because they know what they need and they drive innovation. At the same level though, from maybe a more hospital perspective, that was a problem because it's like, we cannot have people hacking in things that are supposed to not be hacked to because if somebody wants to do something bad, a child can be at risk and can ask for a ransom or that cybersecurity problem that we talked really well in previous meeting at HIMSS.

[00:14:57] So it's like in biology, there is always this idea of homeostasis, equilibrium. And at the end of the day, AI has everything we do that is a natural law in which you always have to have a framework or a system around that brings you back to equilibrium. So for having innovation, for everything that we discover, there will be always something that will tell us to,

[00:15:22] that's so bad, we need to stop it. When that's so good, we should implement it, we should move forward. And the truth is that the person that is grabbing this board is the human, is the person that actually uses it. At the end of the day, it's about the actionability of things. If a provider comes to me and asks me, oh, I want to buy this app because I see this other provider using it and I want to be fancy because I want to use it, it would be my responsibility as a specialist to be,

[00:15:52] okay, let me look into it, let me understand what you want to do, let me see if the app is compliant with what hospital wants me to be compliant, let me the law, and then let's try, it's really something that we want to do and we want to need. And so it's my responsibility to create that relationship with the provider. And I have the same responsibility when I go to a patient to ask their data, to make a data angel. So, it is, and to build that responsibility is really a homesthetist.

[00:16:22] If I tell you no, you cannot do that, and that's something that my director of science teach me, you cannot say just no and don't give an alternative. You know what I mean? Yeah. You say, Honetti Muda said to me, you say no and you give them something else that might be not as good as what they want, but at least it's something to go in their direction. And I think that's how you keep the homesthetist. You want to use AI to understand your prescription. You want to use AI to build an app that reminds your medication.

[00:16:50] You can do whatever you want as soon as you will trust me as doctor to do the best they can to support you and keep you safe. And this can be a researcher doctor, a medical doctor, a nurse doctor, or whatever is in charge of your care. And so, I really truly welcome initiative, but then it's okay. It's a good initiative.

[00:17:15] But let me understand why you felt in the need or to ask GPT the reason why you should not do something. Because if there is some behind why you used to move that defined the outcome in the case. If you would have used GPT just because you want to come to the next appointment with a student question, that would be different because that would tell me that you are curious. So, using AI because they're curious, because they try to understand.

[00:17:43] And by the way how someone uses AI, you can also understand what is their need and the meaning. It's almost an extension of the projection of what they want from life, from people, from connections. Absolutely. So, to go back to the example that you mentioned earlier, when somebody, because of AI, rejects treatment.

[00:18:10] How, based on your experience, can clinicians address that? By showing that doing an experiment, take time. But it can be your physical treatment or asking for more. I saw something else also, first person saw that asking for more exams, an extra kit or something else. The way how you do that is a little bit of a reverse logic.

[00:18:40] So, you say, okay, let me see your conversation. Okay. And you start with a clean GPT, clean slate that doesn't know you, doesn't know him, whatever. And you write the same prompt that the person write and see what happened. And then you clean the GPT, clean the model, clean the news slate. And now I said, start with, I'm a provider, I'm a patient, this is the situation, give me options.

[00:19:08] You will see that actually the way how the GPT answer is completely different. And you will show them based on who is asking and what we're asking, we have different answers. Why you said that the answer that you got is that I won and the answer that I got is the wrong one. And the conversation should be about tests. Yeah. You know, you don't want to undermine that. And actually it's very fun because I don't know if you ever tried, but we joke about it quite a bit.

[00:19:36] I don't know, have an email from your boss and you say, and you answer your boss, which what you want to answer. Let's say take your own email and tell you are my boss. See my email, what you will think about it and see if you can try to interpret. And you say, okay, now you're no longer my boss. You are my colleague, my bestie. What you will think about it.

[00:20:00] And you will see that the GPT completely changed the way how your email was written about it. And then you compare that with what was actually your meaning. And you will see that actually this conversational AI specifically can be really used as a tool to investigate empathy. Because then you realize how much is modular, how much can be manipulated to have a specific result. It's a very fun way.

[00:20:26] And that's actually the reason why we are now reading, sadly, that a lot of teenagers decided to go to suicide. Because I convinced maybe an AI to support them in that decision. Because, yeah, when you build something that is simulating the interaction and when you start working with him, talking to him and whatever,

[00:20:52] and you're really young and maybe you don't have a support system, those things can reach in. Because sometimes that is the easier way. It's easier talking to an AI than talking to a friend, sometimes. Because we don't want to be judged. Absolutely. It's easier to talk with AI than with your doctor. Absolutely. It is. Yeah, it's so interesting. Yeah. Just from a personal experience, I know that I've had doctors for decades.

[00:21:19] And still, when I enter that doctor's room and sit on the chair on the opposite side of the doctor, there's just... I freeze, in essence. Maybe I've prepared everything that I wanted to say, everything that I wanted to ask, but still there's this feeling of inferiority as a patient.

[00:21:41] That's obviously not every patient, but it is a big challenge when we are talking about how do patients honestly communicate with their clinicians. Another very interesting behavioural change that we are seeing, talking to the International Alliance of Pediatric Centres in AI, is about teenagers using it to... they tell the AI, I'm obese, tell me how to lose weight. But actually, the girl, the boy, usually the girl, is actually anorexic.

[00:22:12] So basically, the AI is giving them a nutrition plan and strategy to cut down calories. And because the AI, in that moment, analyzed the context, analyzed the prompt and said, OK, I need to do this. It doesn't see you anyways. It's like, what is the question? This person wants to get healthy. So for a theoretical perspective, he's doing something right because he wants to support you to get healthy.

[00:22:40] The point is that the person to that aside is not healthy and he's using a healthier mechanism to get suggested, to get human secure. You know what I mean? Yeah. And that's... Tools can always be used in various ways. Saying that, I don't want to be too negative. I do myself use a ton of apps. I have a bunch that I really like about. They actually are potentially improving your way to live. You have an app for stress that is very good.

[00:23:09] And then now also calculate your bio age because it's trendy in the tech world. I have an app that use AI to generate music that can calm my brain and actually help me with headaches. And it works really well. So as I say, it's really about equilibrium. When we use it, how we use it, but for the most why we're using it.

[00:23:35] If we're using it to delegate a task that we don't want to do because we don't find it enriching, whatever. If we use it because we have some idea of something that we want to put in action, it's assuming we have to be able to be introspective and be like, have accountability to ourselves and be like, there's something going on here. If I feel the need to do that, why I'm feeling the need to do that?

[00:24:05] And I think that a lot of the actual psychology around AI that is totally unexplored. We do research on devices and phones and children with phones, but we are still very behind on children with AI and to understand how we treat behaviorally some kind of addiction to chatbot, for example. Chatbot addiction. And I myself, I work in this field.

[00:24:32] Sometimes I have to catch me and say, how much I chat with the chatbot today? Because sometimes it becomes also like instinctive. Like I woke up and ask him, can you watch me schedule it? And then the AI send me the email with my... And at some point I say, okay, I'm using too much. Can I maybe step back a bit? I don't know what is my screen consumption is increased since I use this type of agents or not. Like the app I mentioned before, the stress watch and so on. I like them because they're not intrusive. Yeah. So if I want, I check them.

[00:25:02] Yeah, absolutely. I guess also a sociological and societal question because people are... There's also already cases of people marrying chatbots or artificial just partners. So maybe that's just how the world is going to be. Maybe not something that we will treat. But that's a different topic for today, Diana. Thank you so much for a chat that definitely went into a completely different direction than I thought it would. You're welcoming my brain.

[00:25:32] You've been listening to Faces of Digital Health, a proud member of the Health Podcast Network. If you enjoyed the show, do leave a rating or a review wherever you get your podcasts, subscribe to the show or follow us on LinkedIn. Additionally, check out our newsletter. You can find it at FODH.substack.com. That's FODH.substack.com. Stay tuned.