How Does AI Work For Medical Note Taking and Risk Scoring? (Augmedix, HDAI)

[00:00:00] Dear listeners, welcome to Faces of Digital Health, a podcast about digital health and how healthcare systems around the world adopt technology with me, Tjasa Zajc. If you're still trying to wrap your head around the use of AI in healthcare, this episode will give you an idea about the use of generative AI to create clinical notes during an interaction between a doctor and a patient.

[00:00:28] And also, how can AI be used to create risk prediction scores for patients? Augmedix is a healthcare technology company that delivers ambient medical documentation and data solutions. Their clinician-controlled mobile app uses generative AI to instantaneously create a fully automated draft medical note after each patient's visit.

[00:00:57] I spoke with the CEO, Manny Crackerys, about the how. How is their data model built? What kind of technology do they use in their product? Manny also explained what are the biggest challenges in the industry at the moment and why the company is not covering revenue cycle management or RCM-related codes in their data structuring processes. This is going to be the first part of this episode.

[00:01:27] The second part is unrelated to generative AI and illustrates how existing medical data can be used to create risk prediction tools for medical care. You will hear from Nassib Jamoun, founder and CEO of the Health Data Analytics Institute, which is an analytics company that is developing risk modeling methodology to ease clinical decision-making

[00:01:53] by assigning patients different risk scores based on their medical history. This enables clinicians to design follow-up protocols based on an individual's potential health deterioration. Both discussions were recorded at Health and you can read a longer article about insights related to generative AI from health in our newsletter,

[00:02:19] which you can find at fodh.substag.com and I added the direct link in the show notes.

[00:02:41] If we start with a very general question, generative AI has definitely marked 2023 in terms of the revolution that is expected in healthcare to reduce physician burnout, to reduce the burden with just typing in things in the computer. What's your impression or reflection of this year and what generative AI has brought into the space?

[00:03:05] Generative AI, as we understand it, has essentially enabled us to think about how to use technology to remove certain manual processes that have traditionally taken place in healthcare that have added a great deal of cost to the process of delivering healthcare. And it's manifested itself in many areas in healthcare.

[00:03:35] We're focused on the documentation of the encounter that occurs between doctor and patient and using generative AI to help us automate that process. So it's been very profound in 2023. What about for you specifically? What has 2023 changed? We use LLMs to help us with the note creation process. It is not the only piece of technology we use to do that, but we use it in conjunction with other technology

[00:04:04] to be able to deliver a product that is expected of us by our customers in healthcare. Because you need to deliver something in our business, in our sector, that is extremely accurate. Because mistakes can be very consequential to our customers. So we don't want to make mistakes. What's the most challenging for you in the development of these solutions with generative AI?

[00:04:34] There are many different models of generative AI. Some are very powerful general purpose tools. Some are much smaller but more nimble tools, more specific to what you're trying to do. And for us, the work is focused on finding the right balance between the very powerful large tools

[00:04:59] and the more very specific or narrower tools where you need a very narrow set of pieces of output from. So we use both to get to that ideal output. What data do you train your models on? We use the medical notes that we get in addition to structured data that we generate,

[00:05:26] feed it back through our machine learning process to adjust our models, to produce results that we think are more appropriate for whatever we're trying to document. And some of the learning occurs in a quality assurance process where we compare a note generated by one model based on the same transcript.

[00:05:54] We run it against a different model, different inputs, and we compare the two. And where we see a divergence, we make adjustments to the specific model that we think needs the adjustment. So we enhance the learning process that way. And quite frankly, LM's have made machine learning a lot easier, a lot faster. Where do you see the biggest challenges in the space more broadly, especially with the rising competition that has occurred this year?

[00:06:23] The biggest challenge I think is in PR. There's a lot of noise that's being made by companies because quite frankly, anybody can enter the space and say, look, I can generate a fully automated medical note. I'm brand new to the industry, but I can do it very cheaply because I'm using LLMs. I can get some open source LLM model, which we did over two years ago, and generate something for our customers.

[00:06:50] But the customer, if it's a large enterprise, which we focus on, will not accept that. They will need a lot more than what you can generate from an open source large language model. And even if you get a large language model like GPT-4, it's very powerful. You need to augment that with other know-how and technology to be able to deliver, as I said before, with a product that your customers expect.

[00:07:20] Even though LLMs are like a technology leveler, they level the playing field in terms of being able to come into the game, they're table stakes today. They've been commoditized. You've got to do a lot more than just come in with table stakes if you want to win. One of the investors at a panel said that basically what differentiates companies or startups in the generative AI space

[00:07:46] is the access to the data libraries that you have, which you already mentioned. And the question that everybody should have an answer to is, what's your hallucination rate? Because there will be one. So how do you tackle that? And how would you answer that question? It's a fantastic question. And it's very relevant. So we have generated more than 6 million medical notes since we started. We started this 11 years ago. So we have a big head start over everybody else.

[00:08:15] And we generate over 70,000 medical notes a week. Okay, so that's a lot of data that you could train your models with. And the key to any modeling, whether it's generative AI or any other form of NLP, it's data. And the input data that you provide is critical to how accurate the output's going to be from that model.

[00:08:42] And if you think about general purpose tools like GPT-4, their input data comes from the universe of information that's available. Maybe one-tenth of 1% of that information is actually relevant to the prompt you're going to put in front of that big, monstrous model.

[00:09:05] And you're not going to get the precise kind of answer that you want from putting that very specific prompt in front of this massive amount of data. We prefer to match models with data. We control the data. We have the data sets. We don't go outside to secure other data sets. And like I said, we have a huge advantage over everybody else because we've been generating data for 11 years. So how large is basically the data set?

[00:09:36] I'm just wondering because you mentioned that you have your own data. Can you maybe talk a bit about that, how you basically got access to the data? How many customers do you work with? Do you automatically get their data when you start working with them? You've got to define what data means, their data. So we use data to generate the medical note and we generate data in the process of creating the medical note.

[00:10:02] One of the distinguishing features of us relative to everybody else is we do more than just produce a flat file at the end of the medical note process. So a medical note is just a flat file. Okay. It's not a data file. In our process, because of the technology we use, we also generate structured data, traditional databases of very detailed metadata for every single encounter. Okay. And that is the data that we use to teach our models.

[00:10:32] It's not the medical note itself, which is a flat file, which is very hard to mine and to use. We actually generate traditional data from the process of creating the note. And that's what we harness to teach our models. We have a lot of it. And it's a byproduct of the process that we use to generate medical notes. So when you say structured data, can you elaborate on that a bit further?

[00:10:58] When you do the text to structure the data so it has data is in categories, do you also then annotate that to ICD codes or other codes? We do E&M coding today. And we're in the process of doing ICD-10 codes. We're not going into the business of RCM. Okay. This information is intended to make their job easier.

[00:11:24] And by that, what I mean is today, the state of the industry is that the RCM companies are essentially asked to count the dead after the war has been fought. They're working with incomplete information, very sparse information, sometimes inaccurate information, and asked to file a claim on behalf of their customers.

[00:11:44] We're delivering to them not only a comprehensive, accurate medical note, but the structured data behind it, all the detail, all that metadata that they need to support a particular claim. And that, we believe, is going to help them achieve better claim results for their customers who are our customers. I know that we talked about PR before and basically the challenges on the market with a lot of noise.

[00:12:10] But how would you say that you differentiate yourself from your competition? We are focused on being, you know, talking when we have something to say. That's of substance. And so we will just, when we have something of substance to say, we will say it. If we don't, we won't. So we're not in a PR battle. We're in a battle to win customers. PR does not win customers, at least not in the long term.

[00:12:39] What are some of the common questions that you get from the customers because of all the noise? From our customers, not money, not much. They love our product. And you can speak to our customers and they'll tell you, please don't take this product away from us, whichever of the products they use of ours. So we don't get questions from our customers about the press that they hear from other companies. At least it's not coming back to me.

[00:13:06] But we, it's the only weapon that a startup has is press. It's the only ammunition they have. It's, that's to be expected. No matter what industry you're in, healthcare, semiconductor, software, it doesn't matter. Any startup, that's how they start. They've got press and that's the only weapon they've got. We've got customers, we've got products, we've got revenue, we're public.

[00:13:31] It's going to take a lot of time and effort and money for any startup to get to where we are. Okay. Anything else that you would like to add about the industry in general and where it's going, where you're going? I've been at this company for five years and for the first three and a half, I'd say four years, I sensed the industry was slow moving.

[00:13:53] I came not from healthcare, I came from software and there's been a sea change in the last, I'd say 14 months. And I see the industry adopting technology like generative AI much more aggressively than any other technology that I've seen them adopt during my tenure, brief tenure in this industry.

[00:14:19] So I find that very exciting and it's reflected in the growth of the industry, our particular industry, which is medical note documentation. It is growing very rapidly and it's because we are able to harness generative AI in a very meaningful and profound way to deliver a very high value proposition to customers, not just our company, but our competitors as well.

[00:14:45] And the industry is benefiting already from generative AI and will only increase the benefit over time. Moving away from generative AI, companies are also developing risk scoring algorithms to ease care management on an institutional level. At Health 2023, Health Data Analytics Institute was presenting their work at Houston Methodist and Cleveland Clinic.

[00:15:15] This is a short discussion with Nassib Jamoun, founder and CEO of the Health Data Analytics Institute. Can you tell me a little bit about how you create your models for risk score predictions?

[00:15:34] It's a clinical tool and we tend to be a little bit more hesitant with clinical predictions compared to just notes summarization when it comes to the use of AI in healthcare. How do you do the risk scores and we'll go from there? Sure. It's an approach that we've been working on since the early 2000s. We've published several papers on the methodology and it uses a number of techniques that are well understood.

[00:16:04] They're very transparent. In fact, in every publication we have put out on the methodology out there, we've included in the supplement a spreadsheet that has the coefficients for the models and how they work. And typically, it's a statistical summary of all the historical codes that are part of the patient's history up to that point in time.

[00:16:29] And by training the models on the largest national database available, we're able to have fairly robust coefficients and therefore very robust predictions for the patient. Having said that, I think a lot of people try to think of AI as magic. Our approach is to think of it as a statistical operational tool.

[00:16:55] We personally do not believe that any of the models out there are of diagnostic quality or should be used as diagnostic. They should be used as tools that summarize, synthesize data, and operationally gets all the clinicians on the same page relative to where the patient's status is at.

[00:17:20] But ultimately, the decision of what tests to run or to verify all the details is going to come down to clinical decision making. And I think at this point in time, we don't believe AI should be interfering with that. How much configuration of the model do you do on the individual institution level? How many different institutions do you work with?

[00:17:45] We have today about a million patients who are part of the Medicare ACO program on our system. And we are working with two large institutions, Houston Methodist and Cleveland Clinic. We're also talking to several others about deploying. We do not have to do much in terms of configuration for institutions to get up and running with our baseline models.

[00:18:13] In fact, at Houston Methodist, when we tested the performance of the models straight out of the box as downloaded from the Medicare environment and applied to the Houston Methodist population, the accuracy for those models was equivalent or better than what we've published on the original data. And these models are applied to all patients in the system, not just Medicare patients, all adults over 18.

[00:18:41] Can you mention which types of risk scores do you provide? We have a large family of risk scores that range from measures of mortality at different time intervals between in hospital all the way out to one year or longer, 30 days, 90 days, or even a week or two after an encounter.

[00:19:06] We also have measures of adverse events leading to an admission like a stroke or myocardial infarction or pulmonary composites like respiratory failure or pneumonia, an infection composite, acute kidney injury, falls, deep vein thrombosis, pulmonary embolism, etc. Heart failure.

[00:19:34] And we have a fairly rich list. But if we go to an institution and, for example, at Houston Methodist, their ICU wanted a predictor for delirium, so we went back and built that predictor and delivered it to them within a very short time frame. So if we don't have it, we deliver it. So that's the adverse events. We also have predictors of utilization. How often a patient should be seen?

[00:20:01] What kind of labs or imaging they may receive? And we also have predictors of cost. And the reason all those are important, because in value-based care, you have to understand the intersections between outcomes, utilization, and cost. And that forms the basis for our predictive suite of models that we offer our partners. And we customize it based on the line of service or the clinical service that's likely to use it.

[00:20:31] I'm sure you did a lot of quantitative research in terms of the impact of these predictive models. Can you mention any of the numbers or results of the use of the models in clinical practice? We're just at the very beginning, and now the clinicians are coming up with a dizzying array of use cases. One of the first use cases right now that we have deployed at Houston Methodist

[00:20:57] is trying to flag patients at high risk for mortality and readmission and take the highest 20% of patients in that category and set up their postoperative follow-up within three to seven days. So the higher risk patients get seen sooner. And to make sure that to the extent there's something preventable or avoidable, they can address that.

[00:21:24] We will start recording the outcomes associated with that intervention soon. We wanted to give the team time to get up and running and get all their processes in place. There are several other use cases that have emerged. And what I'd like people to think about AI in healthcare today, in terms of these early days, it's about creating AI literacy in the mind of clinicians or the care team,

[00:21:53] getting them to read quantitative measures of risk, getting them to think about integrating them into their workflows and operating more efficiently with this information in hand. But every time one of those protocols goes into formal implementation, we intend to work with Houston Methodist and the Cleveland Clinic to document the impact. And I'm certain that while you cannot save every patient

[00:22:21] from the complications of their condition, by being more vigilant as a team across the organization, you're going to help a lot of patients. And I'm certain there'll be a significant impact on outcomes, on utilization, and on cost. That means the patient wins, the system wins. And more importantly, our job is to make it easier for clinicians to find all this information so they can spend their time

[00:22:49] focusing on the patient rather than doing a search in their EHR. You've been listening to Faces of Digital Health, a proud member of the Health Podcast Network. If you enjoyed the show, do leave a rating or a review wherever you get your podcasts, subscribe to the show, or follow us on LinkedIn. Additionally, check out our newsletter. You can find it at fodh.substack.com. That's fodh.substack.com.

[00:23:19] Stay tuned.