Kelvin Wursten, leader of PointClickCare's Data Science team, explains how they are utilizing AI to help solve complicated supply vs. demand calculations in hospital emergency departments, as well as the challenge of balancing building awesome technology while still prioritizing the user's needs.
Kelvin Wursten, leader of PointClickCare's Data Science team, explains how they are utilizing AI to help solve complicated supply vs. demand calculations in hospital emergency departments, as well as the challenge of balancing building awesome technology while still prioritizing the user's needs.
EPISODE 08
[INTRODUCTION]
[00:00:01] KW: Then that model, what it does is it's taking all of those inputs and finding those underlying relationships and patterns across that data to predict the patients that are at the highest risk of readmission.
[00:00:13] RS: Welcome to How AI Happens, a podcast where experts explain their work at the cutting-edge of artificial intelligence. You'll hear from AI researchers, data scientists, and machine learning engineers as they get technical about the most exciting developments in their field and the challenges they're facing along the way. I'm your host, Rob Stevenson, and we're about to learn how AI happens.
[INTERVIEW]
[00:00:44] RS: Today on How AI Happens, we're going to be looking at a specific instance of how AI can be used to measure, predict and optimize challenging calculations regarding supply and demand. PointClickCare is a company applying predictive AI models to the tremendous challenge of optimizing the American healthcare system. PointClickCare has constructed a TensorFlow model to weigh a multitude of factors surrounding patient demographics, as well as emergency department bandwidth, in order to create a more effective and optimized hospital experience.
To learn more, I caught up with Kelvin Wursten, who leads a team of data scientists for PointClickCare. Kelvin shared some of the considerations that inform this model, as well as where the responsibility for ensuring these technologies have relevant use cases truly lies.
[00:01:34] KW: I joined Collected Medical, a PointClickCare company just over three years ago, and lead our data science team here. My role here is focused on supporting our hospital health plan and ACO customers. We build machine learning models to deliver insights in real-time, to treating providers and care coordinators. We also provide consulting analytics to show the impact on healthcare costs, utilization and healthcare outcomes through the use of our care coordination software.
One of the main challenges and problems we are pointing our AI technologies at is the problem of healthcare over-utilization. Throughout the United States, there is and has been for many decades, just significant over-utilization of healthcare services in the industry, which results in excess cost, as well as poor patient outcomes. For example, nearly one in five patients are readmitted to the hospital following an inpatient stay. There continues to be that significant over-utilization of hospital emergency departments as well over those less costly and often, more effective urgent care centers, or primary care visits.
[00:02:40] RS: Unnecessary readmission, let's call it, is one example, contributing to patient over-utilization. It strikes me that this is a very delicate, complicated supply versus demand question. I'm curious, just how does the model work? How does the model weigh all these factors and come out with an optimal solution?
[00:02:59] KW: In the model that we built, we're taking the real-time data we get across hospitals across the country. We're building in inputs for things like, patient demographics, chronic conditions, and patient visit history, as well as other inputs to put those into a deep neural network model. Then that model, what it does is it's taking all of those inputs and finding those underlying relationships and patterns across that data to predict the patients that are at the highest risk of readmission.
Essentially, it's looking for patterns that the human brain may not be easily able to find. To give you an example, Rob, we actually had a client ask us about a specific patient and why their readmission risk was so low, because they looked at the patient and they said, “Oh, this patient has a stage four cancer diagnosis.” At face value, that looks like a patient that would be at high risk of going back to that hospital for another readmission.
When we dug into that specific patient, what we found was that that patient was actually discharged to hospice. By definition, when you're going to a hospice care, that means they're no longer trying to continue to treat you in a hospital setting, and they're trying to work on end-of-life care and palliative care to improve quality of life and avoid hospital stays going forward. The model was actually able to catch that and build that into the model and came up with a very low score for that patient, as opposed to the high score you might think of when you just see that stage four cancer diagnosis.
[00:04:25] RS: The downstream effects of solving patient over-utilization are clearly huge, as is I should point out, satisfactorily solving any complex supply versus demand calculation. I wanted to dig deeper into PointClickCare’s TensorFlow model, what particular inputs the data science team deems predictive, and how they decided that TensorFlow approach was right for them.
[00:04:47] KW: Some of the features are those demographics, which include age and gender, that chronic conditions and the diagnoses those patients have and then looking at their visit history. In addition, it would include things like, number of historical addresses where we're looking for, is there a pattern of housing insecurity, as that can be an indicator as well of risk for readmission. It's using all of those features to look for those patterns and identify those patients at highest risk of readmission, that the model was powered on.
When we built the model, we went through and tried 20 or so different models. We ended up using TensorFlow in our final model, because it fit best with our current infrastructure, and it resulted in the best model that we have tried. We also looked at trying a boosted tree models, some linear regression models and XG boost models as well.
As we run the models, what we're looking at is the different metrics of success around each of the models. Specifically, what we focus on is typically the precision and sensitivity of the models. We look at the confusion matrices to see, what are the true positives, true negatives, false positives and false negatives. Then when we're talking with clients, specifically, we talk about precision, which is of those that are predicted to readmit what percent actually had a readmission, so that they can see, if they don't have a model, they may be looking at a 10% to 20% readmission rate, if they're doing outreach to every single patient, which means that 80% to 90% of their work is essentially not necessary.
Then using a machine learning model, it can get them to a significantly higher precision, or of those that they outreach to the percent that actually had a readmission. In the way we built our models, what we're looking for is, how do we provide insights to our users that can fit directly into their workflow? You can build out a great model that has high accuracy and is very accurate. If there's no workflow to put that model into, it's just going to sit and not get used.
What we look for is, we work with our clients, do discovery on what are their current workflows look like, and then design our models to fit into those workflows. Work with our product team to do that discovery, and then getting that user feedback, so we can build the right models. As an example, I've seen in the past eight, that we consider doing a model to predict the risk of dementia, or Alzheimer's. To make this meaningful, the challenge is you need a workflow that can fit into, because it's not like a care manager is going to call up the patient and say, you're at risk of having Alzheimer's, because that's not a story they want to hear. There may not be a good intervention that they can do to get them on a good care pathway.
With a subsequent example being that internally, we've been looking at predicting patients that are at risk of significant overutilization of hospital emergency departments. Going to the ED, instead of going to their primary care doctor. Those ED visits are significantly more costly than going to your primary care doctor. Initially, we thought, “Hey, this would be a great model, that the emergency departments of hospitals would love to use.”
Upon doing some discovery with some of those EDs, many of them just didn't have the care management staff to actually utilize that model, because they were too busy. What we've done is design a model that can be used by the hospitals. We're also then talking about how can we also give this to maybe a health plan case Manager, or the case manager at an ACO, or to the primary care doctor, who might have more time and bandwidth to do some follow-up with those patients, to help get them to alternatives to using the Eds, and getting them to visiting their primary care doctor, instead of that high-cost ED.
[00:08:31] RS: From an AI products development standpoint, how much of this would you say, falls on the practitioner, on the data scientist on the ML engineer, for example? Because they can do amazing things, right? It all has to be steeped into a really usable real-world application that has to fit in the hands of a consumer, and then be useful. I'm curious, as you're directing your team, how much of these kinds of questions are they servicing, versus like, I don't know, man. I'm just going to get an R and do my work?
[00:09:03] KW: It's a critical component, because over time, the tools for developing machine learning models have gotten better and better, and just the technical skills that you need to build a model, those barriers have continued to decrease over time and it becomes easier and easier on the technical side. As models become easier to build the critical pieces and having the background for and the healthcare of knowledge and background to understand your users, so that you build a model that will work for them, instead of a model that just sits and doesn't get used.
I continue to coach my team around, okay, what does the customer want? What do they say they want? What do they actually need? How do you design for what they would actually use in real life? Then, once we design those models, we send it out to our customers to use and get feedback and continue to iterate on those models to make sure it's meeting their needs and can really be used to improve those healthcare outcomes.
[00:09:58] RS: Let's say, you have a massive data set, you have a team of brilliant data scientists, and they all understand the need to build with the user in mind. How do you make sure you're asking the right questions of your model, and presenting the results in a manner that your user can well, use?
[00:10:14] KW: One of the first problems is just getting that user adoption. Because if you just say, “Here's a model,” and don't really explain it to them, and just say, “Go use this risk model,” and it's a machine learning model, it doesn't immediately mean that they are going to adopt it and use it. What we've been doing a lot of work on is just putting in context setting for the models that we've developed with our customers, so that when we're talking to them, we can explain it to them in terms that they understand.
For example, we typically don't use the term AUC, or area under the curve, to show the accuracy of our models, because it's often overly technical and hard for them to understand how to fit into their workflow. Instead, we'll focus on things like the precision and recall and talk about, okay, if you have a 100 patients you're going to outreach to for in a readmission model perspective, then with no model, then you would expect that 10% to 20% of them readmitting. You're going to call a 100 patients to get to that 10 or 20, that actually are at risk of readmitting.
Then, when you talk about the goals of using a model is to shrink the number of patients they actually have to outreach to. Maybe they only call 30, or 40. Of those 30, or 40, they're going to get half or more, would actually readmit. They're much more efficient and effective in their outreach. It's explaining that to them of it is not just a machine learning model score, but it's actually going to improve how you use it in your workflow, so that they can see the value of it themselves.
[00:11:50] RS: Yeah. It's such a tricky thing. Because, again, in the example you shared earlier, where the output might be more interesting to a primary care doctor, and someone who has a front office worker who can call someone and bring them in, as opposed to an emergency department, who don't have the resources to be reaching out to all of their patients. There's just a business approach there that needs to be refined. I'm wondering if AI's adoption, AI is widespread nature is somewhat hamstrung by the need to really figure out who it's the most useful to and put it in their hands, as opposed to the ability to generate the insight.
[00:12:30] KW: For sure. When I go back to that example on the emergency department, and how they're too busy to do a lot of that follow-up care, it's also making sure that you have the tools, or that those clinicians have the tools to make it easy for them as well. They may say they're too busy to really utilize that model, but they might be thinking about using that in a complex intervention, where they're doing follow up after they've left the ED and they may not have time for that.
If you can give them to a pathway that's a lot easier, like for example, adding a note to their EHR about, “Hey, this patient is at risk of significant future emergency department use. We recommend they meet with their primary care doc.” They mention their primary care doc's name, here it is. You can do some relatively easy things to help put them on a better path, even for those clinicians who may not have a lot of time. It's around, here's the model and this will help you prioritize which patients to focus on, and then here's some simple workflows, and workflow tasks that they can do that will really drive positive impacts.
[00:13:35] RS: Yeah, that makes sense. Going back to just the supply and demand of it all, how are you measuring the supply? You are not inventing, but trying to figure out, what is the optimal use for all these available hospital beds, the amount of individuals working, how long it takes to treat someone based on what they're admitted for. It just strikes me as a fantastically complicated calculation.
Then there's two sides, right? Because that is different. The ability to deliver the care is different from the measuring the people seeking it, right? What is that half of the model look like? How are you building that and factoring in all these variables?
[00:14:13] KW: For sure. It’s a challenging piece of the equation. Because if you look at the US healthcare system, where historically, it has been so much focused on fee for service, which is, the more services you provide, the more you get paid. As the system shifts to more value-based care, which is paying for value and the outcomes that you deliver, there's that natural tension between more services and more effective and better patient outcomes. As we develop the machine learning models that we're developing, we have to always be meeting each of our customers, or our clients where they're at stay in that, they're usually straddling that fee for service world, and the value-based care world. How do we help them be effective in the different workflows they have today?
On the ED side, they actually can lose money if there's fewer ED visits at a hospital, even though that's often the least effective, or the most costly place for those patients to receive care. Then it's around helping those hospital customers see that you may lose a little bit of money on the ED side, but that gives you more time to spend with those more complex patients, where you can get paid more to offset some of those costs as well. It's about understanding their workflows enough, so that you can give them the model that will continue to be financially viable, as well as proving patient outcomes that continues to be one of the major challenges of just building any model and healthcare of balancing the financial and economic incentives with the desires to improve patient outcomes.
[00:15:46] RS: Yeah. It's so fascinating. I mean, you get into just incentives here, right? Surely, PointClickCare was founded with this belief that optimizing the healthcare experience and getting more care to more people is just an inherent good, right?
[00:16:02] KW: Yeah. I mean, it's getting the right care to the right patients. That's critical and reducing a lot of the inefficiencies in the healthcare space, because there are so many inefficiencies there.
[00:16:13] RS: How much would you say AI tools is just like, finding inefficiencies, and then applying technology to make people more optimized and efficient? I'm seeing all these parallels, the supply and demand of it all, the really understanding incentives in a workflow and just trying to alleviate it. This has implications. If you solve the supply and demand equation, this feels it has implications for loads of industries.
[00:16:38] KW: I completely agree with that. It's interesting, because it will often talk to some of our health plan customers about doing some of that post-discharge follow up with a readmission model. Sometimes, we'll get to a point where they'll say, “Well, we're just calling everyone and we want to keep calling everyone.” In that instance, it may not be very effective. If they're not willing to change to something that is more efficient and targeted, then there's not really a use case for a machine learning model. It's all around finding those customers who are looking for those ways to be more effective and efficient, and then delivering a model that will help them get there. It can help reduce a lot of the inefficiencies throughout our whole healthcare system.
[00:17:20] RS: Yes, exactly. I'm curious to hear you wax a little poetic here, Kelvin. I want to know, surely, you inhale lots of content about AI. This problem would have to fascinate you, for you to work on it such that you have. I'm curious, maybe it's to do with healthcare and PointClickCare, or just the industry at large. What is it about this technology, the current state of it, the implications of it, the future of it, that really fascinates you, and really has you excited?
[00:17:48] KW: If I take a step back and just think about where we are as an industry, often in the AI space, the biggest challenge can be just getting access to the data. I think, with all of the limitations from HIPAA around the use of PHI data, is a complex problem of just being able to use healthcare data to be able to develop models and having enough of that healthcare data to develop those models. I think, that's one of the challenges. Because even within any single given organization, they are not going to have all of the data, all of the medical data on their patient populations that they're treating. This information challenge around, how do you get all of the data?
Then, once you have all of the data, it just opens up so much of what you can do in terms of improving healthcare outcomes and predicting different ways to improve workflows and care coordination and reducing that over-utilization. I think, as an industry, we're still stuck so much on just that data acquisition piece, that we're just starting to unlock what we can do as an industry in the AI space and the models that can be built.
[END OF INTERVIEW]
[00:19:05] RS: How AI happens is brought to you by Sama. Sama provides accurate data for ambitious AI, specializing in image, video and sensor data annotation and validation for machine learning algorithms, in industries such as transportation, retail, e-commerce, media, medtech robotics and agriculture. More information, head to sama.com.
[END]