Good afternoon to all those who are attending today's webinar, Dominique Hussain by the Public Health Healthcare Delivery SIG. My name is Mayor Brumani, I'm a transplant epidemiologist at Western University in London, Canada. I'll be your moderator today. So one of the goals of our SIG, Public Health SIG, this year was to focus on activities around patient-centered care, in other words, how we could serve our patients better. Last year, we started off by looking at the hot topic of palliative care, and now we're moving into the emerging area of patient-reported outcomes. So I'd like to now introduce our two presenters for the day. For the first half of the talk, we'll have Dr. Manisha Verma, who's currently Assistant Professor of Medicine at the Sydney Kimmel Medical College, which is part of the Einstein Health Network in Philadelphia, Pennsylvania. Many of you are aware that Dr. Verma was awarded a prosthesis grant by PCORI in 2017 to prepare the effective palliative care models. She's also gone on to win many awards through the Albert Einstein Society, including the Innovation as well as Research Awards. She has significant expertise in health service research and is a prominent researcher in the field of patient-centered outcomes research. In the second half of the talk, we'll have Dr. Sisodia, who is a gynecologic oncologist at Massachusetts General Hospital, as well as an Assistant Professor of Obstetrics at Harvard Medical School. And so many of you would be asking yourselves why we have a gynecologist presenting at one of our hepatology talks, but Dr. Sisodia is currently the Medical Director for Patient Reported Outcomes at Partners Healthcare. She is actually responsible for the implementation, management, and strategy for patient reported outcomes within the Partners enterprise. And during her tenure, which is quite impressive, she has grown the program to be the largest in the world, including 213 clinics over a broad geographic location. And they've actually been able to collect over 8 million patient reported questionnaires. And so with this, within her capacity, she's also seen a creation of a data warehouse, which houses these patient reported outcomes. But really, the ultimate goal would be to map them to individual patient characteristics and outcomes. So with that being said, I'll let Dr. Verma take over and begin the webinar. All right. Thank you, everyone. And good afternoon. Thank you for being here. And thanks again to the Public Health Special Interest Group for allowing us to discuss the implementation science of patient reported outcomes. Thank you, Mayo, for giving all the background introduction. I think it's very helpful. I will straight move forward to the next slide. Me and Dr. Sisodia have no conflicts of interest to disclose. I am really glad that Dr. Rachel Sisodia is able to join us to this webinar, be given her expertise at a large scale, patient reported outcomes, collection of data, and implementation. So the learning objectives of our webinar today would be to discuss briefly what is patient-centered care, what are its key biomarkers, what are the role of pros in our liver disease research and practice. We did a small pilot project, so I'm going to discuss that implementation of patient reported outcomes assessment within a small hepatology practice. We'll go over some of the key steps we should know and be familiar if we are ready to do a pro integration into our specialty care. And then Dr. Sisodia will discuss large-scale implementation of pros led by Partners Health being with her under her esteemed guidance and leadership at Partners Health in different specialties. So we'll move on. So patient-centered care is one of the key objectives of our delivery reform in specifically the U.S. It is providing care which is respectful and responsive to individual patient preferences, needs, and values. And we really want to ensure that patient's values guide all the clinical decisions, which is the key concept of patient-centered care. Here you will see that patient reported outcomes actually form the foundation for an evolving patient-centered healthcare system. These are outcomes which are important to patients, families, and the communities. These can inform research, policy, health services improvement with an ultimate goal to improve the outcomes and patient engagement within our healthcare system. So why actually pros? I mean, these are like all the clinicians are supposed to know what their patients are feeling. What does the concept of pro come from and why are we so much interested in it now? So this is not a newer concept. This has been back in 1970s and 80s. There were some questions which were asked directly from patients, specifically palliative care, where people were being responsive to patient's pain, their sleep disturbance issues, and fatigue back then. But in early 2000s, it was FDA which took the lead in developing some core concepts of patient-reported outcomes and in using those to inform drug development and even device development. So patient-reported outcomes are critical to capture the patient's voice, understand how the illness and treatment affect people, and establish how well services and treatments really address what matters most to them. And moving forward, they are becoming key performance measures, and they'll be directly or actually are directly linked to the level of reimbursements for all practices on outpatient, inpatient, and everywhere. So this table shows some of the key concepts or key terms of what we are going to discuss during the webinar. So patient-reported outcomes, as I talked before, is any symptoms, health behaviors, functional status, mental status, which comes directly from the patient without an interpretation of the clinician at that point. So it's kind of anything which a patient tells us is all patient-reported outcomes. For example, for our liver disease, we need to know depression in our liver disease patients, how much fatigue they have, their sleep disturbance, physical health, mental health. All these are key concepts of patient-reported outcomes. And when you add an M to it, it becomes a questionnaire. So patient-reported outcome measures are all tools which are used to assess or measure patient-reported outcomes. And most of the time now, these are validated tools which have been validated in specific populations or even in general populations, and they are either domain-specific or symptom-specific. For example, the PHQ-9 is a personal health questionnaire which is used to assess depression. Patient-reported outcomes measurement information system. The PROMIS is an NIH-supported tool and kind of holds arena of patient-reported outcomes where researchers who were experts in qualitative interviews and experts in psychometric properties, they came together to develop a whole resource of patient-reported outcomes which are available through the NIH now. It was available for free for research purposes and for clinical purposes, you have to pay a little bit, but now these are integrated within electronic health records. So there is an NIH resource of PROMIS which is available through Google, and I would be glad to share the links to everyone so you can look up and learn more about it. It is domain-specific, used for the general population, can be used for specific diseases, and is a really good resource for specialty care. Then you have CLDQ, which is our well-known chronic liver disease questionnaire, which has been used since 1999, which was developed by a prominent researcher in patient-reported outcomes, it's Dr. Yannasi at Inova, and these questions are domain-specific and symptom-specific as well. Then you have patient-reported experience measures, or TREMs, so these are tools to measure patients' experience during receiving care at hospitals or outpatient clinics, and currently these are being very, very important if some of you are working in administration, you look at these reports very frequently to see how satisfied your patients are. So this consumer assessment of healthcare providers and system survey, the CAHPS survey is sent to patients after any of their clinical interaction to see what patients' experience was, not only with their provider, but also with the front desk, with the entire clinical service they received. Then PROPM is another new term which has been integrated and will be very familiar moving forward, so these patient-reported outcomes are now also becoming the performance measures. They can be used to assess performance of a clinician or of a whole system. For example, there is a percent of patients who had major depression, which is assessed by a PHQ-9 score of more than 9, and the National Quality Forum would like to know how many of these patients had a follow-up score of less than 5 at 6 months. So these are performance measures which will be linked with the reimbursement. So value of PROs in clinical research and practice is actually threefold. You can use them as clinical endpoints in research where we have increasingly used these in liver disease research, and I will go over some of the recent articles which have been published where PROs are used as clinical endpoints. They can be used as prognostic markers, we know about this for a long time, and then the most recent and the most interesting concept is using PROs as intervention tools to really inform the care and improve our care for our patients and their families. So as clinical endpoints, the PROP-UP trial, which has been recently published in the Journal of Hepatology, it's kind of a trial in which they assess the role of PROs in understanding the regular DAA regimen for our patients. It's a multi-central prospective observational study where the primary aim was to characterize patient-driven outcomes before, during, and after regimens of DAA therapy, the direct acting antiviral therapy. And they really found that patients who achieved SBR, which is sustained biological response, had improvement in PRO scores while those who did not had either worsening or no change in symptoms, clearly identifying the clinical role for these questionnaires into our routine care for hep C patients. Here in this article, they clearly outlined the role of PROs linking with the side-process scores, specifically in NASH use. What they found was that the authors here in this study, the authors analyzed the data from phase three stellar clinical trials of NASH enrolling patients in 27 countries. So this data comes from a rich resource of patients, not only in the US, but across as well. So what they were doing is they were testing the safety and efficacy of cellular assertive and reducing fibrosis progression in NASH, but they also assessed PROs at baseline and follow-up time points. Almost 1,600 patients were enrolled and they used three PROs, as you will see in this figure. It's the CLDQ, chronic liver disease questionnaire, which has six domains. And then they used SF36, which is the short form 36, a generic tool, and the work productivity activity index to see what impact the disease of NASH has on their work productivity. Given that NASH, we all know, is the rising burden for liver diseases, all predominantly becoming fatty liver. So what they found was that patients with NASH and advanced fibrosis have a significant impairment in quality of life, which is even greater in those with cirrhosis. As you will see here, the dark blue bars are patients with cirrhosis, the light blue are those with emerging fibrosis, and the ones which are white are population norms. You will clearly see that our patients are significantly deteriorated than the general population. This study really helped clear up the fact that patients with NASH have no symptoms. I mean, they're all symptomatic. They all suffer from deteriorated quality of life. And it also informs the need for development of medicines or other resources which can help improve growth and also the fibrosis scores in these patients. So what they also found was very interesting, I would like to say, is that fibrosis is the only histologic feature of NASH which was associated with patients' long-term outcomes, including impairment of health-related quality of life. So given that, the disease burden, I feel, will be most relieved by a treatment that will primarily target the state of fibrosis in patients with NASH. Another study from European NAPLD registry where they assessed growth in NAPLD, what they did was that this comes from three European centers where they enrolled patients with NAPLD and other chronic liver diseases. And this figure, they used CLGQ, our commonly used chronic liver disease questionnaire, for liver disease-specific quality of life. And they found that all patients had deteriorated quality of life as fibrosis grading increased from steatosis to fibrosis. So it's clear that quality of life is impaired as the disease progresses. They also found, which is not reported right here, is that fatigue and sleeping quality were the most frequently reported symptoms in our population of fatty liver. So the striking finding of this analysis in a well-characterized European cohort really contrasted the published data on predictors of overall and liver-specific mortality in this case. So this study really highlights the link of impaired health-related quality of life with the liver-parenchymal inflammation. So now, which is the most important part, is PROs as an intervention toolkit. What we did here at Einstein was a small study funded by the Ames Center. We were really thankful to receive these funds where we used PROMIS-29, the tool of intervention within hepatology. So the primary objective of this study was integration of PROs as an experimental intervention before the clinical consult of patients with chronic liver disease and or STD as a patient practice. The three key elements which we used were the patient-provided training, PROMIS-29, using PROMIS-29 through the NIH Assessment Center for the initial results, and capturing what the provider did using those results, using a post-study visit service. So what PROMIS-29 does is it's 29 questions which assesses seven domains, which is fatigue, sleep disturbance, anxiety, depression, pain interference, and the risk-typical function. Each domain is assessed using four questions, and using the NIH Assessment Center, you can get the results stacked. When you click submit, this is the report which we were able to share to our patients and providers. Here, you can see the red one really shows the worst symptoms. So higher the score, the higher the domain assessed. So higher fatigue score means higher fatigue. On the other hand, higher physical function score means better physical function. And also, there was a question on pain intensity. So what the T-scores here are, the good thing about PROMIS is you can compare it to a 50, which is the norm for our population, for the U.S. general population. So anything beyond 50 means it is the worst. Anything below 50 means it's good, specifically for the domains which you are looking at. The only two domains where the scores are reversed are physical function and social function. So the score report was very much easier to understand. We enrolled 75 patients with a mean age of 60 with chronic liver disease, like over eight years as the common. And the CLDQ scores at baseline were 4.8. So what we found was the PROMIS-29 scores at baseline were very low as compared to the general population, specifically for anxiety and fatigue. We also found that physical function was severely impaired as the childhood score increased. So paired mean differences in PROMIS scores, what we found was that there was an improvement in some of the PROMIS domains from baseline to three months as a result of this PRO intervention, where typically fatigue and anxiety improved clinically for this population. But the improvement did not sustain at six months because there was no intervention at time two, which is three months. We integrated the PROs at baseline, and that's the point when the clinician and the patient were able to discuss the PRO results during the clinical consultation. After the clinical consultation, all the three-month and six-month questionnaires were done by phone because the purpose was, again, feasibility and seeing if there was any effect. And even this pilot study showed that there was a benefit. What we found, we also did some exploratory analysis. We found that greater physical impairment at baseline predicted greater intensity of pain at six months. Greater sleep disturbance at baseline was an independent predictor of patient mortality at six months. We also assessed satisfaction with the use of PROMIS-29 during our clinics. We found that most patients really found this useful to convey physical and emotional well-being, and the majority found it useful in helping discuss their individual symptoms with the providers. The provider survey after the clinical consultation visit really revealed that providers also found it to be useful. In 79% of the clinical interactions, providers found that PRO-8 provided additional information that helped structure the medical interview because you were able to know that the patient had fatigue, so you were able to cut down the questions you had to ask. You were able to directly discuss the things which were important to the patients. And we also found that in the 88% of the interactions, providers reported that T-scores really helped patient education to address management of specific symptoms. So I will just outline some of the key steps which we all or anyone who is interested to integrate PROs in their routine care should know. Step one is always outlining the goals for collecting PROs. Are you collecting it for research or for clinical practice or for both? Then a very important point is selecting the patients, the setting, and the timing of assessments. Are you targeting all chronic liver disease patients or only hep C or only fatty liver? Because that will determine what the questions are you want to select. The setting. We want to understand whether we are collecting PROs with an outpatient or with an inpatient setting. And the timing of assessments. Usually before the clinical consultation visit is the perfect timing because this gives time for the providers to react on the PRO scores. Determining which questionnaires best address your patient needs and will support clinical care is a very important point which we all need to understand as we aim to include PROs within our routine care. Another challenge which I have faced and I feel is very important to know about is the mode of administration and scoring of the questionnaires. Because it is very important to have the PRO scores available to you immediately for clinical interpretation. So either you can do it on paper, but the con about that is you cannot get them scoring stat or to do it electronically. Using electronic platforms, either you have it through your electronic medical records, which is the ideal platform. Because in that way you can have it right there and you can see the scores and you will learn more about it from our next presenter's talk. Because they integrated it with an EPIC which is really ideal and very useful for the patients and for the clinicians. Because you can see the graphs over time. You can do a lot more clinically when you look at the PRO scores within medical information. On the other hand, when you do an outside vendor or through an NIH assessment center, you need a separate login, you need to have a separate portal to capture those scrolls, so that becomes a little bit challenging, but it is better than nothing. And another important point is developing the processes for reporting of results to both patients and providers. Timing really matters for this one. Another point which we faced as very challenging was facilitating score interpretations, because we all know how to act upon MELD scores. We know, okay, this is the cutoff, we need to begin transplant evaluations. On the other hand, for PRO scores, it is very challenging to make cutoffs because it is so individualized. What we have learned from our pilot study is that the PRO scores really give you some backgrounds to act upon, but your individual clinical decision really has to take the next step. Then evaluating the impact on improving clinical practice is, again, data is always helpful not only for research and for clinical practice, so you need to know how it is helping you improve the care. So the pros to understand or to consider is always, point one, think if you are using PROs to create clinical efficiencies and what is the primary goal. Integrating PROs within electronic medical records is quite challenging, but it is the idealistic way. And ensuring the PROs are clinically relevant to your patient population is very important, and hence the need to integrate patient input to begin with is always very helpful. Some of the system barriers, I will go over some of the barriers briefly, and then I will hand it over to my next presenter. Some of the system barriers which are important to think about and to consider are lack of IT support, which is always a big barrier, but it is modifiable. Either we can train our clinicians to do the programming or support IT workers from our research fundings or select some standard measures, as I did for my PROMIS study, in which standard measures exist, so you can use them as your resource. Difficulty pooling data is an important part to think about because it is important to know where are you standardizing your database into. Proprietary PRO tools sometimes require payment and funding needs to be involved in those, so it is important to consider the pools or PRO tools which are available within public domain. Missing research data is always the challenge, specifically for PROs involving lots of questionnaires, people skip questions, and then it is very difficult to analyze PRO data with missing questions. You need a lot more statistical support and input as to how are you going to analyze that missing data points to reduce so that we don't have inferred wrong findings. And a few other things, including limited data exporting, limited designer understanding of the IT or informed consent of the patients required for research purposes, or even within the clinical rules as well, so it is important to consider these. Some of the clinician barriers which are reported in literature of PRO research and which we face were time constraints. Providers we know, hepatologists are super busy, there is always a clinical priority, so we need to redesign workflows and use extenders, advanced practice providers, or have some financial incentives linked to them so that the integration of PRO becomes a standard and it's not just taken as an add-on work. The desire for single login, everybody wants to log in at one place and then do everything right there, so PROs within EHR is the idealistic way, as I said before. Sometimes there is lack of familiarity between our providers of what these PROs are, so there is a need to inform the clinicians through further education and training as to what these PROs can be, what is the benefit, so a brief training within two-hour windows can be very, very helpful. Sometimes people have negative attitudes, I mean, there is always a team which can operate better with a good champion, so having the chair or the administration be the local clinical champion can really promote clinician innovation and interest. Lack of self-efficacy and outcome expectancy can be a barrier, but if you take recommendations to act and respond to the scores through training program of the hepatologists during fellowship early on would be very helpful. Some of the patient barriers can be that the patients really don't want to respond to the PROs, but sometimes you can make them and do it, and I feel what we have learned is that patients really would like to get their voices heard, and that that becomes a really good facilitator for patient-equal outcome correction. Still, our field of hepatology is very naive in the field of PRO, and there's a lot more we need to do. There exists a high need to integrate PROs within our specialty care. Consideration of feasibility on workflow impact and patient burden, display of results, administration frequency, all needs to be carefully considered while you do a clinical change. There are opportunities for education and training in PRO integration, and future research is needed to inform clinical interpretation, actionability, and based on standardized cutoff scores for our particular conditions. Now I would like to hand it over to Dr. Rachel Fesodia so she can discuss her really important lessons learned and give us some guidelines of how we can do further on with our PRO research. Thank you. Thank you so much, Mani. So, as I said before, I am a GYN oncologist, so thank you for allowing me to speak to you guys today. I'm really thrilled to be with you, and I want to talk to you not about GYN oncology but about our experience at Partners Healthcare collecting PROMs globally sort of across the enterprise, and hopefully share some of the lessons we've learned with that. And I always like to start with this slide and say that as an oncologist, right, no one has ever said to me, no patient has ever said to me, oh, I'm so glad I didn't go to the ICU after your surgery, right? No one has ever said to me, I'm so glad I wasn't readmitted. What people are worried about, right, is am I going to be able to go back to work? What is my quality of life going to be? Am I going to survive? Am I going to be able to get on the floor and play with my children? And these are the things people care about. And historically, we haven't been able to measure those, right? So, we measure what we can. We measure things like 30-day readmissions, ICU admissions, but really that's not what matters to patients. And as you all know, collection of PROs is rapidly becoming the standard of care. This is one of my favorite studies, it's a randomized clinical trial of patients with metastatic malignancy at Memorial Sloan Kettering. And the patients were randomized, either to just routine clinical care or routine clinical care plus collection of PROs. And there wasn't anything high-tech, right? If the patient flagged positive on some PRO, people responded in a common-sense, pragmatic way to that. So, for example, if a patient said that they were having belly pain, right, they might refer them to palliative care or they might tweak some meds. And within that study, just by routinely collecting the patient's voice and acting on it in a pragmatic fashion, they bought a six-month overall survival benefit. And as people who also take care of very critically ill patients, you all know that to buy a six-month overall survival benefit in cancer generally requires hundreds of millions of dollars or billions of dollars to bring a new drug to market. And this we could just do with just asking the patient's voice. So, I'd like to talk to you a little bit about our program. We're a large integrated healthcare system in Boston, Massachusetts. It includes the Brigham and Women's Hospital and Massachusetts General Hospital, which are the teaching hospitals for Harvard, as well as three other specialty hospitals, seven community hospitals, 6,000 physicians in a primary care network. And all together, we cover the lives of about 4 to 5 million patients in the New England area. And in 2014, we launched an enterprise-wide clinical PRO program. And what's important to note about this was that all launches were voluntary. We didn't force this on anyone. And the primary goal had to be clinical care. Now, obviously, we're a heavy research environment, and clinicians were going to perform research on these outcomes, but that could not be the goal of the collection. The collection primarily had to be to help patients. And that for any given condition, prompts had to be standardized. So, if we had docs at the Brigham and Women's collecting in rheumatoid arthritis, docs collecting at Mass General had to also agree to the same tools to be used. And over the last five years, we've really ramped up very quickly. We've now collected more than 7 million questionnaires. And we have more than 375 patient-reported outcome measures built within our EPIC catalog. We're in 213 clinics and 90 specialties. We now actually contract on this with various payers. We've been in contracts with payers for about four years now, reporting patient-reported outcomes for various disease sites. And our leading specialties are ortho and oncology and neurosurgery and psychiatry. But really, we're sort of all over the place. And I like to always put this slide in because we talk about PROs as a very abstract format. But for a lot of us that haven't launched this, we're like, what do they actually look like? So, in our system, this is what they look like. If you present for hip pain to an ortho clinic, you're given a tablet, and this is what your questions look like. Or if you fill them out on the portal in the week before, this is what your questions will look like. And you just toggle through, answer the whole questionnaire, give the tablet back, and you're done. For the doctor, this is the current view in EPIC. And as you can see, there's some problems here. And they speak to some of the challenges Dr. Verma outlined, where you can see that this is literally a live view from my clinic at the patient's PROs. And you can see how difficult it is to sort of find pertinent positives. This is the new provider interface that's supposed to be coming. There's a lot of work in this area. I will say the EHRs do recognize that there's a significant limitation here, but they're working on it. And this is another sample, again, of what a full PRO looks like. This is a questionnaire that we administer for patients getting spinal fusion. You can see the 10 questions that the patient has, and they click their response. And then here in our enterprise, what we will often do to feed data back to doctors is we will convert these into low S regressions, particularly for surgery, although we can do it for any disease site. And because you're going to see a lot of these, I want to orient you for a moment. You can see here the vertical line on the graph is surgery. The preoperative score is in the blue to the left. The postoperative score is in red to the right. And the shading around it is the 95% confidence interval. So this is an example of what we would like to see, right? A patient has a spinal fusion. They have significant leg pain before. And then after surgery, it drops, right? And the benefit is sustained. And so let's talk about how we use PRO data throughout the enterprise. I'm going to give you guys some sample uses. In the beginning, there was a lot of pressure on all our clinics. If they did launch, we said, oh, you can use it in all these ways. You can use it at the bedside. You can use it for dashboards. You can use it to convince external audiences. And what we actually found is that most doctors are most comfortable in one of these four buckets. And most people actually don't use it in multiple ways. Some do, but not all. And we've decided that's okay, right? Using it some is better than not using it at all. Our primary care doctors primarily prefer to use it in the clinic. They all say, if you give me one more dashboard, I just can't stand it. I can't look at any more. Our orthopedics primarily use, or excuse me, orthopedic surgeons primarily use it to demonstrate the efficacy of their procedures, which obviously are considered elective. You can see a low S curve below that looking at the effects of knee surgery. Comparative effectiveness, we will use that throughout the enterprise. This is a dashboard by our bariatric surgeons comparing themselves to each other, comparing themselves to other bariatric surgeons in the country. And we will also use it for contracts. And here are some of the high focus conditions for payers and both CMS in the bottom right. And we do report on some of those. And I like to tell anyone who's considering a launch that it is, sorry, I just got distracted by a little question there. It's okay to operate in any of those buckets. And so some of the things that we can do with PROMs, and I use surgery here because it's the easiest example for people that are new to PROMs, but these can be done with anything. And so I would challenge you to think about them in your own line of work. So if we wanted to say which surgeon gets the best outcome after knee replacement or which hospital? Well, here's an example of our tool. This is the COOS PS. This is a validated tool for things that matter to you if you have knee pain, right, and things you can or cannot do. And so if we ask patients to complete that pre- and post-op, this is about 9,000 data points. And you can see that higher is better here. And so pre-op, the patients are at about a 50, and then post-op, they come up to about close to a 70. And one of the things that we focus on a lot is the minimal clinically important difference, meaning what do patients have to get to actually see a benefit? And for this, it's about eight. So as an enterprise, we perform well above what patients need to get to see a benefit. But what if you wanted to know if our different hospitals, one does a better knee than another? So I can plot that, right, Mass General, Brigham and Women's, and another community hospital. And you can see the curves largely look the same, right, and the confidence intervals overlap throughout much of it. And so as a hospital administrator, when I see this, I say, that's great, right? A patient can come to any of our hospitals, get the same quality of operation, and maybe we should focus on doing them at the Newton-Wellesley, which is much less expensive than the two AMCs. Or what if we wanted to look at our surgeons, right, not our hospitals? And so this is an example of the top three highest volume knee surgeons in the enterprise. And again, this is what you want to see, right? Patients get better, not a lot of variation in the enterprise. And I would argue that's exactly what you want, and PROMS helped demonstrate that. Here's another example. If you wanted to stratify by ASA, do patients who are sicker do worse? And that's true, right? They all get a delta, but they're never, if you're an ASA of three, you're never going to have as good a knee function as someone who's an ASA of one. And again, you can think about this in your own patient population and stratifying people, and for variation and sort of risk stratification, it's a very powerful tool. And so now let's do a more advanced exercise, right? Let's see how things can interact. And so let's look at the COOS and PROMIS-10. So you guys just looked at the COOS with me. Now we're going to add to this PROMIS-10, which is a great sort of global survey for mental health and physical health. And most patients in our enterprise are administered this at some sort of visit throughout their healthcare. And here we can see that for patients who have poor mental health, they are not going to have as good an outcome after a knee replacement as a patient who has above average or even average mental health. And this is with all other things being equal. So one of the things we talk about as an enterprise because of this sort of data is that should patients who are suffering from poor mental health pre-op, right, can we refer them to mental health or social work before and get them in a better position so that they actually may get more gains from surgery, right? And that's an area of active investigation that we're looking at right now to see if we can artificially shift people onto different curves by addressing needs that we wouldn't have seen before. The other thing we like to do with PROMIS data is sort of tell stories. And we like to say that this data really illustrates realities that we may not otherwise have been aware of. This is an example of the PSC-17. This is a validated PROM to assess psychosocial problems in children. And when kids are little, the parents will fill this out for them. And then as they age, they fill it out themselves. You can see here that it's a classic Likert scale and looks at symptoms, sort of anxiety, depression, sadness, oppositional, defiant sort of behaviors, and antisocial behaviors. And we also query patients in our enterprise on my platform about social determinants of health. And so this is a graph looking at how social determinants of health, which are the factors that we all live in, right, where health care occurs, our home, our community, our finances, our family, and look at how those juxtapose with anxiety and depression in children. And what we found is, in this survey, these were all Medicaid patients in Massachusetts, so they're all poor, right, but some of the families are starting to struggle. So some of the families were flagging positive on food insecurity, meaning, and the question was, how often do you run out of food before you have money to buy more? The utility question was, how often this year have you been worried that your utilities would be shut off? And child elder care was, how often do you have difficulty caring for your children or finding someone to watch your children or other adults? And you can see here that just a child being poor, an average child on Medicaid here in Massachusetts has an 8.7 percent chance of flagging as either having anxiety or depression on the PSC-17. But once families start to decompensate, right, once food insecurity or utilities or child care in particular events come into play, that rate skyrockets and ultimately triples. And so PROs allow us to sort of figure out our most at-risk patients here, and when you compare them with PROs in other aspects of life, they become incredibly powerful. One of the ways things that we've looked at using our scale is, how can we start a successful PROM program? And so we actually did a study, right, very classic, so we did an observational study of 231 clinics within 56 specialties who launched via our centralized standardized pathway. And I should say, at this time, we have a central team at Partners of 10 that supports this effort throughout the enterprise. So we have devoted great resources to this. And what we looked at was the mean collection rate per clinic, and we excluded people per the criteria here, and we really just wanted to see what predicts for success or failure. And I will also tell you, for the ordinal regression, we defined success as collecting at a rate greater than 50 percent. And the reason we came up with that number is, frankly, it's hard to get at, and also because you want a number that's robust enough to make relevant clinical or research inferences about that population and sort of affect people on a macro level. And so the clinical characteristics that we felt were salient using, like, tenets of basic implementation science were the following. We looked at year of launch, right? We knew that clinics that occurred or that launched very early on in the life of our program, we had a suspicion that they were at higher risk. We looked at the number of platforms the clinic collected on. Early on in our career, we collected on a third-party vendor. And then two years into our program, we rolled everyone over to Epic. Did they collect PROMs on paper before? Where were they located? Were they forced to collect PROMs, right? I said all launches were voluntary, but a department chair could force other physicians in the department to collect PROMs. Did they have an administrator that downloaded weekly collection rates and then intervened with the front desk staff if they flagged? Did more than 50 percent of the doctors show up for training on where to find PROMs, how to use them? Was the clinic included in a payer incentive, right, were we required by a private payer to collect? Was there a local clinical champion? And what happened in that first month of collection? What was the collection rate? This goes into my data analysis. You guys will have these slides available. I think it's pretty routine data. But what you can see here, at the time of this study, we'd only collected just over 4 million. And we had 205 clinics available for analysis. Dr. Verma pointed out to me that we did not have any clinics that were focused on hepatology. So, there you go. And our largest clinics in medicine were internal medicine clinics, and our largest volume in surgery was orthopedics, as highlighted in yellow. And you can see that most of our clinics were in the AMC. Most had some administrative surveillance. Most actually had a chair that was heavily involved in collecting PROMs. Just over half collected pre-existing paper PROMs. Just over half were included in a payer incentive. Physician engagement was present in just under half. Just under half were early adopters. A third had a local clinical champion. And how we define that, just, you know, for purposes of the analysis, was a physician who was a non-chair who was responsible and who had agreed to take responsibility for clinical prompt collection in that division. And then did they undergo a platform conversion, which is important for people that start on third-party and then want to roll to EHR. Ultimately, just over half of our clinics became successful and collected at a rate of greater than 50 percent. About a third have a mixed performance. And 16 percent failed. And 10 percent collection can be obtained via the portal, just via patients doing it, with almost no clinical engagement. So that's why we designated that as a failure rate. And there was really no common denominator in the underperforming specialties. And what we found on a linear regression was the following. So, location did not matter. Being in a payer contract did not matter. Having a chair mandate for collection didn't matter. Having a clinical champion wasn't statistically linked to success, although it got close. And interestingly, having to undergo a platform conversion didn't seem to impact as well. But what we did find was that pre-existing prompts collection on paper predicted for better performance. Administrative surveillance really lifted performance. And the most key was really the docs being trained and the docs being on board. So, it's important to state here that in our enterprise, we are a very physician-autonomous enterprise. So, we don't, for right or for wrong, we don't force docs to do a lot of things. And so, for physicians to show up to this training, for the most part, that was elective and voluntary or the division director asked them to. But it wasn't tied to credentialing. It wasn't tied to ability to practice. And so, having more than half the doctors willing and engaged in this gave you a list of about 30% on your mean collection rate. And one of the really important findings was, as we suspected, clinics who we launched within the first two years of the program were more likely to fail. And they were associated with a worsening collection rate. And so, one of the things I say there, in addition to the initial month rate, is it's really important to get it right. And so, I always tell our whole team the initial month of a clinic's launch will resonate in eternity for that clinic. If the Wi-Fi gets botched, if the clinic assignment logic goes down, if some key person that you really needed on board is not on board, if you didn't get your workflows nailed down and they have a rocky launch, it's almost impossible to recover from that. And that's a really important point. And particularly, in your very first clinics, which are the most vulnerable, you're learning with them. You have to really do an all-hands-on-deck to get them through that. And so, in conclusion, what I would say is that PROMS data, we found in our experience, it can be used for any of, I call them the four C's, clinic, convince, comparative effectiveness, or contracts of payers. Some clinics will use it for all four, but not everybody does, and that's okay. And people shouldn't feel pressured to use it in a way that's not organic to their specialty. Routine clinical collection PROs has been associated with improved outcomes, including improved overall survival. And widespread collection, this is the future, and it is feasible, but it requires a lot of work. And the most important ingredients to this are going to be engaging your physicians, right? And that may mean working with them on why it's important, working with them on developing algorithms for pertinent positives, working on them to understand how to use this to hasten their workflow as opposed to slowing them down. Whatever it takes, and it's different in every clinic, really. There's no one-size-fits-all approach. You also need to have administrators overseeing their own. So, it is not feasible for a physician to, or it's not reasonable, I should say, to be going out to the front desk staff during clinic and asking them, you know, where are the tablets? Are people getting the tablets, right? You need strong administrative presence for those people to encourage collection. And I also say that were we to do it over again, you have to really protect your early adopters, and you have to really protect that initial month of collection. And you want to make sure all your ducks are in a row and that the whole month sort of goes well, and that you've accounted for every potential thing that can go wrong. And for people on the phone call that are looking to launch within, be like one of the first people within their enterprise, I always say it's really important to make sure everyone knows the first month will be a little bumpy, right? There will be problems that pop up just like anything else, but they kind of need to hang with it and sit tight. And generally over a few months, all the kinks will work out, but you really need to hold the hand of those clinics when they first start. And so I think that's it. I'm not sure, Mayor, if you want to then take over and open the floor to questions for the balance of the time. Thanks, Rachel. And Manisha, I'd like to say that, I mean, a lot of our patients, I think, for the attendees on the webinar, we see patients with decompensated NASH cirrhosis, and we often wonder, is there ascites or encephalopathy controlled? Have we done the varicose screening, and is our HCC screening? But what you're really showing is that sometimes patients don't really even care about that, and they have certain feelings that they want answered. And sometimes in our clinic visits, we totally miss the point. So I think Manisha has shown that, you know, there is some early evidence around PROMs, and there's some implementation barriers, but you've also shown how others on the webinar could be successful about collecting patient-reported outcomes. So there has been some chat online, and one of the questions, Manisha, was, you know, there's a lot of questionnaires out there, and so do we use the PROMIS-29, CLDQ, or the SF-36? And then where do we get these questionnaires from, and are they available online? Is there a cost to them? Yeah, thank you, Mayur, for all the summary, and thank you, Rachel, for sharing all the details. I mean, that's really impressive, the amount of work you have put in. So taking the questions, I feel, through the chat, I opened, I sent a link which is called healthmeasures.net. That gives you all the details about the NIH resources available through PROMIS, the patient-reported outcomes measurement information system. So the good thing about, I'm going to vouch for PROMIS and then also include others. So based on our literature, we have seen that it is a good idea to include a disease-specific questionnaire and a generic questionnaire, so as to have a holistic view. But again, clinical integration is quite challenging. The more questions you add, the less data you get, the more confusion you create. So it's very important to be picky in those matters. The reason I chose PROMIS 29 was because it has domains which are, in general, very important for the overall patient's health, and it can be compared to the U.S. general population. Because our, you would say, literature within liver diseases is so naive that we don't even, that there is a need to actually realize that it is important, first of all, and that, yes, our patients are affected. So I used PROMIS 29 initially. And the other thing is, when you look for the goal, as I was pointing before, is your goal only comparative effectiveness research, or is your goal clinical integration? If the goal is clinical integration, we need to find tools which can be auto-scored and which have stacked results available. So that, because none of the clinician has time to look through 29 questions individually and see, oh, your fatigue was A, B, and C. On the other hand, if you have a score, it looks just like your liver function test. For example, if the AST is 100, you know it's worse. Similarly, if you have a T-score with a physical function score of 69, 75, you know the patient is physically unfit. So you can, so those numbers are going to hit your brain directly just as you look at your lab values. So it's very important to have numbers in front of your face to really understand the pros during clinical use. Of course, in research, it is important, yes. So that's the good thing about PROMIS. The chronic liver disease questionnaire is liver disease specific, and it has all the 29 questions, which are very semantic to PROMIS. Some of the questions were quite similar. Some of the questions are a little bit different. The good thing about that questionnaire is it is disease specific. So you can really assess your outcome measures very clearly. But until or unless there is a way to autoscore it, stat, and create those domain scores right in front of you, and then create cutoff values to understand what is the norm and what do you mean by worse, I think then it can be clinically used. Till date, it has been used mainly for research purposes to see the impact of a treatment. So CLGQ has been very commonly used as an outcome measure to assess the effect, just like in the PROPOP trial, the large trial where they enrolled more than 1,500 patients to look for hep C effects. They used it to assess the outcome of an intervention. So those tools can really help us understand, once the data is published, that this regimen is more effective in improving the abdominal score domain, or so-and-so. So those tools are also important. But again, going back to the question of clinical integration, we need something which is very precise, to the point, can be clinically used immediately. So I think those things are important to consider when you make these decisions. And these tools are available online. CLGQ is available online, but it is not in the public domain. You need to go through some approval from the person who has developed this. So I know there is a process, I mean, I can share it with Dominique, and she can forward it to all the attendees. But it is available, yes. For research purposes, you don't have to make much payment. Now, Manisha, one other question. A lot of our decompensated serotics are quite sick, and there's confounders. So how do you take that into account when patients are filling out the PROMIS questionnaires? And what do you do at your institution when you have patients with ascites and encephalopathy, whether it be overt or even minimal hepatic encephalopathy? I think that's a very important and very good question, Mayur. I think these confounders, that's why it's all good medicine. Asking patients is important, but then you trust your own gut when you actually go and talk to the patients. We cannot have these pros. It's very hard to say, if the pro score of a person with fatigue is more than 75, please prescribe A or prescribe B. It's very hard to do that since these are so individualized. These change over day from morning to afternoon, depending on patient's work schedule, depending on patient's mood, how they are responding, how the things were at home, how the things were at work. So these can inform the care, but there has to be an individual discussion between the patient and the clinician to really make clinical plans. That is why it is very important to consider the timing of these assessments. Just like Rachel mentioned, before and after surgery. I mean, we could do these before and after the transplant to really understand whether the transplant helped the patients improve their quality of life. And for clinical assessment, you need to take your clinical judgment. I know Rachel mentioned very, very importantly, you need to be a good doctor. You need to understand and take your clinical opinion when you actually treat the patients. Rachel, there was a question from the audience, and maybe this is more directed to you, but what are some strategies that you use with the hospital administration to spread pros or proms across the whole enterprise? That's a great question. So I like to say that in the early years, like when we were first starting back in 2014, we would have opened a proms program in a veterinary clinic, like if they would have let us. We knew it was important. We happened to benefit from very high power people who believed in PROs. There's a whole long culture that goes into it, but people believed it was the future. And so we were funded. And then initially what we did was we went to other physicians that we knew were sort of innovators and were willing to take risks. And we asked them if they would be willing to start in their clinics. And so we had a handful in oncology, orthopedics, neurology. And then after that, things sort of gained traction, right? Because as you guys all know, doctors are buddies with other doctors. You went to med school with other people. And so it sort of organically grew on its own. There have been very few clinics. Oh, and let me back up as well. For us in the enterprise, if like one hospital full of radiation oncologists starts to collect proms and show their outcomes, then the other hospitals want to do it as well, right? They realize that it looks a little funny if one group is collecting this data. They really care what patients think and sharing it, and other people aren't. And so it sort of became an organic thing. And we do have, as I said, we have payer incentives for that. But I will tell you an example of that is we are required to submit proms in spine surgery to three separate payers. And one of our neurosurgery clinics just doesn't want to collect proms. They're not interested. They feel like it interacts with a research study they're doing. And so we don't force them. Like we hit the contract based on the other neurosurgery clinics in the enterprise. So we really hold by that line of like, you can't force people to do this. It's just not going to work. It will not work. It's just never going to work. And they're not going to look at it, right? And so and then the patient actually has a worse experience because they took the time to fill this out. No one looks at it. So mainly, we just try to work people. We work with people. We also, frankly, treat people with a lot of respect. And I saw a question that popped up and said, like, what's the government? And I can get more into the weeds on that. But like, I am not a liver specialist. And I'm a GYN oncologist. And I happen to be in charge of this program. But I try to do what the hepatologists tell me would make sense, right? If you guys as a group said, we want these questionnaires assigned at this periodicity for these conditions, we set that up, right? We don't push back. We don't offer opinions. We don't, as long as it's a validated tool and it like globally makes sense, then we do it. And so we find that when you give people wide autonomy to be good physicians and do what they think matters, they're more apt to do it. So I would say we kind of take a pragmatic, respectful approach to our colleagues on that. And that's really helped. Definitely. You're taking a quality improvement approach to things where you get buy-in with a small group and then expand forward. And another question that kind of came up is, how did you train your ancillary staff? So allied health professionals or your nurse practitioners, physician assistants that you might have? Yeah. So every clinic workflow is different, as you all know. And we have 200 different, you know, 205 plus clinics, 205 different workflows. And so we have a project specialist that goes to the clinic if they want to launch. Our basic setup is this. They go to the clinic. They sort out kind of how many, their highest volume patients per hour, how many tablets they need, make sure their Wi-Fi works. They then train the front desk staff on the EPIC workflow of seeing if a patient has questionnaires. That's fast. You can do that in like 30, 45 minutes. And then generally the physicians in charge of the clinic decide what makes sense for the workflow. So in some places, the MAs review the responses and will highlight to the patients if they're, or excuse me, the physicians, if they're pertinent positives. In others, the physicians directly review them. In some, you know, the ortho clinics, they actually have the patients complete them after the visit with the surgical scheduler. Because as I said before, those guys have said, like, we don't want to use this in the clinic. We want to use it to see the benefits of our surgery and for variation. And so we take essentially the lead from the clinic. And we have them, project specialists, literally go during clinic and kind of stand there and wait for like a moment to sneak in and say, hey, can I show you where to find the prompts? Hey, can I show you how to do this? We don't, we try really hard not to add work. We don't schedule meetings at 6 p.m. to go through it. We meet people where they are. Right. Let me add one more thing to Rachel's point. I think you made very good points, Rachel, that we need a physician champion and there is a way we can train the providers. But I am also familiar that within oncology, what they have used is automatic alerts. So it's not a work burden for the clinical staff who are already overburdened. And it's not a work burden for the providers as well. For example, for PHQ-9, which is a very commonly used tool to get reimbursements increased within the oncology care model, which is similar to a patient-centered care model, they have cutoffs. If the PHQ-9 score is more than 15, it sends an alert to the provider and to the social worker. So something like that, which can be automated, really has proven to be helpful to the clinicians and to the staff. That's right. Anisha, another question for you here. You know, inflammatory bowel disease, they've actually incorporated PROs into their clinical trials. And you mentioned a couple of studies in Hep C as well as NASH talking about PROs. But do you think there should maybe be a standard of care and moving forward in all clinical trials for drug development and other trials that PROs should be mandated? So that's a very good question and really very important to my heart because I really like to have these PROs as a part of routine care and as a part of all our clinical trials where we can have the drugs being informed by the patient-reported outcomes and see or actually be able to report that this drug really helped improve the depression in our patients or really helped improve fatigue or really helped improve sleep issues or really helped improve itching, muscle cramps. Like our patients suffer from a lot of symptoms. The symptom burden is very high. And based on most recent literature, the symptom burden is almost as high as other non-liver cancers. So to make that more important, it is very important for all of us as to consider clinical trials which are informed by patient-reported outcomes. Absolutely. Great. There was another question about, maybe this is for you, Rachel, that is it easier for patients to do it on a patient portal or a tablet? What has your experience been? Yeah. So there's actually data on this as well. Patients much prefer to fill it out on their own phones in a portal. And there's a higher likelihood of them being honest. There's a higher likelihood of them completing it. That being said, just a unique situation at Partners is we have a really inefficient, clunky patient portal. And so at our enterprise, we're 84% tablet in clinic and 16% portal. But I will tell you, in other large-scale deployments, most people are the reverse of that. They're very heavy on the portal. And they keep a few tablets, like in clinic, if people didn't fill them out. Yeah. There's just a part to ask if study versus phone versus tablet in the clinic. And Dominique can provide that link for you after the webinar is over. I don't see any other questions that are popping up. I'd like to thank all the webinar attendees for joining us this Thursday afternoon. And I'd especially like to thank Dr. Verma and Dr. Sisodia, who's not even in hepatology, for sharing her experience in patient report outcomes. And definitely, we have something to learn from your work. So thank you again.

webinar

patient-reported outcomes

PROs

healthcare integration

patient-centered care

PROMIS-29

Chronic Liver Disease Questionnaire

physician engagement

clinic care

patient outcomes