I would like to welcome you all on behalf of the Clinical Research Committee of the American Association for the Study of Liver Disease, and on behalf of my co-organizers, Dr. Sherry Rogal from University of Pittsburgh Medical Center in Pittsburgh. My name is Jasmohan Bajaj, and I'm professor of medicine at the Richmond VA Medical Center and Virginia Commonwealth University, and we are thrilled to organize this first of two seminars on and clinical research workshop on artificial intelligence or AI for the non-AI-ologist. This as I said is the first in the series of two workshops spread across two consecutive liver meetings. A deeper dive into AI will be done on the next liver meeting, which will hopefully be in person. The overall goal is to introduce clinicians and scientists to AI or artificial intelligence as a tool for understanding and addressing questions in hepatology. This program spans several applications of AI in hepatology, including datasets, radiology, pathology, and multi-omics. This interactive session will spur further dialogue and research into this field and provide a platform to build on for future clinical research workshop events. The objectives are to understand the facets of AI, including deep learning, machine learning, and others, and Dr. Rogal will cover this in her introduction after me. We also want to describe applications of AI across the several domains of hepatology, such as biopsy, radiological analysis, and outcome prediction, and understand the role of AI in multi-omic and administrative dataset analysis. We have an exciting lineup in front of us, which will focus on multiple aspects for clinical and translational research, and while all the presentations are prerecorded, there will be live Q&A at the end of both sessions, so please tune in for that and engage with the speakers and everyone else. The first 25-minute talk is shared between Dr. Rogal and I, which is an introduction. Then the next 20 minutes is done by Dr. Meena Bansal, Professor of Medicine at Mount Sinai in New York, who will discuss big data, big problems, how to apply AI methods to administrative data analysis, followed by another 25 minutes from micro to macro, applying AI to multi-omic databases given by Dr. Mamta Bhatt, who is in the University of Toronto, followed by live Q&A. After this, the second session will start, which again talks about radiomics, applying AI to imaging by Dr. Qi from Guangzhou in China, who has published a lot in this field, then applying AI to pathology data given the increasing need for us to actually mechanize this and make this so that we can get a lot of this data together and intercalated by Dr. Mazin Nuruddin from Cedars-Sinai, and bringing it all home is our incoming chair of the Clinical Research Committee, Dr. Rohit Luma from University of California in San Diego, who will talk to us of the journey of a thousand miles, first steps towards building AI into hepatology research, and to avoid pitfalls. This is followed by the last Q&A, and hopefully we'll have an exciting session. Thank you again for joining us. Handing over to co-chair, Dr. Sherry Rogal. Hi, I'm Sherry Rogal, and I'm here today to provide a very basic introduction to artificial intelligence. This is meant for the non-AIologists. Thank you, Kim Watt, for that title, and thank you to my co-moderator, Dr. Bajaj, for introducing this session. I want to start out by saying that I'm certainly not an AI expert, and I apologize to my computer scientist uncles for what's about to happen here today. AI was first described in the 50s as a name for the idea that every aspect of learning or any other feature of intelligence could, in principle, be so precisely described that a machine could be made to stimulate it. Webster's describes it as a subfield of computer science that delves into how machines can imitate human intelligence, typically by learning or problem solving. AI can be divided into weak and strong subtypes. You may recognize the image on the right as Siri. Siri is a form of weak AI, meaning that it's developed to focus on a specific task. In contrast, strong AI is a hypothetical term to describe a machine with at least equal intelligence to humans, and typically it has some self-awareness. As of now, this is just a concept, so this is a picture of Hal from Space Odyssey, but it is a little creepy how much he looks like Siri, don't you think? Let's go through a couple more definitions. We talked about how AI is a generic term for when machines mimic humans in terms of problem solving and learning. Machine learning falls within artificial intelligence, and it refers to algorithms basically built from training data. You basically use this method of data analysis to automate analytical model building. Systems learn from what's called this training data to identify patterns and make decisions with minimal human intervention. Within this, there are different types of learning. There's supervised and unsupervised learning. Supervised learning involves when your data go in in a labeled format. Things like regression or random forest models are forms of supervised machine learning. Unsupervised learning takes unlabeled data and looks for patterns that would emerge from it without human intervention. That said, you still have to check to make sure that the output makes sense. Within machine learning is deep learning, but before we get to that, we need to understand neural networks. The next question is, what in the world is a neural network? An artificial neural network, or ANN, is a group of connected nodes or artificial neurons. They're meant to act like neurons in the brain, meaning that each node or neuron receives inputs that get processed to generate outputs. If the output reaches a certain threshold, it can trigger other adjacent nodes. This may seem elusive, but if you're doing regression, you're already doing this process. You're using an AI algorithm without maybe even realizing it. It's important to also see that as the inputs come in, they get weighted to form a function, which is an output that determines whether it activates or not the output or how in what way it activates which outputs. Again, when you have multiple of the so-called neurons, you have an artificial neural network, or ANN. When you have at least three layers in the middle between your input and your output, that's called a deep neural network. Today, we're going to talk about three types of deep neural networks, including multilevel perceptrons, MLP, convolutional neural networks, or CNN, and recurrent neural networks, or RNN. Let's start on this word salad adventure, shall we? I drew that eight myself, in case you were wondering. These are the most common kinds of neural networks. Multilayer perceptrons are considered the most simple form of deep neural network. I didn't make up this example, by the way. This is a classic. Say you have handwritten digits, and you want the computer to be able to interpret them and tell you which digits they are. The computer basically can take this eight and read it in as a series of pixels. Then the pixels trigger all the different nodes. Each of the pixels would be represented on a node. Then it would be weighted by how much it's shaded. If I were writing it in pencil, it would be darker for the darker line and lighter for a lighter line. That could be an example of how you'd weight each cell. Then what would have to happen next? To get it to understand that it's an eight, each layer needs to be able to process and identify patterns from the layer before it. The first layer would just read in the data and say where there was writing. You would go through, and eventually your network would recognize higher order combinations. For example, by the time you get to this point, maybe the computer is recognizing that you have a loop up high, a loop down low for an eight. Then when it recognizes that pattern, it would output that this is an eight into that last column of nodes. That's the most simple kind of neural network in terms of deep neural networks or the multi-layer perceptron. You've been able to turn a drawing into data. At the end, you categorized it as a digit. Each of the layers recognizes higher and higher complexity. Multi-layer perceptrons work in a forward direction. The layers are all fully connected to each other. Each node connects to every node in the next layer. Then within each node, the sum of the outputs from one layer feed into the next. Multi-layer perceptrons are fully connected. They require a lot of computational energy in a sense. They read in vectors. They're a little bit limited in what they can do. If you need more complicated reading abilities for more complicated images, for example, then you'll need to use a convolutional neural network or CNN. CNN uses matrices rather than vectors. It's able to compact complex data into more compact forms. It's really good for doing spatial relations and complex images. Next we have, this is just an example of a journal article where they used CNN to read in pathology data. Then they applied risk models. You'll see this throughout the medical literature. In general, CNN is considered almost like doing computer vision. It's when the computer is seeing images. It works in a feedforward way. It's what we use for most image data. It does facial recognition technology, pathology data. They use it in drug discovery to some extent. But there are some limitations to CNN. CNN approaches can't remember things. Whereas if you use CNN, you can categorize a bird as a bird. It's good at saying what something is or isn't or putting it in categories, but it won't be able to tell you where that bird is flying. So if you need to do something like that, then you would need to use something called a recurrent neural network or RNN. This is what's used for natural language processing, NLP or text translation, speech analysis, and videos, things that require an assessment of things over time. So analyzing video data, for example, if you needed something that could analyze colonoscopy data or your endo reports, for example, and needs to be able to sequence words in a row or images in a row, you would use RNN. In conclusion, artificial intelligence is understandable and it's relevant to clinicians. You're probably using AI already in your life, whether it's Siri or natural language processing. And the most common forms of AI that you're going to read about and hear about today are CNN, which is used for imaging, and RNN, which is used for NLP and other sequential processes. Thank you so much for listening. We're really excited to hear from our other speakers today. Thanks to Dr. Bajaj and Rogal for inviting me to join this clinical research workshop. You just heard about different AI methodologies, and the goal of my presentation is to share how administrative data can be leveraged for AI predictive algorithms and provide insight into the type of problems that can arise from using these large data sets and perhaps strategies to circumvent or mitigate those problems. I have no relevant disclosures to this presentation. So, first of all, what is administrative data? Most hospitals or health systems have access to data generated and collected within their four walls, such as hospital and ambulatory encounters, DRG codes, diagnosis-related groups, billing data, readmission information, length of stay, procedure codes. But we know that often patients seek care elsewhere, outside of our four walls, either unintentionally in the setting of an emergency or intentionally for consultative care. For patients in value-based care or contracts, meaning that we take risk for them, we often have relationships with other institutions and health systems, health information exchange systems, so we can receive an abundance of data in the form of things like ADT event notifications, which are notifications of a patient's admission, ADT event notifications, which are notifications of a patient's admission, discharge, or transfer to another healthcare facility, HIE notifications. We can get census data, educational records, social determinants of health data, which I'm going to come to in a moment. But the most common form of administrative data used in clinical research is claims, whether it be paid or unpaid. And when we have a relationship with an insurer, they provide, on a monthly basis, claims files that we can then use to look at utilization of care, whether it be within our own system or elsewhere. So just a moment on social determinants of health, because I think that it's a topic that many people talk about, and I think it's important to understand what are we really trying to solve for. So in the neighborhood around Mount Sinai, if you were to take the number four train and stop at 85th Street, the average household income is about $180,000. The life expectancy is about 85 years, better than Japan. But if you just continue on that same train just a few stops further, the average household income drops to $45,000. Life expectancy drops 10 years, dropping about 3.2 years for every stop on the subway. So you could imagine, or you could argue, that zip code and household income is a good surrogate marker for poor health outcomes that could be obtained by census data. But we need to have an even more nuanced approach. Patients in the same area may be challenged with different issues. Some may have food insecurity, while others may have housing insecurity or transportation issues. Understanding the drivers of poor health outcomes for particular patients will allow us to better substratify the population to facilitate impactable interventions and drive improved health outcomes. Now, administrative data are able to pick up social determinants of health data if Z codes are used. So Z codes are a category that capture things like problems related to education literacy, occupational exposure to risk factors, problems related to your social environment, et cetera. But until that becomes more robust and commonplace, many health systems are partnering with third party vendors to obtain this big data. So for example, we have purchased data from a company called Centerforce. Centerforce has a comprehensive proprietary data set that includes 255 million robust individual profiles, 8.4 billion questions and answers across every zip code in the country, over 25 proprietary and third party national data sets that are all combined to provide an output that includes 121 comprehensive determinants of health scores across eight different categories, 26 engagement scores, meaning how likely is the patient to engage with the provider or the health system in whatever that intervention may be. And they also provide ethnographic data. So this information helps us become even more strategic addressing the issues that are most relevant to a particular patient. So why does all of this big data matter? Ultimately, we have to recognize that we have limited healthcare resources and there is tremendous heterogeneity within diseases that requires risk stratification. Multiple factors converge that cause different clinical trajectories and outcomes. Some are obvious and some may be hidden in clues within the data that is not readily seen. To decrease healthcare spend and improve outcomes, data will be critical to identify those most likely to benefit from a specific intervention in a timely fashion. So from a health system perspective, we know that we care for a large complex population. And in order to effectively manage this population, we need to be able to risk stratify them and identify impactable factors. So if we think about our large complex population, we need to identify those that are at highest risk for developing different diseases. Let's take, for example, NAFLD. But then within each disease, we need to recognize that there is variable progression within disease states. For example, some will be fast fibrosis, some may be slow fibrosis. In addition, there's a complex interplay between different diseases, variable impactability of interventions, and there is an important impact of social determinants and environmental factors, as I just mentioned. The ultimate goal is to enhance, extend, and expand human capabilities and allow us to deliver the right type of care at the time and place patients need that the most. So we only get claims for contracts where we have a special relationship with the payer. We do not have claims for all insurance carriers. So for those who are uninsured, we get no claims. So when we think about the pros and cons of using administrative data for AI algorithms to support liver research, we need to keep these in mind. So first of all, on the pros, they are large data files. They include info on a wide variety of programs or activities. They may include information that is not in our electronic health record, such as care received elsewhere, pharmacy claims. And it provides us population-level data that would be cost-prohibited to obtain via a prospective clinical research…in prospective clinical research. So what is the downside? We can't control what data is collected. And the biggest issue is the completeness of the data is difficult to assess. You don't know what you're missing if you don't know. Many of the payers also mask behavioral health or other sensitive condition claims. So you're missing that whole group of diagnoses. We know that there's a lot of inaccuracy and incompleteness in diagnosis codes. If it wasn't coded for, it does not exist. We also know that there is inherent claims lag. By the time we get a claims file, it's six months after the claim was actually dropped. And we only have access to claims where value-based arrangement exists with the payer, as I mentioned earlier. Despite that, there are a number of applications of machine learning to using administrative data to help predict clinical outcomes. And I just wanted to share one example more as kind of what we need to think about in utilizing this data, and it could be leveraged for any question that you're aiming to answer. So in this case, the authors were provided a 5% sample of Medicare fee-for-service claims, part A and B, from 2008 to 2011, that was linked to publicly available US Census data by county code. They had almost 3 million Medicare beneficiaries, and about 770,000 had an admission during 2009 and 2011. They split the group into a training set and a validation set. They used HCC codes, DRG codes, CPT codes, excuse me, ICD-9 procedure codes, as well as a clinical classification software. Their machine learning risk model was meant to be used as a clinical decision support tool on admission and discharge. So providers knew what is the likelihood, in this case, the primary outcome of mortality in 30 days, and what the secondary outcome was, likelihood of a 30-day readmission or the development of a new HCC diagnosis category. Overall, using their model, the event rate was about 8.7% for 30-day mortality, with an area under the curve of about 0.88. A Breyer score reflects the accuracy of a prediction model and is on a scale of zero to one. So the lower the Breyer score, the better the accuracy of that prediction model. In terms of their secondary outcome, they had a 9.1% re-hospitalization rate and an area under the curve of their prediction model of 0.73 with a Breyer score of 0.07. Their adverse event rate range from just 0.23% to 6%, but the key is that their model performed well in predicting 17 out of the 23 adverse events and only performed moderately well for the other six. But what are the issues? As I mentioned, the uninsured have no claims, but they still have utilization and risk of events. This information of this data set or algorithm was created on Medicare fee-for-service patients. And so it's not generalizable to the Medicaid population or the commercial population. The most important thing is that a model is only as good as the data and the coding. So if the coding was not accurate, the event didn't happen. There are inherent incentives for coding because it does dictate reimbursement in some situations. And sometimes the centralized billing office will truncate claims and not put all of the diagnoses on to the claim. And so for all of these factors, we may be missing a lot of very important information. And so I think what this shows us is that administrative data is necessary and can be very helpful, but in and of itself may not be sufficient. There is variable information in the data sets. So it's really the combination of data sets that will be needed to address incompleteness. And as we'll get into, I think the key is, is that we need to enrich administrative claims with other forms of data, including natural language processing, EHR data, pharmacy data, and laboratory values. And so now I wanna turn our attention in thinking about what are the inherent challenges or considerations that we need to have when we talk about these different AI or machine learning algorithms or models. They can be broken down into four broad categories, the model integrity, the model applicability, model transparency, and model security. First, we must be aware of the potential for machine learning algorithms to not only introduce bias, but to propagate systemic existing biases. In terms of introducing bias, we can break that down into social bias, wherein inequity in care delivery systematically leads to a suboptimal outcome for one group, and statistical bias, wherein our algorithm produces a result that differs from the true underlying estimate due to inadequate sampling across a diverse population, the heterogeneity of effects, or the potential measurement error of a specific predictor variable. So for example, if we are using claims data to look at the prevalence of hepatic encephalopathy among alcoholic cirrhotics. Again, for those that are uninsured, you will have no claim, but you will still have the diagnosis or the disease. So if you're only using claims data to create your algorithm, you're introducing bias into your model. As most clinical data is generated as a consequence of real human decisions, implicit bias in our current care could also be reinforced. So for example, if you are more likely to diagnose a condition like hypertension or encephalopathy in your white patients, ultimately it will look like Hispanic or black patients are less likely to have hypertension or hepatic encephalopathy. Then your predictive model and thus clinical decision support will reinforce that bias. So what are some more examples of these types of biases and what can we do to proactively avert falling into these traps? So if there is a low sensitivity of a particular risk score, in this example, the Framingham Risk Score, which predicts cardiovascular outcomes, but you can imagine perhaps a non-invasive assessment of liver fibrosis where the population examined was predominantly white. Therefore, statistical bias is introduced when the training sample does not include the full population of interest. Therefore, we can address this by proactively oversampling minority subgroups and or tailoring our predictions or scores for specific subgroups. If diagnoses are delayed due to social determinants, such as transportation to the clinic, it will look as though those patients develop that disease at a later stage. So by creating flags for model uncertainty in high-risk subgroups, we can try to minimize this bias. Always remember that artificial intelligence and machine learning are limited by the quality of data on which they are trained. If data are missing, whether it be administrative data or EHR data, the algorithms that are generated on that data may be similarly biased. So you can base your predictions on more upstream data at presentation of illness rather than subsequent data follow-up. So the goal is really to achieve algorithmic fairness. How do we do that? We recognize upfront that there's potential bias in the training process. We recognize that human decisions are feeding back into that training. If you don't code for hypertension as an example, then the claims data that will never show up. We may have implicit bias in diagnosing certain conditions in certain populations. And then if the clinician follows the AI-based prompts, assuming that the algorithm knows better, they will be trapped into automation complacency, which will then self-perpetuate the bias. We can try to deal with proxies like zip codes or insurance status of markers of socioeconomic status. Though, as I mentioned, those proxies may not be granular enough. Sometimes collecting new data may be better than trying to clean up messy historical data. So how can we reduce bias in AI and actually leverage AI to maybe identify implicit bias and bring it to the attention of the provider in real time? So for example, if a clinician is deviating from the AI-based recommendation, they can be prompted to explain why are they choosing that path, causing them to perhaps recognize their own implicit bias in decision-making. If documentation in the notes suggests a diagnosis, the physician can be prompted to drop a specific code on the bill so that it ends up in the claims file. The greater the dependency on unbiased data sources, the better. And again, perhaps using upstream events prior to any clinical decision can help mitigate some of these risks. Wolf and colleagues published the Probase tool, which poses questions to assess the risk of bias in prediction models, to help assure the selection of appropriate training sets in algorithm development. We need to always have continuous tracking to ensure outputs are not reinforcing existing social bias and always oversampling underrepresented populations can help mitigate some of the statistical bias. By recognizing bias and focusing on model integrity, we still need to ensure that our model is applicable across diverse populations and across different health system. If an algorithm is developed using data with one source, such as Medicare claims, as I said, it may not be applicable to other lines of business, such as Medicaid or commercial. Different radiology imaging systems may store and track information differently such that that might not be applicable in another system. Similar EHR systems vary tremendously and elements therefore detected in one EHR system that may be important for a predictive algorithm may not be able to be applied to another EHR system. And of course, clinicians have various ways in which they describe things. And so these models must account for language variations. And one of the biggest risks too is that new and rare diseases may be missed because they just haven't been coded for and weren't part of the training set. And then what about model transparency? We're talking about transparency to the patient, transparency to the provider of the health system, transparency to the employer, transparency to the insurer. Inherently, people will distrust an algorithm when it deviates from their expectations and appears to be disadvantageous to them. So providing detailed information can sometimes help to align human expectations with those eventual outcomes. Too much transparency may provide more opportunities for objections empowered by confirmation bias. And those who are not disadvantaged by decision have little incentive to delve into the details of how that decision was actually made. Often a better performing model in terms of some accuracy metric might be in conflict with other goals such as human understanding of predictions. There is no rule of thumb on how much performance improvement is sufficient to justify using less interpretable or transparent estimators, but it's best that researchers have clear justification for choosing an estimator and recognize or explain any decrease to lead to distrust. This becomes particularly relevant when you think about proprietary algorithms which are kind of opaque or black box. There is a risk of harm when applied to clinical practice without oversight. And again, patterns may not make sense leading to distrust by the clinicians. No matter what, a model must ultimately meet accepted standards for clinical benefit, just as for clinical therapeutics and predictive biomarkers. They need to have excellent positive and negative predictive values. There must be consequences of false positives and false negatives. Therefore, we must develop hazard prevention protocols and ensure that there are dual safety mechanisms for crucial clinical processes like medical diagnoses or treatment decisions. Doctors make the decisions, but hopefully AI can help inform them with data generated insights. Now, what about model security? Insurers are increasingly demanding direct access to health system EHRs, and they already have the claims. So likely they're using AI type models, not likely, I know they are, to stratify potential risk as well. And that information could be used to determine premium structure even before a patient has actually had any of those healthcare outcomes. There is obviously large sharing of data across multiple stakeholders, including the employers and the insurers. And we always need to remember that there could be incentives for adversarial attack. That means inputting kind of a subtle data element into the algorithm that can result in suboptimal or incorrect decisions. The example of one pixel in a benign mole making the algorithm then say that it's likely malignant, requiring resection or at least biopsy. So we must always keep adversarial attack in mind. So we come to administrative big data. Is it more problem than it's worth? The key is, is no matter how complex the method, a data set that is poor in quality or poorly informative for a given question will not be useful, even if large in size. Bigger is not always better. For example, analyzing claims that may be appropriate to predict an outcome of hospitalization, which is well captured and carefully adjudicated in the data. But using such data to identify covariates predictive of a diagnosis may be fraught because claims data are known to be subject to significant diagnostic misclassification or under diagnosis errors. Machine learning methods may have deceptively high accuracy but predict the wrong outcome, such as predicting the probability of being diagnosed with a condition, not the probability of actually having the condition. Hence measurement and selection biases apply to machine learning methods as much as to any form of secondary data analysis. In the end, administrative data can bring great insights into clinical care, but used in isolation are not sufficient. We need to have very, we must include variable information from various data sets. The combination of those data sets are needed to address that incompleteness. And we need to enrich administrative claims with natural language processing, pharmacy data and lab values. All that being said, there are so many promising applications for AI in liver disease research, some of which you're hearing about from our speakers in this session. We can increase efficiency by targeting patients most at risk for adverse outcomes, administrative data plus EHR. Many algorithms have been helpful in augmenting or improving the efficiency of clinicians through the rapid evaluations of radiologic or pathologic images. Again, some of what you're hearing about during this session. I focus on the use of natural language processing to capture unstructured data elements. This also has the effect of reducing physician burnout, given our over-dependence on machine learning models to pick up documentation structured fields. By being able to pick up what's written in a note by a provider, we're decreasing the burden on them in terms of documentation. And at the end, we can improve quality with better informed clinical decision support embedded in the clinical workflows, ensure adherence to evidence-based guidelines, identify outliers, not only in clinical care, but also identification of implicit bias because certain providers are consistently deviating from the AI-based recommendation. So I think that we can all agree there is tremendous potential for AI and liver research. From all of our speakers today, we are learning how omics coupled with radiology, pathology, data from the EHR, and information from large administrative data can help us both in disease detection and risk stratification. Administrative data can be incredibly helpful, but needs to be combined with other data sets to improve performance and predictive models. We must acknowledge potential bias and put in strategies to mitigate that bias. Recognizing bias and limitations of our model proactively will improve applicability, and titrating transparency and security will be critical. Ultimately, AI has the ability to enhance, extend, and expand our capabilities. Because of the heterogeneity and complexity of liver disease, hepatology is a perfect arena for AI applications. Thank you, and I'll be happy to answer questions during the Q&A, or feel free to email me offline. Thanks, and enjoy the rest of the meeting. Hello, everyone. I'd like to thank the organizers for the opportunity to present to you as part of this clinical research workshop on AI. And my topic is Micro to Macro, Applying Artificial Intelligence to Multi-omics Databases. I'm a staff hepatologist and clinician scientist at the Edgmarrow Transplant Program at the University Health Network in Toronto, and an assistant professor within the Division of Gastroenterology and Hepatology in Toronto. And so I run a translational research program using tools of systems biology, as well as machine learning to various types of data. And this is to inform the mechanistic basis of disease post-transplant. Here are my disclosures. So I'll start off by talking about the types of multi-omics data, and then the applications of artificial intelligence to multi-omics data, liver specific applications to date, and finally future directions. So we'll start off with what are the types of multi-omics data. So diseases arise due to alterations of different biological entities in humans. So there are 20,000 protein coding genes, over 30,000 mRNAs, 2,300 micro RNAs, 20,000 proteins, and 114,000 metabolites in the human body. And as people who do research in liver disease, you surely appreciate that the liver is the metabolic factory of the human body, and that there are different levels of regulation in liver disease. Significant advances in biotechnology in recent years have made the generation of complex omics data feasible, and these make precision medicine even more possible. Genomics refers to sequencing and analysis of an organism's genome, and their single nucleotide polymorphisms, copy number variants, whole exome, whole genome sequencing, as well as mitochondrial DNA, among the type of genomic data that can be generated. There's also transcriptomics, which is the expression readout of the genome. And so you can look at bulk gene expression data, gene expression arrays, RNA sequencing, single cell sequencing, as well as spatial transcriptomics. And so you can see in the lower left-hand corner, there is a figure illustrating how different areas of a liver lobule were found to have different transcriptomic gene expression levels. So clearly there is this varied approach that can be used to transcriptomic data in liver disease. Epigenetics refers to heritable alterations, not due to changes in DNA sequence. And so these involve modifications such as histone modification, acetylation, deacetylation, as well as DNA methylation most commonly. Non-coding RNAs refer to RNA molecules not translated into protein, and this offers a different level of regulation. And so you have micro RNAs, long non-coding RNAs, small nuclear RNAs, as well as small nucleolar RNAs. Proteomics is also another method by which you can generate high throughput data. And as I mentioned, the liver is the metabolic factory in the human body. And so using mass spectrometry techniques, one can generate proteomic data. Additionally, you can generate phosphoproteomic data to evaluate the protein signaling cascade in the liver. Metabolomics involves quantitative measurements related to the physiological state of an organism. And so you can generate lipidomic, glycomic, and amino acid data. And these really offer wonderful insight into the liver's metabolic functions in the context of different diseases. The microbiome has really exploded in terms of data generation in recent years. And so you have 16S RNA sequencing data at its most basic, shotgun sequencing, metagenomic sequencing, metaproteomic, metatranscriptomic, and from different sites, including the intestinal microbiome, biliary, circulating microbiome in the blood, oral, skin, et cetera. So really there is this proliferation of high throughput data at different levels that is feasible these days. However, there are certain caveats to single omics data. So the analysis of any single omics data is limited to identifying a variation. At most, it is a correlation between one or two types of bioentities. The outcome is limited to a reactive process rather than providing insight into causation. For example, an upregulated mRNA may not increase its target gene expression, target protein expression. And if it is an enzyme, the protein will influence its associated metabolites. So therefore, different omics are truly interrelated, as you can appreciate in the figure in the lower right-hand corner. So there are so many omics that can be generated, and ultimately they are interrelated. So therefore, we have to be careful not to fall into this trap, which is the single omics trap. So you see the various blindfolded people looking at the different aspects of the elephant. And so this is the trap of single omics data. Now, using multi-omics really will provide a more holistic view of a specific condition. So really having the various layers of data can provide insight into the true nature of the elephant. So next I'll get into applications of artificial intelligence to multi-omics data. First of all, we need to talk about data integration. So once the data is analyzed, one needs to look at integrating data. So integration of data is feasible with specialized tools, and these include tools such as matrix factorization, Bayesian factor analysis, Gaussian latent variable model, generalized linear regression, Bayesian integrative clustering. So there are various types of tools that may be used to integrate data. And so these can provide deeper insights into the mechanistic aspect of disease. Now, I will talk to you about a few specific tools that can be used for data integration. So here I will present about omicsnet. So this is a way to intuitively explore molecular interactions and regulatory relationships among genes, transcription factors, microRNAs and metabolites. Biological networks play increasingly an important role in omics data integration and interpretation. And so a network building interface allows integration that can be visually explored in three-dimensional space. So you can see here that looking at these various types of data, so transcription factor gene interactions, protein-protein interactions, microRNA gene interactions, metabolite-protein interactions, these various levels of data can then be brought together and explored to assess for interactions and potentially providing insight into the mechanistic basis of disease. And so, for example, in a network, the nodes that are close to each other are assumed to share related biological functions. Here's another tool for data integration. It's called MUNCUT, so multilayer NCUT. And here this uses a clustering approach and this is tailored to multilayer omics data. It sufficiently accounts for both across and within layer connections. So, for example, you may have a profiling study that collects measurements on copy number variations, gene expression, and proteins. And so two variables, so the dots, would be connected by a line if the corresponding variables are interconnected. And so you have two ways here that are illustrated, so the MUNCUT clustering and the k-means clustering, and these can provide insights as to the different layers of data and how they cluster together to provide insight into disease. So next I'll get into how one can integrate and apply AI to multi-omics and clinical data. So you can use supervised approaches for prognostic prediction and unsupervised approaches for clustering into patient disease subgroups. So here we provide the example of cancer patients on one side and healthy controls on the other side. There's three different layers of data here, genomic, transcriptomic, and proteomic data, to which artificial intelligence algorithms could be applied and clustering people into different subtypes, which are then associated with specific survival probabilities. And this can overall, this clustering can provide insight into potential therapeutic targets. Here is one tool that allows for such integration. So this is called the Similarity Network Fusion Tool, which was published in Nature Methods some years ago, and this allows for clustering of various types of data. So here you have the example of DNA methylation, mRNA, microRNA, and these are clustered to provide insight into different types, subtypes of disease. So this SNF tool allows for aggregation of data types on a genomic scale. There are two steps, construction of a sample similarity network for each data type, and then integration of networks into a single similarity network using a non-linear combination method. And these are robust to noise and data heterogeneity. The fused networks can efficiently identify subtypes of disease by clustering, and so you can appreciate that here. So there are various subtypes of a specific type of cancer, which was the example provided in this paper, which was looking at GBM, glioblastoma multiforme, and they identified different subtypes of disease using these different layers of data and applying a machine learning algorithm to these different layers of data. And so they were able to generate predictive labels for new samples based on the constructed network. So next I'll speak about another tool called NetDx, and so this is a machine learning method to integrate multimodal patient data and build a patient classifier, and it addresses risk stratification. So the samples are split into train and test samples. Training samples are subjected to feature selection using regularized regression. The test patients are classified by the highest similarity, and so patient networks that combine the training and test sets are then integrated. And so you can see here, you can appreciate that there are different types of data, so the gene expression, metabolomic, and mutation data that can divide these patients into low-risk versus high-risk patients, so it allows for risk stratification. And so the average model performance is computed by running the process over several train and test splits, and the features with consistently high scores are used to classify an independent validation set. And so you can see here, this is the actual methodological framework that then allows for generation of the average model performance for a given data set, so you get an hour rock, an area under the rock curve, and an AUPR accuracy readings that ultimately provide you insight into the model's performance. So here I'll provide an example in cancer. The cancer space is really where AI approaches to multiomics have really started to develop. So I'll give you an example of a paper published in Nature recently where they looked at meningiomas, and so they had a discovery set and a validation set. They had different types of data here, so methylation, RNA, and CNA data, so copy number variant data, and so they had these different types of data which allowed them to integrate multiple data types into a unified analysis using cluster-by-cluster assignments. And so this study really allowed for definition of molecular taxonomy for meningiomas with direct clinical relevance. So you can see here that these subtypes that were created by these three different layers of data allowed for prediction of survival. And ultimately, what the authors proposed there is that this could inform more precision therapeutics. So next I'll get into liver-specific applications to date. So we had published this review last year looking at the applications of machine-learning tools in liver disease and transplantation, and really the papers at that time that we reviewed were very specific to clinical and laboratory data and the applications of ML tools to clinical and laboratory data, including longitudinal patterns that could be assessed to provide predictions and risk stratification. Now, such studies cannot provide a more mechanistic insight into liver disease, and so what I would propose is that integrating such multi-omics data, as I mentioned a few slides ago, would really provide such a mechanistic insight. And so the application of such tools in liver disease is still in its infancy, and I will present to you two studies specifically where such an approach has been used. So we'll talk about first this paper that was published last year, wherein they looked at predicting and elucidating the etiology of fatty liver disease using a machine-learning model. And so they used high-performance random forest to develop various prediction models. They used machine-learning on various types of molecular data to identify novel molecular features associated with NAFLD, and they combined these data with conventional clinical variables to predict NAFLD. These were data from the IMI Direct Consortium. Eighteen prediction models were developed. They generated rock curves to evaluate model performance by measuring the area under the curve, and basically they determined that the addition of detailed multi-omics data, so genetic, transcriptomic, proteomic, and metabolomic significantly improved the predictive utility of their models. And so in this study, they really applied these machine-learning algorithms to data from deeply phenotyped patients, so 1,514 patients, to identify sets of highly informative variables for the prediction of NAFLD. Now I'll give you another example. So I'm sure most of you are aware of the Cancer Genome Atlas and really what that has provided to the cancer world in terms of multi-omics data in a variety of cancers to inform prediction of prognosis as well as potentially inform therapeutics and subtyping cancers. So clearly for the HCC data, so hepatocellular carcinoma, there's various types of data, including copy number variations, mutations, methylation, mRNA, microRNA, as well as various clinical data associated with these patient samples, such as survival, overall survival, and disease-free survival. So what this study has demonstrated, so this was a study published last year in Frontier Genetics, and they identified 93 potential driver genes using these different levels of data. So they looked at the methylation, the mutations, the copy number variations, and so you can see here that there were 93 potential driver genes identified, and then here you see 10 genes that were driven by mutations and nine genes driven by copy number variation. And what they determined through this study was that these findings would be important for informing prevention and individualized treatment as well as its exploration of potential therapeutic targets for hepatocellular carcinoma. And so these are some other studies wherein this sort of AI applied to the multi-omics data has been used, and these are all based on using the TCGA HCC data. So you can see here that clearly there has been a significant interest in using AI to these multi-omic data. So in conclusion, I'll give you a sense of the future directions I see in terms of application of AI to multi-omics data in liver disease. So I think the challenges in applying AI to multi-omics data in the liver realm is the heterogeneity of data from multiple omics sources. The data may not have been processed in a unified way. There's the possibility of technology There's the possibility of technical artifacts, including batch effects. There is certainly a need for large multi-omics data sets, and certainly there's a paucity of this in hepatology at this point. High computational power would be required for analysis. There is a risk of overfitting machine learning models due to limited sample size, so less than hundreds or thousands. And so far, a lot of this literature wherein AI is applied to multi-omics data has been limited to the cancer space. And so these are the future directions I see. So omics data is becoming more cost effective and will become more accessible. Multi-omics data analysis is still an underdeveloped area of research, but very promising and fast growing. And it really needs a combinative effort from clinicians, biologists and computational analysts to use this wealth of data that is generated to provide a more precision medicine approach to the practice of hepatology. There are several other factors like lifestyle and environmental effects that could be integrated to add new dimension to these analyses. And so the key takeaways in terms of the application of AI to multi-omics data. So clearly, this would provide additional value from a mechanistic standpoint beyond clinical laboratory and radiomic data. The challenge, though, is to generate omics data in a large number of patients. And I believe the future is really bright if we are able to generate such large amounts of multiple layers of data, because this could really inform a precision medicine approach to the diagnosis and therapeutic approaches in hepatology. So I'd like to acknowledge the contributions of Christina Baciu, a scientific associate and computational biologist in my lab, as well as Elisa Fassini, a research analyst to developing this presentation. And I'd like to thank you for your attention and happy to take questions. Thank you. Hi, welcome to all of the clinical research workshop live Q&A session. I'm Jasmohan Bajaj and I'm honored to have organized this course with my colleague, Dr. Rogal. And I really thank Dr. Bansal and Dr. Bhatt for these amazing talks. And this is focused on the basis of navigating AI and liver disease research right now. But as I said, there's going to be another one next year as well. So we hope you have a chance to view the presentations and we've got a lot of questions already. But please feel free to continue to put the questions together. Another programming note, the second workshop part today, it was supposed to be later, but it's now going to start at 5.30 p.m. Eastern. We are going to wrap up this question and answer session, at least the live part, by 5.20 or 5.25 so that you can get a little break, log off, and then log back in. So we want to make sure that we get to all these wonderful questions. Dr. Bansal, one of the questions that we had was, how does the ASLG want to put things together so that people who have interest in the AI and machine learning field can get together and not be across several silos? Yeah, no, clearly there are some emerging themes in this meeting. I think, as you said, single cell and AI. So in terms of AI, you know, it's one of those disciplines that obviously crosses a lot of our disease-specific SIGs. So just like our public policy SIG, it makes sense to think about having an AI SIG that could include representation from all of the different other SIGs, right? And come together to think about this. So absolutely makes tremendous sense. And I'm glad somebody suggested it in the chat. So you heard it first, Dr. Bansal. Please email Dr. Bansal and everyone at ASLG about your demand for the ASLG SIG on machine learning and AI. So questions, Shari, you want to ask some questions as well that came through? Yeah, absolutely. The questions in the chat were excellent. And thank you to both speakers. I definitely learned a lot from your presentations. So the first question is for Dr. Bansal, but Dr. Bhatt, also feel free, please, to chime in. It says, fantastic talk. Thank you. What level of validation is needed for an EHR-based AI tool for outcome prediction? Well, I think, you know, think about algorithms as any other test, really. It has to be assessed as its positive predictive value, what's the sensitivity, specificity. And I think that these algorithms are not one and done, right? It's an iterative process. And it's a continuous learning. And we need to ensure that our algorithm is actually predict how good was it, right? As we move forward, how well did it predict what we were saying? And I think as somebody else mentioned in the chat, I mean, our field is so dynamic, right? So, you know, there's a new test, there's a new this, if we don't, if it's not a continuous iterative process, we'll be like stuck back in, you know, early, you know, we won't move forward. So I think it has to be a continuous process. You always have to say, how well did my algorithm predict what actually happened? These are clinical outcomes. So you know what actually happened. And so you're able to go back and say, how well did my tool predict this outcome? So we're slowly learning that this is not completely different from what we've already doing. We still have to do the hard work of actually making sure it's applicable to the right person, right time, right setting. And this is not just like a magic silver bullet that will come and rid us of all our problems. Is that the overall theme here? Yeah, I think so, Jess. And I completely agree with what Meena said. So I think, you know, you can develop an algorithm based on a training set, and then you need to validate in the test set. And then there is a specific consideration in hepatology. Over the last years, we've had effective hepatitis C antiviral therapy. And certainly some of our algorithms may perform differently in that hepatitis C era as compared to this new era that is relatively hepatitis C free. And in fact, we encountered that in a study that we recently published in that we were asked this question by the reviewers, you know, does your algorithm perform as well in the non hepatitis C era. So then we have to actually split our data set into the pre hepatitis C treatment era and the post hepatitis C treatment era. And interestingly, say the ranked features that predicted the specific outcome of interest differed. So they were different. You know, the rank features based on the SHAP values that we identified were different for those two eras. So I think certainly algorithms need to be say, validated in those different contexts. And, you know, you may find that there are different predictors depending on the, on the different eras. So Dr. Bansal, a very important thing that you touched upon that is really at the core of everything is how to make sure that we reduce bias at every level and whether it be social determinants of health or just things that we want to make sure that the rich don't keep getting richer. That's the ultimate kind of situation. Yeah, it's absolutely critical. And I think it's really important to focus on these AI algorithms because lack of data doesn't mean that that doesn't exist. And I think, you know, even if you look in epic for, you know, in the EHR and you look at the demographic data, half the time, there's nothing filled in for ethnicity. We've done this. We've tried to look, that's why we got some of that center force data to try to get better insight from all these other areas to overlay because as physicians, we're only limited And so really where the big data comes in is how can that supplement what we don't know? Do we know who has issues with transportation or housing issues? We would have no idea. Census data helps, obviously. You have some socioeconomic status data. But again, those are just kind of surrogate markers for some of these other things. And so I think that, one, we need to be hyperacutely aware of this issue when we're developing our algorithms and almost in a way overcompensate, right? Like you overcompensate and make sure that you are including as much of that information as possible in creating your algorithm. The other thing is, is like if you think about generalizability, when you do a clinical trial and your disease burden, like are the people in the trial represent the people who have the disease? So if you think about, okay, if you've enrolled a NASH trial and that one is 90% white and there's like a very small number of Hispanic patients in that, right? Now you're saying that that clinical trial, like the efficacy of this drug is X, but you are missing whether or not it's applicable to that other population. There's no different with an AI algorithm that's based on this. So if you're not including the population that it's relevant to, you will then be propagating that bias because you'll think they don't have the disease. And we know that that's really problematic. And so thinking about whether your algorithm has the proportion of patients that you know are in your population. So if you know your population has certain percentages of ethnicities, you want to make sure your algorithm represents that. And an interesting thing that you brought up is because with Mount Sinai, you deal with an insured population. If someone is not insured, they don't show up. But we have two other experts here, Dr. Rogal and Dr. Bhatt, who Dr. Rogal does a lot of research in VA, which is, you know, the VA system is largely insured. Most people who show up there are insured by the VA. And Dr. Bhatt is from Canada, where there is a public healthcare system. So it would be interesting for their input into this, because the system there would really capture a lot more people, whether the depth is what we needed. Any thoughts, Mamta or Sherry? Yeah, sure. Yeah, in fact, it was a question that came up in my mind during Meena's talk, you know, with respect to the claims. So here in Ontario, there is a large database called the ICES or the Institute of Clinical and Evaluated Sciences. So that framework allows you to, say, retrieve data based on diagnostic codes that have been entered by physicians. And certainly there's no issue of claims and so on. So it's really based on diagnostic codes at the time of admission. And I suppose it's quite similar to, say, the VA system. So I think there we have, say, the access to blood test results and these outcomes, et cetera. So it would be very interesting to see whether, say, outcome prediction differs in a claims-based setting as opposed to, say, a public healthcare setting. And I suppose the VA is similar in that way. I did want to add on. Sorry. Just with respect to the minority and the socioeconomic status, like one interesting thing that you could do is just see whether, say, socioeconomic status or racial slash ethnic identity predicts. So as a ranked feature, whether that predicts the specific outcome. And then you can really dig deeply into seeing whether it's a specific ethnicity that is predicting an adverse versus a positive outcome. Yeah. And to kind of follow up on that, Dr. Lumba, who Free Advertisement is going to be speaking in the next part of this session, asks the question if it would be better from a global perspective to validate U.S. algorithms, or is it that the variables are so different in each country or region that we should be developing different models for different geographical regions? Like how much did the context matter? One thing on claims that I just wanted to add, you know, in certain systems, you get paid based on the severity and complexity of the patient. So there is an inherent incentive in some systems to code a lot. But if you're getting a lump sum payment, you're in some kind of like capitated model or if you don't bill, really, right, like you're not going to pick up all of those diagnoses codes because you don't have to, you know, kind of go through it. There's no need to do that. And so that's actually, you know. Like real estate location, location, location. And this sort of speaks to another question or a couple of questions just sort of about applicability of the different models. So one question from the chat was what is the cost of developing AI models and are there interoperability issues for wider use in clinical practice? And similarly, another participant asked whether to what extent you need a computational scientist versus just hepatologists. Not just hepatologists, we're all hepatologists. So I can answer, say, Rohit's question if that's all right. I thought that was answered already, like the global one. The global one? Okay. Yeah. I mean, in terms of... I think maybe because we are running a little short of time, maybe think about cost and the... Basically, this is how to start. If someone is starting, who all do we need to actually get in touch with to help us kickstart this operation? Yeah. Well, I would recommend reaching out to experts within your region. So say, you know, for example, I interact with experts at the University of Toronto and we have regular meetings. We have co-supervised students and postdoctoral fellows. So it involves, say, active engagement on both sides. So both parties have to be actively involved to, first of all, understand the language. So for me, it's been a learning process in terms of learning the lingo and the language in the last years. And then from their side, they're learning the clinical context. So it involves a very active crosstalk. And I think you'd be surprised, like I've felt in my experience, people are very open to interacting with clinicians. They're very excited to learn about the clinical problems that they could help solve. So I would encourage people to reach out to, say, local experts. And if not that, then reaching out beyond, because a lot of the world is now operating virtually. So a lot of research meetings can happen virtually as well. Amina, you agree? If you know more about AI than your collaborators, then you probably... No, I think you need collaborators. I mean, I think unless you are really, you know, you're a hepatologist, you know what clinically makes sense for the inputs, which will help in terms of the physician end of it, right, believing the model. Like if it's got all this stuff that you're like, what, what does that have to do with anything? You're going to have some kind of skepticism on the model. So I think it has to be a collaboration between the clinician hepatologist. And I mean, if you're double, you know, if you're also trained, wonderful. But I think you need partners in this. And I think it's important for people who are in the AI field listening to this, if they've somehow stumbled into this, please also make sure that, you know, you engage with us. Because I remember once Dr. Sanyal said that one of the medical record people were thrilled because they found 100% correlation between hemoglobin A1C and diabetes, and they were ready to bring out the champagne. But you want to make sure that this is something that gives them context as well. So people that we collaborate with spend an excruciatingly long time discussing with us as to what are the relevant features, what are the features that you can actually do, you could talk about. And I think a very important question from Dr. Lai is, is it worth it? It goes to Dr. Bansal's thing is, is it worth it in the end? Does it actually give us that much bang for the buck? If you put all of these together, and it actually ends up like, say, suppose, improving accuracy by 0.5, I guess it would be all dependent on context, I guess. Yeah, I think it depends on the context. And I think it, to some extent, it remains to be seen. I would think that if you have a more robust data set, in other words, bigger is not better, but if it has the detail that you need, I think that complete data, which I think is hard to get, but if you have complete data, I think it will bear fruit in terms of being better. It's kind of like, you know, getting that scalpel personalized approach. How much better I think remains to be seen for what outcome you're actually looking at. And that's why you need an area under curve, like comparison, like how good is it? Is it better than, you know, something much simpler or not? I think that remains to be seen. We did publish in the Red Journal that the readmission risk was similar to Mel's score. After all that, you know, but you had to compare it to Mel's score or something. You don't know until you know, right? Like, that's the thing. Is one test better than another? Is FIB4 better than, I don't know. Let's see. I'd like to comment on that further. So I think you can use established algorithms or established tools, but then computer scientists have that ability to generate or create a new tool depending on the clinical questions. So, you know, sometimes it may be an ensemble of different machine learning tools that are available out there. So say random forest, SVM, et cetera, like it may be a combination of those tools. The other thing is, I think the advantage of machine learning tools, especially the deep machine learning tools, is the ability to look at longitudinal data and patterns in the longitudinal data to inform a prediction. And that is also keeping in mind the different interrelationships and hidden patterns that are present in that longitudinal data. So I think, but nonetheless, I agree, you know, depending on the clinical question, it could be that an advanced biostatistical tool can perform as well as a machine learning tool. But that really is, I agree with Dr. Bunsalt, that that would be dependent on the clinical question. Okay. I think we have time for one more, but we are keeping track of all your questions and they will all be answered in the next session if you haven't gotten an answer yet. So just to keep people coming to the next session. So Dr. Chung asks a question about standardization. So to minimize heterogeneity of these datasets, shouldn't there be efforts to develop standards regarding the methods around omics acquisition and performance? He was specifically thinking about microbiome, but maybe true of other areas as well. Yeah, I think that's a great question, Dr. Chung. So I think this really demands a multicenter efforts and really coming together with a consensus as to what are the methods that would be used for a specific clinical outcome question. Like say, for example, the NASH CRN has come together as a multicenter effort, the cirrhosis network. So these are all networks that have the opportunity to really create that uniformity and minimize heterogeneity in the acquisition of omics data. I think the problem is like, say, for example, if you look at the gene expression omnibus, for example, there may be a variety of, say, arrays or RNA sequencing depth, et cetera, that might have been used. And so a bioinformatician will have great difficulty in, say, integrating all these different datasets. And that's a major problem. So I think, you know, having such multicenter efforts is really what is crucial to driving the field forward. Awesome. Well, thank you to everyone. Thank you to the speakers. And thank you to the audience for joining us in today's lively Q&A. We want to thank all the presenters for participating in their awesome presentations. And we want to just share a couple of facts. So if you didn't get your question answered, you should come to part two. But TLM DX offers a unique opportunity to network with your presenters through the platform. And then if you didn't have a chance to view the whole presentation, you can. It will be available online until February of 2022. We hope you enjoyed the session and we hope you enjoy the rest of the liver meeting. And we hope to see you in five minutes in the other room. Thanks, guys.

Artificial intelligence

Liver disease research

Datasets

Radiology

Pathology

Multi-omics

CNN

RNN

Predictive algorithms

Clinical decision support

Risk stratification

Data integration