A unique workshop co-hosted by HealthTech Build & the Martin Trust Center for MIT Entrepreneurship
Facilitators Leo Anthony Celi, Dawei Liu, and Erez Kaminski led deep dive discussions in three breakout sessions.
Leo Anthony Celi
Principal Research Scientist at the MIT Laboratory for Computational Physiology, ICU Physician at Beth Israel Deaconess, Associate Professor of Medicine at Harvard Medical School
Discussion highlights: The Harms of AI Bias
- A key challenge to deploying AI in healthcare is the amount of bias that exists in AI data sources and the AI development process.
- Large Language Models (LLMs) can provide especially poor medical assistance as they are commonly trained on biased data. For example, a CT scan may often be recommended if you are white, but if you are black, the same CT scan may not be recommended.
- The underlying data sets, such as electronic health records and clinical trial data used to create AI, often come from a single predominant population (i.e., white and/or male). For instance, the Framingham Heart Study is used all over the world, yet it could be very inaccurate for a person living in Uganda. The data is also shaped by the biased decision-making of the people who input the data.
- Part of the problem is that data scientists designing AI have a short-term cycle. The validation of the data model isn’t sufficiently valued; the focus is on trees and neural nets. There should be more questions asked about which populations might be missing from the database or how the clinical decisions in the database might be biased. Moreover, the data scientists who built the AI may not be involved when it’s actually deployed in the real world among diverse populations.
- To fix this, we need a better recognition of the extent of the problem and new approaches. We need more experiments with data sets, for example, generating simulated databases where the world is fair.
- We also need to consider the AI developers and ask if their lived experiences are diverse or if they share the same blind spots as the people who generated the data. One solution is to encourage AI teams to include a social scientist in their group, which can help them ask the right questions.
Dawei Liu
Global Artificial Intelligence Lead at Olympus
Discussion highlights: Challenges of AI data
- There are significant challenges to building AI systems to assist with image recognition. For example, polyp recognition in colonoscopy screenings is immensely challenging because it involves bringing together disparate and inadequately labeled data sets.
- Different companies have manufactured the machines in current operation for colonoscopies over a period of decades. The scopes employ different software to generate and store the images in different ways and often these image files will not indicate which manufacturer and model was used to generate them. The hospitals and clinics that operate the machines also have varying ways to store the imagery data. It is typically stored separately from medical records data that could provide context about the patient and identification of polyps from the scan.
- These image files are very large and contain an immense amount of data, while only a very small amount of the video will actually show the moment that a polyp is identified.
- For the AI researcher who wants to aggregate a colonoscopy data set, the process might start with a series of negotiations with individual clinics to acquire access to the data, conditional on compliance with patient consent and perhaps other requirements of an IRB.
- It’s important to know what population was used to create the data set so that steps can be taken to reduce bias. Once the data set has been aggregated, experienced colonoscopy readers must review the images and determine which images show polyps.
- Typically the most experienced physicians are not involved in this work and multiple steps of validation may be needed. In fact, the validation process is likely more expensive than the entire process of AI software creation, especially in healthcare which require human (rather than machine) validation and compliance with local and federal regulations.
Erez Kaminski
CEO and Founder of Ketryx Corporation
Discussion highlights: FDA Regulation and AI
- In recent years, the systems which the FDA reviews and regulates are becoming more complicated, thus the regulatory submission process is becoming more complex.
- At its essence, the FDA wants to have evidence that your device does what you say you say it does, which means identifying and documenting how patient risks are managed.
- The validation of an AI system includes:
-Validation – Providing objective evidence that your system does what it says it does
-Risk-based approach – Telling the FDA what’s riskiest and documenting how these risks are addressed
-Continuous integration and deployment (CI/CD) – This step is very challenging since the FDA approval will require very extensive documentation that needs to be updated and tested with each deployment
- To date, nearly all the AI devices that have been approved by the FDA have been in radiology imaging.
- Large Language Models (LLMs) are in the early stages with the FDA and it’s still unclear where these models will fit. It’s likely that LLM healthcare models initially will be for very narrow medical areas that can be clinically validated.
