Combatting Hallucinations in FDA’s AI

One of the goals of the current administration is to implement the use of AI in FDA’s approval process. Proponents of the policy argue that the technology can increase the speed at which FDA can review materials and approve new drugs. Critics, however, worry both that the technology will be used to replace existing workers at the agency and also that the results will be unreliable. Marcel Botha, CEO of 10XBeta, spoke with Pharmaceutical Executive about the technology and the risks and benefits of its use at FDA.

PE: How can FDA prevent hallucinations?
Botha: We can't have our AI do that when it comes to critical analysis of core ingredients and component structures that are required. These are elements where a slight deviation makes something safe or not. There's a degree of testing that that needs to happen before you can release this as a fully-fledged bot. We've seen many of the consumer AI applications gone through both private and public testing and required significant rework.

I think that the data sets are important. From the conversations I've had with professors at CSAIL and MIT, we were all preoccupied with a large language model in healthcare and how it should really be the opposite of what other LLMs are like: a small data model that's hyper specific to the application that we're solving for. That means not ingesting data that is not relevant and training our AIS only on data that's relevant to the science. The specific application that we are scrutinizing creates some of these anomalies, and having the human-in-the-loop is still a significant requirement for validating or verifying the quality of the FDA scrutiny process.

We must find the balance between how we train the model, determining how much data that's relevant to the model, and the what the purpose of the human loop is if we want to get back to the pandemic warp-speed-approval-criteria. There are other ways to shorten the process, such as limiting the amount of human testing from two cycles to one run of testing. But again, that that imposes risk.

FDA requires more post market surveillance for drugs that might have an outlier risk that's not been fully quantified or captured. Post-market-surveillance and feeding that back into the learning models might become a valuable source of future data.