Fractional Chief Medical / Product / Clinical Safety Officer for HealthTech companies. Specialist in Clinical AI / GenAI
A great post from Stephen Gilbert which includes an explainer and link to his most recent paper in Nature Reviews: Cancer. As we are all no doubt aware, current medical device regulation essentialy prevents generalist AI and LLMs from being used in medical devices. Sure, lots of people are developing products for use in the healthcare setting, but they are chosen with great care to ensure they don't fall foul. Meanwhile, countless patients and clinicians are making use of LLMs in an uncontrolled manner. How do we square this circle? The paper, a neat 2 pages, covers this nicely. Perhaps we reign in LLMs to narrow use cases? Maybe we develop a mandatory human in the loop solution and wrestle with automation bias? What IS clear is that blocking GenAI in healthcare isn't a long term strategy.
Professor - Medical Device Regulatory Science, News and Views Editor, Nature Portfolio - Digital Health
Just published in Nature Reviews Cancer https://rdcu.be/dESF9 (Stephen Gilbert & Jakob Nikolas Kather Else Kröner Fresenius Center for Digital Health): How large language models (LLMs) / large multimodal models in health are and will be regulated and be implemented is one of the most polarised arguments of our time ..(in my bubble) You should read this whether you think that LLMs (and related general AI models, foundation models and generalist AI approaches to medical diagnosis and treatment) are peaking in their hype cycle, or, you think we are just seeing the peak of the iceberg of their potential (my view). - There is increasing experiential and published evidence of these approaches being highly practically useful for general purpose reasoning – both non-medical and medical - in task we would traditionally have considered 'intelligent' tasks, including 'zero/one-shot' reasoning - Naturally there are philosophical arguments of whether these have any potential in 'general artificial intelligence'. This is not the focus of the article which concentrates on the practical utility of models capable of generalist tasks in medical decision support - Current medical device regulatory frameworks, that already apply to these approaches https://rdcu.be/dfLOJ, do not allow generalist medical reasoners in medical decision support. These are excluded as the international medical device frameworks require specific pre-design for tasks and do not allow interaction with 'zero/one-shot’ intelligent reasoning. Required are human-designed devices, conceived by the human mind, for specific (usually narrow) tasks - Our argument is that intelligent reasoning approaches in medical decision support should not be excluded, by default, through regulations developed before they were conceived possible - THIS IS NOT TO SAY THAT THESE APPROACHES SHOULD NOT BE REGULATED. Rather that intelligent decision support using multimodal data, for person-specific medical support questions, should not be a priori excluded by regulatory frameworks - We discuss the implications of regulatory blocking the access of doctors to such approaches, which will be available to patients to use on their own medical data (whether available as certified medical device products or not, as these technologies will filter through fast open international internet access) - this point of view may be attacked from either side - those with the view that we have seen no shift in the development rate of these technologies in the last 2 years and those that think there should be no regulation at all - Although generalist medical artificial intelligence approaches may not (quite) be ready for centre stage, nonetheless, the approach of regulations should not be 'you shall not pass' on the basis regulatory frameworks developed prior to the concepts in question. - regulation is a human creation and needs to be fit for purpose and address new and large challenges head on for current and approaching technologies
CEO | A Healthier Democracy | Physician
3moKeith Grimes 👍🏽, your discussion on the challenges of integrating generalist AI and LLMs in medical devices is both timely and critical. The juxtaposition of strict regulations with the uncontrolled use of LLMs by clinicians and patients presents a significant dilemma.