The latest study on ChatRWD™ beta has shown that it surpasses Big Tech LLMs in physician trust and the ability to provide complete answers with high-quality evidence. The results, published in a preprint on arXiv, highlight ChatRWD's superiority in delivering reliable and accurate real-world evidence for healthcare.
LLMs have increasingly been considered as potential sources of evidence, summarizing information from prior literature learned during LLM training. While LLMs have displayed impressive performance in responding to natural language queries in various medical domains, they are prone to hallucinating reference materials or treatment guidelines and may produce “recommendations” that are inappropriate.
Atropos Health tested ChatRWD beta, OpenEvidence, and three “out-of-the-box” LLMs (ChatGPT-4, Claude 3 Opus, and Gemini Pro version 1.5) to answer the same clinical questions context of care delivery.
Key Findings: ChatRWD beta outperformed other LLMs with a remarkable 94% completion rate. Independent physician reviewers found ChatRWD beta to provide the best answer to 60% of the questions. Read the full paper to see more results at arXiv.
Atropos Health's Chief Medical Officer and co-founder, Dr. Saurabh Gombar, emphasized the importance of rigorously measuring and verifying the reliability of AI technologies in healthcare. ChatRWD's ability to generate evidence for both well-studied and novel questions sets it apart.
“Built on the core technology GENEVA OS™ (Generative Evidence Acceleration Operating System), ChatRWD is not tied to any single LLM ,” said Neil Sanghavi, President of Atropos Health. “GENEVA OS is deployed locally...and it does not expose any patient-level clinical or claims data to LLMs; the LLMs in ChatRWD are used to create the query and design the study, but GENEVA OS is used to execute the query - without any connection to the LLMs.”
The full launch of ChatRWD is on the horizon, promising to democratize access to transparent, reliable, and personalized real-world evidence for all. Read more and sign-up for the ChatRWD beta program at https://hubs.ly/Q02F0Kvk0
#AtroposHealth #ChatRWD #RealWorldEvidence #HealthcareInnovation #AIinMedicine #LLM #LargeLanguageModels
