The study utilized a between-subjects design, involving 1,298 participants divided across three treatment groups and a control, to evaluate decision-making in clinical scenarios. Participants assessed medical urgencies through clinical vignettes while using an LLM chat interface, focusing on real-life medical decisions. Before treatment, a demographic survey was completed, followed by a post-survey on experiences. Data collection, conducted via the Dynabench platform and Qualtrics, faced API issues affecting some participants, prompting adjustments in participant replacements. Each participant addressed two randomly assigned medical scenarios, assessing healthcare needs and reasoning, with responses scored against expert standards. Three leading large language models (LLMs)—GPT-4o, Llama 3, and Command R+—were employed to gauge effectiveness compared to the control, which lacked LLM assistance. Statistical analysis of the findings signifies the relevance of LLMs in healthcare settings. Detailed protocols received ethical approval from the University of Oxford, ensuring adherence to research standards.
Source link
