Study: ChatGPT gave ‘mostly correct’ answers to questions about cardiovascular disease prevention

ChatGPT provided “mostly relevant” answers to questions about cardiovascular disease prevention. research letter published in JAMA.

The researchers compiled 25 questions about key concepts in heart disease prevention, including risk factor counseling, test results, and drug information. They asked questions to the AI ​​chatbot three times, and each set of answers was scored by a doctor.

The clinician rated responses as relevant, inappropriate, or unreliable if the chatbot responses differed in each set. The reviewers also evaluated responses in two contexts: first as a patient-facing platform and then as draft responses to patient messages sent to the physician for review.

The study found that 21 out of 25 questions were considered appropriate in both contexts, and four were rated as inappropriate in both scenarios. In three of the four sets of inappropriate answers, all three answers were incorrect, and in one set there was only one inappropriate answer.

WHY IS IT IMPORTANT

The researchers noted several limitations in their analysis. Currently, the chatbot is not intended for medical use, and cardiovascular disease prevention cannot be covered with just 25 questions.

They said that in future studies, investigators could use more reviewers to rate responses, or create a formal response scoring system that is less dependent on the subjective opinion of the clinician.

“The results show the potential of interactive AI to assist clinical workflows by expanding patient education and communication between patients and physicians on general issues of cardiovascular disease prevention. For example, such an application may provide interactive responses to simple queries on information platforms or generate automatic draft responses for patients. emails for clinicians,” they wrote. “It should be explored whether these approaches can improve readability, as previous studies have shown poor readability of some online CVD patient materials.”

BIG TREND

There are a number of companies promoting chatbots for medical purposes. Wysa, which creates a chatbot that helps users get cognitive behavioral therapy for problems such as low mood, anxiety and stress. raised $20 million last summer.

The other company behind the mental health chatbot, Woebot Health, received a $9.5 million investment from Leaps by Bayer almost a year ago. In 2021, Woebot received FDA breakthrough device designation for digital therapy for postpartum depression.

Symptomatic check of the chatbot Ada Health, which announced the The $30 million Series B expansion in February 2022 recently launched a digital COVID-19 care journey co-developed with Pfizer.