Researchers from Torino, Italy, utilized Chat Generative Pretrained Transformer (ChatGPT), one of the most widely used chatbots, developed by OpenAI, to generate nutrition advice using prompts mimicking typical user questions to investigate the accuracy of advice provided.
Results showed that ChatGPT offered mostly clear advice, however, when the chatbot handled overlapping conditions, limitations emerged, resulting in some contradictory or inappropriate advice, the authors noted in the Journal 'Nutrients'.
“Although ChatGPT exhibited a reasonable accuracy in providing general dietary advice for NCDs, its efficacy decreased in complex situations necessitating customized strategies,” they concluded.
ChatGPT reliability
NCDs account for 74% of global deaths, with cardiovascular diseases (CVD) being a major contributor: According to global estimates, there are approximately 43.8 million cases of type 2 diabetes (T2DM), 18.5 million cases of hypertension and 1.2 billion cases of non-alcoholic fatty liver disease (NAFLD) worldwide.
Lifestyle and dietary factors play a crucial role in NCDs, and online health information searches are on the rise.
ChatGPT, which uses machine learning algorithms to generate responses to text-based queries, mimicking human-like conversation, is gaining popularity as a source of personalized and immediate medical advice.
There is growing application of artificial intelligence (AI) in the domain of patient education and support, and an increasing interest among healthcare professionals to explore the tool's potential to enhance patient care, diagnosis and treatment.
However, there is limited literature on the use of ChatGPT for improving nutrition knowledge in patients with different chronic NCDs.
One recent study explored the potential of ChatGPT in delivering nutritional information and concluded that it could not replace the expertise of a registered dietitian, particularly when addressing complex medical conditions, while other research found that the chatbot performed on par with human dietitians in delivering accurate answers.
Therefore, the objective of the study was to compare the nutritional information supplied by ChatGPT (version 3.5) regarding different NCDs requiring dietary advice with the nutritional recommendations from international guidelines.
Assessing nutritional guidance
During November 2023, the researchers assessed the appropriateness of dietary advice given by ChatGPT through the formulation of a set of prompts for various conditions: Dyslipidemia (hypercholesterolemia and hypertriglyceridemia), arterial hypertension, type 2 diabetes mellitus (T2DM), obesity, NAFLD, chronic kidney disease (CKD) and sarcopenia.
Questions were designed by a panel of experts (medical doctors and registered dietitians) using language and sentence structure that replicated how patients might inquire with a healthcare professional.
Three prompts were tested for each medical condition: “Could you provide guidance on planning an optimal diet to manage [disease]?”; “What are the dietary recommendations for [disease]?”; and “I have [disease], what should I eat?”.
The dietary advice given by ChatGPT was compared with nutrition guidelines, and then the chatbot’s capacity to manage a complex case was investigated by introducing more intricate scenarios involving the coexistence of multiple conditions.
The expert panel assessed and designated ChatGPT’s responses as “appropriate” if the content aligned with the guidelines, “inappropriate” if in contradiction of the guidelines, “not supported” if not confirmed in the guidelines, and “not fully matched” if the responses did not fully fulfill the guideline recommendations.
ChatGPT advice that was not specifically focused but promoted an overall healthy diet was classified as “general advice”. Any dietary guideline recommendation that was not present in the ChatGPT response was labeled as “missing”.
Results showed that overall, ChatGPT offered clear advice, with the appropriateness of responses ranging from 55.5% (sarcopenia) to 73.3% (NAFLD).
Including both categories of “appropriate” and “general advice”, the overall accuracy of nutritional recommendations reached 100% for sarcopenia.
However, two recommendations contradicted guidelines: one conflicting with mealtime and snack guidelines from the European Association for the Study of Obesity; another advising supplements such as vitamin E, omega-3 fatty acids and antioxidants in NAFLD, while the European Society for Clinical Nutrition and Metabolism (ESPEN) assert that these supplements cannot be endorsed while further data on efficacy is pending.
A single suggestion for T2DM was found to be “unsupported”, while many recommendations for various NCDs were deemed to be “not fully matched” to the guidelines despite not directly contradicting them.
Additionally, the researchers found that limitations emerged when the chatbot handled overlapping conditions, resulting in some contradictory or inappropriate advice.
"Our findings indicated that the chatbot was unable to combine different recommendations, which resulted in contradictory or inappropriate advice that could have confused users," the authors concluded.
The authors also noted that the use of ChatGPT in nutrition education could be limited by the patient’s digital literacy, and while younger individuals may readily embrace technology, older patients may not be as willing to do so and might encounter difficulties understanding the platform.
Journal: Nutrients
doi: https://doi.org/10.3390/nu16040469
“Is ChatGPT an Effective Tool for Providing Dietary Advice?”
Authors: Ponzo, V. Et al.