AI-based attempt to find supplement safety signals in Twitter critically flawed, expert says

The new research is the work of a team of professors in bioinformatics at the University of Minnesota and the University of Florida.  The research was published recently in the journal JAMIA Open​.

The authors’ stated purpose was “to develop a deep learning pipeline to detect signals on dietary supplement-related adverse events (DS AEs) from Twitter.”​

More than 200,000 tweets analyzed

To do this they looked at 247,807 tweets ranging from 2012 to 2018 that mentioned both dietary supplements and adverse events. The researchers narrowed this down to 2,000 tweets which they then subjected to further comparative analysis to find the best tools to apply to the entire data set.

The authors compared the performance of several neural language models to analyze the tweets.  They divided this into two separate tasks.  The ‘concept extraction task’ is were terms that correspond to dietary supplements and adverse events were identified.  In the second, the  ‘relation extraction task,’  the relations between these terms were identified.

A neural language model called DeBERTa-CRF performed best on concept extraction while another referred to as RoBERTa proved superior at relation extraction.

The team then assembled what they called “an end-to-end deep learning pipeline”​ to detect adverse event signals for dietary supplements within the tweets using the two neural language models.  The assertion is that they found adverse events that were not recorded elsewhere, by comparing the results of the latest study to a database assembled by some of the same researchers called iDISK (Integrated Dietary Supplement Knowledge Base)​​.

Samples of tweets the researchers said according to their analysis implied dietary supplement adverse events were:

• “Fish oil does not help or prevent heart disease or Alzheimers. It *does* increase prostate cancer. Do not take it” ​
• “Melatonin sure does help me sleep but it also causes some really trippy dreams” ​
• “Too much vitamin C or zinc could cause nausea, diarrhea, and stomach cramps. check your dose” ​
• “Vitamin D2 is a patented drug similar to vitamin D, but is not natural. It's been responsible for the majority of toxicity from vitamin D.”​

In addition to what the researchers asserted was evidence of adverse events from taking supplements, they also claim to have found what they called ‘deficiency AEs’ or tweets that showed the Twitter user suffering a health consequence from not​ taking a supplement.

“We found that our proposed deep learning pipeline not only retrieved DS AEs and DS indications recorded in the current DS knowledge database but also discovered DS AEs and DS indications that were only reported in tweets,” ​the researchers concluded.

Expert: Approach has fundamental flaws

Rick Kingston, PharmD, is a clinical professor in the School of Pharmacy at the University of Minnesota as well as being president of scientific and regulatory affairs at SafetyCall International, which helps drug makers and supplement firms monitor adverse events and manage recalls.  At the request of NutraIngredients-USA Kingston weighed in the research. 

Kingston said many of his clients do monitor Twitter for various purposes, but finding verifiable adverse event information there is not one of them.  Even properly reported adverse events, either reported to manufacturers or directly to FDA, often contain incomplete or incorrect information. The content of a tweet, often fired off with mere seconds of contemplation and with no guarantee the originator has any direct knowledge of or has ever consumed the product in question, would be of no value in this context, in his view.

“The authors state ‘Currently, to the best of our knowledge, while there are many pipelines that extract and identify drug AEs, there is no existing pipeline to extract and identify DS AEs from Tweets.’ This is for good reason.  It takes reliance on adverse event hearsay to a whole new level,” ​he said.

Mischaracterization of adverse event reporting

Kingston said the way the authors characterized how adverse events are currently handled for dietary supplements reveals a fundamental misunderstanding of the system.

“It’s not a ‘voluntary’ process for manufacturers that have mandatory reporting obligations to report to FDA any alleged AE meeting the criteria for Serious Adverse Event (SAE),”​ Kingston said.

“​Adverse event data collection for dietary supplements has both strengths and limitations not discussed by the authors.  Social media has specific challenges and is not typically an opportunity for a two way street of information sharing between reporters and those documenting or mining such data.  Consumers are typically reluctant to share sensitive medical history information as well,”​ he added.

“​Lastly, this model has no adjustment for market penetration and as such there they have no denominator to do reasonable comparisons.  Frankly, I think this model has serious flaws as a safety signal generating tool,”​ Kingston concluded.

Source:​ JAMIA Open​​​
Volume 4, Issue 4, October 2021, ooab081, https://doi.org/10.1093/jamiaopen/ooab081
Deep learning models in detection of dietary supplement adverse event signals from Twitter​
Authors: Wang Y, et al.