AI Health Prediction Models Built on Unverifiable Datasets Raise Serious Concerns

Study Uncovers Troubling Gaps in AI Health Data Provenance

Artificial intelligence models designed to predict health risks such as stroke and diabetes may be built on datasets whose origins cannot be verified, according to new research published in BMC Medicine. The study, led by researchers at Queensland University of Technology (QUT) and the Australian Center for Health Services Innovation (AusHSI), examined two widely downloaded health datasets hosted on Kaggle, a popular online platform for sharing datasets and machine-learning resources. The findings highlight a critical flaw in the foundation of some AI-driven clinical tools.

Datasets Used in Over 125 Peer-Reviewed Studies

The two datasets in question were found to have been used in 125 peer-reviewed studies, despite providing almost no information about where the data came from, how they were collected, or whether they represented real patients. Lead author Alexander Gibson from QUT’s School of Public Health and Social Work and AusHSI expressed shock at the discovery. “It was an enormous surprise to come across something like this,” Gibson said. “These datasets exhibit unusual patterns that raise serious questions about their authenticity and suitability for clinical research.”

Clinical Impact and Patent Citations

Three prediction models based on the data showed evidence of use in clinical practice. One model was cited in a medical device patent, and the models were referenced in 86 review articles. This suggests that despite the questionable provenance of the underlying data, these models have influenced real-world medical decisions and innovations.

Zero Score on Essential Data-Provenance Criteria

The study assessed the datasets using the internationally recognized TRIPOD+AI reporting framework, which evaluates the transparency and completeness of prediction model studies. The datasets scored 0 out of 9 on essential data-provenance criteria, indicating a complete lack of verifiable information about their origins. Gibson warned that this should be a red flag for journals, developers, and clinicians. “Prediction models built on data of unknown provenance have no place in clinical decision-making. Without trustworthy data, the outputs are unreliable and risk misleading clinicians and harming patients,” he said.

Call for Stronger Disclosure Requirements

The authors recommend that journals, funders, and data repositories strengthen requirements for data-source disclosure. They also suggest that the two Kaggle datasets be removed to prevent further misuse. Seven articles that used these datasets have already been retracted from journals for being unreliable. The results of the study have also updated the Collection of Open Science Integrity Guides, which provides resources for ensuring research integrity.

Broader Implications for AI in Healthcare

Gibson noted that the issue reflects a broader challenge as AI tools proliferate in healthcare. Without robust data provenance standards, the risk of deploying flawed models into clinical practice increases. The study underscores the need for rigorous validation of datasets before they are used to train AI models that could affect patient outcomes.

Recommendations for the Field

• Journals should require detailed data provenance information for any study using AI prediction models.
• Funders should mandate transparency in data collection and sharing practices.
• Data repositories like Kaggle should implement verification processes to ensure datasets meet minimum provenance standards.
• Clinicians should be cautious about adopting AI tools without clear evidence of data reliability.

The findings serve as a cautionary tale for the AI and medical communities, emphasizing that the promise of AI in healthcare can only be realized if the data underpinning these models is trustworthy.

This article is based on reporting by Medical Xpress. Read the original article.