Suicide Prediction Models Show Significant Innaccuracy

A recent review of suicide prediction models (SPMs) has determined that, despite advances in technology and methods, they currently offer limited practical application in predicting suicide mortality.

The review examined results from 64 unique prediction models across 5 countries with more than 14 million participants and concluded that advances in statistical modeling have done little in the way of improving SPMs. For federal organizations, such as Veterans Affairs or the Department of Defense, these models could report inaccuracies up to 99%.

A team of military and civilian based investigators sought to compare the predictive validity of published models and to explore the impact of implementing SPM in a large health system. They screened records from 7306 abstracts and identified 43 articles for review. Of those 43, 17 met their eligibility criteria.

Criteria included that all articles be peer-reviewed, formatted in English, focused on populations that were 18 years or older, and had target outcomes of death by suicide or suicide attempts. Investigators stated that due to the strict inclusion criteria and low risk of bias across studies, the quality of evidence should be considered high and represent the current state of science on SPMs.

After analysis of the available information, investigators came to the conclusion that while statistical modeling has advanced, it is still far from being applicable in accurately predicting suicide. They determined that while research is growing due to advancements in analytics and efforts to pool large data sets, performance of models suggests prediction models offer limited practical use. Predictive models that were used in the military, Veterans Affairs, and civilian health systems all had notable flaws.

In Veterans Affairs and Department of Defense models, which accounted for a total 7 of the SPMs examined, only 25% to 50% of individuals who died by suicide would be correctly identified as at-risk prior to death. Further attesting to the investigators’ conclusion is that 99 out of every 100 individuals predicted to die by suicide will not.

In addition to inconsistency in accurately predicting suicide rates, the investigators felt that falsely identifying an individual as at-risk for suicide could lead to potential adverse psychosocial and health system implications.

In regards to military personnel, whose health conditions are often known to their chain of command, potential negative effects could present concerns about how that information will be used and what effects it could have on one’s career, family life, and social network.

Investigators added that future studies should include model targeting more common outcomes, and that any predictive model that demonstrates statistical feasibility needs to be evaluated within a clinical research framework. Prior to testing new models, they advised that clinicians should determine what is an acceptable positive predictive value in order to determine if the model should be pursued.

“The increasing availability of large health care data sets coupled with advances in machine learning analytics have introduced opportunities to develop and pilot advanced SPMs that can analyze diverse, large-scale data set elements to potentially improve classification accuracy,” they wrote.

A corresponding author did not respond to requests for comment at the time of publication.

Investigators ultimately concluded that the review led to a comprehensive summary of the current state of the evidence on SPMs. While the idea of having a prediction model that can accurately analyze data to classify patients into a suicide risk rating is ideal, findings from the review show sizable limitations of current SPMs.