How Accurate are Cardiac Risk Calculators?

Review

DeFilippis, AP, et al. Analysis of Calibration and Discrimination Among Multiple Cardiovascular Risk Scores in a Modern Multiethnic Cohort. Annals of Internal Medicine. 2015 Feb; 162:266-275.

Methods

This prospective epidemiological study compares the capabilities of the new 2013 American Heart Association (AHA) and American College of Cardiology (ACC) ASCVD risk calculator against several alternative cardiovascular risk calculators and explores the potential effect of preventive therapy on risk overestimation in the AHA-ACC-ASCVD model.

The 4227 participants in this study were from an age- and sex-balanced, multiethnic cohort in the Multi-Ethnic Study of Atherosclerosis (MESA). To meet parameters of the risk calculators of interest, age range of the study population was limited to 50-74 years and all participants were free of cardiovascular disease (CVD), coronary heart disease (CHD), and diabetes at baseline.

Participants were followed every 9-12 months by phone for a median of 10.2 years for specific endpoints of myocardial infarction; definite or probable angina; resuscitated cardiac arrest; stroke (not transient ischemic attack); or death due to CHD, stroke, atherosclerosis, or other CVD. These observed events were compared against their respective predicted values, as generated from 5 risk calculators — 3 tools based on the Framingham Risk Score (FRS-CHD, FRS-CVD, and ATPIII-FRS-CHD, CVD), the Reynolds Risk Score (RRS), and the new AHA-ACC-ASCVD.

To explore the effect of preventative therapy on risk estimation, further analysis was performed including only participants not receiving any cardiovascular preventive therapy (use of aspirin, anti-hypertensive medications, lipid-lowering agents, and revascularization.)

Results and Outcomes

The authors used discordance as a primary metric of how well the estimated and observed numbers of cardiovascular events compared against one another. Greater discordance signified a larger gap between the two values and vice versa.

In male participants, all risk calculators overestimated the cardiovascular end points they were designed to predict, at all levels of cardiovascular risk. The greatest level of discordance in men was found using ATPIII-FRS-CHD (37-154%), meaning the predicted number of events could be more than 2.5 times that of the observed number. The lowest level of discordance was found using RRS (9%). The new AHA-ACC-ASCVD tool overestimated risk by 86%.

In female participants, the FRS-CHD, FRS-CVD, ATPIII-FRS-CHD, and AHA-ACC-ASCVD calculators also overestimated risk at all levels of cardiovascular risk. In contrast, the RRS underestimated risk by 21%. The levels of discordance were found using 8% FRS-CVD and 67% using the AHA-ACC-ASCVD .

In participants with AHA-ACC-ASCVD risk scores between 7.5%-10% the AHA-ACC-ASCVD calculator produced a 186% and 71% overestimation in risk among men and women, respectively. This was of particular significance as the 2013 AHA-ACC-ASCVD guidelines consider 7.5% to be the threshold for initiating medical therapy.

Using another metric, the C-statistic, the authors also showed little difference in the ability amongst all 5 methods to discriminate between individuals having and not having an event.

As for the question of preventive therapy, when the analysis was limited to participants not receiving any therapy at baseline and all follow-up visits, an overestimation of cardiovascular risks occurred for all risk scores.

Conclusions

The 5 risk scores examined show no significant difference in discrimination capabilities and 4 of these models, including the new AHA-ACC-ASCVD, overestimate cardiovascular risk in an age- and sex-balanced, multiethnic cohort without baseline CVD. The use of preventive therapy does not appear to correlate with risk overestimation and importantly, risk overestimation within the new AHA-ACC-ASCVD model may have implications on early and/or overtreatment of patients for CVD risk.

Commentary

As the leading cause of mortality in adults, CVD is a widely prevalent disease in the United States, accounting for 1 out of every 4 deaths (1). As such, there is a major emphasis on developing tools to objectively and accurately assess the risk for developing CVD. The FRS-CHD was originally designed with this purpose incorporating known risk factors such as age, sex, diabetes, blood pressures, cholesterol levels, and smoking (2) to evaluate for 10-year cardiovascular risk. Since then, reiterations of this model and alternative risk scores have been developed, taking into account different composite variables including stroke, parental history of premature CHD, and high-sensitivity C-reactive protein.

Most recently, AHA-ACC-ASCVD model was designed using data from racially and geographically diverse cohorts. This led to several perceived improvements such as separate risk calculators for distinct patient groups, such as African Americans, Caucasians, males, and females. Despite this, several investigator groups have identified a potential issue with the AHA-ACC-ASCVD risk score to be the overestimation of risk in certain cohorts (3, 4, 5, 6).

In this study, the calibration of this new risk estimator was further evaluated in comparison to alternate models using a large cohort of over 4000 individuals with diverse ethnic composition, including white, African-American, Hispanic, and Chinese. The length of the study is ideal for evaluating the risk score models of interest, as all are designed to predict 10-year risks for cardiovascular events. This reduces the likelihood of any falsely low rates of observed events While cardiovascular events were assessed initially by self-reporting, the authors validated these outcomes with copies of medical records, and when possible, the Centers for Medicare and Medicaid billing database. Per authors, no more than 9% - a high-end estimate — of myocardial infractions and strokes could have been missed among study participants.

A major potential limitation of this study is its external validity. All participants in this study were without any CVD at baseline, making the cohort much healthier than the general population. This could have led to a uniquely low incidence of observed cardiovascular events and subsequent over-exaggeration of risk. The authors acknowledge this possibility and try to reconcile that similar selection biases for healthy cohorts can be found in other published studies, from which risk score models are derived.

Another potential caveat lies in the exclusion of patients with diabetes. The authors justified this decision on the basis that 2 of the 5 risk score models were not designed to assess such patients. Given that diabetes is a significant cardiovascular risk factor, this exclusion could have both altered the effectiveness of the other 3 risk score models and artificially lowered the observed incidence of cardiovascular end points obtained.

The authors attempt to address this pitfall by showing inconsistent effects on risk score calibrations following inclusion of participants, who screened positive for diabetes. These individuals, however, were screened on the basis of hemoglobin A1C (HbA1C) calculations extrapolated from baseline blood glucose measurements as opposed to directly measured HbA1C levels — a more reliable method to diagnose diabetes, not obtained in this study.

Lastly, the geographic localization of the cohort to 6 distinct communities in the United States adds to the possibility that participants enrolled in this study may not be a representative sample of the general population.

Overall, this study provides much needed insight into current and prior models for assessing CVD. Specifically, it reveals that the majority of our available models, including the most recent AHA-ACC-ASCVD, favor sensitivity (identifying high risk) over specificity (identifying low risk) in determining CVD risk. Further investigation is necessary to explain how and why these overestimations, if validated, exist. A reasonable next study may be to examine more closely the effect of specific variables, namely diabetes, on calibration and discrimination capabilities of various risk assessment tools. Alternatively, a similar study using more homogeneous subgroups, e.g. by race/ethnicity, instead of a heterogeneous composite cohort could provide useful information to help refine our current CVD risk stratification models.

On a larger scale, this study also raises important questions about how we approach risk as a medical community. This paper emphasizes the calibration of CVD risk stratification models can affect the tenuous balance between over- and under-treatment of heart disease.

For example, under the new AHA-ACC-ASCVD guidelines, many individuals may be needlessly taking medications, all of which carry significant adverse effects including bleeding (from aspirin) and debilitating myopathies, as well as self image (having an increased risk) and insurer bias. The the cost of such overestimation can be a financial burden on our patients and healthcare system. Statins, oft viewed as benchmarks of cost-effectiveness in secondary prevention, are not nearly as cost-effective as agents for primary prevention (7, 8). On the other hand, data from this paper also suggest if a model like RRS were to be adopted, there would be a much lower rate of overestimation for heart disease. As a result, fewer patients would fall victim to iatrogenesis, yet many more might slip through the cracks untreated, leading to an accumulation of easily preventable morbidity and mortality.

The process of optimizing risk scores is akin to the classic dilemma of “picking your poison.” There does not appear to be a good solution as of now and efforts to better understand the mechanics underlying our current risk stratification models are certainly necessary. Equally as vital is a clearer picture of what the treatment goals should be for CVD prevention. Should we lower our threshold for treatment at the cost of exposing healthy people to medications or reserve therapy — and its associated harms – for only those with definite disease? These are only a couple of the questions that warrant meaningful consideration. After all, a map can only be so useful without a destination in mind.

References:

1. CDC, NCHS. Underlying Cause of Death 1999-2013 on CDC WONDER Online Database, released 2015. Data are from the Multiple Cause of Death Files, 1999-2013, as compiled from data provided by the 57 vital statistics jurisdictions through the Vital Statistics Cooperative Program. Accessed Feb. 3, 2015.

2. Wilson PW, D’Agostino RB, Levy D, Belanger AM, Silbershatz H, Kannel WB. Prediction of coronary heart disease using risk factor categories. Circulation. 1998;97:1837-47. [PMID: 9603539]

3. Ridker PM, Cook NR. Statins: new American guidelines for prevention of cardiovascular disease. Lancet. 2013;382:1762-5. [PMID: 24268611] doi:10.1016/S0140-6736(13)62388-0

4. Goff DC Jr, Lloyd-Jones DM, Bennett G, Coady S, D’Agostino RB, Gibbons R, et al; American College of Cardiology/American Heart Association Task Force on Practice Guidelines. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation. 2014;129:S49-73. [PMID: 24222018] doi:10.1161/01.cir.0000437741.48606.98

5. Kavousi M, Leening MJ, Nanchen D, Greenland P, Graham IM, Steyerberg EW, et al. Comparison of application of the ACC/AHA guidelines, Adult Treatment Panel III guidelines, and European Society of Cardiology guidelines for cardiovascular disease prevention in a European cohort. JAMA. 2014;311:1416-23. [PMID: 24681960] doi:10.1001/jama.2014.2632

6. Muntner P, Colantonio LD, Cushman M, Goff DC Jr, Howard G, Howard VJ, et al. Validation of the atherosclerotic cardiovascular disease Pooled Cohort risk equations. JAMA. 2014;311:1406-15. [PMID: 24682252] doi:10.1001/jama.2014.263

7. Brandle M, Davidson MB, Schriger DL, Lorber B, Herman WH. Cost effectiveness of statin therapy for the primary prevention of major coronary events in individuals with type 2 diabetes. Diabetes Care. 2003;26:1796-801. [PMID: 12766112]

8. Mitchell AP, Simpson RJ. Statin cost effectiveness in primary prevention: a systematic review of the recent cost-effectiveness literature in the United States. BMC Res Notes. 2012;5:373. [PMID: 22828389] doi:10.1186/1756-0500-5-373