If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Validation of an Integrated Risk Tool, Including Polygenic Risk Score, for Atherosclerotic Cardiovascular Disease in Multiple Ethnicities and Ancestries
A new integrated risk tool combines clinical and polygenic risk scores.
•
Prediction is significantly enhanced in multiple ethnicities and ancestries.
•
Ethnicities include Black and White, ancestries include African and European.
•
This is the first positive cross-ethnicity cross-ancestry validation of such a tool.
The American College of Cardiology / American Heart Association pooled cohort equations tool (ASCVD-PCE) is currently recommended to assess 10-year risk for atherosclerotic cardiovascular disease (ASCVD). ASCVD-PCE does not currently include genetic risk factors. Polygenic risk scores (PRSs) have been shown to offer a powerful new approach to measuring genetic risk for common diseases, including ASCVD, and to enhance risk prediction when combined with ASCVD-PCE. Most work to date, including the assessment of tools, has focused on performance in individuals of European ancestries. Here we present evidence for the clinical validation of a new integrated risk tool (IRT), ASCVD-IRT, which combines ASCVD-PCE with PRS to predict 10-year risk of ASCVD across diverse ethnicity and ancestry groups. We demonstrate improved predictive performance of ASCVD-IRT over ASCVD-PCE, not only in individuals of self-reported White ethnicities (net reclassification improvement [NRI]; with 95% confidence interval = 2.7% [1.1 to 4.2]) but also Black / African American / Black Caribbean / Black African (NRI = 2.5% [0.6–4.3]) and South Asian (Indian, Bangladeshi or Pakistani) ethnicities (NRI = 8.7% [3.1 to 14.4]). NRI confidence intervals were wider and included zero for ethnicities with smaller sample sizes, including Hispanic (NRI = 7.5% [−1.4 to 16.5]), but PRS effect sizes in these ethnicities were significant and of comparable size to those seen in individuals of White ethnicities. Comparable results were obtained when individuals were analyzed by genetically inferred ancestry. Together, these results validate the performance of ASCVD-IRT in multiple ethnicities and ancestries, and favor their generalization to all ethnicities and ancestries.
Introduction
Current US guidelines for the primary prevention of cardiovascular disease are based on the quantification of an individual's predicted risk of atherosclerotic cardiovascular disease (ASCVD) over the following 10 years using the ASCVD pooled cohort equations tool (ASCVD-PCE).
2019 ACC/AHA Guideline on the primary prevention of cardiovascular disease: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines.
The tool combines information from multiple clinical risk factors including age, sex, blood lipid levels, blood pressure, history of diabetes, smoking or anti-hypertensive treatment, and racial identity (entered as “White,” “African American” or “Other,” where “Other” is treated algorithmically as “White”). Although genetics is a known risk factor,
it is not directly included in ASCVD-PCE (family history of ASCVD is assessed separately, outside of the tool). Polygenic risk scores (PRSs), which combine information across thousands of common genetic variants in the human genome, can be added to the ASCVD-PCE tool, but previous studies have reported variable predictive performance.
Additionally, these studies focused primarily on individuals with European ancestries, but it is known that the predictive accuracy of a PRS tends to attenuate in individuals with non-European ancestries.
We therefore undertook a clinical validation study of a new 10-year ASCVD risk prediction tool (ASCVD-IRT) that integrates a PRS for ASCVD with the established ASCVD-PCE tool, paying particular attention to its predictive performance in non-European ancestries and non-White ethnicities.
Methods
Following recent guidelines on the use and reporting of race, ethnicity and ancestry,
Race and Genetics Discussion Group Guidelines on the use and reporting of race, ethnicity, and ancestry in the NHLBI Trans-Omics for Precision Medicine (TOPMed) program.
we clarify that in this study we use the term “ethnicity” to refer to social categories including both race and ethnicity, and we do not distinguish “race” from “ethnicity.” In the testing cohorts described below, data on ethnicity were collected from questionnaire, census and other self-identification data, allowing us to infer that ethnicity was self-reported. We use the term “ancestry” or “genetic ancestry” to refer to inferences from genetic data. We note that the concepts of ethnicity and ancestry are correlated but not synonymous.
To infer ancestry, we used a method based on principal components derived from genetic data to infer membership to one of 5 high-level ancestry groups conforming to those used by the 1000 Genomes Project
(Sub-Saharan African [AFR], Native/Indigenous American [AMR], East Asian [EAS], European [EUR], and South Asian [SAS]; see Supplementary Materials for details, OTHER indicates mixed inferred ancestry). We note that this method is more accurately described as producing “ancestry-like” genetic similarity relationships.
We present performance evaluations for both ethnicity and ancestry groups, as the former relate to important social categories while the latter allow us to examine the ancestry-specific PRS attenuation issue
We tested the performance of the ASCVD-IRT using data from the Atherosclerosis Risk in Communities (ARIC) cohort, the Multi-Ethnic Study of Atherosclerosis (MESA), and UK Biobank (UKB). All individuals in these studies gave informed consent. Legal and ethical approval for our use of ARIC and MESA data is provided by the Western Institutional Review Board (Study Number 1264897, IRB Tracking Number 20192201). See Supplementary Materials for UKB approval information.
All 3 cohorts are prospective, contain participants that were extensively examined at baseline, and have continuing follow-up (via annual phone calls for ARIC and MESA, via linkage to national electronic healthcare and death records for UKB). ARIC comprises over 15,000 adults from predominantly 2 study-defined racial/ethnic groups (“Black” and “White”), from defined populations in 4 sites in the USA, aged 45 to 64 years when recruited between 1987 and 1989.
MESA comprises over 6,000 adults from 4 study-defined racial/ethnic groups (“African American,” “Chinese American,” “Hispanic,” and “White/Caucasian”), recruited primarily via phone call invitation to 6 sites in the USA, aged 45 to 84 years and free of cardiovascular disease when recruited between 2000 and 2002.
We carefully selected 88,666 UKB individuals from multiple ethnicities (labels defined from questionnaire data) for the “IRT testing” subgroup, in order to ensure that the IRT testing subgroup was maximally enriched for incident ASCVD cases from non-European ethnicities and ancestries, and to ensure the independence of these individuals from other UKB subgroups that were used to construct and train the IRT (see Supplementary Materials for further details).
We excluded individuals with cardiovascular disease or on cholesterol lowering medication at baseline, and also those related to others in the cohort at greater than third degree relative level according to the genetic inference method described in Bycroft et al.
A separate bridging analysis was performed on individuals free of cardiovascular disease who were on cholesterol lowering medication at baseline, which indicated similar performance for this subgroup (Supplementary Figure 1). White participants in MESA and ARIC were excluded from testing due to sample overlap with GWASs used to construct the ASCVD PRS (MEGASTROKE_EUR
respectively - see Supplementary Table 1). Black ARIC individuals were included, but it should be noted that they also formed part of the cohort data used to train the ASCVD-PCE model (outside of this study).
It may therefore be expected that the absolute prediction performance of both ASCVD-PCE and ASCVD-IRT is somewhat elevated in this group. However, our validation focused on the comparative performance of these 2 tools, which is not expected to be biased. All individuals in our testing cohorts were selected so as to be independent and unrelated to any individuals used in the training of the PRS or the IRT model.
To define ASCVD cases, we used outcomes that closely matched the ASCVD-PCE tool. In MESA we used “CARDIOVASCULAR DISEASE (CVD), HARD,” and in ARIC we used a union of “MI (myocardial infarction), heart attack, or fatal CHD (coronary heart disease) by December 31, 2004” and “Definite or probable ischemic incident stroke by December 31, 2004.” The ASCVD definition for UK Biobank is described in Supplementary Materials.
The ASCVD-IRT tool is a function of the score obtained from the currently established ASCVD-PCE tool
2019 ACC/AHA Guideline on the primary prevention of cardiovascular disease: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines.
and a PRS for ASCVD, trained from multiple datasets that each represent individuals from multiple ancestry groups and from different geographies. Ten GWAS datasets for different ASCVD subtraits were meta-analyzed to derive the PRS, and an additional 4 cohorts were used to train the PRS effect size. Further details regarding the construction of ASCVD-IRT are provided in Supplementary Materials.
We assessed predictive performance of ASCVD-IRT via relative performance comparisons to ASCVD-PCE, focusing in particular on differences in sensitivity and specificity and on the sum of these differences, also known as the Net Reclassification Improvement.
We used ASCVD events in the following 10 years to define cases and we used scores above and below a certain risk threshold to define positive and negative results. This relative performance approach is justified because ASCVD-PCE provides a strong basis for comparison, being the currently established and recommended tool for 10-year ASCVD risk prediction in the US.
2019 ACC/AHA Guideline on the primary prevention of cardiovascular disease: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines.
We note that absolute values of sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) are difficult to interpret on their own. ASCVD-IRT and ASCVD-PCE are not diagnostic tests but risk predictors of an uncertain future event, where even under the best of conditions a “positive” result has a measured probability of being a noncase and a “negative” result has a measured probability of being a case. For example, we label an individual with a risk of 8% a “positive” result, but they only have an 8% chance of becoming an ASCVD case, assuming the model is correct. But while absolute values are difficult to interpret, relative improvements remain good indicators of performance improvements. In particular, simultaneous improvements in both sensitivity and specificity are strongly indicative of superior test performance.
There is also a logical choice for which threshold to use for binary compartmentalization into “positive” (high-risk) and “negative” (low-risk) results, as required for metrics such as sensitivity and specificity. The same guidelines recommending ASCVD-PCE for risk prediction also lay out recommendations for actionable risk thresholds above which intervention is advised. This threshold is 7.5% risk over 10 years for most individuals, but is reduced to 5% for individuals presenting with additional risk factors, one of which is South Asian ethnicity.
2019 ACC/AHA Guideline on the primary prevention of cardiovascular disease: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines.
We therefore applied the same thresholds for our clinical performance assessment, applying the 5% threshold for individuals of South Asian ethnicities or ancestries and 7.5% for all others.
Further details on the statistical methods used to calculate performance metrics and their confidence intervals are provided in Supplementary Materials.
Results
Tables 1 and 2 show the subgroup sample sizes, after exclusions, in each IRT testing cohort. Figure 1 and Supplementary Table 2 summarize the relative performance improvements of ASCVD-IRT over the currently established tool (ASCVD-PCE). When meta-analyzed across all testing cohorts, ASCVD-IRT shows significantly improved performance (as measured by 95% CI) across NRI, sensitivity, and specificity (combined results [with 95% CI]: NRI = 3.0% [1.7 to 4.3]; delta-sensitivity [equivalent to NRI in cases] = 1.5% [0.2 to 2.8]; delta-specificity [equivalent to NRI in noncases] = 1.5% [1.3 to 1.7]). Positive predictive value (PPV) and negative predictive value (NPV) were also significantly improved (Supplementary Table 2). The NRI is also significantly positive in all 3 cohorts, with the largest point estimates seen in the 2 US cohorts (Figure 1a, Supplementary Table 2). The within-cohort point estimates for changes in sensitivity, specificity, log(PPV), log(NPV) and Harrell's C
Figure 1Relative performance of ASCVD-IRT over the currently established ASCVD-PCE tool, split by cohort and combined via meta-analysis across cohorts. (a) Net Reclassification Improvement (NRI), with 95% confidence intervals (CI). (b) Delta-sensitivity (equivalent to NRI in cases) with 95% CI. (c) Delta-specificity (equivalent to NRI in noncases) with 95% CI. Vertical dotted lines at zero indicate the null hypothesis of no change.
We proceeded to investigate relative performance patterns within ethnicity (Figure 2, Supplementary Table 3) and ancestry (Supplementary Figure 2, Supplementary Table 4) groups (groups with >50 ASCVD cases across men and women are shown, metrics for groups with fewer cases can be found in Supplementary Tables 3 to 4). Both figures show similar patterns. The overall NRI remains significantly positive when individuals are meta-analyzed for the 2 largest groups we have data for – those corresponding to White or Black / African American / Black Caribbean / Black African self-reported ethnicities and EUR or AFR genetic ancestries. The significantly positive NRI results for Black / African American / Black Caribbean / Black African ethnicities (combined NRI = 2.5% [0.6 to 4.3]) and AFR ancestry (combined NRI = 2.2% [0.4 to 4.1]) are especially noteworthy, given the reported attenuation of PRS performance in individuals of African genetic ancestries.
The sample sizes and case numbers for other ethnicities and ancestries are lower, meaning that in some contexts it is not possible to reject the null hypothesis that there is no change in NRI. We note that there are no instances of significantly negative performance, whereas there are several instances of significantly positive NRI performance (for MESA-AMR, UKB “Indian, Bangladeshi, or Pakistani,” and UKB-SAS), and point estimates are also generally positive. The changes in sensitivity (equivalent to NRI in cases), specificity (equivalent to NRI in noncases), log(PPV), log(NPV) and Harrell's C are either significantly positive or not significantly different from zero (Figure 2b, c, Supplementary Figure 2b, c, Supplementary Tables 3, 4). We also find that ASCVD-IRT has, across the same large ethnicity and ancestry groups, a larger NRI than that of a tool constructed in the same way as ours, but using an alternative PRS previously shown to have good cross-ancestry performance (the coronary artery disease PRS of Inouye et al,
Figure 2Relative performance improvements of ASCVD-IRT over the currently established ASCVD-PCE tool, split by self-reported ethnicities. (a) Net Reclassification Improvement (NRI), with 95% confidence intervals (CI). (b) Delta-sensitivity (equivalent to NRI in cases) with 95% CI. (c) Delta-specificity (equivalent to NRI in noncases) with 95% CI. Vertical dotted lines at zero indicate the null hypothesis of no change.
Performance metrics for smaller groups (with fewer than 50 cases across men and women) are reported in Supplementary Tables 3 and 4. As the sample sizes are small, it is to be expected that most of the reported CIs in these smaller groups overlap zero. For example, UKB contains a small group of Chinese ethnicities (n = 979, with 6 cases), and MESA contains an even smaller group of Chinese American ethnicities (n = 5, with 1 case). The case numbers in these 2 groups are too low to provide an accurate estimate of performance.
We next proceeded to investigate relative performance patterns in 4 sex-by-age subgroups (Figure 3, Supplementary Table 5). This analysis reiterates patterns previously reported for individuals of European ancestries in UKB and using a different IRT built from a coronary artery disease PRS.
The overall NRI performance in 3 of the 4 subgroups is significantly positive, with the strongest performance seen in younger middle-aged men (40 to 54 year old) (NRI = 10.3% [5.7 to 15.0]). The NRI estimates within cohorts are either significantly positive (for ARIC 40 to 54 year old men and women) or not significantly different from zero. It is noteworthy that the largest NRI point estimate in ARIC, comprising individuals self-reporting as Black, is also for younger middle-aged men.
Figure 3Relative performance improvements of ASCVD-IRT over the currently established ASCVD-PCE tool, split by 4 sex-by-age subgroups. (a) Net Reclassification Improvement (NRI), with 95% confidence intervals (CI). (b) Delta-sensitivity (equivalent to NRI in cases) with 95% CI. (c) Delta-specificity (equivalent to NRI in noncases) with 95% CI. Vertical dotted lines at zero indicate the null hypothesis of no change.
UKB sex-by-age subgroups vary in their sensitivity and specificity patterns (Figure 3b, c, Supplementary Table 5), with younger middle-aged men and women (40 to 54 year old) showing significant increases in sensitivity and smaller but significant decreases in specificity, while older middle-aged men (55 to 69 year old) show the opposite pattern. The 2 US cohorts are more balanced, with no significantly negative performance estimates in any subgroup, while some effects remain significantly positive (delta-specificities for ARIC 40 to 54 year old men and women, ARIC 55 to 69 year old women, and MESA 55 to 69 year old men and women).
Detailed tables of sensitivity, specificity, PPV, NPV, Harrell's C, and their comparative differences to ASCVD-PCE (deltas and NRI) are provided in Supplementary Tables 2 to 5). We also provide an equivalent Supplementary Table 6 for an analysis carried out using a different version of the IRT that used the same PRS and PRS coefficients but was integrated with the QRISK2 score that is recommended for use in the UK.
National Institute for Health and Care Excellence Cardiovascular disease: risk assessment and reduction, including lipid modification (Clinical guideline [CG181]).
Our results indicate that ASCVD-IRT, a new tool for estimating 10-year ASCVD risk that incorporates a PRS for ASCVD, outperforms the existing standard-of-care tool ASCVD-PCE, and that this improvement extends across ethnicities and genetic ancestries. The 2 US-based cohorts used in our validation (ARIC and MESA) are drawn, in part, from minority US ethnic groups, allowing us to demonstrate that, in addition to individuals of White ethnicities or European ancestries, the significant improvement of ASCVD-IRT is also seen in individuals with Black or African American self-reported ethnicities and African genetic ancestries.
Data in other ethnicities and ancestries are more limited. However, the UKB contains a reasonably large group (n = 3,688, with 136 cases) of individuals of South Asian (“Indian, Bangladeshi, or Pakistani”) ethnicities, and a significantly improved performance of ASCVD-IRT is seen in this group as well. MESA contains a smaller group of Hispanic ethnicities (n = 823, with 55 cases), and although the NRI point estimate (7.5%) is positive, there was insufficient power to reject the null hypothesis of no change. Groups of East Asian ethnicities are even smaller in the IRT testing data (n = 979, with 6 cases, of Chinese ethnicities in UKB; n = 5, with 1 case, of Chinese American ethnicities in MESA), and this resulted in poor estimation of NRI and wide 95% CIs that extend to either side of zero.
The extent to which the results from larger groups can be generalized to support the clinical use of ASCVD-IRT in individuals from ethnicities and ancestries that are poorly represented in the IRT testing data is an important question. Three lines of evidence support such a generalization. First, we have data from additional cohorts (described in Supplementary Materials) that indicate the ASCVD PRS has an effect size in individuals of Hispanic and East Asian ethnicities that is positive, significantly different from zero and comparable in size to that seen in individuals of White ethnicities, with an equivalent pattern also seen across genetically inferred ancestries (Supplementary Figure 4). Although these cohorts lack the necessary longitudinal and covariate information to calculate ASCVD-PCE at baseline, and therefore could not be used for IRT testing, these results permit the inference that the strong predictive performance seen at the PRS level should transfer to the IRT level. Second, in line with previous work,
we observe a low (and statistically nonsignificant) correlation between PRS values and ASCVD-PCE scores in the IRT testing cohorts (ARIC: r = 0.026, 95% CI −0.018 to 0.070 [Fisher's z-method]; MESA: r = 0.002, 95% CI −0.043 to 0.047; UKB: r = 0.002, 95% CI −0.005 to 0.008). This increases our confidence that the ASCVD PRS acts largely independently of ASCVD-PCE, and strengthens the inference that PRS results should therefore transfer to the IRT level. Third, both population genetic theory and prior data indicate that individuals of African ancestries should be most affected by attenuation in PRS effect size.
so we conclude the addition of a PRS to ASCVD-PCE should lead to further predictive enhancement. But for clinical utility, the ASCVD-IRT tool requires gains in performance that are not only statistically significant but also clinically meaningful.
On this latter point, we note that large gains are seen especially in younger middle-aged men (40 to 54 year old), not only in this study (overall NRI=10.3% [95% CI 5.7 to 15.0]), but also in a previous study on individuals of European ancestry in UK Biobank, where an NRI of 15.4% (95% CI 11.6 to 19.3) was observed for coronary artery disease outcomes.
Furthermore, a large effect is also seen in younger middle-aged ARIC men of Black ethnicity in the current study (NRI = 8.5% [95% CI 0.4 to 16.7]), suggesting that this effect generalizes to other ethnicities and ancestries. We note that ASCVD is a more heterogenous condition than coronary artery disease, which may explain the observed drop in NRI.
There are limitations to this study. Tools that predict future risk, such as ASCVD-PCE and ASCVD-IRT, require larger datasets for validation than diagnostic clinical tools that typically have high sensitivity, specificity, PPV, and NPV. Thus, our conclusions remain data limited. We demonstrate significant gains in performance in some ethnicities and ancestries, but generalization to other groups requires additional inferential steps. More data should be collected to further validate and optimize ASCVD-IRT, improve risk prediction, further incorporate variable genetic ancestry among individuals, and assess performance gains in different subgroups. We also note this study does not address the question of analytical validation, and additional evidence is required to demonstrate the value of specific sampling and genotyping or sequencing protocols for the accurate computation of risk scores.
In conclusion, using data from multiple ethnicities and ancestries, we have shown improved predictive performance of the ASCVD-IRT tool over the currently established ASCVD-PCE tool in multiple cohorts, multiple ethnicities, and multiple ancestries. To our knowledge, this is the first time, for any disease, that an integrated risk tool combining a current clinical risk tool and a PRS has been successfully validated across multiple ethnicities and ancestries.
Author Contributions
Michael E. Weale: Writing - Original draft preparation, Methodology, Investigation, Supervision. Fernando Riveros-Mckay: Writing - Review & Editing, Methodology, Formal analysis, Software, Validation, Investigation, Visualization. Saskia Selzam, Eva Gradovich: Data Curation, Software, Resources. Priyanka Seth, Rachel Moore, Carla Giner-Delgado, Duncan Palmer, Daniel Wells, Ayden Saffari: Methodology, Software, Resources. William A. Tarran, R. Michael Sivley: Software, Resources, Validation. Alexander S. Lachapelle: Conceptualization, Project administration. Hannah Wand, Lee Shoa Long Clarke, Joshua W. Knowles, Jack W O'Sullivan, Euan A Ashley: Writing - Review & Editing. Gil McVean: Conceptualization, Writing - Review & Editing, Project administration. Vincent Plagnol, Peter Donnelly: Conceptualization, Writing - Review & Editing, Supervision, Project administration.
Acknowledgments
We acknowledge and thank the participants and investigators of all the datasets used in this study. Please see Supplementary Materials for detailed acknowledgements.
Disclosures
Peter Donnelly and Gil McVean are partners in Peptide Groove LLP. All other authors declare no competing interests.
2019 ACC/AHA Guideline on the primary prevention of cardiovascular disease: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines.
00207-1&id=gr1.jpg){kind=link}
00207-1&id=gr2.jpg){kind=link}
00207-1&id=gr3.jpg){kind=link}