The following is a guest post from Michael McWilliams, MD, PhD; Michael E. Chernew, PhD; Bruce E. Landon, MD, MBA, MSc; and Aaron L. Schwartz, PhD.
We have received multiple requests from researchers, reporters, and other stakeholders for additional information that might explain the difference between the reduction in total Medicare spending in the first year of the Pioneer ACO model estimated by our recent study and the reduction estimated by another study conducted by evaluators sponsored by the Centers for Medicare & Medicaid Services (CMS) and authored by Nyweide et al.
We found a 1.2% reduction in spending, whereas the CMS-sponsored study found about a 3.6% reduction. This post is intended to describe key differences in approach and findings and to interpret those differences in estimates in the context of the larger policy debate.
Before discussing the differences, we should emphasize that the conclusions of the two studies are far more similar than different. Both studies, together with a previous study we published, provide consistent evidence that Pioneer ACOs have reduced Medicare spending with unchanged or improved quality of care. Moreover, both studies find that savings were similar for ACOs that dropped out of the Pioneer model after the first year and those that continued to participate. This finding is emphasized in our study and can be found in the appendix of the Nyweide et al. paper.
That the ACOs subsequently withdrawing achieved savings, let alone equal savings, is perhaps more important than the exact magnitude of the overall savings because of the important implications of this finding for the direction of Medicare ACO payment policy, which has received much attention as the rules for the Medicare Shared Savings Program (MSSP) are revisited. As we discuss in detail in a prior paper, a related white paper, and our paper on the Pioneer model (excerpted below), current regulations may discourage participation even among ACOs that lower spending.
Taken together, these findings have important implications for payment policy in Medicare ACO programs. First, given the lack of a relationship between estimated savings and continued participation in the Pioneer program, sustaining or expanding participation in a Pioneer-like ACO program will probably require greater and more reliable rewards for ACOs that reduce spending than those currently in place. For example, our findings support the consideration of increased shared-savings rates and a benchmarking approach that would account for local spending growth and would sever or weaken the link between ACO benchmarks and savings in preceding contract periods; currently, this link diminishes incentives to achieve and maintain increased efficiency. Second, as analysts score proposed changes to ACO payment rules, they should consider lost savings from organizations that withdraw from the ACO programs in response to current incentives. Stronger incentives to participate in ACO programs would diminish the share of savings appropriated by Medicare for a given ACO but could lead to more ACOs generating savings.
Comparison of studies
The two studies share the same basic difference-in-differences design, comparing beneficiaries attributed to Pioneer ACOs with a control group of local beneficiaries. The differences discussed below relate to relatively nuanced analytic choices in how the two studies define Pioneer ACOs, attribute beneficiaries to ACOs, and define the control group.
Readers not interested in the methodological differences between the papers but just the implications of them for findings may want to skip to the section titled “Differences in comparability and findings.”
Differences in Pioneer ACO definitions
The CMS-sponsored study used the provider identifiers submitted by Pioneer ACOs identifying the providers included in the ACO contracts (maintained in provider extract files by CMS), whereas we determined the providers in each contract from Pioneer ACO websites, converting provider names to national provider identifiers (NPIs). We do not believe this difference in provider data contributed significantly to the difference in overall savings, as we have confirmed that 90% of NPIs in our Pioneer ACO definitions are included in the 2012-2013 Pioneer provider extract files used by the CMS-sponsored study (96% when the 2014 extract file is also considered), and 90% of the NPIs in the 2012-2013 Pioneer extracts were included in our ACO definitions.
Differences in beneficiary attribution
We attributed beneficiaries to Pioneer ACOs or to non-ACO providers (the control group) in each year from 2009 to 2012 based on beneficiaries’ concurrent use of primary care services in each year (perhaps somewhat confusingly, attribution based on concurrent use is called “retrospective attribution” and is currently used in the Medicare Shared Savings Program). The CMS-sponsored study used the Pioneer prospective alignment rules to attribute beneficiaries based on use of primary care services in the years preceding each study year (i.e., using the beneficiaries actually aligned to ACOs under the model for performance years and analogously aligned beneficiaries for the pre-contract years).
We chose retrospective attribution to ensure consistent categorization of beneficiaries in each year, including in the pre-contract years of our study. The prospective attribution methodology used in the CMS-sponsored evaluation introduced a slight asymmetry in attribution between the pre-contract and post-contract periods. Specifically, it used 3 years of utilization to prospectively align beneficiaries in 2012 and 2013, but only 1.5-2.5 years to determine alignment in 2010 and 2011 due to NPIs being uniformly present in claims only starting in 2008.
Conceptually, retrospective attribution supports analysis of patients served by Pioneer ACOs, most of whom were prospectively aligned under the Pioneer model, and the rest of whom received care primarily from Pioneer ACO physicians and thus would have been affected by any spillover effects of Pioneer ACOs’ efforts to constrain spending for patients covered by the risk contract onto other patients they served. Prospective attribution supports analysis of patients aligned to an ACO under the Pioneer model, some of whom were no longer served primarily by the ACO during the year of interest and would have been less affected by efforts to constrain spending than other prospectively aligned patients.
Thus, we would not expect our use of retrospective attribution to bias estimates toward the null to a greater degree than prospective attribution, as both prospective and retrospective attribution lead to the inclusion of beneficiaries in the intervention group who may have not been affected as much by the intervention. Indeed, in a sensitivity analysis attributing beneficiaries in each study year prospectively based on utilization in the prior year only (to maintain consistency over the study period), we found savings that were no greater than what we estimated using retrospective attribution.
As we discuss in detail in our paper’s appendix, another difference is that we did not count physician visits in nursing facilities as primary care services used for beneficiary attribution, whereas the CMS-sponsored study did in accordance with the definition of primary care services in Medicare ACO rules for the purpose of beneficiary attribution. Because post-acute care providers are under-represented in Pioneer ACO contracts relative to the surrounding delivery system, we found that excluding these services from the attribution algorithm improved the comparability of the ACO and control groups, as evidenced by similar levels of spending in the pre-contract period.
Differences in control group definitions
To maximize comparability between the ACO and control groups, we included in our control group only FFS beneficiaries who used at least one primary care service (and were therefore eligible for attribution to an ACO or non-ACO provider in a given year). In other words, in the ACO programs, a beneficiary must have used at least one primary care service to be attributed to an ACO. Therefore, we required the same for the control group.
In contrast, the CMS-sponsored study considered any beneficiary not aligned to an ACO to be part of the control group as long as the beneficiary resided within the ACO’s service area (defined as the set of counties where Pioneer providers were located and contiguous counties) and met the same coverage requirements as ACO-aligned beneficiaries (we also applied similar enrollment requirements). This approach included beneficiaries with no primary care visits in the years used for prospective attribution.
Our control group also differed in that we did not require beneficiaries in the control group to live in specific areas defined by the location of Pioneer ACO providers. Instead, we achieved geographic similarity between the ACO and control groups through our statistical modeling (specifically, by including fixed effects for HRRs, allowing them to vary each year). That is, whatever the geographic distribution of patients attributed to an ACO was, we compared each ACO-attributed beneficiary to non-ACO beneficiaries living in the beneficiary’s HRR. Thus, beneficiaries in the control group living in HRRs with fewer ACO-attributed beneficiaries contributed less to estimates than control group beneficiaries living in HRRs with more ACO-attributed beneficiaries.
Differences in comparability and findings
Whether because of these or other nuanced methodological differences, several differences in the results reported by the two studies stand out. In our study, all observed sociodemographic and clinical characteristics of patients were similar between the ACO and control groups in the pre-contract period, and differential changes in these characteristics from the pre- to post-contract periods were negligible. These similarities suggest that there were no compositional changes in observed or unobserved characteristics of our comparison groups that biased our estimated savings upward or downward.
In contrast, the ACO and control groups in the CMS-sponsored study significantly differed in the pre-contract period in key characteristics (e.g., age and sex) and exhibited significant differential changes in those characteristics in the post-contract period. A weighting technique was used by the CMS evaluators to restore consistent balance in patient characteristics between the comparison groups throughout the study period, but these differences in observables raise the concern that other unobserved characteristics not accounted for in the weighting adjustment also differentially changed between the comparison groups.
More importantly, both spending levels and trends over the three-year pre-contract period (2009-2011) were similar for the ACO and control groups in our study, supporting a key assumption in the difference-in-differences approach—that we would not expect spending in the ACO group to differentially change in the absence of an intervention.
In the CMS-sponsored study, pre-contract spending was 5.4% higher in the ACO group than in the control group, and spending trends in the two-year pre-contract period (2010-2011) were not as parallel. In the CMS-sponsored study, spending was already growing more slowly in the ACO group than in the control group from 2010 to 2011. Assuming this convergence in spending already underway would have continued into the post-contract period in the absence of the Pioneer model, the preceding trends would explain about one-third of the estimated savings in 2012, reducing the estimate from 3.6% to 2.5%.
Finally, adjusted spending for the control group in the CMS-sponsored study grew by 4.9% from the 2010-2011 pre-contract period to 2012, while overall unadjusted per-beneficiary FFS spending in Medicare actually declined slightly in 2012, as documented in national reports (e.g., see chart 1-2 in this MedPAC report). Unadjusted and adjusted spending growth in our control group before 2012 was consistent with the national trend in FFS spending, and we confirmed that this was the case, too, for beneficiaries in our control group specifically residing in Pioneer ACOs’ primary HRRs (the HRR where most of an ACO’s attributed patients resided). The increase in spending for the control group in the CMS-sponsored study likely contributed to the estimated differential reduction in spending for the ACO group.
Conclusions
For these reasons, we believe the true savings achieved by Pioneer ACOs in the first year of the program is likely closer to the more modest estimate of 1.2% produced by our methodological approach. However, the more important conclusion is that two independent evaluations of the Pioneer program now provide consistent evidence of spending reductions with unchanged or improved performance on measures of quality and patient experiences. We applaud CMS for publishing their work and believe there is much to be learned from multiple studies of important reforms because different perspectives and methodological choices deepen the evidence base and provide richer information than any one study could provide alone.
We hope this examination of methodological differences between the two studies is informative for future evaluations and underscores the importance of assessing the impact of significant policy changes with multiple evaluations. We would encourage stakeholders to focus less on the exact amount of savings achieved by the Pioneer program and more on the lessons that can be drawn from these two studies for the challenging task at hand—implementing changes to the rules governing ACO programs that will allow providers and Medicare to build on a promising start.
