HCI3 Update from the Field: Confidence Interval

Submitted by francois.debrantes@hci3.org on Friday, February 15, 2013 - 01:48

Newtown, CT – February 15, 2013

A confidence interval is a simple measure of significance. It tells us, with a degree of certainty, whether or not a measure, a benchmark, or a goal has been achieved or exceeded – For example, in a 2008 paper published in the AJMC, we showed with a degree of certainty that BTE-recognized physicians have lower costs of care and deliver higher quality than non-recognized peers. That certainty was further established last year in an Issue Brief on a similar topic. Conversely, last year, the Congressional Budget Office found that there was little (if any) certainty that Medicare's past demonstration programs on "ACOs" had yielded any savings. Strangely (or perhaps not when one considers that dogma often trumps facts), that hasn't stopped private and public sector payers from rushing in to grab the elusive ACO savings. Yet the caution expressed by the CBO was fully analyzed and explained in a recent paper by Derek DeLia and colleagues that we referenced last week. We come back to this paper because of its importance in reminding us that the transformation of US health care will succeed or fail depending on our certainty that changes are having the desired effect. And how will we know that ACOs are delivering the goods? The same way we know that BTE-recognized clinicians are more efficient and effective than peers: their cost and quality measures are better than their peers and benchmarks. Professor DeLia's paper suggests that unless the ACOs have lots and lots of a payer's members – somewhere close to 20,000 – the conclusions on performance will simply lack certainty.

What this means to you – If your health plan is touting the latest ACO deal concluded, make sure you ask how many members they have enrolled in that ACO, and then ask the plan to conduct an analysis similar to DeLia's, and provide you with the results. Because if you and your plan can't tell whether or not the ACO is actually saving any money, why are you wasting your time and resources pursuing a chimera? The central point in that paper is self-evident: there is lots of normal variation in costs of care, year over year, for any population of patients, caused by the new onset of certain conditions, the termination of others, and a host of other "probability"-related events. As such, what has to be the magnitude of the change in total costs, from year to year, above or below which one can say, with certainty, that it is caused by something else than this normal variation? The answer, of course, depends on the size of the population, and hence DeLia's recommendation of a large sample size. That's because the smaller the population, the greater the swings caused by fewer individuals having some "probability"-based events. Unfortunately, very few ACOs, including the newly minted ones in the CMMI program, and most of the ones in private sector payer programs, have adequate sample sizes. As such, this rush to grab the elusive ACO savings might end up being the butt of a rhyme, much like the mythic chase for the elusive Pimpernel: We seek them here, we seek them there, those payers seek them everywhere. Are they in sight? – Are they in right? Those elusive savings simply fade into the night. And with them, the needed relief for the average American household. More on that needed relief in The Incentive Cure.


Francois de Brantes
Executive Director
Health Care Incentives Improvement Institute, Inc.
w: www.hci3.org