Reliable Measures

Submitted by on Friday, July 24, 2015 - 12:41

Newtown, CT – July 24, 2015

Is some information better than none? Always, IF the provider of information is honest enough about the caveats – The past two weeks have seen publicly available provider ratings, mostly about surgeons and hospitals, widely released. The first to come out was ProPublica's rating of surgeons using a measure of adjusted rates of complications, and the second was Consumers' Checkbook's rating of surgeons, also using a measure of complications. The third was the US News & World Report's rankings of Best Hospitals. Let's dispense quickly of this latter one because, by and large, it's a bad joke on the American public. The methodology is weak, at best, and the usual suspects of very expensive academic medical centers make the "top grade". The pretense – and it is a pretense – is that somehow these "best" hospitals can be categorized as such because they do a "fabulous" job in a small selected set of conditions that matter only for Medicare patients. It's absurd and pathetic, but then again what should we expect from that type of magazine. So let's turn instead to the more serious attempts to provide insights into the quality of surgical care. Both ProPublica and Consumers Checkbook rate surgeons on specific surgeries and don't presume, rightfully, that a surgeon that has low complications for one type of surgery would have low complications for another. Similarly, and rightfully, they don't assume that a hospital that has surgeons that are all above average for one type of surgery can be deemed good at everything. Overall, the ratings of the surgeons on both sites are similar, although there are notable differences. The more problematic assumption that all three make, regrettably, is that a provider's performance on Medicare patients yields the same results as on patients under 65.

What this means to you – Earlier this month we submitted/resubmitted several of our measures of Potentially Avoidable Complications to the NQF for review and endorsement. We already have some PAC measures endorsed as comprehensive outcomes measures and want to expand on that set. A requirement in submitting new measures to the NQF is to test the reliability of each measure, because a reliable measure is a measure that can distinguish provider performance. The upshot is that PAC rates for procedures and chronic conditions are a reliable way of distinguishing performance and the sample sizes required to achieve a reliable score are relatively low. And here's a summary of the work done. Note that we calculated these measures on commercial data sets and on Medicare data sets and that the results were different, as one might and should expect. The measures submitted for endorsement were purposefully tested on commercial data sets because we want to publish reliable rates of complications that are directly relevant to under 65 Americans. And we know (and have very strong evidence) that ratings of physicians or hospitals change when you change the dataset. The results using a Medicaid data set are significantly different than a commercial data set or a Medicare data set, and blending the data together, or inferring that results that are derived from a specific population are applicable to all is wrong. It's also potentially damaging to the reputation of the providers. So for the sake of all who are forcefully, legitimately and rightfully pushing for transparency on price and quality, ratings publishers that only use Medicare data should add an important caveat to their results: These results only apply to Medicare beneficiaries.