Take That, Atul Gawande!

I’m just kidding with the title; I’m actually a big Atul Gawande fan.  He’s a provocative thinker with a compelling way of explaining complicated issues, and I suspect he is an excellent physician.  Among other things, he’s helped elevate the issue of surgical checklists into broader awareness, such as through his best-selling The Checklist Manifesto.  Now, though, some Canadian researchers are challenging the recent conventional wisdom that surgical checklists actually improve care.

New research by Dr. David Urbach and colleagues studied all Ontario hospitals on a variety of measures, comparing them pre and post-adoption of a mandated surgical checklist.  They found no difference in mortality or complications that could be attributed to the introduction of the checklists.

That’s quite a surprise (or, as the accompanying editorial in The New England Journal of Medicine called it, a “conundrum”).  After all, the WHO had made surgical checklists a major priority, going all the way back to 2007 and backed by their own research, which found almost a 50% drop in mortality and a drop of over a third in inpatient complications.  Other research, such as those by van Klei and by Kwok, has confirmed the value of the checklist, although the former did warn that their research did not find the effect as dramatic as the original WHO research. 

To further complicate things, other recent research, by Krell, et. alia., raised cautions about even using outcome measures like risk-adjusted morbidity or mortality to compare hospitals or to compare surgical outcomes.  In many cases, the authors found, the low numbers of events reduced the reliability of the results.  They warned this was especially true when only using a sampling of cases and without using advanced statistical modeling approaches.

In other words, we may not even know what we think we know, which isn’t nearly as much as we’d like to know.

There could be lots of reasons why Dr. Urbach’s study didn’t produce the same results as earlier studies.  As he pointed out, the early results stating a 50% drop may have overstated the effect.  "That's a really big effect," Urbach said. "Very few things reduce something by 50 percent."    

Dr. Gawande doesn’t seem too worried about the new results, and has already responded to them.  He worries that the data it is based on was “underpowered,” with neither enough cases nor over a long enough period of time, and he suspects that the major cultural changes required for such a change to have an impact didn’t have enough time or organizational backing to produce significant results. 

Other experts have also cautioned that the effect is not so much in just using a checklist as it is in changing the culture, which was unlikely to have fully happened in the time frame of the Ontario study.  Becker’s Hospital Review had a nice article on the challenges of implementing such major cultural changes.  In short, it’s not just about the checklist, not even just about processes in general, but must involve organization-wide changes.  Those are very hard to make. 

We’re still a long way away from truly making patient safety job #1.

After all, a study released just last month found a number of failures in adherence to infection control policies.  One in ten hospitals didn’t have checklists to prevent bloodstream infections, one in four didn’t have them to prevent ventilator-associated pneumonia, and one in three had no policies to prevent catheter-related urinary track infections.  As Patricia Stone, the study’s lead author, said, “Hospitals aren’t following the rules they put in place themselves to keep patients safe.  Rules don’t keep patients from dying unless they’re enforced.”  Worse yet, they may not even have the right rules in place.

If this is progress, I don’t want to know what things used to be like.

In psychology there a something called the “Hawthorne effect,” which is a tendency for subjects to perform better on tasks when being measured.  Although the effect and the original experiments on which it was based have subsequently been largely dismissed on a factual basis, variations of it still seem to pop up – including with checklists.  Researchers at Tulane recently released findings that showed that when patients are informed of, and involved in, surgical safety checklists, use of and compliance with the checklists improved.  As the researchers said, these involved patients not only feel safer but also likely are safer – a “win-win,” as they put it.

We should be demanding this kind of involvement.

The Leapfrog Group has been pushing measuring and publicizing patient safety measures since 2001, with a nifty hospital comparison tool and a hospital safety score, but even they say progress is too slow.  Less than a third of the hospitals that report on their measures – which is less than half of all U.S. hospitals – scored an “A” on the safety score in the 2013 results.  As Leah Binder, the president and CEO of The Leapfrog Group lamented, “we are burying a population the size of Miami every year from medical errors that can be prevented.” 

Some hospitals pick on the Leapfrog scores, especially for getting “penalized” for not participating (!), but the U.S. News & World Report “Best Hospital” ratings have more than their share of critics as well, yet those ratings show up in hospital ads all the time.  Anyone seen any hospitals ads touting their Leapfrog patient safety score?  If they’re not, it’s because we, as consumers, don’t care enough, or understand enough, about it.  Shame on us.

The real question is why hospital management everywhere hasn’t made patient safety more of a focus.  After all, the 2013 HIMSS Innovation Survey still found cost reduction as the top provider priority and in terms of actual initiatives.  It’s not that they don’t care about improving patient safety or the quality of care – those are high priorities as well -- it’s just that those aren’t seen to be as important as costs.  Six sigma/lean manufacturing experts would argue that sustainable cost reduction comes from a quality and process improvement focus, not simple cost-cutting.

As patient safety expert Ashish Jha, MD, MPH, recently told Forbes, “I haven’t heard of any hospital that went out of business because its care was unsafe.  I also haven’t heard about any CEO who got fired because the hospital’s infection rate was too high.  It doesn’t happen, and that’s telling… It’s not what keeps CEOs awake at night.  And until we get CEOs losing sleep about unsafe care, we’re not going to make a big dent in the failures of our health care system.”

Several years ago I mentioned (Gambling on Health Care) a point former Senator Tom Daschle made, that he could get more performance data on virtually any athlete than on virtually any physician, which certainly says something about our priorities.  Hospital data is somewhat more available, although definitely not comprehensive, and what data there is does not tease out important characteristics like which physician(s) and care teams were involved.  Surgeon Smith may do better overall than surgeon Jones, but in hospital A surgeon Jones may perform the best…but only with anesthesiologist Sleepy.  It’s a mess, but so is the uncoordinated care system we use every day.

You can easily assemble fantasy sports teams on the available data sets, but no such luck with trying to pick out your real life health care team.

Look, I don’t really care about checklists in and of themselves, or the very technical patient safety measures Leapfrog collects.  Checklists are a tool, not a magic wand, and like all tools, what matters is how they are used.  It sounds cynical, but maybe one reason why airplane pilots have religiously relied on pre-flight checklists for decades while physicians can’t quite seem to get into them is because the pilots know their lives depend on them too. 

We need to look at patient safety as though our own lives are at stake, because, in the final analysis, they are.

I’ll let Dr. Gawande get the final word, from his book Better: A Surgeon’s Notes on Performance:   “We always hope for the easy fix: the one simple change that will erase a problem in a stroke. But few things in life work this way. Instead, success requires making a hundred small steps go right - one after the other, no slipups, no goofs, everyone pitching in.”