In his GIBLIB lecture, Benjamin (Ben) D. Pollock, PhD, MS, Kern Center endowed scientific director for the Science of Quality Measurement, defines a framework for the ratings and rankings of hospitals
HOSPITAL RANKINGS AND RATINGS
The differentiation between a rating and a ranking is that, like the Leapfrog Hospital Safety Grade ratings shown in the image above, ratings are categorized. In the Leapfrog Hospital Safety Grade, hospitals are categorized into a letter grade of A for top-performing and then down the list B, C, D, or F like a schoolyard.
The CMS Hospital Star Ratings are a similar kind of concept. Hospitals are rated from 1 to 5 stars, with five stars being the best. In a rating, 5-star hospitals are considered equal to each other. The 4-star hospitals are considered equal to each other, etc.
The image above shows an example of a ranking. Mayo Clinic very often receives the number one ranking in the U.S. News and World Report best hospital ranking. And again, this is different from a rating because, in the rating where hospitals are categorized in equal groups here, this is just a pure ranking, number one, number two, number three, and so on down the list.
So when we talk about hospital rankings and ratings, what does good, or what does best really mean? These things could mean different things to different people, and we can measure these things in different ways.
A good analogy to think of is that of a coffee shop. So if you picture the best coffee shop or your favorite coffee shop, what is it that makes it the best or your favorite? Is it the shop with the tastiest coffee? Is it the coffee shop with the friendliest staff? Is it simply the one in the most convenient location or nearest to you? Is it a coffee shop with the most seamless ordering or pickup process? Historically, for ranking or rating hospitals, the framework for good or best generally revolves around the word quality, as in which hospitals provide the highest quality of care?
THE DONABEDIAN FRAMEWORK OF QUALITY OF HOSPITAL CARE
Okay, so what is hospital quality? How do we define that term? Luckily, we have over 50 years of scientific literature, which has really coalesced around a Donabedian framework for quality of care. The Donabedian framework has three pillars Structure, Process, and Outcome.
So, let’s put this back to our coffee shop analogy. If we use the Donabedian framework – picture your favorite coffee shop, what do you like about it? What makes it your favorite?
So, using those three pillars of that framework, Structure would be things like the physical building and the atmosphere. Does the coffee shop have a good vibe? Do they have friendly staff? Do they have a high-tech cappuccino machine? Is there asbestos or mold in the walls? So those are the structural component of your best coffee shop.
Then, the other two pillars are Process and Outcomes. Process would be things like do they have touch screen ordering? How long is the line? Do they have a drive-thru window? And then you get to Outcomes which are probably the most obvious and most well-known, and what people think about when they think about best. And these are things like, does the coffee taste good? Is it the right temperature? Does it have the right amount of caffeine?
STRUCTURE, PROCESS, AND OUTCOME
So, back to the hospital framework. Structure is things like the organizational or physical characteristics of the hospital. How many nurses do they have? How many beds do they have? Do they have a trauma center? Does it conduct academic research? Process is the care delivery steps that occur in the hospital. What services do they offer? What treatments do they have? And what's the care pathway from when you enter the hospital until the time that you leave the hospital?
And then Outcomes are the most well-known: the effects, the results of the care, things like mortality, and readmissions. Did you acquire an infection in the hospital? Did you have a patient safety event? Did you have a fall? Did you get or develop a pressure ulcer? The Outcomes really answer the question, how did the care affect the patient? Did they get better? Did they get worse?
In hospital rankings and ratings, outcomes tend to be given the highest weight in the scoring methodologies because these are the direct and measurable results that seem to matter the most to patients. It makes sense if you think back to the coffee shop analogy, when you think about the process and the structure, you might be willing to wait in line 2 to 5 minutes longer for coffee that tastes better.
So, this brings us to our first multiple-choice question:
The answer is C. nurse staffing ratio. This is not an outcome measure. Mortality, readmissions, and hospital-acquired infections are outcomes. The nurse staffing ratio has to do with the structure of the hospital. So, again, under this Donabedian framework, most hospital rankings and ratings select a handful of structural measures, a few process measures, and some outcomes measures. The key distinguishing feature, however, is the variety of different weighing or scoring approaches.
VARIETY OF DIFFERENT WEIGHING AND SCORING APPROACHES FOR HOSPITALS
The image above shows the U.S. News and World Report hospital ranking on the left, the CMS star rating in the middle, and the Leapfrog safety rating on the right.
The above image is a breakdown showing what percentage of these hospital rankings/ratings are made up of which components of the Donabedian framework. Orange represents the outcomes; especially in the CMS star ratings which are given by far the largest weight. Structure and Process are usually given equal weight to each other, but less than Outcomes.
So, back to our coffee shop analogy: even within a domain such as Outcomes, the Outcomes that matter to different customers or patients can be different. For coffee, do you want the perfect temperature? Do you want the perfect caffeine level? The perfect flavor? Outcomes that matter might be different for different customers. Likewise, some rankings and ratings place a greater emphasis on different Outcomes, such as hospital readmissions, mortality, home discharge, or infection rates. So when we think about hospital Outcomes, how our rankings and rating stakeholders like U.S. News, Leapfrog, or CMS, how are they able to measure hospital outcomes?
INTERPRET HOSPITAL RANKINGS/RATINGS
This brings us to our first important point: when we think about it as a consumer, how can you interpret some of these hospital rankings and ratings? They typically use Medicare claims data, and it's typically on a 2 to 5-year lag.
As you think about the next few years, especially in the era of COVID, most current hospital rankings or ratings have yet to include any data from the pandemic era. And so, over the next few years, you might see some of these rankings and ratings change.
THE PANDEMIC WILL AFFECT FUTURE HOSPITAL RATINGS AND RANKINGS
As we understand how hospitals performed during the pandemic, based on what data is used, rankings and ratings can also completely measure different groups of people. If you have heart disease, do you want to go to a hospital that is highly ranked for heart surgery, or one ranked highly for pneumonia treatment? When we think about a ranking like the U.S. News, they include 15 different specialties and 17 different procedures and conditions. So, it's important to look at the specific procedure and condition that you might be seeking and see how the hospital performs in that aspect.
Thinking back to the coffee shop, what if we compared coffee shops on college campuses to coffee shops within long-term care facilities, which might have a more elderly clientele? When you're looking at hospital rankings and ratings, you need to maybe understand which of these might be more applicable to you.
All these different things can add to the variability of hospital rankings and ratings, and it can cause the same hospital to be rated highly by one organization, but then be rated lower by another organization. For example, only around half of the top 20 ranked hospitals by U.S. News actually receive five out of five stars on the CMS rating in any given year. Likewise, the ratings and grades can have uneven distributions. So, less than 5% of hospitals are typically rated 5 or 5-HCAHPS stars, compared to roughly one in three hospitals, which receive the top grade of A in Leapfrog.
Above is a hypothetical example. On the top, is hospital rating Y, and on the bottom is hospital rating Z. You can see that depending on the clustering algorithm used, you might have a different number of hospitals that achieve 5, 4, 3, 2, or 1 star.
The example here is if we put this in a normal distribution, and we say this hospital achieved the 70th percentile score on this rating, depending on the grouping and rating Y, you see that achieves 3-stars versus a rating Z where that same percentile compared to their peers, they'll be achieving 5-stars. So considering all these differences and limitations, how are consumers supposed to interact with or understand hospital rankings and ratings?
HOW MAYO CLINIC EVALUATES HOSPITAL RANKING AND RATING
Mayo Clinic has analyzed and aggregated several rankings and ratings into a composite index of hospital quality. Specifically, they combine the CMS hospital star ratings, the Leapfrog safety grade, the U.S. News and World Report best hospital ranking, and the HCAHPS star rating. The U.S. News and World Report best hospital ranking ranks hospitals across 15 specialties and 17 procedures and conditions. Although this is subject to change in the annual rankings released, they're constantly adding new specialties, adding new procedures and conditions, or removing old specialties and procedures and conditions.
This ranking also incorporates criteria, such as the number of nurses on staff, other accreditations, and advanced technologies. There's also a strong reputational component for Process. So, about 25% of the U.S. News and World Report ranking is based on what other physicians say about that hospital. For the CMS star rating, this is a federal agency, this is very Outcomes-heavy and the key data include hospital readmission rates, patient safety, patient satisfaction, mortality, and timely and effective care.
The Hospital Consumer Assessment of Health Care Providers and Systems or HCAHPS star rating is a completely different rating than the ones we've described so far, this is a survey that measures patients' perceptions of their hospital experiences. This allows patients to compare hospitals on topics important to them, such as communication with doctors and nurses, as well as noise level and cleanliness. And lastly, the Leapfrog Hospital safety rate, which we've discussed, uses 15 measures of publicly available hospital safety data and combined with seven self-reporting survey answers to produce a patient safety-focused score for each hospital.
The Mayo Clinic ultimately combines all four of the discussed rankings and ratings. They then convert that score into a national percentile from 1 to 99. An example of this is if there are approximately 4000 hospitals eligible for the U.S. News ranking. Hospitals ranked 1st to 40th would be given the 99th percentile. 41st through 80th would be given the 98th percentile, and so on.
Then, hospitals 3960 through 4000 would be in the 1st percentile. Mayo Clinic does this for the Leapfrog safety grade score, the CMS star rating score, and the HCAHPS summary score, where they have an underlying score beneath that 5-star rating. Mayo then averages the percentiles of all four of these rankings and ratings, and this gives them a final composite index score.
The image shows a handful of top-ranked U.S. News hospitals. In each column, you can see the hospital ranking or rating, along with the corresponding percentile that the hospital achieved. And then on the far right side, in the little bubbles, you can see the average that's the Mayo Clinic composite quality index score, the average of the percentiles across those for rankings and ratings with obviously a score closer to 100 being the best and lower down towards one being the worst.
At Mayo Clinic, the goal is to achieve the 90th percentile or better performance in each of the scores. So, while this is useful overall and allows us to dig into areas of potential underperformance, it can be even more nuanced than that for the consumer.
HOSPITAL RANKING AND RATING QUIZ
Here is multiple-choice question number two:
There are four fictional national rating systems. Using these systems, which coffee shop consistently performs the best across all areas? There are question marks to not show you the actual scores in each of these rankings and ratings. Instead, there is only an overall score, which is analogous to the composite quality index score discussed previously.
So based on this, without knowing the individual index rankings or ratings, you would probably say that the answer is, A. It has the highest overall score, 4.75 out of five. It beats B, C, and D.
Which coffee shop would you prefer to attend? And now this is really up to you as the user or consumer, or in the hospital analogy as the patient to choose what you prefer.
For coffee shop A, even though they're the best overall, they scored a four out of five stars on the national taste rating. If taste is your number one component that's important to you, you might go for coffee shop B or C. If you're really concerned about wait time, then coffee shop A might be for you, they have the highest rating on the wait time.
This is effectively how the Mayo Clinic conceptualizes hospital rankings and ratings as an overall score, as well as the individual scores of each ranking and rating. Generally, hospital rankings and rating systems that we've mentioned here are a great place to start your consumer or patient research. But there's one additional very important consideration when it comes to you and your personal health care decision-making. And that is, of course, seeking the advice and counsel of your primary care physician or trusted healthcare advisor.
CONCLUSION
At the end of the day, hospital rankings and ratings, as well as individual experts like myself, cannot maximize your individual experience as a patient. All we can do is measure patient outcomes from a population level and provide data-driven insight into overall hospital quality. Your primary care provider can help you understand referral options, diagnoses, and treatment plans.
