A new version of the Consumer Assessment of Healthcare Providers and Systems (CAHPS®) health plan survey, CAHPS Health Plan 4.0, was introduced last year by the Agency for Healthcare Research and Quality (AHRQ). AHRQ worked with a number of stakeholders, including NCQA and the CAHPS Consortium, to make changes to the survey such that it would improve the clarity of the survey address topics of growing interest to plans, purchasers, and consumers—for instance, ensuring better comprehension of the questions among diverse populations and clarifying the providers referenced in certain questions.  

Resultant changes to the survey included changes to the number of core items, the wording and order of certain questions, and the wording and number of response scales. (An overview of the changes is available here).

NCQA performed a statistical analysis of this year’s results among commercial and Medicaid plans and compared them to the results from 2006 (CAHPS 3.0H) to understand the new survey results. NCQA worked with the CAHPS research team to design and interpret these results.

Statistical Analysis of CAHPS Results, 2006-2007

Key findings of our statistical analyses are:

• CAHPS averages have changed to a greater extent than in years past. Some averages increased while others declined.
• In general, NCQA’s analysis found that 4.0H results across plans and within the 4.0H survey were consistent with past survey results and relationships, though absolute levels of performance may have changed. In other words, 4.0H results provide reliable and valid information about the relative rankings of plans and about strengths and weaknesses within plans.
• Historically, certain composites have been known to be strongly correlated with specific global ratings: for example, a doctor’s Communication composite strongly correlates to his or her overall rating. In general, these relationships in the 4.0H survey were consistent with prior work or changed in ways that were to be expected based on changes to the survey. The only unexpected result was that a decrease in correlation between the Customer Service composite and the Rating of Health Plan measure.
• The Customer Service composite showed changes in average performance, the correlations between years, and the correlations between composites and global ratings.
  Getting Needed Care and Getting Care Quickly showed changes in average performance and correlations over time, but their relationship to other composites and global ratings were similar to previous surveys. This suggests that the new results may capture different aspects of these constructs.
• The remaining composites and ratings showed some changes in average performance, but the correlations between years and the relationship to other composites were consistent over time or changed in ways that were to be expected based on changes to the survey. For example, the correlation between the Doctor Communication composite and the overall Rating of Personal Doctor increased, while the correlation between Doctor Communication and the overall Rating of Health Care decreased.

These results suggest that while the overall survey remains sound, the results varied for some composites and ratings. More information about the findings is available here.  

Changes to NCQA Reporting Products for 2007  

The changes to the CAHPS survey affect Accreditation scoring and a number of NCQA’s public reporting products, as well as a number of evaluative activities outside our purview. As a result, NCQA is taking the following actions:

• Accreditation scoring. Due to significant changes in wording, NCQA will not score the following items in Accreditation:
  • Getting Needed Care
  • Getting Care Quickly
  • Customer Service

As noted in a previous memo to health plans, plans will receive the maximum score on these three composites.

• Benchmarks and thresholds. For the other 4 ratings and 2 composites scored for Accreditation, NCQA will recalculate benchmarks and thresholds based on CAHPS 4.0H and score plans using the new benchmarks and thresholds.
  
• Using results to demonstrate improvement for Accreditation. NCQA will give plans full credit for the QI 13 Service Quality Improvements standard, for which plans generally use CAHPS results over time to demonstrate improvement.
  
• Public reporting. NCQA will report all CAHPS results in Quality Compass except the Customer Service composite. However, the individual items that make up the Customer Service composite will be reported. NCQA will include guidance about using results over time (e.g. identifying items with wording changes; notes that changes in the organization of the survey may affect results) in all public reporting products, including Quality Compass.
  
• America’s Best Health Plans. NCQA’s collaboration with U.S. News and World Report ranks health plans relative to one another based solely on one year’s data. The Customer Service composite will be removed from the America’s Best Health Plans methodology and the remaining CAHPS measures will be reweighted accordingly. An updated methodology will be posted to www.ncqa.org/bestplans as soon as it becomes available. 

NCQA is confident that the survey is valid and reliable. We will continue to work with AHRQ and the CAHPS team to understand the differences that were observed in the CAHPS 4.0H version.  

Should you have questions about how the changes to CAHPS affect NCQA reporting products for 2007, please don’t hesitate to contact NCQA’s Customer Service department at 888-285-7585.  

Thank you for your continued commitment to our shared mission of improving the quality of health care.