Objectives: The Consumer Assessment of Health Plans (CAHPS) survey is designed to collect member experiences with getting medical care. The objective was to evaluate the comparability of answers to CAHPS questions when data are collected by mail and by telephone interview.
Methods: Two studies comparing phone and mail responses used a pretest instrument with parallel samples drawn from Medicaid beneficiaries in California (n = 217 telephone, 97 mail) and adults with chronic conditions who had health insurance through the State of Washington (n = 98 telephone, 109 mail). A third study used a revised instrument with two parallel cross-section samples of adults covered through the State of Washington (n = 446 telephone, 609 mail). Questions covered respondents' experiences with getting medical care through their health plans.
Results: In the first two tests, numerous significant differences were found in the rates at which questions that potentially did not apply to all respondents were answered: some ratings were more positive on the telephone. In the test of a revised instrument, nine of 58 comparisons differed significantly by mode. The systematic differences in response to questions that did not apply to all respondents were greatly reduced. Only one of four ratings and one of seven multi-item composite measures of quality of care were significantly different by mode.
Conclusion: Although further steps to reduce the remaining mode effects are needed, the data indicate that when the revised CAHPS questions are used, mode of data collection will have little effect on the key results.