Pilot Findings

Nationwide CAHPS Survey of Adult Medicaid Beneficiaries Sampling Pilot: Methods and Findings

CMS and its contractor, NORC conducted a pilot study between January and March 2014 to test three methods of sample selection that we propose for the Nationwide Consumer Assessment of Healthcare Providers and Systems (CAHPS) Survey of Adult Medicaid Beneficiaries. In addition to assessing the viability of the sampling specifications used to select samples and quality of the samples drawn, the pilot was also intended to estimate the level of burden on states associated with each sampling method. The three sampling methods each yield a sample file of eligible beneficiaries that can be contacted for the CAHPS survey. States will choose one of these options based on the currency of their data files and state preferences (e.g., whether the state prefers to provide a data extract from which NORC will draw a sample or whether the state prefers to select its own sample).

The three sampling options are:

1. MSIS Option (Option 1): NORC pulls a sample of eligible beneficiaries from approved state MSIS data. The sample is sent back to the state, so contact information (e.g., name, address, phone number) can be appended. The state then sends the contact information back to NORC.

2. Data Extract Option (Option 2): The state sends NORC a file of beneficiaries eligible during the six-month period of interest (currently defined as July 1, 2013 through December 31, 2013), and a long-term care claims file for claims during calendar year 2013. NORC uses this file to select a sample of eligible beneficiaries.

3. State Selects Sample Option (Option 3): The state constructs the sample frame and selects a sample of eligible beneficiaries using NORC’s detailed specifications. The state sends the selected sample to NORC.

It is expected that the first and second options will require the least amount of effort on the part of the state, but the last option will require significant effort because the state will need to select their own sample per specifications provided by NORC. Ultimately, if a state chooses Option 3, the state will first have to create the files needed for Option 2 as a starting point.

Pilot Methods

CMS invited all states receiving an Adult Quality Measures grant and states participating in the T-MSIS pilot to participate in the CAHPS pilot. Nine states either volunteered to participate in, or requested additional information about the expectations of the pilot. After preliminary discussions with these states, we recruited five states to participate in the pilot study: Alabama, Oregon, Rhode Island, Tennessee, and West Virginia. Each state chose at least one sampling option to pursue for the pilot study. Two states (Alabama and Tennessee) chose two options, due to their interest in attaining experience selecting a sample under Option 3. At least two states participated under each option, which provided a range for the level of effort required under each sampling option. Table 1 displays the sampling options pursued by each pilot state.

Table 1. Pilot States by Sampling Options Selected

State	Option 1 (MSIS)	Option 2 (Data Extract)	Option 3 (Select Sample)
Alabama			
Oregon	
Rhode Island		
Tennessee			
West Virginia		

CMS and NORC convened introductory meetings with each pilot state in which NORC introduced the study and provided a guided walk-through of the documentation that lists out the specifications associated with each sampling method. NORC also held follow-up phone calls and responded to email correspondence from states to provide technical assistance, ensure that procedures were followed correctly, and answer state questions. After each state finished the work associated with the chosen option(s), NORC conducted a debriefing phone call with the state to understand the amount of time each option required and to gain feedback to enhance the process.

Pilot Results

Based on the feedback from states for each of the three options, Table 2 shows the level of effort in hours for each state under each option. As shown, there was a wide range of hours needed across different states under each option. Averaging these time estimates together, NORC anticipated Option 1 will require 9 hours, on average; Option 2 will require approximately 57 hours, on average; and Option 3 will require 9 hours in addition to data extraction for a total of 66 hours, on average. The process for Option 3 requires the state to first complete the steps for Option 2 (i.e. data extraction).

Table 2. Time Estimates by Option and State

State	Option 1 (MSIS)	Option 2 (Data Extract)	Option 3 (Select Sample)
Alabama		15 hours across 3 weeks	10 additional hours once data extract is pulled, over 1 additional week
Oregon	16 hours across 1 week
Rhode Island		35 hours across 10 weeks
Tennessee	2 hours*		8 additional hours once data extract is pulled, over 2 additional days
West Virginia		120 hours across 2 weeks

* Projection of hours. Tennessee is experiencing difficulty receiving the MSIS file from NORC, so projected hours for this Option based on their experiences with Option 3.

Through the pilot study, NORC was able to gain feedback on the specifications for each option, as well as the process in general, both of which will help NORC and states during the main survey collection. For example, one state suggested it would be helpful to include the MSIS definitions in the sampling specifications resource because the state staff who are responsible for pulling together the sample for the CAHPS survey may not be the same staff that are responsible for preparing MSIS files for submission to CMS. Another state noted that including more details about file layouts, variables, and variable descriptions would be helpful in the specifications.