External Quality Review of Medicaid MCOs and Supporting Regulations in 42 CFR 438.360, 438.362, and 438.364

OMB Approval No.
0938-0786

EQR PROTOCOL 5
VALIDATION AND IMPLEMENTATION
OF SURVEYS
A Voluntary Protocol for External Quality Review (EQR)
Protocol 1: Assessment of Compliance with Medicaid Managed Care Regulations
Protocol 2: Validation of Measures Reported by the MCO
Protocol 3: Validation of Performance Improvement Projects (PIPs)
Protocol 4: Validation of Encounter Data Reported by the MCO
Protocol 5: Validation and Implementation of Surveys
Protocol 6: Calculation of Performance Measures
Protocol 7: Implementation of Performance Improvement Projects (PIPs)
Protocol 8: Focused Studies
Appendix V: Information Systems Capabilities Assessment (ISCA)

Department of Health and Human Services (HHS)
Centers for Medicare & Medicaid Services (CMS)

Protocol 5
Version 2.0
December 1, 2011

TABLE OF CONTENTS
PURPOSE AND OVERVIEW OF THE PROTOCOL .............................................................3
ACTIVITY 1: IDENTIFY SURVEY PURPOSE(S), OBJECTIVE(S) AND INTENDED USE..... 4
ACTIVITY 2: SELECT THE SURVEY INSTRUMENT...........................................................5
Option 1: Use Existing, Validated Survey Instruments .....................................................5
Option 2: Adapt Existing Surveys ....................................................................................6
Option 3: Develop New Survey Instruments ....................................................................6
ACTIVITY 3: DEVELOP THE SAMPLING PLAN ..................................................................7
Step 1: Identify the Study Population ...............................................................................7
Step 2: Determine the Type of Sampling to be Used .......................................................7
ACTIVITY 4: DEVELOP A STRATEGY TO MAXIMIZE THE RESPONSE RATE ................. 7
Step 1: Specify the Strategy for Contacting Target Respondents ....................................7
Step 2: Maximize the Response Rate ..............................................................................8
ACTIVITY 5: DEVELOP QUALITY ASSURANCE PLAN ......................................................9
ACTIVITY 6: IMPLEMENT THE SURVEY ..........................................................................10
ACTIVITY 7: PREPARE AND ANALYZE DATA OBTAINED FROM THE SURVEY ........... 11
ACTIVITY 8: DOCUMENT THE SURVEY PROCESS AND RESULTS .............................. 11
VALIDATE SURVEYS (OPTIONAL)....................................................................................12
Step 1: Review Survey Purpose(s), Objective(s) and Intended Use .............................. 12
Step 2: Assess the Reliability and Validity of the Survey Instrument .............................. 13
Step 3: Review the Sampling Plan.................................................................................13
Step 4: Review the Adequacy of the Response Rate.....................................................14
Step 5: Review Survey Implementation .........................................................................14
Step 6: Review Survey Data Analysis and Findings/Conclusions .................................. 15
Step 7: Document Evaluation of Survey ........................................................................15
REFERENCES....................................................................................................................16
Attachments
Attachment A: Survey Validation Worksheet

EQR Protocol 5:
Validation and Implementation of Surveys
December 2011

2

PURPOSE AND OVERVIEW OF THE PROTOCOL
This voluntary protocol provides procedures for the administration or validation of managed care
enrollee and other health care consumer surveys. The protocol is also applicable to surveys of
other groups such as beneficiaries and providers. The instructions are intended for EQROs,
States, and other external quality reviewers. Because the protocol may be used for a variety of
purposes, no specific survey instrument, sampling method, or approach to analysis and
reporting is recommended. However, the protocol does provide specific information about the
Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys and reporting
formats, which have been developed by the Agency for Healthcare Research and Quality
(AHRQ) in collaboration with the CAHPS Consortium. States frequently use CAHPS surveys to
evaluate Medicaid beneficiaries’ experiences with managed care.
The protocol assumes the State has determined, independently, or in consultation with the
EQRO, the survey:
1. Goals and objectives;
2. Instrument (i.e., questionnaire); and
3. Intended audience(s) for findings.
The protocol includes the following eight activities:
1. Identify survey purpose(s), objective(s) and intended use;
2. Select the survey instrument;
3. Develop the sampling plan;
4. Develop a strategy for maximizing the response rate;
5. Develop a quality assurance plan;
6. Implement the survey;
7. Prepare and analyze the data obtained from the survey; and
8. Document the survey process and results.
The survey must be designed and administered to produce valid and reliable information.
Validity refers to the degree to which the information is what was intended to be measured.
Reliability refers to: 1) the internal consistency of a survey, and 2) the reproducibility of survey
results when the survey is administered under different conditions (e.g., by different people, or
at different times). This protocol also contains information about validating results of a
completed survey. The EQRO should use a worksheet such as that found in Attachment A to
document activities. Please note that the Attachment A worksheet only provides examples of
the first three activities; it does not outline validation steps for all eight activities listed in this
protocol. Information for the remaining five activities is provided throughout this Protocol.

EQR Protocol 5:
Validation and Implementation of Surveys
December 2011

3

ACTIVITY 1: IDENTIFY SURVEY PURPOSE(S), OBJECTIVE(S) AND
INTENDED USE
In order to develop the survey, the EQRO must determine the survey’s purpose, objective, and
intended use. Therefore, the EQRO should request the State provide answers to the following
questions:
•
•

“What does the State want to learn from the survey?”
“What does the State plan to do with the survey results?”

Examples of survey purposes include:
•
•
•

Monitoring and evaluating the quality of care provided to beneficiaries;
Assisting beneficiaries in choosing among MCOs; and/ or
Informing quality improvement initiatives.

The State should specify the required level of analysis, including populations or subpopulations,
to allow for selection of standardized survey instruments. While the EQRO must collect data on
individual MCOs, these data may also be used to analyze provider groups or the State’s entire
managed care program depending upon what the State is interested in knowing. The State
should provide concise, explicit, and measurable survey objectives. For example, if the purpose
of the survey is to identify how satisfied managed care enrollees are with their health care,
specific objectives might include determining if individual MCO enrollees are:
•
•
•

Satisfied with their access to specialty care;
Involved in planning for their own treatment; and
Satisfied with the quality of their interactions with their primary care provider.

The State should specify how it intends to use survey results, since intended use and audience
will affect the format of the report(s) the EQRO must prepare. Such audiences and uses could
include the following:
•

•

•

Beneficiaries and their families choosing between fee-for-service (FFS) and MCOs or
among MCOs. Increasingly, consumers rely on survey information to inform their choice
of healthcare options. This requires the survey design to allow for MCO - to - MCO
comparisons.
MCO managers and providers identifying areas of superior health service delivery as
well as areas needing improvement. The State will need to develop a policy specifying
whether the MCO may receive individual enrollee survey results or a summary, and
determine if MCO-level results will compromise confidentiality.
State policy makers monitoring how beneficiaries perceive the care they receive under

EQR Protocol 5:
Validation and Implementation of Surveys
December 2011

4

the State’s managed care initiative. In this case, the survey analysis would need to
provide information on the State’s managed care initiative as a whole, on the individual
MCO, and may need to consider a comparison to the FFS or Primary Care Case
Management (PCCM) System.

ACTIVITY 2: SELECT THE SURVEY INSTRUMENT
The State may choose the survey instrument independently or in consultation with the EQRO.
There are three approaches to selecting a survey instrument:
1. Use an existing instrument such as the CAHPS survey;
2. Adapt an existing instrument with additional State-specific supplemental questions; or
3. Develop a new instrument.
The State’s choice should be consistent with the survey purposes, objectives and units of
analysis and should promote the collection of reliable and valid data. The CAHPS survey
instruments and reporting formats have undergone rigorous testing for reliability and validity,
including focus group interviewing, cognitive interviewing, and field-testing. CAHPS also
provides for the addition of MCO or State-specific questions. As these standardized instruments
are used nationally, national and regional benchmarks are available for comparison.
Should the State choose to modify an existing instrument or develop one of its own, it should
establish face and content validity and pre-test the tool for reliability. The State can assess face
and content validity by convening one or more focus groups that include targeted survey
respondents and individuals with subject matter expertise. The State can assess reliability using
the test-retest method in which the survey is administered to the same group at two different
times. A correlation coefficient is calculated and indicates the reproducibility of results.
Correlation coefficients with r-values at or above 0.70 indicate good reliability.

Option 1: Use Existing, Validated Survey Instruments
Due to the costs associated with developing and testing a new survey instrument, the State
should use an existing survey instrument whenever possible; especially one that has undergone
strong reliability and validity testing, such as CAHPS. CAHPS testing included cognitive testing
during the development and evaluation phases, calculation of reliability estimates in a sample of
Medicaid enrollees and private health insurance purchasers, and convening of focus groups to
test relevance of survey concepts and items. In addition, the validated CAHPS surveys include
a free, easy-to-use CAHPS Survey and Reporting Kit. The kit contains a set of mail and
telephone survey questionnaires including Spanish language versions and supplement item
sets for the Adult and Child surveys. The kit covers a variety of topics of potential interest to the
State or MCO, sample reporting formats, and a handbook with step-by-step instructions for
sampling, administration, analysis, and reporting. The handbook’s instructions are easy to
understand and allow for flexibility within the sample and analysis design. The handbook

EQR Protocol 5:
Validation and Implementation of Surveys
December 2011

5

includes a comprehensive, statistical software package and a telephone number and e-mail
address for the technical assistance hotline.
The State can select from a wide variety of other questionnaires. The State should question an
existing instrument’s reliability and validity. Even pre-existing, validated survey instruments may
not have been validated in a Medicaid or CHIP population. Selection of instruments not
validated in the target population may not yield valid or reliable results.
Another advantage of selecting an existing instrument is that use of the same questionnaire,
methods, and analysis for surveys of MCOs, populations, and States, allows the State to
compare its findings against those of other studies. For example, using the standardized
CAHPS survey instruments, methodology, and report formats provides reliable and valid
measures that the State can use to compare performance across MCOs, between MCOs and
fee-for-services (FFS) or primary care case management (PCCM), and across States.
When using an existing survey instrument, the EQRO should document the extent of reliability
and validity testing of the survey instrument.

Option 2: Adapt Existing Surveys
The State may decide to adapt an existing survey by adding or deleting items, modifying
questions, or using only certain groups of questions relevant to the State’s survey objectives.
Adapting an existing survey is easier if CAHPS is used, as the CAHPS survey instruments have
been specifically designed with opportunities to customize the questionnaire by adding
questions from a set of optional supplement items, as well as State-designed questions. This
has the advantage of providing a validated instrument that allows comparison, while
accommodating special questions of particular interest, either to the State, or to each MCO.
Most pre-existing questionnaires are not designed to accommodate such modification and
require additional testing.
Modifying an existing questionnaire provides the State with the flexibility to add or change the
data to be collected while providing many of the advantages of using a pre-existing
questionnaire. However, adding, deleting, or modifying questions may undermine the validity
and reliability of the questions, as well as the survey overall. Validated questionnaires are tested
“as a whole,” and modifications can change the focus and purpose of the questionnaire. When a
State adapts an existing questionnaire, it should obtain the advice of an individual
knowledgeable in survey design (preferably knowledgeable about the original survey) to provide
advice on the modification and how to appropriately test for reliability and validity.

Option 3: Develop New Survey Instruments
The State may decide to use or develop a new survey when the purposes and objectives of the
study require answers to questions that are not addressed by existing instruments. A well-

EQR Protocol 5:
Validation and Implementation of Surveys
December 2011

6

designed instrument can capture information that is of interest and relevant to the questions
under study. However, assuring the reliability and validity of new surveys is costly and time
consuming. Without such reliability and validity testing, surveys can have methodological flaws
making the results suspect. When developing new EQR survey instruments, the State should
involve an expert in survey design and assure testing of the instrument for validity and reliability.
In addition to the cost, this approach has the disadvantage of having limited benchmarks for
comparison of results.

ACTIVITY 3: DEVELOP THE SAMPLING PLAN
The EQRO should develop a sampling plan that represents all eligible enrollees within the
MCO. If using CAHPS, the CAHPS Health Plan Surveys Reporting Kit contains a handbook with
step-by-step instructions for sampling. The handbook’s instructions are easy to understand and
allow for flexibility in the sample and analysis design. The handbook also includes a
comprehensive, statistical software package. In addition, HEDIS®1 CAHPS technical
specifications provide specific instructions for sampling and administration to ensure
comparability with other MCO survey results.

Step 1: Identify the Study Population
The EQRO must define the population to be studied (e.g., all Medicaid or CHIP beneficiaries
enrolled in an MCO or all children with special health care needs) and the data sources from
which to draw the sample (the sampling frame).

Step 2: Determine the Type of Sampling to be Used
There are two basic types of sampling: probability and non-probability Probability sampling is
subject to the laws of chance and includes simple random sampling and stratified random
sampling of specific populations. Non-probability sampling is based on the decisions of those
administering the survey and not on random chance. Because of the risk of biased results and
the obstacles to statistical analysis, non-probability sampling is discouraged. For more
information, see Appendix II.

ACTIVITY 4: DEVELOP A STRATEGY TO MAXIMIZE THE RESPONSE
RATE
The EQRO should develop a strategy for contacting and following up with the sample
respondents that will maximize the response rate.

Step 1: Specify the Strategy for Contacting Target Respondents
1

®

HEDIS is a registered trademark of the National Committee for Quality Assurance (NCQA).

EQR Protocol 5:
Validation and Implementation of Surveys
December 2011

7

The EQRO should develop a strategy for locating and contacting the individuals selected in the
sample. Mail and telephone surveys are the most practical and widely used data collection
modes. Following up by telephone with those who do not respond to a mail survey results in
higher response rates than either method alone.
Other frequently used methods are Internet surveys and interactive voice recognition (IVR)
surveys. These methods can combine automated and human contact for different aspects of the
survey. Internet surveys typically yield low response rates and can be subject to bias resulting
from under-coverage and non-response attributed to the lack of Internet or email access for
select respondents and households. Initial studies of IVR techniques suggest that further study
is necessary before widespread adoption can be feasible. Personal interviews are expensive
and are seldom used outside of research environments.
The EQRO should identify the specific data it needs to administer the survey such as:
• Individual’s full name;
• Address;
• Home and cell phone numbers;
• E-mail address;
• Date of birth;
• Primary language;
• Name of the individual’s MCO; and
• Length of enrollment.
The EQRO should collect as complete information as possible and consider that some
information may be verified through the State’s eligibility files or the MCOs’ enrollee files. The
EQRO should document its plans to locate and contact respondents, including sending names
in the sample to a telephone number look-up vendor or using a change-of-address database
vendor.
The strategy should include steps the EQRO will take if the respondent rates are lower than
expected. These steps may include mailing a reminder postcard or second survey, making a
follow-up phone call to non-respondents to a mail survey or conducting repeat calls in a
telephone survey. The EQRO should track and follow up on the number of respondents that
could not be contacted or failed to respond.

Step 2: Maximize the Response Rate
The survey plan should specify the response rate established by the State; procedures for
handling missing data; and the methods for calculating response rates. The CAHPS developers
suggest that the target response rate for administering CAHPS to Medicaid beneficiaries range
from 40 percent to 50 percent. They also recommend that survey vendors focus on strategies

EQR Protocol 5:
Validation and Implementation of Surveys
December 2011

8

that promote high response rates and develop a plan of corrective actions if the response rate
falls short of the goal.
Research suggests a number of strategies to improve response rates. These include:
•

•
•
•

Including a cover letter that emphasizes survey sponsorship (e.g., on State government
letterhead signed by the Agency Director), and includes:
a. Purpose of the survey;
b. A guarantee of anonymity and confidentiality;
c. Selection criteria for participation;
d. Benefits to the respondent; and
e. How to return the survey;
Using personalized correspondence with respondents (e.g., addressing all
correspondence to the respondent by name);
Providing stamped return envelopes using first class postage; and/or
Using follow-up contacts (e.g., reminder postcards, second mailing of the questionnaire,
telephone contact or special postage mailing of a second questionnaire).

The EQRO may need to arrange for translation of surveys and correspondence into languages
other than English. Failure to provide surveys in the respondent’s primary language may result
in excluding vulnerable segments of the population. The State and MCOs should have
information about beneficiary language. If this information is not readily available, the EQRO
might translate a sentence in the introductory letter into the most common languages in the
area, inviting the individual to call for more information or to request a specific translation.
During the call, the respondent may request a translated survey, complete the survey over the
phone, or schedule a more convenient telephone interview time.

ACTIVITY 5: DEVELOP QUALITY ASSURANCE PLAN
The EQRO should develop a quality assurance plan that includes processes to monitor,
evaluate and review all aspects of the survey administration procedure. The purpose of a quality
assurance plan is to document a strategy of reviews and audits to assure compliance with the
appropriate processes.
The quality assurance plan should clearly identify
• Who performs the check;
• What checks are performed;
• How the checks are performed;
• Frequency of these checks;
• Percentage of records that are to be checked, and
• Corrective actions required if issues are identified.

EQR Protocol 5:
Validation and Implementation of Surveys
December 2011

9

The following checks should be performed during survey administration:
• Sampling—ensure the sampling plan is followed and an initial contact is attempted for
every sampled member.
• Mail—review contents of mailing packet, such as cover letter and questionnaire, for
accuracy, print smearing, fading, or misalignment.
• Telephone—review training of the interviewers and telephone scripts for accuracy.
Monitor live telephone interviews to confirm that interviewers read questions verbatim
and accurately capture the response given.
• Internet—review the programming and content for accuracy.
• IVR—review scripts for accuracy and monitor to confirm the script is read verbatim and
the system accurately captures the response entered/spoken by the respondent
• Review data entry/capture of returned mail surveys for accuracy.

ACTIVITY 6: IMPLEMENT THE SURVEY
Based on the information obtained in Activities 1, 2, and 3, and the decisions made in Activities
4, 5 and 6, the EQRO should prepare a work plan to govern the implementation of the survey.
The work plan should specify routine aspects of project management including key staff and
their responsibilities, timelines, and deliverables; how to document each phase of the survey
process; and a quality assurance work plan to proactively identify and resolve problems. The
work plan should specify the number, format, and content of the reports for submission to the
State. The work plan should include a description of any reports that the EQRO will be
responsible to publicly release, if this is part of the EQRO’s scope of work.
Key methodological issues to be addressed in the work plan include the following:
• Specifications and procedures for formatting, reproducing, and distributing the survey
questionnaire;
• Procedures for assuring the confidentiality of the data in compliance with HIPAA
regulations;
• Data collection, data entry, and data quality controls;
The EQRO should determine procedures for handling responses that fail edit checks,
treatment of missing data, and procedures for determination of usable/complete
surveys. For a survey to be considered “complete,” the State or EQRO should
establish a pre-determined number of questions that must be answered by the
respondent.
• Data analysis plan including statistical methodology;
The EQRO should involve a statistician in developing an analysis plan that supports
the State’s objectives for the survey.
• Production of data files and their format and delivery;
If feasible, the EQRO should provide the State with a mock-up of survey results prior
to administering the survey. This will help assure that the survey information is
consistent with the State’s planned use of results.

EQR Protocol 5:
Validation and Implementation of Surveys
December 2011

10

•

Procedures for assessing quality of data collection activities.

The EQRO should obtain State approval of the work plan prior to implementation and
implement the survey in accordance with the approved work plan.

ACTIVITY 7: PREPARE AND ANALYZE DATA OBTAINED FROM THE
SURVEY
Once the surveys have been completed and returned, the EQRO must prepare the data for, and
conduct, the analysis. This includes data quality control procedures (e.g., cleaning and editing),
data analysis, and production of data files.
Consistent with the implementation plan, the EQRO should implement procedures for
responses that fail edit checks, address missing data, and remove data from surveys
determined to be unusable. The EQRO should document the reasons for all exclusions or
adjustments of data used for the analysis.
Following the analysis plan, the EQRO should generate frequency distributions for each survey
question and calculate statistics, such as measures of central tendency. In addition, the EQRO
should examine differences in survey results among MCOs, between MCOs and the FFS or
PCCM population, or between MCOs in the State and nation or region. The comparisons should
follow the analysis plan approved by the State.
Consistent with the purposes and objectives of the survey, the EQRO could analyze and report
on sub- populations within each MCO. For example, the State may be interested in whether
responses differ significantly across geographic locations, racial/ethnic groups, socio-economic
groups, or other identifiable subgroups.

ACTIVITY 8: DOCUMENT THE SURVEY PROCESS AND RESULTS
Statistical graphs should accompany narrative text to aid comparison and interpretation. For
example, bar graphs and comparison charts, such as those recommended by CAHPS, convey
important information about the performance of each MCO and indicate meaningful differences
among MCOs.
The EQRO should prepare and submit reports in the agreed format documenting the survey
process and results, including:
1. Survey purpose and objectives;
2. Survey implementation and analysis (Activities 2 - 7), including challenges
encountered, lessons learned, and recommendations for improving future efforts;
3. Data obtained, including raw data files and analyses;
4. Public reports, presentations or web site designs developed for public reporting.

EQR Protocol 5:
Validation and Implementation of Surveys
December 2011

11

5. Conclusions drawn from the data;
6. A detailed assessment of each MCO’s strengths and weaknesses with respect to
access, quality, and/or timeliness of health care furnished to enrollees; and
7. Methodologically appropriate, comparative information about all MCOs, as
determined by the State.

VALIDATE SURVEYS (OPTIONAL)
This is a voluntary activity used to assess the methodological soundness of a completed survey.
Many MCOs contract with NCQA-certified survey vendors to conduct CAHPS 4.0H surveys
following a standardized protocol. The State may rely on such surveys without further validation.
However, some MCOs, States, and local healthcare collaboratives conduct or sponsor surveys
using other instruments or methods. The State needs assurance that the results of these other
surveys are valid and reliable.
The activities in this protocol focus on a review of survey procedures. They do not include
collecting survey data anew from the initial survey respondents to verify their responses.
The protocol specifies seven activities the EQRO should follow to assess the methodological
soundness of a completed survey:
1. Review survey purpose(s) and objective(s) and intended use.
2. Assess the reliability and validity of the survey instrument.
3. Assess the sampling plan.
4. Assess the adequacy of the response rate.
5. Assess implementation.
6. Review survey data analysis and findings/conclusions.
7. Document evaluation of survey.
The EQRO should use a worksheet, such as that found in Attachment A, to document its
activities. The EQRO should identify documentation that it reviewed the survey procedures and
note its findings for each activity listed. The EQRO should document the absence of
documentation for a particular activity because that is relevant to the EQRO’s assessment of
survey validity.

Step 1: Review Survey Purpose(s), Objective(s) and Intended Use
The State, MCO or a health care collaborative may have conducted a survey prior to the
EQRO’s engagement. To understand and evaluate the adequacy of the prior survey to assess
access, timeliness, or quality of care, the EQRO should communicate with the entity(ies) that
sponsored or administered the survey to understand the survey’s purpose(s), objective(s), and
intended use. See Part 1, Activity 1 for more information about defining the survey purpose and
intended use.

EQR Protocol 5:
Validation and Implementation of Surveys
December 2011

12

Step 2: Assess the Reliability and Validity of the Survey Instrument
A survey may have been administered using:
• A pre-existing, validated survey instrument;
• An adaptation of a pre-existing instrument with additional State-specific supplemental
questions; or
• A new instrument developed specifically for the particular survey under review
The EQRO should review the evidence about the reliability and validity testing of any survey
instrument used. The EQRO need not conduct independent validity and reliability testing of the
survey instrument; however, it should evaluate whether such testing was done. The EQRO
should consider the adequacy of the survey’s reliability and validity testing in determining
whether to rely upon the survey findings to inform its analysis and evaluation of access, quality,
and timeliness of health care. (See Part 1, Activity 2 for more information about how States and
EQROs ensure the validity and reliability of survey instruments.)

Step 3: Review the Sampling Plan
The EQRO should assess the sample plan documentation for the following:
• Definition of the study population;
• Specifications for the sample frame;
• Type of sampling used;
• Adequacy of the sample size; and
• Sample selection procedures.
The level of detail involved in this review requires that the EQRO use professional statisticians.
The EQRO must evaluate whether the sample selected was sufficiently representative of the
study population for the EQRO to have confidence in the survey findings. (See Part 1, Activity 3
and Appendix B for more information about sampling.)
Identify the Study Population
The EQRO should document what population the survey was designed to study (e.g., all
Medicaid or CHIP beneficiaries enrolled in MCOs, or all children with special health care needs).
A good sample is representative of the population to be studied in terms of characteristics that
could be associated with differences in survey results (e.g., race, age, MCO affiliation).
Review The Sample Frame
Once the EQRO understands how the target population was defined, it must assess the
construction of the survey’s sample frame. The EQRO’s objective is to determine whether the
sample frame’s construction was appropriate to the survey objectives, or if it could bias the
survey results.
Review The Type Of Sampling Used
EQR Protocol 5:
Validation and Implementation of Surveys
December 2011

13

The EQRO should evaluate whether the sampling method used was appropriate to the survey
purpose. For more information, see Appendix II.
Review Adequacy of Sample Size
Two factors influence the determination of the appropriate sample size for a survey:
1. The acceptable margin of error; and
2. The confidence levels.
The EQRO should determine an acceptable margin of error for the survey results in relation to
the States need to assess access, timeliness, or quality of care. The EQRO should determine
whether the sample size was appropriate for the survey’s original purpose, as well as its
adequacy to inform the EQRO’s current quality review efforts. The sample size should be
determined by a statistical software program or a statistician. See Step 5 for more information
on determining an appropriate sample size.
Review The Sample Selection Procedures
The EQRO should review the sample selection procedures including reviewing the statistical
program or other process used to generate the sample. The EQRO should determine the extent
to which the selection of sample members was conducted to protect against bias.

Step 4: Review the Adequacy of the Response Rate
The EQRO should assess the response rate, whether the survey employed a reasonable
method for calculating the response rate, potential sources of non-response and bias, and the
extent to which the response rate weakens or strengthens the generalizeability of the survey
findings. The EQRO should consider national standards for acceptable survey response rates in
evaluating the response rate achieved. See Part 1, Activity 5 for more information on response
rate targets.

Step 5: Review Survey Implementation
The EQRO should review the survey implementation and quality assurance plans and
documentation of the survey administration. It should assess whether survey implementation
conformed to the plans. The EQRO should specifically consider the following:
• The presence and comprehensiveness of a survey quality assurance plan;
• Problems detected and corrections implemented during the survey process;
• How the survey questionnaire was administered, including formatting and distribution of
mailed surveys or scripting and training of telephone surveys;
• Confidentiality procedures followed; and
• Data collection, data entry, and data quality control methods used, including reports of
missing data, data that failed edit checks, and incomplete or unusable surveys.

EQR Protocol 5:
Validation and Implementation of Surveys
December 2011

14

Step 6: Review Survey Data Analysis and Findings/Conclusions
The EQRO should review how the survey data were analyzed, including the statistical
procedures used and comparisons made. The EQRO should assess whether the analysis was
appropriate to the survey purpose, whether appropriate statistical tests were applied, and how
well the survey findings were supported by the data. See Part 1, Activities 7 and 8 for more
information about analysis and presentation of survey findings.

Step 7: Document Evaluation of Survey
Using the information obtained from Activities 1 - 6, the EQRO should assess whether the
findings can be generalized to the population from which the sample was drawn. The EQRO
should document these conclusions and provide written findings on:
• The survey’s technical strengths and weaknesses;
• The limitations/generalizeability of survey findings;
• Conclusions drawn from the survey data;
• Assessment of access, quality, and/or timeliness of healthcare furnished to beneficiaries
by the MCO if not done as part of the original survey report; and
• Comparative information about all MCOs, as appropriate.

EQR Protocol 5:
Validation and Implementation of Surveys
December 2011

15

REFERENCES
Agency for Healthcare Research and Quality: CAHPS Overview
http://www.cahps.ahrq.gov/content/cahpsOverview/OVER_Program.asp?p=101&s=12, last
accessed 11/18/2010.
Babbie, E. “The Practice of Social Research.” 12th Edition. Chapter 9. Wadsworth Publication
Company: Belmont, CA. 2010.
Brown, J., Nederend, S., Hays, R., Short, P.F., & Farley, D. “Special Issues in Assessing Care
of Medicaid Recipients.” Medical Care. Volume 37, Number 3: MS79-88, MS106-MS115. March
1999.
Cella, D., Hernandez, L., Bonomi, A., Corona, M., Vaquero, M., Shiomoto, G., Baez, L.
“Spanish Language Translation and Initial Validation of the Functional Assessment of Cancer
Therapy Quality-of-Life Instrument.” Medical Care. Volume 36, Number 9: 1407-1418. 1998.
Converse, P.D., Wolfe, E.W., Huang, X., Oswald, F.L. “Response Rates for Mixed-Mode
Surveys Using Mail and E-mail/Web.” American Journal of Evaluation. Volume 29, Number 1:
99-107. 2008.
Dillman, Don A., Sinclair, Michael D., Clark, Jon R. “Effects of Questionnaire Length,
Respondent-Friendly Design, and a Difficult Question on Response Rates for OccupantAddressed Census Mail Surveys”. Public Opinion Quarterly. Volume 57: 289-304. 1993.
Dillman, Don A., Mail and Telephone Surveys: The Total Design Method. New York, A WileyInterscience Publication, 1978.
Don A. Dillman, Corresponding Author Contact Information, E-mail The Corresponding Author,
Glenn Phelps, Robert Tortora, Karen Swift, Julie Kohrell, Jodi Berck and Benjamin L. Messer.
“Response rate and measurement differences in mixed-mode surveys using mail, telephone,
interactive voice response (IVR) and the Internet.” Social Science Research. Volume 38, Issue
1. 1-18. March 2009.
Donelan, Karen. “How Surveys Answer A Key Question: Are Consumers Satisfied With
Managed Care?” Managed Care, Volume 5, Issue 2: 17-24. February 1996.
Gallagher, P.M., Fowler, F.J., & Stringfellow, V.L. “Respondent Selection by Mail: Obtaining
Probability Samples of Health Plan Enrollees.” Medical Care. Volume 37, Number 3: MS50-8.
March 1999.
Gold, Marsha & Eden, Jill. “Monitoring Health Care Access Using Population-Based Surveys:
Challenges in Today’s Environment, Information for Policymakers.” Methods Brief, August 1998.
EQR Protocol 5:
Validation and Implementation of Surveys
December 2011

16

Harris-Kojetin, L.D., Fowler, F.J., Brown, J.A., & Sweeney, S.F. “The Use of Cognitive
Testing to Develop and Evaluate CAHPS 1.0 Core Survey Items.” CAHPSTM Medical Care.
Volume 37, Number 3: MS10-21. March 1999.
Hays, R.D., Shaul, J.A., Williams, V.S.L., Lubalin, J.S., Harris-Kojetin,L., Sweeney, S.F., &
Cleary, P.D. “Psychometric Properties of the CAHPSTM 1.0 Survey Measures. Consumer
Assessment of Health Plans Study.” Medical Care. Volume 37, Number 3: MS22-31. March
1999.
Heerwegh, D., Loosveeldt, G. “Face-to-face versus web surveying in a high-internet-coverage
population.” Public Opinion Quarterly. Volume 72, Number 5: 836-846. 2008.

END OF PROTOCOL

EQR Protocol 5:
Validation and Implementation of Surveys
December 2011

17