CMS-10549_MCBS Generic Clearance_Supporting Statement A

Supporting Statement A
for Request for Generic Clearance:
Questionnaire Testing and Methodological Research for the
Medicare Current Beneficiary Survey (MCBS)

Contact Information:
William S. Long
Contracting Officer’s Representative, Medicare Current Beneficiary Survey
Office of Information Products & Data Analytics
Office of Enterprise Management/CMS
7500 Security Boulevard, Mail Stop B2-29-04
Baltimore, MD 21244
(410) 786-7927
[email protected]
(410) 786-5515 (fax)

January 22, 2015

1

Table of Contents
A.
JUSTIFICATION ................................................................................................................ 4
A1. Circumstances Making the Collection of Information Necessary ................................... 4
A2. Purpose and Use of Information Collection ..................................................................... 9
A3. Use of Information Technology and Burden Reduction .................................................. 9
A4. Efforts to Identify Duplication and Use of Similar Information ...................................... 9
A5. Impact on Small Businesses and Other Small Entities .................................................... 9
A6. Consequences of Collecting the Information Less Frequently ........................................ 9
A7. Special Circumstances Relating to Guidelines of 5 CFR 1320.5..................................... 9
A8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside
Agencies .................................................................................................................................... 10
A9. Explanation of Any Payment or Gift to Respondents .................................................... 10
A10.
Assurances of Confidentiality Provided to Respondents ........................................... 10
A11.
Justification for Sensitive Questions .......................................................................... 10
A12.
Estimates of Annualized Burden Hours and Costs ..................................................... 11
A13.
Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers 12
A14.
Annualized Costs to the Federal Government ............................................................ 12
A15.
Explanation for Program Changes or Adjustments .................................................... 12
A16.
Plans for Tabulation and Publication and Project Time Schedule ............................. 12
A17.
Reason(s) Display of OMB Expiration Date is Inappropriate .................................... 12
A18.
Exceptions to Certification for Paperwork Reduction Act Submissions.................... 12

2

LIST OF ATTACHMENTS
Attachment 1: 60-day Federal Register Notice

3

A.

JUSTIFICATION

A1.

Circumstances Making the Collection of Information Necessary

This new OMB generic clearance request is for MCBS Questionnaire Testing and
Methodological Research. This request encompasses development and testing of MCBS
questionnaires, instrumentation, as well as methodological experiments. The Medicare Current
Beneficiary Survey (MCBS) (OMB No. 0938-0568) is an in-person, nationally-representative,
longitudinal survey of Medicare beneficiaries that is sponsored by the Centers for Medicare &
Medicaid (CMS) and directed by the Office of Information Products and Data Analytics
(OIPDA) in partnership with the Center for Medicare and Medicaid Innovation (CMMI). The
MCBS operates under legal authority through the Medicare Prescription Drug, Improvement and
Modernization Act. The MCBS is the most comprehensive and complete survey available on the
Medicare population and is essential in capturing data not otherwise collected through CMS
operations. The activities under the MCBS are currently conducted by contract with NORC at
the University of Chicago.
The MCBS captures beneficiary information, whether aged or disabled, living in the community
or facility, or serviced by managed care or fee-for-service. Data produced as part of the MCBS
are enhanced with CMS administrative data to provide users with more accurate and complete
estimates of total health care costs and utilization. The MCBS has been continuously fielded for
more than 20 years and consists of three annual interviews per survey participant. The MCBS
has been at the forefront of in-person survey collection and data processing, most notably as one
of the first surveys to successfully 1) implement a computer assisted personal interview (CAPI)
and 2) match survey and claims data to adjust and correct for underreporting in survey reported
health care utilization. The CMS vision for the MCBS is to continue to provide unique, highquality and high-value data in a timely manner, continue to break ground in innovative, efficient
and analytically powerful new areas of survey administration, design and development, and to
increase the survey’s ability to develop, monitor, assess and evaluate the impact of CMMI care
delivery and payment models. To succeed in these areas, CMS requests this generic clearance
for research and testing activities to accomplish the following goals:
• Improve data quality and accuracy by evaluating and revising existing questionnaire
items;
• Address emerging policy and program issues by testing new questionnaire items;
• Reduce respondent burden by improving questionnaire items, response categories, and
questionnaire flow;
• Reduce survey costs and implement efficiencies by improving questionnaire items and
interview flow, as well as considering new methods and modes of data collection;
• Increase response rates by improving respondent materials and improving questionnaire
content and flow to reduce survey length.
This clearance contains approval for seven types of potential research activities: 1) cognitive
interviewing, 2) focus groups, 3) usability testing, 4) field testing, 5) respondent debriefing
questionnaire, 6) split ballot and other methodological experiments, and 7) research about
4

incentives. CMS will submit individual collections under this generic clearance to OMB, and
will provide OMB with a copy of questionnaires, protocols, and debriefing materials in advance
of any testing activity.
NORC at the University of Chicago, under contract with CMS to administer the MCBS, will
conduct the research testing. NORC employs methodological specialists, who will collaborate
with CMS to examine questionnaire components from MCBS and compare those existing
questions and methods with those “state of the science” in other federal agencies, or other
academic or professional institutions. Specific topics to be addressed will be outlined in
individual collection requests under the generic clearance. All data collection and analysis will
be performed in compliance with OMB, Privacy Act, and Protection of Human Subjects
requirements.
The general methods proposed for each type of research activity under this clearance are
described below.
Cognitive Interviewing. Cognitive pretesting is an important innovation in the development and
testing of survey questionnaires that has emerged over the last 20 years. Its chief strength is in
providing a structured methodology for ascertaining whether the respondent has understood the
questions in the way CMS and researchers intend them to be understood, and to assess the ability
of respondents to provide meaningful and accurate information. Cognitive interviewing is done
through the administration of questions by a specially trained and experienced cognitive
interviewer, followed by probes to ascertain comprehension, memory, judgment processes, and
topic sensitivity. A secondary purpose is to make sure that issues pertinent to the research are
covered adequately. The cognitive interviewing process often includes techniques, such as
observation and coding of respondent behaviors (e.g., responses of “don’t know” and requests
for question clarification), and in-depth debriefings with respondents, survey methodologists and
interviewers to fully understand the functioning of a survey questionnaire.
Cognitive interviewing offers a detailed depiction of question interpretations and processes used
by respondents to answer questions—processes that ultimately produce the survey data.
Cognitive interviewing is useful not only for investigating individual questionnaire items, but
also for understanding how contextual factors, such as instructions to the respondent or question
order can influence response and contribute to measurement error. As such, the method offers an
insight that can transform understanding of question validity and response error.
Respondents are typically not selected through a random process, but rather are selected for
specific characteristics such as age, health status or some other attribute that is relevant to the
type of questions being tested. Because the goal is to identify the presence of problems, as
opposed to making estimations or causal statements, a randomly drawn sample is not required.
The interview structure consists of respondents first answering a draft survey question and then
providing explanations to reveal the processes involved in answering the test question.
Specifically, cognitive interview respondents are asked to describe how and why they answered
the question as they did. Through the interviewing process, various types of question-response
5

problems that would not normally be identified in a traditional survey interview, such as
interpretive errors and recall accuracy, are uncovered.
Data collection procedures for cognitive interviewing are different from survey interviewing.
While survey interviewers strictly adhere to scripted questionnaires, cognitive interviewers use
survey questions as starting points to begin a more detailed discussion of questions themselves:
how respondents interpret key concepts, their ability to recall the requested information and to
formulate an answer, and the appropriateness of response categories. Because the interviews
generate narrative responses rather than statistics, results are analyzed using qualitative methods.
This type of in-depth analysis reveals problems in particular survey questions and, as a result,
can help to improve the overall quality of the MCBS. Results of cognitive interviews will be
used to make questionnaire design decisions that minimize survey response error; to enhance our
understanding of the question response process; to develop better standards for questionnaire
design; and to improve data collection procedures.
Cognitive interviewing methodology identifies problems that are missed by traditional field tests.
Field interviewers may not be sufficiently trained to identify questionnaire problems, and such
tests are often conducted too late to allow for substantial revisions to be made. Nevertheless,
field tests are a vital complement to cognitive interviews because they can provide important
information about how new or revised questions perform in a production environment.
Focus Groups. Focus groups are used to obtain insights into target respondent perceptions,
attitudes, and experience during questionnaire and materials development and testing. Focus
groups are usually composed of 8 - 10 people who have characteristics similar to the target
survey population, or subgroups of the target population. The groups are conducted by a
professional moderator who keeps the session on track while allowing respondents to talk openly
and spontaneously. The moderator uses a loosely structured discussion outline, which allows
him/her to change direction as the discussion unfolds and new topics emerge. The interactive
nature of a focus group often encourages a richer discussion than would have been possible in
individual interviews.
Usability Testing. Research on computer-user interface designs for computer-assisted
instruments is often referred to as “usability testing.” This research examines how survey
questions, instructions, and supplemental information are presented on computer instruments
(e.g., CAPI or Computer Assisted Self-Interviewing (CASI) instruments), Audio ComputerAssisted Self-Interview (ACASI), and investigates how the presentation affects the ability of
users to effectively utilize these instruments. Authors of computer-assisted instruments make
numerous design decisions: how to position the survey question on a computer screen; how to
display interviewer instructions that are not to be read to respondents; the maximum amount of
information that can be effectively presented on one screen; how supplemental information such
as “help screens” should be accessed; whether to use different colors for different types of
information presented on the screen; and so on. Research has shown that these decisions can
have a significant effect on the time required to administer survey questions, the accuracy of
question-reading, the accuracy of data entry, and the full exploitation of resources available to
help the user complete his or her task.
6

Usability testing has many obvious similarities to questionnaire-based cognitive research, since it
focuses on the ability of individuals to understand and process information in order to accurately
complete survey data collection. It is also somewhat different, in that the typical user can be an
interviewer (in the case of CAPI instruments) as well as a respondent (in the case of
CASI/ACASI instruments). It also focuses more heavily on matters of formatting and
presentation of information than traditional cognitive testing.
Field tests. Under this research program, field tests are defined as small data collection efforts of
500 cases or less, conducted among subsamples of the MCBS sample. Their main objective is to
evaluate changes to the questionnaire and/or data collection procedures in order to inform the
overall survey. To maximize efficiency and reduce costs, field tests conducted under this
clearance would occur within the production environment. Usually, field tests will be utilized
after cognitive interviews or additional survey research methods are completed that tested new or
revised questions, revisions to questionnaire flow, or data collection methods. Generally, field
tests will be conducted in the respondent's household. Professional MCBS field interviewers will
be trained to administer these test questions or changes to the instrument flow. A subset of these
interviews may be observed by a survey professional from CMS and/or NORC. In cases
involving observation, as the interviewer conducts the interview, the observer compiles notes
regarding respondent misunderstandings or difficulty answering, or questions that interviewers
have difficulty administering, or difficulties with new data collection methodologies, which help
to identify potential question revisions. In addition, CMS staff may conduct analysis of outcome
data such as response rates and response distributions to key items, para-data (e.g., response
times), interviewer observations, and respondent debriefing data. Subject matter staff are
debriefed on these findings and if changes are required, the results of the field test will be used to
modify the questionnaire or data collection procedures for follow-up field tests prior to
recommending changes to the production instrument.
Respondent debriefing questionnaire. In this method, standardized debriefing questionnaires are
administered to respondents who have participated in a field test. The debriefing form is
administered at the end of the interview, and contains questions that probe to determine how
respondents interpret the questions, whether they have problems in completing the
survey/questionnaire, or whether they have questions or concerns about new procedures being
tested. This structured approach to debriefing enables quantitative analysis of data from a
representative sample of respondents, to learn whether respondents can answer the questions, and
whether they interpret them in the manner intended by the questionnaire designers. The
debriefing questionnaire would be administered by professionally trained MCBS field
interviewers.
Split ballot and other methodological experiments. This methodological experiment involves
testing alternative versions of questionnaires, at least some of which have been designed to
address problems identified in questionnaires from previous survey rounds. The use of multiple
questionnaires, randomly assigned to permit statistical comparisons, is the critical component.
We anticipate this type of data collection to occur mainly in-person as part of a field test.
In any split ballot experiments conducted under this clearance, alternative questionnaire versions
7

will be tested. The number of versions tested and the number of cases per version will depend
on the objectives of the test. We cannot specify with certainty a minimum panel size, although
we would expect that no questionnaire versions would be administered to less than fifty persons
in a split ballot experiment.
Split ballot experiments that incorporate methodological questionnaire design experiments will
have a larger maximum sample size (up to several hundred cases per panel) than field tests using
other pretest methods. This will enable the detection of statistically significant differences, and
facilitate methodological experiments that can extend questionnaire design knowledge more
generally. CMS will consult with OMB prior to submission regarding split ballot tests with
sample sizes over 1,000.
Research about incentives. In the original design of the MCBS, $3.00 was provided to each
community survey participant at each interview. In the early 1990’s CAPI laptop battery life
technology could be questionable, especially if an interviewer was conducting multiple
interviews in the course of a day. Therefore, interviewers were instructed to plug in their
laptops, if they could, while conducting the in-person interview. Keeping in mind that many of
the MCBS survey participants live on limited incomes and being mindful of any potentially
added costs, our interviewers offered $3.00 to cover any cost associated with the electrical usage
during the interview. This approach was cleared in the original OMB clearance and all
subsequent applicable clearances.
In 2008 the MCBS was faced with a challenging budget year. As a result, CMS in consultation
with the existing MCBS contractor at the time, determined that the $3.00 electrical usage
compensation was no longer a necessity. Laptops were common place in the community and
there wasn’t the apprehension associated with plugging them into a respondent’s outlet that there
once was. In actuality, the $3.00 compensation was now seen as a very small form of
appreciation by most of the survey participants. Starting in 2009 the compensation was phased
out over the course of four years for continuing survey participants. We began eliminating the
compensation for all new panels entering the survey.
Independent of the prior use of the $3.00 compensation, CMS has seen a small but steady drop in
response rates over time. Response rates for the incoming Supplemental panel has gone from 84
percent in 2001 to just under 70 percent in 2013. These respondents come into the survey each
Fall and their cooperation rates have a long lasting impact to the quality of the data over the four
year period of participation. Therefore, incentives to improve the response rates of the
Supplemental panel would be targeted to gaining initial cooperation. Of similar interest would
be incentive experiments targeted to reduce attrition over the life of enrolled respondents.
CMS would like to evaluate what impact incentives could have on the MCBS response rate.
This evaluation would at first consist of conducting an environmental scan of the state of the
science on respondent incentives in longitudinal surveys and other Federal surveys. From these
findings CMS would consult with OMB about the various kinds of experiments that would both
inform the statistical community at large as well as provide information about improving the
quality of MCBS data and potentially reducing survey costs.

8

A2.

Purpose and Use of Information Collection

The information collected will be used by CMS staff to evaluate and improve the quality of the
data in the MCBS survey. The MCBS has remained virtually unchanged in methodology and
content since it was first fielded in 1991, while the state of the science has adapted to the ever
changing Medicare health care related survey environment. To address a need for
modernization, the MCBS through its contactor will conduct cognitive interviews, focus groups,
usability testing, field tests, split ballot and other methodological experiments, respondent
debriefings, and research on incentives.
The qualitative and quantitative data collected under this testing research program will aid CMS
in its overarching goals for administering the MCBS: improving data quality; addressing
emerging issues; reducing respondent burden; reducing survey costs and implementing
efficiencies; and increasing response rates.
A3.

Use of Information Technology and Burden Reduction

Appropriate technology will be used during testing to keep respondent burden at a minimum. All
cognitive testing will be facilitated by an interviewer, however automated data collection
methods such as Computer Assisted Personal Interviewing (CAPI) and Audio Computer
Assisted Self Interview (ACASI), as well as web-based interviews may be used to reduce
respondent burden. Field testing will employ the usual CAPI data collection method used on the
MCBS. These automated data collection methods reduce the burden on respondents, as both
eliminate the need for respondents to read items and write responses.
A4.

Efforts to Identify Duplication and Use of Similar Information

This testing and methodological research program does not duplicate any other questionnaire
design work being done by CMS or other Federal agencies. No information to be obtained from
the proposed testing currently exists. The research may involve collaboration with staff from
other agencies. All efforts will be collaborative and no duplication in this area is anticipated.
A5.

Impact on Small Businesses and Other Small Entities

There will be no impact on small businesses or other small entities. We do not anticipate that any
small business entities or other small organizations will be involved in this research program.
A6.

Consequences of Collecting the Information Less Frequently

This clearance involves one-time data collection for each testing activity. If the research program
is not conducted, the quality of the data collected in the MCBS would suffer.
A7.

Special Circumstances Relating to Guidelines of 5 CFR 1320.5

None of the special circumstances listed by OMB apply to this MCBS research program.
9

A8.
Comments in Response to the Federal Register Notice and Efforts to Consult
Outside Agencies
A 60-day Federal Register Notice will be published for this clearance request. A copy of the
notice is included as Attachment 1.
A9.

Explanation of Any Payment or Gift to Respondents

Respondents for testing activities conducted in the laboratory (i.e., cognitive interviews, usability
testing and focus groups) under this clearance will receive a small stipend. This practice has
proven necessary and effective in recruiting subjects to participate in this type of small-scale
research, and is also employed by other Federal cognitive laboratories. The standard incentive
for participation in a cognitive interview is $40 for adults, and for participation in a focus group
it is $50 - $75 unless approval is granted by OMB on a case-by-case basis to pay a higher
incentive. Respondents for methods that are generally administered as part of field test activities
(that is, split ballot experiments and respondent debriefing) will not receive payment unless there
are extenuating circumstances that warrant it.
A10.

Assurances of Confidentiality Provided to Respondents

All respondents who participate in research under this clearance will be informed that the
information they provide is confidential and that their participation is voluntary. All participants
in cognitive research will be required to sign written notification concerning the voluntary and
confidential nature of their participation.
For field testing activities, the current MCBS cover letter contains a reference to the Privacy Act
of 1974, as amended. A handout sheet provided to the household respondent at the door and the
nursing home administrator and proxy respondents contains a statement of confidentiality
consistent with the Privacy Act of 1974 and the Paperwork Reduction Act of 1995. Interviewer
training stresses the importance of maintaining confidentiality. The household interviewer's
manual specifically addresses this and it is part of the training for the interviewers (both
household and nursing home). Procedures have been established to maintain and ensure
confidentiality. These include computer security procedures (laptop password encryption).
Any data published will exclude information that might lead to the identification of specific
individuals (e.g., ID number, claim numbers, and location codes). CMS will take precautionary
measures to minimize the risks of unauthorized access to the records and the potential harm to
the individual privacy or other personal or property rights of the individual.
All MCBS survey staff directly involved in MCBS data collection and/or analysis activities are
required to sign confidentiality agreements. Furthermore, all MCBS patient-level data are
protected from public disclosure in accordance with the Privacy Act of 1974, as amended.
A11.

Justification for Sensitive Questions

None of the questions that are included in the current MCBS questionnaires are of a sensitive
nature. However, it is possible that some potentially sensitive questions may be included in
10

questionnaire items that are tested under this clearance. One of the purposes of the testing is to
identify such questions, determine sources of sensitivity, and alleviate them insofar as possible
before they are incorporated into the main MCBS questionnaires.
A12.

Estimates of Annualized Burden Hours and Costs

Table 1 is based on the maximum number of data collections expected on an annual basis. The
total estimated respondent burden and costs are calculated below. Please note that for Field Tests
and Split Ballot or other Experiments, our plan is to conduct these efforts in Production with
existing respondents. Therefore, the burden for their time is captured in the MCBS clearance,
0938-0568.
Table 1. Estimated Annual Reporting Burden, by Anticipated Data Collection Methods
Number of Frequency of Hours Per
Respondents
Response
Response
Cognitive Interviews
(10 questionnaires, 30 respondents
each)
Focus Group Interviews
(10 sessions, 10 respondents each)
Usability testing sessions
(10 sessions, 10 respondents each)
Respondent Debriefing
Questionnaires
(2 field tests, 500 respondents each)
TOTAL

300
100
100

Total Hours

1

2.00

600

1

2.00

200

1

1.50

150

1

10/60

167

1000
1,117

1,500

The estimated annualized costs to respondents is based on the Bureau of Labor Statistics (BLS)
data from May 2013, http://www.bls.gov/oes/current/oes_nat.htm. The mean hourly wage for all
occupations is $22.33.
The estimated annualized annual costs are outlined in Table 2.
Table 2. Estimated Annual Costs
Wages
Cognitive Interviews
(10 questionnaires, 30 respondents each)
Focus Group Interviews
(10 sessions, 10 respondents each)
Usability testing sessions
(10 sessions, 10 respondents each)
Respondent Debriefing Questionnaires
(2 field tests, 500 respondents each)
TOTAL

$22.33
$22.33
$22.33
$22.33

Total Hours
600

$13,398

200

$4,466

150

$3,350

167

$3,729

1,117
11

Total Costs

$24,943

A13.

Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

None.
A14.

Annualized Costs to the Federal Government

At this time, we cannot anticipate the actual number of participants, length of interview, and/or
mode of data collection for the surveys to be conducted under this clearance. Thus, it is
impossible to estimate in advance the cost to the Federal Government. Costs will be covered by
CMS under the existing MCBS budget.
A15.

Explanation for Program Changes or Adjustments

There are no changes in burden, as this is a new request for a generic clearance.
A16.

Plans for Tabulation and Publication and Project Time Schedule

This clearance request is for questionnaire development activities and for developmental work
that will guide future questionnaire design. The majority of laboratory testing (cognitive
interviews, focus groups) will be analyzed qualitatively. The survey designers and
methodologists serve as interviewers and use detailed notes and transcriptions from the in-depth
cognitive interviews to conduct analyses. Final reports will be written that document how the
question performed in the interviews, including question problems as well as the phenomena
captured by the survey question. Reports are used to provide necessary information to guide
designs for redesigning a question prior to fielding as well as to assist end users when analyzing
the survey data. For field test activities, qualitative and quantitative analysis will be performed
on samples of observational data from household interviews in order to determine where
additional problems occur. Because CMS is using state-of-the-science questionnaire
development techniques, methodological papers will be written which may include descriptions
of response problems, recall strategies used, and quantitative analysis of frequency counts of
several classes of problems that are uncovered through the cognitive interview and observation
techniques.
A17.

Reason(s) Display of OMB Expiration Date is Inappropriate

No exemption is requested.
A18.

Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to this certification statement.

12