CMS-10549_MCBS Generic Clearance_ Supporting Statement B

Supporting Statement B
for Request for Generic Clearance:
Questionnaire Testing and Methodological Research for the
Medicare Current Beneficiary Survey (MCBS)

Contact Information:
William S. Long
Contracting Officer’s Representative, Medicare Current Beneficiary Survey
Office of Information Products & Data Analytics
Office of Enterprise Management/CMS
7500 Security Boulevard, Mail Stop B2-29-04
Baltimore, MD 21244
(410) 786-7927
[email protected]
(410) 786-5515 (fax)

January 22, 2015

Table of Contents
B. STATISTICAL METHODS ..................................................................................................... 3
B1. Universe and Respondent Selection ..................................................................................... 3
B2. Procedures for Collecting Information ................................................................................ 3
B3. Methods to Maximize Response ......................................................................................... 6
B4. Testing of Procedures .......................................................................................................... 7
B5. Individuals Consulted on Statistical Aspects and Individuals Collecting/Analyzing Data 7

B. STATISTICAL METHODS
B1. Universe and Respondent Selection
The data collected under this clearance request will be used for survey research and testing
activities rather than to produce estimates about populations. Consequently, many of the smallscale methods proposed (cognitive testing, focus groups, usability testing) will involve
respondents selected either to cover a broad range of demographic subgroups of interest to the
MCBS, or for specific characteristics related to the specific content being tested. For other
methods, including field tests, split ballot experiments or other methodological experiments,
statistically representative subsamples of existing MCBS panel respondents will be selected for
inclusion. A description of the plans for selecting respondents for each individual test will be
provided to OMB at the time the testing plans are submitted.
While survey research employs a quantitative methodology and relies on a relatively large
population-based probability sample to support statistical inference and representativeness,
cognitive testing, usability testing and focus groups conducted in a laboratory usually employ a
qualitative methodology and generally rely on relatively small samples. Unlike survey research,
the primary objective of such testing is not to produce statistical data that can be generalized to
an entire population. Rather, the objective is to provide an in-depth exploration of particular
concepts, processes and/or patterns of interpretation. Cognitive interviewing samples generally
do not achieve full inclusivity of all social and demographic groups. However, as a general rule,
respondents are selected for inclusion based on similarity of characteristics to MCBS
respondents.
For field tests, split ballot experiments and other methodological experiments, the purpose, scope
and size of the testing require the use of subsamples of MCBS respondents currently enrolled in
the survey.
B2. Procedures for Collecting Information
Most of the data collection anticipated under this clearance request will be conducted through inperson interviews, whether through field tests, cognitive interviews, focus groups, or usability
testing. CMS will submit individual collections under this generic clearance to OMB, and will
provide OMB with a copy of recruitment materials, questionnaires, consent forms, protocols, and
debriefing materials in advance of any testing activity. The general recruitment, screening and
interviewing procedures are described below.
Recruitment
Laboratory respondents for cognitive testing, focus groups and usability testing will usually be
recruited by means of flyers and other advertisements posted in public places, newspaper
advertisements, or word-of-mouth. Prior experience at federal and private cognitive laboratories
has shown that advertisements in local newspapers and flyers attract a large pool of potential
respondents. These recruitment mechanisms have been productive in the past for obtaining a
diverse group of respondents to help us determine potential sources of error in survey questions.
To test questions that are targeted toward specific subgroups (i.e. by age, race/ethnicity, or other
factors), the advertisement or flyer may be developed to identify appropriate respondents. Direct

contact to solicit support from senior centers, employers, and/or social or service organizations is
occasionally used as a way to recruit potential participants. In these cases, a flyer is provided to
a contact person who then either posts the flyer or distributes it to members of the organization.
For field tests and methodological experiments, cases to be included will be pre-identified in the
MCBS sample, based on the desired sampling mechanism for each test. MCBS interviewers will
use the same procedures as used in the actual MCBS interview to explain the survey and acquire
informed consent.
Screening and Scheduling Procedures
The first contact with potential laboratory research respondents occurs in response to flyers or
advertisements. Interested persons leave contact information (name and telephone number) on
voicemail. The MCBS staff person then calls the person back, gives a brief description of the
nature of the study, i.e., one-on-one interview or focus group, where the interview/focus group
takes place, video/audio taping procedures, and the remuneration to be offered. The MCBS staff
person also determines through a brief series of questions whether the volunteer possesses the
desired research characteristics (e.g., age, gender, and other demographic characteristics). If the
person does meet the selection criteria and agrees to participate in the cognitive interview or
focus group, he/she is scheduled for an interview or focus group session. Telephone numbers
and minimal demographic information (e.g. name, age, gender, etc.) are obtained for all
scheduled participants. For those callers who are ineligible for the study because they are not
members of a subgroup of interest, no identifying information will be retained. Recruitment data
will be maintained in separate, encrypted files with password protection. Any hardcopy materials
will be stored in locked files and access limited to authorized study personnel.
For field tests and methodological experiments, cases to be included will be pre-identified in the
MCBS sample based on the desired sampling allocation for each test, so no separate respondent
screening should be necessary. MCBS interviewers will use the same interview scheduling
procedures as are currently in place for the main MCBS data collection.
Interviewing Procedures
Cognitive Testing Interviews. Participants scheduled for in-person cognitive testing interviews
will usually travel to the contractor facilities. On rare occasions, a participant may be unable to
travel to the intended location (e.g. an individual may be housebound or have limited mobility).
In such cases, the interview may be conducted in his/her home or at a location normally
frequented by the participant, such as a senior center. To reduce the number of "no shows" for
cognitive interviews, participants scheduled more than a week in advance receive a reminder
telephone call by the MCBS staff the day prior to the scheduled interview.
When the respondent arrives for their scheduled interview, he/she is greeted by the MCBS staff
person. The participant will then be brought to the interview room and asked to read (or have
read aloud to) a consent form. The form contains a brief description of the study, assurances of
confidentiality for the participant, explains the voluntary nature of the study, and describes the
risks and benefits. The need for recording the interview (audio or video) is explained and the
respondent is asked to sign the consent form. In the rare instance that the participant consents to
the cognitive interview, but not to recording it, the session will be carried out but not recorded.
If the respondent grants consent to record the interview but changes his/her mind while the
session is being recorded, the interviewer will ask for verbal consent to retain the interviewing

materials and the portion already taped.
The interviewer, usually a survey methodologist or other project staff, will begin the cognitive
interview by reading a more detailed explanation of the purpose of the interview and the
procedures to be used. Interviewing procedures vary depending on the specific testing technique
to be applied. The selection of the technique is determined by the nature of the project, or the
stage of development of the questionnaire or set of questions under study. The most commonly
used method is the cognitive interview with concurrent probing. In these interviews, respondents
are presented draft survey questions and asked to “think aloud” about how and why they
answered as they did. The interviewer usually probes extensively to ascertain the degree of
comprehension and the recall processes involved.
If possible, the cognitive interview will be conducted in the mode intended for the instrument,
either face-to-face, telephone, or self-administered using a paper or web-based instrument. For a
telephone interview, the respondent is called from one room to another and in-person debriefing
follows.
As described in Supporting Statement A. section A.9, participants will receive a small stipend.
They will be asked to sign a receipt form indicating receipt of the remuneration. Immediately
following the interview, any hardcopy interviewing data (e.g. questionnaires) will be separated
from the consent form and signed receipt form, so that no demographic information will be
associated with the individual’s name. Signed consent forms will be stored separately from any
data collection from the participant. Audio and video recording files will be stored on NORC’s
secure servers and the files deleted from the recording device.
Focus Groups. Participants scheduled for focus groups will usually travel to the contractor
facilities. To reduce the number of "no shows" for focus groups sessions, participants scheduled
more than a week in advance receive a reminder telephone call by the MCBS staff the day prior
to the scheduled session. When participants arrive they are greeted by MCBS project staff and
directed to the focus group room. Participants are given a consent form to read (or to have read
to them by project staff). The consent form contains a brief description of the study, assurances
of confidentiality for the participant, explains the voluntary nature of the study, and describes the
risks and benefits. The need for recording the focus group sessions (audio and/or video) is
explained and the participant is asked to sign the consent form.
MCBS staff person(s) will moderate the focus group. Before discussion begins, the moderator
will distribute name tags and will tell respondents to pick a name to put on the name tag.
Respondents will be told that they do not have to use their real names. The moderator will then
describe the process of the focus group and ask if there are any questions. After all questions are
answered, the moderator will then begin the focus group discussion following the moderator
guide designed for that particular study.
Once the focus group has concluded, participants will receive a small stipend. They will be
asked to sign a receipt form indicating receipt of the remuneration. Immediately following the
interview, any hardcopy materials completed by the participant will be separated from the
consent form and signed receipt form, so that no demographic information will be associated
with the individual’s name. Signed consent forms will be stored separately from any data
collection from the participant. Audio and video recording files will be stored on NORC’s secure

servers and the files deleted from the recording device.
Usability Testing. Participants scheduled for usability testing will usually travel to the
contractor’s facilities. When the participant arrives for their scheduled interview, he/she is
greeted by the MCBS staff person. The participant will then be brought to the testing room and
asked to read (or have read aloud to) a consent form. The form contains a brief description of the
study, assurances of confidentiality for the participant, explains the voluntary nature of the study,
and describes the risks and benefits. The need for recording the interview (audio or video) is
explained and the respondent is asked to sign the consent form. In the rare instance that the
participant consents to the cognitive interview, but not to recording it, the session will be carried
out but not recorded. If the respondent grants consent to record the interview but changes his/her
mind while the session is being recorded, the interviewer will ask for verbal consent to retain the
interviewing materials and the portion already taped.
The MCBS project staff person will begin the testing session by reading a more detailed
explanation of the purpose of the interview and the procedures to be used. Interviewing
procedures vary depending on the specific testing technique to be applied. Usability could be
evaluated with cognitive testing (think aloud, debriefing questions), behavior coding (e.g.,
coding errors made by the interviewer or respondent while trying to navigate the instrument),
and analysis of interview completion time.
As described in Supporting Statement A. section A.9, participants will receive a small stipend.
They will be asked to sign a receipt form indicating receipt of the remuneration. Immediately
following the interview, any hardcopy interviewing data (e.g. questionnaires) will be separated
from the consent form and signed receipt form, so that no demographic information will be
associated with the individual’s name. Signed consent forms will be stored separately from any
data collection from the participant. Audio and video recording files will be stored on NORC’s
secure servers and the files deleted from the recording device.
Field Tests. Field tests will be conducted with subsamples of the main MCBS cohorts, as part of
regular survey data collection cycle. The test questions will be incorporated into the
questionnaires and administration will follow MCBS protocols.
Field testing will be conducted by MCBS field interviewers. Cases to be included will be preidentified in the MCBS sample, based on the desired sampling allocation. For all interviews
administered in this manner, the field interviewer will follow approved MCBS survey
administration procedures. As time and resources allow, a subset of the interviews may be
observed by CMS staff or MCBS project staff and observations manually recorded to allow for
systematic analysis. In addition, CMS or project staff may conduct analysis of outcome data
such as response rates and response distributions to key items, paradata, interviewer
observations, and respondent debriefing data. Subject matter staff are debriefed, and findings are
used to modify the questionnaire for follow-up field tests or incorporation into the main MCBS
questionnaires.
B3. Methods to Maximize Response
For the purpose of conducting cognitive interviews and focus groups, CMS will rely on our
contractor to advertise in local newspapers and flyers as well as reaching out to former survey

participants that have been retired from the MCBS, to attract a large enough pool of potential
laboratory research respondents. These recruitment mechanisms have been productive in the
past for obtaining a diverse group of respondents to help us determine potential sources of error
in survey questions. For those questionnaires that target specific subgroups, special recruitment
procedures will be developed to identify respondents. After laboratory volunteers have been
recruited, the probability of the respondent failing to show is minimized by making reminder
phone calls to volunteers.
As separate evaluation is proposed to examine the current methods to maximize response rates
within the full MCBS collection. Once identified, proposed strategies will be cleared by OMB
before implementation. In general, callbacks will be used to maximize response rates in field
tests and methodological experiments. For cognitive interviews and focus groups, participants
will receive a stipend for their participation. Reminder phone calls and/or letters to participants
will be used to encourage them to keep their appointments. Tallies will be kept of the number of
non-respondents to all testing activities. More specific information will be contained in the
description provided to OMB at the time the individual testing plans are submitted.
B4. Testing of Procedures
This submission is a request for authorization to conduct tests of questionnaires, instrumentation
and methodologies typical in survey research programs and to build on this research through
field tests and methodological experiments. We expect that the testing and experiments
conducted under this clearance will result in improved data quality and lower respondent burden.
B5. Individuals Consulted on Statistical Aspects and Individuals Collecting/Analyzing
Data
Kirk Wolter, Ph.D.
Executive Vice President
NORC at the University of Chicago
55 East Monroe Street
30th Floor
Chicago, Illinois
(312) 759-4206
[email protected]