CMS-10549_Supporting_Statement_Part_A 04.29.20_clean

This generic clearance request is for MCBS Questionnaire Testing and Methodological Research. This request encompasses development and testing of MCBS questionnaires, instrumentation, as well as methodological experiments. CMS will submit individual collection requests under this generic clearance, and will provide OMB with a memo explaining the specific purpose and procedures for each collection, as well as copies of all questionnaires, protocols, consent forms, and debriefing materials in advance of any testing activity. The Medicare Current Beneficiary Survey (MCBS) (OMB No. 0938-0568) is an in-person, nationally-representative, longitudinal survey of Medicare beneficiaries that is sponsored by the Centers for Medicare & Medicaid (CMS) and directed by the Office of Enterprise Data and Analytics (OEDA) in partnership with the Center for Medicare and Medicaid Innovation (CMMI). The MCBS operates under legal authority through the Medicare Prescription Drug, Improvement and Modernization Act. The MCBS is the most comprehensive and complete survey available on the Medicare population and is essential in capturing data not otherwise collected through CMS operations. The activities under the MCBS are currently conducted by contract with NORC at the University of Chicago.

The MCBS captures beneficiary information, whether aged or disabled, living in the community or facility, or serviced by managed care or fee-for-service. Data produced as part of the MCBS are enhanced with CMS administrative data to provide users with more accurate and complete estimates of total health care costs and utilization. The MCBS has been continuously fielded for more than 25 years and consists of three annual interviews per survey participant. The MCBS has been at the forefront of in-person survey collection and data processing, most notably as one of the first surveys to successfully 1) implement a computer assisted personal interview (CAPI) and 2) match survey and claims data to adjust and correct for underreporting in survey reported health care utilization. The CMS vision for the MCBS is to continue to provide unique, high-quality and high-value data in a timely manner, continue to break ground in innovative, efficient and analytically powerful new areas of survey administration, design and development, and to increase the survey’s ability to develop, monitor, assess and evaluate the impact of CMMI care delivery and payment models. To succeed in these areas, CMS requests a non-substantive change of this generic clearance for research and testing activities to accomplish the following goals:

Improve data quality and accuracy by evaluating and revising existing questionnaire items;
Address emerging policy and program issues by testing new questionnaire items;
Reduce respondent burden by improving questionnaire items, response categories, and questionnaire flow;
Reduce survey costs and implement efficiencies by improving questionnaire items and interview flow, as well as considering new methods and modes of data collection;
Increase response rates by improving respondent materials and improving questionnaire content and flow to reduce survey length.

This clearance contains approval for seven types of potential research activities: 1) cognitive interviewing, 2) focus groups, 3) usability testing, 4) field testing, 5) respondent debriefing questionnaire, 6) split ballot and other methodological experiments, and 7) research about incentives. CMS will submit individual collections under this generic clearance to OMB, and will provide OMB with a copy of questionnaires, protocols, and debriefing materials in advance of any testing activity.

NORC at the University of Chicago, under contract with CMS to administer the MCBS, will conduct the research testing. NORC employs methodological specialists, who will collaborate with CMS to examine questionnaire components from MCBS and compare those existing questions and methods with those “state of the science” in other federal agencies, or other academic or professional institutions. Specific topics to be addressed will be outlined in individual collection requests under the generic clearance. All data collection and analysis will be performed in compliance with OMB, Privacy Act, and Protection of Human Subjects requirements.

The general methods proposed for each type of research activity under this clearance are described below.

Cognitive Interviewing. Cognitive pretesting is an important innovation in the development and testing of survey questionnaires that has emerged over the last 20 years. Its chief strength is in providing a structured methodology for ascertaining whether the respondent has understood the questions in the way CMS and researchers intend them to be understood, and to assess the ability of respondents to provide meaningful and accurate information. Cognitive interviewing is done through the administration of questions by a specially trained and experienced cognitive interviewer, followed by probes to ascertain comprehension, memory, judgment processes, and topic sensitivity. A secondary purpose is to make sure that issues pertinent to the research are covered adequately. The cognitive interviewing process often includes techniques, such as observation and coding of respondent behaviors (e.g., responses of “don’t know” and requests for question clarification), and in-depth debriefings with respondents, survey methodologists and interviewers to fully understand the functioning of a survey questionnaire.

Cognitive interviewing offers a detailed depiction of question interpretations and processes used by respondents to answer questions—processes that ultimately produce the survey data. Cognitive interviewing is useful not only for investigating individual questionnaire items, but also for understanding how contextual factors, such as instructions to the respondent or question order can influence response and contribute to measurement error. As such, the method offers an insight that can transform understanding of question validity and response error.

Respondents are typically not selected through a random process, but rather are selected for specific characteristics such as age, health status or some other attribute that is relevant to the type of questions being tested. Because the goal is to identify the presence of problems, as opposed to making estimations or causal statements, a randomly drawn sample is not required.

The interview structure consists of respondents first answering a draft survey question and then providing explanations to reveal the processes involved in answering the test question. Specifically, cognitive interview respondents are asked to describe how and why they answered the question as they did. Through the interviewing process, various types of question-response problems that would not normally be identified in a traditional survey interview, such as interpretive errors and recall accuracy, are uncovered.

Data collection procedures for cognitive interviewing are different from survey interviewing. While survey interviewers strictly adhere to scripted questionnaires, cognitive interviewers use survey questions as starting points to begin a more detailed discussion of questions themselves: how respondents interpret key concepts, their ability to recall the requested information and to formulate an answer, and the appropriateness of response categories. Because the interviews generate narrative responses rather than statistics, results are analyzed using qualitative methods. This type of in-depth analysis reveals problems in particular survey questions and, as a result, can help to improve the overall quality of the MCBS. Results of cognitive interviews will be used to make questionnaire design decisions that minimize survey response error; to enhance our understanding of the question response process; to develop better standards for questionnaire design; and to improve data collection procedures. Because of the programming costs involved, cognitive testing is always administered by a trained questionnaire methodologist using a paper form that simulates asking the questionnaire items via CAPI.

Cognitive interviewing methodology identifies problems that are missed by traditional field tests. Field interviewers may not be sufficiently trained to identify questionnaire problems, and such tests are often conducted too late to allow for substantial revisions to be made. Nevertheless, field tests are a vital complement to cognitive interviews because they can provide important information about how new or revised questions perform in a production environment.

Focus Groups. Focus groups are used to obtain insights into target respondent perceptions, attitudes, and experience during questionnaire and materials development and testing. Focus groups are usually composed of 8 - 10 people who have characteristics similar to the target survey population, or subgroups of the target population. The groups are conducted by a professional moderator who keeps the session on track while allowing respondents to talk openly and spontaneously. The moderator uses a loosely structured discussion outline, which allows him/her to change direction as the discussion unfolds and new topics emerge. The interactive nature of a focus group often encourages a richer discussion than would have been possible in individual interviews.

Usability Testing. Research on computer-user interface designs for computer-assisted instruments is often referred to as “usability testing.” This research examines how survey questions, instructions, and supplemental information are presented on computer instruments (e.g., CAPI or Computer Assisted Self-Interviewing (CASI) instruments), Audio Computer-Assisted Self-Interview (ACASI), and investigates how the presentation affects the ability of users to effectively utilize these instruments. Authors of computer-assisted instruments make numerous design decisions: how to position the survey question on a computer screen; how to display interviewer instructions that are not to be read to respondents; the maximum amount of information that can be effectively presented on one screen; how supplemental information such as “help screens” should be accessed; whether to use different colors for different types of information presented on the screen; and so on. Research has shown that these decisions can have a significant effect on the time required to administer survey questions, the accuracy of question-reading, the accuracy of data entry, and the full exploitation of resources available to help the user complete his or her task.

Usability testing has many obvious similarities to questionnaire-based cognitive research, since it focuses on the ability of individuals to understand and process information in order to accurately complete survey data collection. It is also somewhat different, in that the typical user can be an interviewer (in the case of CAPI instruments) as well as a respondent (in the case of CASI/ACASI instruments). It also focuses more heavily on matters of formatting and presentation of information than traditional cognitive testing. In addition, usability testing can be informative to the development of web-based surveys. While MCBS does not currently include a web-based design, future investigations could focus on testing web-based responses, especially for facility data collection.

Field tests. The main objective of MCBS field tests is to evaluate changes to the questionnaire, administration mode and/or data collection procedures in order to inform the overall survey. To maximize efficiency and reduce costs, field tests conducted under this clearance can either occur as a stand-alone supplement or can occur within the production environment. Usually, field tests will be utilized after cognitive interviews or additional survey research methods are completed that tested new or revised questions, revisions to questionnaire flow, or data collection methods. Generally, field tests will be conducted in the respondent's household. Professional MCBS field interviewers will be trained to administer the field test. A subset of these interviews may be observed by a survey professional from CMS and/or NORC. In cases involving observation, as the interviewer conducts the interview, the observer compiles notes regarding respondent misunderstandings or difficulty answering, or questions that interviewers have difficulty administering, or difficulties with new data collection methodologies, which help to identify potential question revisions. Field tests could also be used to test changes in mode of survey administration. In addition, CMS staff may conduct analysis of outcome data such as response rates and response distributions to key items, para-data (e.g., response times), interviewer observations, and respondent debriefing data. Subject matter staff are debriefed on these findings and if changes are required, the results of the field test will be used to modify the questionnaire or data collection procedures for follow-up field tests prior to recommending changes to the production instrument.

Respondent debriefing questionnaire. In this method, standardized debriefing questionnaires are administered to respondents who have participated in a field test. The debriefing form is administered at the end of the interview, and contains questions that probe to determine how respondents interpret the questions, whether they have problems in completing the survey/questionnaire, or whether they have questions or concerns about new procedures being tested. This structured approach to debriefing enables quantitative analysis of data from a representative sample of respondents, to learn whether respondents can answer the questions, and whether they interpret them in the manner intended by the questionnaire designers. The debriefing questionnaire would be administered by professionally trained MCBS field interviewers.

Split ballot and other methodological experiments. This methodological experiment involves testing alternative versions of questionnaires, at least some of which have been designed to address problems identified in questionnaires from previous survey rounds. The use of multiple questionnaires, randomly assigned to permit statistical comparisons, is the critical component. We anticipate this type of data collection to occur mainly in-person as part of a field test.

In any split ballot experiments conducted under this clearance, alternative questionnaire versions will be tested. The number of versions tested and the number of cases per version will depend on the objectives of the test. We cannot specify with certainty a minimum panel size, although we would expect that no questionnaire versions would be administered to less than fifty persons in a split ballot experiment.

Split ballot experiments that incorporate methodological questionnaire design experiments will have a larger maximum sample size (up to several hundred cases per panel) than field tests using other pretest methods. This will enable the detection of statistically significant differences, and facilitate methodological experiments that can extend questionnaire design knowledge more generally. CMS will consult with OMB prior to submission regarding split ballot tests with sample sizes over 1,000.

While all data collection for the MCBS is currently conducted in-person using computer-assisted personal interviewing (CAPI) mode, future methodological experiments may involve testing other modes of data collection, such as self-administered paper questionnaire (SAQ), self-administered web survey, or computer-assisted telephone interviewing (CATI). The generic clearance request would enable CMS to test various modes of questionnaire administration to enhance data quality and reduce respondent burden.

Research about incentives. In the original design of the MCBS, $3.00 was provided to each community survey participant at each interview. In the early 1990’s CAPI laptop battery life technology could be questionable, especially if an interviewer was conducting multiple interviews in the course of a day. Therefore, interviewers were instructed to plug in their laptops, if they could, while conducting the in-person interview. Keeping in mind that many of the MCBS survey participants live on limited incomes and being mindful of any potentially added costs, our interviewers offered $3.00 to cover any cost associated with the electrical usage during the interview. This approach was cleared in the original OMB clearance and all subsequent applicable clearances.

In 2008 the MCBS was faced with a challenging budget year. As a result, CMS in consultation with the existing MCBS contractor at the time, determined that the $3.00 electrical usage compensation was no longer a necessity. Laptops were common place in the community and there wasn’t the apprehension associated with plugging them into a respondent’s outlet that there once was. In actuality, the $3.00 compensation was now seen as a very small form of appreciation by most of the survey participants. Starting in 2009 the compensation was phased out over the course of four years for continuing survey participants. We began eliminating the compensation for all new panels entering the survey.

Independent of the prior use of the $3.00 compensation, CMS, similar to other national surveys, has seen a small but steady drop in response rates over time. Response rates for the incoming panel has gone from 84 percent in 2001 to 55 percent in 2016. These respondents come into the survey each Fall and their cooperation rates have a long lasting impact to the quality of the data over the four year period of participation. Therefore, incentives to improve the response rates of the incoming panel would be targeted to gaining initial cooperation. Of similar interest would be incentive experiments targeted to reduce attrition over the life of enrolled respondents.

CMS may, in the future, request approval to evaluate what impact incentives could have on the MCBS response rate. This evaluation would at first consist of conducting an environmental scan of the state of the science on respondent incentives in longitudinal surveys and other Federal surveys. From these findings CMS would consult with OMB about the various kinds of experiments that would both inform the statistical community at large as well as provide information about improving the quality of MCBS data and potentially reducing survey costs.

A2. Purpose and Use of Information Collection

The information collected will be used by CMS staff to evaluate and improve the quality of the data in the MCBS survey. The MCBS has remained virtually unchanged in methodology and content since it was first fielded in 1991, while the state of the science has adapted to the ever changing Medicare health care related survey environment. To address a need for modernization, the MCBS through its contactor will conduct cognitive interviews, focus groups, usability testing, field tests, split ballot and other methodological experiments, respondent debriefings, and research on incentives.

The qualitative and quantitative data collected under this testing research program will aid CMS in its overarching goals for administering the MCBS: improving data quality; addressing emerging issues; reducing respondent burden; reducing survey costs and implementing efficiencies; and increasing response rates.

A3. Use of Information Technology and Burden Reduction

Appropriate technology will be used during testing to keep respondent burden at a minimum. All cognitive testing will be facilitated by an interviewer, however automated data collection methods such as Computer Assisted Personal Interviewing (CAPI) and Audio Computer Assisted Self Interview (ACASI), as well as web-based interviews may be used to reduce respondent burden. Field testing will employ the usual CAPI data collection method used on the MCBS but could also include tests of other modes of survey administration. These automated data collection methods reduce the burden on respondents, as both eliminate the need for respondents to read items and write responses.

A4. Efforts to Identify Duplication and Use of Similar Information

This testing and methodological research program does not duplicate any other questionnaire design work being done by CMS or other Federal agencies. No information to be obtained from the proposed testing currently exists. The research may involve collaboration with staff from other agencies. All efforts will be collaborative and no duplication in this area is anticipated.

A5. Impact on Small Businesses and Other Small Entities

There will be no impact on small businesses or other small entities. We do not anticipate that any small business entities or other small organizations will be involved in this research program.

A6. Consequences of Collecting the Information Less Frequently

This clearance involves one-time data collection for each testing activity. If the research program is not conducted, the quality of the data collected in the MCBS would suffer.

A7. Special Circumstances Relating to Guidelines of 5 CFR 1320.5

None of the special circumstances listed by OMB apply to this MCBS research program.

A8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agencies

The 60-day Federal Register notice published on October 27, 2017 (82 FR 49816). No comments were received. The 30-day Federal Register notice was published on January 23, 2018.

A9. Explanation of Any Payment or Gift to Respondents

Respondents for testing activities conducted in the laboratory (i.e., cognitive interviews, usability testing and focus groups) under this clearance will receive a small incentive. This practice has proven necessary and effective in recruiting subjects to participate in this type of small-scale research, and is also employed by other Federal cognitive laboratories. The standard incentive for participation in a cognitive interview is $40 for adults, and for participation in a focus group it is $50 - $75 unless approval is granted by OMB on a case-by-case basis to pay a higher incentive. Respondents for methods that are generally administered as part of field test activities will not receive payment unless there are extenuating circumstances that warrant it.

A10. Assurances of Confidentiality Provided to Respondents

All respondents who participate in research under this clearance will be informed that the information they provide is confidential and that their participation is voluntary. All participants in cognitive research will be required to sign written notification concerning the voluntary and confidential nature of their participation.

For field testing activities, the current MCBS cover letter contains a reference to the Privacy Act of 1974, as amended. A handout sheet provided to the household respondent at the door and the nursing home administrator and proxy respondents contains a statement of privacy consistent with the Privacy Act of 1974 and the Paperwork Reduction Act of 1995. In addition, on October 16, 2006, CMS published in the Federal Register a notice of a modified or altered System of Record (SOR) that assigned a new identifying number to the SOR (System No. 09-70-0519).

Interviewer training stresses the importance of maintaining privacy. The household interviewer's manual specifically addresses this and it is part of the training for the interviewers (both household and nursing home). Procedures have been established to maintain and ensure privacy. These include computer security procedures (laptop password encryption).

Any data published will exclude information that might lead to the identification of specific individuals (e.g., ID number, claim numbers, and location codes). CMS will take precautionary measures to minimize the risks of unauthorized access to the records and the potential harm to the individual privacy or other personal or property rights of the individual.

All MCBS survey staff directly involved in MCBS data collection and/or analysis activities are required to sign confidentiality agreements. Furthermore, all MCBS patient-level data are protected from public disclosure in accordance with the Privacy Act of 1974, as amended.

A11. Justification for Sensitive Questions

None of the questions that are included in the current MCBS questionnaires are of a sensitive nature. However, it is possible that some potentially sensitive questions may be included in questionnaire items that are tested under this clearance. One of the purposes of the testing is to identify such questions, determine sources of sensitivity, and alleviate them insofar as possible before they are incorporated into the main MCBS questionnaires.

A12. Estimates of Annualized Burden Hours and Costs

Table 1 is based on the maximum number of data collections expected on an annual basis. The total estimated respondent burden and costs are calculated below. Please note that for Field Tests and Split Ballot or other Experiments, our plan is to conduct these efforts in Production with existing respondents. Therefore, the burden for their time is captured in the MCBS clearance, 0938-0568.

Table 1. Estimated Annual Reporting Burden, by Anticipated Data Collection Methods
	Number of Respondents	Frequency of Response	Hours Per Response	Total Hours
Cognitive Interviews	100	1	2.00	200
Focus Group Interviews	100	1	2.00	200
Usability testing sessions	50	1	1.50	75
Respondent Debriefing Questionnaires	500	1	10/60	83
Field Tests (Standalone Supplement)	10,949	1	0.25	2,737
TOTAL	11,699			3,295

The estimated annualized costs to respondents is based on the Bureau of Labor Statistics (BLS)

data from May 2019, http://www.bls.gov/oes/current/oes_nat.htm. The mean hourly wage for all occupations is $25.72.

The estimated annualized annual costs are outlined in Table 2.

Table 2. Estimated Annual Costs
	Wages	Total Hours	Total Costs
Cognitive Interviews	$25.72	200	$5,144.00
Focus Group Interviews	$25.72	200	$5,144.00
Usability testing sessions	$25.72	75	$1,929.00
Respondent Debriefing Questionnaires	$25.72	83	$2,134.76
Field Tests (Supplemental Standalone)	$25.72	2,737	$70,395.64
TOTAL		3,295	$84,747.40

A13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

None.

A14. Annualized Costs to the Federal Government

At this time, we cannot anticipate the actual number of participants, length of interview, and/or mode of data collection for the surveys to be conducted under this clearance. Thus, it is impossible to estimate in advance the cost to the Federal Government. Costs will be covered by CMS under the existing MCBS budget.

A15. Explanation for Program Changes or Adjustments

Previously, the estimated burden for this generic clearance was 558 hours to reflect the maximum burden contemplated annually over the three years as stated in section A12 of this document. An increase in 2,737 hours is being requested to accommodate standalone supplemental field testing conducted outside of the production environment (MCBS production is fielded under 0938-0568). This request would bring the total estimated burden for this generic clearance to 3,295 hours.

A16. Plans for Tabulation and Publication and Project Time Schedule

This clearance request is for questionnaire development activities and for developmental work that will guide future questionnaire design. The majority of laboratory testing (cognitive interviews, focus groups) will be analyzed qualitatively. The survey designers and methodologists serve as interviewers and use detailed notes and transcriptions from the in-depth cognitive interviews to conduct analyses. Final reports will be written that document how the question performed in the interviews, including question problems as well as the phenomena captured by the survey question. Reports are used to provide necessary information to guide designs for redesigning a question prior to fielding as well as to assist end users when analyzing the survey data. For field test activities, qualitative and quantitative analysis will be performed on samples of observational data from household interviews in order to determine where additional problems occur. Because CMS is using state-of-the-science questionnaire development techniques, methodological papers will be written which may include descriptions of response problems, recall strategies used, and quantitative analysis of frequency counts of several classes of problems that are uncovered through the cognitive interview and observation techniques.

A17. Reason(s) Display of OMB Expiration Date is Inappropriate

No exemption is requested.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to this certification statement.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Supporting Statement for Request for Clearance:
Author	Karen Whitaker
File Modified	0000-00-00
File Created	2021-01-11

CMS-10549_Supporting_Statement_Part_A 04.29.20_clean