CMS-10632.Supporting Statement Part A

Evaluating Coverage to Care

Supporting Statement Part A for OMB Approval

(CMS-10632; 0938-New)

A. Justification

The Office of Minority Health (OMH) of the Center for Medicare & Medicaid Services (CMS) of the Department of Health and Human Services (DHHS) is requesting Office of Management and Budget (OMB) clearance for the Evaluation of Coverage to Care (C2C) study.

A.1. Circumstances Making the Collection of Information Necessary

A.1.1. Overview of Request

CMS OMH has contracted with the RAND Corporation to evaluate From Coverage to Care (C2C). This section provides an overview of the study and discusses its objectives and the need for the proposed information collection.

From the beginning of the Affordable Care Act’s implementation, the Centers for Medicare & Medicaid Services, Office of Minority Health (CMS OMH) recognized that achieving better health and reduced health care costs would require individuals to take an active role in their health care and regularly use primary and preventive care services. To address this need, CMS OMH launched From Coverage to Care (C2C) in June 2014. C2C was designed to help consumers understand what it means to have health insurance coverage, how to find a provider, when and where to seek appropriate health services, and why prevention and partnering with a provider is important for achieving optimal health. It was also designed to equip health care providers and stakeholders in the community who support consumers’ connection to care with the tools needed to promote consumer engagement and to promote changes in the health care system that improve access to care. As part of C2C, CMS produced a range of consumer-oriented materials, both web-based and in print. The most in-depth of the print materials is an eight-step booklet titled “A Roadmap to Better Care and a Healthier You.” Based on the need for the information to be communicated in smaller, more digestible packets, booklets were developed to correspond to each of the eight steps. Four of the most popular pages of the Roadmap have been made available as single-page handouts for easier distribution. These materials are currently available in eight languages, including English, Spanish, Arabic, Chinese, Haitian Creole, Korean, Russian, and Vietnamese.

Since the national launch in 2014, CMS has disseminated C2C through speaking engagements, webinars, and meetings sponsored by CMS regional offices. CMS fills product orders and recently completed a redesign of the C2C website. C2C has grown to address emerging needs of consumers, as well as stakeholders or organizations that work with and support consumers, across the full continuum of health insurance and care: plan selection, enrollment, finding a provider, and engaging in care over time.

RAND spent the past year designing and preparing for this evaluation to assess C2C’s impact on consumer health insurance literacy and care utilization. This evaluation will also help CMS understand how C2C is spread within a community and disseminated to consumers, and in turn how best to maximize C2C’s impact. The next three years will be dedicated to implementing the evaluation described in this submission. We are proposing four data collection activities: 1) A cross-sectional survey of organizations that have ordered and used the materials with consumers; 2) A cross-sectional survey of consumers, drawn from the Knowledge Networks panel, to measure the association between C2C and consumer knowledge and behavior; 3) semi-structured interviews with staff from a limited set of community organizations as part of a case study; and (4) focus groups of consumers as part of a case study. The case study will be conducted in a community where English is not the preferred language, and where C2C materials in another language (e.g., Spanish, Arabic, Chinese, Haitian Creole, Korean, Russian, and Vietnamese) were used with consumers.

A.1.2. Study Context and Rationale

The proposed evaluation is intended to achieve two separate but related goals. First, it will provide information that will support CMS OMH to maximize the impact of C2C at the local, state, and national levels. The evaluation will support selecting the best dissemination activities to maximize the spread of C2C, developing new messaging on preventive visits and primary care, and creating additional products to fill critical gaps. Second, findings from the evaluation will help CMS to assess whether and how C2C improves consumer health literacy and care utilization.

Although C2C includes a range of print and on-line materials and resources, it is not considered to be a “program” or “initiative” with guidance on what and how the materials should be used or implemented within the community. As a result, organizations have significant flexibility to use, and even customize the materials, and to implement them in a way that meets their needs. Conversations that occurred with six organizations during the planning year for this evaluation found that organizations range from passive distribution of materials (e.g., leaving them on a resource table) to very active distribution (e.g., walking consumers through specific pages or sections of the Roadmap). With this flexibility, however, comes the challenge of understanding how C2C is being spread within a community and the impact that C2C is having on organizations and consumers. As discussed in more detail in the sections that follow, this study seeks to gain that understanding by collecting much needed data to inform future directions of C2C to most effectively and efficiently maximize its impact.

This study is guided by a conceptual framework (see Exhibit A.1.1.) that outlines the diffusion of C2C, based on Rogers’s Diffusion of Innovation theory.¹ In Rogers’s theory, the innovation-decision process involves five steps: knowledge, persuasion, decision, implementation, and confirmation. In the C2C model, CMS OMH outreach and dissemination efforts are intended to spread knowledge of C2C and present a compelling case for its value to consumers and organizations that work with and support consumers. These efforts include a wide range of dissemination and outreach activities at the national, state, and local levels, including webinars, blogs, community events, and presentations. Organizations within the community then decide to use C2C, place product orders, disseminate materials to consumers, and diffuse C2C more broadly within the community by sharing C2C messages and materials with other community stakeholders.

C2C materials and messages are intended to produce changes in consumers’ health insurance literacy and knowledge, beliefs, and intentions around health care seeking, which then may translate into changes in consumer behaviors through participation in primary and preventive care and reductions in unnecessary care. The shading in Exhibit A.1.1. indicates the temporality of the four stages of C2C diffusion and the possible strength of anticipated impacts from one stage to the next.

The information collected for the study will support a richer understanding of the steps required for impact at the consumer level, including dissemination from CMS to organizations, and from organizations to consumers. It will also help uncover the ways in which organizations and consumers are meeting with success and continued gaps and challenges in supporting consumers’ connections to care. Such insights, coupled with detailed information on the types of organizations that have placed product orders and insights into the effectiveness of C2C materials in a language other than English, will provide a wealth of information for the CMS Office of Minority Health. The resulting data will inform decisions about programmatic improvement, including how C2C is disseminated and the need for new or refined materials or messages, and identify information gaps to be addressed in future data collection efforts.

Exhibit A.1.1 Conceptual Framework of C2C Impact

A.2. Purpose and Use of the Information Collection

As noted above, the purpose of this evaluation is to assess whether and how C2C improves consumer health literacy and care utilization and to provide information that will support CMS OMH to maximize the impact of C2C at the local, state, and national levels. The evaluation will support selecting the best dissemination activities to maximize the spread of C2C, developing new messaging on preventive visits and primary care, and creating additional products to fill critical gaps. .

To design an evaluation that best aligns with CMS needs and interests, the RAND team worked with CMS to identify priority research questions for the evaluation. Four priority research questions were identified, and subquestions were developed to provide more specificity and context for each of the priority questions. These questions and subquestions, shown below and in Exhibit A.2.2, capture information on high-priority objectives that will support decisions about the direction of C2C.

1. How successful was the spread and uptake of C2C?

2. What did organizations do with the C2C materials/messages?

3. How did consumers use C2C materials/messages?

4. To what extent is C2C associated with changes in care utilization?

The four research questions map to the conceptual model above (see Exhibit A.1.1). Each research question serves a unique purpose in helping CMS better understand C2C impact and potential next steps, and as a result has its own set of outcomes. Examples of these outcomes can be found in the rounded boxes on the right hand side of Exhibit A.1.1.

These study objectives will be accomplished through the four data collection components shown in Exhibit A.2.1. Exhibit A.2.2. shows how the evaluation questions map to the data collection components. These components are:

• A cross-sectional survey of organizations that placed product orders. Based on contact information available from the CMS product ordering reports, the full population, or universe, of organizations that have placed a product order will be invited to complete a short online survey to obtain basic information about ordering organizations themselves, how they learned about C2C, which materials they use and how, lateral interactions with other organizations in their communities relevant to C2C or its topics, and additional C2C resources that would be useful to them. This survey instrument is attached in Appendix A.

• A cross-sectional survey of consumers selected from the Knowledge Networks panel. We will conduct a cross-sectional web-based survey of members of a probability-weighted panel in order to understand whether and how C2C dissemination efforts thus far may have impacted consumers. This survey questionnaire, described more fully below, is attached in Appendix B.

• Semi-structured interviews with staff from a limited set of community organizations as part of case studies (non-English). C2C has been translated into seven languages (Spanish, Arabic, Mandarin, Haitian Creole, Korean, Russian, and Vietnamese) and CMS is particularly interested in learning more about how these materials are being used and the impact they are having in communities where the primary language is not English or Spanish. We will conduct up to three case studies that will include a purposive sample of 5-8 key staff of organizations in the selected community who have used translated C2C materials. The discussion guide is attached in Appendix C.

• Focus group(s) of consumers as part of a case study (non-English). As part of each case study, we will also conduct a focus group with 7-12 consumers to assess consumer perceptions of the value and impact C2C has had on their understanding of health insurance and decisions to seek care. Smaller focus groups may be held to accommodate the schedules of the participants. As part of the focus group, individuals will be asked to complete a one-page survey that will help us to characterize focus group participants. Personally identifiable information, including name, will not be collected on the form. The facilitator guide for the focus groups and this one-page survey are attached in Appendix D.

• Secondary data analysis. In addition to four data collection elements under review, the evaluation also includes analyses of existing data, including CMS product ordering reports, national surveys, and claims data. Although these data to not require OMB review and approval, we provide a brief overview of this work and how we estimate C2C saturation in small geographic areas (the exposure variable) in Appendix E. This appendix may provide helpful context for how some of the research questions will be addressed. Our estimates of C2C saturation are also important for developing the sampling frame of the online consumer survey and the selection of the case study location.

Exhibit A.2.1. Data Collection Components
Component	Study Population	Mode
Survey of organizations	Organizations that ordered C2C materials	Online survey
Survey of consumers	Consumers selected from the Knowledge Networks panel	Online survey
Case study: semi-structured interviews	Organizations in case study location who ordered C2C products in a language other than English	Semi-structured interview
Case study: focus group	Consumers residing in case study location who have been exposed to C2C materials/messages	Focus group

Exhibit A.2.2. Evaluation Questions by Data Collection Component
Research Questions	Org. Survey	Consumer Survey	Interview	Focus Group	Existing Data
1. How successful was C2C spread and uptake?	X		X		X
How many systems and organizations were reached?					X
Which areas have more/less penetration?					X
What are effective strategies for dissemination from CMS to organizations?	X		X		X
How did organizations share the information with other organizations (e.g., spread within community) and what are best practices?	X		X
2. What did organizations do with C2C materials/messages?	X		X		X
How did organizations use the materials with consumers and what are best practices?	X		X
Which C2C materials/messages did they use?	X		X		X
Which materials/messages did organizations find most helpful, and why?	X		X
What factors influenced what materials/messages organizations decided to use and how to use them?	X		X
3. How did consumers use C2C materials and messages?		X		X
Which of the materials/messages did consumers find helpful, and why?		X		X
Did C2C contribute to a change in consumer knowledge and understanding of health insurance coverage and how it works (e.g., health insurance literacy)?		X		X
Did C2C contribute to a change in consumer knowledge or understanding of how to access care as a result of C2C messages or materials?		X		X
4.  To what extent is C2C associated with changes in care utilization?		X		X	X
Did the use of preventive services change (e.g., primary care, vaccination)?		X		X	X
Did use of ED services change?		X		X	X
Did the fraction of the population with an identified PCP change?		X			X
Did use of behavioral health care change?		X		X

A.3. Use of Improved Technology and Burden Reduction

The two survey-based data collection components of this study (the online surveys of organizations and consumers) will be administered through computer-assisted data collection programs to reduce response burden on respondents. The surveys will be hosted and programmed by DatStat and Knowledge Networks. Both data collection platforms are used to help manage the data collection process including questionnaire layout (including skip patterns), sample management, fieldwork monitoring, and final dataset production. The systems allow a user to begin the survey, save responses, and go back later to complete the remaining items. DatStat and Knowledge Networks also offer an email address and toll-free hotline, which provides technical assistance for respondents. Through both systems, respondents can participate in the survey via the Internet and smart phones. In instances when the Internet is not available, Knowledge Networks can support administration by paper; in addition, Knowledge Networks also has capability for phone surveys.

Because the two surveys are not slated to launch until the following fiscal year (FY18), we are not currently working with either vendor and do not have illustrative screen shots for these surveys. However, both platforms have been in existence for years and fielded countless studies of similar size and scope. As a result, they both provide easy to use interfaces for respondents. Upon OMB request, we can submit screen shots prior to study launch.

The components of this study that involve existing data, focus groups, and stakeholder interviews are not conducive to computerized interviewing.

A.4. Efforts to Identify Duplication and Use of Similar Information

The C2C Evaluation is designed to complement, not duplicate, existing information and to provide information that does not currently exist. For example, the organizational survey does not collect information that can be obtained from the CMS OMH product ordering reports. Existing information about program size, location, and types and quantities of products ordered will be matched with the survey data to provide a more complete database.

Similarly, for the consumer survey, we will utilize demographic information collected on an annual basis from Knowledge Networks and will purchase additional data from them on characteristics relevant to our study including prior health insurance coverage (see Appendix B for more information on these variables).

In sum, these data collection efforts will provide CMS with comprehensive up-to-date information that is not available from any other source. The resulting data will help to inform future directions of C2C.

A.5. Impact on Small Business or Other Small Entities

The impact on small businesses is minimal. For the survey of organizations, only one person will be asked to complete the survey, and the survey will last no more than 20 minutes. Individuals can complete the survey at a time that is convenient for them (e.g., does not have to be completed during work hours). The study may impact small businesses, but the impact is designed to be minimal and businesses can select their level of involvement. We do anticipate conducting semi-structured interviews with a limited number of staff (not more than 3 per organization). These will be conducted in person, but at a time that is convenient for the individuals that will not impact their workflow. Organizations in the community will also be asked to help spread the word to consumers who may be interested in participating in the focus group, and may also provide meeting space for the focus groups, but this will not be required of organizations and organizations will be compensated for their time and resources used in support of this project.

A.6. Consequences of Collecting Information Less Frequently

All study components are one-time collection efforts.

A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

There are no special circumstances for the proposed data collection.

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Park 1320 (60 FR 44978, August 29, 1995), CMS published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. The 60-day Federal Register notice published on November 2, 2016 (81 FR 76538). No public comments were received.

During the planning year, the study team interviewed six organizations and three consumers about the study approach and topics for inclusion. Their feedback helped to shape the evaluation design. Once the protocols were developed, three organizations were consulted through cognitive interviews to obtain their input on the clarity of the online survey instrument for organizations and to get their feedback on any potential issues associated with completing the survey. During the 60-day comment period, the draft survey instruments and interview protocols were revised to reflect this feedback.

A.9. Explanation of Any Payment or Gift to Respondents

Incentives for web surveys have been found to reduce refusals, increase the odds ratio of starting (1.19), and completion (1.27).² While incentives may affect sample composition, they have not been found to affect response quality. Incentives have also been found to increase response rates without introducing response bias.³

For the organizational survey, we plan to give $25 to the person completing the survey as incentive for participation. Respondents to the consumer survey will receive points for each survey they complete. We will work with GfK, who run the Knowledge Networks Panel, to determine the actual point value for the survey. We anticipate the point value to be between 5,000 and 10,000 points, which roughly equates to $5 to $10 depending on the type of award selected by respondents when redeeming. Points can be redeemed for cash, merchandise, gift cards, or game entries. For the case studies, we plan to offer organizations that assist us up to $250 to cover their time and resources required to assist us with the identification and recruitment of consumers, and rental of facilities to hold the focus group. The actual dollar amount will be commensurate with the level of effort. The exact dollar amount will be based on their level of involvement. Participants who complete a semi-structured interview and consumers who participate in the focus group will also receive financial incentives. The exact dollar amount of these incentives will be decided upon once the location of the case study site is selected but will not exceed $75 per person. We will select a dollar amount that is reasonable and reflective of the cost of living in the area (e.g., to cover transportation, child care) but not too high as to be considered coercive.

A.10. Assurance of Privacy Provided to Respondents

All persons who participate in this data collection, either through the web survey or the focus group, will be assured that the personally identifiable information they provide will be kept private to the fullest extent allowed by law. Informed consent from participants will be obtained to ensure that they understand the nature of the research being conducted and their rights as survey respondents. Respondents who have questions about the consent statement or other aspects of the study will be instructed to call the RAND principal investigators or the administrator of RAND’s Institutional Review Board.

For both online surveys, the consent script will appear on an introductory screen and the survey will only continue after the respondent has given his or her consent to participate. For those participating in the focus group or semi-structured interviews as part of the case study, the consent script will be read in the appropriate language as part of the introduction to the discussion and the group facilitator will not proceed unless oral consent is given. These consent scripts are included as part of the survey instruments and focus group protocols provided in Appendices A, B, C, and D.

The study will have a Data Safeguarding Plan to further ensure the privacy of the information that is collected. For the organizational survey, RAND will assign a data ID to each respondent that will enable us to link survey responses with known information about the organization and history of product orders. Neither names of respondents nor any other kinds of personal identifiers of the respondent will appear in the survey or interview data. The consumer survey is administered by Knowledge Networks. After the survey has been fielded, Knowledge Networks will send RAND the data that has been stripped of any information that would identify the respondent, per their standard protocols. Personally identifying information is never revealed to clients (i.e. the RAND Corporation) or other external parties without explicit respondent approval and a client-signed nondisclosure agreement. RAND will not have access to any identifying information, thus participant confidentiality will be maintained. For the case study, data collection forms and notes will have an ID to indicate organization but they will not include information on the staff member(s) we meet with.

All participants will be informed as part of the consent process that their participation in this study will be kept confidential and that the information received will be aggregated prior to presenting it to CMS. If any quotations are used in reports, they will not be attributable to a specific individual or organization.

A.11. Justification for Sensitive Questions

Data collection elements that involve organizations contain no personally sensitive questions. Data collection elements involving consumers do ask about health care utilization, but are done so using standardized questions common in large-scale surveys and ask only about timing of these events (e.g., when was the last time you saw your primary care provider) and do not ask about specific health conditions or concerns. As such, the level of sensitivity for these data collection elements is low. Although the level of sensitivity is low, individuals who do not wish to report on their health care utilization (e.g., of health screens or emergency department utilization) may skip these questions.

A.12. Estimates of Annualized Burden Hours and Costs

This proposed data collection efforts do not impose a financial burden on respondents. Respondents will not incur any expenses other than the time spent answering the web survey questions and the semi-structured interview questions. The estimated annual burden for study respondents is shown in Table A.12.1.

Survey respondents will fall within two groups. The first group consists of representatives from organizations who can speak to how C2C is being used within their organization. These individuals will participate in the online survey and a small sample will participate in semi-structured interviews as part of the case study. The second group will be consumers, drawn from the Knowledge Network panel (to participate in the online survey) or the community (to participate in the case study focus group). To compute the total estimated annual cost, the total burden hours were multiplied by the average hourly wage for each participant according to wage data compiled by the Bureau of Labor Statistics. For organization representatives, we used data for 2014 median weekly salary for full-time employees over the age of 25 with a bachelor’s degree or higher ($30.60 per hour assuming an average work week of 40 hours).⁴ For consumers, we are primarily interested in collecting data from individuals who were likely exposed, or could have been exposed to C2C through community organizations or during open enrollment as they enroll in Medicaid or purchase insurance through their health insurance exchange. As such for consumers, we used the same data source to obtain the 2014 median weekly salary for full-time employees over the age of 25 with some college or associate degree ($19.27 per hour assuming an average work week of 40 hours). Some consumers will have a high school degree, or less than a high school degree, but we use the some college category as an upper estimate of total annual cost.

Exhibit A.12.1 summarizes the reporting burden on respondents to the various instruments submitted for OMB clearance. The annual number of respondents for the organization survey is estimated to be between 400 and 600, given likely response rates to an online survey. We use the upper end of this estimate in our calculation of annual burden and cost estimates. For the number of respondents to the consumer survey, we are using the number of respondents necessary to achieve adequate power (n=2,800). Response times were estimated from pre-tests with three potential respondents to the organizational survey and known completion times for existing measures included in the consumer survey. The annual burden is estimated from the total number of completed surveys and interviews and the minutes taken per instrument.

Exhibit A.12.1 Annual Burden and Cost Estimates
Data collection	Annual Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Total Burden Hours	Average Hourly Wage	Total Annual Cost
Organizational survey	600	1	0.33	198.0	$61.20	$12,117.60
Consumer survey	2,800	1	0.33	924.0	$38.54	$35,610.96
Case study: semi-structured interviews	24	1	.75	18	$30.60	$550.80
Case study: focus group	36	1	1.0	36.0	$19.27	$693.72
Total Annual Estimate				1176		$48,973.08

A.13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

The only potential additional cost may be transportation costs incurred by consumers to travel to and from the focus group. The financial incentive is intended to help offset this cost.

A.14. Annualized Cost to the Federal Government

The total cost for the three years of data collection planning, implementation, and reporting is $925,516 or $308,505 per year. These costs include all four data collection elements including protocol development for each component of the evaluation including the development of sampling frames, costs for the survey vendors to program and implement the two on-line surveys, recruitment of participants, data collection and data coding, and the data analysis and report preparation.

A.15. Explanation for Program Changes or Adjustments

This submission to OMB is a new request for approval. While no changes were made as a result of public comment during the 60-day review period, edits were made to the organizational survey at this time as a result of three cognitive interviews conducted by RAND during this period. These edits clarified and streamlined the organizational survey.

A.16. Plans for Tabulation and Publication and Project Time Schedule

A.16.1. Analysis Plan

The analysis plan for the C2C Evaluation will be tailored to address each of the study’s research questions. Specifically, the analyses will aim to assess (1) how successful was the spread and uptake of C2C; (2) what organizations did with the C2C materials/messages; (3) how consumers used C2C materials/messages; and (4) to what extent is C2C associated with changes in care utilization. In this section, the analysis plan for the information collected through the two online surveys: organization survey and consumer survey is described. Then the analysis plan for the information collected through the semi-structured interviews and focus groups as part of the case study is discussed.

Analysis of Information from Organizational and Consumer Surveys. The analysis of the information collected in the online organizational and consumer surveys will have two main components: (1) development of nonresponse weights for organizational surveys and (2) descriptive analyses.

(1) Development of nonresponse weights. As discussed in Section A.9, we will use a number of strategies to maximize organizational response rates. However, some nonresponse is expected so appropriate statistical procedures to correct for any potential nonresponse bias will be used. With some characteristics of all organizations known a priori, based on geographic characteristics and through information available from ordering histories, non-response weights will be designed based on those program characteristics that will allow for reweighting the sample of survey respondents to be similar to the population of organizations that placed orders for C2C materials. Logistic regression models will be used to predict the propensity of an organization participating in the survey and the inverse of the propensity will be used to construct non-response weights. These obtained weights will be used throughout the analyses for inference.

The consumer survey will draw participants from the Knowledge Networks Knowledge Panel®. This probability-based sample will be provided with sampling weights that will be used in analyses to obtain nationally representative findings. A more detailed explanation of the sample calculation is available in Section B.1.2 of Supporting Statement B.

(2) Descriptive analyses. In order to address the study questions outlined above, summary statistics of the different survey questions (means, percentages) and cross tabulations or two or more questions will be reported in aggregate and for subgroups. For both organizational and consumer surveys, the primary subgroups of interest will be based on the level of C2C saturation within the geographic area where the organization is sited or the consumer resides. Other subgroups of interest for the organizational survey include:

• Perception of C2C value and helpfulness;

• Urban versus rural location; and

• Demographics of the population served by the organization, based on census characteristics (e.g., racial/ethnic characteristics, rates of insurance.

For the consumer survey, we will identify similar subgroups based on individual-level characteristics, rather than geographic characteristics. Additional organization and consumer subgroups that are not anticipated in advance of the data collection may emerge as part of the descriptive analysis.

T-test or chi-squared tests will be conducted to test for differences across subgroups. Multivariable regression models will also be used to examine the factors associated with key outcomes of interest related to organizational use of C2C materials. For consumer surveys, the primary predictor will be level of C2C saturation.

Analysis of Information from Semi-Structured Interviews. Drawing on state-of-the-art practices for analyzing qualitative data,⁵ including grounded theory,^{6 ,7} the analysis plan for the semi-structured interviews with key staff from organizations and focus groups of consumers will include: (1) identifying themes, (2) building and applying a codebook, (3) describing themes, and (4) identifying patterns.

(1) Identifying themes. To identify themes (the abstract constructs that researchers may identify before, during, and after data collection), a variety of techniques, including those from the analytic tradition of grounded theory, to read a sample of transcripts to look for examples that suggest processes, actions, assumptions, and consequences will be used. Metaphors, repetitions across informants, and shifts in content that may indicate relevant themes will be explored. Text management software (e.g., ATLAS.ti) will be used to review texts and mark instances where each theme occurs.

(2) Building and applying a codebook. To increase inter-coder reliability and the validity of the findings, a codebook will be developed using standard procedures. Qualitative codebooks, similar to quantitative codebooks, list each theme (rather than each variable) accompanied by a detailed description, inclusion and exclusion criteria, and exemplars. Once the codebook is complete, we will meet to familiarize ourselves with a set of standardized procedures for marking chunks of text that pertain to each theme. To practice the procedures, we will select a random sample of transcript sections that we will each code independently. On completion, the coding will be reviewed as a group. Disagreement among coders will suggest where the codebook may be ambiguous and confusing. Ambiguities will be fixed and additional exemplars will be included in the codebook. The training will continue until coders are familiar with the codebook and can consistently identify and mark each theme when it appears in sample texts. Upon completion of the training, two coders will analyze each interview. The first coder will take the first pass at marking the text for themes. The second coder will re-examine the text to assure that no themes have been missed. In an exploratory analysis like this, we are more concerned with finding all examples of a theme and less concerned with calculating a measure of inter-coder agreement. Using two coders helps us accomplish this goal.

(3) Describing themes. Once coding is complete, we will use the coding software to retrieve all instances of each theme. We will review these instances and describe the theme by presenting segments of text—paraphrases of cases and verbatim quotes from informants—as typical and atypical examples of concepts. We will also examine the distribution of the theme across all groups.

(4) Identifying patterns. There are two types of patterns we will explore: (1) “cross group” themes that cut across all respondents and (2) “within-group” thematic similarities and differences that appear within respondent sub-groups. We will examine to what degree themes are central or peripheral to group members and how they might be distributed across various group characteristics. Finally, we will identify how themes from the qualitative interviews are linked to survey data. For example, we may sort the qualitative themes and map to categories in the quantitative data. Then, we will note the extent to which the interview data expand or align with a quantitative data point, or contradict a survey finding.

A.16.2. Time Schedule and Publications

Exhibit A.16.1 details the timeline for surveys with organizations and consumers and Exhibit A.16.2 details the timeline for the case study.

Exhibit A.16.1 Timeline for Surveys with organizations and consumers
Task	Planned Start Date	Planned End Date
Acquisition of survey frame and file preparation	OMB approval	220 days after OMB approval
Data collection activities	30 days after OMB approval	180 days after OMB approval
Data analysis	90 days after OMB approval	450 days after OMB approval
Prepare and submit report	300 days after OMB approval	450 days after OMB approval

Exhibit A.16.2 Timeline for Case Study
Task	Planned Start Date	Planned End Date
Sample selection	60 days after OMB approval	220 days after OMB approval
Data collection activities	90 days after OMB approval	360 days after OMB approval
Data analysis	90 days after OMB approval	420 days after OMB approval
Prepare and submit report	300 days after OMB approval	450 days after OMB approval

A final report and at least one research brief will be prepared and delivered five weeks before the project end date (September 2019). The report will have an executive summary that will present key findings as well as answers to the study research questions. The full report will describe the study purpose and research questions, detail the approach and data collection methodology, present the results of the analysis of both the survey data and the case study, and discuss implications of the findings for the CMS Office of Minority Health.

A.17. Display of OMB Expiration Date

All instruments involved in this data collection will display the number and expiration date of OMB approval. The OMB number and expiration date will be displayed on the first Web page of the online survey of organizations and the online survey of consumers. The OMB number and expiration date will be read at the start of each semi-structured interview and focus group.

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this data collection.

List of Appendices Under Separate Cover

Appendix A	Organization survey and consent form
Appendix B	Consumer survey and consent form
Appendix C	Case study: Semi-structured interview guide and consent form
Appendix D	Case study: Focus group protocol and consent form
Appendix E	Additional study components of relevance to the C2C evaluation that do not require OMB review
Appendix F	Recruitment scripts for online surveys, semi-structured interviews and focus groups
Appendix G	Letter of support from CMS to be included with recruitment scripts for organizations

1 Rogers, Everett M., Diffusion of Innovations, 5th ed., New York: Free Press, 2003.

2 Singer and Ye (2013), The Use and Effects of Incentives in Surveys, Annals of the American Academy of Political and Social Science, 645(1): 112-141.

3 Young et al. (2015), Unconditional and conditional incentives differentially improved general practitioners' participation in an online survey: randomized controlled trial, Journal of Clinical Epidemiology, 68(6): 693-697.

4 Bureau of Labor Statistics. Usual Weekly Earnings of Wage and Salary Workers Fourth Quarter 2014. (Table 5: Quartiles and selected deciles of usual weekly earnings full-time wage and salary workers by selected characteristics, 4^th quarter 2014 averages, not seasonally adjusted). http://www.bls.gov/news.release/archives/wkyeng_01212015.htm

5 Bernard H. Social research methods: Qualitative and quantitative approaches. Thousand Oaks, CA: Sage; 2000.

6 Strauss A, Corbin J. Basics of qualitative research: Grounded theory procedures and techniques. Newbury Park, CA: Sage; 1990.

7 Glaser B, Strauss A. The Discovery of Grounded Theory: Strategies for Qualitative Research. New York: Aldine; 1967.