OMB Control Number: 0938-NEW
Expiration Date: TBD
Supporting Statement – Part B
The Centers for Medicare & Medicaid Services (CMS) is interested in collecting information to assess beneficiaries’ satisfaction, understanding, and experience with the Montana Health Economic Livelihood Partnership (HELP) Demonstration in the State of Montana. This data collection effort will help characterize beneficiaries’ experiences in general under HELP, as well as specifically how experiences and understanding vary among key subgroups. This data collection effort includes two primary components: 1) a beneficiary survey in two waves; and 2) a qualitative component encompassing two site visits, key informational interviews, and focus groups. These are detailed below and in the Appendices.
The respondent universe of the beneficiary survey includes current HELP enrollees and HELP disenrollees. The sampled members of these two populations will receive questionnaires eliciting information on the various aspects of understanding, cost considerations, satisfaction and experience with the HELP plan. The two surveys differ as appropriate to the targeted respondents. Table 1 summarizes the number of beneficiaries in the two respondent universes. We propose to select a random sample. We will select an enrollee sample of 2,187 to obtain approximately 700 completed interviews.
For the disenrollee survey, we propose to select all available HELP disenrollees into our sample to obtain up to 700 completes since we believe the numbers will be small due to the relatively short duration of the HELP plan. The disenrollee population would include individuals who have voluntarily disenrolled for various reasons and those who were disenrolled due to premium non-payment.
Survey Instrument |
Members Selected into Sample |
Target Completed Responses |
HELP Enrollee |
2,187 |
700 |
HELP Disenrollees |
2,187 |
700 |
Totals |
4,374 |
1,400 |
Based on previous response rates associated with this target population, we assume a response rate of approximately 32 percent.
The focus groups will include HELP enrollees. Informational interviews with key stakeholders will include Medicaid officials, state budget and financial officers, health plan administrators, health care providers, health care industry representatives, consumer advocates, and community based enrollment assistors, and tribal or Indian Health Services representatives among others. All participants selected for the qualitative component will rely on the voluntary participation of the individuals. Throughout the recruitment process as well as at the beginning of each focus group, researchers will emphasize that participation is voluntary; all comments will be kept private to the extent allowed by law.
The participant list identified for focus group recruitment will be derived from enrollment files. The SSS Team will request from the State of Montana limited information for current HELP enrollees from the State HELP enrollment files. This file will serve as the sampling frame.
. Apart from being currently enrolled in the HELP demonstration, individuals will also meet the following criteria--adult enrollees (ages 19-64 years) whose primary language is English and who are continuously enrolled in the HELP program for at least four (4) months at the time the lists are drawn, and live in Helena, Billings, Blackfeet Indian Reservation or Havre. For each sample person we will request the following information from the HELP enrollment files: name, contact information (street address, phone number(s), email address if available), age, gender, race/ethnicity, and income level (preferably as a percentage of the federal poverty level).
The identification and selection of specific individuals for the informational interviews will occur through an iterative process. First, researchers will examine public records, periodicals, and literature to develop a preliminary roster of individuals and organizations to be considered. Second, a series of telephone conversations will occur with state officials in order to draft a roster of stakeholders, verify roles and determine if they currently play key roles relevant to HELP. Third, after the key individuals are verified and identified researchers will contact key organizations (beyond state officials) to verify the appropriate experts and leaders most important to interview, and to inquire whether there are other stakeholders the researchers may have overlooked that should be added to the list. Lastly, when the stakeholder list is finalized interviews will be scheduled to occur during the site visit.
This plan assumes enrollee demographic information will be available to us at the time of sampling. We plan to draw a random sample from both the enrollees and disenrollee populations. Our general aim is to quantify characteristics of the target populations with a reasonable amount of precision. For example, the margin of error in estimating a population proportion from either sample (enrollees or disenrollees) would be no more than 3.7%. This calculation assumes an underlying population proportion of 50 percent, the most conservative assumption (i.e., requiring the largest sample) for computing margin of error in sample estimation. If the underlying population proportion is closer to 20%, then the margin of error in estimating it would be approximately 3%.
These sample sizes would also allow reasonably accurate comparisons of proportions between subgroups. For example, we would have approximately 80% power to detect a difference of 10percentage points or more between urban and rural beneficiaries at the 95% level of significance. This calculation assumes approximately equal numbers of respondents from urban and rural environments, and population proportions of 55% and 45% in the two groups, respectively.
The two versions of the HELP beneficiary survey will require different sampling considerations for selecting the target populations for each questionnaire. To account for the complex sampling requirements (i.e., to classify the sampling frame into strata), we will primarily be looking at beneficiary enrollment variables. Possible variables will include, for example: length of time enrolled in HELP, failure to pay premiums on time, area of residence (urban or rural), and federal poverty level.
After controlling for the variables needed to identify the appropriate beneficiaries for each survey questionnaire, we will draw random samples from HELP enrollees and disenrollees. All standard SSS Survey Operations administration specifications will be followed – including removing duplicates and ensuring that only one member of a household is surveyed.
The beneficiary survey will be administered via three data collection modes: mail questionnaire, phone follow up for non-respondents, and online (web) questionnaire. The web option will be optimized to ensure that survey participants can complete the questionnaire on mobile devices (e.g., cell phones, tablets, etc.). All survey questionnaires and materials will be made available in Spanish. Bilingual interviewers will also be available as needed.
Participants will receive a cover letter inviting everyone to participate in the survey (by mail or online), as well as the paper mail-in survey. Non-respondent participants will receive a maximum of two reminder cards and two paper mail-in surveys accompanied by a prepaid return envelope. Non-respondents will be followed up by phone after the second reminder card has been mailed.
Interviewers will contact non-respondents and attempt to complete the survey via telephone with them after the second questionnaire reminder card is sent. The SSS Team will make up to ten calls per non-respondent to attempt to collect the survey data. During the entire telephone follow-up period, respondents who already have completed the questionnaire and returned it by mail or completed it online will be suppressed from phone interviewing.
The survey management system will automatically schedule callbacks and allow an interview to be resumed at a later date if necessary. The system is also designed to accommodate multiple questionnaire versions so that the appropriate version is used based on the respondent’s tracking number.
A less frequent or delayed data collection would not serve the purposes of completion of the evaluation of the HELP demonstration, approved through December 31, 2020. It is crucial that the data collection be completed on time for both the first round and second round of surveys, focus groups, and informational interviews. The second round of data collection will occur approximately one year after the first round to allow for comparisons over time.
The data collection team will implement a number of procedures to maximize response rates. Cover letters and surveys will be mailed to all sample members. Mailing reminders and phone follow ups will occur over the course of the data collection period. Approximately one week after mailing the initial survey, a reminder card will be mailed to all sampled members who have not yet completed the survey. Four weeks after mailing the initial questionnaire, a second survey and cover letter will be sent to all non-respondents. A second reminder postcard will be mailed to all non-respondents approximately one week after mailing the second survey packet. Three weeks after the second questionnaire is mailed, telephone follow up will be initiated to members who have not yet responded and completed a mail or web survey (and who have not refused to participate). In addition, we will support bilingual interviewing at our phone facilities (English-Spanish). We anticipate that these procedures, the offer of multiple modes for completion of the survey, and the payment for completing the survey will result in an overall response rate in the 32 percent range, with some variation among program groups.
Subsequent to the fielding of the beneficiary surveys we will conduct a non-response analysis. Survey non-respondents will be compared to survey respondents on demographic and other characteristics to see if there is a systematic difference between those who responded and those who did not. Sampling weights will adjust for any differential response, as needed.
The beneficiary survey was developed from a number of previously vetted and fielded beneficiary surveys. Several existing beneficiary surveys were examined during survey development including: The Consumer Assessment of Healthcare Providers and Systems (CAHPS), CAHPS Qualified Health Plan Survey, Nationwide Medicaid CAHPS Survey, CAHPS Supplemental Items for Adult Questionnaires (CAHPS Healthy Plan Survey 4.0), Behavioral Risk Factor Surveillance System (BRFSS), National Health Interview Survey (NHIS), Iowa Wellness Plan, Healthy Indiana Plan 1.0 Beneficiary Surveys: Enrollee and Leaver Survey, the Healthy Indiana Plan 2.0 Beneficiary Surveys (state evaluation): HIP Basic; HIP Plus; Never member, no POWER account contribution made, and the Healthy Indiana Plan 2.0 Beneficiary Surveys (federal evaluation): Current enrollees; New enrollee; Disenrollees and lockouts. Revisions to the survey instruments have been made based on the instrument pretest, feedback from CMS, and review by survey experts. No public comments were received from the publication of the 60-day and 30-day Federal Register Notices.
Questionnaires completed online and via phone will be subject to programmed Quality Control procedures to ensure data have been captured correctly. Mail survey data will be dual-entered and adjudicated where there are discrepancies.
Our evaluation of HELP has three main research goals:
Provide in-depth understanding of the design, implementation, ongoing operation and impacts of HELP
Provide a detailed examination of beneficiary experience under HELP
Estimate impacts HELP on health insurance coverage, access, service use, affordability as well as quality of care and health and health behaviors
Collecting both qualitative and quantitative data will help to inform and achieve these goals.
The qualitative components (focus groups and informational interviews with key stakeholders) will enable assessment of how the launch of HELP proceeded and also to identify the successes and challenges in establishing and administering HELP. Additionally, the informational interviews will provide important insights into how major stakeholders to HELP perceive the operations and effectiveness of the program, from its beginnings to its maturity. Focus groups will further enrich the evaluation by capturing the “voices” of adults affected by HELP, providing valuable details about their experiences and concerns, details that cannot be obtained in the beneficiary survey. Apart from providing an in-depth understanding of the design, implementation and operations of HELP and a detailed consumer perspective on the program, data from informational interviews will inform our impact analyses by guiding these analyses and providing valuable context for interpreting the results.
The beneficiary survey will serve as a vital data collection opportunity for information on other policies being tested under the HELP demonstration, including the 90-day grace period for the non-payment of premiums, and premium credits and copays. The beneficiary survey instruments have been designed to adequately collect beneficiary experience and understanding in various survey domains including access to care, affordability, premiums and copays, satisfaction, and HELP enrollment.
The majority of the survey questions have been adapted from or directly taken from previously vetted, tested, and/or fielded beneficiary surveys.
Survey instrument testing activities occurred shortly after the 60-day public comment period concluded. A total of 8 beneficiaries participated in in-person respondent debriefings over the course of two days (January 4, 2017 – January 5, 2017) in Billings, Montana. The survey instrument testing participants were current and previous HELP enrollees.
Overall, the survey instrument testing did not result in any major survey revisions. Given that the majority of the survey questions have been adapted from or directly taken from previously vetted, testing, and/or fielded beneficiary surveys, the survey team did not anticipate any major revisions. Additionally, the respondent debriefing focused on newly developed questions or adapted beneficiary survey questions.
Data Collection Instruments Revision Summary
While the data collection instruments and associated materials did not receive any public comments during the 60-day public comment period, the beneficiary survey instruments were pretested. However, the State of Montana provided comments on the draft federal evaluation design. Revisions were made to the appropriate data collection instructions.
Revisions to the beneficiary surveys were in response due to the survey instrument testing; however, moderate beneficiary survey revisions were made in response to the comments provided by the State of Montana. The beneficiary survey crosswalk clearly identifies what changes were made and includes the justification for such changes.
The beneficiary survey revisions can be characterized by the following: (1) improved clarity and flow of survey and/or survey questions and (2) more focused examination of research questions through expansion or omission of components within a survey domain.
Survey revisions focused on improving the survey flow and clarity of both survey questions and answer choices provided can be seen across both survey instruments. These revisions were considered minor as they did not change the nature or intent of the survey questions.
Moderate changes were made to the beneficiary survey instruments in response to comments provided by the state of Montana. These changes were represented by the addition of survey questions and/or survey sections, addition of survey questions to one of the surveys, and the omission of some survey questions. The survey questions and sections added included questions focusing on insurance coverage before enrolling in HELP, the comparison of previous health insurance (if any) to HELP, an examination of the 12-month continuous eligibility policy in HELP, and emergency room related questions in the disenrollee survey. The survey questions that were omitted include all transportation related questions.
Revisions to the focus group moderator guide and informational interview guide were in response to the comments provided by the State of Montana. The moderator guide and interview guide crosswalks clearly identify what changes were made and includes justification for such changes.
The interview protocol was modified to include the addition of a Tribal or Indian Health Services representatives as a stakeholder type. The focus group moderator guide was modified to include content specific to enrollees living in the Blackfeet Indian Reservation, MT and in Havre, MT.
The following persons outside of CMS contributed to, reviewed, and/or approved the design, instrumentation and sampling plan:
Name |
Affiliation |
Telephone Number |
Niranjana Kowlessar, PhD |
Social & Scientific Systems, Inc. |
301-628-3275 |
Linda Piccinino, MPS, DrPH (ABD) |
Social & Scientific Systems, Inc. |
301-628-0244 |
Tim Powers, MS |
Social & Scientific Systems, Inc. |
301-628-3039 |
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Statement – Part B |
| Subject | Supporting Statement – Part B |
| Author | itsupport |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |