SHIP Supporting Statement - B - Final - 082817

SHIP Customer Satisfaction Survey Supporting Statement

B. Collections of Information Employing Statistical Methods

The agency should be prepared to justify its decision not to use statistical methods in any case where such methods might reduce burden or improve accuracy of results. When Item 17 on the Form OMB 83-I is checked, "Yes," the following documentation should be included in the Supporting Statement to the extent that it applies to the methods proposed:

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The respondent universe for this survey project will be the list of all individuals who completed SHIP Medicare counseling sessions occurring one week in the Medicare non-enrollment period (spring/summer) and one week in the Medicare Open Enrollment Period (October 15 – December 7). Based on initial samples, there are roughly 18,000 non-redundant individuals who receive SHIP Medicare counseling services in a given week. This number is the total for 18 states/territories, which is the number of states to be surveyed in a given year. ACL will focus on a non-redundant respondent universe so as not to bias this study towards individual opinion or place an undue burden on individual respondents.

To generate a sample with a 95% confidence level at the national level (with all 54 states/territories represented), 384 responses will be required (n = 3,301,031 SHIP Medicare counseling sessions in 2016). SHIP anticipates collecting 75 completed surveys per state/territory, for a total collection of 4,050 completed surveys, or 1,350 surveys per year. This larger collection will enable ACL to make state-to-state comparisons, which is an important feature of this survey. Specifically, state-to-state comparisons will allow ACL to identify which states/territories are providing the best services to their customers, and what best practices can be shared across states. The larger collection will also provide each state/territory with sufficient information to take local action to improve service.

SHIP will stratify the list of 54 states and territories by data collection method and Medicare-eligible population. The data collection method refers to how each state/territory collects and enters its records of counseling sessions. The majority of states/territories (29 of 54) directly enter counseling records into SHIP’s NPR reporting system, but the remaining states/territories upload data in batches at the end of each month. To ensure that the batch upload states/territories will be able to provide weekly samples twice per year, we will limit these states/territories to Years 2 and 3 of the survey administration period, thereby allowing for technical assistance to these states/territories based on our experience in Year 1, if necessary.

Once the list of states/territories has been determined, ACL will randomly select a group of individuals who received Medicare counseling from SHIP in the previous week. Table B-1-1 indicates how many completed surveys ACL expects each year. We anticipate a response rate of 80 percent for the collection.

Table B-1-1: Expected Number of Completed SHIP Surveys by Year

Period	States	Responses Needed for a Representative Survey 4	Expected Responses
Year 1	18	380	1,350
Year 2	18	380	1,350
Year 3	18	380	1,350
GRAND TOTAL	54	1,140	4,050

⁴ These response totals are based on 36,000 counseling sessions across 18 states, or two weeks’ worth of counseling sessions in 18 states/territories. The estimate of 36,000 counseling sessions is based on 2016 NPR system data. The expectation of two weeks’ worth of data is based on the survey process outlined in question 16 of the Justification. See Table A-16-1 for more information.

2. Describe the procedures for the collection of information including:

   1. Statistical methodology for stratification and sample selection,

The SHIP survey will be stratified at the state/territory level, to determine which 18 states and territories are surveyed in Year 1, Year 2, and Year 3. The states/territories will be stratified by the number of Medicare-eligible residents in the state/territory and the data collection method used by the state/territory. Once the Year 1, Year 2, and Year 3 states/territories have been established, ACL will send out instructions and conduct a training call to prepare grantees within the states/territories to produce weekly counseling data at two points during the year:

1. Outside of Medicare Open Enrollment Period (spring/summer)

2. During Medicare Open Enrollment Period (October 15 – December 7).

During the week identified for sampling, SHIP counselors will notify individuals who request Medicare counseling that a third party might contact them the following week to assess their satisfaction with their counseling experience. On the Monday following the week identified for sampling, ACL will download the complete counseling session data for that week, for each of the 18 states/territories in that year’s survey. The contractor will then randomly select potential respondents to call, with the expectation that each state/territory will generate 37 responses (outside Medicare Open Enrollment Period) and 38 responses (during Medicare Open Enrollment Period). All completed responses will be entered electronically into a password-protected Microsoft Excel database, where ACL can track state/territory progress in survey completion and maintain data for analysis.

2. Estimation procedure,

ACL will derive its total sample based on three factors:

1. The number of unique (i.e., non-redundant) individuals who have received SHIP Medicare counseling at two points each year (those that received service during the Medicare Open Enrollment Period and those that received service outside of the Medicare Open Enrollment Period);

2. The number of respondents required to achieve a 95% confidence level; and,

3. A margin of error of +/- 5%.
   
   
   

3. Degree of accuracy needed for the purpose described in the justification,

In order to generate a national sample, only 384 responses are required across all three years and 54 states/territories. As a result, ACL is oversampling both nationally (4,050 total responses) and at each state/territory (75 responses). A state/territory that generates fewer than 75 responses will be unlikely to impact the national reporting related to this survey. However, states/territories that fail to generate 75 completed responses may not be available to participate in state-to-state comparisons, and states/territories with low response totals will also lose some of the meaning in their individual data sets.

4. Unusual problems requiring specialized sampling procedures, and

There are no unusual problems requiring specialized sampling in this collection.

5. Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

A specific feature of this survey is the ability to compare performance between states/territories. In order to do so, oversampling in each state/territory is necessary. As a result of the extra response needs, the survey will need to occur across three years. Trying to limit the full data collection to one year would place an undue burden on ACL and would potentially reduce the quantity and quality of responses obtained.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Based on ACL's review of similar HHS surveys, an 80% response rate is expected. In order to achieve this rate of response, several best practices will be used, including:

1. Using a notable agency for data collection:

All letters, surveys, and email announcements will contain ACL's seal and logo, so as to improve the confidence respondents have with the survey.

2. Advance Warning of the Survey Request:

Each respondent will be notified during his/her counseling session that he/she may be contacted in the following week to assess their satisfaction with the counseling experience. Respondents will also receive a telephone invitation to schedule a time for survey completion, assuming they are not available to complete the survey at the time they are initially contacted.

3. Computer-Assisted Telephone Interviewing:

This method of collecting data is less burdensome and time consuming than traditional paper surveys.

These methods will minimize non-response. ACL will also conduct an analysis of non-response bias for the survey and for each important question on the survey. If significant bias is identified, ACL will propose changes to the survey to accommodate this bias and/or correct the results in a systematic manner. Since the annual survey is new, ACL has not compiled enough data to provide an analysis of non-response bias. However, ACL will conduct this analysis when enough data has been collected.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

ACL conducted pilot testing prior to survey clearance approval. A small sample of state directors (n = 3) and potential respondents (n = 6) were asked to comment on the survey and its questions. These tests allowed ACL to understand the burden placed on individual respondents, specifically in terms of survey length. As a result, ACL identified methods of collecting relevant information in advance of the survey to reduce survey length and improved question wording to reduce the overall burden. Discussing the overall survey process with state directors from California, Florida, and Ohio also allowed ACL to understand that potential respondents who have received SHIP Medicare counseling should be contacted within 1-2 weeks of their counseling experience in order to improve response rates and quality of response.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Three members of the ACL staff will play a primary role in this project:

1. Katherine Glendening - Social Science Analyst, 202-795-7350, [email protected]

2. Rebecca Kinney - SHIP Program Manager, 202-795-7375, [email protected]

3. Josh Hodges - Director, Office of Healthcare Information and Counseling, 202-795-7364, [email protected]

This submission was prepared in consultation with contractors from Coray Gurnitz Consulting, specifically David Spak, M.P.P., P.M.P., ([email protected]) and Hunter Gray, B.A. ([email protected]). Both Mr. Spak and Mr. Gray can be reached at 703-527-7001.

With respect to specific tasks, the following individuals were responsible for:

4. Designing the data collection: Ms. Kinney, Mr. Hodges, Mr. Gray, and Mr. Spak.

5. Collecting the data: Ms. Glendening, Mr. Gray, and Mr. Spak.

6. Analyzing the data: Mr. Spak.

3