SMP Supporting Statement - B - Final - 082817

SMP Customer Satisfaction Survey Supporting Statement

B. Collections of Information Employing Statistical Methods

The agency should be prepared to justify its decision not to use statistical methods in any case where such methods might reduce burden or improve accuracy of results. When Item 17 on the Form OMB 83-I is checked, "Yes," the following documentation should be included in the Supporting Statement to the extent that it applies to the methods proposed:

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The respondent universe for this survey project will be the number of individuals who attend SMP group education sessions in a given year. Based on the most recent data available (2014), there were over 14,000 education session presentations, with a total audience of 450,000 individuals. As ACL will sample 17-18 states/territories in a given year, the annual sample will be one-third of the total population, or roughly 150,000 individuals each year. To generate a sample with a 95% confidence level each year 383 responses will be required (see Table B-1-1 below).

SMP will exceed the sampling requirement by collecting 75 completed surveys per state/territory, for a total collection of 3,975 completed surveys (75*53 = 3,975), with 1,350 surveys collected in years 1 and 2, and 1,275 collected in year 3. This larger collection will enable ACL to make state-to-state comparisons, which is an important feature of this survey. Specifically, state-to-state comparisons will allow ACL to identify which states/territories are providing the best services to their customers, and what best practices can be shared across states. The larger collection will also provide each state/territory with sufficient information to take local action to improve service within budgetary constraints.

SMP will stratify the list of 53 states and territories by the average number of education session attendees per month. Once the list of states/territories has been determined, ACL will instruct the states/territories about the criteria used to survey individual state/territory presentations. These criteria will be applied to each state/territory’s education sessions, in order to create a representative sample of the state/territory’s SMP education session attendees.

A traditional sampling methodology cannot be applied to the attendee population because events are only scheduled 2-3 weeks before they occur, and therefore the entire universe of attendees is only known at the end of a given year. ACL will survey education session attendees immediately after the delivery of an SMP presentation, which should both reduce burden on ACL and the public and increase the quality of responses received. The sampling criteria that will be applied to each state/territory’s education session events includes:

1. Surveying a specific site no more than once;

2. Sampling from at least five unique (i.e., non-redundant) presenters in each state/territory;

3. Surveying no fewer than five events and no more than 20 events in each state/territory; and,

4. Surveying no more than two events per month in each state/territory.

Once an event has been chosen, the SMP presenter will give event attendees an opportunity to complete a comment card survey at the conclusion of the presentation. Table B-1-1 indicates how many completed surveys ACL expects each year, with 17-18 states/territories surveyed each year and each state/territory obtaining 75 completed surveys. We anticipate a response rate of 80 percent for the collection.

Table B-1-1: Expected Number of Completed SMP Surveys by Year

Period	States/ Territories Surveyed	Responses Needed for a Representative Survey*	Expected Responses
Year 1	18	383	1,350
Year 2	18	383	1,350
Year 3	17	383	1,275
GRAND TOTAL	53	1,149	3,975

*These response totals are based on 150,000 education session attendees across 17-18 states or one-third of the 450,000 attendees in 2014.

2. Describe the procedures for the collection of information including:

   1. Statistical methodology for stratification and sample selection,

The SMP survey will be stratified at the state/territory level to determine which 18 states and territories are surveyed in Year 1 and Year 2, and which 17 states and territories will be surveyed in Year 3. The states/territories will be stratified by the average number of group education session attendees per month. Once the Year 1, Year 2, and Year 3 states/territories have been established, ACL will send out instructions and conduct a training call to prepare states/territories to conduct, collect, and store survey data.

Specifically, each state/territory will be responsible for identifying upcoming events where comment card surveys will be distributed to attendees. Any event can be selected, so long as it is consistent with the limiting criteria applied to each state’s/territory’s survey (see specialized sampling procedures, below). Once an event has been chosen, the state/territory will notify ACL to ensure that the collection is consistent with the limiting criteria.

Assuming an event has been approved for sampling, the state/territory presenter will bring comment card surveys to his/her presentation. Each comment card will display the location and presenter’s name. Prior to the presentation, the presenter will notify attendees of the opportunity to complete a comment card survey. Once the presentation is over, the presenter will instruct the attendees on where to obtain a survey, how to complete the survey, how long it should take, and how to ask questions about the survey. Once the surveys are completed, the SMP presenter will collect the surveys and then scan them to a third-party contractor for data collection, storage, and analysis. This third-party contractor will assist the state/territory SMPs in tracking the progress of their survey efforts. Survey collection will occur year-round, or until a state/territory has collected 75 completed surveys.

2. Estimation procedure,

ACL will derive its total sample based on three factors:

1. The number of SMP group education session attendees in a given year;

2. The number of respondents required to obtain a 95% confidence level; and,

3. A margin of error of +/- 5%.
   
   
   

3. Degree of accuracy needed for the purpose described in the justification,

In order to generate a national sample of the 53 states/territories, only 384 responses are required. As such, ACL is oversampling both nationally (3,975 total responses) and at each state/territory (75 responses). As a result, a state/territory that generates fewer than 75 responses will be unlikely to impact the national reporting related to this survey. Also, states/territories that do not generate 75 completed responses may not be available to participate in state-to-state comparisons, and states/territories with low response totals will also lose some of the meaning in their individual data sets.

4. Unusual problems requiring specialized sampling procedures,

The SMP survey will not perform a typical sampling procedure, given that state/territory officials are not currently tracking attendee contact information. Even if this information was available, attempting to obtain attendee feedback at some point after the event would be less reliable and add to the respondent’s burden. The ability to obtain feedback immediately following a group education session is a more valuable and less burdensome collection of information.

Given the lack of traditional sampling, ACL will instruct states/territories on specific criteria to apply when selecting events to survey. Specifically, ACL expects states/territories to:

• Survey a specific site no more than once.

• Survey at least five unique (i.e., non-redundant) presenters in each state/territory.

• Survey no fewer than five events and no more than 20 events in each state/territory.

• Survey no more than two events per month in each state/territory.

ACL will monitor each state’s/territory’s collection on an ongoing basis, and will inform a state/territory when it is over- or under-scheduling relative to a specific piece of criteria. ACL will also require states/territories to inform ACL of an upcoming session chosen for survey administration to ensure that none of the criteria above are violated.

5. Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

A specific feature of this survey is the ability to compare performance between states/territories. In order to do so, oversampling in each state/territory is necessary. As a result of the extra response needs, the survey will need to occur across three years. While the data collection will occur over a multi-year period, each entity will be surveyed only once. Trying to limit the full data collection to one year would place an undue burden on ACL and the grantees, and would potentially reduce the quantity and quality of responses obtained.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

ACL is confident in its ability to obtain “reliable” data through its use of sampling criteria to select education sessions available for information collection. ACL will notify education session attendees that comment card surveys will be available for completion prior to the SMP presentation, and will emphasize the importance of obtaining individual feedback in order to inform its continuous improvement efforts.

At most education sessions ACL will only require 5-10 responses in order to meet the more rigorous expectation of 75 responses per state.

Despite these relatively low response needs, ACL will take steps to ensure adequate responses, including:

1. Using a notable agency for data collection:

All surveys will contain ACL and SMP seals and logos, so as to improve the confidence respondents have with the survey.

2. Description of the survey request, both before and after presentations:

Attendees will be made aware of the option to complete comment cards both before and after the SMP presentation, with specific instructions on where to find the surveys, how to complete them, the number of questions (8), and how long it should take an individual attendee to complete the survey (5 minutes).

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

ACL conducted pilot testing prior to survey clearance approval. Nine (9) individuals attending two (2) distinct state/territory education sessions were asked to comment on the survey and its questions. These two (2) tests allowed ACL to improve question wording and better understand the burden placed on individual respondents, specifically in terms of survey length. Discussing the overall survey process with state directors from Pennsylvania and Connecticut also allowed ACL to understand how best to capture a representative sample of respondents.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Three members of the ACL staff will play a primary role in this project:

• Katherine Glendening - Social Service Analyst, 202-795-7350, [email protected]

• Stacey Platte - SMP Program Manager, 202-795-7420, [email protected]

• Josh Hodges - Director, Office of Healthcare Information and Counseling, 202-795-7364, [email protected]

This submission was prepared in consultation with contractors from CG Strategy, specifically David Spak, M.P.P., P.M.P., ([email protected]) and Hunter Gray, B.A. ([email protected]). Both Mr. Spak and Mr. Gray can be reached at 703-527-7001.

With respect to specific tasks, the following individuals were responsible for:

• Designing the data collection: Ms. Platte, Mr. Hodges, Mr. Gray, and Mr. Spak.

• Collecting the data: Ms. Glendening, Mr. Gray, and Mr. Spak.

• Analyzing the data: Mr. Spak.

3