NHE OMB Part B_Final_rev

PART B: JUSTIFICATION for Evaluation of NHE GRANTS to Address the opioid Crisis

OMB No.

october 2020

PART B: STATISTICAL METHODS

The evaluation of the National Health Emergency (NHE) Dislocated Worker Demonstration Grants to Address the Opioid Crisis involves interviews with both individuals and focus groups. Information on the burden associated with the individual interviews and the focus groups was provided in Part A above, and copies of all protocols and question lists are attached to this supporting statement.

These grants enable states to test innovative approaches to address the economic and workforce-related impacts of the opioid epidemic. The evaluation of the NHE Grants to Address the Opioid Crisis offers a unique opportunity to build knowledge about the implementation of these approaches, identify perceived challenges and promising practices, and share information with grantees and other stakeholders as they seek to address the opioid crisis. This exploratory study is examining the implementation of grants that are preliminary in nature. These findings will not be used to make statistical inferences that are more broadly generalizable. Mathematica Policy Research and its subcontractor Social Policy Research Associates have been contracted to conduct an implementation evaluation. As required by the Paperwork Reduction Act (PRA), this package requests clearance for the following four implementation study data collection activities:

1. Semistructured interviews with administrators and staff

2. Interview respondent information forms

3. Focus groups with NHE program participants

4. Participant focus group information forms

B.1. Respondent universe and sampling methods

1. The universe of potential sites for this evaluation includes the six grantees that received NHE grants in July 2018. All six grantee sites will be included in the implementation study. This section outlines the respondent universe, proposed methods for respondent selection, and the expected response rates for the two types of respondents participating in the proposed research. Due to the nature of the sample, including the data gathered through the interview respondent information forms and the participant focus group information forms, it will not be feasible to make statistical interferences about these type of programs, the broader set of staff who provide grant services, or about participants who are engaged in those services.State and local administrator, staff, and partners. No statistical methods will be used in selecting interviewees from the following categories: (1) state administrators, (2) state grant directors, (3) state partners, (4) local subgrant directors, (5) frontline staff, (6) employers, and (7) local partners. Across the six NHE grantees, the study team estimates that the universe of state administrators, state grant directors, and state partners most involved in activities that affect NHE grant operations totals 10 individuals per site (or 60 individuals across sites). The study team also plans to interview a purposeful sample of 20 local subgrant directors, frontline staff, employers, and local partners per site (or 120 individuals across sites). As demonstrated in Table B.1, the study team plans to interview the universe of state administrators, grant directors, and partners and a purposeful sample of local subgrant directors, frontline staff, employers, and local partners and anticipates a 100 percent response rate for these interviews. The following describes how state-level administrator, staff and partners, and local-level subgrant directors, frontline staff, employers, and partners will be selected:

• The study team estimates the number of state administrators, state grant directors, and state partners will vary from site to site but estimates a universe of 10 administrators, grant directors, and partners per site (or 60 individuals across sites). The study team plans to interview the universe of these state-level respondents.

• The number of local-level subgrant directors, frontline staff, employers, and local partners will vary by grantee depending on the number of geographic areas in which each grantee is operating. During the course of each site visit, the study team will visit two local areas, on average, and interview the local subgrant director, frontline staff, employers, and local partners most involved in activities that affect NHE grant operations in each of these local areas. We anticipate that each local area will have a universe of 10 individuals in these categories, and we will interview all 10 individuals in each of the ocal areas selected, for a total of 20 local interviewees per grantee.

To select the local areas for the site visit (for grantees providing services in many locations), we will work with the identified grantee liaison to identify local areas in which to conduct the field work. To the extent possible, we will work with the liaison to identify geographically diverse local areas. The number of local areas will vary by grantee but we estimate that on average, each grantee is providing services in 4 local areas, for a total universe of 24 local areas.

1. NHE participants. The study team will collect data from up to 10 NHE participants at each visited site to gather data on their backgrounds, employment experience, motivation for participation, and description and assessment of training received. The universe of NHE participants across the six grantees is estimated to be 3,500 (with an average of approximately 600 participants per site over the life of the grant). We will conduct focus groups with the following two participant types: (1) people receiving health care occupational training through the grant, and (2) crisis-affected individuals receiving employment services through the grant. To select focus group participants, the study team will use a sample of convenience and rely on NHE grant staff to identify and invite 20 participants (10 from each category) who have received NHE grant services (for a total of 120 individuals across sites), with the expectation that up to 10 per site are likely to attend, for a total of 60 participants across sites (and a response rate of 50%). The data collected from NHE focus group participants will not be generalized to the broader universe of NHE program participants.

Table B.1 summarizes this information, including the sampling method, estimated sample sizes, and assumptions about response rates for each data collection activity and respondent type by site.

Table B.1. Sampling and response rate assumptions, by data collection activity and respondent type

Respondent type	Sampling method	Estimated universe across 6 sites	Sample (per site)	Estimated response rate (percent)	Estimated responses (per site)	Estimated responses (across sites)
State administrators, staff, and partnersa	Census	60	10	100	10	60
Local administrators, staff, and partnersa	Purposeful	240	20	100	20	120
NHE participantsb	Convenience	3,500	20	50	10	60
---	---	---	50	---	40	240

^a The study team anticipates 100 percent of people who participate in the interviews will complete a respondent information form.

^b The study team anticipates 100 percent of people who participate in the focus groups will complete a participant information form.

B.2. Procedures for the collection of information

Understanding the implementation of the NHE grants requires data collection from multiple sources. The implementation evaluation data collection instruments included in this clearance request include the protocols that will be used to conduct in-person interviews and focus groups during site visits for all six grantees. Over the course of each mulitday site visit, the site visit team will spend time at the state level, interviewing the state grantee and state-level partners, and at select local areas participating in the grant. The study team will collect a rich set of qualitative data from NHE administrators, frontline staff, partners, and participants. The data covered by this clearance include semistructured interview data, focus group data, and demographic and contextual data from interview respondent information forms and participant focus group information forms. Next, we describe each of these data collection activities in detail.

Semistructured administrator and manager interview and information form data. During each visit, two members of the study team will conduct semistructured, in-depth interviews with the universe of administrative staff responsible for the NHE grant, including the (1) state administrator, (2) state grant director, (3) state partners, (4) local subgrant directors, (5) frontline staff, (6) employers, and (7) local partners. The data collected using the semistructured interview protocols will focus on the rationale behind grant plans and activities, the context of the state’s opioid crisis and other initiatives addressing it, and implementation progress, perceived successes, and perceived challenges. The master protocol will be used to create individual discussion guides based on the respondent type before the site visits.

The interview respondent information form will sent by email to administrators and staff who will be interviewed during in-person site visits. In cases where respondents do not complete the form in advance, it will be administrated before the start of each administrator and staff interview. This short questionnaire includes basic background information, such as their highest education level and experience.

Focus group and participant information form data. During each site visit, the study team will conduct two focus groups. Separate focus groups will be conducted with people receiving health care occupational training through the grant, and crisis-affected individuals receiving employment services through the grant. The focus groups will gather information about the opioid use context in participants’ communities, the services and training they have received through the grant, their assessment of these services and trainings, and their overall experiences with the program.

At the beginning of each focus group, the study team will hand out a paper information form to each participant. This brief, three-page questionnaire will gather demographic characteristics and other relevant information including how participants were referred to the program, how long they have been involved, and their previous work experience. The questionnaires will not ask for personally identifiable information. The study team anticipates 100 percent of people who participate in the focus groups will complete a participant information form.

The following describes the data collection procedures specific to each type of respondent:

• State and local administrators, staff and partners. The study team will conduct interviews and information form data collections with up to 10 state administrators, staff, and partners and 20 local administrators, staff and partners at each of the six sites. For each site visit, the study team will work with the state grantee to purposefully select two local areas to visit. In both the state and local areas, the study team will work with grantee staff to identify administrators, staff, and partners involved in the NHE grant and invite them to participate in interviews.

• NHE participants. As previously mentioned, the study team will conduct two participant focus groups and information form collections in each site. These will include one focus group with people receiving health care occupational training through the grant and one focus group with crisis-affected individuals receiving employment services through the grant. For each of the focus groups, the study team will use a sample of convenience and rely on NHE staff to identify and invite 10 participants, with the expectation that up to 5 are likely to attend. Therefore, across the six sites, approximately 120 participants will be invited to attend the focus groups and approximately 50 percent, or 60 are anticipated to attend. A convenience sampling method is appropriate here because the study team will not have access to participant contact information or be able to reach participants without the assistance of NHE staff.

1. Statistical methodology, estimation, and degree of accuracy

The main type of data collected from the interview and focus group respondents will be qualitative information about staff’s experiences and insights implementing the NHE grant or, in the case of participants, their motivations for participating in the NHE and their experiences while doing so. Thus, no statistical methodology (such as sample stratification) or estimation will be needed in the analysis of the interview or focus group data. The qualitative data collected will be analyzed using qualitative coding software such as NVivo or ATLAS.ti. This coding will enable us to identify key themes across sites and enable the study team to describe NHE grants. Because the implementation study is examining grant implementation, study findings will apply only to the NHE grantees and will not be more broadly generalizable.

The data gathered through the interview respondent information forms and participant focus group information forms will be tabulated using descriptive methods (including simple frequencies, cross-tabulations, and means, when appropriate) to provide contextual information about the characteristics of participants who provide the qualitative interview and focus group data. We will not use these data to make inferences about the broader set of staff who provide grant services, or to participants who are engaged in those services.

2. Unusual problems requiring specialized sampling procedures

As mentioned previously, the semistructured interview and focus group data will be used to describe the NHE grants, including the perspectives of frontline staff, partners, and participants. Although the study team plans to use sampling methods for the local administrator, staff, and partner interviews and NHE participants, the state administrator, staff, and partner interviews will involve the universe of respondents. Using a census for this set of respondents will ensure that the study captures the diversity of grantees’ experiences and the perspectives of multiple respondents in each state. Without talking to all the key state administrators and frontline staff, the study team might miss important information regarding the implementation of the NHE grant.

B.3. Methods to maximize response rates and minimize nonresponse

As indicated in their grant agreements, all six NHE sites included in the evaluation are expected to participate in the implementation study data collection activities. As we schedule site visits, we will also explain the nature of the visits, so that state and local staff and partners know what we expect of them when they agree to participate. Site visitors will work with site staff to ensure that the timing of the visit is convenient for all staff involved.

Semistructured interviews with state and local administrators, staff, and partners and interview respondent information forms. The study team will be flexible in scheduling interviews with state and local administrators, staff, and partners to accommodate the particular needs of respondents. Furthermore, data collectors will meet with in-person interview respondents in their office locations or in a central location that is well-known and accessible.

Although the study team will try to arrange interviews that accommodate respondents’ scheduling needs, there might be instances when a respondent is unable to meet while the team is on site; when this happens, a member of the study team will request to meet with the respondent’s designee or schedule a follow-up call at a more convenient time. With these approaches, the study team anticipates a 100 percent response rate for staff interviews, as has been achieved on similar qualitative data collection efforts, such as those for the Workforce Investment Act Adult and Dislocated Worker Programs Gold Standard Evaluation, the Evaluation of the Linking to Employment Activities Pre-Release grants, the Evaluation of the Summer Youth Employment Initiative, and the Impact Evaluation of the Trade Adjustment Assistance Program.

We will send the interview respondent information forms by email to each respondent in advance of the discussion, and bring hard copies to administer in-person for respondents who do not complete the form in advance. The information form is a single, one-page form, and the questions are written in clear and straightforward language. The average time required for respondents to complete the information form is estimated to be approximately 5 minutes for respondents. Previous experience collecting such demographic and contextual information from staff, such as the Evaluation of the Linking to Employment Activities Pre-Release grants, suggests that 100 percent of interviewees will complete the information form.

Focus groups and participant information forms. To encourage participation in the focus groups, the study team will use methods that have been successful for numerous other Mathematica studies, including providing easy-to-understand outreach materials, strategically scheduling focus groups at convenient times and locations, and offering incentives to encourage participants to respond.

• Outreach materials will be used to help sites recruit participants for the focus groups. These materials will (1) describe the study, its purpose, and how the data collected will be used; (2) highlight DOL as the study sponsor; (3) explain the voluntary nature of participation in focus groups; and (4) provide a phone number and email address for questions that focus group members might have. Outreach materials for participants will be clear and succinct and convey the importance of the focus group data collection.

• The study team will consider respondents’ schedules and availability when scheduling the focus groups to maximize response. In addition, data collectors will meet with respondents in locations that are convenient.

• In addition to these strategies, we also plan to offer incentives for the focus group participants. Unlike the NHE staff and partners who are invested in the grant and evaluation, NHE participants will be more difficult to locate and invite to participate. Therefore, to encourage participation in the focus groups, the study team will use methods that have been successful for numerous other Mathematica studies, such as the Evaluation of Linking Employment Activities Pre-Release and the Evaluation of Youth Career Connect, including offering the focus group participants a $40 gift card incentive. Although this is a nominal amount that is not likely large enough to be coercive for participants, the payment will serve two purposes: (1) to facilitate recruitment by increasing the likelihood that an NHE participant will agree to participate in the focus group, and (2) to acknowledge that participants’ time is valuable and that participants may incur some costs (e.g., transportation and/or child care) to attend.

We will hand out the participant information forms at the beginning of each focus group. The information form is a single, three-page form, and the questions are written in clear and straightforward language. The average time required for respondents to complete the information form is estimated to be approximately 5 minutes for participants. Previous experience collecting such demographic and contextual information from focus group respondents suggests that 100 percent of focus group attendees will complete the information form.

Methods to ensure data reliability. We will use several well-proven strategies to ensure the reliability of the interview and focus group data collected during the site visits. First, site visitors, all of whom already have extensive experience with this data collection method, will be thoroughly trained in aspects particular to this study, including how to probe for additional details to help interpret responses to interview and focus group questions. Second, this training and the use of the protocols will ensure that the data are collected in a standardized way across sites. Finally, all interview and focus group respondents will be assured that their responses will be reported anonymously; reports will never identify respondents by name, and any quotes will be devoid of identifying information, including site name.

B.4. Tests of procedures or methods to be undertaken

All procedures, instruments, and protocols to be used in the NHE evaluation will be reviewed by content and methodological experts to ensure clarity and optimal ordering of the questions. During the first visit of the six site visits, the research team will assess the visit agenda—including how data collection activities should be structured during each site visit—and ensure it is practical, given the amount of data to be collected and the amount of time allotted for each data collection activity. Based on its experience with the first site visit, the study team will train all site visitors on the data collection instruments to ensure a common understanding of the key objectives and concepts as well as fidelity to the protocols. The training session will cover topics such as the study purposes and research questions, data collection protocols, procedures for scheduling visits and conducting on-site activities (including a review of interview and focus group facilitation techniques and procedures for protecting human subjects), and post-visit files and summaries.

B.5. Individuals consulted on statistical aspects of design and on collecting and/or analyzing data

Consultations on the methods used in this study were part of the study design phase to ensure technical soundness. The individuals listed in Table B.2 were consulted on the methods discussed in this submission to the Office of Management and Budget.

Table B.2. Individuals who were consulted for the NHE evaluation

Mathematica Policy Research P.O. Box 2393 Princeton, NJ 08543-2393 (609) 799-3535

Jeanne Bellotti Area Leader for Labor and Employment (609) 275-2243 Dr. Jillian Berk Associate Director (202) 264-3449 Dr. Crystal Blyler Senior Researcher (202) 250-3502 Michaela Vine Senior Researcher (617) 674-8358

Staff responsible for overseeing the collection and analysis of data are listed in Table B.3.

Table B.3. Individuals who will oversee the collection and analysis of data for the NHE evaluation

Mathematica Policy Research P.O. Box 2393 Princeton, NJ 08543-2393 (609) 799-3535	Dr. Jillian Berk Associate Director (202) 264-3449 Dr. Crystal Blyler Senior Researcher (202) 250-3502 Michaela Vine Senior Researcher (617) 674-8358
Social Policy Research Associates 1330 Broadway, Suite 1426 Oakland, CA 94612  (510) 763-1499	Kate Dunham Director of the Workforce and Human Services Division (510) 788-2475

1