Part B_TechHire_2nd_package_2018_6_7

^{PART B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS}

In this document, we discuss the statistical methods to be used in the data collection activities for the Evaluation of Strategies Used in the TechHire and Strengthening Working Families Initiative (SWFI) Grant Programs. This study is sponsored by the Chief Evaluation Office (CEO) and Employment and Training Administration (ETA) in the U.S. Department of Labor. The purpose of the evaluation is to identify whether the grants help low-wage workers obtain employment in and advance in H-1B industries and occupations and, if so, which strategies are most helpful. CEO has contracted with Westat and its subcontractor, MDRC, to conduct this evaluation. The evaluation includes three components: an implementation study, a randomized controlled trial (RCT) study, and a quasi-experimental design (QED) study.

This request for clearance includes the following data collection activities:

• Grantee survey

• Semi-structured telephone interviews with grantees

• Partner contact information form

• Partner survey

• Semi-structured telephone interviews with partners

• Site visit interviews with grantees and partners

A subsequent OMB submission will seek clearance for the 18-month follow-up participant survey.

B.1 Respondent Universe and Sampling Methods

B.1.1 Respondent Universe

Grantee Survey

The grantee survey will be fielded to the universe of 53 grantees. No attempt will be made to draw inferences to populations other than respondents.

Grantee Semi-Structured Interviews

All 53 grantees will be asked to participate in semi-structured telephone interviews.

Grantee Partner Information Template

Prior to the partner survey fielding, grantees will be asked to complete a template about their grant partners to assemble the sampling frame. The study team reviewed grantee proposals and identified organizations listed as partners of grantees. Approximately 1,200 partners were identified. The study team will ask grantees to review and update the list, as well as to provide current contact information and involvement level (not involved, low, medium, and high) of each partner.

Partner Survey

The evaluation will include a survey of all partner organizations. The partner survey will be fielded to partners who have medium and high involvement in the grant and who have contact information. Partners with no or low involvement may not be aware of grant activities, and only those who are involved are relevant to understanding the education and training and supportive services provided by grantees. Grantees will rate the level of involvement of each partner in grant activities in the partner information template as part of the process of framing the sample.

While the precise number of partners with medium or high involvement is impossible to know prior to data collection, it is safe to assume that the number will be less than the number identified by the study team’s initial review of applications. Nevertheless, we assume 1,200 partners will be invited to complete the survey. No attemp will be made to draw inferences to populations other than respondents.

Partner Semi-Structured Interviews

A subset of 53 partners will be selected for semi-structured telephone interviews. TechHire grantees must be part of a primary partnership that includes members of the workforce investment system, education and training providers, and business-related organizations. Grantees are also required to have at least three employer partners or an industry association. Partnership requirements for SWFI are similar with the additional requirement of a partner that administers or funds childcare services. Partners will be selected in two steps. First, approximately 10 grantees will be purposefully selected to represent a range of organization types and training strategies. Second, approximately 5 to 6 partners will be purposefully seleted from each of the grantees to reflect a variety of partner roles and experiences and to answer research objects about partnerships.

Site Visit Interview

The study team will conduct sites visits to the five RCT sites. Each visit will be 2-3 days in length, and will include hour interviews and observations of program operations. The study team will interview grantee program staff, such as the Program Director, Program Manager/Coordinator, Data Analysts, and other support staff at each grantee site. The study team will also interview grantee partners, including training partners and employer partners. The site visits will use a modular interview guide in which questions will be asked based on the staff member’s knowledge and role. Interviews wills be conducted with 10 grantee program staff and 8 partners during each site visit, for a total of 50 grantee staff and 40 partners. Each interview will take one hour. We will not use any statistical methods in the selection of grantee staff or partners.

B.2 Procedures for the Collection of Information

B.2.1 Statistical Methodology for Stratification and Sample Selection

No sampling methods will be used to select grantees for the survey or semi-structured telephone interviews because the universe will be expected to participate.

For the partner survey, the study team has assembled a list of partner organizations identified by grantees in their proposals. The study team will ask grantees to update the list and provide current contact information. If a grantee cannot provide contact information, the study team will conduct Internet searches and telephone calls to locate the information. However, we anticipate that these efforts will largely be unnecessary as grantees are likely to know this information and be motivated to provide it. Grantees will also be asked to indicate the level of involvement of the partner. The study team will select partners who have a medium or high level of involvement in the grant for the partner survey.

For the partner telephone interviews, the study team will conduct interviews with a purposefully selected sample of partners of grantees to represent a diverse set of grantee and partnership characteristics.

B.2.2 Estimation Procedures

Data collected from the grantee and partner surveys will be analyzed using simple descriptive statistics. Because the grantee and partner surveys include the universe of grantees (i.e., all 53 grantees) and partners, no estimation procedures will be used. Key measures from the grantee survey include the percentage of grantees with forged new partnerships with various types of partners for the grant and the percentage of grantees that experienced various implementation challenges. Partner survey data will be analyzed at both the partner and grantee levels. At the partner level, key measures include the percentage of partners who are involved in various grant activities (e.g., recruitment, curriculum development, etc.) and the percentage of partners who plan to continue working with the grantee after the grant expires. Partner survey responses will also be rolled up by grantee to summarize key measures such as the percentage of grantees that have partners who are involved in various roles or who plan to contribute funds or resources after the grant expires.

Data collected from the semi-structured telephone interviews with grantees and partners and site visits will provide in-depth qualitative information about implementation at the five RCT sites; no estimate procedures will be used. Analysis will be purely descriptive.Data will be analyzed using the constant comparative method, in which qualitative analysis is seen an iterative process through which hypotheses and understanding of implementation are continually refinsed. The study team will identify themes in the interviews to be used to develop a preliminary coding scheme. This initial coding scheme will detail the codes and sub-codes that we will use to categorize and organize all data. It will reflect both a priori categories of interest (e.g., types of systems level activities for child care) as well as themes that emerge from the data (e.g., common challenges that arise in implementation).

B.2.3 Degree of Accuracy Needed

No power analyses were conducted because the analysis will be qualitative and descriptive.

B.2.4 Unusual Problems Requiring Specialized Sampling Procedures

None.

B.2.5 Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

There is no use of less than annual data collection cycles because of the relatively short duration of the grant programs being evaluated.

B.3 Methods to Maximize Response Rates and to Deal With Issues of Nonresponse

B.3.1 Methods to Maximize Response Rates

Grantee Survey

Figure B-1 shows the planned data collection methodology to be used for the grantee surveys. The grantee survey will be a web survey data collection. All 53 grantees will receive an advance email from DOL encouraging them to participate in the web survey. Two weeks following the advance email, Westat will email an invitation with study information that contains the web survey URL and a unique PIN that will enable the grantee to access the survey. The invitation will also provide a phone number and email address to the Westat survey support center to assist grantees with any questions they may have about the web survey.

Two weeks following the initial invitation email from Westat, grantees will start to receive weekly email reminders if they have not completed the web survey. Because DOL has stated that grantee participation in the national evaluation is mandatory, we expect a 100 percent response rate. Westat will call any grantees that do not respond after the final email reminder. If any grantees have not responded to calls, Westat will ask DOL to reach out to them to remind them to complete the survey.

Partner Survey

Figure B-2 shows the data collection approach for the partner survey. Similar to the grantee survey, the partner survey will use a web survey to collect the data. We expect a 60 percent response rate to the partner survey based on response rates for similar surveys conducted for the recent Health Profession Opportunity Grant National Implementation Evaluation (HPOG-NIE). HPOG-NIE conducted web surveys of partners, stakeholders, and employers and achieved response rates that ranged from 33 to 58 percent depending on the survey. We anticipate that the response rate will be on the higher end of this range owing to the focus on modertly and very involved partners.

The partner survey will include an advance letter from DOL explaining the study and the importance of their particpiation. Westat will send the initial email invitation to all partners with study information that contains the URL and a unique PIN for the web survey. Partners will start to receive weekly email reminders if they have not completed the web survey approximately two weeks after the invitation mailing. Westat will also send a reminder postcard to partners two weeks after the intial invitation email. All invitations and reminders will include the Westat survey support center hotline and email address. We will ask the grantees to encourage participation among their partners during data collection. If partners have not responded after three email reminders, Westat will conduct reminder telephone calls to partners, with the option to complete the survey over the telephone if they desire.

Figure B-1. Proposed data collection flow for the Grantee Survey

Figure B-2. Proposed data collection flow for the Partner Survey

B.3.2 Nonresponse Bias Analysis

Even though intensive methods will be used to increase response rates to the parter survey, non-response bias is still a concern. Survey nonresponse can bias the estimates if the outcomes of survey respondents and nonrespondents differ, or if the types of individuals who respond to the survey differ across the program and control groups. We will use several methods to assess the effects of survey nonresponse.

We will conduct a non-response bias study to determine the extent to which partner survey respondents and non-respondents differ, using data available from grantee applications, the grantee survey, and the partner contact information form on partner and grantee characteristics that may be related to partner non-response. We will compare the responding and non-responding partners on the following characteristics: type of grantee organization, type of partner organization, and level of involvement of the partner.

If the non-response bias study indicates bias, then we will use the “propensity score method” for nonresponse weight adjustment. Its strength is its ability to use as many auxiliary variables as available, and to use interaction terms in the model. The goal of nonresponse weight adjustment is to use auxiliary information available for the population that is predictive of nonresponse behavior and adjust the analysis weights to take account of the differential nonresponse propensities.

We do anticipate a great deal of missing data in the grantee or partner surveys. The surveys will be pretested and item nonresponse will be monitored in the initial stages of data collection. Respondents with missing data on an item will be excluded from the analysis of that item.

B.4 Test of Procedures or Methods to be Undertaken

The grantee and partner surveys were pretested with fewer than 9 respondents each. The grantee survey was pretested with 3 current grantees. The partner survey was pretested with 6 partners of the grantees included in the pretest. Each respondent completed the survey on paper and the evaluation team conducted cognitive interviews over the telephone with respondents to determine question clarity, ease of responding, and time to complete the surveys. The surveys were revised based on feedback from grantees and partners. The partner contact information form was also pretested with the 3 grantees to ensure clarity of instructions and determine timinig. When the survey is fielded to all grantees, to minimize burden, those who participated in the pretest will be asked to review the answers they provided during the pretest to ensure that they are still accurate and up-to-date. Partners who participated in the pretest will not be contacted again to complete the survey. The final instruments are include in this OMB clearance package.

B.5 Individuals Consulted on Statistical Methods and Individuals Responsible for Collecting and/or Analyzing the Data

B.5.1 Individuals Consulted on Statistical Methods

There are no statistical aspects of the design and thus, no one was consulted on statistical aspects of the design.

B.5.2 Individuals Responsible for Collecting/Analyzing the Data

Westat

Dr. Joseph Gasper (240) 314-2470

Mr. Wayne Hintze (301) 517-4022

MDRC

Dr. Richard Hendra (212) 340-8623

Ms. Kelsey Schaberg (212) 340-7581

1