201504-1291-002_Part_B_10-02_clean_final

Supporting Statement for the

Paperwork Reduction Act of 1995

Part B. Collections of

Information Employing Statistical Methods

Demonstration and Evaluation of the Short-Time Compensation Program

October 2, 2015

U.S. Department of Labor

200 Constitution Ave., NW

Washington, DC 20210

PART B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

The U.S. Department of Labor (DOL) is requesting clearance for information collections to conduct (1) in-depth interviews with state agency officials and employers, and (2) surveys of employers. These data collections are essential elements of the implementation study and the rigorous impact evaluation of the demonstration of the Short-Time Compensation (STC) program.

1. Respondent Universes and Sampling Methods

   1. Respondent universes for the employer surveys

There will be two surveys of employers in each of Iowa and Oregon. One survey will use a questionnaire containing three questions to collect data from a sample of the employers in each state that are eligible to participate in the STC program. This survey will be referred to as the Short-Form Survey. The other survey, the Long-Form Survey, will use a questionnaire containing either 33 (for Oregon) or 34 (for Iowa) questions to collect data from employers that requested information from their respective state Unemployment Insurance (UI) agency about developing an STC plan. The questionnaire for the long-form survey includes the three questions present on the short-form questionnaire.

The two employer surveys in each state are part of a larger study (demonstration and evaluation of the STC program), funded by the Department of Labor. In addition to the employer surveys, this larger study contains randomized experiments in each state that are currently ongoing and precede the fielding of the proposed employer surveys and the post-intervention implementation study interviews. In Iowa, the preceding experiment is a randomized controlled trial (RCT), in which all STC-eligible employers that did not have STC plans established immediately prior to the beginning of the RCT were randomly assigned to either a treatment or a control group, and only employers assigned to the treatment group receive specific interventions to inform them about STC and promote the use of STC. In Oregon, the preceding experiment consists of an RCT in the Portland area and a quasi-experimental design (QED) in the remainder of the state. For the QED, employers in a particular set of counties outside the Portland area were assigned to a treatment region and employers in the remainder of the counties outside the Portland area were assigned to a comparison region. We plan to use the assignments of employers in the Iowa and Oregon STC experiments to define the respondent universes and strata for the proposed employer surveys.

Employers eligible for the STC experiments were those in business immediately prior to the start of the experiment and likely to meet STC plan requirements, but without current STC plans. In Iowa, all employers covered by UI that have five or more employees can establish STC plans. The number of such Iowa employers was 28,692, and 34 of these employers had an STC plan sometime during the period from October 2012 through September 2013. Hence, Iowa had 28,658 employers eligible to participate in Iowa’s RCT experiment.

In Oregon, all employers covered by UI who have three or more covered employees can establish STC plans, and 182 of these employers established STC plans sometime during the period from October 2012 through September 2013. For Oregon, an RCT involving 24,661 employers is being conducted in the Portland area. Outside of the Portland metro region, the balance of the state was divided into a treatment region (containing 15,059 eligible employers) and a comparison region (containing 15.059 eligible employers) for conducting the QED.

The combined universe for the long- and short-form employer surveys will be all employers that were eligible for the STC experiments, plus employers that were not eligible because they had an STC plan in effect immediately prior to the start of the STC experiments (meaning that all STC-eligible employers in the state are included). Employers that are no longer in business in each state will be ineligible. The universe for the long-form survey in each state will consist of the employers that had an STC plan in effect immediately prior to the start of the STC experiments plus other STC-eligible employers who requested information from their respective state UI agency about developing an STC plan either before or during the STC experiments. In Iowa, we expect the long-form survey universe to contain no more than 300 employers (between 2008 and 2015). In Oregon, we expect the long-form universe to contain about 1,300 employers (between 2010 and 2015). The universe for the short-form survey in each state will be employers in the combined universe that are not in the long-form survey universe. The short-form-survey universes will contain approximately 28,400 employers in Iowa and approximately 52,800 employers in Oregon.

2. Strata for the employer surveys

Because of the expected small sizes of the long-form universes, we plan to invite all employers in each state’s long-form universe to participate in the long-form survey. For collecting short-form data, we plan to select a stratified sample from each state’s short-form universe. The Iowa short-form universe will contain two major strata: one for employers assigned to the treatment group of the RCT and the other for employers assigned to the RCT’s control group. The Oregon short-form universe will contain four major strata: treatment and control strata for the RCT and treatment and comparison strata for the QED. The purpose of the major strata is to create analysis domains for comparing the short-form responses by employers assigned to the experiments’ treatment groups with those from employers assigned to the control groups. The experiments used blocking variables—including NAICS sector, workforce region, and employer’s annual Unemployment Insurance benefit charges—to randomly assign employers to the treatment and control groups, so the distributions of these variables are very similar within the RCT major strata in each state. The two QED major strata in Oregon provide geographical stratification. Within the major strata in each state we plan to create five substrata based on each employer’s number of employees. The number of employers in each of the five substrata within a major stratum will be approximately the same. Table B-1 indicates the approximate number of employers in the major strata of the short-form universe for each state. Proportional allocation will then be used to determine the sample sizes for the strata.

Table B-1. Approximate number of employers by short-form survey major strata

State	Major Stratum		Approximate number of employers
	Experiment	Assignment
Iowa	RCT	Treatment	14,200
	RCT	Control	14,200
	Total		28,400
Oregon	RCT	Treatment	12,000
	RCT	Control	12,100
	QED	Treatment	14,600
	QED	Comparison	14,100
	Total		52,800

3. Sampling unit, response rates, target sample sizes, and fielded sample sizes for the employer surveys

Sampling units will be individual UI-employer account numbers. Multi-establishment employers with only one UI account number will be treated as only one sampling unit.

For both the short-form and long-form surveys, the primary mode of data collection is online. Employers (the director of human resources) will first receive a mailed invitation with instructions for accessing the online survey. Reminder letters will be mailed, and for the short-form employers only, the second and third reminder letters will include a paper survey. Employers that do not respond online or by mail will be contacted by telephone to complete the survey by telephone. As described more fully in section 4.3, a test of survey mode will be conducted on the short-form survey, with 70 percent of the short-form employers invited initially to complete the survey online, and 30 percent invited initially to complete the survey on paper.

The literature on establishment surveys indicates that national survey response rates vary, depending on whether the survey is mandated or voluntary, with voluntary surveys in the range of about 30 to 40 percent, on average, whereas mandatory surveys can reach as high as 80 or 90 percent. At the state level, similar results are found for voluntary surveys. For example, Pickreign and Whitmore (2012) report response rates between 35 and 40 percent between 2004 and 2010 for the California Employer Health Benefits Survey.¹

For the short-form survey, we are assuming that the online/mail response rate will be 40 percent and that the response rate for the telephone follow-up will be 65 percent. Thus, for the short-form survey, the assumed weighted response rate is 0.40 + (1‑0.40)*0.65 = 79 percent. For the long-form survey, we are assuming that the online/mail response rate will be 35 percent and that the telephone follow-up response rate will be 30 percent. Thus, for the long-form survey, the assumed weighted response rate is 0.35 + (1‑0.35)*0.30 = 54.5 percent.

To determine the fielded sample size for the short-form survey, it is necessary to consider not only its assumed response rate but also any expected losses in precision due to adjustments for nonresponse that increase the variability of the analysis weights. We assume that for the short-form survey the increase in variability of the weights caused by nonresponse adjustments will require that the actual sample size of completed cases be 20 percent larger than the effective sample size, which is the size of a simple random sample that has the same precision as a given stratified sample. Precision calculations are based on the effective sample size, but burden calculations are based on the actual sample size.

All employers in the long-form universe will be invited to participate in the long-form survey. For the short-form survey, large enough samples of employers will be selected as shown in Table B-2, which contains proposed target effective sample sizes, actual sample sizes, and fielded sample sizes for the short-form survey.

Table B-2. Target effective sample sizes, actual sample sizes, and fielded sample sizes by major short-form survey strata

State	Major Stratum		Target actual sample size	Target effective sample size	Fielded sample size
	Experiment	Assignment
Iowa	RCT	Treatment	500	417	633
	RCT	Control	500	417	633
	Total		1,000	833	1,266
Oregon	RCT	Treatment	225	188	285
	RCT	Control	225	188	285
	QED	Treatment	275	229	348
	QED	Comparison	275	229	348
	Total		1,000	833	1,266

1.4 Respondent universes and sampling methods for the in-depth interviews.

In-depth interviews will be conducted with state agency officials and employers. The universe for the interviews of state agency officials is ten in Iowa and 20 in Oregon. All of the personnel involved in the demonstration and recommended by each of the state agencies will be interviewed for the implementation study. The contractor and states have been working together to implement the demonstration and identify relevant personnel and stakeholders to interview.

In Iowa, the universe for the in-depth interviews of employers is the set of employers who were assigned to the treatment group of the RCT. In Oregon, the universe for the in-depth interviews of employers is the set of employers who were assigned to the treatment group of the RCT or to the treatment region of the QED. Purposive sampling will be used to select a sample from each in-depth employer interview universe.

The contractor will select employers that are part of the treatment sample for the employer survey, including a mix of those that do and do not become STC users during the demonstration period in order to compare how they experienced or reacted to the intervention and their respective reasons for participating or not. The universe of treatment employers who became STC users during the treatment period is expected to be about 22 in Iowa and 88 in Oregon. Purposive selection will be used to capture variation across industry sectors, firm size, urban/rural location, and prior usage of STC program under different UI taxing requirements.² Non-users will also be purposively selected and to the extent possible, matched with the user sample.

Selecting the small purposive sample from the treatment sample of the employer survey will allow for comparison of answers from the qualitative interviews and the quantitative survey data

and thus, inform interpretation of the survey results and the impact data.

2. Statistical Methods

2.1. Statistical methodology for stratification and sample selection for the employer surveys

As noted in the response to the previous question, all of the employers in the long-form universe will be invited to participate in the long-form survey. The long-form survey will be administered to the universe of employers who requested information about the STC program, including all employers that participated in an STC plan. For the short-form survey, which contains only three questions, the primary analysis goals will be to estimate population proportions for the possible response options and to compare sub-population proportions for the possible response options among the assignment groups for the STC experiments.

In Iowa, the treatment and control groups for the RCT are equal in size. Consequently, in the two major strata associated with Iowa’s RCT assignment groups, equal size samples will be selected for collecting short-form employer data. The sample selected in each stratum will be an equal probability sample. To increase slightly the precision of resulting estimates, the employers in each stratum will be sorted by the employer’s number of employees and systematic sampling with a random start will be used. The use of systematic sampling will yield a sample in each stratum of the associated universe that will be more representative than would result from the use of simple random sampling.

In Oregon, there are more employers in the treatment and comparison QED regions than in the treatment and control groups for the Portland Area RCT. In the RCT, the treatment and control groups are equal in size; whereas in the QED, the treatment and comparison regions are approximately equal in size. The major stratum sample sizes for Oregon’s short-form survey (see Table 2) are nearly proportionately allocated, with an adjustment so that the RCT-treatment-plus-QED-treatment sample size is equal to the RCT-control-plus-QED-comparison sample size. Within each Oregon stratum, employers will be sampled for short-form data collection in the manner described above for Iowa.

2.2. Estimation procedures, variance estimation, and degree of accuracy needed for the employer surveys

Employer survey data, supplemented with UI and STC administrative data will be used to determine if the interventions had a statistically significant impact on program awareness for treatment employers. Employer survey data and STC program data will also be used to identify which interventions and delivery mechanisms are the most effective. Employer survey data will capture the administrative burden on employers by asking for the number of hours of employer staff devoted to specific administrative tasks (e.g., application; set up plan; set up and processing claims). The survey will also provide data on the benefits that employers may derive from participating in STC, such as surviving a business downturn and retaining skilled employees.

For both the short- and long-form surveys, analysis weights will be computed so that the sum of the weights for the respondents adds up to the total number of employers in each universe. (Section 3.3 describes the procedure that will be used to calculate the analysis weights.) Moreover, within each short-form survey stratum the sum of the weights for the respondents will add up to the total number of employers in the stratum. In addition to satisfying these additivity conditions, the analysis weights will reflect differences in stratum sampling rates and will adjust for different response rates by demographic categories, defined by variables—such as industry, number of employees, and headquarters location—present on the files used to assign employers to the STC experiments.

Both full-sample weights and replicate weights will be developed. The full sample weights will be used to compute weighted totals, means, and proportions for the universes and for the subpopulations defined by the STC-experiment assignments and by variables on the files used to assign employers to the STC experiments. Because the three questions on the short-form questionnaire survey are also on the long-form questionnaire, it will be possible to compute estimates for these three questions for the combined universe and for various subpopulations of the combined universe. The replicate weights will be used to compute associated variance estimates.

Several approaches will be used to handle missing data. The use of analysis weights to compute weighted estimates will handle missing data due to employer-level nonresponse. For employers that answer some but not all of survey items, the employer’s data will be reviewed and if pre-specified key items (e.g., awareness, how learned of program) are not answered or if a particular number of items are not answered the employer’s data will be treated the same as employer-level nonresponse. If the employer’s data are determined to be usable, then a missing response to a particular item will be treated as a separate response category.

A major analysis goal is to compare the average responses to the three short-form questions between the set of treatment employers and the set of control/comparison employers. For example, the first question is about whether the respondent is aware of the state STC program. Say that 60 percent of the respondents in the control group and 75 percent of the respondents in the treatment group report that they are aware of the STC program. The employer survey’s data users will want to know if this difference is statistically significant (using a two-sample T test) or if this difference likely is due solely to the sampling variability present in these two estimates.

Table B-3 shows the minimum-detectable effect (MDE) for comparing treatment and control group estimated proportions using a one-tailed 95-percent confidence test, assuming the number of completed questionnaires will be 500 for each group, and the design effect due to unequal sampling rates is 1.2.³ (Hence, the assumed effective sample size for each group in Iowa is 500/1.2=417.)⁴ Since the MDE is 8.2 percent when the control group estimate is 60 percent, it follows that the difference between the treatment-group estimate of 75 percent and the control group estimate of 60 percent is statistically significant. (The MDEs in Table B-3 were computed in Excel using equation 12 in Lachin (1981).⁵)

Table B-3. Minimum detectable effects (MDEs) for comparing proportions estimated for treatment and control groups. Power is 80 percent for a one-tailed 95-percent-confidence test.

Proportion estimated for control group (%)	MDE (%)	Smallest detectable proportion for treatment group (control + MDE) (%)
20	7.3	27.3
40	8.5	48.5
50	8.6	58.6
60	8.2	68.2
70	7.6	77.6
85	5.6	90.6

The design for the STC study, involving both the STC experiments and the employer surveys has been reviewed by an advisory committee and by the Department of Labor. In particular, both the advisory committee and the Department of Labor have concluded that the MDEs in Table 3 are sufficient to support policy making activities.

The long-form survey is intended to provide descriptive information about the subset of employers who had contact with the state agency about the STC program, particularly those employers with an STC plan, not to support comparisons between the treatment and the control groups. So, the analysis will be descriptive, based on weighted totals, means, and proportions on key variables of interest (ever contacted state agency about STC program; ever applied for STC; had an approved STC plan; used STC; administrative burden; and benefits of the program).

2.3. Problems requiring special sampling procedures

None.

2.4. Periodic data collection to reduce burden

This request is for a one time data collection.

3. Methods for Maximizing Response Rates and Addressing Nonresponse

3.1 Handling non-response to the in-depth interviews

The interviews with state agency staff and employers will be conducted approximately 4-6 months after the end of the 12-month demonstration periods in each state, or about February 2016 for Iowa and March 2016 for Oregon. The response rate for the state agency staff interviews is expected to be 100 percent. If anyone is unexpectedly not available during the in-person site visit, the interview will be rescheduled and conducted by telephone. Based on the pretest, we expect an 80 percent response rate for employer interviews. Given that purposive sampling will be used, substitution of alternatives with similar characteristics is appropriate and will be employed. A letter of invitation will be mailed on state agency letterhead to the employer’s “Director of Human Resources,” the person most likely to be aware of the state STC program. The letter will encourage participation in the interviews and will be followed by a telephone call to schedule the interview. A minimum of three calls will be made.

3.2 Handling non-response to the employer surveys

To maximize response rates to the employer surveys, two separate employer instruments will be used to cover two different kinds of employers: those who ever inquired about the state STC program and all other STC-eligible employers.

The short-form survey consists of only three questions pertaining to awareness of the STC program and will be addressed to a random sample of the treatment and control employers of the STC demonstration project. This survey is expected to take only one to two minutes to complete. The long-form survey consists of 33 (for Oregon) or 34 (for Iowa) questions on awareness, participation, administration, and firm characteristics and is expected to take 12 minutes to complete. The long-form survey is addressed to the universe of employers who contacted the state agency about the state STC program (between 2008 and September 2015 in Iowa and between 2010 and September 2015 in Oregon). Most of these employers have established a relationship with the state STC agency and are expected to be willing to respond to the survey because of that relationship.

These surveys will be conducted approximately 16-18 months after the launch of the 12-month demonstration (i.e., 4 to 6 months following the end of the interventions). The field period is expected to be twelve weeks in duration to provide sufficient time to conduct follow-up on non-respondents.

Dillman, Smyth and Christian (2009)⁶ describe the tailored design method for surveys, noting the importance of developing a set of survey procedures that interact and work together to encourage all people in the sample to respond to the survey. These procedures focus on contact letters, the questionnaire and follow-up contacts. Following this method, a letter of invitation will be mailed on state agency letterhead to the employer’s “Director of Human Resources,” the person most likely to be aware of the state STC program. The letter will encourage participation in the survey and provide information to access the survey online for 70 percent of employers; the other 30 percent will receive the letter of invitation with a paper survey and business reply envelope (see section 4.3 about test of survey mode). As described below, there are several reminder letters and telephone calls planned to increase the response rates. Both short-form and long-form surveys will include a telephone follow-up with the remaining non-respondents.

Short-form survey process. The selected employers will receive a letter inviting them to participate in the survey. The letter received by 70 percent of the sample will include instructions and an access code for logging into the online survey. After two weeks, a reminder letter will be sent that again provides the instructions and access code for logging into the survey. After another two weeks, a third letter with a paper version of the short-form survey will be mailed. For the remaining 30 percent of the sample, the two letters containing the paper survey will be sent before two reminder letters that provide instructions and an access code for logging into the online survey. One week after the fourth mailing, an email reminder (with instructions and an embedded link to the online survey); will be sent to those Iowa employers for whom an email address is available (about 30% of Iowa sample); email addresses are not available for Oregon employers. About the same time, telephone follow-up with remaining non-respondents will begin, offering the option to complete the survey by telephone.

Long-form survey process. The universe of employers who contacted the agency about STC will receive a letter inviting them to participate in the survey. The letter will include instructions and an access code for logging into the online survey. Reminder letters (that include the instructions and access code for logging into the survey) will be sent every two weeks for a total of two reminders. After one more week, an email reminder (with instructions and an embedded link to the online survey) will be sent to Iowa employers for whom an email address is available. A week later, telephone follow-up with remaining non-respondents will begin, including the option to complete the survey by telephone.

The contractor will maintain a survey management system to track survey response and non-response as well as the extent to which surveys were “not deliverable.” The system will be used to count the number of days since the launch of the survey to trigger necessary follow-ups.

The contractor will provide telephone follow-up on all employers that did not complete the survey within the first eight weeks of the survey. These non-respondents will be encouraged to complete the survey by telephone. Up to four weeks of telephone follow-up will be conducted using experienced telephone interviewers. The interviewers will be trained on the survey instruments and use Computer Assisted Telephone Interviewing (CATI). When speaking with respondents, interviewers will remind employers about the purpose of the survey and the importance of their participation.

We expect to achieve a 40 percent response rate to the initial online-plus-mail data collection for the short-form survey because of its very short length (three questions), administering it online, and conducting extensive follow-up activities by mail and telephone. In addition, the initial mailing to introduce the survey will be on state agency letterhead from the UI agency director and addressed to the Director of Human Resources. Pretest employers said that the Director of Human Resources is the best person to receive the survey and that the invitation coming from the state agency will be opened and read.

3.3 Weighting procedures for the employer surveys

Analysis weights will be calculated for completed employer surveys to allow for unbiased estimates of population and sub-population proportions. The analysis weights will be the product of a base weight and a post-stratification adjustment. The base weight is the reciprocal of the probability of selection for each employer. Thus, for employers in the long-form universe, the base weight is equal to 1. For employers in the short-form universe, the base weights will be the reciprocals of the stratum sampling rates.

The post-stratification adjustments are designed to reduce the potential bias caused by differences between responding and non-responding employers. These adjustments modify the base weights so that they aggregate to sub-group totals computed from the files used to assign employers to the STC experiments. The post-stratification adjustments have the effect of reducing variance.

3.4 Non-response bias analysis for employer surveys

When unadjusted analysis weights are used to estimate the population mean for some item, the non-response bias present in the resulting estimator is equal to the product of the nonresponse rate and the difference between the average value of the item for respondents and the average value of the item for nonrespondents. Hence nonresponse rates are one measure of potential nonresponse bias. After the completion of data collection and the calculation of the analysis weights, base weighted response will be calculated for each employer survey and for domains defined by administrative data variables on the files that were used to assign employers to the STC experiments. Possible domains include industry type, size categories based on number of employees, assignment to treatment versus control, and geographical domains. AAPOR Response RTE Formula RR3 will be used to calculate the response rates.

To estimate the difference between the average value of an item for respondents and the average value of the item for nonrespondents, it is necessary to have data for the item for all the units in the sample, not just the responding units. Thus, the administrative data present on the files used to assign employers to the STC experiments, along with the base weights will be used to estimate the difference in the population means of respondents and nonrespondents for available data variables. This will allow the estimation of non-response biases present in estimated means calculated with the base weights. Another way to estimate such biases is compute population means from all the administrative data for all employers on the files used to assign employers to the STC experiments and then subtract these values from weighted means calculated from the base weights and the administrative data associated with respondents. The reductions in nonresponse bias resulting from the post-stratification adjustments to the base weights will be determined by using the adjusted weights to repeat the estimation of the nonresponse bias present in the administrative data variables.

4. Tests of Procedures or Methods

4.1. Pretesting and cognitive interviewing for the employer survey

A pretest (with cognitive interview) of the long-form version of the employer survey was conducted using a paper version of the instrument in Iowa in early December, 2014. Five employers participated, two who had prior use, two who are current users, and one who expressed interest in but did not sign up for STC. The survey consisted of 35 questions. A senior researcher from Westat administered the survey and conducted the cognitive interview at each employer’s office. Survey response times ranged from 8 minutes to 12 minutes, or an average of 10 minutes.

Included with the instrument was an advance letter intended to be mailed 6 weeks in advance of the survey. The intent of the letter was to introduce the survey, encourage participation, and ask for the contact information for an individual at the firm who should receive the information needed to access the online survey. Pretest participants provided comments on the letter and the instrument. Based on the results of the pretest and interviews, the letter and instrument were revised. For example, the cover letter will include be addressed directly to Director of Human Resources because that is the person most likely to know about the STC program. As a result, the letter will not ask for a contact person to be identified. The survey was revised to provide a “range of values” rather than asking for a precise number on some items (reducing burden) and to include a “not applicable” response for employers that used STC before the state’s temporary not charging period began.

After the pretest, the contractor recommended that two surveys be used rather than one, to better focus survey content to specific kinds of respondents (those who ever inquired about the state STC program within a specific time period and all others), and possibly improving response rates. Two surveys would facilitate a sampling process that better meets the needs of the evaluation to determine the impact of the demonstration by using a stratified random sample for the survey that measures employer awareness (the short-form survey) and use of the universe of employers who inquired with the state UI office about the STC program (long-form survey). The long-form survey is the revised version of the pretested instrument.

The short-form survey includes only the first three questions from the long-form survey because a large portion of the pretested survey would not apply to most potential respondents, (those who never inquired about the program). By reducing the size of the instrument, employers that did not inquire about STC are expected to be more cooperative to answer the few questions that are critical to the evaluation about employer awareness. The long-form survey is relevant to the small number of employers who inquired about the state’s STC program.

Determining the extent of employer awareness and how employers learned about the program is considered critical information for the evaluation. It requires a representative sample of the universe of eligible employers. In contrast, the number of employers that inquired about the program is a very small relative to the number of eligible employers. So, a survey of the universe of these employers is feasible to learn about their experiences with the STC program.

4.2 Pretesting of the in-depth interviews of state agency staff and employers

The interviews guides of state officials and employers were pretested in December 2014 in Iowa. Five or fewer respondents were asked the same questions. The breakdown of types of positions and questions for state officials was: Leadership (n=2); Management (2); Assorted Technical (4). The interviews with leadership and management lasted 45 to 60 minutes and those with assorted technical officials (claims, benefits, and communication) lasted 15 to 30 minutes. Five employers who previously participated in the STC program were interviewed during the pretest period. The employers were purposively selected for heterogeneity across several characteristics, including firm size, industry, rural or urban location, and usage of STC (including treatment and control employers and one current STC user).

The interview guides were revised based on the experience of the interviewers during the pretest. For the post-demonstration interviews, the three separate guides for leadership, management, and technical interviews have been merged into two (leadership/management and technical).

4.3 Experiment of online first versus mail first

There is little empirical evidence as to the best mode for conducting a very short survey of employers. As part of the information collection described above, we plan to test two different data collection approaches for conducting the short-form survey by comparing an online survey approach and a mail survey approach. One reason for testing these two approaches is that reliable email addresses for the sample are not available to send a direct link, so a letter will need to be mailed. Second, an online survey may not be a more efficient and effective means of reaching employers. The short-form survey can fit on one sheet of paper and be completed in one to two minutes when received, then placed in a business reply envelope and mailed back. In contrast, with the online survey, the employer has to go to the web, type in the URL, enter username and passcode, and then answer the questions. It is possible that mailing a paper survey could produce a higher response rate than sending instructions for participating in an online survey.

For the web-first approach, 70 percent of short-form survey employers will receive a letter of invitation by mail to participate in an online survey, with letter follow-ups (where the third and fourth reminder letters include a paper survey). Under the mail survey approach, 30 percent of short-form survey employers will receive the invitation letter with a paper survey (and no mention of online survey), followed by a reminder letter with a paper survey, then reminder letters with invitation and instructions to complete the survey online (without a paper survey included). The results of the two approaches will be compared on their respective response rates at two-week intervals.

The results of the experiment of “online first” versus “mail first” will be used to provide the U.S. Department of Labor advice for conducting future short surveys of employers. The expectation is that employers are more likely to complete an online survey than a mail survey. If there is a statistically significant difference in the rate of response of employers, then DOL will have evidence for future short surveys of employers as to the method that provides a higher rate of response (when employer email addresses are not available).

The results can also impact the conduct of the short-form survey. If the response rate after the first two weeks for “online first” is significantly higher than for “mail first,” then the offer of an online-option will be provided by the third week of the survey rather than waiting to the fifth week (as planned). Alternatively, if the response rate for “mail first” is significantly greater, then the mail option will be offered earlier than the planned seventh week of the survey (i.e., included in the reminder letter). If there is no significant difference in response rates, no changes will be made to survey procedures, and overall response rates of online and mail will be compared at the conclusion of the study

5. Individuals Contacted on Statistical Aspects and/or Analyzing Data

The following individuals (see Table B-4) consulted on statistical aspects of the design and will also be primarily responsible for actually collecting and analyzing the data for the agency.

Table B-4. Persons and/or entities involved in data collection and/or analysis of data

Name	Agency/Company/Organization	Telephone Number
Katharine G. Abraham	Director, Maryland Center for Economics and Policy Professor of Economics and Survey Methodology	301-405-3489
Frank J. Bennici	Senior Study Director, Westat	301-738-3608
George Ghanem	Research Associate, Social Dynamics, LLC	301-990-1105
Susan N. Houseman	Senior Economist Upjohn Institute for Employment Research	269-385-0434
Susan N. Labin	Vice President, Social Dynamics, LLC	301-990-1105
Chris O’Leary	Senior Economist Upjohn Institute for Employment Research	269-385-0407
Richard S. Sigman	Senior Statistician, Westat	240-453-2783

1 Pickreign and Whitmore, Evaluating the Effectiveness of Two Strategies to Improve Telephone Survey Response Rates of Employers (2012) downloaded from www.amstat.org/meetings/ices/2012/papers/301963.pdf.

2 The taxing requirements changed as a result of the 2012 authorizing legislation. Also, see the U.S. Government Accountability Office’s 2012 Designing Evaluations (GAO-12-208G) for the importance of selecting “to represent particular conditions of interest” in qualitative studies (p. 34).

3The design effect of 1.2 is an estimate of the loss of precision due to variability of weights due to non-response adjustment (i.e., a 20 percent increase in variation).

4 Summing across treatments in the RCT and QED yields an effective sample size of 417; similarly for summing across control and comparisons.

5 Lachin, J. M. (1981). Introduction to Sample Size Determination and Power Analysis for Clinical Trials, Controlled Clinical Trials, 2, 93-118.

6 Dillman, Smyth, and Christian (2009), Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method. 3^rd ed., John Wiley & Sons, Inc. Hobokun, New Jersey.