STC Employer Survey Supporting Statement PtB final 4 3 2014

STC OMB Pt. B Page 12 of 12

Employers Survey of Short-Time Compensation Program

OMB SUPPORTING STATEMENT PRA PART B

Collection of information involving statistical methods

This information collection request is for an employer survey conducted as part of the Employer Survey of Short-Time Compensation (STC) Program study conducted for the U.S. Department of Labor (DOL). This project will study employers’ experiences, awareness, and perspectives of the STC program in four purposely selected states.

1. Respondent Universe and Sampling

The sampling universe is all employers who are subject to unemployment insurance (UI) laws in the four selected states. The time frame for an employer’s inclusion in the sampling frame will be from 2009 through 2011. The total sample size for this study across all 4 states over the 3 years comprises nearly 4,000 employers. The sampling unit is unique employers in each state.¹

1. State Selection

Currently, 25 states and the District of Columbia have STC provisions in their UI laws.² The 17 states that enacted STC laws prior to 2010 have a long history with administering the STC Program. IMPAQ, in collaboration with DOL, selected 4 of the 17 states with enacted STC laws to participate in this study. These four states--Kansas, Minnesota, Rhode Island and Washington --provide a rich cross section of recent STC experiences and offer a wide variation of STC use by employers.³ For example, Washington currently has roughly 1,400 active STC plans, while Minnesota has roughly 65 active STC plans.⁴

State selection was based on each state’s:

• STC program characteristics.

• Geographic diversity.

• Experience working with employers to develop STC plans.

• Data system capacity to help IMPAQ identify employers using STC.

Exhibit 1 presents the information that is available in each of the four states. We will use these data to identify employers who applied to participate in STC.

Exhibit 1: State Agency Data Systems and Employer Characteristics

Employer Information	Study State
	Kansas	Washington	Minnesota	Rhode Island
Employer identification number				
Business name				
Mailing address				
Physical location address if different than mailing address	*	**		N/A
Employer representation/contact	*	**		
Telephone number				
Email address				**

Notes: Information captured during telephone interviews with state workforce agency personnel in October 2012. All elements reflect data from each state workforce agency’s (SWA) integrated data systems unless otherwise indicated.

*Spreadsheet.

**Tax Files

In addition to having the necessary data systems to identify appropriate employers, these states also vary in their experience with STC, statutory language, rules, state agency practices and use. Exhibit 2 presents the variation in statutory requirements. Kansas and Rhode Island, for example, do not require employers to maintain health insurance for workers while an employee is receiving STC; Washington and Minnesota do. In addition, Minnesota does not require, by law or practice, employee concurrence in lieu of union agreement to the STC plan; the other three states do.

Exhibit 2: Selected Current STC Plan Approval Requirements

Selected Plan Requirements	Study State
	Kansas	Washington	Minnesota	Rhode Island
Maintenance of health insurance	No	Yes*	Yes**	No***
Maintenance of pension contributions	No	Yes*	Yes**	No***
Union concurrence/sign-off	Yes*	Yes*	Yes**	Yes*
Employee concurrence in lieu of union	Yes*	Yes*	No	Yes****

Note: Information captured during telephone interviews with SWA personnel during October 2012. Information reflects state practices at the time of the interview.

*State statute

**State agency practice only

***Plan must indicate by state statute how this is treated

****Statutory notification only

Employers’ tax rates increase or decrease depending on employers’ individual experience rating with the UI program. As indicated in Exhibit 3, the four states vary in their unemployment rates and in their type of UI tax system. Minnesota experienced the lowest annual average unemployment rate of the four states in 2011 (6.4 percent) while Rhode Island experienced the highest unemployment rate (11.3 percent).⁵ Minnesota and Washington are Benefit Ratio states while Kansas and Rhode Island are Reserve Ratio states. The four states are geographically dispersed: one in the Northeast region, two in the West North Central region, and one in the Pacific region.

Exhibit 3: Characteristics of Selected STC States’ UI Tax Systems

Selected Plan Requirements	Study State
	Kansas	Minnesota	Rhode Island	Washington
2011 unemployment rate	6.70%	6.40%	11.30%	9.20%
Type of UI system	Reserve Ratio	Benefit Ratio	Reserve Ratio	Benefit Ratio
Region	West North Central	West North Central	Northeast	Pacific
Year STC program enacted	1988	1994	1991	1983

Source: BLS Regional and State Unemployment 2011 Annual Averages report, CRS Report Compensated Work Sharing Arrangements as an Alternative to Layoffs, September 2011, and Comparison of State UI Laws report 2011.

In sum, the four states vary in geographic region, characteristics of their UI tax systems, STC Program application rates, and STC plan approval requirements. In addition, the states’ data systems capture the relevant information needed to draw the employer sample.

b. Employer Selection

To obtain meaningful answers to the primary research questions, we need to identify and survey both STC and non-STC employers. Exhibit 4 illustrates the different employers groups that will receive the survey. As indicated in the exhibit, some employers apply to participate in STC while others do not. Among employers who apply for the program, some receive approval while a small number of others may not.⁶ Among those who do not apply, some employers inquire about the state’s STC program while others do not. The four states that we have selected have adequate UI administrative data to identify employers who apply and do not apply for STC and those employers that have approved and disapproved plans. We will survey employers in each of the four groups presented in Exhibit 4.

Exhibit 4: Employer Selection Process

A relatively small number of employers have approved STC plans (Group 1). Minnesota has roughly 65 currently active plans; Kansas has roughly 130 active plans, of which around 50 percent of the employers are multi-plan employers; Rhode Island has roughly 200 active plans with roughly 80 percent of these being multi-plan employers; and Washington has roughly 1,400 currently active plans.⁷ Given the relatively small number of employers in Groups 1, 2, and 3, we plan to survey all available employers in these groups.

Employers that do not inquire about the STC program (Group 4) will be randomly selected to reflect the distribution of industries of employers in the state with STC approved plans. Stratifying by industry will help balance the sample and prevent over representation by employers who may not be suitable for using STC. Suitable strata will be determined once we have obtained the STC administrative data and can identify the prevalent industries and characteristics of employers with an approved STC plan (Group 1). No more than 10 strata will be used. The states will be providing employer NAICS codes at the 5 digit level. We will use the two digit NAICS when performing the stratification. We expect manufacturing, real estate and retail to be the most prevalent industries using the STC program. IMPAQ expects to have received all of the states administrative data by the end of second quarter 2014.

Exhibit 5 summarizes the sampling strategy in each state for the four employer groups identified above. We will attempt to survey nearly 4,000 employers. An expected response rate of 80 percent will mean 3,200 unique employers completed the survey.⁸ While the survey is in the field IMPAQ will monitor the response rate daily. In the event the response rate appears to be drastically lower than 80 percent, IMPAQ will explore options for achieving a higher response rate. These options may include either extending the period that the survey is in the field, performing additional outreach to employers that have not responded or expanding the sampling frame. Any such decision will be made in consultation with USDOL and the federal project officer.

Exhibit 5: Sampling Strategy by State

Employer Group		Study State
		Kansas	Minnesota	Rhode Island	Washington
Group 1	STC plan approved	Census	Census	Census	Census
Group 2	STC plan not approved	Census	Census	Census	Census
Group 3	Inquired about STC	Census	Census	Census	Census
Group 4	Did not inquire about STC	Stratified Random Sample	Stratified Random Sample	Stratified Random Sample	Stratified Random Sample

IMPAQ will select the interview sample from the universe of employers who are subject to paying UI taxes (subject employers) in each of the four states during the reference years. The average number of subject employers between 2009 and 2011 in Kansas was 70,600; in Minnesota, the average was 132,300; in Rhode Island, the average was 33,000; and in Washington, the average was 208,000.

Exhibit 6 presents estimates of the number of employers in each of the employer groups. For groups 1 through 3, we will interview all the employers in the group. We will then select a sample of employers in group 4 equal to the total number of employers in groups 1 through 3. For example, in Kansas, based on information received, we estimate that there will be 158 unique employers with approved STC plans from the years 2009 through 2011. Furthermore, we estimate that there will be 61 unique employers who submitted STC plans which were not approved, and 61 unique employers that inquired about STC. Combining the total number of employers in groups 1 through 3, we will survey a total of 280 randomly selected employers that did not apply or inquire about STC.

The sample size estimates in the other three states required additional assumptions about the number of likely employers. For group 1, we used the number of approved plans reported in 2011.⁹ To develop the sample population size for groups 2 and 3, we used the proportions found in Kansas.¹⁰ The resulting total sample available for the survey is expected to be 3,980. Approximately one-half of the sample will be in groups 1 through 3 and the other half in group 4.

Exhibit 6: Estimated Number of Employers to be Surveyed

Employer Group		Study State				Total
		Kansas	Minnesota	Rhode Island	Washington
Group 1	STC plan approved	158	65	200	700	1,123
Group 2	STC plan not approved	61	25	77	270	433
Group 3	Inquired about STC	61	25	77	270	433
Group 4	Did not inquire about STC	280	115	354	1,241	1,990
Total		560	230	708	2,481	3,979

c. Time-Period for Sample Selection

As illustrated in Exhibit 7, STC use peaked in calendar year 2009 with 288,618 STC first payments.¹¹ This was roughly two percent of all UI first payments made during 2009. IMPAQ’s conversations with STC managers at SWAs confirmed that STC use peaked in 2009 in the four states and continued at historically high rates through 2011. As a result, to generate a sufficient sample size, the time period for inclusion in the sampling frame will be calendar years 2009, 2010 and 2011.

Exhibit 7: STC Use from 1995 through 2011

Source: ETA UI AW 5159 and AR 5159.

2. Procedures for Collecting Information

The objective of this study is to understand employers’ awareness and perceptions of the STC Program, including how various elements of the program affect their interest and participation. The study will provide information on how these factors vary with employer characteristics, recent trends in STC benefits paid to employees (including variation across states and how such variation relates to state STC provisions), and how the Federal STC legislation may impact existing state-level laws related to STC programs.

To achieve this objective, we will administer an employer survey using a Web-based data collection survey tool. The Web-based survey will be supplemented by a telephone follow-up survey of non-respondents. If an employer does not complete the Web-based survey, IMPAQ’s telephone survey center will follow up to solicit the employer’s responses.

a. Statistical Methodology, Estimation, and Degree of Accuracy

IMPAQ will conduct a census of employers in Group 1 (STC plan approved), Group 2 (STC plan not approved), and Group 3 (did not apply but inquired about STC). We will employ a stratified sampling plan to populate Group 4 (did not apply and did not inquire about STC). We will stratify on industry because employers in certain industries are more likely to apply for STC. For stratification, we will use STC administrative data to identify the prevalent industries associated with STC. We will then use UI administrative records to identify employers in these industries that did not inquire or apply to participate in STC. These employers will then be grouped into strata by industry using the NAICS 2012 classifications. We will use no more than 10 strata. Employers within each stratum will be selected with equal probability.

We will account for the stratified sampling design in our analyses using Stata’s survey procedures. These operations enable the user to define the survey design for the dataset, thereby taking into account the specific characteristics of the survey data. Failure to analyze the data without considering the sampling characteristics can result in inaccurate point estimates and/or inaccurate estimates of standard errors. Specifically, we will use Stata’s “svyset” command to define the primary sampling unit and identify the strata and weight variables. After defining these characteristics, we will use the “svy:” prefix for all subsequent commands and calculations. This enables Stata to perform the computations while taking into account the survey design. Conveniently, Stata stores these characteristics with the dataset, eliminating the need to re-define the sampling design with each dataset use.

Estimation Procedures. Employer survey data from participating and non-participating employers will enable us to examine the factors related to the employer participation decision, a key focus of this study. Our analyses will include descriptive statistics (tabulations, means, standard deviations, etc.) of:

• All available employer characteristics (industry, number of employees, years in business, annual revenue, etc.).

• STC participation rates and STC Program use (the number of quarters in which firms used STC).

• Reasons for participating in the STC Program (retain skilled workers, improve employee morale, and avoid future recruitment and training costs).

• Reasons for not participating (lack of awareness of the program, the administrative burden of participation, higher UI charges, or production technologies that limit the potential for reducing working hours).

• Employer and employee interest (as reported by employers) in participating.

In addition to descriptive analyses, we will use multivariate regression models to produce a more detailed characterization of the relationships between employer characteristics and participation in the STC Program. We will estimate the following logistic regression model using pooled employer survey data:

The dependent variable (Y) is equal to 1 if the employer participates in the STC program and zero otherwise. Control variables include:

• , which includes all available employer characteristics in the survey data (e.g., industry, size, annual revenue) and a constant term.

• , includes identifiers for each of the four states.

• , a mean zero disturbance term.

We will repeat this analysis for each state and use t-tests to determine if the estimated coefficient associated with each employer characteristic is statistically significant. The results of these models will allow us to assess whether significant differences exist in the likelihood of participation across employer characteristics and the direction of those differences. For example, we can assess whether larger firms are less likely to participate in STC than other employers, controlling for other employer characteristics.

b. Unusual Problems Requiring Specialized Sampling Procedures.

There are no unusual problems requiring specialized sampling procedures.

3. Methods to Maximize Response Rates and Data Reliability

a. Response Rates

We define response rate as the proportion of completed and partially completed Web and phone interviews to the number of eligible employers identified from the UI administrative data in four states. The response rate is crucial for this project as a high response rate ensures study external validity by accurately inferring the characteristic of STC participating firms and the extent to which the STC Program is adopted by employers in the sample. Additionally, the pattern of response rate among employers may reveal varied interests in the STC Program and the relationship between employer characteristics and such interest.

The IMPAQ staff has accumulated extensive experience in survey research and developed a profile of strategies to achieve high response rates. These strategies--essentially efforts to decrease the workload the survey demands from survey respondents--increase the perceived reward of completing the survey. For example, IMPAQ staff members pay a great amount of attention to the user-friendliness of the Web-based questionnaire and the phone interview protocols, enabling respondents to quickly comprehend the questions, make accurate responses, and proceed smoothly to the end of the survey. Specifically, IMPAQ learned from our experience that online survey respondents prefer a clear description of computer actions at point of use, a scrolling rather than a screen-to-screen format, progress bars, and the option to skip questions. In addition to incorporating these design features in our online questionnaire, we have also carefully crafted each question to make it easily comprehensible and concise.

Another strategy to increase response rate is to motivate respondents by increasing the perceived reward of survey completion. For example, we will email a motivational cover letter to all employers in the sample that will introduce the study’s Government sponsor and explain the purpose of the survey as well as the potential applications of the survey findings. The cover letter not only increases respondents’ trust but also shows the relevance of the survey content to employers. Logistically, the letter serves as the first point of contact to verify contact information before survey delivery and to pre-notify employers of the online survey availability. We will send the cover letter email approximately 1 week before the questionnaire becomes available online.

For online survey non-respondents, we will follow up with three waves of phone interviews. We will recruit and extensively train experienced interviewers on data collection procedures, including methods for promoting cooperation among sample members. Interviewers especially skilled at encouraging cooperation will be available to persuade reluctant respondents to participate and will be assigned to attempt conversions with respondents who initially refuse (except for hostile refusals). We will translate the survey into Spanish and use bilingual interviewers to conduct surveys in Spanish if necessary. Call scheduling in CATI will allow respondents to select the time most convenient for them to be surveyed

b. Addressing Bias due to Non-Contact and Non-Response

While we expect a relatively high response rate, we will investigate our data to determine if there is any non-response bias. Specifically, we will conduct a non-response bias analysis to assess whether the survey sample is representative of employers in the four states. Two kinds of bias may arise from missing responses in the surveys. “Non-contact bias” results from the failure to locate the respondent despite repeated email and telephone calls. This may be for various reasons such as the employer has gone out of business, moved, or restructured. “Non-response bias” results from the respondent’s failure to answer a particular question, either because the respondent refused to answer, or the respondent did not know the answer.

We will address non-contact bias by comparing characteristics of employers we cannot reach and that of employers we do reach. We will use UI administrative data, not subject to non-response, to examine differences in perceptions of STC. A consistent pattern between employer characteristics and the chance of failure to locate suggests a skewed sample, i.e., employers with certain characteristics tend to be more difficult to reach and, therefore, less represented in the analytic sample. Results of such bias analyses will help us understand the study sample and the population to which we can generalize the study findings. Study findings cannot and will not be generalized to all states with STC programs. The non-response bias analysis will be used to see how generalizable the results are to the 4 study states.

In addition to comparing characteristics of respondents and non-respondents, we will adopt appropriate methods to adjust for non-respondent bias based on the specific nature of missing survey responses. For example, if a response is not missing at random and certain background characteristics fully explain the difference in response rate, we will use re-weighting methods that assign larger weights to underrepresented respondents. Specifically, we will use a logistic regression model of the probability that a sample member responded to the survey to estimate the probability of survey response to construct appropriate weights for each respondent. The initial logistic regression model will be estimated using all available employers’ background characteristics. The model can be expressed by the following equation:

The dependent variable in this model (S) is the likelihood that the participant responds to the survey. S equals 1 if the participant responds to the survey and 0 otherwise. The initial model includes all available participant characteristics (X) in the UI administrative data, such as company size and industry. Inclusion of all available background characteristics in the model is important for identifying which variables are strong predictors of non-response. To identify the final set of covariates for the non-response model (i.e., the variables that are strong predictors of non-response), we will use the following measures of predictive ability and goodness of fit:

• McFadden’s Pseudo R-squared statistic – This measure captures the percent of the variation in the likelihood of responding to the survey that is explained by participant characteristics. This statistic is constructed as follows:¹²

• Akaike’s Information Criterion (Akaike 1974) – This statistic measures the efficiency of the model in predicting the outcome based on the number of covariates. This statistic is constructed as follows:¹³

We will use these statistics to assess which set of observable characteristics are strong predictors of survey non-response. After identifying this set of characteristics, we will estimate the final version of the model using these characteristics. After estimating this model, we will use the results to calculate the predicted likelihood of survey response based on each participant’s characteristics, as follows:

In words, the survey non-response weight for each participant is the inverse of the predicted probability of response, . We use these weights in the analyses to make the sample representative of all employers in the four states. This method is a widely accepted practice in program evaluations for controlling for survey non-response and for making estimation results representative of all program applicants.¹⁴

We will also conduct sensitivity analyses using alternative methods to address non-response bias, such as case wise deletion and dummy adjustment. The former involves deleting the entire survey response record if dependent variables have missing data. The latter requires creating a dummy variable to indicate the missing status (0 if not missing data; 1 if missing data) when independent variables in the analytic model have missing data. We will then incorporate the dummy missing indicator and the independent variables in the analytic model as predictors.

c. Reliability of Data Collection

The survey question design ensures that the respondents can easily understand them. Revisions were made to the draft questionnaire based on an internal review, a review by technical advisors, and a review by DOL.

The use of Computer Assisted Telephone Interviewing (CATI) and Computer Assisted Web Interviewing (CAWI) to conduct the survey also helps ensure data reliability. Both systems can control question branching (reducing item nonresponse due to interviewer error), as well as modifying wording (providing memory aids and probes and personalizing questions), and constructing complex sequences that are not possible to produce or are less accurate in other survey administration systems. The probes, verifications, and consistency checks are built into the system, enabling the CATI/CAWI system to standardize procedures. These procedures ensure the reliability of the data collection methods and the data collected through those methods.

Lastly, IMPAQ will monitor each interviewer’s work using silent call-monitoring equipment and video monitors that display the interviewer’s screen in real-time for supervisors.

4. Tests of Procedures or Methods

IMPAQ conducted a pretest of the survey instrument using printed versions of the questionnaire. Each pretest was monitored to determine if the respondent’s answers were consistent with the intent of the question. The pretest enabled IMPAQ to check the accuracy of the internal skip patterns of the survey, the instructions to responders in the Web-based instrument, instructions to interviewers for the phone-based instrument, the ease with which interviewers can interpret question wording, and the apparent ability of respondents to follow the wording and sequence of questions. Project staff monitored all pretests and took extensive notes on question wording, skip logic, and the overall flow of the instrument.

5. Individuals Consulted on Statistical Methods

Exhibit 8 lists the persons outside of the Employment and Training Administration who contributed to, reviewed, and/or approved the design, instrumentation, and sampling plan:

Exhibit 8: Consulted Individuals

Name	Affiliation	Telephone Number
Jacob Benus (Project Director)	IMPAQ International	(443) 367-0088
David Balducchi (Principal Investigator)	IMPAQ International	(703) 276-8494
Stephen Wandner (Senior Technical Advisor)	Urban Institute	(301) 785-6670
Russell Saltz (Research Associate)	IMPAQ International	(443) 367-0088

1 An employer can submit more than one STC application.

2 Prior to 2009, 17 states had STC language in their UI laws. These states were Arizona, Arkansas, California, Connecticut, Florida, Iowa, Kansas, Maryland, Massachusetts, Minnesota, Missouri, New York, Oregon, Rhode Island, Texas, Vermont, and Washington. In 2010, Maine and Pennsylvania enacted programs. In 2011, Colorado, District of Columbia, New Hampshire, and Oklahoma adopted STC. In 2012, New Jersey and Michigan enacted STC bills. This number does not include Louisiana, which enacted an STC law in 1986, but does not operate a program.

3 We have received informal acknowledgement that these four states desire to participate in this study, and we are currently seeking formal agreements with the state to participate. In the event that one or more states do not formally agree to participate in the study, we will solicit participation from an alternative and comparable state.

4 Kansas currently has roughly 130 active STC plans; Minnesota has more than 60 active STC plans; Rhode Island has more than 200 active STC employers; and Washington has roughly 1,400 active STC plans. However, these figures do not represent the number of unique employers with approved STC plans. Employers can have more than one active STC plan at a time (i.e. multi-plans). For example, roughly 50 percent of Kansas’ plans are multi-plans and 80 percent or more of Rhode Island’s plans are multi-plans. These states had significantly more STC plans in 2009 than they do today. In addition, these are current numbers. The survey sampling frame will include the years 2009 through 2011.

5 The U.S. annual average unemployment rate in 2011 was 8.9 percent.

6 An employer’s STC plan will not be accepted if it fails to meet state law requirements that may include whether the employer is delinquent in paying contributions, unable to commit to the program, or is either in negative balance or a maximum rated tax status. These state law requirements and reasons for non-acceptance vary by state.

7 The number of approved plans and number of unique employers using STC will be different because employers can have multiple approved plans at the same time. In addition, employers can have multiple plans across time. The number of unique employers in the sampling frame will be less than the number of approved/denied plans. Information on the number of plans is based on phone interviews with SWA individuals.

8 IMPAQ expects 80 percent survey response rate because employers who participate in (or known about) STC are more likely to have working relationships with the state UI agencies, or because of the premium their companies place on human capital and job preservation programs. Similar studies, such as the USDOL Evaluation of Individual Training Account Demonstration and the Green Jobs Impact Evaluation, have achieved an 80 percent or higher response rate.

9 In Washington, we used 50% of approved 2011 plans due to the very large number of approved plans in Washington. These employers will be randomly selected with equal probability.

10 Sample size estimates were developed using the following proportions: (1) the rate of STC plans not approved in each state is equal to 39% of the approved plans; (2) group 3 was set equal to group 2; and (3) group 4 was set equal to the sum of groups 1, 2, and 3.

11 Current federal reporting requirements only yield data on STC claimants, not on participating STC employers.

12 Note: is the full model prediction using all control variables; M_Full is the full model prediction without predictors; and M_Intercept is the estimated likelihood of response based on the logit model.

13 Note: k is the number of parameters in the estimated model; and is the maximum value of the likelihood function based on the estimated model.

14 Source: McConnell et al., 2006; Trenholm et al., 2007; Benus et al., 2009)