Revised -- Final GATE II Part A 5 8 2012

Supporting Statement, Telephone Evaluation, Project GATE II

A. Justification

This information collection request is for the follow-up survey to be conducted as part of the Growing America Through Entrepreneurship II (GATE II) demonstration and evaluation project. This project will evaluate three U.S. Department of Labor (DOL) grants to provide microenterprise services to older and rural Workforce Investment Act (WIA) dislocated workers.

1. Circumstances Necessitating the Data Collection

In June 2008, the U.S. Department of Labor (DOL) awarded grants to states for implementing self-employment training programs modeled after the successful Project GATE (Growing America Through Entrepreneurship) demonstration. These grants, known as GATE II, were awarded to four states: Minnesota, Alabama, North Carolina, and Virginia. Two of the four grantees, Alabama and North Carolina, are targeting services to dislocated workers in rural areas, while Minnesota and Virginia are targeting services to older dislocated workers. In awarding the four GATE II grants, DOL sought to assess the effectiveness of self-employment training programs for rural dislocated workers and for older dislocated workers. These worker groups face unique challenges when seeking reemployment, thus self-employment training may offer an opportunity for reemployment through self-employment.

While four state grants were awarded under GATE II, one of the states (Minnesota) was unable to implement the required random assignment design. As a result, Minnesota will not be included in the proposed survey or the impact evaluation.

There are many similarities between the implementation of the original Project GATE (see Benus et al., 2009) and GATE II. There are, however, also a number of important differences. Key features of GATE II:

• Eligibility is restricted to Workforce Investment Act (WIA) eligible dislocated workers.

• In North Carolina and Alabama, GATE II provides self-employment training to dislocated workers in rural areas.

• In Virginia, GATE II provides self-employment training services to older dislocated workers. The minimum age threshold for program participation is 45 years.

• In North Carolina and Virginia, individuals are randomly assigned to the treatment or to the control group. In North Carolina, the random assignment probabilities are 75 percent treatment and 25 percent control; in Virginia the random assignment probabilities are 50 percent treatment and 50 percent control. Only those assigned to the treatment group are offered GATE II services.

• In Alabama, plant closings, not individual workers, were originally randomly assigned. To improve program enrollment, Alabama has adopted an individual random assignment model similar to North Carolina and Virginia.

An evaluation of the GATE II grants is necessary for policymakers and program developers to determine whether the model can be successfully implemented for dislocated workers who are interested in pursuing self-employment. A follow-up surveys is the only way to collect information on self-employment experiences, receipt of microenterprise services, self-employment earnings, and household income.

a. The GATE II Demonstration

A major outreach effort has been launched in each state to recruit participants for GATE II. This effort included placing brochures, flyers, and posters at DOL’s One-Stop Career Centers and at community-based organizations throughout the program’s target area. The program also advertised in local newspapers and on the Internet. Individuals were able to register for the program by mail, by telephone, or via the Internet.

Everyone who registers for the program receives an invitation to attend a GATE II orientation session. At the orientation session, a trained counselor uses a state-customized video to introduce the GATE II program. The video provides attendees with a description of the demonstration’s evaluation design (including random assignment), a realistic description of the challenges of self-employment, and a description of GATE II and its services. An important purpose of the orientation is to provide enough information for individuals to make an informed decision about whether or not to pursue self-employment.

In order to be eligible for GATE II, the applicant must be a U.S. citizen (or lawfully able to work in the US) and must have a proposed business that is legal and appropriate for support by DOL. Individuals already self-employed but interested in developing their business further are also eligible for the program.

Self-employment services provided by GATE II include:

• Assessment. A trained business development counselor conducts assessments one-on-one. At this assessment, the trained counselor determines the service needs of the participant and makes a referral to existing self-employment providers for further GATE II services.

• Structured Training Course. The training course consist of a series of classes on topics such as developing the business idea, writing a business plan, marketing, accounting, legal issues, cash flow, financing, and other topics.

• One-on-One Counseling. The participant meets with a trained counselor for technical assistance. As part of the technical assistance, counselors assist individuals in completing their business plans. For those in need of financing for their businesses, the counselors will also provide technical assistance in applying for loans.

b. The Evaluation

The GATE II grants will be evaluated using an experimental design in Alabama, North Carolina, and Virginia. In these three states, individuals who meet the eligibility criteria (rural or older dislocated worker, a valid business idea and permitted to work in the U.S.) are randomly assigned to either a program or control group. Members of the program group are eligible to receive GATE II services; members of the control group are not eligible to receive GATE II services. Members of the control group, however, are eligible to receive all other self-employment services currently available in the community.

The evaluation will address the following key questions:

(1) What are the impacts of GATE II on participants’ labor market and self-employment outcomes?

(2) Does the program increase the use of self-employment services?

(3) Does the program lead to an increase in the completion of business plans and applications and receipt of loans?

(4) Does the program increase the likelihood of self-employment?

(5) Does the program promote employment and other aspects of economic development?

(6) Does the program increase employment, earnings, and satisfaction with employment and reduce the receipt of UI and public assistance?

(7) Is GATE II effective in rural areas and for older workers?

(8) Does the effectiveness of the program vary by population subgroup?

Addressing these questions will enable us to assess whether GATE II was an effective policy to assist dislocated workers interested in self-employment improve their labor market outcomes in Alabama, North Carolina, and Virginia. The results of the study will provide insights on the potential impacts of self-employment training for dislocated workers who are interested in pursuing self-employment. However, since the GATE II states have been chosen purposively, combined with the fact that the study has small sample sizes, GATE II impacts cannot be generalized to a wider population with a known degree of statistical precision.

c. Data Collection

Data for the evaluation will be collected from four sources: application form, a survey, UI administrative records, and site visits. Application form data were collected by the state. Specifically, each GATE II applicant was required to provide information on their socioeconomic characteristics (gender, race, age, education, etc.) and labor market outcomes (tenure with prior employer, prior wages, industry, occupation, etc.) at the time of application.

As discussed more fully under A6 (page 8), the survey is needed because it is the best source of data for important outcomes, such as self-employment experiences, receipt of microenterprise services, and household income.

The survey will be administered by telephone to all sample members approximately 18 months after random assignment. The survey will be used mainly to collect information about the sample members’ experiences since random assignment, including employment history and receipt of microenterprise services during the 18 months following random assignment. The survey will also include questions on receipt of self-employment services, experience in starting a business, and wage and salary employment during the time period following random assignment. In addition, the survey will include questions about experiences with self-employment and wage and salary employment prior to random assignment. The survey instrument is provided in Appendix A.

For the impact evaluation, we will also obtain Unemployment Insurance (UI) administrative data from the four participating states. The UI administrative data contains information on receipt of UI benefits, quarterly earnings from wage and salary jobs, and other employment information. We will also use data collected during the site visits to inform the impact analysis.

2. How, By Whom, and For What Purpose the Information is to be Used

The survey data will be used to measure outcomes for members of the program and control groups in the five broad areas described below and listed in Table 1.

• Receipt of Self-Employment Services from GATE II and Other Providers. Training, technical assistance, and other self-employment services that are not funded by GATE II are available in all the sites to both the program and control group members. An important outcome is the extent to which GATE II increases the receipt of different types of services and the intensity and quality of the services received.

• Completion of Business Plans and Loan Applications. GATE II is designed to assist participants in completing formal written business plans and loan applications. GATE II may increase the number of business plans. It may also increase the number of completed and successful loan applications.

• Business Development. By providing training and technical assistance, GATE II aims to increase the success of business development, create employment, and promote economic development. GATE II may increase the number of businesses started and increase the size and success of these businesses. Businesses may be larger in terms of sales, profits, number of employees, and payroll. They may provide more benefits to their employees.

• Employment. By assisting people to start their own business, GATE II may increase employment of sample members, increase their earnings, and their satisfaction with their employment. By increasing self-employment, GATE II may decrease other types of employment. Hence, it is important to measure both types of employment – self-employment and employment working for other people. Project GATE II may also affect the employment of sample members’ spouses. The direction of the effect on spouses’ employment is uncertain, however. Spouses may work more because of the uncertain income of self-employment, or less if the business is successful and household income increases or the sample member has less time for child care and other household activities.

• Household Income and Receipt of Public Assistance. By changing the employment outcomes of participants, GATE II may change household income and the degree to which the participant is self-sufficient. Self-sufficiency will be measured by the receipt of UI, welfare benefits, and other forms of public assistance.

TABLE 1

OUTCOMES FOR THE GATE II EVALUATION

Receipt of Self-Employment Services

Receipt of structured training courses (duration and whether complete)

Receipt of one-on-one technical assistance (amount)

Participation in peer support groups (amount)

Receipt of mentoring services (amount)

Receipt of other services (amount)

Payment for services

Ways in which services assisted business development

Satisfaction with services

Business Plans/Application for Loans

Whether completed business plan

Loan applications (how many places applied and types of loans)

Business Development

Financing of business

Success in obtaining loans

When business started

Whether business still exists at follow-up

Income produced by business

Sales, expenses, profits of business

Type of business

Number of jobs created by business (whether for family or others)

Payroll of business, fringe benefits offered

Whether business is located in an economically distressed area

Employment

Time spent in self-employment

Time spent working for someone else

Industry/occupation of job

Earnings

Hours worked

Receipt of fringe benefits

Satisfaction with employment

Spouse’s employment

Household Income and Receipt of Public Assistance

Household income

Availability of health benefits

Receipt of UI, Trade Readjustment Allowance, and Trade Adjustment Assistance

Receipt of food stamps, cash assistance (e.g. TANF), SSI, Veterans’ payments, and Social Security

The survey will also include questions on the barriers to starting a business. Information on such barriers may point to ways that GATE II services can be improved. Policymakers will be able to use the results of the impact evaluation to assess whether GATE II is effective and whether it should be replicated on a larger scale with rural and older dislocated workers. The findings will also be useful to other microenterprise training providers in developing their programs.

The outcome data collected from sample members will be used together with data collected from the service providers during site visits to the four sites. The information collected during site visits will be used in a process analysis which can provide important contextual information on the effectiveness of the GATE model in different environments. For example, the process analysis will help us to assess whether the program is particularly effective in certain types of rural areas and in certain environments. The process analysis will also help to provide information about whether GATE would be effective if replicated in other sites.

3. Use of Improved Technology to Reduce Burden

Computer Assisted Telephone Interviewing (CATI) will be used to conduct the follow-up survey. CATI was selected because telephone surveys are more cost-effective and impose a lower burden on respondents than in-person surveys. CATI is more cost effective than paper and pencil interviewing for many reasons, including the fact that CATI programs accept only valid responses and can be programmed to check for logical consistency across answers. Interviewers are thus able to correct errors during the survey, eliminating the need to call respondents a second time to obtain missing data. Also, calls will be made through an auto-dialer, linked to the CATI system, virtually eliminating dialing error. In addition, the automated call scheduler will simplify scheduling and rescheduling of calls to respondents at their convenience and can assign cases to specific interviewers, for example, those who work in the evening and/or those who are fluent in Spanish.

4. Efforts to Avoid Duplication

The survey will collect key information about sample members. No other survey data collection effort has been conducted or has been planned to collect similar information.

The study will also use administrative records data where possible. Specifically two kinds of administrative data will be used in the impact evaluation:

• UI Benefits Data: UI agency administrative records on UI eligibility and benefit receipt will be collected from the GATE II states in the study and used in the analysis. Questions concerning UI benefit receipt will not be asked on the survey.

• Wage Records: Quarterly wage records will be collected from the GATE II states to obtain summary information on employment and earnings by quarter. As the wage records exclude self-employment earnings and earnings from some other jobs, the survey also includes questions about employment and earnings. Additional detail on employment such as industry, occupation, hours worked, the hourly wage, and fringe benefits not available from wage records will be collected on the survey.

While these administrative data are an important supplementary data source for the impact evaluation, they are not sufficient to conduct the impact evaluation. Survey data are the most important data source for the impact evaluation of GATE II.

5. Methods to Minimize Burden on Small Businesses or Entities

Some sample members will become self-employed and establish small businesses. Since self-employment is the major outcome of interest, these individuals will be asked questions about their businesses. Only sample members will be asked questions about their business; the extent of the questions will be limited. We expect that the questions about small businesses will add approximately 10 minutes to the survey for individuals who have self-employment experience. Surveys will not be conducted with non-sample businesses or entities.

6. Consequences of Not Collecting the Data

The survey will provide the primary source for data for sample members on the following outcomes:

• Self-employment training and services,

• Completion of business plans and application for loans,

• Self-employment experiences and earnings,

• Employment working for someone else, and

• Income and receipt of public assistance.

Therefore, if the survey were not conducted, the evaluation would be unable to assess the impacts of GATE II services on these outcomes.

7. Special Data Collection Circumstances

In all respects, the data will be collected in a manner consistent with federal guidelines. The statistical survey will produce valid and reliable results that can be generalized to the universe of study, and it will include only statistical data classifications that have been reviewed and approved by OMB. It will include a pledge of confidentiality that is supported by authority established in statute or regulation and by disclosure and data security policies that are consistent with the pledge. It will not unnecessarily impede sharing of data with other agencies for compatible confidential use.

8. Federal Register Notice

The Federal Register notice soliciting public comments was published on May 5, 2011 (vol. 76, p 25723). No comments were received.

9. Respondent Payments

To enhance the response rate, we propose to offer $15 incentive payment to respondents. The strategy of providing compensation for participation draws on an extensive literature documenting its importance in achieving high levels of cooperation with surveys and IMPAQ’s previous experience in surveying displaced workers.

There is a vast literature providing evidence that compensation can increase the response rates to surveys and lower the cost of data collection without compromising the quality of the data. Evidence of increased response rates has been found in several research areas such as health (Halpern et al. 2011, Griffin et al. 2011, Kristin et al. 2009), company-sponsored surveys (Rose et al. 2008), veteran affairs (Coughlin et al. 2011), program participation (Martin et al. 2001), among others. Also, monetary incentives have been shown to increase participation across survey modes including mail surveys (Church 1993, Trusell and Larrakas, 2004), telephone surveys (Brick et al. 2006, Singer et al. 2000, Gelman et al. 2003) and online surveys (Balajti et al. 2010). Despite the variation in context and target populations, these studies show that incentives are a cost-effective mechanism to increase response rates. We take advantage of this evidence and design an incentive mechanism tailored to the specificities of our project.

In addition, IMPAQ has a longstanding experience implementing surveys of similar content and length as the GATE II-IMPACT EVALUATION survey and among very similar target populations (i.e. dislocated workers participating in a USDOL reemployment program) in which participant incentives were both offered and not offered. For example, IMPAQ conducted surveys between 2004 and 2009 of participants associated with an earlier evaluation of GATE I. We obtained an 80% response rate using $15 participant incentives and 2 hours of interviewer time per completed survey. In contrast, for a survey of participants associated with an evaluation of the Reemployment and Eligibility Study, which did not utilize a participant incentive, IMPAQ obtained response rates less than or equal to 60%.  Both surveys were directed to the treatment and control groups of dislocated workers participating in a USDOL reemployment program evaluation and addressed outcomes related to reemployment, future wages and utilization of UI benefits.

Building on this experience, we propose an incentive scheme offering $15 payments to respondents. Although offering an incentive to complete the survey incurs the additional cost of $15 per completed survey, it is also expected to reduce the amount of time required to obtain that completed survey and thus produce a saving to the government. IMPAQ believes that offering respondents $15 incentive payment will yield the government savings of $15,840. The approach to this estimate follows. By offering a $15 incentive to our sample, we expect to achieve an 80% response rate (i.e., 1980 sample * 80% response rate = 1,584 completed interviews). Furthermore, with this $15 incentive, we estimate the average time to complete an interview will be 2 hours. Assuming an average cost of $25 per interviewer hour, the total cost of interviewer time is $79,200 (1,584*$25/hour*2 hours). In addition, the cost of incentives is $23,760 (1584*$15). Thus, the resulting total cost to achieve an 80% response rate with incentives is $102,960.

As previously mentioned, in the absence of a $15 incentive, a conservative expectation of the response rate is 60%(i.e., 1,188 completed interviews). To reach the targeted 80% response rate we would need a total of 1,584 completed interviews (or an additional 396 respondents). We believe that completing the required 1,584 interviews without the incentive, will require a substantial increase in interviewer time. Based on our experience, respondents will not be as cooperative and it will take additional telephone call attempts and increased time for refusal conversion. We estimate that, without incentives, the average interviewer time needed to complete the required 1,584 interviews will be 3 hours. As such, the cost of interviewer time will be $118,800 (1,584*$25/hour*3 hours). Thus, without incentive payments, the total cost of completing the interviews will be $118,800.

In summary, the total cost of the proposed survey with $15 incentive payments is $102,960 and the total cost without incentives is $118,800. Thus, offering respondents $15 incentive payment yields expected savings of $15,840.

10.  Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.

All individuals interviewed will be informed that information gathered will not be attributable directly to the respondent and will only be discussed among members of the evaluation team.  Terms of the DOL contract authorizing data collection require the contractor to maintain the privacy of all information collected, unless written permission is provided by the program applicant or participant.  Accordingly, individual privacy will be protected to the fullest extent permitted by law.

a.    Protection of Personal Information

Telephone surveys will be conducted by IMPAQ staff at IMPAQ’s call center in Columbia, MD.  It is IMPAQ policy to efficiently protect all information and data, in whatever media they exist, in accordance with applicable Federal and state laws and contractual requirements.  All program participants will receive unique identification codes which will be stored separately from personally identifying information.  Researchers from IMPAQ who play a role in data collection and analysis will be trained in proper procedures for data handling and will be prepared to describe these procedures in full detail, and to answer any related questions raised by survey respondents.  Access to all data that identify respondents will be limited to IMPAQ staff who have a data collection or analysis role in the project, unless written permission is provided by the survey respondent.  Such data will be needed for assembling records and assuring data alignment.  Any data sent to DOL will not contain personal identifiers or any other identifier that would allow individual identification of study participants, except as authorized in writing by the program applicant or participant.

In conjunction with IMPAQ’s data policy, all staff members are required to:

• Comply with a Confidentiality Pledge and Security Manual procedures to prevent the improper disclosure, use, or alteration of confidential information. Staff may be subjected to disciplinary and/or civil or criminal actions for knowingly and willfully allowing the improper disclosure or unauthorized use of information.

• Access information only on a need-to-know basis when necessary in the performance of assigned duties.

• Notify their supervisor, the Project Director, and the organizational Security Officer if information has either been disclosed to an unauthorized individual, used in an improper manner, or altered in an improper manner.

• Report immediately to both the Project Director and the organizational Security Officer all contacts and inquiries concerning information from unauthorized staff and non-research team personnel.

b.    Protection of Data

The security procedures implemented by IMPAQ cover all aspects of data handling for hard copy and electronic data.  All hardcopy materials will be shipped to the contractors using Federal Express or an equivalent system that allows for package tracking; if any item is delayed or lost it will be investigated immediately.  All completed hardcopy documents will be stored in locked file cabinets or locked storage rooms when not in use.  Unless otherwise required by DOL, these documents will be destroyed when no longer needed in the performance of the project. 

c.     Background checks and security

Evaluation team members working with this data will have previously undergone background checks.  These may include filling out an SF-85 or SF85P form, for example, authorizing credit checks, and having fingerprints taken.

11. Questions of a Sensitive Nature

The survey of GATE II sample members contains a minimal set of items that may be considered sensitive in nature. These questions are related to the success of businesses (Section C in the questionnaire), receipt of individual and household income (Section D in the questionnaire), and public assistance receipt (Section E in the questionnaire).

All questions in the current survey, including those deemed potentially sensitive, have been pretested and used extensively in prior surveys with no evidence of harm. Questions about the success of the business are necessary to measure the effect of GATE II on economic development. Questions about income and public assistance receipt are necessary to measure the economic well-being of study participants.

12. Participant Hour Burden of the Collection of Information

The total participant hour burden for information collected for the follow-up survey is 792 hours as shown in the Table 3 below. This participant hour burden estimate is based on actual pretests of the survey, which averaged 25 minutes to complete; with five additional questions requested by DOL, it is estimated that each survey will take 30 minutes to complete.

Table 3: Minimum Detectable Effects (MDE) of the $15 Incentive

Cite/reference	Total Respondents	Frequency	Total Responses	Average Time per Respondent	Burden (hours)
GATE II Follow-up Survey	1,5841	Once	1,584	30 minutes	792
Totals			1,584		792

The total burden cost of collecting this information is $12,672. This cost represents 30 minutes to complete the survey multiplied by the number of completers (1,584) and by an estimated average hourly wage of $16 per hour.² This burden cost is offset by the $15 respondent payment.

13. Estimated Total Annual Cost Burden to Respondents and Record Keepers

There will be no start-up or ongoing financial costs incurred by respondents. There are no record keepers.

14. Estimated Annualized Cost to the Federal Government

The cost to the Federal government of conducting the survey is $608,648 (as shown in Table 4), which is the total contractor cost of conducting the survey.

Table 4: Survey Costs

Item	Cost
Research Staff Labor	$75,155
Call Center Staff Labor	$125,584
Fringe Benefits	$62,908
Overhead	$131,823
Total Labor Costs	$395,469
Subcontractors	$10,000
Total Other Direct Costs	$69,373
G&A	$88,320
Subcontractor Handling Fee	$400
Total Costs	$563,563
Fixed Fees	$45,085
TOTAL	$608,648

15. Changes in Burden

This is a new, one time data collection effort counting as 792 hours towards ETA’s Information Collection Budget.

16. Tabulations, Publication Plans and Project Schedule

a. Tabulations

The survey data will be used together with administrative data on UI receipt and quarterly earnings, and data collected via site visits to address the following three broad questions:

• What is the overall impact of GATE II on the receipt of self-employment services, completion of business plans, application for loans, business development, employment, household income, and receipt of UI and public assistance?

• Do the impacts of GATE II differ by economic or demographic characteristics of the sites, the service environment, or the way in which GATE II is implemented?

• Do the impacts vary for different subgroups of the population?

Estimating Overall Impacts. The analysis will begin with a comparison of the average outcomes of sample members in the program group with those in the control group. The randomized design ensures that, given large enough sample sizes, there will be no systematic observable or unobservable differences between program and control group members except for the acceptance into GATE II. However, it is possible that due entirely to chance, the program group may differ from the control group in some systematic way. To check for this occurrence, the two groups will be compared. T-statistics can be used to indicate statistical significance of these differences.

More precise estimates can be obtained by using regression methods, such as linear regression models, that control for any random differences in the baseline characteristics of program and control group members. Our regression models will use weights to adjust for non-response in the follow-up surveys (a detailed discussion of survey non-response weights is provided in Part B, Section 3a). The regression model can be expressed by the following equation:

The dependent variable in this model (Y) is the participant post-random outcome of interest (e.g., likelihood of starting a new business after program entry, likelihood of being self-employed at the time of the survey, and self-employment earnings). Control variables include:

• , which equals 1 if the participant was in the treatment group and 0 otherwise.

• X, which includes all available participant characteristics that may affect the outcome of interest (age, gender, race, education, prior employment, self-employment experience, etc.), as well as fixed effects for the program site.

• e, which is a zero mean disturbance term that captures the unobserved factors that affect the outcome of interest.

The parameter of interest in this model is α, the regression-adjusted treatment effect of GATE II on the outcome of interest. This parameter represents the intent-to-treat effect, that is, the impact of being assigned in the GATE II treatment group. The above model will be estimated separately for each outcome of interest. The statistical techniques used to estimate this model depend on the form of the dependent variable. For example, if the dependent variable is continuous (e.g., self-employment earnings and total earnings), then least squares techniques will be used to estimate the model. However, if the dependent variable is binary (e.g., likelihood of starting a new business and likelihood of self-employment), logit or probit maximum likelihood methods will be used to estimate the model.³

Once we estimate each model, we will use t-tests to assess whether estimated impacts are statistically significant. Specifically, we will implement the following steps:

• Estimate the variance of the treatment parameter (a) – The variance of the treatment effect is estimated as follows:⁴ where: is the survey non-response weight for each individual ; and SST= is the total sample variation in (Wooldridge, 2009).

• Calculate the t-statistic for the treatment parameter – Using the estimated treatment parameter (a) and its variance, we will calculate the t-statistic for the treatment parameter as follows: . This statistic will be used to assess if the treatment parameter is statistically significant at the 5 percent level.

Differences-in-means and estimates of the coefficient α in the regression described above will provide estimates of the impact of acceptance into GATE II. Because random assignment will occur after an orientation at which individuals are told about the program and the challenges of self-employment, we expect that a high proportion of those who are accepted will participate. However, some who are accepted may still decide not to participate in the program. Obtaining estimates of the impact of GATE II on those who actually receive GATE II services may be of policy interest. Assuming that GATE II has no impact on those who are accepted into the program but do not receive services, the impacts on those who receive services can be computed by dividing the impact estimates based on those accepted into GATE II by the proportion of program group members who participate in the program.

Estimating Impacts for Rural and Older Workers. The above regression model estimates the impact of GATE II, assuming that the impact of the program is identical for rural and older workers. To assess if the impact of the program is different between rural and older workers, we will modify the above model as follows:

The dependent variable in this model (Y) is the participant post-random outcome of interest and control variables include:

• , which equals 1 if the participant was a rural worker in the treatment group and 0 otherwise.

• , which equals 1 if the participant was an older worker in the treatment group and 0 otherwise.

• X, which includes all available participant characteristics and fixed effects for the program site.

• e, which is a zero mean disturbance term that captures the unobserved factors that affect the outcome of interest.

There are two parameters of interest in the above model: 1) α, which the GATE II impact for rural workers and 2) γ, which is the GATE II impact for older workers. Based on these parameters, the impact of GATE II for rural workers is equal to (α+δ), while the impact of GATE II for older workers is equal to (γ+δ). To assess if these impact estimates are statistically significant, we will use t-tests, as described above. Furthermore, we will use t-tests to assess if the program’s impact for rural workers is statistically different from the program’s impact for older workers.

Estimating Impacts by Site and Participant Characteristics. In addition to assessing program impacts for rural and older workers, it is of interest to assess whether the program had differential impacts by site and participant characteristics. Site characteristics of interest include: rural/urban status, availability of microloans, strength of local economy, and differences in GATE II implementation. Participant characteristics of interest include: gender, education, prior self-employment experience, and UI receipt. For example, to estimate if the program had differential impacts for men vs. women participants, we will estimate the following model:

The dependent variable in this model (Y) is the participant post-random outcome of interest and control variables include:

• , which equals 1 if the participant was in the treatment group and 0 otherwise.

• , which equals 1 if the participant was male and in the treatment group and 0 otherwise.

• X, which includes all available participant characteristics and fixed effects for the program site.

• e, which is a zero mean disturbance term that captures the unobserved factors that affect the outcome of interest.

There are two parameters of interest in the above model: 1) α, which the GATE II impact for female participants workers and 2) a+γ, which is the GATE II impact for male participants. To assess the statistical significance of a, we will use t-tests, as described above. To assess if the program’s impact for male participants (α+γ) is statistically significant, we will use an F-test:⁵

Based on this statistic, we will be able to assess if the program’s impact for male participants workers is statistically significant. Finally, to assess if the impact of the program was statistically different for male and female participants, we will use t-tests to examine the statistical significance of γ: if γ is statistically zero, then there is no gender differences in program impacts. Similar analysis can be conducted for other characteristics of interest. We should note, however, that sample sizes may not be large enough to reliably estimate impacts by all site and participant characteristics of interest. For instance, in the above example, small sample sizes may not allow us to determine whether γ is statistically significant. Furthermore, if impact analyses by gender are conducted separately for older and rural workers, small sample sizes may cause collinearity/confounding issues. At this point, it is difficult to assess the feasibility of conducting subgroup analysis based on sample limitations; however, once the survey data are collected, we will be able to determine the feasibility of these analyses.

b. Publication Plans

The final report on GATE II will be submitted to DOL in draft form in April 2012 and in final form in June 2012. The report will describe the results of the evaluation that will use data from the follow-up survey as well as UI administrative data and data collected as part of the process analysis. A public use dataset will be prepared from the survey data and the UI administrative data, along with a data dictionary and other documentation.

Throughout the data cleaning and analysis process, IMPAQ will maintain the highest security standards to ensure that personally identifiable information (PII) are protected from unauthorized access. IMPAQ has implemented extensive data security procedures and infrastructure to ensure data confidentiality.

c. Time Schedule

The project began in April 2010 and will end in June 2012. The design and survey instruments were prepared in Spring/Summer 2010. The demonstration began September 2008. The sample intake period will end either in June 2011 or when we have reached our sample goals, whichever is earlier. The follow-up survey will be conducted between August 2011 and January 2012.

17. Reasons for Not Displaying Expiration Date of OMB Approval

The expiration date will be displayed on the advance letter and on the hard copy version of the questionnaire.

18. Exceptions to the Certification Statement 19

There are no exceptions taken to item 19 of OMB Form 83-1.

References

Brick, J. Michael, Mary Collins Hagedorn, Jill Montaquila, Shelley Brock Roth, Chirsotpher Chapman. Impact of Monetary Incentives and Mailing Procedures: An Experiment in a Federally Sponsored Telephone Survey. National Center for Education Statistics, U.S. Department of Education. NCES 2006-066. March 2006.

Halpern, Scott D.; Kohn, Rachel; Dornbrand-Lo, Aaron; Metkus, Thomas; Asch, David A.; Volpp, Kevin G. (2011) “Lottery-Based versus Fixed Incentives to Increase Clinicians' Response to Surveys” Health Services Research, 46 (5), pp. 1663-1674.

Griffin JM, Simon AB, Hulbert E, Stevenson J, Grill JP, Noorbaloochi S, Partin MR (2011) “A comparison of small monetary incentives to convert survey non-respondents: a randomized control trial” BMC Medical Research Methodology, ISSN: 1471-2288, Vol. 11, pp. 81.

Kristin M.; Cook, J. R.; Jensen-Doss, A. (2009) “Do Noncontingent Incentives Increase Survey Response Rates among Mental Health Providers? A Randomized Trial Comparison” Administration & Policy in Mental Health & Mental Health Services Research, 36 (5), pp. 343-348.

Coughlin, Steven S., Aliaga, P., Barth, S., Eber, S., Maillard, J., Mahan, Clare M., Kang, Han K., Schneiderman, A., DeBakey, S., Vanderwolf, P. and Williams, M. (2011) “The Effectiveness of a Monetary Incentive on Response Rates in a Survey of Recent U.S. Veterans” 2011. Survey Practice, February

Rose D S, Sidle S D, Griffith K H (2007) “A Penny for Your Thoughts: Monetary Incentives Improve Response Rates for Company-Sponsored Employee Surveys” Organizational Research Methods, 10 (2), pp. 225-240.

Martin, E. Abreu, d. and Winters, F. (2001) “Money and Motive: Effects of Incentives on Panel Attrition in the Survey of Income and Program Participation” Journal of Official Statistics, 17: pp. 267-284.

Church, A.H. (1993) “Estimating the Effect of Incentives on Mail Surveys: A Meta-Analysis”, Public Opinion Quarterly, 57, pp. 62-79.

Trussell, N., and Lavrakas, P.J. (2004) “The Influence of Incremental Increases in Token Cash Incentives on Mail Survey Response: Is there an Optimal Amount?” Public Opinion Quarterly 68 (3): pp. 349-367.

Federally Sponsored Telephone Survey”. National Center for Education Statistics, U.S. Department of Education. NCES 2006-066. March.

Singer, E., Van Hoewyk, J., and Maher, Mary P. (2000) “Experiments with Incentives in Telephone Surveys”, Public Opinion Quarterly, 64 (2), pp. 171-188.

Gelman, A., Stevens, M. and Chan, V. (2003) “Regression Modeling and Meta-Analysis for Decision Making: A Cost-Benefit Analysis of Incentives in Telephone Surveys”, Journal of Business and Economic Statistics, 21:pp. 213-255.

 Balajti, Ilona; Daragó, László; Ádány, Róza; Kósa, Karolina. (2010) “College Students' Response Rate to an Incentivized Combination of Postal and Web-Based Health Survey” Evaluation & the Health Professions, 33 (2), pp. 164-176.

Wooldridge, Jeffrey M., Introductory Econometrics, South-Western Cengage Learning, 2009

1 Our goal is to have an 80% response rate from a sample of 1,980 respondents, yielding 1,584 completed surveys.

2 The average wage for UI recipients reported in a recent study of this population (Needels et al. 2002) is $16 per hour.

3 Logit and probit techniques are specifically designed to estimate models where the dependent variable is binary. Least squares models can be used to estimate binary outcomes but they may be imprecise when the dependent variable has low variation. For this reason, logit and probit are preferable to least square techniques when the dependent variable is binary.

4 Note: K is the number of parameters in the estimated model; j is the respective variable in question, N is the total number of treatment and control group participants; is the R-squared from regressing T on all other independent variables and e is the regression error.

5 Note: WSS_R is the weighted sum of square residuals for the restricted model (i.e., a model where we restrict α+γ to be zero); WSS_U is the weighted sum of square residuals for unrestricted model (i.e., the estimated model); N is the total number of treatment and control group participants; and k is the number of parameters in the estimated model.

IMPAQ 23 OMB