OMB Supporting statement A and B 11 06 2006
OMB Supporting statement A and B 11 06 2006.doc
Project Gate
OMB: 1205-0444
PROJECT GATE:
OMB Supporting Statement
August 8, 2006
Sheena McConnell
Shawn Marsh
Nuria Rodriguez-Planas
Submitted to:
U.S. Department of Labor Employment and Training Administration 200 Constitution Ave., NW Room N-5637 Washington, DC 20210
Project Officer: Jonathan Simonetta
|
Submitted by:
IMPAQ International, LLC 10420 Little Patuxent Parkway, Suite 300 Columbia, MD 21044 Telephone: (443) 367-0088 Facsimile: (443) 367-0477
Project Director: Jacob Benus
Co-Principal Investigators: Sheena McConnell Walter Corson |
CONTENTS
CHAPTER Page
a. justification 1
1. Circumstances Necessitating the Data Collection 1
2. How, By Whom, and For What Purpose the Information Is to be Used 5
3. Use of Improved Technology to Reduce Burden 8
4. Efforts to Identify Duplication 9
5. Methods to Minimize Burden on Small Businesses or Entities 10
6. Consequences of Not Collecting the Data 10
7. Special Data Collection Circumstances 11
8. Federal Register Notice 11
a. Federal Register Notice and Comments 11
b. Consultations Outside of the Agency 11
9. Respondent Payment 12
10. Confidentiality 12
11. Questions of a Sensitive Nature. 13
12. Hour Burden of the Collection of Information 14
13. Estimated Total Annual Cost Burden to Respondents and Record Keepers 18
14. Estimated Annualized Cost to the Federal Government 18
18. Changes in Burden 18
16. Tabulations, Publication Plans, and Project Schedule 18
17. Reasons for Not Displaying Expiration Date of OMB Approval 21
18. Exception to the Certification Statement 19 21
b. collection of information involving statistical methods 22
1. Respondent Universe and Sampling 22
2. Analysis Methods and Degree of Accuracy 23
3. Methods to Maximize Response Rates and Data Reliability 26
a. Response Rates 26
b. Reliability of Data Collection 27
4. Tests of Procedures or Methods 28
5. Individuals Consulted on Statistical Methods 29
APPENDIX
A: Survey Instrument
B: Explanations for Inclusion of Survey Questions
C: Federal Register Notice
D: Advance Letter
a. justification
This clearance package seeks approval to extend the follow-up survey that is being conducted as part of the Project GATE (Growing America Through Entrepreneurship) demonstration and evaluation of services for microenterprise development. The demonstration and its evaluation is being conducted by the U.S. Department of Labor (DOL) in conjunction with the Small Business Administration (SBA); it is being implemented, under contract to DOL, by IMPAQ International and its subcontractors—Mathematica Policy Research (MPR), Battelle Memorial Institute, and the National Center on Education and the Economy (NCEE).
1. Circumstances Necessitating the Data Collection
Many individuals have the motivation and skills to develop small businesses but lack business expertise and access to financing. Recognizing this untapped potential, DOL teamed with the SBA to create a demonstration program designed to assist individuals interested in self-employment to develop their businesses—Project GATE. An evaluation is necessary for policymakers and program developers to determine whether Project GATE should be replicated on a larger scale. A survey is the only way to collect information on self-employment experiences, receipt of microenterprise services, and household income. The demonstration and evalaution are conducted under the authority granted ETA in Section 171 of the Workforce Investment Act (WIA). Readers can look up the information at the following link. http://www.oalj.dol.gov/public/ina/refrnc/acwia.htm.
a. The Demonstration
A major outreach effort was launched to recruit persons for Project GATE. This effort included placing brochures, flyers, and posters at DOL’s one-stop career centers and at community-based organizations. The program advertised in local newspapers and on the Internet. Individuals registered for the program by mail, by telephone, via the Internet, or at specially-designed kiosks at one-stop centers. Notices about the program were sent to recipients of unemployment insurance (UI). Special attention was paid to outreach to immigrant populations, low-income urban populations, and rural populations by promoting Project GATE through local media.
Everyone who registered for the program was invited to a GATE orientation. At the orientation, a trained counselor described the services of the One-Stop Center. A video also was presented at the orientation to provide attendees with a description of the demonstration evaluations design (including random assignment), a realistic description of the challenges of self-employment, and a description of Project GATE and its services. People who are still interested in the program after the orientation submitted an application form to complete. Although anyone who attends an orientation could receive an application form, one purpose of the orientation was to provide enough information that people with unrealistic expectations about self-employment or the services available under Project GATE would choose not to apply.
Eligibility for Project GATE was broad. It was designed to serve almost anyone—either employed or unemployed--interested in starting a business. The program was open to anyone who is 18 years of age or over and a U.S. citizen (or lawfully able to work in the US) and as long as the applicant’s proposed business is legal and appropriate for support by DOL. Persons already self-employed but interested in developing their business further were also eligible for the program. However, the program was focused on the development of microenterprises--usually defined as a sole proprietorship, partnership, or family business that has fewer than five employees and requires a loan of less than $25,000 (Walker and Blair 2002).
Self-employment services provided by Project GATE included an assessment, a structured training course, and technical assistance.
Assessment. The assessment was conducted one-on-one at one of the SBA’s Small Business Development Centers (SBDCs). At this assessment, a trained counselor determined the service needs of the participant and made a referral to existing self-employment providers for further GATE services.
Structured Training Course. The training courses consisted of a series of classes on topics such as developing the business idea, writing a business plan, marketing, accounting, legal issues, cash flow, and financing.
Technical Assistance. The participant met with a trained counselor for technical assistance. As part of the technical assistance, counselors assisted individuals in completing their business plans. For those in need of financing for their businesses, the counselors provided assistance in applying for loans from SBA’s Microloan program and other funding sources.
GATE was implemented in seven sites in three states—Maine, Minnesota, and Pennsylvania. The three sites in Maine--the workforce investment areas around Portland, Bangor, and Lewiston—were all predominantly rural areas. Two sites were in Minnesota. One was in an urban area--Minneapolis-St. Paul—and the other was the rural area centered around Duluth in Northern Minnesota. The two sites in Pennsylvania were both urban—Philadelphia and Pittsburgh.
b. The Evaluation
GATE is being evaluated using an experimental design. Individuals who submit an application for GATE in each site were asked to sign an “application agreement” which gives their consent for participation in the study and for the release of administrative data. Signing the application agreement was a condition of eligibility for GATE—those who do not sign were not eligible for GATE services. Those who met the minimal eligibility criteria (18 years old or older, a valid business idea and permitted to work in the U.S.) were randomly assigned to either a program or control group. Members of the program group were eligible to receive GATE services. Members of the control group were not eligible to receive GATE services, although they were not prohibited from receiving self-employment services from other sources.
The evaluation is addressing three key questions:
Is Project GATE Viable? What are the challenges in implementing the program? Does it address the barriers to participation? Does an interagency model for the program work? Is the outreach effective in reaching immigrants? How does the implementation of the program vary across sites?
Does the Program Work? Does the program increase the use of self-employment services? Does the program lead to an increase in the completion of business plans and applications and receipt of loans? Does the program increase the likelihood of self-employment? Does the program promote employment and other aspects of economic development? Does the program increase employment, earnings, and satisfaction with employment and reduce the receipt of UI and public assistance? Is it effective in both rural and urban areas and in different service environments? Does the effectiveness of the program vary by population subgroup?
Is the Program Cost-Effective? Do the benefits of the program exceed its costs from the perspective of the participants, the government, and society as a whole?
Addressing these questions involves conducting process, impact, and benefit-cost analyses.
c. Data Collection
Data for the evaluation will be collected from the survey, administrative data from the program, the state UI agency, and site visits. As discussed more fully under A6 below, the survey is needed because it is the only source of data for important outcomes, such as self-employment experiences, receipt of microenterprise services, and household income.
The follow-up survey is being administered by telephone to all sample members twice: approximately 6 months after random assignment and then again 18 months after random assignment. Together the two interviews provide a complete history of employment and receipt of microenterprise services for the 18 months following random assignment. The first follow-up survey is used to collect information about the sample members’ experiences over the first 6 months after random assignment. The second follow-up survey is used to collected information about the subsequent twelve months. Interview attempts are made with all sample members at 18 months after random assignment, even with those who did not complete a 6-month survey. Those sample members interviewed for the first time at 18 months after random assignment will be asked about their experiences for the full 18 months after random assignment.
Although the questions asked at the 6 and 18 month follow-up are similar, we expect that respondents will spend more time at the first follow-up interview answering questions about the receipt of self-employment services and experiences starting a business and more time at the second follow-up interview answering questions about their business and employment experiences. Questions about experiences with self-employment and working for someone else prior to random assignment are asked only at the respondent’s first interview.
The survey instrument is provided in Appendix A. Appendix B summarizes the reasons for the inclusion of each question in the survey.
The administrative data from the program together with the data collected from the site visits provide data for the process analysis. The UI administrative data will provide data for the impact analysis on the receipt of UI benefits and quarterly earnings from jobs where the sample member is not self-employed.
2. How, By Whom, and For What Purpose the Information is to be Used
The survey data will be used to measure outcomes for members of the program and control groups in five broad areas described below and listed in Table 1.
Receipt of Self-Employment Services from GATE and Other Providers. Training, technical assistance, and other self-employment services that are not funded by GATE are available in all the sites to both the program and control group members. An important outcome is the extent to which Project GATE increases the receipt of different types of services and the intensity and quality of the services received.
Completion of Business Plans and Loan Applications. Project GATE is designed to assist participants in completing formal written business plans and loan applications. Project GATE may increase the number of business plans. It may also increase the number of completed and successful loan applications, especially those to the SBA Microloan program.
Business Development. By providing training and technical assistance, Project GATE aims to increase the success of business development, create employment, and promote economic development. Project GATE may increase the number of businesses started and increase the size and success of these businesses. Businesses may be larger in terms of sales, profits, number of employees, and payroll. They may provide more benefits to their employees.
Employment. By assisting people to start their own business, Project GATE may increase employment of sample members, increase their earnings, and their satisfaction with their employment. By increasing self-employment, Project GATE may decrease other types of employment. Hence, it is important to measure both types of employment--self-employment and employment working for other people. Project GATE may also affect the employment of sample members’ spouses. The direction of the effect on spouses’ employment is uncertain, however. Spouses may work more because of the uncertain income of self-employment or less if the business is successful and household income increases or the sample member has less time for child care and other household activities.
Household Income and Receipt of Unemployment Insurance and Other Public Assistance. By changing the employment outcomes of participants, Project GATE may change household income and the degree to which the participant is self-sufficient. Self-sufficiency will be measured by the receipt of UI, welfare benefits, and other forms of public assistance.
TABLE 1
OUTCOMES FOR THE GATE EVALUATION
Data on most of these outcomes is being obtained from the two follow-up surveys. Data on the receipt of UI and quarterly earnings data are obtained from administrative records maintained by each state.
The outcome data are used to estimate the overall impacts of the program. It is used together with the process analysis data, collected from the service providers and on site visits to the seven sites, to determine whether the program works in some settings but not others. This can provide important contextual information on the effectiveness of program models in different environments that is useful for program designers. For example, we may be able to determine whether the program is effective in urban and rural areas and in environments with different intensities of existing services. This will also provide information about whether GATE would be effective if replicated in other sites.
We can also estimate the impact of the program on different subgroups of the population. This allows program developers to target their program more effectively.
Even if the program is effective, its impacts may not be large enough to justify the expenditures on the program. The impact estimates will be used together with estimates of the cost of the program to determine whether the program is cost-effective.
The surveys also collect information on the barriers to starting a business. This information will be used to ensure that self-employment services are tailored to the needs of their clients. Information collected in the surveys may point to ways that Project GATE services can be improved.
Policymakers will use the results of the evaluation to assess whether Project GATE is cost-effective and should be replicated on a larger scale. The findings will also be useful to other microenterprise training providers in developing their programs.
3. Use of Improved Technology to Reduce Burden
Computer Assisted Telephone Interviewing (CATI) is being used to conduct the survey. CATI was selected because telephone interviews are more cost-effective and impose less burden on respondents than do in-person interviews. CATI is more cost effective than paper and pencil interviewing for many reasons, including the fact that CATI programs accept only valid responses and can be programmed to check for logical consistency across answers. Interviewers are thus able to correct errors during the interview, eliminating the need to call back respondents to obtain missing data. Also, calls will be made through an auto-dialer, linked to the CATI system, virtually eliminating dialing error. The automated call scheduler will simplify scheduling and rescheduling of calls to respondents at their convenience and can assign cases to specific interviewers, for example, those who are fluent in Spanish.
4. Efforts to Avoid Duplication
This survey is being conducted to collect key information about sample members. No other survey data collection effort has been conducted or has been planned to collect similar information.
The study is also using administrative records data where possible but since these data are not sufficient to conduct the study, survey data will be needed to supplement the administrative data. Specifically four kinds of administrative data will be used.
Project GATE Administrative Records on Program Application: Project GATE administrative records on program application provide some limited background data on characteristics of program and control sample members. These items are not included in the follow-up surveys, except for items such as marital status and household composition that could change after application.
Project GATE Administrative Records on Service Receipt. Project GATE administrative records on self-employment services are available for program group members. These records are used in the process analysis to describe service use. They cannot, however, be used in the impact analysis since they will not be available for control group members and since program group members may receive self–employment services outside of GATE. Hence the survey includes questions on self-employment service use for both program and control group members so that comparable data is collected for both groups.
UI Benefits Data: UI agency administrative records on UI eligibility and benefit receipt is being collected from the three states in the study and used in the analysis. Questions concerning UI benefit receipt are not asked on the survey.
Wage Records: Quarterly wage records are being collected from the three states to obtain summary information on employment and earnings by quarter. As the wage records exclude self-employment earnings and earnings from some other jobs, the survey also includes questions about employment and earnings. Additional detail on employment such as industry, occupation, hours worked, the hourly wage, and fringe benefits not available from wage records are collected on the survey.
5. Methods to Minimize Burden on Small Businesses or Entities
Some sample members will become self-employed and establish small businesses. Since self-employment is the major outcome of interest, these individuals are asked questions about their businesses. However, only the sample member and not other people in the business are asked questions, and the extent of the questions will be limited. We expect that the questions about small businesses add approximately 18 minutes to the interview for individuals who have self-employment experience.
6. Consequences of Not Collecting the Data or Collecting It Less Frequently
The survey provides the primary source for data for sample members on the following outcomes:
Self-employment training and services
Completion of business plans and application for loans
Self-employment experiences and earnings
Employment working for someone else
Income and receipt of public assistance
Therefore, if the survey were not conducted, the evaluation would be unable to assess the impacts of GATE services on these outcomes.
The survey is being administered twice—6 and 18 months following random assignment. It is being administered twice to minimize recall error. Information on sample members’ receipt of self-employment services is collected primarily on the 6-month interview since most services will be delivered in the first 6 months. Waiting to ask these questions until the 18-month interview would lead to substantial recall error. We also think that it is important to wait some time before asking sample members’ about their experiences in trying to start a microenterprise. Six months would be too short a period and hence we expect that the 18-month interview will obtain most of the information about these experiences.
7. Special Data Collection Circumstances
In all respects, the data will be collected in a manner consistent with federal guidelines. The statistical survey will produce valid and reliable results that can be generalized to the universe of study, and it will include only statistical data classifications that have been reviewed and approved by OMB. It includes a pledge of confidentiality that is supported by authority established in statute or regulation and by disclosure and data security policies that are consistent with the pledge. It will not unnecessarily impede sharing of data with other agencies for compatible confidential use.
8. Federal Register Notice
a. Federal Register Notice and Comments
In accordance with the Paperwork Reduction Act of 1995 (PRA95) [44 U.S.C. 3506 (c) (2)(A)]., the public was given an opportunity to review and submit comments on August 17, 2006 with the publication in the Federal Register (Vol. 71, No. 159, p 47531) of a Sixty Day Notice. No comments were received concerning the data collection, survey, or burden estimate. Three requests were received asking for information on how to become a Project GATE client. Each request was answered with the appropriate contact and project information.
b. Consultations Outside of the Agency
The following individuals were consulted in designing the data collection plan and developing the questionnaire:
Name |
Affiliation |
Telephone Number |
Jacob Benus |
IMPAQ International |
(443) 367 0379 |
Alicia Schoua-Glusberg |
IMPAQ International |
(410) 484 2500 |
Terry Johnson |
Battelle Memorial Institute |
(206) 525 3130 |
Wayne Vroman |
Urban Institute |
(202) 261 5573 |
Sheena McConnell |
Mathematica Policy Research |
(202) 484 4518 |
Walter Corson |
Mathematica Policy Research |
(609) 275 2398 |
Shawn Marsh |
Mathematica Policy Research |
(609) 936 2781 |
Nuria Rodriguez-Planas |
Mathematica Policy Research |
(202) 264 3449 |
Irma Perez-Johnson |
Mathematica Policy Research |
(609) 275 2339 |
Peter Schochet |
Mathematica Policy Research |
(609) 936 2783 |
Alan H. Dorfman |
Bureau of Labor Statistics |
(202) 691 7378 |
No unsolvable problems were identified by any of these individuals.
In addition, a pretest was conducted with nine respondents. Minor modifications were made to the survey as a result of the pretest as discussed later in Section B4.
9. Respondent Payments
Respondents are offered $15 for completion of each wave of the survey. This respondent payment is intended to boost response rates. The strategy of providing compensation for participation in the study draws on an extensive literature documenting its importance in achieving high levels of cooperation with surveys. Research has shown, for example, that even modest compensation can increase the response rates to surveys and lower the cost of data collection without compromising the quality of the data (Singer 2002; Singer et al. 1999a and 1999b).
10. Confidentiality
IMPAQ International is following procedures for assuring and maintaining confidentiality consistent with provisions of the Privacy Act. Respondents receive information about confidentiality protection in an advance letter describing the survey (presented in Appendix D) and again at the outset of the interview as part of the interviewer's introductory comments. Respondents are informed that all information they provide is treated confidentially. Interviewers are trained in confidentiality procedures and are prepared to describe these procedures in full detail, if needed, or to answer any related questions from the respondents. For example, if asked about confidentiality, the interviewer explains that the answers will be combined with those of others and presented in summary form only and that the answers will not affect past or future eligibility for any programs.
All data items that identify respondents are kept only by the contractor, IMPAQ International and its subcontractors--MPR and Battelle--for use in assembling records data and conducting the interviews. Any data received by DOL does not contain personal identifiers thus precluding individual identification.
In addition, the following safeguards are employed to carry out confidentiality assurances:
All employees at IMPAQ and the two subcontractors have signed a confidentiality pledge that emphasizes the importance of confidentiality and sets forth the obligations of staff.
Identifying information is maintained in a separate file from interview data. The files are linked only with a sample identification number.
Access to link-files containing sample identification numbers connecting the research data and the respondents' identification is limited to a few individuals who have a need to know this information.
Access to any hard-copy documents is strictly limited. Physical precautions include use of locked files and cabinets, shredders for discarded materials, and interview control procedures.
11. Questions of a Sensitive Nature
The survey of Project GATE sample members contains a minimal set of items that may be considered sensitive in nature. These questions are related to the success of businesses (C1-C57 in the questionnaire), receipt of individual and household income (D1-D22 in the questionnaire), and public assistance receipt (E1-E8 in the questionnaire). As described in item A10, all respondents are assured of confidentiality at the outset of the interview. This survey requires the collection of the social security numbers (SSN) of respondents so that we can obtain unemployment insurance (UI) benefits and wage records data on the sample members. The SSN is used as an identifier in those data. All survey responses are held in strict confidence and reported in aggregate, summary format, eliminating the possibility of individual identification.
IMPAQ International is complying with the requirements of the Privacy Act of 1974 in collecting all information. All questions in the current survey, including those deemed potentially sensitive, have been pretested and used extensively in prior surveys with no evidence of harm. Question about the success of the business are necessary to measure the effect of Project GATE on economic development. Questions about income and public assistance receipt are necessary to measure the economic well-being of study participants and to conduct the benefit-cost analysis.
12. Remaining Hour Burden of the Collection of Information
The remaining hour burden for information collected for the follow-up survey is 267 hours as shown in the attached table. This hour burden estimate is based on actual pretests of the survey which averaged 40 minutes to complete.
Activity |
Total respondents |
Frequency |
Total responses |
Average time per response |
Burden (hours) |
GATE 18-month follow-up survey (in 2007) |
400 |
once |
400 |
40 minutes |
267 hours |
Totals
|
|
|
|
|
267 hours |
The total burden cost of collecting this information is $4272. This cost represents 40 minutes to complete the survey multiplied by the number of remaining completers (267) and by an estimated average hourly wage of $16 per hour.1 This burden cost is offset by the $15 respondent payment.
13. Estimated Total Annual Cost Burden to Respondents and Record Keepers
There will be no start-up or ongoing financial costs incurred by respondents. There are no record keepers.
14. Estimated Annualized Cost to the Federal Government
The total cost to the Federal government of conducting the survey is $1.5 million, which is the total contractor cost of conducting the survey.
15. Changes in Burden
This is a one time data collection effort, nearly finished, counting as 267 hours towards ETA’s Information Collection Budget, or -4,000 hours towards the currently approved 1205-0444 Collection Budget.
16. Tabulations, Publication Plans and Project Schedule
a. Tabulations
The survey data will be used together with administrative data on UI receipt and quarterly earnings, data collected from the service providers, and data collected via site visits, to address the following four broad questions:
What is the overall impact of GATE on the receipt of self-employment services, completion of business plans, application for SBA Microloans and other loans, business development, employment, household income, and receipt of UI and public assistance?
Do the impacts of GATE differ by economic or demographic characteristics of the sites, the service environment, or the way in which GATE is implemented?
Do the impacts vary for different subgroups of the population?
Do the benefits of GATE outweigh the costs?
Estimating Overall Impacts. The analysis will begin with a comparison of the average outcomes of sample members in the program group with those in the control group. The randomized design ensures that there will be no systematic observable or unobservable differences between program and control group members except for the acceptance into Project GATE. T-statistics can be used to indicate statistical significance of these differences.
More precise estimates can be obtained by using regression methods to control for any random differences in the baseline characteristics of program and control group members. In their simplest form, these models can be expressed by the following equation:
Y = + X + P + e,
Where:
Y is an outcome variable
X is a vector of control variables
P is an indicator that equals 1 for program group members and 0 for control group members
is the intercept term
is the vector of regression coefficient for the control variables
is the measure of the impact of Project GATE on accepted applicants
e is a random disturbance term that is assumed to have a mean of zero conditional on X and P, and is interpreted as the unobserved factors that affect Y.
The statistical techniques used to estimate the regression-adjusted impacts depend on the form of the dependent variable, Y. If the dependent variable is continuous, then ordinary least squares techniques will produce unbiased estimates of the parameter . However, if the dependent variable is binary—for example, whether the sample member started a business—then consistent parameter estimates can be obtained by using logit or probit maximum likelihood methods. If the dependent variable is censored or truncated, such as hours worked in a given quarter, then tobit maximum likelihood or two-stage procedures will be used.
Control variables in the vector X will include any variables that may affect the outcome that are not affected by the intervention. Hence, they will include demographic and socio-economic characteristics (such as age, gender, race/ethnicity) at the time the sample member applies to GATE. It will also include measures of experience in employment and self-employment prior to random assignment. This will include whether the sample member has prior managerial experience, which is viewed as important in the success of starting and running a business.
Differences-in-means and estimates of the coefficient in the regression described above will provide estimates of the impact of acceptance into Project GATE. Because random assignment will occur after an orientation at which people are told about the program and the challenges of self-employment, we expect that a high proportion of those who are accepted will participate. However, some who are accepted may still decide not to participate in the program. Obtaining estimates of the impact of GATE on those who actually receive GATE services may be of policy interest. Assuming that GATE has no impact on those who are accepted into the program but do not receive services, the impacts on those who receive services can be computed by dividing the impact estimates based on all program and control group members by the proportion of program group members who receive services.
Estimating Impacts by Site Characteristics. Although the samples are not large enough to reliably estimate impacts by individual site, we will estimate impacts for different site characteristics. Site characteristics of interest include:
Whether it is rural or urban
Richness of the self-employment service environment
Extent to which SBA Microloans are available
Degree to which economy is based on small businesses
Unemployment rate and strength of the economy
Differences in how Project GATE is implemented, such as availability of different types of providers and the average length of assessment
Information on these site characteristics will be obtained from published sources and during visits to the sites.
We will estimate impacts for a particular site characteristic by comparing the mean outcomes of program and control group members in sites with that characteristic. For example, we will estimate program impacts in rural sites by comparing the outcomes of program and control group members in rural sites. To estimate regression-adjusted impacts by site characteristic, we will slightly modify the regression models by adding an interaction term that is the product of the program group indicator (P) and the indicator for the site characteristic of interest. We will conduct statistical tests to gauge both the statistical significance of the impact estimates by site characteristic and the difference in impacts by site characteristic.
Estimating Impacts by Population Subgroup. We will also estimate impacts for subgroups of the population. Subgroups of interest include those defined by the following characteristic at random assignment:
Demographic characteristics, such as age, gender, race/ethnicity, marital status, household size
Education level
Immigrant status
Whether already self-employed
Prior self-employment experience
Prior work experience and earnings, including whether they have had managerial experience
Income and self-sufficiency
Receipt of UI benefits
Data for some of these baseline characteristics that can easily be recalled, such as prior self-employment and work experience, will be obtained from the first follow-up survey. Data on other baseline characteristics will be obtained from the application form.
We will estimate the impacts by population subgroup using similar methods to those we will use to estimate impacts by site characteristic.
Estimating Benefits and Costs. Government programs are generally viewed as cost-effective if social benefits exceed social costs. In assessing the desirability of the program from a policy perspective, however, it is also important to account for the distributional impacts of the program. Hence, we will estimate benefits and costs from the perspectives of participants and taxpayers as well as from the perspective of society as a whole.
The potential benefits of Project GATE include:
Increased Earnings of Program Participants. As earnings measure productivity, society benefits from an increase in earnings of participants, whether from increased self-employment or from working for someone else. The estimated impact of GATE on earnings from all employment plus an estimate of the cost of fringe benefits received will be used as an estimate of this benefit.
Increased Earnings of Persons Hired by Participants’ Businesses. If there is unemployment in the community, any impact of the program on the total number of people hired by the participants’ businesses is a social benefit. This will be estimated from the estimated impact of Project GATE on the total payroll of participants’ businesses plus the cost of fringe benefits received by employees.
Increased Tax Payments. As participants earn more, they pay more in income, sales, and payroll taxes, and possibly corporate income taxes. The increase in taxes represents a transfer from participants to taxpayers, and is neither a benefit nor a cost to society. This transfer will be estimated using impacts on earnings and published estimates of the effects of increased income and corporate profits on tax payments.
Reduced Use of UI and Public Assistance. The reduction in UI and public assistance is a benefit to taxpayers, but a cost to program participants. Again, except for the cost of administering the programs, the reduction in benefits is neither a benefit nor a cost to society as a whole. This benefit will be estimated using the estimated impacts on receipt of the assistance and published estimates of the administration costs of the programs.
Satisfaction. It may be that even if earnings do not increase, participants are more satisfied with work in their own businesses than they were when working for someone else. Although we will not be able to place a dollar value on the impacts on satisfaction, we will present the impacts on satisfaction along with the dollar value of benefits.
The potential costs of Project GATE fall into two main categories. The first category is the increased use of self-employment services. We will estimate these additional costs from the impact of the program on the amount of self-employment services received and estimates of the costs of each type of service obtained during the site visits. Second, by increasing the application and receipt of loans, society will bear the additional costs of the loans that are not covered by loan repayments. These will be estimated from the estimates of the impact of the program on loan applications and receipt of loans and estimates of the administrative and other costs of each loan.
b. Publication Plans
The final report on Project GATE will be submitted to DOL in draft form in April 2007 and in final form in June 2007. The report will describe the results of the evaluation that will use data from both follow-up surveys as well as UI administrative data and data collected as part of the process analysis.
c. Time Schedule
The project began in July 2002 and will end in June 2007. The design and survey instruments were prepared between December 2002 and March 2003. The demonstration began in September 2003. The sample intake period ended in June of 2005. The first follow-up survey started in March of 2004. The second follow-up survey began in March of 2005 and will end in March 2007.
17. Reasons for Not Displaying Expiration Date of OMB Approval
The expiration date will be displayed on the advance letter and on the hard copy version of the questionnaire.
18. Exceptions to the Certification Statement 19
There are no exceptions requested in OMB Form 83-1.
b. collection of information involving statistical methods
1. Respondent Universe and Sampling
For each of the seven sites, the study population includes all individuals eligible for Project GATE who applied for Project GATE services beginning in Summer 2003. The sample intake period lasted 22 months with Project GATE enrolling 4168 participants. Half of the eligible applicants were randomly assigned to be offered Project GATE microenterprise services (the program group) and half were not offered GATE services but could seek pre-existing microenterprise services (the control group).
Follow-up interviews at 6 and 18 months following random assignment will be attempted with all 4,168 sample members. Based on experiences with similar surveys, we expect to obtain approximately an 80 percent response rate in each wave of the survey.
2. Procedures for the Collection of Information
a. Statistical Methodology, Estimation, and Degree of Accuracy
The primary objective of the Project GATE evaluation is to provide statistically valid and reliable estimates of the incremental effects of Project GATE self-employment services on key outcomes, including self-employment, weeks worked, and earnings. Use of a classical experimental design, in which applicants are assigned randomly to program and control groups, ensures that measured impacts represent valid estimates of the effects of the demonstration services. The measured impacts are internally valid for those sites. Since the three states and the sites within the states have been chosen purposively, they cannot be generalized to a wider population with a known degree of statistical precision.
Impacts will be estimated by computing differences in mean outcomes between individuals in the program group (that is, those offered Project GATE services) and individuals in the control group. Simple differences of means will be computed, but we will also adjust for random differences at intake using multivariate regression. The regression adjustments will increase the precision of the impact estimates. More detail on estimation procedures is included in our discussion of tabulation plans under item A16.
Given this design the main question is whether the impact estimates will be precise enough to detect policy relevant impacts. To answer that question, Table 2 shows minimum detectable impacts for comparisons across the full sample (that is, comparisons of 2,000 program group members and 2,000 control group members) for three key labor market outcomes—(1) the percentage ever self-employed, (2) total weeks employed over a 12 month period, and (3) earnings from all jobs over a 12 month period (self-employment and wage and salary jobs). We also show minimum detectable impacts for subgroups such as individuals in urban sites (1,300 programs and 1,300 controls) or rural sites (700 programs and 700 controls).
TABLE 2
MINIMUM DETECTABLE IMPACTS FOR KEY LABOR MARKET OUTCOMES AT 18 MONTHS AFTER RANDOM ASSIGNMENT
|
|
Minimum Detectable Impacts |
||
Program Group/ Control Group |
Available Sample |
Percentage Self Employed |
Weeks Employed |
Earnings from All Jobs |
2,000/2,000 |
1,600/1,600 |
3.9 |
2.6 |
1,326 |
1,300/1,300 |
1,040/1,040 |
4.8 |
3.3 |
1,645 |
700/700 |
560/560 |
6.5 |
4.5 |
2,242 |
Note:
The calculations assume (1) a 95 percent confidence level with an 80
percent level of power; (2) a one-tail test for the percentage
self-employed, but a two-tail test for weeks employed and earnings;
(3) a reduction in the variance of 20 percent owing to the use of
regression models; (4) 40 percent of the control group will become
self-employed; (5) a standard deviation of 30 for weeks employed and
$18,000 for earnings, which are consistent with findings from
previous studies of similar populations; and (6) a response rate of
80 percent on the 18 month interview. The minimum detectable impacts
(MDI) are calculated using the following formula:
where
= 2.5 for a one-tail test and 2.8 for a two-tail test,
is the standard deviation of the variable, R2
is the variance
explained by the regression model, r is the response rate, and n is
the size of the program and control group.
Based on a review of results from previous studies of self-employment program impacts, we believe that a sample of 2,000 program and 2,000 control group members will meet precision targets for impacts for the full sample and key subgroups. As seen in the table, the minimum detectable impact is 3.9 percentage points for the percentage ever self-employed, 2.6 weeks for weeks employed, and $1,326 for earnings. Minimum detectable impacts for important subgroups-urban and rural sites--are larger given the reduction in sample size.
Previous analyses of self-employment programs suggest that program impacts may exceed these minimum detectable impacts including those for the urban and rural subgroups. For example, Benus et al (1995) found impacts of the Self-Employment Assistance program on the percentage self-employed of 22 percentage points in Massachusetts and 12 percentage points in Washington, impacts on weeks employed of nine weeks in Massachusetts and five weeks in Washington, and impacts on annual earnings of $6,000 in Massachusetts and $300 in Washington. Except for the $300 earnings impact in Washington, we could detect these impacts even with smaller sample sizes. However, impacts for GATE could be lower than these earlier findings imply since existing self-employment services are substantial in some sites and control group members may receive those services. Having the ability to detect impacts for even smaller subgroups than those shown in the table (for example, prior self-employment experience and the urbanicity of the site) will also be valuable.
b. Unusual Problems Requiring Specialized Sampling Procedures.
There are no unusual problems requiring specialized sampling procedures.
c. Periodic Cycles to Reduce Burden
The GATE survey is being administered twice—6 months and 18 months after random assignment. While burden could be reduced by administering the survey once (at 18 months) and collecting information for the entire 18-month period, we believe that this approach would lead to recall problems that would affect the quality of the data. We expect that most sample members will receive self-employment services from GATE or other sources within the first six months following random assignment and hence we expect the initial survey to be the primary source of information on these services. At 18 months after random assignment, sample members may not remember in detail their experiences receiving services. Administering the survey again at 18 months is, however, necessary since a follow-up period longer than 6 months is needed to observe individuals’ attempts to become self-employed. An even longer follow-up period might be desirable but is not possible given the time constraints for the study.
3. Methods to Maximize Response Rates and Data Reliability
a. Response Rates
In the first wave of the survey, Project GATE achieved an 82% response rate. We expect an 85% percent response rate for the second wave of interviews. Several strategies are being used to achieve this high response rate. First, before interviewing begins, an advance letter describing the purpose and sponsorship of the survey is mailed to potential respondents (the letter is presented in Appendix D). This advance letter assures potential respondents that the caller is conducting a research interview and not soliciting donations or selling anything. Letters are sent approximately one week before the sample is released to the CATI call scheduler. The letter requests up-to-date contact information and provide a toll-free call-in number.
Second, experienced interviewers are recruited and extensively trained. These interviewers are thoroughly trained on data collection procedures, including methods for promoting cooperation among sample members. Interviewers especially skilled at encouraging cooperation are available to persuade reluctant respondents to participate and are assigned to attempt conversions with respondents who initially refuse (except for hostile refusals). The interview is translated into Spanish and bilingual interviewers are used to conduct interviews in Spanish.
Third, call scheduling in CATI allows respondents to select the time most convenient for them to be interviewed.
Fourth, detailed contact information provided by sample members when they apply for GATE is used to help locate sample members for the follow-up interviews. This information includes not only an address and telephone number but an e-mail address and cell phone number, if available, and the names and addresses of three relatives or other individuals who will know how to contact them. If these sources do not provide sufficient information to contact the sample member, extensive use is made of various on-line databases to try to locate sample members who have moved.
Finally, a $15 incentive is used on each round of the survey to encourage participation. Past research (Singer 2002, Singer et al. 1999a and 1999b) indicates that incentive payments help boost survey response rates. They are particularly valuable in panel studies like this one where respondents are contacted and surveyed more than once.
When the survey is completed we will conduct an analysis of nonresponse to assess whether the survey sample is representative of the initial population of GATE applicants. In particular we will examine whether any differences in response rates between program and control group members may affect the findings. This analysis will use background data collected on the application form including demographic data and data on prior self-employment experiences. Quarterly wage record data on post random assignment earnings, not subject to nonresponse, will be used to examine differences in earnings. Sample weights will be assigned to adjust for differences between responders and nonresponders in important background characteristics.
b. Reliability of Data Collection
The draft questionnaire was built extensively on questionnaires developed for other U.S. Department of Labor studies, including the UI Self-Employment Demonstration Follow-up Survey; Comprehensive Assessment of Self-Employment Assistance Programs (OMB number 1205-0412); 1992 Economic Census Characteristics of Business Owners Survey (OMB number 0640-0022); the Job Search Assistance Experiment Survey (OMB number 1205-0367), and the Survey of UI Recipients (OMB number 1205-0405).
The questions were designed to ensure that they would be easily understood by respondents. Revisions were made to the draft questionnaire based on an internal review, a review by technical advisors to Project GATE, a review by DOL, and a pretest.
The use of CATI to conduct the survey also helps ensure the reliability of the data. It controls question branching (reducing item nonresponse due to interviewer error), modifies wording (providing memory aids and probes and personalizing questions), and constructs complex sequences that are not possible to produce or are less accurate in hard-copy surveys. The probes, verifications, and consistency checks are built into the system and standardizes the procedures. These procedures ensure the reliability of the data collection methods and the data collected through those methods.
Lastly, IMPAQ International monitors each interviewers’ work using silent call-monitoring equipment and video monitors that display the interviewers’ screen.
4. Tests of Procedures or Methods
Nine pretests of the current survey instrument were conducted. The pretests assessed the content and wording of individual questions, the organization and format of the questionnaire, respondent burden time, and potential sources of response error. The pretest results were used to modify the questionnaire and the pretest results were incorporated in the initial OMB submission.
5. Individuals Consulted on Statistical Methods
The following persons outside of ETA contributed to, reviewed, and/or approved the design, instrumentation and sampling plan:
Name |
Affiliation |
Telephone Number |
Jacob Benus (Project Director) |
IMPAQ International |
(413) 367 0379 |
Terry Johnson |
Battelle Memorial Institute |
(206) 525 3130 |
Wayne Vroman |
Urban Institute |
(202) 201 5573 |
Sheena McConnell |
Mathematica Policy Research |
(202) 484 4518 |
Walter Corson |
Mathematica Policy Research |
(609) 275 2398 |
Peter Schochet |
Mathematica Policy Research |
(609) 936 2783 |
Alan Dorfman |
Bureau of Labor Statistics |
(202) 691 7378 |
references
Benus, Jacob M., Terry R. Johnson, Michelle Wood, Neelima Grover, and Theodore Shen. “Self-Employment Programs: A New Reemployment Strategy, Final Report on the UI Self-Employment Demonstration.” Unemployment Insurance Occasional Paper No. 95-4. Washington, DC: U.S. Department of Labor, Employment and Training Administration, 1995.
Needels, Karen, Walter Corson, and Walter Nicholson. “Left Out of the Boom Economy: UI Recipients in the Late 1990s.” ETA Occasional Papers 2002-2003, Washington, DC: U.S. Department of Labor, Employment and Training Administration, 2002.
Singer, Eleanor. “The Use of Incentives to Reduce Nonresponse in Household Surveys.” In Survey Nonresponse. Edited by R. Groves, D. Dillman, J. Eltinge, and R. Little. New York: John Wiley and Sons, 2002.
Singer, Eleanor, John Van Hoewyk, Nancy Gebler, Trivellore Raghunathan, and Katherine McGonagle. “The Effect of Incentives on Response Rates in Interviewer-Mediated Surveys.” Journal of Official Statistics, vol. 18, no. 2, 1999a.
Singer, Eleanor, Robert M. Groves, and Amy Corning. “Differential Incentives: Beliefs About Practices, Perceptions of Equity, and Effects on Survey Participation.” Public Opinion Quarterly. Vol. 63, 1999b.
Walker, Britton, and Amy Kays Blair. 2002 Directory of U.S. Microenteprise Programs. Washington, DC: The Aspen Institute, 2002.
1 The average wage for UI recipients reported in a recent study of this population (Needels et al, 2002) is $16 per hour.
| File Type | application/msword |
| File Title | MEMORANDUM |
| Author | Rosita Turkel |
| Last Modified By | naradzay.bonnie |
| File Modified | 2006-11-06 |
| File Created | 2006-11-06 |
© 2024 OMB.report | Privacy Policy