U.S. DEPARTMENT OF LABOR
SUPPORTING STATEMENT
Evaluation of the Aging Worker Initiative
New Information Collection Request
A. JUSTIFICATION
1. Explain the circumstances that make the collection of information necessary:
The collection of information via site visits, phone reconnaissance, and additional program data, is necessary to evaluate the Aging Worker Initiative (AWI), a $13 million demonstration program testing new strategies for helping workers 55 and older stay in the labor market and remain productive. Ten competitive grants, awarded to local organizations, are being used to build partnerships and test innovative approaches in training, job development and specialized counseling assistance, among other interventions, geared to the target population.
The proposed collection of information will provide the data needed to meet the evaluation objectives, which are:
to provide a comprehensive picture of the activities undertaken by grantees and how they were designed to help older workers develop skills and secure employment in high-demand, high-growth occupations;
to describe the administrative arrangements, partnerships, funding, expenses, and performance in meeting the goals of the demonstration; and
to identify promising strategies, practical lessons, and useful products for the workforce investment system, other organizations serving aging workers, and regional economies.
2. Explain how, by whom, how frequently, and for what purpose the information will be used. If the information collected will be disseminated to the public or used to support information that will be disseminated to the public, then explain how the collection complies with all applicable Information Quality Guidelines:
Information is to be collected via two rounds of site visits to the ten sites and two rounds of phone reconnaissance with the ten grantee administrators. The list of questions to be used in the site visits and phone reconnaissance can be found in the attached set of protocols for different respondent groups.
Respondents during site visits include program administrators, front line staff, partner agency staff, participants, and employers. All administrators and project managers will be interviewed as will most of the front-line staff. Grantee administrators will be asked to identify front-line staff and partner agency staff for interviews.
The first round of visits will focus on early implementation and its challenges, while the second round of site visits will focus on the latter stages of implementation and sustainability. Also, the second but not the first site visit will involve focus groups with participants as well as focus groups and individual interviews with employers.
Program participants will be those currently in training or other service and those have recently exited the program. Participants will be invited to participate in focus groups by grantee administrators or local project managers but only a small sample of participants that have experienced a broad range of services will be contacted.
Employers will be identified by grant administrators or project managers, who will provide a list of employers for the evaluators to contact for individual or small group interviews. The list will include all employers involved in initial planning or in offering potential internship or job openings. It is anticipated that there will be two to three employers per site, and that these will include all the employers directly involved in the projects.
As per the grant agreement, grantees must provide program data to the contractor, including the data collected for the quarterly reporting and the Common Measures (as already required in the OMB-approved “High Growth and Community-Based Job Training Grants: General Quarterly Reporting Forms & Instructions”) and other data specific to the demonstration. A list of the additional data to be provided is included as an attachment to this statement. To simplify provision of all data, however, grantees can use, on a voluntary basis, free software tailored to the AWI demonstration and developed by the Employment and Training Administration (ETA) of the U.S. Department of Labor.
The master protocol was developed by the evaluation contractor, Social Policy Research Associates (SPRA) and its subcontractor, Mathematica Policy Research (MPR), while the data elements were developed by the Employment and Training Administration (ETA) in the U.S. Department of Labor (USDOL) in collaboration with SPRA. SPRA and MPR will conduct all data collection, including the site visits and phone calls, and will also receive data from the grantees. Questions from the protocol will be directed at the informants most likely to be able to respond and separate question lists will be prepared and used with different respondents.
Compliance with Information Quality Guidelines on utility, integrity, and objectivity will be achieved through the following:
The information from the evaluation of the AWI will be used to inform policy regarding services to aging workers in the public workforce system and in determining future research priorities. Information will also be shared with state and local practitioners on lessons learned, promising practices, and effective strategies has the potential to enhance the quality of services delivered to the target population. Interim and final reports from the evaluation, concerning the AWI overall and individual sites, will be available to the general public in the ETA Research Publication Database at http://wdr.doleta.gov/research/keyword.cfm. Oral briefings also will be provided to ETA and other public officials. Accurate, clear, complete and unbiased reports will be developed as a result of the contractor’s editorial review (included as part of the contract itself), and by extensive internal review at ETA.
Objectivity, defined as accurate, reliable and unbiased information, will be achieved by conducting site visits using a protocol that will include standardized questions and activities for site visitors. Consistent and reliable information will be obtained by providing rigorous training to all site visitors on use of the protocol and their role as objective recorders and observers, and having the site visitors record information immediately after and during face-to-face individual and group interviews and from direct observations. Information from each site will be recorded in individual site visit reports and the information will be recorded in a matrix, as appropriate.
Integrity of the information will be maintained by the observance of data security measures, both in the field and in the office, including password protected laptops for recording site visit data and password protection and limited access to desktop computers on which grantee data is stored, consistent with the level of data security needed for the type of information to be acquired (which does not include any personally identifiable information).
3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological techniques or other forms of information technology:
The collection of information during site visits necessarily involves face-to-face contact via individual and group interviews in order to gain insights into program operations and to obtain products that result from the grants, such as new curricula or software. Technological techniques have limited applicability for the interviews and for direct observation of facilities, classes, and intake or counseling sessions. However, transmission of products such as curricula will be conducted via computer and Internet, as appropriate.
Administrative data will be collected through direct data uploads to the contractor’s secure FTP site or via CD, as per the grantee’s preference. ETA has also made an effort to simplify grantees’ inputting of participant data and provision of quarterly report and evaluation data by offering to grantees free software tailored to the AWI demonstration.
4. Describe efforts to identify duplication:
Since the AWI is a new grant program, there have been no other efforts to collect data via site visits or phone reconnaissance. Since grantees are already collecting information for the quarterly reports and Common Measures, an effort has been made to build on the existing data and reporting requirements, to avoid duplication and make data collection as easy as possible. For example, the contractor will be reviewing documents at each site and will only confirm responses as necessary, and will not repeat questions.
5. If the collection of information involves small businesses or other small entities, describe the methods used to minimize burden:
Information will be acquired from small non-profit organizations that administer the AWI grants, small partner organizations, and small businesses that are involved in the project. Efforts to minimize burden on these small entities include limiting the number of site visits, using shorter phone reconnaissance interviews, requesting only a limited set of data elements, and providing a small easy-to-use data system to input participant data (both for the evaluation and for the quarterly reports and Common Measures).
6. Describe the consequences to the Federal program or policy activities if the collection is not conducted or is conducted less frequently:
If the collection is not conducted, there will be only limited, and no summary, information about the nature of the AWI project and individual grants. Information on the various interventions, who received these services, the administrative issues encountered, and the possible effectiveness of the AWI grants will not be available, nor will there be systematic and unbiased information on successes or challenges encountered. Development of new policy or demonstration projects would not be informed by the experiences under these grants.
To conduct the data collection less frequently, say by using less than two site visits, would not allow for an understanding of what occurred during different phases of the grants, nor of the problems that arose at different points in time.
7. Explain any special circumstances that require the collection to be conducted in a manner inconsistent with OMB guidelines:
No such circumstances are anticipated.
8. Provide information on the PRA Federal Register Notice that solicited public comments on the information collection prior to this submission. Summarize the public comments received in response to that notice and describe the actions taken by the agency in response to those comments. Describe the efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported:
The public was given an opportunity to review and comment (Federal Register Notice, dated November 19, 2010, Volume 75, Number 223, Pages 70949 – 70950, with comments due on or before January 18, 2011). No comments were received on this information collection request
The public was given an opportunity to review and comment in a Federal Register Notice, dated November 19, 2010, Volume 75, Number 223, Pages 70949 – 70950, with comments due on or before January 18, 2011.
Two comments were received. One commenter stated that the proposed collection of information is a necessary evaluation component to provide data and anecdotal records for determining implementation progress and accomplishments and could be useful in disseminating information on successful methods and activities. However, the commenter felt that the proposed number of burden hours appeared to underestimate the time grantee staff will utilize in regard to the site visits and for inputting data. Response: The burden hours as proposed represent an expected average among different sites, based on contractor experience with similar information collection activities. The burden hours therefore have not been changed.
Another commenter questioned the value and cost-efficiency of the project and recommended that the funds for the project be utilized for reducing the deficit. Response: It was not clear if the respondent was questioning the value of data collection, the evaluation, or the underlying grants. The data collection is part of an evaluation that was procured through a competitive process in which cost was a critical factor. The evaluation itself will look at various outcomes, including how efficiently funds were used in providing services to the target populations; therefore the degree to which the funds in the grants were used cost-effectively is not yet known. No change to the information collection request has been made.
Consultations on the research design, data sources and collection approaches occurred during the study’s design phase and will continue to take place throughout the study. The purpose of such consultations is to ensure the technical soundness of the study and the relevance of its findings. The contractor and subcontractor have provided substantial input to DOL for the evaluation. The table below displays the senior technical staff from these organizations that were consulted in developing the research design, the data collection plan, the site visit protocol, the phone reconnaissance script, and the new data elements.
TABLE 2
CONTRACTOR TECHNICAL STAFF
Name |
Affiliation |
Telephone Number |
Dr. Ronald D’Amico |
Social Policy Research Associates |
(510) 763-1499 |
Jeffrey Salzman |
Social Policy Research Associates |
(510) 763-1499 |
Jill Leufgen |
Social Policy Research Associates |
(510) 763-1499 |
Michelle Derr |
Mathematica Policy Research |
(202) 484-4830 |
9. Explain any decisions to provide payments or gifts to respondents, other than remuneration of contractors or grantees:
No payments or gifts will be offered to grantee administrators and staff, partner organizations, or to business persons who participate in the evaluation of the AWI grants.
10. Describe any assurance of confidentiality provided to respondents and the basis for assurance in statute, regulation, or agency policy:
Grantee and partner organizations administrators and staff , who provide information during site visits or phone reconnaissance, will receive information about how information will be shared, consistent with the program design regarding nondisclosure of information from specific sites, (i.e., not to disclose information about specific sites that might cause respondents to avoid candid responses) .
Focus group participants (both program participants and employers) will also receive assurances about the nondisclosure of information that they provide before the group discussion begins, and no personally-identifying information about them will be recorded. There will be no PII provided by the focus group participants and all findings will be summarized in a way to prevent any identification of individual respondents.
All information gathered during site visits and phone reconnaissance, as well as administrative data provided by local sites, will be kept by SPRA and MPR for use in identifying and analyzing the services and their effectiveness. No information received by the U.S. Department of Labor, Employment and Training Administration will contain personal identifiers, which will thus preclude individual identification. In addition, the following safeguards, consistent with provisions of the Privacy Act, will be used by research team members to assure privacy in the collection of site visit, phone, and administrative data [covered by a System of Records Notice, ETA-15, published April 8, 2002 (67 FR 16898 et seq)]:
Access to data with respondents’ contact information will be limited to a few persons who have a need to know this information and
Access to any hard-copy documents will be strictly limited. Physical precautions will include use of locked files and cabinets, shredders for discarded materials, and interview control procedures.
The research team will also use standard methods to guard against inadvertent disclosure1 of information in tabular results of administrative data, using the following guidelines:
No cell shall be reported if the number of respondents is less than 10 and
No single cell shall solely account for a row or column total.
Should these conditions be violated in initial tabulations, rows or columns will be combined, as necessary, until the conditions are satisfied.
11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private:
There are no questions of a sensitive nature regarding sexual behavior or attitudes and religious beliefs, but information is being requested on individuals’ educational attainment and whether they are ex-offenders, homeless, or lack a work history. This information is needed to understand who received services, what types of services different groups of individuals received, and how successful the interventions were for different subgroups.
12. Provide an estimate in hours of the burden of the collection of information:
|
Average # Respondents Per Site |
Total # Respon-dents |
Hours/ respondent |
Total hours |
Average cost/hour1 |
Total cost |
Round 1 site visit |
18 |
180 |
1.33 |
216 |
$29.82 |
$7,127 |
Round 2 site visit |
25 |
250 |
1.33 |
333 |
$30.92 |
$10,296 |
Data/MIS-participants |
685 |
6,850 |
0.5 |
3,425 |
$20.25 |
$69,356 |
Data/MIS-Grantee staff |
1 |
10 |
40.0 |
400 |
$32.50 |
$13,000 |
Telephone Reconnaissance – Two rounds |
1 |
10 |
2.0 |
20 |
$34.94 |
$698 |
Focus Group Respondents |
6 |
60 |
.75 |
45 |
$20.25 |
$911 |
Total |
730 |
7,300 |
|
4,439 |
|
$101,338 |
13. Provide an estimate of the total annual cost burden to the respondents or record-keepers resulting from the collection (excluding the value of the burden hours in #12 above):
There are no additional costs to the respondents or to the record-keepers, who already have the equipment to collect the data.
14. Provide estimates of annualized cost to the Federal government:
The estimated total cost per year to the Federal government is $210,854.33, based on the total cost of the contract for the evaluation of $563,563 plus $69,000 in Federal staff costs for a total of $632,563, divided by 3 years.
Actual costs of the information collection alone, under the contract with the Federal government, are as follows: $77,580 in 2010, $7,500 in 2011, and $73,632 in 2012, for a total cost of $158,712, for 20 site visits, 10 phone reconnaissance efforts, and acquisition of quantitative data. This figure includes the contractor’s loaded labor costs for all activities and travel costs associated with the site visits. As note above, though, the total cost of the evaluation over the length of the contract, including data collection, design, analysis, writing, and management is $563,563.
In addition, estimated costs to the Federal government for all government personnel are estimated to be $69,000 and include the following:
Staff level management of the evaluation (including development of the initial Request for Quotation, Procurement Review Board review packages, development of new data elements, and oversight and guidance over 42 months): $60,000;
Oversight by other USDOL agencies, including the Office of the Solicitor and the Office of the Assistant Secretary for Policy, including review of the OMB package, review of the multi-year contract: $5,000; and
Review and publication of the two papers resulting from the evaluation: $4,000
15. Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB 83-I:
Not applicable. This is a new collection of information and will use 4,439 hours.
16. For collections whose results will be published, outline the plans for tabulation and publication:
Site Visit and Phone Reconnaissance Data: Phone reconnaissance responses will be recorded and detailed site-visit write-ups will be developed. The information from the phone reconnaissance and the site visits will be tabulated in detailed matrices. Cross-site analysis will be conducted to identify similarities and variations among the sites and to summarize findings on the AWI as a whole.
There will also be specific qualitative methods to evaluate critical elements of the AWI, such as:
Partnership Analysis. One of the most important components of the AWI is to catalyze partnerships that will create new or improved services by including other public agencies and private sector businesses. The evaluation will document and compare partnerships in different sites before and after the AWI grant award.
Curriculum Analysis. Some grantees are developing new curriculum, which will be reviewed rigorously, using standard educational review criteria, to determine originality and the extent to which course materials are tailored for specific aging worker sub-groups (such as those who may need remediation or specific skill upgrades). The review will also explore the extent to which curricular materials have benefited from recent industry association standards.
17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons why display would be inappropriate:
No such approval is sought and the expiration date and control number will be provided wherever appropriate.
18. Explain each exception to the certification statement identified in Item 19 of the
OMB 83-I:
Not applicable.
1 See Report on Statistical Disclosure Limitation Methodology, Subcommittee on Disclosure Limitation Methodology, Statistical Policy Office of the Office of Management and Budget, 1994.
1 Costs are derived from the Bureau of Labor Statistics (BLS), National Compensation Survey, 2008 from tables for Full-Time State and Local Government Workers and Full-Time Private Sector Workers. Costs for unemployed participants are derived from the 2008 national estimate for the cost of volunteer labor created by The Independent Sector, which is based on BLS data.
| File Type | application/msword |
| File Title | SUPPORTING STATEMENT |
| Author | Richard Roberts |
| Last Modified By | naradzay.bonnie |
| File Modified | 2011-10-13 |
| File Created | 2011-10-13 |