Final OMB Supporting Statement Oct 6 2006.rtf

SUPPORTING JUSTIFICATION FOR OMB PACKAGE

THE OFFICE OF COMMUNITY SERVICES (OCS) EVALUATION INITIATIVE

NEW INFORMATION COLLECTION QUESTIONNAIRE

[OMB Control Number: 0970-XXXX/ Date: XX/XX/XXXX]

Debbie A. Powell

Acting Director

Administration of Children and Families (ACF)

Department of Health and Human Services (DHHS)

Office of Community Services (OCS)

370 L’Enfant Promenade, S.W. - 5^th Floor

Washington, DC 20447

Phone: (202) 401-6968

Fax: (202) 401-5726

[email protected]

Re-Submitted to Robert Sargis, OIS

Report Clearance Officer

On October 6, 2006

By Academy for Educational Development (AED)

Rose Ann M. Rentería, Ph.D.

Senior Research and Evaluation Officer at Tel. 202-884-8608

table of contents

PART A. justification 2

A1. Circumstances Making the Collection of Information Necessary 3

A2. Purpose and Use of Information Collection 3

A3. Use of Improved Information Technology and Burden Reduction 4

A4. Efforts to Identify Duplication and Use of Similar Information 4

A5. Impact on Small Businesses or Other Small Entities 4

A6. Consequences of Collecting the Information Less Frequently 4

A7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 4

A8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency 4

A9. Explanation of Any Payment or Gift to Respondents 5

A10. Assurance of Confidentiality Provided to Respondents 5

A11. Justification for Sensitive Questions 5

A12. Estimates of Annualized Burden Hours and Costs 6

A13. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency 6

A14. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers 7

A15. Annualized Cost to the Federal Government 7

A16. Explanation for Program Changes or Adjustments 7

A17. Plans for Tabulation and Publication and Project Time Schedule 7

A18. Reason(s) Display of OMB Expiration Date is Inappropriate 8

A19. Exception to Certification for Paperwork Reduction Act Submissions 8

PART B. Collection of INformation Employing STatistical Methods (THIS IS LINKED TO PART I TEMPLATE, Question NO. 12 , OMB Part 1 of ICRAS) 9

B1. Respondent Universe and Sampling Methods 9

B2. Procedures for the Collection of Information 9

B3. Methods to Maximize Response Rates and Deal with Non-response 9

B4. Tests of Procedures or Methods to be Undertaken 10

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 10

REFERENCE 11

FOUR APPENDICES (Submitted Under Separate Cover)

1. Attachment A: Final Federal Register Notice No. 1 (PDF)

2. Attachment B: One Public Comment to Federal Register Notice No. 1

3. Attachment C: Final Federal Register Notice No. 2 (PDF)

4. Attachment D: Cover letter to Grantees on OCS Initiative

SUPPORTING JUSTIFICATION FOR OMB PACKAGE

THE OFFICE OF COMMUNITY SERVICES (OCS) EVALUATION INITIATIVE

NEW INFORMATION COLLECTION QUESTIONNAIRE

[OMB Control Number: 0970-XXXX/ Date: XX/XX/XXXX]

PART A. JUSTIFICATION

A1. Circumstances Making the Collection of Information Necessary

One of the current priorities at the OCS is to improve performance and accountability. OCS leadership has been very clear about monitoring the programs more closely in order to better measure success and to understand and replicate the programs that excel. OCS works in partnership with states, communities, and other agencies to provide a range of human and economic development services and activities, which ameliorate the causes and characteristics of poverty and otherwise assist persons in need. OCS continues to have great success in the realm of poverty reduction and community development. However, measuring that success systematically has not always been easy. Thus, OCS is in the process of creating a more performance-based environment, with greater emphasis on accountability and achieving results. Movement in the direction of evaluation has gained considerable momentum with the implementation of the Office of Management and Budget’s (OMB) Performance Assessment Rating Tool (PART), but is anchored in the programs’ legislative requirements.

Legislative Requirement

This questionnaire is part of a contract that addresses evaluation strategies for three programs administered by Office of Community Services (OCS): Community Economic Development (CED), Rural Community Facilities (RF), and Job Opportunities for Low-Income Individuals (JOLI). The Legislative requirement for two of these programs, i.e., the RF and CED programs, is in Title IV of the Community Opportunities, Accountability, and Training and Educational Services Act (COATS Human Services Reauthorization Act) of Oct. 27, 1998, Pub. L. 105-285, section 680(b) as amended. This legislative directive states that “The Secretary shall require all activities receiving assistance under this section to be evaluated for their effectiveness. Funding for such evaluations shall be provided as a stated percentage of the assistance or through a separate grant awarded by the Secretary specifically for the purpose of evaluation of a particular activity or group of activities.” Under Title V, section 505, of the Family Support Act of 1998, Pub. L. 100-485, section 505(f), JOLI was initially a demonstration program that required local evaluations of each project. When JOLI was reauthorized in 1996 (Pub. L. 104-193--Aug. 22, 1996), it no longer had demonstration status and evaluation requirements. As a result, a formal evaluation for the JOLI programs has not been conducted since the 1996 Pub. L. reauthorization. At this time, OCS is interested in a formal evaluation to assess the JOLI program.

Achieving OCS’ Stated Priorities and OMB PART

OCS has chosen to evaluate all three of these programs through a separate contract awarded by the Secretary using the Office of Management and Budget’s (OMB) Performance Assessment Rating Tool (PART) in order to critically and comprehensively review the overall design and effectiveness of each program. The evaluation initiative contract provides the central office with the mechanism to ensure that all programs evaluated will have consistent data that is in agreement with the direction of OMB and provides the Secretary with information on program efficiency and effectiveness.

A2. Purpose and Use of Information Collection

The primary purposes are to document and systematically evaluate the program performance of two OCS discretionary grant programs in qualitative and quantitative terms. Each of the OCS discretionary grant programs – CED and JOLI – will be assessed using qualitative and quantitative evaluation methods that capture key information about program and grantee-level performance in four general areas: (1) program purpose and design; (2) strategic planning; (3) program management; and (4) program results. The evaluation activities will expand data collection and analysis to improve the validity and generalizability of evaluative and program impact findings. The data collected with this questionnaire will provide guidance for OCS in creating future program announcements with new evaluation definitions and expectations so that future CED and JOLI grantees will have evaluation plans that will produce PART level outcomes. With the results from the survey OCS will be able to target evaluation training at specific levels of grantees and to make adjustments to the requirements based on the grantees capabilities. The questionnaire data provides baseline data for future evaluations.

A3. Use of Improved Information Technology and Burden Reduction

The questionnaire will be administered using a web-based questionnaire for electronic submission. This technique does not require paper from the respondents. To reduce burden on respondents, the questionnaire is designed to last on average 1.5 hours, and respondents are instructed not to use more than three hours to complete the questionnaire or to conduct new data collection projects. The subcontractor (Academy for Educational Development, AED), will assist respondents that encounter information technology barriers to reduce paper submission and will conduct telephone interviews as requested; the questionnaire only will be used to collect the survey data during this telephone conversation as no other interviewing steps will be employed.

A4. Efforts to Identify Duplication and Use of Similar Information

The collection of information avoids unnecessary duplication. There are no similar data available from other studies because this evaluation is specific to OCS programs and OCS grantees. Although some of the grantee performance data requested is similar to the data included in the grantee annual reports, the questionnaire contains more specific performance measures than currently required by OCS. It will provide a significantly more detailed and accurate picture of how the grantees are performing. A review of grantee’s annual reports revealed that the reports are not uniformly submitted and therefore the data varies between reports. Standardized reporting is necessary to compile accurate program data.

A5. Impact on Small Businesses or Other Small Entities

The collection of information reduces burden on small entities. The evaluation has been designed with the minimal burden on the grantees by creating an electronic form and requiring no new data collection. The questionnaire should not take longer than an hour to complete and grantees will be instructed not to take over 3 hours. They will also be instructed to only gather data from their current and accessible files and not to call participants or new business to gather new information, but reporting only on the data they have already collected. The questionnaire is a snapshot revealing both how many jobs the grantees have created and how many grantees are already collecting the data to provide baseline data for future evaluations.

A6. Consequences of Collecting the Information Less Frequently

Respondents will be asked to participate in a one-time only questionnaire. One of the benefits of using close- and open-ended questionnaires is assuring that all pertinent information is gathered at one time. Respondent’s responses will provide comprehensive information on program impacts and experiences during a 45 minute to 1 hour questionnaire administration, eliminating the need for multiple data collections thereby reducing the burden on the respondent. There are no legal obstacles to reducing the burden.

A7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

This data collection fully complies with the guidelines of 5CFR 1320.5.

A8. Comments in Response to the Federal Register Notice and Effort to Consult

Outside the Agency

a. Federal Register Notices

The first notice required in 5 CFR 1320.8 (b)(3) was published in the Federal Register, Volume 71, Number 2, pages 348-349, on January 4, 2006. A copy of this notice is provided in Attachment A. One public comment was received on January 4, 2006. See Attachment B for the comment. The public comment is beyond the scope the evaluation initiative as it deals with illegal migration to the U.S.

The second notice required in 5 CFR 1320.8 (b)(3) was published in the Federal Register, Vol. 71, Number 2, page 28869, on May 18^th 2006. A copy of this notice is provided in Attachment C. No public notices have been received by the contractor (AED) as of October 5, 2006.

b. Effort to Consult Outside the Agency

Efforts were undertaken to consult a CED and JOLI content and program expert, Dr Mark Lelle, who has worked with AED for two years, to obtain his views on the availability of data, the clarity of instructions, disclosure, and the development of the proposed information collection.

A9. Explanation of Any Payment or Gift to Respondents

The respondents will not require a gift or payment.

A10. Assurance of Confidentiality Provided to Respondents

The data collected from the questionnaire from CED and JOLI grantees will be treated in a confidential manner. The topics focus on the program outcomes and grantee experiences. There are no questions of an inherently sensitive nature in the questionnaire, but it is possible that some grantees could see any questions about their projects, asked of them by the OCS, as a type of evaluation of their project work. This possibility will be allowed for by having a contractor (Academy for Educational Development) rather than OCS staff respond to questions about the questionnaire and conduct telephone conversations due to information technology barriers with grantees, and by indicating to the grantees in the invitation letter (contained in Attachment C) and at the time of any telephone conversations that: their participation is voluntary and will not affect how they are evaluated on the project they had funded through the OCS nor their future eligibility for funding from OCS, they may refuse to answer any questions, names and all other identifying information will be removed from reports before they are shared with OCS or others; and the comments of all respondents will be combined in reports and documents about the project so that no comments can be linked with any particular project. The contractors will also tell the grantees that no OCS employee will listen in on telephone conversation that may occur.

Each respondent will be assigned a unique identifier that will be substituted for all identifying information in the survey data results. Lists linking names to unique identifiers will also only be available to AED and will be kept in a secure location. Both hard and electronic copy versions of the questionnaires with identifying information will be kept in a locked data storage room and on a secure password-protected network, respectively, only accessible to researchers at the contracting organization, AED. All files will be maintained at AED for the duration of the project. Electronic and hard copies of the questionnaires with identifying information and the lists linking respondent identifying information to unique identifiers will be destroyed upon completion of the project. AED will maintain copies of the questionnaires from which the identifying information has been removed for 7 years.

A11. Justification for Sensitive Questions

There are no questions of an inherently sensitive nature included in this data collection (See the CED and JOLI questionnaires, which are attached to this request package).

A12. Estimates of Annualized Burden Hours and Costs

a. Burden Hours

The burden estimates is stated in the final Federal Register Notice No. 1 for the OCS evaluation initiative (See Attachment A):

Annual Burden Estimates

---------------------------------------------------------------------------------------

Instrument Number of Number of Average burden Total

Respondents responses per hours per burden

Respondent response hours

---------------------------------------------------------------------------------------

Questionnaire for OCS

--CED Grantees 147 1 1.5 220.5

---------------------------------------------------------------------------------------

Questionnaire for OCS

--JOLI Grantees 25 1 1.5 37.5

---------------------------------------------------------------------------------------

Estimated Total Annual Burden Hours: 258

b. Annualized Cost

The project’s cost burden is presented in Table 1. The hourly wage rate is derived from statistics provided by the 2005 National Compensation Survey (Department of Labor, 2005). The OCS grantees are mostly all executives working as directors of a community-based organization. The mean hourly rate for the respondents was therefore calculated as the overall mean of the mean hourly rate for senior executives in community-based organizations (mean of $36.22).

Table 1—Annualized Burden Hours
Type of Respondent	Number of Respondents	Number of Burden Hours	Hourly Wage Rate	Total Respondent Cost
CED Grantees	147	1.5	$36.22	$7.987
JOLI Grantees	25	1.5	$36.22	$1,358
				$9,345

A13. Comments in Response to the Federal Register Notice and Effort to Consult

Outside the Agency

a. Federal Register Notices

The first notice required in 5 CFR 1320.8 (b)(3) was published in the Federal Register, Volume 71, Number 2, pages 348-349, on January 4, 2006. A copy of this notice is provided in Attachment A. One public comment was received on January 4, 2006. See Attachment B for the comment. The public comment does not provide a specific question or concern to address as the comment focuses on illegal migration in the U.S., which is beyond the scope the evaluation initiative.

The second notice required in 5 CFR 1320.8 (b)(3) was published in the Federal Register, Vol. 71, Number 2, page 28869, on May 18^th 2006. A copy of this notice is provided in Attachment C. No public notices have been received by the contractor (AED) as of October 5, 2006.

b. Effort to Consult Outside the Agency (OCS)

Efforts were undertaken to consult a CED and JOLI content and program expert by the contractor to obtain his views on the availability of data, the clarity of instructions, disclosure, and the development of the proposed information collection. Specifically, Dr. Mark Lelle, a consultant in evaluation and planning, was consulted. Dr. Lelle has worked with the contractor (AED) for two years conducting site visits at CED and JOLI sites.

A14. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers

Neither respondents nor record keepers are subject to capital and start-up costs or operation and maintenance costs and purchase of services.

A15. Annualized Cost to the Federal Government

The primary cost to the Federal government for conducting this project is the cost of securing a contractor to carry out the bulk of the project tasks. The costs associated with hiring a contractor to complete this study are estimated at $1,479,514 (for three program years: Base Year and Two Option Years. From September 2004-2005, Base year ($599,703) was exercised. Option Year 1 ($466,648), was exercised from September 2005 to September 2006; and, Option Year 2 ($453,163) is currently being exercised at this time until September 2007. The total annual cost to the government is therefore $493,171 ($1, 479,514/3). All costs related to the study have been funded entirely out monies set aside for the administration, management, and evaluation of the OCS programs, including CED and JOLI programs.

A16. Explanation for Program Changes or Adjustments

This project is a new data collection.

A17. Plans for Tabulation and Publication and Project Time Schedule

a. Tabulation

Data Entry. Each questionnaire submitted by the respondent will be coded into an Excel spreadsheet, which will be automatically updated by the AED information technology staff. An electronic and hard copy of each questionnaire will be delivered to the AED research and evaluation staff by the AED information technology staff. Upon receipt, each hard copy questionnaire will be logged in and then stored in a secure data storage room. Electronic questionnaires will be created in a standard word processing program—Microsoft Word. The respondent’s name will then be removed from each electronic file and replaced with a unique identifier. The resulting electronic file will be saved on a secure directory available only to the research staff, while the original disk and the hard copy of the questionnaire will be stored in a secure place. Each electronic questionnaire will be reviewed, and all identifying information throughout the questionnaire will be removed.

Data Analysis. Quantitative and qualitative content analysis will be conducted to determine the key themes rising from respondent’s comments around program impact and experiences. Toward this end, each questionnaire will be uploaded into Excel and SPSS, a quantitative data analysis software package, and qualitative data will be coded. The initial coding structure uses descriptive codes based on the questions in the questionnaire. Each grouping of comments from a respondent will therefore be identified as relating to the plan for dissemination and translation of their research results.

b. Plan for Publication

One type of formal documentation will be required. This will consist of a detailed report, with an associated executive summary, that can be shared throughout OCS and elsewhere in the federal government. This report, and especially the executive summary, will have as their primary focus concrete findings and recommendations for the federal government.

c. Project Time Schedule

Table 2—Project Time Schedule presents the projected timeline for scheduling the field administration of the questionnaire, conducting telephone interviews using the questionnaire for grantees with information technology barriers, data analysis of the survey, and preparation of reports.

Table 2—Proposed Project Time Schedule
Activity	Number of Months After OMB Approval
Mail introductory letters and schedule telephone interviews using the questionnaire to collect survey data from grantees with information technology barriers	1
Field questionnaire and conduct telephone interviews noted above	2-3
Data analysis	3-5
Production of reports for OCS	5-6

A18. Reason(s) Display of OMB Expiration Date is Inappropriate

Not applicable. The data collection instrument will display the OMB approval expiration date for this data collection.

A19. Exception to Certification for Paperwork Reduction Act Submissions

We are not seeking exception to certification for the Paperwork Reduction Act Submission for this data collection.

PART B. Collection of INformation Employing STatistical Methods

This submission requests OMB clearance to enable the gathering of information from the 172 OCS grantees whose CED and JOLI projects were funded with start dates between fiscal years 2001 and 2004. It is the intent of this project to survey as many of these OCS grantees and, thus, no statistical sampling methods are employed. Information will be collected via the use of a closed- and open-ended questionnaire. Descriptive statistics and content and thematic analysis will be used to conduct quantitative and qualitative analysis of the data. Thus both statistical and non-statistical information on the respondent and the information collection procedures for the project are described below.

B1. Respondent Universe and Sampling Methods

This data collection does not employ statistical methods as all active CED and JOLI grantees with start dates between 2001 and 2004 will be surveyed. This project will attempt to survey the active OCS grantees (i.e., lead grantee contact persons identified by OCS) of all 172 CED and JOLI projects with start dates between fiscal years 2001 and 2004. It is preferable to interview all of the grantees rather than a sample for several reasons. First, since OCS funding amounts and project time lines change from year to year and from project to project, there is a great variety of CED and JOLI projects that make up the currently active CED and JOLI grantee sites. Second, for CED projects, there is a desire to survey over 100 grantee leads to strengthen the statistical methods (e.g., descriptive statistics) employed. Third, for JOLI projects, there are a small number of active grantees that can respond to the questionnaire, making it desirable for researchers to survey as many JOLI grantees leads, with the desire to survey 80 percent or more grantees to truly understand the full range of experiences.

B2. Procedures for the Collection of Information

This data collection does not employ statistical methods as all active CED and JOLI grantees with start dates between 2001 and 2004 will be surveyed. Data will be collected through the close- and open-ended questionnaire, which will be administered on the Internet web or by US mail distribution. For surveys administered via the web, the web site for the OCS evaluation initiative will be used, which has been developed by the Academy for Educational Development (the evaluation contractor to OCS) with guidance from key OCS staff in 2005 and 2004. The OCS grantee lead per CED and JOLI projects will receive a letter from OCS requesting their involvement in the project (See Attachment D). Each cover letter will direct respondents to the OCS evaluation initiative web site and provide a unique password to ensure that the lead grantees complete the correct questionnaire section. An additional, duplicate letter from OCS will be used as necessary towards the total response rate of 80 percent or more per OCS program. For grantee leads unable to use web and/or information technology, an AED evaluator will schedule a telephone interview time to administer the questionnaire and to collect the data. All interview data will be entered in the web-based questionnaire by AED staff as needed. Each section of the questionnaire corresponds to several core topics such as CED and JOLI programmatic impacts and outcomes, performance measurement and reporting, and a self-assessment regarding grantee organizational capacity.

B3. Methods to Maximize Response Rates and Deal with Nonresponse

In order to encourage as many grantees as possible to participate in this project, lead grantees will receive a letter from OCS requesting their involvement in the evaluation with user-friendly instructions on the http address, password usage, and how to begin the questionnaire. If a lead grantee does not respond to the initial communications, the OSC Federal Program Officer for the evaluation initiative will send the letter a second time to the lead grantee, encouraging his or her participation in the project. As needed, AED will schedule a telephone interview with lead grantees to complete the questionnaire when information or computer technology is not present at the grantee site location.

B4. Tests of Procedures or Methods to be Undertaken

The questionnaires are reviewed and approved by key OCS staff, including Debbie Brown (OCS’ CED team leader) and Thom Campbell (OCS’ JOLI leader), which was completed on February 23, 2006, and March 13, 2006, respectively. The questionnaires were internally and externally pre-tested by March 31, 2006, by an AED information technology and web design specialist. The pre-testing was done with internal AED volunteers until no additional usability enhancements could be suggested, and no usability concerns were raised. There were two valid external tests—one volunteer CED grantee and one volunteer JOLI grantee (one of the original 2 CED testers was unable to login due to the fact that this was on a development server which required two separate logins). This will not be an issue when the survey is launched. The CED tester said that aside from the login issue, he felt that the survey was user-friendly. The JOLI tester reported that ‘overall - the survey looks really good’ and ‘the usability is great.’ This tester completed the survey in less than 20 minutes.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

This data collection does not employ statistical methods. OCS has contracted with the AED, under the direction of Dr. Rose Ann Rentería, to help design the data collection, conduct the follow-up e-mails and telephone interviews, and assist with the analysis of the data. Dr. Rentería can be reached as (202) 884-8608. Ms. Debbie A. Powell, Federal Project Officer, and key OCS staff members (Debra Brown and Thom Campbell) participated in the design of the data collection questionnaires. Ms. Powell will be responsible for receiving and approving AED contract deliverables. She can be reached at (202) 401-6968.

REFERENCE

Department of Labor; U.S. Bureau of Labor Statistics (2005). National Compensation Survey: Occupational Wages in the United States, July 2004. [Summary 05-04.].

13