Supporting Statement A - CWCC Implemenation and Evaluation Templates

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Building Capacity to Evaluate Child Welfare Community Collaborations to Strengthen and Preserve Families (CWCC) Grantee Local Evaluation Plan and Implementation Plan Templates

Formative Data Collections for Program Support

0970 – 0531

Supporting Statement

Part A

February 2020

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers: Mary Mueggenborg and Laura Hoard

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a generic information collection under the umbrella generic, formative data collections for program support (0970-0531).

• Description of Request: The Children’s Bureau (CB), within the Administration for Children and Families (ACF), in the Department of Health and Human Services (HHS) funds cooperative agreements under the Child Welfare Community Collaborations (CWCC) initiative. As part of their agreements, grantees must 1) develop and execute a project implementation plan; and 2) develop and execute a local evaluation plan. This data collection clearance covers an 1) an implementation plan template that grantees can use to plan a project that meets CB requirements and 2) an evaluation plan template that grantees can use to systematically and thoroughly document and develop an evaluation that meets CB requirements. This information is for grantee, evaluation technical assistance provider, implementation technical assistance provider, and ACF use only. We do not intend for this information to be used as the principal basis for public policy decisions.

• Time Sensitivity: CWCC grantees are required to submit finalized implementation and evaluation plans to CB by July 31, 2020. Grantees are currently underway in planning for implementation and designing evaluations. These templates will help grantees in this process, so approval as soon as possible is requested.

A1. Necessity for Collection

As part of their cooperative agreements, grantees must design and execute their own project implementation plan and their own local evaluation plan. Grantee completion of the implementation and evaluation plan templates included in this information collection request will allow CB to ensure that the implementation and evaluation plans meet CB requirements as stated in the Funding Opportunity Announcement for the cooperative agreement.

There are no legal or administrative requirements that necessitate this collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose

Purpose and Use

This information collection provides CWCC grantees with a systematic way to document all required elements of their individual local evaluation plans and implementation plans, through use of an implementation plan template (Instrument 1) and evaluation plan template (Instrument 2). CB developed the implementation plan template alongside their implementation TA contractor, James Bell Associates (JBA). OPRE developed the evaluation plan template alongside their evaluation TA contractor, Abt Associates (Abt). The templates allow for a standardized review of all CWCC grantee evaluation plans and implementation plans by ACF and their evaluation TA contractor or implementation TA contractor. CB will approve grantees’ implementation and evaluation plans, to allow use of cooperative agreement funds for evaluation efforts and project implementation. The evaluation TA provider will use data collected to design and execute ongoing evaluation TA; the implementation TA provider will use the data collection to design and execute ongoing implementation TA; and grantees will execute their projects based on their implementation plans and local evaluations based on the information contained in their evaluation plans (as documented in the templates).

This proposed information collection meets the following goals of ACF’s generic clearance for formative data collections for program support (0970-0531):

• Planning for provision of programmatic or evaluation-related training or technical assistance (T/TA).

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker, and it is not expected to meet the threshold of influential or highly influential scientific information.

Research Questions or Tests

We do not have any research questions, nor are we conducting any tests. These templates are systematic ways to ensure all CWCC grantees have developed detailed implementation and evaluation plans to guide their work.

Study Design

The 9 CWCC grantees funded in September 2019 for 5-year cooperative agreements will be asked to complete the implementation and evaluation plan templates during the spring and summer of 2020, upon OMB approval. Final drafts of the implementation plans are due May 29, 2020. Final drafts of the evaluation plans are due on July 31, 2020. Grantees may be asked to revise and/or expand on the information provided in the templates to meet CB requirements for the CWCC’s local projects and evaluations. Once the final drafts are approved, grantees will submit a final version of their implementation and evaluation plans, using the templates as a guide. Data will be collected during this single time period of initial completion and revision.

The evaluation plan template follows best practices of evaluation plan development, documents components of evaluation execution that need to be explicated before implementing an evaluation, and provides quality indicators of each of those components to ensure grantees’ evaluations meet CB requirements for rigor and quality. Similarly, the implementation plan template follows best practices for implementation plan development and documents components of implementation that must be articulated before implementing a program to ensure grantees’ funded projects meet CB’s requirements for quality and completeness.

Data collected via the implementation and evaluation plan templates are not intended to be generalizable beyond the CWCC grantees, but will represent all of the proposed implementation and evaluation activities and rationales for all CWCC grantees.

Data Collection Activity	Instrument(s)	Respondent, Content, Purpose of Collection	Mode and Duration
Documentation of CWCC grantee implementation plans	Implementation Plan Template (Instrument 1)	Respondents: CWCC grantee staff Content: Components of their local implementation plans, including a theory of change, a detailed description of program activities and interventions, implementation teams, implementation readiness and capacity building activities, activities to initiate and sustain implementation, plans for quality assurance and continuous quality improvement, communication and dissemination activities, work plans, and implementation timelines Purpose: To support grantee’s implementation of their funded projects through systematic documentation of their implementation plans	Mode: Written responses in a template Duration: 10 hours
Documentation of CWCC grantee evaluation plans	Evaluation Plan Template (Instrument 2)	Respondents: CWCC grantee and their contracted local evaluator staff Content: Components of their planned local evaluation, including research questions, methodology, analysis plans, and data security/privacy plans Purpose: To support grantee implementation of their required evaluations through systematic documentation of their evaluation plans	Mode: Written responses in a template Duration: 8 hours

Other Data Sources and Uses of Information

Completed evaluation plan templates will inform evaluation TA efforts. The evaluation TA contractor also draws from information provided by grantees and their evaluators through oral communication and email requests, and guidance from ACF, to design evaluation TA.

Completed implementation plan templates will inform implementation TA efforts. The implementation TA contractor also draws from information provided by grantees through oral communication and email requests, and guidance from ACF, to inform and design implementation TA efforts.

A3. Use of Information Technology to Reduce Burden

Grantees and evaluators will receive Microsoft Word versions of the evaluation and implementation plan templates. They will be able to complete the templates by cutting and pasting information from their plans, and will be able to submit the documents electronically.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

As grantees are developing and implementing unique CWCC grant projects, and designing evaluations specific to their CWCC grant activities, no similar information is already available. No unnecessary information is being requested of program staff or grantees. These instruments do not ask for information that can be reliably obtained through other sources.

A5. Impact on Small Businesses

Some of the organizations grantees have contracted with to provide evaluation services may be small businesses. Providing editable templates and electronic submission of the completed evaluation plan template are designed to minimize burden on these entities. In addition, many grantees are partnering with state and local government entities (e.g., schools, courts) and local service providers in developing their implementation plans and implementing their projects. Providing an editable implementation plan template and allowing for the electronic submission of the completed template will minimize burden on these entities.

A6. Consequences of Less Frequent Collection

Data will only be collected once through each of the two plan templates. A nonsystematic collection of grantee implementation and evaluation plans would hinder CB review of plans for approval and may result in grantees’ failure to document and plan for important elements of their projects and respective evaluations.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the overarching generic clearance for formative information collection. This notice was published on October 11, 2017, Volume 82, Number 195, page 47212, and provided a sixty-day period for public comment. A subsequent notice, updated with more specific information, was published on June 18, 2019, Volume 84, Number 117, page 28307, and provided a thirty-day period for public comment. During the notice and comment periods, no substantive comments were received.

Consultation with Experts Outside of the Study

ACF consulted the project evaluation TA contractor, Abt Associates, and the implementation TA contractor, James Bell Associates, when preparing the templates.

A9. Tokens of Appreciation

No tokens of appreciation will be provided.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

The implementation and evaluation plan templates ask for information on organizational and individual roles and responsibilities related to program implementation and evaluation, respectively. Specific staff members may be listed by name. This information is limited to their professional responsibilities as employees of organizations participating in a CWCC grant. Staff names and affiliations are publicly available.

Assurances of Privacy

Respondents will be informed of all planned uses of data, and that their participation is voluntary. The submitted completed templates will not be shared beyond ACF and the TA contractors. Grantees may publicize their implementation and evaluation plans, so the information submitted within could potentially be made public. TA contractors may also present aggregated details from grantees evaluation plans, such as evaluation designs or instruments used, via presentations or written materials at an evaluation conference as context for how and why evaluation TA was provided.

Data Security and Monitoring

Grantees will submit their completed implementation and evaluation plan templates via email, which will be saved in secure project files. We do not intend to disseminate any non-aggregated or identifiable information collected via these templates.

A11. Sensitive Information ¹

No sensitive information will be collected.

A12. Burden

Explanation of Burden Estimates

We have estimated the total number of hours it will take to complete each template. We have estimated that two people will be involved in the completion of each template at each grantee, for a total of 36 people involved in the information collection. We expect each grantee response to take 10 hours for the Implementation Plan Template and 8 hours for the Evaluation Plan Template.

We estimate the average hourly wage for respondents to be $34.07, to be the average hourly wage of “social and community service managers” (11-9151) as determined by the U.S. Bureau of Labor Statistics National Occupational Employment and Wage Estimates (U.S. Department of Labor, May 2018; https://www.bls.gov/oes/current/oes_nat.htm#21-0000).

Estimated Annualized Cost to Respondents

Instrument	No. of Grantees	No. of Respondents per Grantee Response	No. of Responses per Grantee (total over request period)	Avg. Burden per Grantee Response (in hours)	Total/Annual Burden per Grantee (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
Implementation Plan Template (Instrument 1)	9	2	1	10	90	$34.07	$ 3,066.30
Evaluation Plan Template (Instrument 2)	9	2	1	8	72	$ 34.07	$ 2,453.04
Totals:	9	4	2	9	162	$ 34.07	$ 5,519.34

A13. Costs

There are no additional costs to respondents.

A14. Estimated Annualized Costs to the Federal Government

Cost Category	Estimated Costs
Instrument Development and OMB Clearance	$7,075
Field Work	$ 404
Publications/Dissemination	$ 0
Total costs over the request period	$7,479
Annual costs	$7,479

A15. Reasons for changes in burden

This is for an individual information collection under the umbrella formative generic clearance for program support (0970-0531).

A16. Timeline

Activity	Start Date	End Date
Grantees and their evaluators plan their local evaluation	9/30/19	7/31/20 or when evaluation plan template is approved
Grantees design their local projects and plan for project implementation	9/30/19	7/31/20 or when implementation plan is approved
Grantees and evaluators receive OMB approved evaluation plan template	Upon OMB approval
Grantees receive OMB approved implementation plan template	Upon OMB approval
Grantees complete and submit completed implementation plan template	5/29/20	5/29/20
Grantees and their evaluators submit completed evaluation plan template	7/31/20	7/31/20
ACF requests revisions to evaluation plan templates, if needed	8/31/20	12/31/20
ACF requests revisions to implementation plan, if needed	6/26/20	6/26/20
Grantees and their evaluators submit revised, completed evaluation plan templates, if needed	9/15/20	12/31/20
Grantees submit revised, completed implementation plan templates, if needed	7/31/20	7/31/20

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Instrument 1: Implementation Plan Template

Instrument 2: Evaluation Plan Template

1 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

10