PACT - Field Data Collection - ICR Supporting Statement A - Final - 4-19-12

Parents and Children Together (PACT) Evaluation:

OMB Supporting Statement

for the Data Collection Necessary to Select Grantees for the Study

Part A: Justification

January 18, 2012

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

7^th Floor, West Aerospace Building

370 L’Enfant Promenade, SW

Washington, D.C. 20447

Project Officers:

Nancye Campbell

Seth Chamberlain

This information collection request (ICR) is for clearance to collect information for the Parents and Children Together (PACT) evaluation of a subset of Responsible Fatherhood (RF) and Healthy Marriage (HM) grants authorized under the Claims Resolution Act of 2010 (public law 111-291). The Responsible Fatherhood and Healthy Marriage (RFHM) grantees represent the “next generation” of grantees that build on what has been learned by earlier grantees and take a more comprehensive approach to serving families. The evaluation is being undertaken by the U.S. Department of Health and Human Services, Administration for Children and Families (ACF) and is being implemented by Mathematica Policy Research and its partner, ICF International.

Work under PACT will be carried out in stages with different types of information collection in each stage. Thus, requests for clearance will be submitted in stages as work progresses. This first submission requests clearance for “field data collection,” that is, to collect information from grantees and key partners that will inform selection of a subset of grantees for evaluation. The information will be collected via telephone calls and in-person conversations either at grantee meetings or at the grantees’ organizations. The submitted discussion guide, if approved, will be used for this information collection. Subsequent submissions will request clearance for further data collection instruments (e.g. baseline and follow-up instruments for the impact study; e.g. interview protocols for the implementation and qualitative studies). These instruments will be developed after we select the grantees to be evaluated and additional design work has been completed.

While this document requests clearance only for the data collection necessary to inform grantee selection, it discusses the entire plan for the study. Because the study is still in its design phase, some of the details of the plan may change, as each stage may influence subsequent stages.

1. Circumstances Making the Information Collection Necessary

a. Policy Background

The past several decades have witnessed sweeping changes in family structure. In 1980, 77 percent of children lived with two married parents; by 2010, this figure had fallen to only 66 percent (U.S. Census Bureau 2011). Families have also become more complex: nearly one in five fathers now has children with more than one woman (Guzzo and Furstenberg 2007). While many children do well living only with one parent, research suggests that on average children do better when they have two involved parents (McLanahan 2009). These changes in family structure, their attendant consequences for children, and recent changes in welfare policy set the stage for new investments in family strengthening programs and in policy research on fatherhood and marriage.

As one response by the federal government, the Deficit Reduction Act of 2005 created the Responsible Fatherhood (RF) and Healthy Marriage (HM) grant program, authorizing ACF to provide up to $50 million for RF grants and $100 million for HM grants each year from 2006 to 2010. This funding represented an “unprecedented financial commitment by the federal government to support marriage and fatherhood programs” (U.S. Government Accountability Office 2008). Under this Act, awards were made to 226 grantees to provide one or more of eight allowable healthy marriage services or responsible fatherhood services including parenting classes, marriage and relationship education, and economic stability services.

The Claims Resolution Act of 2010 re-authorized this grant program, evenly allocating the $150 million between responsible fatherhood and healthy marriage funding ($75 million for each). New 3-year grants were awarded in September 2011 to 55 RF and 60 HM grantees.

b. RFHM Grants

RF grantees funded by the most recent re-authorization of the program must offer services in each of three categories: (1) healthy marriage; (2) responsible parenting; and (3) economic stability. HM grantees must provide one or more of eight allowable activities, which include marriage and relationship skills programs, mentoring, divorce reduction programs, and education in high schools on the value of marriage, relationships skills, and budgeting. The solicitations for applications for both RF and HM grants “strongly encouraged” applicants to offer more comprehensive services than just the traditional healthy marriage or responsible fatherhood services. They also encouraged partnerships with other organizations in the community to provide these services. The solicitations also stated that ACF had a “particular interest” in parents who are eligible for, are receiving, or have received TANF previously, and other low-income and at-risk parents including immigrants and refugees.

c. Prior Research

With regard to evaluations of RF programs, few rigorous studies of RF programs have been conducted. Of the 60 impact and implementation studies of programs for low-income fathers included in a recent systematic review of the evidence (Avellar et al. 2011), only 13 used a rigorous evaluation design. These program evaluations generally showed at least one statistically significant favorable impact, but most did not result in a compelling pattern of positive impacts.

With regard to HM evaluations, beginning in 2002, ACF sponsored two large-scale, multi-site evaluations utilizing random assignment of enrolled couples to evaluate the effects of programs offering healthy relationship and marriage skills and supportive services to unmarried parents having a child together (Building Strong Families [BSF]) and of similar services for low-income married couples with children (Supporting Healthy Marriage). ACF also sponsored an evaluation of community-wide healthy marriage programming (Community Healthy Marriage Initiative) utilizing a quasi-experimental design. Interim results from the BSF evaluation were released in May 2010 showing no significant differences when data from all eight programs were aggregated; however, in site-specific analyses, a positive pattern of impacts in one site and a negative pattern in another were observed (Wood et al. 2010). A report on the 36-month impacts is expected in 2012. Interim (12 month) impact findings from the Supporting Healthy Marriage evaluation and final results from the Community Healthy Marriage Initiative evaluation are expected in early 2012.

d. Current Request: Field Data Collection

The PACT Evaluation will build on prior evaluations by ACF and others on RF and HM services to low-income couples and fathers. Both RF and HM grantees may be evaluated under PACT.

The overall objective of the PACT evaluation is to document and test the “next generation” of RFHM programs. To meet that objective, the evaluation has two major, inter-related components (discussed in section “1e” below): (i) experimental impact studies with complementary implementation and qualitative studies; and (ii) qualitative/implementation studies.

As previously mentioned, this request relates to a “field data collection,” that is, the collection of information from grantees and key partners that will inform selection of a subset of grantees for the impact/implementation studies and the qualitative studies. It is expected that qualitative data collection will be conducted in all impact/implementation sites, to assist in better understanding impact findings. It is expected that an additional, separate set of programs would be recruited as qualitative/implementation study sites without impact data collection (e.g. when those programs do not meet criteria for impact evaluations, but still present some particular feature of program design or target population that warrants detailed study – see next section on “Site selection criteria”). However, this approach may be revised at a later point, based on what is found through the field data collection.

Site selection criteria. The policy interest of the grantee’s service delivery approach, the population targeted, or the context will be among the primary factors in site selection for both the impact/implementation and qualitative components. Specifically, grantees that demonstrate prior experience and a well developed approach to helping low-income individuals attain and sustain employment, including the use of subsidized employment, and grantees that demonstrate strong partnership with the child support agency to actively review child support orders and assist with modifications, as appropriate, are of high policy interest to ACF. Most grantees identify key characteristics of the population they expect to serve such as incarcerated parents, recently released parents, or predominantly low-income Hispanic or African American, for example. Final decisions about sites to be included in the impact/implementation study will give more weight to sites serving populations that are similar, in order to support attainment of adequate samples for sub-group analyses. That is, all things equal, a grantee serving multiple small populations of refugees would have lower priority for consideration as an impact/implementation study site, since we will not be able to conduct analyses of such small subgroups. The operational context within which the grant program is operated will also be considered in the site selection process. Grantees operating in contexts similar to that of many other grantees, as compared to very unique settings or operating structures, will be given greater consideration for selection into the impact/implementation study to facilitate the translation of the impact/implementation findings to the broader field. For example, a grantee operating within a therapeutic mental health facility is not as common as grantees operating programs within multi-service community based organizations.

An impact/implementation study site will also need to meet the following three key criteria: (1) a random assignment evaluation can be successfully implemented at the grantee’s program—it must be possible to collect the necessary baseline information, to insert random assignment into the program’s intake procedures, and to form a control group that does not receive the same or similar services to those offered the program group; (2) the program must be able to enroll enough participants to meet sample size requirements; and (3) it must be plausible that the program can lead to impacts that are detectable with the planned sample size (estimated to be at least 400 study enrollees per site – power analyses are offered in Supporting Statement B).

Grantees recruited into the qualitative study, and not into the impact/implementation study, need not meet the criteria for being an impact/implementation study grantee, but will have to present some particular feature of program design or target population that warrants detailed study. Grantees that demonstrate commitment to improving father-child relationships through the inclusion of multiple activities and services such as substantial hours of parenting education and fathering skill building and facilitated father-child activities are a high priority for ACF. Grantees that specifically address fathers’ disconnection from and difficulty with traditional service systems and include approaches to help men overcome isolation and offer navigational supports such as intensive case management and peer supports are also of high priority for ACF. Given the difficult economic environment, additional learning through the qualitative study from participants in programs that have a strong focus on employment and job placements in in-demand jobs (but do not meet other criteria for inclusion in the impact/implementation study) are of primary interest. Examples of target populations that may warrant detailed study include noncustodial fathers, fathers with multiple partner fertility, i.e., adults who have had children with multiple partners, and incarcerated or recently-incarcerated men.

While the aforementioned criteria are most salient to ACF prior to discussions with grantees, these discussions may lead ACF to tailor or adjust criteria as appropriate to the field of grantees available for evaluation. Updated criteria, resulting from these discussions, will be included in subsequent ICRs.

Site selection process. The process of selecting grantees for the study will involve stepwise winnowing of potential study candidates. To minimize burden on grantees, the first winnowing will be based on careful reading of the applications, information obtained on the web, and other available information on grantees. Based on this information, some grantees will be determined to be unlikely candidates for study (for example, if the grantee will not be able to recruit a sufficient number of participants). However, for the grantees not excluded based on this information, more information will need to be obtained to determine whether they are suitable for inclusion in the study (for example, whether the program design or target population warrants detailed study).

Per the current request to OMB, further information on each grantee will be obtained by semi-structured discussions with administrators or managers of RFHM grant programs and their proposed community partners slated to provide substantial services to participants. The submitted instrument, a discussion guide, will be used to guide these semi-structured discussions. The initial “script” at the beginning of the Discussion Guide, which relays the authority of the collection, purpose, use, voluntary nature, and privacy offered, will either be read to respondents (e.g. during telephone discussions) or distributed to respondents (e.g. via email prior to discussions, or by hand prior to beginning discussions). With the exception of the script, it is not expected that all the information on this discussion guide will be collected from all grantees or partners. Some of the information will already have been collected from the application or other sources. Moreover, if in collecting the information we find that a program will not meet the evaluation criteria, at that point no further information will be collected from the grantee or partner agency. Table A.1 summarizes the data collection proposed with this ICR.

Table A.1. Summary of Data Collection Plan – Field Data Collection – CURRENT REQUEST

Activity	Respondents	Mode	Schedule
Selection of Study Grantees
Discussions	Grantee (or partner organization) staff	Telephone (or in-person) discussions	Upon approval through Fall 2012

Table A.2 links all questions posed in the discussion guide with the site selection criteria.

Table A.2. Site Selection Criteria and Discussion Guide Question – Field Data Collection – CURRENT REQUEST

Criterion	Information Needed	Question
Random assignment can be successfully implemented Baseline information can be collected Possible to insert random assignment into intake period Similar services not available to control group	Strength of management Existence of current data collection procedures, or evidence of involvement in similar data collection Details of intake process Availability of other services in the community	A1-2 C11 C3 G1-3
Program must be able to enroll a sufficient sample size	Target number for recruitment, participation, and completion, including especially: sites serving populations that are similar, in order to support attainment of adequate samples for sub-group analyses Number of recruitment staff and whether vacancies are filled	C6-10 B3-6, D2
Plausible that program can lead to detectable impacts	Program retention Experienced management, including especially: prior experience and a well developed approach to helping low-income individuals attain and sustain employment History of providing services Partnerships, including especially: partnership with the child support agency Staffing Locations and hours Program funding Challenges	C8-10 A1-A3 B1-2, C1-2 B7-8 D1-7 C12-13 E1-3 C14, C16, H1
Program/population warrants detailed study	Types of services, curriculum used, including especially: grantees operating in contexts similar to that of many other grantees; grantees that demonstrate commitment to improving father-child relationships; grantees that specifically address fathers’ disconnection from and difficulty with traditional service systems; and grantees that have a strong focus on employment and job placements in in-demand jobs Partnerships Population served, including especially: noncustodial fathers; fathers with multiple partner fertility, i.e., adults who have had children with multiple partners; and incarcerated or recently-incarcerated men. Community context	C3-4 B7-8 C5 F1-6

The discussions will be conducted by staff at Mathematica and ICF International who have been trained on the discussion guide and study protocols. The discussions will occur either by telephone or in person.

e. Overview of the Remainder of the PACT Evaluation

As mentioned previously, the PACT evaluation has two major, inter-related components: (i) experimental impact studies with complementary implementation studies; and (ii) qualitative studies.

i. Experimental Impact and Complementary Implementation Component

We expect to select no more than 15 grantee programs for the impact/implementation evaluation component. The minimum number anticipated to be included is four, though this will be determined after discussions with grantees. The actual number of grantees to be included will be determined by the estimated sample size that can be generated by the set of grantees. We anticipate including a set of grantees that are offering strong programs and that, combined, will generate at least 1,800 sample members over two years. We anticipate that this will be approximately four grantees with samples of between 400 and 500. Past evaluations have demonstrated effect sizes of 0.1 or greater on relationship outcomes, and between $1,000 and $2,500 in increased income. A sample of 1,800 will position the evaluation to detect impacts of this size:

Table A.3. Estimated Minimum Detectable Differences in Study

Sample size (T and C)	Effect size	Annual Earnings
400	0.20	$2,754
600	0.16	$2,265
800	0.14	$1,962
1,800	0.09	$1,308
2,500	0.08	$1,110
3,600	0.07	$925

Assumptions: response rate = 80%; R-squared = 50%; control group earnings = $15,000

Furthermore, a sample of 1,800 will permit subgroup analyses of approximately 25%, or 400. Power analyses and justification for findings effects with samples of 400 are provided in Supporting Statement B, question 2.

A primary goal of this component is to provide rigorous estimates of the effectiveness of the programs. It will address: (1) whether RFHM programs are effective at improving multiple family related outcomes of participants and their families; (2) whether program effectiveness varies by population served; and (3) whether effectiveness varies by program type (e.g., structure, services provided, how services are implemented, or community context). Information collection activities for the impact/implementation component are numbered below, and then detailed in Table A.2.

This component will use an experimental design. Program applicants who are interested in and eligible for a RFHM program will be randomly assigned to either a program group which will be offered participation in the program, or a control group which will not be offered participation in the program. We expect in most RF sites that the unit of random assignment will be individuals (fathers) rather than couples. In most HM sites the unit of random assignment will be couples rather than individuals. However, this will be customized to reflect the target population (couples or individuals) of a specific site – and in all cases random assignment of the unit of analysis will be conducted..

The plan is for the grantees to begin random assignment in fall 2012 and continue random assignment for about two years. About 400 fathers or couples are expected to be randomly assigned in each evaluation grantee program. (Statistical power is discussed in Part B.)

Information will be collected twice for the impact component. First:

1. Baseline survey in person or via audio-computer assisted survey (ACASI). A survey will be conducted of fathers or members of couples, as appropriate, in both the program and control groups prior to random assignment.

Second, follow-up data will be collected from sample members at about 12 months after enrollment in the program. These follow-up data will include:

2. Follow-up survey, via a computer-assisted telephone interview (CATI). A survey will be conducted of fathers or members of couples, as appropriate, in both the program and control groups.

3. Participant-completed diaries. The diaries will consist of easy-to-complete checklists and closed-ended questions about interactions with partners or children. The advantage of diaries is that they can collect data that are sensitive to the intervention and do not suffer from recall bias. We will ask study participants to collect data for about one week.

4. Observations in the study participant’s home. Direct observations of interactions can provide rich information unavailable from self reports. Our current plan is to observe father-child interactions.

5. Administrative data. Rich information is available from child support enforcement records collected by state agencies and from earnings and employment data from the National Directory of New Hires.

RFHM programs could affect a wide range of outcomes. Outcomes of interest may differ by grantee but will include:

• Service receipt. Many of the services provided by RFHM programs are also available in the community. Hence, it will be important to collect data on the services received by members of both the program and control groups.

• Father involvement. This includes paternity establishment, time spent with the child, activities conducted with the child, and formal and informal child support.

• Child support. This includes whether there is a child support order in place, the amount of the child support order, amount paid, and the extent of child support debt and arrears.

• Parenting and co-parenting. This includes the quality of parenting, stress of parenting, and trust in and communication with the other parent of child.

• Parent relationships. This includes measures of the parents’ relationship status (for example, whether married, divorced, cohabiting, romantically involved, or not romantically involved but co-parenting) as well as the quality of the parents’ relationship.

• Economic self-sufficiency. This includes employment, earnings, hours worked, and the quality of the job.

• Family and parent well-being. This includes living arrangements, family functioning, material hardship, fertility, as well as the parents’ physical and mental health, substance abuse, criminal activity, and social support.

• Attitudes, expectations, knowledge, and skill acquisition. The programs could affect attitudes towards marriage, commitment, or fatherhood, expectations about future relationships, knowledge of parenting and the importance of fathers, and the acquisition of relationship and parenting skills.

The goal of the complementary implementation study is to provide a detailed description of the grantee activities included in the impact study component—how they are implemented, their participants, the contexts in which they are operated, and their operating practices. The detailed descriptions will assist in interpreting program impacts and identifying program features and conditions necessary for effective program replication or improvement.

Data on the programs will be collected from three main sources:

6. Site visits. During three rounds of multi-day site visits, data collection will involve interviews with managers and staff of the grantees and partner organizations, focus groups of participants, observations of program and staff activities, and reviews of participant case records.

7. Partner organization survey. A web-based survey of about 20 partner organizations in each site will occur at the beginning of the intake period and again near the end. This survey will describe the organizations that provide services to the population, details on services available, and the partnerships between the organizations.

8. Study Management Information System (MIS). Program staff will be asked to record information on the services provided to study participants in a study MIS.

Table A.4 summarizes the data collection activities envisioned for the impact and implementation component, though this is subject to change. Subsequent requests for information collection will be submitted, along with any changes to the table below.

Table A.4. Summary of Data Collection Plan – Impact and Implementation Component

Activity	Respondents	Mode	Schedule
Impact/Implementation
1) Baseline	Program applicants	Hard-copy form or ACASI. Administered or facilitated by grantee staff	Fall 2012-Fall 2014
2) Follow-up Survey	Study participants	CATI	Fall 2013-Fall 2015
3) Diaries	Study participants	Self-administered checklists	Fall 2013-Fall 2015
4) Observations	Study participants	Videoed in-home father-child observations	Fall 2013-Fall 2015
5) Administrative data	Study participants	Existing data	Spring 2012-Fall 2015
6) Site visits	Grantee and partner staff and participants	Semi-structured interviews, focus groups, observations, case-file review	Early 2013, late 2013, and early 2015
7) Partner organization survey	Staff at community organizations	Web-based survey	Early 2013, Summer 2014
8) Study MIS	Grantee staff	Web-based MIS	Fall 2012-Fall 2014
ACASI = Audio computer-assisted survey; MIS = Management Information System CATI = computer-assisted telephone interview

ii. Qualitative Component

We expect to select no more than 15 grantee programs for the qualitative evaluation component. The goal of the qualitative component is to obtain a richer and more nuanced understanding of the factors that influence the experiences, choices, and behavior of fathers or couples who are (or could be) served by RFHM grantees. While the impact/implementation component is squarely focused on the RFHM grantee activities and services, the qualitative studies are more oriented to exploring how individual beliefs and experiences connect to and are affected by the communities in which they live, their family relationships, and the programs and organizations that touch their lives. Information collection activities for the qualitative component are numbered below, and then detailed in Table A.3.

A subset of up to 15 grantees will be included in this study component, which will be separate from the subset included in the impact and implementation component.

Data collection for this component will include:

1. Site visits. Activities during these site visits will include “drive arounds” in which staff take notes and audio-record comments about infrastructure, housing, commercial activity, and street life. Additional data collection activities will include interviews with program staff, observations of program activities, and case record reviews.

2. Study MIS. The current plan is to ask program staff to enter data into the study MIS about participants’ service receipt.

3. Characteristics survey. The current plan is to ask program participants to complete a survey about their characteristics when they enroll in the program.

4. Partner organization survey. The web-based survey of partner organizations administered in the impact/implementation study sites will also be administered in the qualitative study sites.

5. Nonparticipant telephone interviews. As engagement in the program is often a challenge for both RF and HM programs, two rounds of interviews will be conducted with about 15 people at each study grantee who enrolled in a program but participated little or not at all.

6. In-depth in-person conversations with program participants. At each site, up to 15 individuals in the program will be followed closely throughout the study. Interviews with these individuals, which will occur in the respondent’s home if possible, will be guided by a semi-structured protocol, and will explore relationships with current and former partners as well as children, community involvement, and program experiences. These interviews will occur annually and be about two hours in length.

7. Check-in calls. Four check-in telephone calls will be made to the 15 program participants in each site who participate in the in-depth, in-person conversations.

8. Diaries. Diaries may also be used to capture program participants’ experiences and perceptions.

Table A.5 summarizes the data collection activities envisioned for the impact and implementation component, though this is subject to change. Subsequent requests for information collection will be submitted, along with any changes to the table below.

Table A.5. Summary of Data Collection Plan – Qualitative Component

Activity	Respondents	Mode	Schedule
Qualitative Component
1) Site visits	Grantee staff and participants	Semi-structured interviews, observations, case-file review	Fall 2012-Fall 2015
2) Study MIS	Grantee staff	Web-based MIS	Fall 2012-Fall 2015
3) Characteristics survey	Program participants	Hard-copy form or ACASI Administered or facilitated by grantee staff	Fall 2012-Fall 2015
4) Partner organization survey	Staff at community organizations	Web-based survey	Fall 2012-Fall 2015
5) Nonparticipants telephone interview	Program enrollees who do not participate in the program or participate very little	Semi-structured telephone interviews	Fall 2012-Fall 2015
6) In-depth participant interviews	Selected program participants	In-person	Fall 2012-Fall 2015
7) Check-in calls (follow-ups to in-depth participant interviews)	Selected program participants	Semi-structured telephone interviews	Fall 2012-Fall 2015
8) Diaries	Selected program participants	Self-administered surveys	Fall 2012-Fall 2015

MIS = Management Information System

2. Purpose and Use of the Information Collection

The information obtained through the PACT evaluation is critical to understanding a new generation of RF and HM programs—the services they provide, the experiences of their participants, and their effectiveness. This information can be used to inform decisions related to future investments in this kind of programming as well as the design and operation of such services.

As described above, different types of information will need to be collected to carry out the study objectives. A future submission will address baseline data which will be needed to document the demographic characteristics of fathers or couples who enroll and obtain initial information on factors that could explain study outcomes (e.g., level of father involvement, couple relationship status), and, importantly, to ensure that experimental and control groups are equivalent.

A future submission will also seek approval for elements in a study MIS that will allow for consistent and systematic collection of program participation data (e.g., number of services/sessions attended; number and type of contacts between staff and participants; number and type of referrals to other services) as necessary to document grant program implementation and provide context for interpretation of the impact findings. Additionally, for the implementation analyses, approval of site visit/discussion protocols to document the perspectives and experiences of key grantee managers and staff and a survey of partner organizations will be sought.

A future request will also be made for approval of diaries, observational protocols, and a follow-up survey instrument to provide information on the outcome measures needed for the impact analyses (e.g., level/quality of father involvement; parenting interactions; parent/couple relationship status and quality).

The purpose of the specific information collection requested in this package—collecting information from grantees—is to obtain more detailed information needed to allow ACF to select grantees for inclusion in the evaluation as an impact/implementation study site or as a qualitative study site.

3. Use of Improved Information Technology and Burden Reduction

Whenever possible within the constraints of the specific type of information collection, we will use technology to reduce burden. The telephone surveys (for the follow-up surveys), for example, will be conducted by CATI. CATI reduces respondent burden by automating skip logic and question adaptations, and eliminating pauses required for interviewers to determine what question to ask. By reducing interviewer error, it also reduces the need to call back respondents. The study MIS will be web-based and include appropriate links to relevant fields for each type of entry to reduce burden.

The information collection requested in this ICR, collected through semi-structured discussions with the grantees or partners, is not conducive to the use of information technology, such as computerized interviewing. Telephone conversations offer the best opportunity to tailor interviews to the specific grantee (or partner) with the least imposed burden on the grantee (or partner). If in-person conversations are determined to be more appropriate, the grantee will not be expected to travel for the semi-structured discussions.

4. Efforts to Identify Duplication and Use of Similar Information

At each stage of the evaluation, we will ensure that we do not collect information that is available elsewhere. For example, we will only collect data in the follow-up diaries that we are not able to collect through the telephone survey.

For the information collection requested in this ICR, no information will be collected from the grantees until a thorough review has occurred of the grantee’s application and other available information (e.g., grantee websites). No information that is available from existing sources will be collected directly from the grantees. Our proposed calls or visits with grantees will seek to supplement existing information with additional information deemed necessary to conduct a thorough and informative review and assessment of the suitability of candidate grantees for inclusion in either the impact/implementation component or the qualitative component.

5. Impact on Small Businesses or Other Small Entities

No small businesses (that are not RFHM grantees or their partners) are expected to be involved in data collection.

6. Consequences of Not Collecting Information or Collecting Information Less Frequently

Not collecting information for the PACT evaluation overall would limit the government’s ability to document the kinds of activities implemented with federal funds, and their effectiveness.

With regard to the data collection requested in this ICR, not collecting the requested information would substantially limit the value of the investment the government will make in this evaluation. As previously mentioned, this ICR is for a field data collection, in order to select grantees for the PACT evaluation. Identifying grant programs that are policy-relevant and that are best able to support the design and data collection requirements of an experimental evaluation and/or the qualitative data collection is crucial to ensuring that findings from the study are relevant to program administrators; federal, state, and local policymakers; researchers; and the parents who could benefit from the program services. Without approval for speaking with grantees and their partners, we risk selecting grantees that are either not as relevant to answering current policy questions, that cannot support evaluation activities or that would not constitute valid tests -- if the counterfactual (the services available to control group members) are not substantially different from grantee services.

Information required to select grantees for the PACT evaluation will be collected only once, thus no repetition of effort is planned. The anticipated schedule for future, additional information collection activities is presented in Table A.1.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

There are no special circumstances for the proposed data collection.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

In accordance with the Paperwork Reduction Act of 1995 (public law 104-13) and the Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity (August 12, 2011, volume 76, number 156, pages 50225-50226). The notice provided 60 days for public comment. The Department specifically requested comments on: (a) whether the proposed collection of information is necessary for the proper performance of the functions of the agency, including whether the information shall have practical utility; (b) the accuracy of the agency’s estimate of the burden of the proposed collection of information; (c) the quality, utility, and clarity of the information to be collected; and (d) ways to minimize the burden of the collection of information on respondents, including through the use of automated collection techniques or other forms of information technology. A copy of this notice is attached as Appendix A.

The discussion guide was developed by staff at the ACF, Office of Planning, Research, and Evaluation (OPRE) building on similar instruments approved by OMB and used in prior studies. The discussion guide was reviewed by staff at Mathematica.

During the notice and comment period, the government received a request for a copy of the instrument and a comment that expressed disapproval of both the programs and research. No public comments on the proposed grantee discussion guide were received.

9. Explanation of Any Payment or Gift to Respondents

No payments to respondents are proposed for this information collection. Payments to respondents for participation in future data collection are expected to be warranted and will be discussed in subsequent submissions.

10. Assurance of Confidentiality Provided to Respondents

Respondents will be told that the information they provide will be shared with ACF and used to select grantees. Programmatic information they provide may be presented in publicly-available reports, but no personal identifying information will be attached to the information or included in the report.

11. Justification for Sensitive Questions

There are no sensitive questions in the collection of information from grantees. The justification for sensitive questions on the subsequent data collections will be provided in subsequent packages.

12. Estimates of Annualized Burden Hours and Costs

The following section details burden and cost associated with the PACT evaluation, including the data collection proposed with this package. Importantly, burden and costs associated with the remaining portions of the project are initial estimates and subject to change.

a. Burden Hours and Costs for Field Data Collection – CURRENT REQUEST

With regard to the current proposed information collection request (field data collection), Table A.6 presents the reporting burden on grantee and partner organization staff for responding to the discussion guide. There are 116 current Responsible Fatherhood and Healthy Marriage grantees; we are expecting to hold with discussions with, on average, approximately 1.3 staff members per grantee (since we will not likely speak to some grantees after reading their applications, while with others we will need to hold conversations with 2 or more persons), for a total of 150 respondents. Response times were based on estimates of the time taken in similar interviews for similar studies such as the Evaluation of Adolescent Pregnancy Prevention Approaches (formerly OMB 0970-0360, now OMB 0990-0382). The burden is estimated from the total number of expected respondents (150) and the time required for each discussion (about one hour). Hence, the total burden to respondents is expected to be 150 hours (150 respondents x 1 hour).

Table A.6 also provides the total estimated annualized cost of the burden for the current information collection request of $4,401. The average hourly wage for staff at the grantee organizations and their partners ($29.34) is the average hourly wage of “social and community service managers” taken from the U.S. Bureau of Labor Statistics, National Compensation Survey, 2009. This proposed information collection does not impose an additional financial burden on respondents other than the time spent answering the questions contained in the discussion guides.

Table A.6. Estimate of Burden and Costs for the PACT Evaluation – Field Data Collection –

CURRENT REQUEST

Activity/Respondent	Annual Number of Respondentsa	Number of Responses Per Respondent	Average Burden Per Response (Minutes)	Total Annual Burden Hoursa	Average Hourly Wage	Total Annualized Cost
Selecting Study Grantees Discussions/ grantee and partner organization staff	150	1	60	150	$29.34a	$4,401
a Burden estimates are annualized over one year.

b. Burden Hours and Costs for Impact and Implementation Component

Table A.7 summarizes the total estimated reporting burden for the impact/implementation component of the study. As the design of the impact/implementation component is not finalized, the burden estimates for that component are very approximate. Assuming the maximum number of sites—15 in the impact/implementation study—the total annualized burden is estimated to be 12,201 hours. Figures are estimated as follows:

1a) Baseline/program applicants. It is expected that approximately 421 fathers will be invited to participate in the study, per site (as well as mothers associated with those fathers in half the sites), and that 95%, or 400 per site, will accept the opportunity to participate. 9,000 total baseline surveys are estimated: 400 fathers in each of 15 sites (6,000) and mothers associated with those fathers in half the sites (3,000). 9,000 annualized over three years is 3,000.

1b) Baseline/grantee staff. 30 staff members – 2 per site – are estimated to administer the 9,000 surveys, which equals 300 surveys per staff member. 300 annualized over three years is 100.

2) Follow-up surveys/study participants. A response rate of 80% is expected. If 9,000 survey members complete the baseline, then 7,200 follow-up surveys will be completed. 7,200 annualized over three years is 2,400.

3) Diaries/study participants. Same as #2 above: A response rate of 80% is expected. If 9,000 survey members complete the baseline, then 7,200 follow-up surveys will be completed. 7,200 annualized over three years is 2,400.

4) In-home observations/study participants and children. 9,600 in-home observations are estimated of fathers and their children for 80 percent of all fathers (0.80 x 6,000=4,800) and one of their children (4,800). 9,600 annualized over three years is 3,200.

[5) Administrative data. These data will not place additional burden on respondents.]

6a) Site visits/grantee staff. 1,125 interviews are anticipated, with 25 staff at each of 15 sites in each of three rounds of site visits. 1,125 annualized over three years is 375.

6b) Site visits/program participants. 720 participants are estimated to be involved in focus groups, with 8 participants per focus groups, two focus groups anticipated at each of 15 sites, in each of three rounds of site visits. 720 annualized over three years is 240.

7) Partner organization survey/organization staff. 600 responses are anticipated: 20 respondents in each of 15 sites, administered twice (over three years of data collection). 600 annualized over three years is 200.

8) Study MIS/grantee staff. We anticipate 6,000 father or father/mother participants, and 26 MIS entries are estimated per father or father/mother,¹ for a total of 156,000 MIS entries over the course of three years. 30 staff members – 2 per site – are estimated to collect MIS data on these father or father/mother participants (this equals a caseload of 200 over the course of 3 years). Each staff member will make 5,200 entries over the course of 3 years (156,000 ÷ 30 = 5,200). 5,200 annualized over three years is 1,733.

Table A.7 also provides the total estimated annualized cost of the impact/implementation component, which is $156,683. The total estimated cost is computed from the total annual burden hours and an average hourly wage for the respondent. The average hourly wage for staff at the grantee organizations and their partners ($29.34) is the average hourly wage of “social and community service managers” taken from the U.S. Bureau of Labor Statistics, National Compensation Survey, 2009. The average hourly wage of the respondents is estimated from the average hourly earnings ($4.92) of study participants in the Building Strong Families study (Wood et al. 2010). The average hourly earnings is lower than minimum wage because many study participants were not working. We expect that to be also the case for the RFHM participants.

Table A.7. Estimate of Burden and Costs for the PACT Evaluation –

Impact and Implementation Component

Activity/Respondent	Annual Number of Respondentsa	Number of Responses Per Respondent	Average Burden Per Response (Minutes)	Total Annual Burden Hoursa	Average Hourly Wage	Total Annualized Cost
Impact/Implementation Component
1a) Baseline/program Applicants	3,000	1	35	1,750	$4.92	$8,610.00
1b) Baseline/grantee staff	30	100	35	1,750	$29.34	$51,345.00
2) Follow-up survey/study Participants	2,400	1	50	2,000	$4.92	$9,840.00
3) Diary/study participants	2,400	1	70	2,800	$4.92	$13,776.00
4) In-home observations/study participants and their children	3,200	1	25	1,333	$4.92	$6,558.36
[5) Administrative data – will not place additional burden on respondents]
6a) Site visits/grantee staff	375	1	60	375	$29.34	$11,002.50
6b) Site visits/program Participants	240	1	90	360	$4.92	$1,771.20
7) Partner organization survey/organization staff	200	1	30	100	$29.34	$2,934.00
8) Study MIS/grantee staff	30	1,733	2	1,733	$29.34	$50,846.22
Total	11,875			12,201		$156,683.28

^a All burden estimates are annualized over three years.

[5) Administrative data – will not place additional burden on respondents]

c. Burden Hours and Costs for Qualitative Component

Table A.8 summarizes the total estimated reporting burden for the qualitative component of the study. As the design of the qualitative component is not finalized, the burden estimates for that component are very approximate. Assuming the maximum number of sites—15 in the qualitative study—the total annualized burden is estimated to be 2,303 hours. Figures are estimated as follows:

1) Site visits/grantee staff. 1,125 interviews are anticipated, with 25 staff at each of 15 sites in each of three rounds of site visits. 1,125 annualized over three years is 375.

2) Study MIS/grantee staff. We anticipate 3,000 father or father/mother participants, and 26 MIS entries are estimated per father or father/mother,² for a total of 78,000 MIS entries over the course of three years. 15 staff members – 1 per site – are estimated to collect MIS data on these father or father/mother participants (this equals a caseload of 200 over the course of 3 years). Each staff member will make 5,200 entries over the course of 3 years (78,000 ÷ 15 = 5,200). 5,200 annualized over three years is 1,733.

3a) Characteristics survey/study participants. 1,500 total surveys are estimated: 200 fathers in 5 sites (1,000), and mothers associated with those fathers in half the sites (500). 1,500 annualized over three years is 500.

3b) Characteristics survey/grantee staff. 15 staff members – 1 per site – are estimated to administer the 1,500 surveys, which equals 100 surveys per staff member. 100 annualized over three years is 33.

4) Partner organization survey/organization staff. 600 surveys are estimated: 1 person in 20 partner organizations in 15 sites, each surveyed twice. 600 annualized over three years is 200.

5) Telephone interviews/study nonparticipants. 450 surveys are estimated: 30 nonparticipants in each of 15 sites. 450 annualized over three years is 150.

6) In-depth interviews/study participants. 225 interviews are anticipated: interviews with 15 individuals in each of 15 sites. 225 annualized over three years is 75.

7) Check-in calls/study participants. 900 check-in calls are anticipated: 4 check-in calls per person interviewed in #6 “in-depth interviews” are anticipated. 900 annualized over three years is 300.

8) Diaries/study participants. 30 program participants are estimated to complete diaries; 30 annualized over three years is 10. 30 entries are estimated per participant.

Table A.8 also provides the total estimated annualized cost of the qualitative component, which is $51,159.78. The total estimated cost is computed from the total annual burden hours and an average hourly wage for the respondent. The average hourly wage for staff at the grantee organizations and their partners ($29.34) is the average hourly wage of “social and community service managers” taken from the U.S. Bureau of Labor Statistics, National Compensation Survey, 2009. The average hourly wage of the respondents is estimated from the average hourly earnings ($4.92) of study participants in the Building Strong Families study (Wood et al. 2010). The average hourly earnings is lower than minimum wage because many study participants were not working. We expect that to be also the case for the RFHM participants.

Table A.8. Estimate of Burden and Costs for the PACT Evaluation –

Qualitative Component

Activity/Respondent	Annual Number of Respondentsa	Number of Responses Per Respondent	Average Burden Per Response (Minutes)	Total Annual Burden Hoursa	Average Hourly Wage	Total Annualized Cost
Qualitative Component
1) Site visits/grantee staff	375	1	60	375	$29.34	$11,002.50
2) Study MIS/grantee staff	15	1,733	2	867	$29.34	$25,437.78
3a) Characteristics survey/ study participants	500	1	35	292	$4.92	$1,436.64
3b) Characteristics survey/ grantee staff	15	33	35	289	$29.34	$8,479.26
4) Partner organization survey/organization staff	200	1	30	100	$29.34	$2,934.00
5) Telephone interviews/ study nonparticipants	150	1	30	75	$4.92	$369.00
6) In-depth interviews/study Participants	75	1	120	150	$4.92	$738.00
7) Check-in calls/study Participants	300	1	30	150	$4.92	$738.00
8) Diaries/study participants	10	30	1	5	$4.92	$24.60
Total	1,640			2,303		$51,159.78

^a All burden estimates are annualized over three years.

d. Burden Hours and Costs for Entire Study

Table A.9 summarizes the total estimated reporting burden and costs for the entire study (combining totals from Table A.4, A.5, and A.6). As the design of the impact/implementation and qualitative components is not finalized, the burden and cost estimates for those components are very approximate. Assuming the maximum number of sites—15 in the impact/implementation study and 15 in the qualitative study—the total annualized burden is estimated to be 14,654 hours, and the total estimated information collection costs is estimated to be $212,244.06.

Table A.9. Estimate of Burden and Costs for the PACT Evaluation – All Components

Activity/Respondent	Annualized Number of Respondentsa	Total Annualized Burden Hoursa	Total Annualized Cost
Selecting Study Grantees	150	150	$4,401
Impact/Implementation Component	11,875	12,201	$156,683.28
Qualitative Component	1,640	2,303	$51,159.78
Total	13,655	14,654	$212,244.06

^a All burden estimates are annualized over three years except for the data collection required for selection of grantees to include in the study, which is annualized over one year.

13. Estimates of Other Total Cost Burden to Respondents and Record Keepers

These information collection activities do not place any capital cost or cost of maintaining capital requirements on respondents.

14. Cost to the Federal Government

The grantee discussion guide was drafted by staff in ACF, OPRE. The cost of federal government employees’ time drafting the discussion guide is estimated at $2,400. Discussions with grantees will be carried out by the evaluation contractor, at a cost of about $100,000.

15. Explanation for Program Changes or Adjustments

This submission to OMB is a new request for approval.

16. Plans for Tabulation and Publication and Project Time Schedule

a. Plans for Tabulation

With regard to the current ICR, the purpose of the field data collection effort is not to collect data for statistical analysis; rather, it is to identify sites for inclusion in the impact/implementation or qualitative studies. Information required for the selection of grantees will be collected from the grantee applications and other available sources and summarized in a grantee-specific profile document. The information from these documents will be summarized in tables that show which grantees appear to meet key criteria for inclusion in either study component (discussed above). No further information will be collected on grantees that clearly do not meet criteria for inclusion in either study component. Supplemental information will be collected via discussions with grantees – i.e., the instrument submitted with this package. As more in-depth, supplemental information is collected, it will be added to the grantee-specific documents and the summary tables will be updated.

The goal of the impact analysis is to compare observed outcomes for program participants with outcomes for members of a control group that were not offered RFHM services. We will use the experience of the control group as a measure of what would have happened to the program group members in the absence of the RFHM program. Random assignment of fathers/couples to a program and a control group ensures that the two groups of fathers/couples do not initially differ in any systematic way on any characteristic, observed or unobserved (nevertheless, we will still statistically compare the groups to establish equivalence between the groups at baseline). Any observed differences in outcomes between the program and control group couples can therefore be attributed to the program with a known degree of precision.

Differences of means or proportions in outcomes between the program and control group will provide unbiased estimates of the impacts of the RFHM program. More precise estimates will be obtained using regression models to control for random differences in the baseline characteristics of program and control group members. In their simplest forms, these models can be expressed by the following equation:

(1) Y = X΄ß + δ P + e,

where:

Y is an outcome variable

X is a vector of control variables (including an indicator for each site)

β is the vector of regression coefficient for the control variables

δ is the measure of the impact of the RFHM program

P is an indicator that equals 1 for program group members and 0 for control group members

e is a random error term that is assumed to have a mean of zero conditional on X and P, and is interpreted as the unobserved factors that affect Y.

The statistical techniques used to estimate the regression‑adjusted impacts depend on the form of the dependent variable, Y. If the dependent variable is continuous, then ordinary least squares techniques will produce unbiased estimates of the parameter δ. However, if the dependent variable is binary—for example, whether the father lives with the child—then consistent parameter estimates can be obtained by using logit or probit maximum likelihood methods. If the dependent variable is censored or truncated—for example, earnings or total income—then tobit maximum likelihood or two-stage procedures will be used.

Control variables in the vector X will include any variables that may affect the outcome that are not affected by the intervention. Hence, X could include the characteristics of the individual or couple for which data are collected on the baseline information form, including the status and quality of the relationship at baseline. X could also include baseline characteristics that can be easily recalled and were measured using the follow-up survey (such as incarceration prior to random assignment).

For both the implementation and qualitative components of the study, the information obtained from site visits and interviews with participants and staff will be organized using a variety of structured formats, including: narrative site summaries to describe the grantee site and its context and highlight themes; detailed logic models to illustrate each grantee’s program design and theory of change, and timelines to track key stages of program development and implementation; tables to describe characteristics of participants and utilization patterns of service receipt; and tables that show predefined measures of success in implementation.

b. Time Schedule and Publications

This study is expected to be conducted over a five-year period that began September 30, 2011. Review of grantee applications and collection of other existing information (e.g. from grantee websites) has already begun. Collection of more in-depth data from the grantees and their partners will begin once OMB clearance is received. To keep to the planned schedule, all grantees must be selected by June 2012.

Table A.10. Schedule for the Evaluation

Activity	Date
Selection of grantees for inclusion in evaluation Intake period for impact study Report on early findings on implementation study Implementation mid-term report First report on qualitative analysis Final implementation report Second report on qualitative analysis Final impact report Final report on qualitative analysis	March-June 2012 September 2012-August 2014 Spring 2013 Spring 2014 Fall 2014/winter 2015 Winter 2016 Fall 2015 Summer/fall 2016 Summer/fall 2016

With regard to the current information collection request, no publication is planned.

17. Reason(s) Display of OMB Expiration Date is Inappropriate

All instruments will display the expiration date for OMB approval.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

REFERENCES

Avellar, Sarah, M., Robin Dion, Andrew Clarkwest, Heather Zaveri, Subuhi Asheer, Kelley Borradaile, Megan Hague Angus, Timothy Novak, Julie Redline, and Marykate Zukiewicz. “Catalog of Research: Programs for Low-Income Fathers.” Report submitted to the U.S. Department of Health and Human Services, Administration for Children & Families, Office of Planning, Research, and Evaluation. Washington, DC: Mathematica Policy Research, June 2011.

Guzzo, Karen Benjamin, and Frank F. Furstenberg, Jr. “Multipartnered Fertility among American Men.” Demography, vol. 44, 2007, pp. 583-601.

McLanahan, Sara McLanahan, Sara. “Fragile Families and the Reproduction of Poverty.” Annals of the American Academy of Political and Social Science. 2009, 621: 111-131.

U.S. Census, 2011. http://www.childstats.gov/americaschildren/famsoc1.asp. Retrieved December 24, 2011.

U.S. Government Accountability Office. Healthy Marriage and Responsible Fatherhood Initiative: Further Progress is Needed in Developing Risk-Base Monitoring Approach to Help HHS Improve Program Oversight. September 2008.

Wood, Robert G., Sheena McConnell, Quinn Moore, Andrew Clarkwest, and JoAnn Hsueh. “Strengthening Unmarried Parents’ Relationships: The Early Impacts of Building Strong Families.” Report submitted to the U.S. Department of Health and Human Services, Administration for Children and Families. Princeton, NJ: Mathematica Policy Research, May 2010.

1 In the Building Strong Families study, two of the three curricula evaluated involved 24 sessions; it is estimated that one MIS entry may need to be made per session. It is further anticipated that two additional MIS entries may also need to be entered for program entry and program exit. This totals to 26 entries.

2 In the Building Strong Families study, two of the three curricula evaluated involved 24 sessions; it is estimated that one MIS entry may need to be made per session. It is further anticipated that two additional MIS entries may also need to be entered for program entry and program exit. This totals to 26 entries.