1. Circumstances Making the Information Collection Necessary
This information collection request (ICR) is for clearance to collect information for the Parents and Children Together (PACT) Evaluation which will evaluate a subset of Responsible Fatherhood (RF) and Healthy Marriage (HM) grants authorized under the Claims Resolution Act of 2010 (Public Law 111-291). This ICR requests clearance for an introductory script (to be read to program applicants), baseline instrument, and information to be collected through a study Management Information System (MIS). These three instruments will be used in impact and implementation/qualitative only evaluations1 of RF programs.2
The evaluation is being undertaken by the U.S. Department of Health and Human Services, Administration for Children and Families (ACF), and is being implemented by Mathematica Policy Research and its partner, ICF International.
The past several decades have witnessed sweeping changes in family structure. In 1980, 77 percent of children lived with two married parents; by 2010, this figure had fallen to only 66 percent (U.S. Census Bureau 2011). Families have also become more complex: nearly one in five fathers now has children with more than one woman (Guzzo and Furstenberg 2007). While many children do well living with only one parent, research suggests that on average children do better when they have two involved parents (McLanahan 2009). These changes in family structure, their attendant consequences for children, and recent changes in welfare policy set the stage for new investments in programs aimed at strengthening families and in policy research on fatherhood and marriage.
As one response by the federal government, the Deficit Reduction Act of 2005 created the Responsible Fatherhood (RF) and Healthy Marriage (HM) grant programs, authorizing ACF to provide up to $50 million for RF grants and $100 million for HM grants each year from 2006 to 2010. This funding represented an “unprecedented financial commitment by the federal government to support marriage and fatherhood programs” (U.S. Government Accountability Office 2008). Under this act, awards were made to 226 grantees to provide three RF services or one or more of eight HM services including parenting classes, marriage and relationship education, and economic stability services.
The Claims Resolution Act of 2010 reauthorized this grant program, evenly allocating the $150 million between RF and HM funding ($75 million for each). New three-year grants were awarded in September 2011 to 55 RF and 60 HM grantees. The PACT Evaluation will provide documentation of the operations of a subset of these grant programs, the characteristics and life experiences of those who apply for services in the selected programs, and, for impact evaluation sites, an assessment of the impact of the programs on a range of outcomes.
Few rigorous studies of RF programs have been conducted to date.3 Of the 60 impact and implementation studies of programs for low-income fathers included in a recent systematic review of the evidence (Avellar et al. 2011), only 13 used a rigorous evaluation design. These rigorous program evaluations generally showed at least one statistically significant favorable impact, but most did not result in a compelling pattern of positive impacts.
Work under PACT will be carried out in stages with different types of information collection in each stage. Thus, clearance will be requested in stages as work progresses. The first submission, approved on April 20, 2012, provided clearance for discussions with grantees that may be considered as sites in the evaluation. OMB Control number 0970-0403 was set for the evaluation. Discussions with grantees are ongoing as this package is being developed. ACF will provide a summary of the information obtained through the discussions to OMB as activity proceeds.
The PACT Evaluation uses three interrelated evaluation strategies: (1) experimental impact; (2) implementation; and (3) qualitative evaluations. More detailed information on the three evaluation strategies is included in Appendix A.
These three strategies are combined into two types of multi-component evaluations:
impact evaluations, complemented with implementation and qualitative evaluations, will be conducted in a subset of grantees to provide rigorous estimates of the effectiveness of the studied programs and information about their operating contexts; and
implementation and qualitative evaluations (without impact evaluations) will be conducted in a separate subset of grantees which present some particular feature of program design or target population that warrants detailed study, but which would not support an impact evaluation (for example, if power analyses indicate that sample size is inadequate).
For ease of communication, in this ICR these two types of evaluations are called “impact” and “implementation/qualitative only,” respectively.
This ICR requests clearance for three data collection protocols:
Introductory script. The script will be read to all program applicants by grantee staff, to introduce applicants to the program, the study, and the baseline survey. (A set of FAQ’s is also provided, should the questions arise.) Applicants who are eligible to participate, and consent, will continue on to complete the Baseline Survey with contractor staff. For more information on the introductory script, see section A.12.
Baseline survey for use in RF programs. The baseline survey will be conducted via CATI (grantee staff will provide a phone for program participants to call, and contractor staff will conduct the survey). The survey will be used to collect information from study participants on their characteristics measured at baseline, contact information used to track the applicants for purposes of follow-up survey completion, and pre-program measures of key outcomes. The consent statement is provided in Appendix B. Appendix C contains a question-by-question justification, lists of key outcome domains and subgroups, links between the domains to be assessed at baseline with those to be assessed at follow-up (provided as the “outcome” column in the question-by-question justification), and descriptions of the surveys from which items are borrowed for the proposed baseline.
Study MIS for use in RF programs. The study MIS will be used to collect information needed to conduct the random assignment of applicants in impact evaluation sites and, importantly, to collect information on participants’ service receipt from the program in all RF study sites.
These three data collection protocols are submitted with this request. Additional follow-on ICR submissions will request clearance for additional instruments and protocols, including those for use in HM programs as needed, to collect follow-up data for the impact analysis, and to collect program implementation and qualitative data.
2. Purpose and Use of the Information Collection
The information to be obtained through the PACT Evaluation is critical to understanding the current field of RF programs—the services they provide, the experiences of their participants, and their effectiveness. This information can be used to inform decisions related to future government investments in this kind of programming as well as the design and operation of such services.
Baseline Survey. Data collected through the baseline survey is crucial for the impact evaluation; when conducted in implementation/qualitative only sites, the baseline will provide critical information on populations served. In particular, these data will be used for six purposes:
Describing the characteristics of participants. The baseline survey will gather descriptive information on study participants at baseline to make it possible to identify the characteristics of fathers who apply to RF programs. In addition to basic demographic information, these data will provide information about the types of challenges faced by fathers who enroll in RF programs (e.g., education level, employment status, housing stability, etc.). In impact sites, such data can also be used to adjust for potential bias that might arise from follow-up survey nonresponse.
Identifying subgroups of interest. Baseline data can be used to identify subgroups for which impacts may differ—for example, it may be that impacts are larger for younger fathers than for older fathers—or to identify subgroups which may be informative to qualitative analyses.
Collecting information that can explain variation in outcomes (in impact sites). Impact estimates obtained from the differences between mean outcomes of program group members and mean outcomes of control group members are unbiased. However, impact estimates obtained using a regression model with covariates that explain some of the variation in outcomes can improve the precision of the estimates. One of the best predictors of most outcomes is the prior value of the outcome. Hence, the baseline survey includes some measures that will also be included on the follow-up survey.
Identifying factors that could predict program participation. The primary impact analysis will focus on the estimated effect of offering grantee services to fathers. Factors at baseline that predict program participation can be used to estimate the impact of receiving different types and intensities of RF program services (as described in section A16). Hence, the baseline survey asks the respondent about his motivation to participate in the program and barriers to his participation. Information collected from grantee staff as part of the study’s MIS (described below) could also be used for this purpose. Predicted probabilities of participation can also be estimated in implementation/qualitative only sites.
Checking that the program and control groups are equivalent (in impact sites). Information on the characteristics of study participants can be used to check the similarity of the program and control groups. Although random assignment produces similar groups, on average, baseline data can be used to verify program-control equivalence for the full research sample and for the sample of respondents to the follow-up survey.
Identifying and tracking study participants. Identifying information includes the study participant’s complete name, sex, date of birth, mailing address, and Social Security number. In impact sites, this information is needed to match with other administrative data (e.g., wage/earnings data, child support data) to assess the impact of the programs on these key outcomes. In addition, personal information along with information on sample members’ telephone numbers, email addresses, social network information, and contact information for up to three relatives or friends is needed to facilitate locating study participants for follow-up survey data collection. Accurate and detailed locating information is essential for achieving high survey response rates.
Study MIS. Data collected through the study MIS are critical to both the impact and implementation evaluations. Data will include basic demographics and program participation (e.g. participant entry into the program, participation, and exit from the program). The data will be used for four main purposes:
Conducting random assignment (in impact sites). The PACT impact evaluation will be a random assignment evaluation. The study MIS, overseen by the evaluation contractor, will determine random assignment after participants have consented and completed the baseline survey. Random assignment is the core of an experimental impact evaluation. It creates a control group that is similar on all baseline characteristics to program participants. For this reason, a random assignment evaluation is often considered the most rigorous program evaluation.
Collecting information on the services provided by RF programs and the extent of program participation. Grantee staff will be asked to report on all services provided to program participants on an ongoing basis. The implementation study will describe what services RF programs offered and the level of participation in those services. Historically, research indicates that many social services programs find it difficult to engage and retain participants—many individuals either never begin participating after enrollment or leave the program before it is completed. Hence, it is important to collect information on both what services the program offers and what services the participants actually receive. This information also will aid in interpreting the impact estimates (by allowing analysis by high or low levels of active participation/dosage).
Monitoring RF programs during the study period. The information gathered through the study MIS will be used to monitor program performance and provide timely feedback to the grantees to help them identify any areas needing attention. The treatment may change during the study period because of monitoring or other reasons. We will document any changes in the program that occur during the study period through the implementation study (or from information obtained from other sources, such as the grant funder). In the impact analysis, we will account for significant program changes by examining the impacts of the program separately for the period before and the period after the program changed.
Estimating the impact of receipt rather than offer of services (in impact sites). Using data from the program group in the MIS, we can estimate the relationship between participant characteristics at baseline (including grantee staff predictions of likely participation) and participation in program activities. This model can then be used to estimate the impact of receipt of services (as described in Section A16).
3. Use of Improved Information Technology and Burden Reduction
Baseline Survey. The baseline survey will be conducted by computer-assisted telephone interview (CATI).4 CATI is a good method for administering interviews with questions with complex skip patterns, the need for interviewer probes, and large numbers of respondents. CATI reduces respondent burden by automating skip logic and question adaptations and by eliminating delays caused when interviewers must determine the next question to ask. CATI is programmed to accept only valid responses based on preprogrammed checks for logical consistency across answers. For example, a father may say that he has only one child in response to the baseline survey question C2. If he then responded to question C4 by saying he had “another child,” the computer program would prompt the interviewer to ask about the discrepancy. Interviewers are thus able to correct errors during the interview, eliminating the need for burdensome and costly call-backs to respondents.
Study MIS. The study MIS will be a web-based application providing easy access while maintaining the security of the data. The web-based application will allow sites to access the MIS without purchasing or installing additional software or changing the configuration of their computers. The system can be accessed from any computer, allowing for ease of entry, while the data are housed on secure servers behind the contractor’s firewall, thereby maintaining data security.
The system has been designed with use by the grantee staff in mind based on experience from prior studies with similar types of service providers. As such, it will be flexible, easy-to-use, and include navigational links to relevant fields for each type of entry to reduce burden on grantee staff and increase the quality and quantity of data collected. The system is designed for multiple users at each organization and will include options for varying levels of system access depending on users’ access needs. For example, administrators or supervisors will have the greatest rights within the system, having the ability to create new users, assign program participants to staff members, and review all activity for the organization. Staff providing direct services to study participants will have the ability to record and review information about participants assigned to their caseload. The various levels of system access allow for streamlining of information; limiting full system access to a small set of staff members promotes increased data security and greater data quality.
4. Efforts to Identify Duplication and Use of Similar Information
At each stage of the evaluation, we will ensure that we do not collect information that is available elsewhere.
Baseline Survey. The baseline survey will not ask for information that can be reliably obtained through administrative data collection. Study participants will be asked to provide limited information on formal child support, as administrative data does not consistently capture child support orders in other states. The baseline survey will ask study participants to report on informal contributions (monetary and in-kind support) that would not be reflected in administrative data. In addition, information on quarterly earnings (reported to the state unemployment insurance agency) will be obtained from administrative data; the baseline survey will ask for earnings in the past month to capture more recent earnings and earnings that may not have been reported to the unemployment agency. Though criminal history information is potentially available through administrative sources, that information will be gathered through the baseline survey because not all states allow administrative data access for research purposes and because administrative data lack key information in some states. Nevertheless, participants will be asked to provide consent for the collection of administrative data on criminal background should that be deemed necessary at a later time.
Study MIS. RF programs do not typically collect all the information that will be gathered by the study MIS. For instance, information required for intake and random assignment is not likely to be available from other sources. Likewise, RF programs often do not have an existing MIS that systematically tracks the information to be included in the service receipt section of the study MIS.
When discussing the requirements for participating in the study with grantees, we will describe the study MIS and its functionality. If a grantee has an existing MIS that tracks information on service receipt, we will review the system and assess the degree to which the grantee’s existing MIS tracks information in a comparable manner to the study MIS. When reviewing data for comparability, we will discuss the grantee’s ability to modify the existing MIS, if needed. For instance, we may identify that most data are comparable, but a few items are different or missing. The grantee may be able to revise its system to have a greater match with the data that will be collected in the study MIS. If the grantee’s MIS tracks the service receipt information needed for the PACT Evaluation, we will establish a data-sharing agreement with the grantee to facilitate using their MIS data for PACT analysis. Grantees may still be required to use the study MIS for the intake and random assignment process, but we will work with grantees that have an existing MIS that can produce study data on service receipt to develop a feasible process for these study components.
5. Impact on Small Businesses or Other Small Entities
No small businesses that are not RF grantees or their partners are expected to be involved in data collection. In the case that RF grantees or their partners are small entities, instruments have been tailored to minimize burden and only collect critical evaluation information.
6. Consequences of Not Collecting Information or Collecting Information Less Frequently
Not collecting information for the PACT Evaluation overall would limit the government’s ability to document the kinds of activities implemented with federal funds and to measure their effectiveness. In particular, the PACT Evaluation represents an important opportunity for ACF to learn about RF programs. If the information collection requested by this clearance package is not conducted, policymakers and providers of RF programs will lack high-quality information on the impacts of these programs as well as descriptive information that can be used to later refine the programs.
Baseline Survey. Without collecting detailed contact information on study participants, the study’s ability to track participants over a 12-month follow-up period would be limited. This would likely lead to a lower response rate and, in impact sites, a greater risk that the impact estimates will be biased by nonresponse. The lack of baseline information would also limit the contractor’s ability to describe the population of RF program participants (in impact and implementation/qualitative only sites) and would limit the analysis of program impacts on subgroups, limiting the contractor’s ability to determine the groups for which the program is most effective. Without data from the baseline survey, impact estimates would be less precise, making small impacts less likely to be detected, and adjustments for nonresponse to the follow-up survey would have to be based on administrative data, which are much more limited. In addition, without baseline information on factors that could predict program participation, it would not be possible to measure the impact of programs on receiving services, rather than being offered services (for more detail, see Section A16).
Finally, the baseline survey yields data that are vital for ensuring that random assignment is properly implemented. In particular, without data from the baseline survey, it would not be possible to test whether the program and control groups were equivalent at baseline on many key measures (such as those not covered by administrative data).
Baseline surveys will be collected only once; thus, no repetition of effort is planned.
Study MIS. Information entered by grantee staff at intake is collected once, prior to submitting an applicant’s case to random assignment. In impact sites, without entry of this information, we would not be able to check if the applicant is already a member of the evaluation sample, which ensures the integrity of random assignment. That is, there are instances in which individuals previously assigned to the treatment group and to the control group come back into an office and reapply for services during the study period. Because a basic principle of an experimental design is that a person can be randomly assigned only once, it is important to determine that an individual has not gone through the process before. The system will always check to determine if a specific individual has been previously randomly assigned. If the individual is found in the system, the worker is informed of their status. In addition, staff predictions of likely program participation would be missing, making it more difficult to estimate impacts on those who actually participated.
Staff will be asked to enter information about services offered to participants (e.g., individuals assigned to the next parenting workshop) and their actual participation and attendance throughout the period of the study. Without information on service receipt, we would not be able to describe the services offered to participants by RF programs and the extent to which program participants received these services. These data are critical to the implementation analysis and to interpreting the findings from the impact analysis.
7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
There are no special circumstances for the proposed data collection.
8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency
In accordance with the Paperwork Reduction Act of 1995, the public was given an opportunity to review and comment through the 60-day Federal Register Notice, published on December 20, 2012 (77FR 4328, document number 2012-1569, pp. 4328-4329). A copy of this notice is attached as Appendix D. The notice provided 60 days for public comment; it also described the entire study and all related burden and therefore we have requested that all subsequent 60-day comment periods be waived.
One comment was made in response to this Federal Register Notice: the commenter expressed the view that the whole evaluation was not worth the resources. There was also a request for the data collection instruments which was fulfilled.
No substantial changes in burden for the baseline are proposed over those proposed in the 60-Day FRN. A reassessment of the burden for the MIS has increased burden proposed in this Supporting Statement compared to that proposed in the 60-Day FRN. Adjustments have been publicized through the 30-Day FRN.
9. Explanation of Any Payment or Gift to Respondents
We propose to offer a $10 payment to applicants who complete the baseline survey. This would occur in both impact and implementation/qualitative only sites. We suggest offering this payment for four reasons:
Increased response to the baseline survey. Completing the baseline survey will take about 30 minutes. Knowing that they will be paid for completion is expected to increase applicants’ likelihood of agreeing to participate in the study and to spend the time to complete the survey. Further, we suggest offering a modest $10 payment to baseline survey respondents to reduce attrition for follow-up data collection.
Reduced attrition for follow-up data collection (in impact sites). In longitudinal studies, providing an incentive for earlier surveys may contribute to higher response rates for subsequent surveys (Singer et al. 1998). Therefore, providing a modest payment at baseline may reduce attrition for follow-up data collection.
Grantee staff cooperation. In general, grantee staff find evaluation very challenging—this is true in implementation/qualitative evaluations, and especially true in impact evaluations where participants are randomly assigned. Directors of RF programs5 have told us that providing a modest payment to all applicants—including those who are ultimately assigned to the control group—demonstrates respect for the applicants’ time and, in impact evaluations, makes it easier for the grantee staff to support the evaluation and the requirement to deny services to control group members.
10. Assurance of Confidentiality Provided to Respondents
The consent statement and all other materials given to study participants will include assurances that the research team will protect their privacy to the fullest extent possible under the law – please see Appendix B for the Consent Statement and FAQ’s (this will be distributed to participants by the case worker, but the actual consent will be provided verbally by the participant after the consent has been read to the participant via the CATI). Several specific measures will be taken to protect their privacy:
Training interviewers in confidentiality procedures. The oral consent process and baseline interview will be administered by telephone interviewers at Mathematica’s Survey Operations Center (SOC). Interviewers will be seated in a common supervised area. As part of the telephone interviewers’ introductory comments, study participants will be told that their responses will be protected and that they will have the opportunity to have their questions concerning the study answered by the interviewer. Interviewing staff will receive training that includes general SOC security and confidentiality procedures as well as project-specific training that includes explanation of the highly confidential nature of this information, instructions to not share it or any personally identifiable information (PII) with anyone not on the project team, and warnings about the consequences of any violations. After receiving training, these staff members sign confidentiality and nondisclosure agreements.
Using CATI for consent and the baseline survey. Administering consent and the baseline survey via CATI eliminates security risks related to shipping hard-copy forms containing PII to Mathematica.
Restricting and logging access to the sample management system (SMS). Some data elements from the baseline survey data will be entered into an SMS to locate sample members for the follow-up survey. This is a sequel server database housed on an encrypted server. A hierarchical architecture will be used to assign user rights to specific individuals who will be able to access the system and enter information only at their own location. All activity in the system will be logged. Unless otherwise required by ACF, the information stored in the SMS will be destroyed when no longer needed in the performance of the project.
Restricting access to the study MIS. Data collected through the study MIS will be housed on secure servers behind Mathematica’s firewall. Access to the study MIS will be restricted by assigning a password to each relevant staff member.
In addition to these study-specific procedures, Mathematica has extensive corporate administrative and security systems to prevent the unauthorized release of personal records, including state-of-the-art hardware and software for encryption that meets federal standards and other methods of data protection (e.g., requirements for regular password updating), as well as physical security that includes limited key card access and locked data storage areas.
11. Justification for Sensitive Questions
Some sensitive questions are necessary in a study of programs designed to affect personal relationships and employment. Prior to starting the baseline survey, all respondents will be informed that their identities will be kept private and that they do not have to answer questions that make them uncomfortable. Table A.1 describes the justification for the sensitive questions included in the baseline survey. Although these questions are sensitive, they have commonly, and successfully, been asked of respondents similar to those who will be in this study (for example, in the Fragile Families and Child Wellbeing Study, the Building Strong Families Study, and the Early Head Start Research Evaluation Project).
Table A.1. Justification for Sensitive Questions – Baseline Survey and Study MIS
Question Topic |
Justification |
Respondent Social Security number |
The respondent’s Social Security number is essential for this evaluation for three reasons. First, it will be used to collect administrative data on the respondents. The Social Security number will allow us to obtain important outcome data on the respondent from child support agencies and the National Directory of New Hires. Second, Social Security numbers will also be used to collect information on the location of the study participant for the follow-up data collection. Third, these numbers will be used as an identifier to link the information collected in the study MIS with the survey data and will allow the study MIS to check whether the person has already been randomly assigned. |
Symptoms of depression
|
Parental depression has been shown to have adverse consequences for child outcomes (Downey and Coyne 1990, Gelfand and Teti 1990). To measure depressive symptoms, we will use eight items from the Patient Health Questionnaire (PHQ-9), which was designed as a diagnostic instrument for depression but can also be used to measure subthreshold depressive disorder in the general population (Martin et al. 2006). The PHQ-9 has been shown to be reliable and valid in diverse populations and has been used in clinical settings to measure symptom improvement and monitor treatment outcomes (Kroenke, Spitzer, and Williams 2001; Löwe et al. 2004). Findings from telephone administrations of the instrument have been shown to be similar to in-person assessments (Pinto-Meza et al. 2005). The PHQ-8 includes eight of the nine items from the PHQ-9; it has been shown to be a useful measure of depression in population-based studies (Kroenke et al. 2009). |
Earnings
|
A key goal of RF programs is to improve fathers’ economic stability. The outcomes of a father employed when he enters the program may be very different than those of a father who enters without employment. The survey asks whether the respondent worked in the past month and, if so, the amount he or she earned in the last month from formal and informal jobs. This question has been asked successfully in many surveys including the Building Strong Families survey (Wood et al. 2010). |
Involvement with the criminal justice system
|
Recent research suggests that a history of incarceration and involvement with the criminal justice system may be fairly common among fathers in the PACT target population (Pearson et al. 2011). Parental incarceration has major negative effects on child and family well-being, reducing the financial support and other types of support the parents can provide to their children and families. Similar questions have been included in other large national studies, such as the Fragile Families and Child Wellbeing Study, the National Job Corps Study, and the Building Strong Families Study. In the Building Strong Families survey, nonresponse was less than 1 percent for these items. |
12. Estimates of Annualized Burden Hours and Costs
Previously Approved Burden: Field Data Collection. Table A.2 summarizes the total estimated reporting burden and costs for the field data collection. This burden was approved on April 20, 2012. Burden was approved for one year; however, to sum with burden requested below, we have annualized the burden over three years in Table A.2.
Table A.2. PREVIOUSLY APPROVED - Estimates of Burden and Costs for the PACT Evaluation – Field Data Collection
Activity/Respondent |
Annual Number of Respondentsa |
Number of Responses Per Respondent |
Average Burden Per Response (Minutes) |
Total Annual Burden Hoursa |
Average Hourly Wage |
Total Annualized Cost |
Selecting Study Grantees Discussions/ grantee and partner organization staff |
50 |
1 |
60 |
50 |
$29.34 |
$1,467 |
a All burden estimates are annualized over three years.
Introductory Script and Baseline Survey. Table A.3 summarizes the proposed total estimated reporting burden for the use of the introductory script and baseline survey. Assuming the maximum number of sites—15 across both the impact and implementation/qualitative only sites—the total annualized burden is estimated to be 1,702 hours. Figures are estimated as follows:
Introductory script by grantee staff. We expect 90 staff members–6 in each of the 15 sites–to provide information about the RF program and the PACT study to the 6,316 applicants. Annualizing the grantee staff members and the applicants results in 30 grantee staff and 2,105 program applicants. (Which equals 70.2 meetings per staff member.) We expect these meetings, which will involve explaining the program services and the fact that the applicant will be randomly assigned to be eligible or not eligible for services, to last approximately 10 minutes. Thus, the total annualized burden for grantee staff is 351 hours (30 staff members holding 70.2 meetings of 10 minute duration). These grantee staff will be meeting with the 2,105 applicants, who will likewise spend 10 minutes per meeting (with one meeting apiece) for an annual burden of 351 hours. For grantee staff and applicants together, the total annual burden of the introductory script will be 702 hours. This introduction will take place prior to the survey.
Baseline survey for study participants. We expect 6,316 applicants during the study intake period. It is assumed that 95 percent of applicants to the program will be found eligible for the study and consent to participate; thus, 6,000 (95%) study participants are expected to complete the baseline survey: 400 fathers in each of 15 sites (6,000). Annualizing 6,000 over three years is 2,000. We expect each survey to last 30 minutes, for a total of 1,000 annualized burden hours.
Table A.3 also provides the proposed total estimated annualized cost of the introductory script and baseline survey, which is $16,425. The total estimated cost is computed from the total annual burden hours and an average hourly wage for the respondent. The average hourly wage for staff at the grantee organizations ($27.86) is the average hourly wage of “social and community service managers” taken from the U.S. Bureau of Labor Statistics, National Compensation Survey, 2010. The average hourly wage of program applicants is estimated from the average hourly earnings ($4.92) of study participants in the Building Strong Families Study (Wood et al. 2010). These average hourly earnings are lower than minimum wage because many study participants were not working. We expect that to also be the case for PACT study participants.
Table A.3. CURRENT REQUEST – Estimate of Burden and Cost for the PACT Evaluation – Introductory Script and Baseline Survey
Activity/Respondent |
Annual Number of Respondentsa |
Number of Responses per Respondent |
Average Burden per Response (minutes) |
Total Annual Burden Hoursa |
Average Hourly Wage |
Total Annualized Cost |
Introductory Script |
|
|
|
|
|
|
1) Grantee staff |
30 |
70.2 |
10 |
351 |
$27.86 |
$9,779 |
2) Program applicants |
2,105 |
1 |
10 |
351 |
$4.92 |
$1,726 |
Baseline Survey |
|
|
|
|
|
|
1) Study participants |
2,000 b |
1 |
30 |
1,000 |
$4.92 |
$4,920 |
Total |
|
|
|
1,702 |
|
$16,425 |
a All burden estimates are annualized over three years.
b Note that the 2,000 “study participants” are part of the 2,105 “program applicants,” as the study participants will all begin as program applicants. (Five percent of program applicants are not expected to agree to participate in the study, thus there are 5% more program applicants than study participants.) The 2,000 study participants do not represent 2,000 individuals in addition to the 2,105 program applicants.
Study MIS. Table A.4 summarizes the proposed total estimated reporting burden for the study MIS. Assuming the maximum number of sites—15 in both the impact and implementation/qualitative only evaluations—the total annualized burden is estimated to be 2,533 hours. Figures are estimated as follows:
Study MIS and grantee staff in the impact and implementation evaluation sites. This burden is based on the number of computer entries grantee staff will make as they enroll and track participation by participants.
Intake MIS entries. We anticipate 7,500 program applicants will have one MIS entry to document intake (prior to random assignment in impact sites), producing 7,500 MIS intake entries: this includes 6,000 impact study participants and 1,500 implementation/qualitative only participants.
Participation MIS entries. Of the 7,500 applicants, we estimate that approximately 4,500 applicants will be eligible to participate in the program—either due to random assignment (3,000 participants in impact sites, i.e. those assigned to the control group will not be eligible to participate in the program) or simply due to applying to the program in implementation/qualitative only sites (1,500 participants). We estimate that these 4,500 program participants will generate an estimated 49 additional MIS entries to document their program participation,6 for a total of 220,500 MIS entries about participation over the course of three years.
The 7,500 intake entries, combined with the 220,500 participation entries, equates to 228,000 MIS entries completed by grantee staff. We estimate that 90 staff members—6 in each of 15 sites—will collect MIS data on these fathers. Therefore, each staff member will make 2,533 entries over the course of 3 years ([7,500+220,500] ÷ 90 = 2,533). We estimate that each entry will take 2 minutes on average. Therefore, when we annualize the grantee staff members – resulting in 30 grantee staff – the total annualized burden is 2,533 hours (30 staff members entering data 2,517 times, each entry taking 2 minutes).
Table A.4 also provides the proposed total estimated annualized cost of the study MIS, which is $70,569. The total estimated cost is computed from the total annual burden hours and an average hourly wage for the staff. As above, the average hourly wage for staff at the grantee organizations ($27.86) is the average hourly wage of “social and community service managers” taken from the U.S. Bureau of Labor Statistics, National Compensation Survey, 2010.
Table A.4. CURRENT REQUEST – Estimate of Burden and Cost for the PACT Evaluation – Study MIS
Activity/Respondent |
Annual Number of Respondentsa |
Number of Responses per Respondent |
Average Burden per Response (minutes) |
Total Annual Burden Hoursa |
Average Hourly Wage |
Total Annualized Cost |
Study MIS |
|
|
|
|
|
|
1) Grantee staff |
30 |
2,533 |
2 |
2,533 |
$27.86 |
$70,569 |
Total |
30 |
|
|
2,533 |
|
$70,569 |
a All burden estimates are annualized over three years.
Combined Total Burden. Table A.5 summarizes the total estimated reporting burden and costs for the previously approved (field data collection) and currently requested (introductory script, baseline survey, and study MIS) ICRs. If the current request is approved, 4,269 hours and $88,016 would be approved for the PACT study.
Table A.5. Estimate of Burden and Cost for the PACT Evaluation – TOTAL Burden Request
Activity/Respondent |
Annual Number of Respondentsa |
Number of Responses per Respondent |
Average Burden per Response (minutes) |
Total Annual Burden Hoursa |
Average Hourly Wage |
Total Annualized Cost |
Collection of Field Data (Approved April 20, 2012) |
||||||
Selecting Study Grantees Discussions/ grantee and partner organization staff |
50 |
1 |
60 |
50 |
$29.34 |
$1,467 |
Introductory Script and Baseline Survey (Currently Requested) |
||||||
Introductory Script |
|
|
|
|
|
|
1a) Grantee staff |
30 |
70.2 |
10 |
351 |
$27.86 |
$9,779 |
1b) Program applicants |
2,105 |
1 |
10 |
351 |
$4.92 |
$1,726 |
Baseline Survey |
|
|
|
|
|
|
2) Study participants |
2,000b |
1 |
30 |
1,000 |
$4.92 |
$4,920 |
Study MIS (Currently Requested) |
||||||
Study MIS |
|
|
|
|
|
|
3) Grantee staff |
30 |
2,533 |
2 |
2,533 |
$27.86 |
$70,569 |
Total |
|
|
|
4,285 |
|
$88,461 |
a Burden estimates are annualized over three years.
b Note that the 2,000 “study participants” are part of the 2,105 “program applicants,” as the study participants will all begin as program applicants. (Five percent of program applicants are not expected to agree to participate in the study, thus there are 5% more program applicants than study participants.) The 2,000 study participants do not represent 2,000 individuals in addition to the 2,105 program applicants.
13. Estimates of Other Total Cost Burden to Respondents and Record Keepers
These information collection activities do not place any additional costs on respondents or record keepers.
14. Cost to the Federal Government
As reported in PACT’s initial ICR for field data collection, the total cost of the PACT study to the federal government is estimated to be $19,225,787. Since the study will last five years, the total cost over this three year request is $11,535,471 and the annualized cost to the federal government is $3,845,157.
15. Explanation for Program Changes or Adjustments
This submission is for additional data collection under the Parents and Children Together evaluation and therefore increases total burden under this information collection request.
16. Plans for Tabulation and Publication and Project Time Schedule
Baseline Survey. Baseline survey data will be used in the impact and implementation/qualitative only analyses.
First, baseline survey data will be used to describe the characteristics of RF program participants across both impact and implementation/qualitative only sites. For each site, we will present tables of frequencies and means for key participant characteristics, including demographic and family information.
Baseline survey data will also be used in the impact analysis to test for baseline equivalence, define subgroups, improve the precision of impact estimates, and estimate factors that predict participation in the program. The goal of the impact analysis is to provide statistically valid and reliable estimates of the effects of RF programs on the outcomes of participants. To do so, we will compare observed outcomes for members of a randomly selected program group—individuals eligible for program services—with outcomes for members of a control group that was not offered program services. We will use the experience of the control group as a measure of what would have happened to the program group members in the absence of the RF program. Random assignment of fathers to a program and a control group ensures that the two groups of fathers do not initially differ in any systematic way on any characteristic, observed or unobserved. Any observed differences in outcomes between the program and control group fathers can therefore be attributed to the program with a known degree of precision.
Though random assignment ensures that fathers in the program and control groups do not initially differ in any systematic way, there might still be chance differences between groups. To confirm that there were no differences between the program and control groups before random assignment, we will statistically compare key characteristics—including outcome measures—between the groups at baseline. In particular, to establish baseline equivalence, we will conduct t-tests and F-tests to test for differences between the two groups both overall and separately by site. In these comparisons, we will use the analytic sample, which is composed of respondents to the follow-up survey.
Baseline survey data will also be analyzed jointly with the follow-up survey data to estimate impacts. Using baseline data in the impact analysis will improve the statistical precision of impact estimates. Differences of means or proportions in outcomes between the program and control group would provide unbiased estimates of the impacts of being offered participation in the RF program (referred to as the intent-to-treat effect, or ITT effect). However, the impact analysis will use regression models to estimate the ITT effect, allowing us to control for random differences in the baseline characteristics of program and control group members.
Finally, data from the baseline survey will be used to estimate the impact of receiving program services, the effect of treatment on the treated (or the TOT effect). In many settings, the TOT effect can be calculated by adjusting the ITT effect for the difference between the program and control groups in program participation rates. However, in this context, RF programs offer a range of services, and as a result participants may take up only some of those services. For example, program participants might only attend some group meetings or might choose to participate in only some components of the program. Because we are interested in understanding how the impact of the programs varies with the type and intensity of services received, the TOT effect must be calculated using quasi-experimental methods—techniques that do not rely solely on the study’s random assignment design (see Wood et al. 2011 for an application of these methods). To estimate the TOT effect, we will use data from the baseline survey and from the study MIS to predict program participation; possible predictors include motivation to change, barriers to participation, grantee staff predictions of participation, and information on referral source. If participant self-reports and grantee staff assessments are predictive of participation among program group members, we will be able to estimate the TOT effect in addition to the ITT effect.
Study MIS. Information from the study MIS will be used in the impact analysis to estimate the effect of participating in program services as described above. In particular, information provided at intake by grantee staff, together with predictors of participation from the baseline survey, will be used with service receipt data to estimate the TOT effect.
Study MIS information will also be used for the implementation evaluation. The implementation study will provide summary statistics for key program features:
Enrollment patterns. For example, the average number of new applicants each month.
Services provided by grantees. For example, the average number of group workshops offered each month, the average number of individual service contacts each month, or the percentage of individual service contacts provided in participants’ homes or in the office.
Participation patterns. For example, the number of participants that engage in a group activity within two months of enrollment and the average number of hours of group workshops received by program participants.
We will analyze data from the study MIS for each grantee at three points in time which correspond to the three implementation reports identified in Table A.4. In each report, we will describe enrollment patterns, services provided, and participation patterns over the previous 12 months. Analyses will use basic descriptive statistics, such as means, proportions, and standard deviations. Later analyses may describe how patterns changed over time, such as from the early to late implementation period.
This study is expected to be conducted over a five-year period beginning on September 30, 2011. This ICR is requesting burden for three years. Baseline data collection is expected to begin in September 2012.
Table A.6. Schedule for the Evaluation
Activity |
Date |
Selection of grantees for inclusion in evaluation Intake period for impact study Report on early findings on implementation study Implementation mid-term report First report on qualitative analysis Second report on qualitative analysis Final implementation report Final impact report Final report on qualitative analysis |
March–November 2012 November 2012–August 2014 Spring 2013 Spring 2014 Fall 2014 Fall 2015 Winter 2016 Summer–fall 2016 Summer–fall 2016 |
In addition to reports on the findings from the impact, implementation, and qualitative studies, PACT will provide opportunities for analyzing and disseminating additional information through special topics reports and research or issue briefs. We will also provide a public or restricted use data file for others to replicate and extend our analysis.
Short research or policy briefs are an effective and efficient way of disseminating study information and findings. Mathematica will develop and produce about five three- to four-page research or issue briefs. Topics for these briefs will emerge as the evaluation progresses but could, for example, summarize key implementation, impact, or subgroup findings or describe the study purpose and grantees.
17. Reason(s) Display of OMB Expiration Date Is Inappropriate
All instruments will display the expiration date for OMB approval.
18. Exceptions to Certification for Paperwork Reduction Act Submissions
No exceptions are necessary for this information collection.
REFERENCES
Avellar, Sarah M., Robin Dion, Andrew Clarkwest, Heather Zaveri, Subuhi Asheer, Kelley Borradaile, Megan Hague Angus, Timothy Novak, Julie Redline, and Marykate Zukiewicz. “Catalog of Research: Programs for Low-Income Fathers.” Washington, DC: Mathematica Policy Research, June 2011.
Downey, G., and J.C. Coyne. “Children of Depressed Parents: An Integrative Review.” Psychological Bulletin, vol. 108, 1990, pp. 50–76.
Gelfand, D.M., and D.M. Teti. “The Effects of Maternal Depression on Children.” Clinical Psychology Reviews, vol. 10, 1990, pp. 329–353.
Guzzo, Karen Benjamin, and Frank F. Furstenberg, Jr. “Multipartnered Fertility Among American Men.” Demography, vol. 44, 2007, pp. 583–601.
Kroenke, Kurt, Robert L. Spitzer, and Janet B.W. Williams. “The PHQ-9: Validity of a Brief Depression Severity Measure.” Journal of General Internal Medicine, vol. 16, no. 9, 2001, pp. 606-613.
Kroenke, Kurt, Tara W. Strine, Robert L. Spitzer, Janet B.W. Williams, Joyce T. Berry, and Ali H. Mokdad. “The PHQ-8 as a Measure of Current Depression in the General Population.” Journal of Affective Disorders, vol. 144, no. 1, 2009, pp. 163–173.
Löwe, B., J. Unützer, C.M. Callahan, A.J. Perkins, and K. Kroenke, “Monitoring Depression Treatment Outcomes with the Patient Health Questionnaire-9.” Medical Care, vol. 42, no. 12, 2004, pp. 1194-1201.
Martin, A., W. Reif, A. Klaiberg, and E. Braehler, “Validity of the Brief Patient Health Questionnaire Mood Scale (PHQ-9) in the General Population.” General Hospital Psychiatry, vol. 28, no. 1, 2006, pp. 71-77.
McLanahan, Sara. “Fragile Families and the Reproduction of Poverty.” Annals of the American Academy of Political and Social Science. vol. 621, no. 1, 2009, pp. 111–131.
Pearson, Jessica, Lanae Davis, and Jane Venohr. “Parents to Work!” Denver: Center for Policy Research, February 2011.
Pinto-Meza A., A. Serrano-Blanco, M.T. Peñarrubia, E. Blanco, and J.M. Haro, “Assessing Depression in Primary Care with the PHQ-9: Can It Be Carried Out over the Telephone?” Journal of General Internal Medicine, vol. 20, no. 9, 2005, pp. 738-742.
Singer, Eleanor, John Van Hoewyk, and Mary P. Maher. “Does the Payment of Incentives Create Expectation Effects?” Public Opinion Quarterly. vol. 62, 1998, pp. 152–164.
U.S. Census Bureau. “Family Structure and Children’s Living Arrangements.” Available at [http://www.childstats.gov/americaschildren/famsoc1.asp.] Accessed December 24, 2011.
U.S. Government Accountability Office. “Healthy Marriage and Responsible Fatherhood Initiative: Further Progress Is Needed in Developing Risk-Base Monitoring Approach to Help HHS Improve Program Oversight.” Washington, DC: U.S. Government Accountability Office, September 2008.
Wood, Robert G., Sheena McConnell, Quinn Moore, Andrew Clarkwest, and JoAnn Hsueh. “Strengthening Unmarried Parents’ Relationships: The Early Impacts of Building Strong Families.” Princeton, NJ: Mathematica Policy Research, May 2010.
Wood, Robert G., Quinn Moore, and Andrew Clarkwest. “BSF’s Effects on Couples Who Attended Group Relationship Skills Sessions: A Special Analysis of 15-Month Data.” Princeton, NJ: Mathematica Policy Research, May 2011.
1 Additional implementation and qualitative data collection instruments will be submitted to OMB for approval at a later date, and a separate ICR with a separate MIS specific to HM programs will be proposed for any HM grants selected for implementation/qualitative evaluation.
2 A separate ICR with a separate baseline instrument specific to HM programs will be proposed for any HM grants selected for impact evaluation.
3 As is discussed below, this ICR relates to instruments for evaluations of RF programs, so only past RF research is discussed here.
4 If it is determined to meet evaluation requirements, in some cases we will provide training for site staff so that they may conduct the CATI interview.
5 For example, February 21, 2012, conversation with Joseph Jones, Jr., President and CEO of Center for Urban Families, a fatherhood program in Baltimore.
6 We estimate that there will be 24 MIS entries to record group sessions, an additional 24 entries to record case management services, and an entry for program exit. Based on the Building Strong Families Study and the speed at which data can be entered, we expect each entry to take about 2 minutes.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | 06997 PACT OMB |
| Subject | OMB Package Part A |
| Author | Sheena McConnell (formatted by Sheena Flowers) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-30 |