Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
Evaluation of the Family Unification Program
OMB Information Collection Request
0970 – 0514
Supporting Statement
Part A
AUGUST 2021
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officer:
Kathleen Dwyer, Ph.D.
Part A
Executive Summary
Type of Request: This Information Collection Request is for an extension without change of an approved information collection (0970-0514). We are requesting three years of approval.
Public housing authorities (PHAs) collaborate with public child welfare agencies (PCWAs) and the local continuum of care (CoC) to identify eligible families and provide them with vouchers and additional services. The current evaluation takes advantage of the recent releases of large numbers of new vouchers. In April 2018, HUD published the “Family Unification Program (FUP) Notice of Funding Availability for Fiscal Years 2017 and 2018” and in October 2019, HUD published the “2019 Family Unification Program Notice of Funding Availability.”
Previous Terms of Clearance: No terms of clearance have been applied to this information collection.
Progress to Date: This multi-site randomized controlled trial evaluation of the Family Unification Program (FUP) was approved on 9/18/2018. FUP provides housing vouchers to homeless or unstably housed families involved in the child welfare system. Since approval, four sites were recruited for cohort 1 (awardees granted vouchers from the FY2018 FUP Notice of Funding Availability (NOFA)), and two sites were recruited for cohort 2 (awardees granted vouchers from the FY2019 NOFA). For all sites, we completed phone interviews for recruitment and development of an evaluation plan. For the four cohort 1 sites, we have completed the first and second of three site visits. For the two cohort 2 sites, we have completed the first site visit. Randomization and referral is complete for the four cohort 1 sites and is on-going for the two cohort 2sites. Across all six sites, we have randomized 822 families as of August 2021. Program data collection is ongoing in all sites. As of August 2021, administrative data have not yet been collected.
Description of Request: The evaluation is examining how the program is implemented in a sample of six sites and will measure impacts on removals into out-of-home care, reunification of children with their parents, and new reports of abuse and neglect. We do not intend for this information to be used as the principal basis for public policy decisions. This request is to extend data collection with no changes. Supporting Statements A and B have been updated to present information in the most current templates and to describe the current status of the information collection.
Timeline: The original timeline assumed that we would recruit the full sample, consisting of individuals at five to ten sites, from the FY2018 NOFA awardees. However, we were only able to recruit four sites from these awardees. To meet sample size requirements, we recruited additional sites from the FY2019 NOFA awardees, thus delaying our timeline. Further, the two sites we selected from the FY2019 NOFA awardees opted to delay the award until September 2020, so their implementation did not begin until the fall of 2020. Given the study design, the timing of data collection is dependent on when sites implement the program (i.e., data collection occurs for 36 months after implementation). Thus, it is expected that data collection will continue until September 2023. While data collection activities for the first four sites have also been delayed as a result of the COVID-19 pandemic, we expect all data collection to be complete well within the proposed new timeline.
A1. Necessity for Collection
The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) is requesting a three-year extension without change of a currently approved information collection (OMB No. 0970-0514).
Although considerable research over the past several decades documents the overlap between child welfare involvement and homelessness, very little is known about the effectiveness of housing vouchers aimed at improving child welfare outcomes and reducing housing instability. The Family Unification Program (FUP), a program of the U.S. Department of Housing and Urban Development (HUD), provides child welfare–involved families with permanent Housing Choice Vouchers. Public housing authorities (PHAs) collaborate with public child welfare agencies (PCWAs) and the local continuum of care (CoC) to administer the program. ACF’s Office of Planning, Research, and Evaluation (OPRE) is conducting a rigorous evaluation of FUP to build the evidence on the effectiveness of housing vouchers for child welfare involved families.
This evaluation includes an impact study to assess the effects of program participation on outcomes of interest and an implementation study to describe and document how the FUP program is implemented in the participating grantee sites. This evaluation is a part of a larger project to help ACF build the evidence base in child welfare through rigorous evaluation of programs, practices, and policies. It also contributes to HUD’s understanding of how housing can serve as a platform for improving quality of life.
ACF awarded a contract to the Urban Institute to conduct this information collection. The purpose of the extension is to complete the ongoing data collection related to the FUP evaluation.
A2. Purpose
Purpose and Use
Data collection activities include interviews with agency heads, program leaders, leaders at partner organizations, and parents; focus groups with front-line staff; and collection of program and administrative data. The purpose of this data collection is to test, through a rigorous evaluation, the impact of participation in the FUP program on preventing children’s placement in out-of-home care and promoting family reunification for children placed in out-of-home care. There has been an increased awareness that homelessness and unstable housing has led to more and longer involvement in the child welfare system. FUP aims to improve child welfare outcomes of homeless and unstably housed child welfare–involved families by providing them with permanent Housing Choice Vouchers. To date, there is limited evidence on FUP’s effectiveness, including one single-site randomized controlled trial (RCT) (Fowler and Chavira 2014) and one quasi-experimental design study (Pergamit, Cunningham, and Hanson, 2017). ACF will use the results to contribute to the evidence base on whether and how housing helps families in the child welfare system. The results from this study will inform HUD about ways to improve the program. Furthermore, ACF and HUD may use the results to identify how to serve unstably housed families in the child welfare system. The information collected is meant to contribute to the body of knowledge on ACF programs and populations. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.
Research Questions or Tests
Impact Study
The impact study seeks to replicate the effects found in the previous RCT (Fowler and Chavira 2014) to build the evidence base on FUP and examine additional child welfare outcomes. The core research questions for the impact study include:
Do FUP vouchers improve child welfare outcomes?
Does FUP reduce the probability that a child is removed and placed into out-of-home care (removal)?
Does FUP increase the probability that a child in out-of-home care is reunified with the child’s family? Does FUP decrease the time to reunification?
Does FUP reduce the number of new reports of child maltreatment?
The program may have impacts on other outcomes beyond preservation and reunification including mediating outcomes. Providing FUP vouchers should reduce homelessness and stabilize housing. These mediating outcomes provide stability for families to engage in the activities they need to undertake to keep their family together or reunify with their children who are in out of home care. Further, through higher and faster rates of reunification and reduction of removals, families may spend less time in the child welfare system. For instance, once a family has reunified they are likely to have their child welfare case closed. Therefore, we are also interested in supplemental impact study research questions:
Does FUP increase the probability that a child welfare case will be closed?
Does FUP decrease the amount of time a child welfare case is open?
Does FUP reduce emergency homeless shelter stays?
Implementation Study
The implementation study research questions focus on describing the design and execution of the model at each site, which can vary based on target populations, supplemental support services, and coordination between the partnering agencies. Documenting this variability provides context for understanding the impact study findings as well as helps identify core components of the FUP model.
Our core implementation study questions include:
Which families are targeted by the public child welfare agency for FUP?
How is the public child welfare agency identifying eligible families?
What types of services are provided along with the FUP housing subsidy?
Which agency provides these services?
What is the nature and frequency of the services?
What data are the public housing authority and public child welfare agency collecting as part of the FUP program?
How is the partnership between the PHA, the PCWA, and the CoC structured?
What are the major implementation challenges and key facilitators to successful implementation of the model?
What share of families who receive FUP vouchers sign a lease and maintain their housing?
What are the barriers and facilitators to a family signing a lease and to maintaining their housing?
What are the relevant aspects of the local demographic, housing, economic, and service environment?
How do these relevant aspects shape the FUP program in each site?
How do families experience FUP?
Which families benefit most from the program and under what conditions?
How do differences across sites in each aspect of their FUP models (target population, identification process, partnerships, housing assistance, case management, support services, and local context) relate to possible outcome differences across sites?
How has the COVID-19 pandemic impacted the structure, implementation, or coordination of services?
Study Design
The study design involves two concurrent components: an impact study and an implementation study. The implementation study provides important context for understanding the impact study results. The two studies will be analyzed simultaneously. Table A1 below describes the data collection that will be completed as a part of both studies.
As discussed in Supporting Statement B, we recruited 6 sites from the PHAs who received more than 40 vouchers through the FY2018 and FY2019 FUP NOFAs. The recruitment was completed. There are two cohorts of sites starting implementation approximately one year apart. The first cohort recruited from the FY2018 FUP NOFA includes Bucks County, PA; King County/Seattle, WA; Orange County, CA; and Phoenix, AZ, and started implementation in 2019. The second cohort recruited from the FY2019 FUP NOFA includes Chicago, IL and Santa Clara County, CA and started implementation in 2020.
Impact Study
The impact study is designed to determine the impacts of FUP on the primary outcomes of family preservation and reunification through a randomized controlled trial (RCT). For the first cohort, as of August 2021, randomization is complete for all sites (Bucks County, PA; King County/Seattle, WA; Orange County, CA; and Phoenix, AZ). Randomization started in the fall of 2020 for the two new sites (Chicago, IL and Santa Clara County, CA) and as of August 2021, is expected to continue for another six months. Across all 6 sites, 822 families have been randomized as of August 2021. As discussed in more detail in Supporting Statement B (B6) the evaluation team will estimate impacts using both Intent-to-Treat (ITT) and Treatment-on-the-Treated (TOT) analyses. The ITT estimate is defined as the difference between the average outcomes for the treatment group and the control group, adjusting for pre-randomization covariates. The TOT estimates the impact of the program per family in the treatment group that are referred and receive the intervention. For the TOT analyses, we define the treated group as those who obtain a lease using a FUP voucher. ITT estimates are of interest to policymakers who want to know whether offering an intervention is effective at addressing the problem it was chosen to solve. TOT estimates are of interest to program and practice stakeholders who want to know how the program impacted those who received the services. The impact study will exclusively use administrative data (instrument 18) in its analysis, which will come from three sources: the PCWA, the PHA, and the CoC.
Implementation Study
The implementation study will capture the differences in each site’s context, allowing for interpretation of findings regarding outcomes measured in the impact study. Special attention will be paid in this analysis to determine the differences in FUP implementation across sites and the extent to which sites are implementing the same program model. Data collection for the implementation study consists of interviews with parents and interviews and focus groups with staff from organizations involved in implementing FUP. Data collection takes place over three virtual or in-person site visits.
The first site visit took place prior to implementation. During this site visit, we set up the randomization process and trained staff on program data collection. The first site visit has been completed for all six sites.
The second site visit occurs once sites have had about half of the families in the treatment group obtain a lease using a voucher, typically around 9-12 months after implementation begins. This site visit focuses on fidelity to the referral process and the process of getting families housed (instruments 1-9). As of August 2021, the second site visit has been completed for the first cohort of sites (Bucks County, PA; King County/Seattle, WA; Orange County, CA; and Phoenix, AZ).
The third site visit will occur once sites have had about half of the families in the treatment group have had a lease using a voucher for 6 or more months, typically around 24-27 months after randomization has begun and will focus on the services provided to families while in housing and families’ ability to maintain their housing (instruments 8-11). As of August 2021, none of the sites have completed the third site visit.
In addition to conducting interviews and focus groups, the Urban Institute asks caseworkers across all sites to complete several forms to help understand each site’s program. The housing status form (instrument 12) will collect information on a family’s current housing situation for the families on the caseworkers’ caseloads at the beginning of the evaluation or entering the child welfare system while FUP vouchers are available. Currently none of the sites are able to collect this information and thus have not completed this form due to competing priorities for the agencies. The agencies have been overwhelmed due to COVID, including facing high staff turnover and being understaffed. They have expressed that they do not have the capacity to complete this form. This will not impact the internal validity of the study but will hinder our ability to answer questions about how FUP families compare to other child welfare involved families. The referral form (instrument 13) collects information on the family being referred including a household roster, housing status, and child welfare status. In all six study sites, there was an existing form used to refer families from child welfare to the housing authority that was either replaced with instrument 13 or from which we could gather necessary information and reduce the number of questions asked of respondents. The referral form along with the family’s unique identification number and whether they are referred to prevent removal of a child (preservation) or to facilitate reunification of a child in out-of-home care (reunification), were entered into the randomization tool (instrument 14). The randomization tool (instrument 14) is an online system that randomizes families to either be referred to FUP or receive services as usual and relays this information to the site. All six sites have used the randomization tool. As of August 2021, four of the sites have finished randomization and are no longer using the randomization tool. The dashboard (instrument 17) collects information on how the family moves through the referral and leasing process including key dates, such as referral date, voucher issuance date, and lease signing date. The dashboard allows the project team to track how families that are randomized to FUP move through the referral pathway and into housing. This dashboard is collected from all six sites on a twice monthly basis and will continue for the first two years of implementation.
The housing assistance questionnaire (instrument 15) and ongoing services questionnaire (instrument 16) collect information on which services the family is receiving through FUP. These data sources allow the project team to compare the services families receive with the description of services in the site’s logic model and across sites. The housing assistance questionnaire (instrument 15) is filled out by the PCWA caseworker or other service provider once for each family immediately after a family signs a lease or upon voucher denial. The on-going services questionnaire (instrument 16) is collected once for each family 6 months after they sign a lease or upon exiting housing if the family exits before 6 months in housing. As of August 2021, all of the sites have started collecting the housing assistance questionnaire and all of the first cohort of sites have started collecting the on-going services questionnaire (instrument 16). As of August 2021, the new sites have not had any families qualify to collect the on-going services questionnaire (instrument 16) yet. The project team will use information from the implementation study to provide important context for findings regarding outcomes of interest in the impact study. Data will be analyzed using qualitative analysis techniques and triangulation of data across data sources.
Table A1 below lays out the timing, respondents, topic, purpose, mode and duration of each instrument to be used for data collection. The following previously approved recruitment and initial site visit activities have been completed:
Guide for Recruitment with PHA and PCWA Administrators
Guide to Develop an Evaluation Plan for PCWA FUP Management
Guide to Develop an Evaluation Plan for PHA FUP Management
Table A1
Data Collection and Research Question Crosswalk
Data Collection Activity |
Instruments |
Respondent, Content, Purpose of Collection |
Mode and Duration |
Second Site Visit
|
Instrument 1- Guide for Implementation Study for PCWA Management |
Respondents: PCWA Management
Content:
Purpose: PCWA management knowledge and perspective |
Mode: In-person or virtually (i.e., phone, video)
Duration: 1 hour |
Second Site Visit
|
Instrument 2- Guide for Implementation Study for PHA Management |
Respondents: PHA Management
Content:
Purpose: PHA management knowledge and perspective |
Mode: In-person or virtually (i.e., phone, video)
Duration: 1 hour |
Second Site Visit
|
Instrument 3- Guide for Implementation Study for CoC Management |
Respondents: CoC Management
Content:
Purpose: CoC management knowledge and perspective |
Mode: In-person or virtually (i.e., phone, video)
Duration: 1 hour |
Second Site Visit
|
Instrument 4- Guide for Implementation Study for Referral Provider Administrators |
Respondents: Referral Provider Management
Content:
Purpose: Referral provider administrator knowledge and perspective |
Mode: In-person or virtually (i.e., phone, video)
Duration: 1 hour |
Second Site Visit
|
Instrument 5- Guide for Implementation Study with PCWA FUP Management (Second) |
Respondents: PCWA FUP Management
Content:
Purpose: PCWA FUP management knowledge and perspective |
Mode: In-person or virtually (i.e., phone, video)
Duration: 1 hour |
Second Site Visit
|
Instrument 6- Guide for Implementation Study for PHA FUP Management |
Respondents: PHA FUP Management
Content:
Purpose: PHA FUP management knowledge and perspective |
Mode: In-person or virtually (i.e., phone, video)
Duration: 1 hour |
Second Site Visit
|
Instrument 7- Guide for Implementation Study Focus Groups with PHA Frontline Workers |
Respondents: PHA Frontline Workers
Content:
Purpose: PHA frontline worker knowledge and perspective |
Mode: In-person or virtually (i.e., phone, video)
Duration: 1.5 hours |
Second & Third Site Visit
|
Instrument 8- Guide for Implementation Study for Parents (Second, Third) |
Respondents: Parents who have signed a lease with FUP
Content:
Purpose: Program participant knowledge and perspective |
Mode: In-person or virtually (i.e., phone, video)
Duration: 1.5 hours |
Second & Third Site Visit
|
Instrument 9- Guide for Implementation Study Focus Groups with Frontline Workers |
Respondents: PCWA Caseworkers and Partner Frontline Staff
Content:
Purpose: PCWA frontline worker and partner agency frontline worker knowledge and perspective |
Mode: In-person or virtually (i.e., phone, video)
Duration: 1.5 hours |
Third Site Visit
|
Instrument 10- Guide for Implementation Study for PCWA FUP Management (Third) |
Respondents: PCWA FUP Management
Content:
Purpose: PCWA FUP management knowledge and perspective |
Mode: In-person or virtually (i.e., phone, video)
Duration: 1 hour |
Third Site Visit
|
Instrument 11- Guide for Implementation Study for Service Provider Management |
Respondents: Service Provider Management
Content:
Purpose: Service provider management perspective |
Mode: In-person or virtually (i.e., phone, video)
Duration: 1 hour |
Program data collection after referral
|
Instrument 12- Housing Status Form |
Respondents: PCWA case workers
Content:
Purpose: Understand how the PCWA is identifying families eligible for FUP |
Mode: Qualtrics or PDF transferred through Secure file transfer protocol (SFTP)
Duration: 0.04 hours |
Program data collection after referral
|
Instrument 13- Referral Form |
Respondents: PCWA case workers
Content:
Purpose: Understand which families are targeted by the PCWA for FUP |
Mode: Qualtrics or PDF transferred through Secure file transfer protocol (SFTP)
Duration: 0.17 hours |
Program data collection after referral
|
Instrument 14- Randomization Tool |
Respondents: PCWA FUP Manager
Content:
Purpose: Randomize eligible families into the treatment or control group |
Mode: Internet-based application
Duration: 0.02 hours |
Program data collection after referral
|
Instrument 15- Housing Assistance Questionnaire |
Respondents: PCWA Caseworker or Services Provider Frontline Workers
Content: Service provision and the application, housing search, and lease up services provided or coordinated for FUP families
Purpose: Understand the types of services provided along with the FUP housing subsidy |
Mode: Qualtrics or PDF transferred through Secure file transfer protocol (SFTP)
Duration: 0.09 hours |
Program data collection after referral
|
Instrument 16- Ongoing Services Questionnaire |
Respondents: Caseworker or Services Provider Frontline Workers
Content: Services provision and types of services FUP families have been referred to or provided
Purpose: Understand the types of services provided along with the FUP housing subsidy |
Mode: Qualtrics or PDF transferred through Secure file transfer protocol (SFTP)
Duration: 0.09 hours |
Program data collection after referral
|
Instrument 17- Dashboard |
Respondents: PCWA and PHA FUP Management
Content: Application submission, voucher issuance, and lease up dates
Purpose: Track treatment group referral, lease signing, and housing stability |
Mode: Secure electronic e-mail or secure file transfer protocol (SFTP)
Duration: 0.17 hours |
First & second administrative data extract
|
Instrument 18- Administrative Data List |
Respondents: PCWA, PHA and HMIS data administrators
Content: Program participant demographics, dates and types of program participant interactions with respondent entities
Purpose: Outcome data to access program impacts |
Mode: Secure file transfer protocol (SFTP)
Duration: 5 hours |
Limitations
Impact Study
The results of the impact study are not designed to be representative or generalizable to all families eligible for FUP but are intended to provide internally-valid estimates of the program’s impact in the six study sites. See SSB, B1 for additional details.
Implementation Study
The results of the implementation study are not designed to be representative of or generalizable to all providers or families who obtain housing through FUP vouchers in the six study sites but are intended to reflect variation in stakeholders’ experiences. See SSB, B1 for additional details
The planned study design combining a quantitative impact study with a qualitative implementation study is the best approach for obtaining the information OPRE needs to better understand the impact of FUP, how the program is currently being delivered, and how implementation varies across sites.
Other Data Sources and Uses of Information
This information collection is the only source of information that will be used for the intended use.
A3. Use of Information Technology to Reduce Burden
Information will continue to be collected as previously described and approved.
Impact study
The two sites that are still randomizing will continue to log into the randomization tool (instrument 14) website and enter the information necessary for randomization. After sites enter a family’s information, they will get immediate feedback with the group to which the family was randomly assigned.
Implementation study
All data collection forms, which include the referral form (instrument 13), the dashboard (instrument 17), the housing assistance questionnaire (instrument 15) and the ongoing services questionnaire (instrument 16), are available in both an electronic and paper format. To reduce burden, sites selected their preferred format. Four sites have elected to use electronic documents while two sites have opted for a paper format. As of August 2021, there are eight remaining sites visit. The evaluation team will audio record the interviews and focus groups to minimize the time needed for potential follow-up to clarify notes. Audio recordings will only occur after the evaluation team has received the respondents’ permission.
A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency
Impact Study
A majority of the data the project team intends to collect already exist in a form required for submission to the federal government through the Administrative Foster Care and Adoption Reporting System (AFCARS).1 This means that the states already track this data. We cannot, however, use AFCARS data directly because it does not include identifiers so we would not be able to identify study families in the data set.
HUD requires PHAs to collect a majority of the administrative data the project team intends to request. However, these data required by HUD do not contain key variables of interest including voucher denial reason and housing exit reason. In addition, these data will not be available in the time period of this project.
Implementation Study
The evaluation team has developed data collection methods to complement or substitute any locally used forms to avoid overlaps in information already collected and to ensure no two forms collect the same information. In addition, if the information collected on a form is available through an administrative data extract, we allow the site to provide data extracts instead of filling out the form. The requested extension will not result in any duplication of effort.
A5. Impact on Small Businesses
As a part of their FUP implementation partnership, some sites have partner organizations that are small organizations. To minimize the burden on program staff, the evaluation team keeps all interviews and focus groups as short as possible, scheduling these discussions at a time that is most convenient for respondents.
A6. Consequences of Less Frequent Collection
Impact Study
The only data collection for the impact study that is at multiple points is the administrative data (instrument 18) which are being collected twice, at one year and two years after the last family is randomized. Each data collection serves its own purpose. The first extract will provide interim findings and allow us to identify any data issues such as ensuring we receive data on all study families and variables are all included and defined correctly, in a timely manner. The second extract will provide our final data for analysis.
Implementation Study
Implementation study data collection in this request is a 1-time data collection. Data collection forms such as the referral form (instrument 13), the randomization tool (instrument 14), the housing assistance questionnaire (instrument 15), and the on-going services questionnaire (instrument 16) each collect different information and are only collected once per family. While the dashboard (instrument 17) is collected every month, the site only needs to add new information since the last dashboard. For interviews and focus groups (instruments 1-11), we do not ask the same information from any individual more than once.
A7. Now subsumed under 2(b) above and 10 (below)
A8. Consultation
Federal Register Notice and Comments
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on 6/9/2021, Volume 86, Number 109, page 30608, and provided a sixty-day period for public comment. During the notice and comment period, we did not receive any comments.
To design the study, the evaluation team based the materials on a previous study of FUP (Cunningham, Pergamit, et al. 2014 & Pergamit, Cunningham, and Hanson, 2017). The previous study was a mixed methods study with both an implementation study and a quasi-experimental impact study. The team consulted with one of the previous study’s principal investigators, Mary Cunningham, on the design of this study.
A9. Tokens of Appreciation
We are not requesting any changes to the previously approved tokens of appreciation provided to respondents.
Impact Study
The impact study relies solely on administrative data, and no tokens of appreciation will be offered for this portion of the data collection.
Implementation Study
The implementation study includes in-depth interviews with parents in the FUP treatment group to understand their experiences with the programs. The project team will give parents who participate in the interviews, which are estimated to take 1.5 hours on average, a $35 token of appreciation in the form of a gift card (or e-gift card if interviews are conducted virtually). While the qualitative data from interviews are not intended to be statistically generalizable to the full population, the study’s findings will be most relevant for policy and practice if the project team is able to secure participation from a wide range of participants, including those with substantial financial challenges or other barriers. This token is intended to offset costs of participation in the study, such as transportation costs, childcare, or other expenses and to help ensure that individuals with more constraints on their ability to participate may take part. The project team anticipates that $35 will serve as a reasonable amount that is high enough to support participation but is not so high as to appear coercive for potential participants.
A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing
Personally Identifiable Information
Impact Study
The impact study will only collect administrative data. The administrative data will not include any personally identifiable information.
Implementation Study
The project team will obtain names, emails, and phone numbers in order to schedule interviews and focus groups with program and agency staff during the site visits. As discussed in more detail in Supporting Statement B, for interviews with FUP families, the Public Housing Authority will administer the informed consent script (appendix A) or the informed consent form (appendix B) to all families who obtained housing using a FUP voucher in order to be able to share parents’ contact information with the project team. The staff will only share this information with the Urban Institute if the family has consented. This information will be used for scheduling purposes, and parent emails or mailing addresses will also be used to send the token of appreciation if they participate in a virtual focus group or interview. Staff will use an outreach call script (appendix E) to recruit parents for the study.
To maintain participants’ privacy, the project team will request verbal consent at the start of each discussion using the informed consent for parents (appendix C). Participants will be provided a physical copy of the consent form before the interview if it is in-person or presented with the consent form via video, email or mail if the visit is virtual.
Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.
Assurances of Privacy
Impact Study
The impact study will only collect deidentified administrative data.
Implementation Study
Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. As specified in the contract, the Urban Institute will comply with all Federal and Departmental regulations for private information. With respondents’ permission, the project team will audio record the interviews and focus groups to minimize time needed for potential follow-up to clarify notes. The project team will ask for consent to record before beginning the interview or focus group.
For in-person interviews, phone interviews, and focus groups with program staff, the project team will use the informed consent document (appendix D) to obtain consent for participation in the study from program staff. This form details the risks and benefits of participating and the level of expected privacy for each participant. Program leaders, staff, program partners, and other stakeholders are categories of respondents not designated as vulnerable populations, and the information the evaluation team will collect is not highly sensitive. The team will ask respondents for factual information about their programs (e.g., what the programs do, the number of people they serve, who is eligible, the outreach and referral process). Because some study participants will be local agency or organization leaders, administrators or staff members, and because the team will name the sites in our reports, individuals reading the reports may be able to attribute particular information or comments to that respondent. The evaluation team will tell respondents about this potential risk.
For in-depth interviews with parents, Urban will use the informed consent for parents (appendix C). This consent statement details the risks and benefits of participating and the level of expected privacy for each participant. Although there are some sensitive questions that will be asked, the questions primarily revolve around the parent’s experience with FUP. Parents will be informed that they may choose not to answer any and all questions during the interview. Information collected during the in-depth interviews with parents will be handled in the same way as with agency personnel, using the project-specific ID created during the referral process.
The project team has also obtained Institutional Review Board (IRB) approval for all data collection (appendix I). An Urban Institute-developed and IRB-approved confidentiality pledge, agreeing to adhere to the data security procedures laid out in the approved IRB submission, will be read and signed during the project training process by all researchers working with the data.
Data Security and Monitoring
The project team will protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The project team has developed a data security plan that outlines how the project will store, transfer and destroy sensitive information as well as the precautions to be taken during each of these activities to ensure the security of those data. The Urban Institute ensures that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements. The data security plan meets the requirements of U.S. federal government agencies and is continually reviewed in the light of new government requirements. Such security is based on (1) exacting company policy promulgated by the highest corporate officers in consultation with systems staff and outside consultants, (2) a secure systems infrastructure that is continually monitored and evaluated with respect to security risks, and (3) secure work practices of an informed staff that take all necessary precautions when dealing with private data.
As specified in their contract, the Urban Institute will use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Urban Institute will securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Urban Institute will ensure that this standard is incorporated into the Urban Institute property management/control system; establish a procedure to account for all laptop computers, desktop computers and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, the Urban Institute must submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or personally identifiable information that ensures secure storage and limits on access.
Our intention is that we will collect from participating sites only de-identified information on FUP participants. We have worked with each site to develop a method to assign each family member a project-specific ID as part of the referral process; the ID will not be related to anything identifiable outside the FUP agencies. Personal identifiers appear on the referral form, but we ask the sites to redact this information before transmitting it as part of the randomization process. The randomization tool (instrument 14), dashboard (instrument 17), the housing assistance questionnaire (instrument 15) and ongoing services questionnaire (instrument 16) will also include only the project-specific ID and not include any identifying information such as name, social security number, address, or phone number. Furthermore, we will request that the project-specific IDs be attached to administrative data (instrument 18) and personal identifiers be removed. Information collected from all interviews and focus groups will not be maintained in a paper or electronic form that includes the individual’s personal identifier. For staff and family interviews, we will maintain a crosswalk of project IDs with individual identifiers to allow us to contact the individuals to clarify responses and/or schedule interviews and focus groups. The crosswalk will be the only document with the individual’s identifiers and will be maintained separately from any documents that include either personal identifiers or project IDs.
The project team will archive the data at the National Data Archive on Child Abuse and Neglect (NDACAN) as required by its contract with ACF with the following provisions:
All personal identifying information will be stripped from the file.
To prevent secondary disclosure, the project team will conduct disclosure analysis and mask, suppress, or categorize any items that could lead to identification of individuals
A11. Sensitive Information 2
ACF is not requesting any changes to the previously approved questions for the interviews and focus groups included in this study. There are no sensitive questions that are asked of program staff. The only sensitive questions that are asked as a part of the data collection are in the guide for the implementation study for parents (instrument 8). The goal of these in-depth interviews with parents is to understand how they have experienced the FUP program including what services they received and how FUP has affected their lives. All sensitive questions are asked purely in the context of how the families experience the program. These questions will be used to describe the program from the family’s perspective. The sensitive topics include:
Mental Health and Substance Abuse Service Receipt. There are questions in the guide that ask whether they received counseling or substance abuse treatment. The guide does not ask about mental health or substance abuse beyond receiving services.
Child Welfare Involvement. All families in the study will have been child welfare involved. In the guide, we ask questions about how the program has impacted their child welfare case.
Material Hardship. The guide also covers how the program has impacted the family’s ability to pay for things they need.
Before starting the in-depth interviews, all respondents will be informed that their identities will be kept private and that they do not have to answer any question that makes them uncomfortable. Although such questions may be sensitive for many respondents, they have been successfully asked of similar respondents in other data collection efforts, such as in the in-depth parent interviews conducted for the supportive housing study of child welfare involved families (Cunningham et al. 2014).
A12. Burden
Explanation of Burden Estimates
The previously approved burden for all the evaluation activities was estimated to be 849 hours per year for three years (2547 total burden hours). We have updated these estimates to reflect the expected status of the information collection in September 2021. This includes removing burden for instruments that are no longer in use and adjusting the expected number of respondents to reflect completed data collection. The number of respondents has also been decreased across all instruments to reflect recruitment of 6 sites; the original request included burden for up to 10 sites.
The following previously approved recruitment and initial site visit activities have been completed, so burden for these have been removed:
Guide for Recruitment with PHA and PCWA Administrators
Guide to Develop an Evaluation Plan for PCWA FUP Management
Guide to Develop an Evaluation Plan for PHA FUP Management
Data collection has begun with the following instruments and therefore the expected number of respondents has been reduced:
Instrument 1 -- Guide for Implementation Study for PCWA Management
Instrument 2 -- Guide for Implementation Study for PHA Management
Instrument 3 -- Guide for Implementation Study for CoC Management
Instrument 4 -- Guide for Implementation Study for Referral Provider Administrators
Instrument 5 -- Guide for Implementation Study with PCWA FUP Management
Instrument 6 -- Guide for Implementation Study for PHA FUP Management (Second)
Instrument 7 -- Guide for Implementation Study Focus Groups for PHA Frontline Workers
Instrument 8 -- Guide for Implementation Study for Parents
Instrument 9 - Guide for Implementation Study Focus Groups with Frontline Workers
Instrument 12 – Housing Status Form (Note: this form has not and will not be collected for most of the sites because of competing priorities, see SSA, A2 for more details.)
Instrument 13 – Referral Form
Instrument 14 – Randomization Tool
Instrument 15 - Housing Assistance Questionnaire
Instrument 16 – Ongoing Services Questionnaire
Instrument 17 – Dashboard
Estimated Annualized Cost to Respondents
We updated cost calculations using current data from the Bureau of Labor Statistics (May 2020 Occupational Employment and Wage Estimates).
The total annual cost burden to respondents is approximately $8,611.44. For program leaders and program partners the figure ($36.13/hr) is based on the mean wages for “Social and Community Services Managers,” job code 11-9151, as reported in the May 2020 U.S. Department of Labor, Bureau of Labor Statistics, Occupational Employment and Wage Estimates.3 For program front-line staff the figure ($25.18/hr) is based on the mean wages for “Child, Family and School Social Workers,” job code 21-1021, as reported in the May 2020 U.S. Department of Labor, Bureau of Labor Statistics, Occupational Employment and Wage Estimates.4 For the compilation and submission of administrative data files, the figure ($25.06/hr) is based on the mean wages for “Statistical Assistants,” job code 43-9111, as reported in the May 2020 U.S. Department of Labor, Bureau of Labor Statistics, Occupational Employment and Wage Estimates.5 Wage data for program participants is based on the federal minimum wage of $7.25/hr as set by the U.S. Department of Labor.
The estimated total annualized burden for this effort is 355 hours and the estimated annualized total respondent cost is $8,611.44. A breakdown of the estimated annualized burden and cost are below in Table A2.
Burden Estimates
Instrument |
No. of Respondents (total over request period) |
No. of Responses per Respondent (total over request period) |
Avg. Burden per Response (in hours) |
Total Burden (in hours) |
Annual Burden (in hours) |
Average Hourly Wage Rate |
Total Annual Respondent Cost |
Guide for Implementation Study for PCWA Management |
2 |
1 |
1.00 |
2.00 |
1 |
$36.13 |
$36.13 |
Guide for Implementation Study for PHA Management |
2 |
1 |
1.00 |
2.00 |
1 |
$36.13 |
$36.13 |
Guide for Implementation Study for CoC Management |
2 |
1 |
1.00 |
2.00 |
1 |
$36.13 |
$36.13 |
Guide for Implementation Study for Referral Provider Administrators |
2 |
1 |
1.00 |
2.00 |
1 |
$36.13 |
$36.13 |
Guide for Implementation Study with PCWA FUP Management (Second) |
2 |
1 |
1.00 |
2.00 |
1 |
$36.13 |
$36.13 |
Guide for Implementation Study for PHA FUP Management |
2 |
1 |
1.00 |
2.00 |
1 |
$36.13 |
$36.13 |
Guide for Implementation Study Focus Groups with PHA Frontline Workers |
6 |
1 |
1.50 |
9.00 |
3 |
$25.18 |
$75.54 |
Guide for Implementation Study for Parents (Second, Third) |
72 |
1 |
1.50 |
108 |
36 |
$7.25 |
$261 |
Guide for Implementation Study Focus Groups with Frontline Workers |
180 |
1 |
1.50 |
270 |
90 |
$25.18 |
$2,266.20 |
Guide for Implementation Study for PCWA FUP Management (Third) |
6 |
1 |
1.00 |
6.00 |
2 |
$36.13 |
$72.26 |
Guide for Implementation Study for Service Provider Management |
5 |
1 |
1.00 |
5.00 |
2 |
$36.13 |
$72.26 |
Housing Status Form |
185 |
31 |
0.04 |
230 |
77 |
$25.18 |
$1,938.86 |
Referral Form |
60 |
10 |
0.17 |
102 |
34 |
$25.18 |
$856.12 |
Randomization Tool |
3 |
200 |
0.02 |
12 |
4 |
$36.13 |
$144.52 |
Housing Assistance Questionnaire |
120 |
3 |
0.09 |
33 |
11 |
$25.18 |
$276.98 |
Ongoing Services Questionnaire |
120 |
3 |
0.09 |
33 |
11 |
$25.18 |
$276.98 |
Dashboard |
12 |
27 |
0.17 |
56 |
19 |
$36.13 |
$686.47 |
Administrative Data List |
18 |
2 |
5.00 |
180 |
60 |
$25.06 |
$1,503.60 |
Total |
|
|
|
|
355 |
|
$8,611.44 |
A13. Costs
There are no additional costs to respondents.
A14. Estimated Annualized Costs to the Federal Government
The total cost for the data collection activities under this request for the next 3 years, including this requested extension, are estimated to be $681,146 for an annualized amount of $227,049. The estimate includes the costs of project staff time to draft the discussion guides, collect the information, analyze the responses, and write up the results.
Cost Category |
Estimated Costs Already Spent |
Estimated Costs for this data collection |
Instrument Development and OMB Clearance |
$107,052 |
$6,043 |
Field Work |
$774,023 |
$498,963 |
Analysis |
$0 |
$117,427 |
Publications/Dissemination |
$0 |
$58,713 |
Total costs over the request period |
$881,075 |
$681,146 |
Annual costs |
$293,692 |
$227,049 |
A15. Reasons for changes in burden
OMB previously approved the burden associated with the implementation and impact components of the FUP evaluation. This request updates the burden estimates to accurately reflect the status of the study and the estimates for the next three years. For example, completed data collection has been removed and adjustments have been made based on actual recruitment numbers. There are no changes proposed to the approved data collections.
A16. Timeline
Table A3 below provides a detailed data collection schedule. For the first cohort recruited from the FY2018 NOFA, awards were granted in November of 2019 with implementation beginning in the spring and summer of 2019. For the second cohort recruited from the FY2019 NOFA, awards were granted in April of 2020 with implementation beginning in the fall of 2020. Preliminary phone calls related to recruitment and evaluation plan development and the first site visit were completed shortly after the grant award. The second site visit, which consists of interviews with program staff, focus groups with front line workers and interviews with parents, will occur 6-9 months after program implementation. Program data collection prior to referral will occur on an on-going basis from program implementation to 12 months after program implementation. The third site visit, which will again consist of interviews with program staff, focus groups with frontline workers and interviews with parents, will occur 18-21 months after program implementation. Program data collection after referral will occur on an on-going basis from program implementation to 24 months after program implementation. Finally, administrative data collection will occur at 24 months and 36 months after program implementation.
Implementation study analysis will be completed at two points: qualitative data analysis will be completed after the third site visit and quantitative analysis will be completed after the receipt of all program and administrative data. Impact analysis will be completed after the receipt of all child welfare administrative data.
The evaluation will result in three publications: a technical report (approximately 18 months after data collection is complete), a practitioner-focused brief (approximately 18 months after data collection is complete), and a journal article (timing to be determined following completion of data collection). See SSB, B7 for additional details.
Table A3
Task |
Description |
Timeframe (after OMB approval) Cohort 1 |
Timeframe (after OMB approval) Cohort 2 |
Second Site Visit |
Interviews and focus groups with PCWA, PHA, and CoC staff, as well as interviews with parents apart of the FUP program. |
Completed |
Month 0-3 |
Third Site Visit |
Interviews and focus groups with PCWA, PHA, and CoC staff, as well as interviews with parents apart of the FUP program. |
Months 0-3 |
Month 12-15 |
Program Data Collection |
Data collection activities such as the randomization tool, referral form, housing status form, housing assistance form, and ongoing services form. |
Months 1-12 |
Months 1-24 |
Implementation Study Analysis |
Implementation coding and analysis; Program data cleaning and analysis |
Months 15-30
|
|
First administrative data extract |
Administrative and outcome measures from the PCWA’s data system, the PHA’s data system, and the CoC’s Homelessness Management Information System (HMIS). |
Month 12 |
Month 24 |
Second administrative data extract |
Administrative and outcome measures from the PCWA’s data system, the PHA’s data system, and the CoC’s Homelessness Management Information System (HMIS). |
Month 24 |
Month 36 |
Analysis |
Administrative data cleaning and analysis |
Months 36-48
|
|
Reporting and Disseminating findings
|
Technical Report, Practitioner-Focused Brief and Journal Article |
Months 48-54
|
|
A17. Exceptions
No exceptions are necessary for this information collection.
Attachments
Instrument 1 -- Guide for Implementation Study for PCWA Management
Instrument 2 -- Guide for Implementation Study for PHA Management
Instrument 3 -- Guide for Implementation Study for CoC Management
Instrument 4 -- Guide for Implementation Study for Referral Provider Administrators
Instrument 5 -- Guide for Implementation Study with PCWA FUP Management
Instrument 6 -- Guide for Implementation Study for PHA FUP Management (Second)
Instrument 7 -- Guide for Implementation Study Focus Groups for PHA Frontline Workers
Instrument 8 -- Guide for Implementation Study for Parents
Instrument 9 - Guide for Implementation Study Focus Groups with Frontline Workers
Instrument 10 - Guide for Implementation Study for PCWA FUP Management (Third)
Instrument 11 - Guide for Implementation Study for Service Provider Management
Instrument 12 – Housing Status Form
Instrument 13 – Referral Form
Instrument 14 – Randomization Tool
Instrument 15 - Housing Assistance Questionnaire
Instrument 16 – Ongoing Services Questionnaire
Instrument 17 – Dashboard
Instrument 18 – Administrative Data
Appendix A – Informed Consent Script
Appendix B – Informed Consent Form
Appendix C – Informed Consent for Parents
Appendix D – Informed Consent for Staff
Appendix E – Outreach Call Script for Parents
Appendix F – Guide for Recruitment with PHA and PCWA Administrators
Appendix G – Guide to Develop an Evaluation Plan for PCWA FUP Management
Appendix H – Guide to Develop an Evaluation Plan for PHA FUP Management
Appendix I – IRB Approval Letter
1 OMB #0970-0422
2 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.
3 “Occupational Employment Statistics: Occupational Employment and Wages, May 2020,” Bureau of Labor Statistics, accessed May 18th, 2021, https://www.bls.gov/oes/current/oes119151.htm
4 “Occupational Employment Statistics: Occupational Employment and Wages, May 2020,” Bureau of Labor Statistics, accessed May 18th, 2021, https://www.bls.gov/oes/current/oes211021.htm
5 “Occupational Employment Statistics: Occupational Employment and Wages, May 2020,” Bureau of Labor Statistics, accessed May 18th, 2021, https://www.bls.gov/oes/current/oes439111.htm
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Hanson, Devlin |
| File Modified | 0000-00-00 |
| File Created | 2021-08-29 |