Phase II Evaluation Activities for Implementing a Next Generation Evaluation Agenda for the Chafee Foster Care Independence Program
Full OMB Information Collection Request
New Collection
Supporting Statement
Part A
July 2016
Submitted By:
Office of Planning, Research and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C St, SW
Washington, DC 20201
A. Justification
A1. Necessity for the Data Collection
The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) requests permission to conduct formative evaluations of programs that include, but are not limited to, employment and postsecondary education for youth transitioning to adulthood from foster care.
Although considerable research over the past several decades documents the challenging early adult outcomes of youth who have aged out of foster care, very little is known about the effectiveness of interventions aimed at improving youth outcomes. To understand the current program and practice landscape, the U.S. Department of Health and Human Services, Office of Planning, Research, and Evaluation (OPRE) has contracted with the Urban Institute (Urban) and its partner Chapin Hall at the University of Chicago (Chapin Hall) to identify promising programs and services for youth transitioning to adulthood from foster care. ACF will use the information the project team collects to plan future evaluation activities. ACF is engaging in this collection at the agency’s discretion, and these activities help to fulfill the requirement for evaluations of promising programs under the legislation for the Chafee program.
We’re seeking OMB approval to engage in 6 data collection activities to be conducted in association with site visits to programs serving youth transitioning to adulthood from foster care. These activities will support ACF’s work developing an evaluation agenda as part of the project, Phase II Evaluation Activities for Implementing a Next Generation Evaluation Agenda for the Chafee Foster Care Independence Program. The evaluations will help ACF document the current state of interventions in the stated domain areas of interest and determine different programs’ readiness for potential rigorous evaluation in the future.
Data collection activities include interviews with program leaders, partners, stakeholders, and front-line staff; focus groups with program participants; and the collection of administrative data.
Study Background
The John H. Chafee Foster Care Independence Program (CFCIP) was created following the passage of the Foster Care Independence Act (FCIA) of 1999 (Public Law 106-169). The program provides assistance to help youth currently and formerly in foster care achieve self-sufficiency by providing grants to States and eligible Tribes that submit an approved plan. Activities and programs allowable under the CFCIP include, but are not limited to, help with education, employment financial management, housing, emotional support and assured connections to caring adults for older youth in foster care. The FCIA also required ACF to establish a data collection and performance measurement system for the CFCIP.
The Multi-Site Evaluation of Foster youth Programs
The Foster Care Independence Act of 1999 mandates that a portion of Chafee Program funding be used for rigorous evaluation of independent living programs that are “innovative or of potential national significance.” Under this mandate, the current project team conducted the first rigorous evaluation, the Multi-Site Evaluation of Foster Youth Programs, between 2003 and 2011 and found few statistically significant effects on youth outcomes. But other important lessons from the long-term study were that many potential programs were not in a position to be evaluated rigorously for reasons such as their small size, not having clearly defined logic models, or having poor targeting of youth participants, among other challenges. An earlier review of non-experimental studies reached similar conclusions (Montgomery et al, 2006).
Implementing a “Next Generation” Evaluation Agenda
In response to critical limitations in our knowledge about independent living programs, ACF, through this current project, is developing a plan and strategy for future evaluation activities before launching its next rigorous evaluation. In this work, we are continuing and expanding on activities the project team began in 2012 during Phase I of the project.
During Phase I the project team reviewed current research on youth in foster care and developed a conceptual framework outlining core developmental assets youth need for success in adulthood. The team also created a typology of existing independent living programs, identifying 10 different domains and describing available research evidence on each. With ACF guidance, they consulted with researchers, federal staff, and program experts in three areas of special interest to ACF—education, employment, and financial literacy and asset-building programs -- to further aid research planning. Finally, the team outlined potential design options and next steps for narrowing and identifying promising programs for future evaluation.
For the current Phase II activities the team is selecting promising programs to document the state of those promising programs for the field and to assess as well as develop the programs’ readiness for summative evaluation, including formalizing their logic models. Strong logic models explicitly define the population a program intends to affect and makes clear how and why particular program “inputs” (e.g., financial resources and personnel) are expected to produce certain “outputs” (program activities and program participation), and how and why those outputs will lead to clearly defined outcomes/impacts (short-, medium-, and long-term). With OMB approval we will proceed with the formative process and outcome evaluation in each site. The evaluations will assess each program’s readiness for rigorous impact evaluation, and, in some cases, assist programs to become ready for impact evaluation.
A2. Purpose of Data Collection Procedures
The purpose of the data collection procedures, which include site visits, one-on-one interviews (either in-person or by phone) and focus groups, is to gather detailed information about program-sites’ service models, target populations, inputs, outputs, intended outcomes, and program implementation. The data collection will also gather detail about program-sites’ readiness for rigorous evaluation.
Research Questions
Our core study questions for understanding program site models and implementation include:
Who does the program target?
How many youth are served?
What is the outreach and referral process?
What are the eligibility requirements?
What is the program’s logic model?
What are the primary inputs, outputs, and intended outcomes?
What is the context of the program? What other similar or related services or programs serve the same population?
Our core study questions for assessing program sites’ readiness for rigorous evaluation include:
How well does the program’s logic model reflect actual program practice?
How valid or reliable are the data and measures programs currently use to evaluate outcomes?
What would the burden on program staff be to implement new measures if necessary?
Is the program large enough to make rigorous evaluation feasible?
What further program development would be required for a rigorous impact evaluation?
Is a rigorous impact evaluation feasible?
Study Design
The study design involves formative evaluations that among other objectives detail each program’s logic model to assess how well the ‘logic’ conforms to each program’s actual operations using readily available data. The data collection activities, including up to 4 site visits, will focus on obtaining information about structural program components (e.g., hours of operation or training, class sizes (if applicable), intake process, process for identifying target participants, eligibility criteria), as well as process and content components. Process components include matching internships to youth interests in an employment program, or mentor-student interactions in a college success program. Content components could include curricula or class schedule offerings, which might involve interviews with staff or reviewing course schedule materials. The collection of this data will be used to detail each program’s logic model and to assess how well the program’s processes, as currently implemented, are likely to achieve intended objectives. The data collection will also be used to document and assess changes to the logic model and procedures resulting from the project team’s review of the program model, procedures, and data.
The data collection, which involves speaking with all staff and stakeholders knowledgeable about the programs, is the best approach for obtaining the information we need to a) better understand the services currently being delivered and document the current state of the field in relation to these services; and b) assess if programs are ready for summative evaluation and/or what programs would need to be ready for rigorous evaluation.
Universe of data collection efforts
The 5 data collection activities will be the following:
Interviews with program leaders. Program leaders include practitioners from state and local agencies and private organizations. We will recruit participants for these interviews via email (see Appendix A). The interviews will be semi-structured and will include informed consent (Appendix C). The goals of the interviews are to understand the program features and history, recruitment and referral processes, administrative program data, logic model, target population, intended outcomes and potential rigorous evaluation designs. Interviews will also cover the plan for the formative evaluation and what will be needed and expected from staff (Appendix C).
Interviews with program partners and other stakeholders. Program partners and other stakeholders include practitioners from state and local agencies, private organizations, providers, or employers (for employment programs), educators (for educational programs), or others who work with the selected program or program participants in some capacity (e.g., referring, hiring, or training participants; providing funding or other support etc.). We will recruit participants for these interviews via email (see Appendix A). The interviews will be semi-structured and will include informed consent (Appendix D). The goal of the interviews is to understand partners or other stakeholders’ relationship to the program and type and frequency of interactions with the program as well as perceptions about program goals and priorities (Appendix D).
Interviews with front-line program staff. Front-line program staff includes staff who work directly with program participants or front-line staff from partnering agencies or organizations. We will recruit participants for these interviews via email (see Appendix A). The interviews will be semi-structured and will include informed consent (Appendix E). The goal of the interviews are to understand staff training and background, history and features of the program, major roles and responsibilities, program data elements and uses, perceptions of participants and participants’ experiences in the program (Appendix E).
Focus groups with program participants. Program participants will include young adults 18 and older who are currently or formerly involved in foster care and eligible for services under the Chafee Foster Care Independence Program. We will ask for program staff assistance in recruiting participants for these focus groups (Appendix B). Participants need to have taken part in the program and be able to speak knowledgeably about their experiences – including successes and challenges. The goal of the semi-structured interviews, which will include informed consent (Appendix F), are to understand how participants learn about the program, their motivation for participating, their understanding of the program components and goals, perceptions of staff, perspectives on what additional help the staff or program could provide, and whether and how the program influences how participants think about themselves or the future (Appendix F).
Compiling and accessing program administrative data files (electronically or hard copies). This activity includes working with program data staff most familiar with administrative data and procedures for collecting and tracking information about program participants and services. The goal of this activity is to assess which data elements are best suited for evaluating the logic model and program inputs, outputs, and outcomes. The goal of the activity is to also identify data elements or data collecting procedures programs may need adopt or track in order to improve their readiness for rigorous evaluation
The project team will collect the information mostly through in-person interviews and focus groups at the program sites. The team will also access data from program staff who compile and submit it to the project team. The team’s communication with program staff will include discussing the types of indicators available in the administrative data and how often and how well it is collected, and whether the data are appropriate for estimating short and intermediate outcomes that conform to the program’s logic model.
Site Visits (interviews and focus groups):
The project team will visit each site approximately 4 times and will follow discussion topic guides for the interviews and focus groups (see Appendices C, D, E, F). The team will use the first visit to interview program leaders and program partners or other stakeholders about the program’s history and components, and program oversight processes. The team’s second site visits will involve interviewing front-line staff and will focus on capturing the referral and service procedures, and intended outputs and outcomes. In addition, we will also use this visit to conduct follow up interviews with program partners and other stakeholders about the program and their organization’s role and responsibilities.
The second site visit will also include focus groups with program participants. The project team will conduct two focus groups with up to 12 participants at each program. The focus groups will last approximately 2 hours and will be conducted by two members of the project team.
The third visit will focus on interviewing front-line staff about their roles and responsibilities. During this visit the project team will speak with the program’s data staff to learn about the administrative data. The project team will also work with the data staff to develop procedures for compiling and securely sending any information the project team would need to analyze the data.
The team’s fourth site visits will focus on any changes to the program model or program procedures resulting from the research team’s review of the logic model and data collection activities from the second and third site visits. The visit will involve interviewing program leaders, front-line workers, program partners and other stakeholders about the program’s capacity to sustain or implement procedures that would support rigorous evaluation.
The team will also present findings from the formative evaluation to program leadership and to seek their feedback. Most data collection efforts will occur at the program site, but the team may conduct some interviews by telephone if staff and other stakeholders are not available during the time of the visits.
Recruiting focus group participants: The study procedures include working with program leaders and staff at the selected programs to recruit participants for the focus groups and to provide the physical space for the discussions. As directed by the project team, program staff will recruit young adult participants age 18 and older who are participating or who have participated in the selected programs. The research team will inform program staff of the purpose of the focus groups so staff are prepared to answer questions posed by prospective focus group participants and will provide program staff with an informational fact sheet to aid with recruitment that will describe the purpose of the study and address other logistical questions (see Additional Project Materials).
Accessing program administrative data: In addition to the site visits, data collection will also involve working with program staff to compile and send administrative data. The communication with program staff will include determining appropriate short and intermediate outcomes that conform to the logic model, and discussing how often and how well the data are collected. The communication will also include establishing a process for program staff to send the project team data including outcome indicators.
A3. Improved Information Technology to Reduce Burden
The study will use semi-structured interviews and focus groups. While some technology, such as computer-assisted instruments, may reduce burden on respondents (e.g., enabling respondents to answer questions on their own time), this study requires direct person-to-person communication. The discussion questions are designed to elicit nuanced responses, and the project team will need to probe appropriately. A computer-assisted survey method would not allow the interview flexibility the project requires.
To reduce respondent burden, the project team will hold small-group interviews and focus groups when feasible. Small-group interviews and focus groups will reduce the overall time that a single organization spends on the study. The project team will try to schedule the interviews and focus groups when the input from multiple respondents with comparable roles in the same organization (e.g., case workers, participants) will increase the efficiency and the amount of information the project team can gather in a single session. As described above, with respondents’ permission, the project team will audio record the interviews and focus groups to minimize time needed for potential follow-up for clarification.
A4. Efforts to Identify Duplication
The information collected will not duplicate information that is already available. The project team will review written documents and organizational materials in advance and use the interviews to fill in missing information – which will make the interviews more efficient. The project is designed to gather details about programs and services that will allow ACF to assess whether the program or services would be a strong candidate for future rigorous evaluation. No other studies are exploring these programs or services with these goals in mind or have collected the information the project team needs to make their assessments.
A5. Involvement of Small Organizations
It is possible that some of the organizations recruited into this study for site visits and concomitant interviews will be small organizations. The team will minimize the burden on program staff by keeping the interviews and focus groups as short as possible, by scheduling the interviews and focus groups at a time most convenient for respondents, holding them on-site, and by not requesting written responses.
A6. Consequences of Less Frequent Data Collection
Potential negative consequences of less frequent data collection would be outdated or inaccurate findings. The study design calls for the minimum number of staff and participant hours necessary for successful and complete data collection. To reduce the time burden on program staff and participants, the project team will conduct the interviews and focus groups as efficiently as possible and will work with program leaders and staff to determine the most appropriate respondents for each interview and focus group.
A7. Special Circumstances
There are no special circumstances for the proposed data collection efforts.
A8. Federal Register Notice and Consultation
Federal Register Notice and Comments
A9. Incentives for Respondents
Program leaders and staff will not receive a cash incentive for participating in interviews.
The project team will give program participants who attend the focus groups $25 in order to encourage participation among those young adults who might not otherwise take part in the research. By encouraging otherwise reluctant young adults, the study reduces the risks associated with nonresponse bias – namely the risk that the research team draws inaccurate or biased conclusions about the program.
Respondent participation is voluntary. The $25 for focus group participants is intended to assist with transportation costs, child care, or other expenses that might prevent some in our target population from participating – i.e., those with the greatest financial challenges or other barriers, and whose absence could contribute to nonresponse bias. Research has shown that such incentives are effective at increasing participation from populations with lower education levels (Berlin et al, 1992) as well as low-income and nonwhite populations (James and Bolstein 1990).
Generally, research designs involving focus groups use incentives to offset the requirements that the design poses on participants to appear at a group meeting at a pre-set location and time (Liamputtong 2011). While research on nonresponse bias often applies to household and other surveys, the often experimental studies conclude that the incentives give an added boost to encourage participation among those with less inherent interest in the topic (Groves, Singer, Corning, 2000; Groves et al 2006), those who are more vulnerable due to income or less education (Knoll et al, 2012), and potentially participants who feel less strongly – whether positively or negatively – about the program, and less motivated to participate without some incentive (Groves, Singer, Corning, 2000).
Groves, R., Couper, M., Presser, S., Singer, E., Tourangeau, R., Acosta, G., & Nelson, L.
(2006). Experiments in producing nonresponse bias. Public Opinion Quarterly, 70(5), 720-736.
Groves, R., Singer, E., Corning A. (2000) Leverage-Saliency Theory of Survey Participation: Description and an Illustration. The Public Opinion Quarterly, Vol 64, 3 (299-308).
Knoll et al, 2012. The use of incentives in vulnerable populations for a telephone survey: a randomized controlled trial. Biomed Central, Research Notes 2012 5:572.
Liamputtong, Pranee (2011). Focus group methodology: Principle and practice. Sage Publications.
Based on the project team’s prior experience with studies of similar young adult populations, $25 is high enough to support participation, but not so high that it is overly generous or that participants would feel the incentive is excessive or coercive. The amount is based on what was previously approved by OMB in past studies for focus groups with similar low-income, hard-to-reach populations, such as the Descriptive Study of Tribal TANF programs ($25, OMB #0970-0411, expiration date October 31, 2013), the Goal-Oriented Adult Learning in Self-Sufficiency (GOALS) Study ($25, OMB # 0970-0472, expiration date January 31, 2018), and the study on Same-sex relationships: Updates to Healthy Marriage and Relationship Education ($25, OMB# 0970-0479 , expiration date April 30, 2017).
To prevent the incentive from being coercive the project team will give participants who show up to the focus group the incentive at the time of the group, regardless of whether an individual ultimately chooses to stay and participate.
A10. Privacy of Respondents
Program Participants
The project team will rely on program leaders and staff at each site to recruit participants for the focus groups and to provide the physical space for the discussions. As directed by the research team, program staff will recruit young adult participants age 18 and older who are currently participating or who have participated in the selected programs. The research team will inform program staff of the purpose of the focus groups so staff are prepared to answer questions posed by prospective focus group participants and will provide program staff with an informational fact sheet to aid with recruitment that will describe the purpose of the study and address other logistical questions (see Additional Project Materials).
Program staff will collect and maintain any contact information necessary for recruitment and to coordinate with focus group participants. The project team members will not collect or receive any contact information for focus group participants. To maintain participants’ privacy, the research team will request verbal, not written, consent, at the start of the focus group. Note that the program staff who help with recruitment may be physically present at the sites of the focus groups when the groups are held to help with access to the building and other logistics, but will NOT be permitted in the focus group itself.
Program Leaders, Staff, Program Partners, and other Stakeholders
Program leaders, staff, program partners, and other stakeholders are categories of respondents not designated as vulnerable populations, and the information the project team will collect is not highly sensitive. The team will ask respondents for factual information about their programs (e.g., what the programs do, the number of people they serve, who is eligible, the outreach and referral process). Since some study participants will be leaders, administrators or staff members, and because the team will name the programs in our reports, individuals reading the reports may be able to attribute particular information or comments to that respondent. The project team will tell respondents about this potential risk.
Information will not be maintained in a paper or electronic system from which they are actually ore directly retrieved by an individuals’ personal identifier.
A11. Sensitive Questions
There are no sensitive questions in this data collection. As noted above, the project team will inform respondents that participation is voluntary.
A12. Estimation of Information Collection Burden
Table 1 below shows estimated burden of the information collection, which will take place over two years. The project team will interview approximately 48 program leaders (roughly 6 leaders across 8 program sites), 80 program partners (up to 10 representatives from partner organizations or agencies in each of the 8 program study sites), 128 program front-line staff, and 200 program participants (up to 16 focus groups with 12 participants each, plus additional interviews with up to 8 participants who may not be part of a focus group). The estimated burden also includes 16 program administrators and staff to review information about the study and to respond to an outreach email for interviews and16 front-line staff to assist with recruiting program participants for focus groups. The total annual number of burden hours for this effort is 832. The project team will interview most program leaders approximately 4 times and direct staff twice. Estimated burden also includes time spent by approximately 24 program data staff (roughly 3 staff across 8 program sites) to spend 24 hours over two years reviewing and submitting data files to the project team.
The total annual cost burden to respondents is approximately $17,018.32, as shown in the second to last column of Table 1. For program leaders and program partners the figure ($33.38/hr) is based on the mean wages for “Social and Community Services Managers;” as reported in the 2015 U.S. Department of Labor, Bureau of Labor Statistics, Occupational Employment and Wage Estimates. For program front-line staff the figure ($22.41/hr) is based on the mean wages for “Child, Family and School Social Workers,” as reported in the 2015 U.S. Department of Labor, Bureau of Labor Statistics, Occupational Employment and Wage Estimates. For the compilation and submission of data files, the figure ($21.26/hr) is based on the mean wages for “Statistical Assistants”, as reported in the 2015 U.S. Department of Labor, Bureau of Labor Statistics, Occupational Employment and Wage Estimates.
Wage data for focus group participants is based on the federal minimum wage of $7.25/hr as set by the U.S. Department of Labor.
TABLE 1: Total Burden Requested Under this Information Collection
Instrument |
Total no. of respondents |
Annual no. of respondents |
Number of responses per respondent |
Average burden hours per response |
Annual burden hours |
Hourly wage rate |
Annual respondent costs |
Appendix A: Outreach email for discussion with Program Administrators and Staff |
16 |
8 |
1 |
1 |
8 |
$33.38 |
$267.04 |
Appendix B: Outreach email for Focus Group Recruiters |
16 |
8 |
1 |
8 |
64 |
$22.41 |
$1,434.24 |
Appendix C: Informed Consent and Discussion Guide for program leaders |
48 |
24 |
4 |
1 |
96 |
$33.38 |
$3,204.48 |
Appendix D: Informed Consent and Discussion Guide for program partners and stakeholders |
80 |
40 |
2 |
1 |
80 |
$33.38 |
$2,670.40 |
Appendix E: Informed Consent and Discussion Guide for program front-line staff |
128 |
64 |
1 |
1 |
64 |
$22.41 |
$1,434.24 |
Appendix F: Informed Consent and Focus Group Guide for program participants |
200 |
100 |
1 |
2 |
200 |
$7.25 |
$1,450.00 |
Compilation and Submission of Administrative Data Files |
24 |
12 |
2 |
12 |
288 |
$21.26 |
$6,122.88 |
TOTALS |
|
256 |
|
|
800 |
|
$16,583.28 |
A13. Cost Burden to Respondents or Record Keepers
There are no additional costs to respondents.
A14. Estimate of Cost to the Federal Government
The total cost for the data collection activities under this current request will be $800,000. The annualized cost is $400,000. The estimate includes the costs of project staff time to draft the discussion guides, collect the information, analyze the responses, and write up the results.
A15. Change in Burden
This is a new data collection.
A16. Plan and Time Schedule for Information Collection, Tabulation and Publication
The formative study will occur over two years. The project team will take notes during each interview and prepare a summary write-up of each program or service. The project team will draft a final report and/or briefs that ACF will release publicly, as part of the goal of these data collection activities is to document the current state of interventions in the stated domain areas of interest for dissemination to the field.
Task |
Description |
Timeframe (after OMB approval) |
Site Visit 1 |
Program leader interviews and introduction to the evaluation process |
Month 1 |
Site Visit 2 |
Participant focus groups; front-line staff, leaders, partners, and other stakeholder interviews |
Months 3-4 |
Site Visit 3, Data Access and Year 1 Analysis |
Determining data indicators, assessing quality and conformity to logic model; providing feedback to sites about data |
Months 6-12 |
Site Visit 4, Data Access and Year 2 Analysis |
Follow up interviews with program leaders, front-line staff, program partners and other stakeholders; Outcome analysis of program data |
Months 14-20 |
Reporting and Disseminating findings (including site visit 5) |
Individual formative evaluation reports |
Months 20-24 |
The individual formative evaluation reports, and the analyses on which the reports are based, will be organized around assessing the falsifiability of key assumptions in each program’s logic model. We will customize each report to the selected program and the assumptions the project team tests. But generally, each report will assess assumptions about 1) program context (e.g., key relations and partnerships with other organizations; availability of competing services), 2) target population, 3) target population participation/engagement, 4) implementation of key program elements, and 5) participant outcomes. Each report will conclude with a description of the project team’s assessment of the steps necessary to bring the program to the point of readiness for experimental evaluation and the likelihood that each step could be undertaken within the time frame of Phase II activities.
A17. Reasons Not to Display OMB Expiration Date
All instruments will display the expiration date for OMB approval.
A18. Exceptions to Certification for Paperwork Reduction Act Submissions
No exceptions are necessary for this information collection.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | OPRE OMB Clearance Manual |
| Author | DHHS |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |