RPG
National Cross-Site Evaluation and Evaluation Technical
Assistance
Supporting
Statement, Part B
For OMB Approval
december
10, 2018
The Children’s Bureau (CB) within the Administration for Children and Families of the U.S. Department of Health and Human Services (HHS) seeks approval to collect information for the Regional Partnership Grants to Increase the Well-being of and to Improve Permanency Outcomes for Children Affected by Substance Abuse (known as the Regional Partnership Grants Program or “RPG”) Cross-Site Evaluation and Evaluation-Related Technical Assistance project. The Child and Family Services Improvement and Innovation Act (Pub. L. 112-34) includes a targeted grants program (section 437(f) of the Social Security Act) that directs the Secretary of HHS to reserve a specified portion of the appropriation for these RPGs, to be used to improve the well-being of children affected by substance abuse. Under three prior rounds of RPG, CB has issued 74 grants to organizations such as child welfare or substance abuse treatment providers or family court systems to develop interagency collaborations and integration of programs, activities, and services designed to increase well-being, improve permanency, and enhance the safety of children who are in an out-of-home placement or at risk of being placed in out-of-home care as a result of a parent’s or caretaker’s substance abuse. In 2017, CB awarded grants to a fourth cohort of 17 grantees, including 2 grantees serving American Indian or Alaska Native (AI/AN) participants, and in 2018 CB plans to award 10 grants to a fifth cohort. The current information collection request (ICR) is for data collection activities associated with the 27 new grantees. The previous 74 grantees were included in previous ICRs (OMB Control Numbers 0970-0353 and 0970-0444).
The RPG cross-site evaluation will extend our understanding of the types of programs and services grantees provided to participants, how grantees leveraged their partnerships to coordinate services for children and families, and the outcomes for children and families enrolled in RPG programs. First, the cross-site evaluation will assess the coordination of partners’ service systems (e.g., shared participant data, joint staff training) to better understand how partners’ collaborative effort affected the services offered to families (partnerships analysis). The cross-site evaluation will also focus on the partnership between the child welfare and substance use disorder (SUD) treatment agencies, to add to the research base about how these agencies can collaborate to address the needs of children and families affected by SUD. Second, the cross-site evaluation will describe the characteristics of participants served by RPG programs, the types of services provided to families, the dosage of each type of service received by families, and the level of participant engagement with the services provided (enrollment and services analysis). Finally, the evaluation will assess the outcomes of children and adults served through the RPG program, such as child behavioral problems, adult depressive symptoms, or adult substance use issues (outcomes and impacts analysis).
The evaluation is being undertaken by CB and its contractor Mathematica Policy Research. The evaluation is being implemented by Mathematica Policy Research and its subcontractor, WRMA Inc. The evaluator is required to advise CB on the instruments grantees use to collect data from program participants for required local evaluations. Grantees will secure approval from their local institutional review boards for collecting these data.
This ICR requests clearance for obtaining from grantees participant data they collect for their local evaluations, and for directly collecting additional data from grantees and their partners and providers, for the cross-site evaluation. Specifically, this ICR requests clearance for the following data collection activities: (1) site visits with grantees, (2) web-based survey of grantee partners, (3) semiannual progress reports, (4) enrollment and services data provided by grantees, and (5) outcomes and impacts data provided by grantees.
In response to federal legislation, the Promoting Safe and Stable Families Program (Section 437(f), Subpart 2, Title IV-B, of the Social Security Act) (42 U.S.C. 629g(f)), as amended by the Child and Family Services Improvement and Innovation Act (Pub. L. 112-34), includes a targeted grants program (section 437(f) of the Social Security Act) that directs the Secretary of HHS to reserve a specified portion of the appropriation for RPGs to increase child well-being, improve permanency, and enhance the safety of children who are in an out-of-home placement or at risk of being placed in out-of-home care as a result of a parent’s or caretaker’s substance abuse.
The overall objective of the RPG cross-site evaluation is to describe and document the performance of the RPG projects, the outcomes for participants enrolled in RPG, and the effectiveness of the grantees’ approaches, as stated in the legislation. To meet these evaluation goals, the RPG cross-site evaluation includes four study components: (1) a partnerships analysis, (2) an enrollment and services analysis, (3) an outcomes analysis, and (4) an impacts analysis. The partnerships analysis will assess the coordination and collaboration of partners’ service systems. The enrollment and services analysis will collect data on RPG participant characteristics and the types of, dosage of, and engagement with services. The outcomes analysis will describe the characteristics of and changes over time in children, adults, and families who participate in the RPG programs. The impacts analysis will include a subset of 22 of the 27 RPG grantees that are implementing rigorous local evaluation designs and can provide outcomes data on both treatment and comparison group members.
The partnerships analysis will assess the collaboration and coordination of services the RPG projects provided for families. The analysis will examine which partners are involved in each project, the roles they play, and the extent of collaboration among partners, such as sharing a vision and goals to integrating assessment and treatment. In addition, the analysis will explore the interagency collaboration and coordination of the child welfare and substance use treatment agencies, specifically examining topics such as competing priorities within each agency, conflicting timelines of recovery and permanency decisions, and conflicting and limited sharing of data between agencies. Advancing the collaboration and coordination of these two agencies is critical to the success of the RPG partnerships because they aim to serve the same families and support their well-being.
We are requesting clearance for the following two instruments associated with this component of the evaluation:
Grantee and partner staff topic guide (Appendix B). During site visits to 21 of the 27 grantees, semi-structured individual interviews will be conducted with the RPG project director, two managers or supervisors for the RPG project, and two frontline staff who are providing services to participants. Individual interviews will take place with three grantee partners who may represent the child welfare agency or substance use treatment provider.For the 6 grantee sites not visited, telephone interviews will take place with one project director and one program manager in each site.
Partner survey (Appendix C). Lead staff of the grantee and partner organizations will be asked to complete a web-based survey. Partner organizations are defined as organizations other than the grantee that provide RPG services to families enrolled in the program or coordinate their services with the grantee. We estimate that five people will fit this criterion for each of the 27 RPG grantees.
The enrollment and services analysis will describe who was served in the RPG projects and how. The analysis will examine how grantees defined and refined their target populations over the course of their projects and why those changes occurred. It will provide an expanded picture of all core services provided to families enrolled in RPG. Core services are the services defined by the grantee that make up its main RPG project. These include, at a minimum, all services funded by the grant, and might include in-kind services provided by partners. The analysis also seeks to describe how engagement varied across participants and services, and how grantees and their partners collaborated to provide the services.
We are requesting clearance for the following two data collections associated with this component of the evaluation:
Semiannual progress reports (Appendix D). Grantee project directors will complete semiannual progress reports with updated information about their projects, including any changes from prior periods. CB has tailored the semiannual progress reports to collect information on grantees’ services, the target population for the RPG program, project operations, partnerships, and grantees’ perceived successes and challenges to implementation.
Enrollment and services data (Appendix E). These data will describe participants’ characteristics at enrollment and the services they receive. Grantees will record the enrollment date for each RPG family or household and demographic information on each family member including date of birth, ethnicity, race, primary language spoken at home, type of current residence, income (adults only), highest education level attained (adults only), and relationship to a focal child in each family on whom data will be collected. Grantees will also record the enrollment date for families or individual family members into RPG services, service contact information for core services, and exit dates for RPG.
The outcomes analysis will include all 27 grantees. This analysis will describe the characteristics of participating families and their outcomes in the five domains: (1) child well-being, (2) family functioning and stability, (3) adult recovery, (4) child permanency, and (5) child safety.
In this ICR, we are seeking clearance for five measures and administrative data elements associated with the cross-site evaluation’s outcomes analysis (Appendix F and G). From the child’s primary caregiver, grantees will collect outcomes data on child well-being, functioning and stability, safety, and permanency on one focal child in each participating family. However, if the child is in out-of-home placement, then grantees will collect child well-being data from the current caregiver. Each grantee will select a focal child at enrollment based on their target populations and planned services (for example, some grantees plan to serve families with infants or toddlers, and others plan to serve adolescents or teens.) Grantees will administer the instruments collecting data on adult SUD recovery from the same adult, unless he or she is not the adult with an SUD or in recovery from an SUD. In those cases, grantees will collect recovery data from a separate adult receiving RPG services who has or had an SUD.
The impacts analysis will include 22 grantees that are implementing rigorous comparison local evaluation designs and can provide outcomes data on both treatment and comparison group members. In this ICR, we are seeking clearance to collect the same elements contained in the outcomes analysis instruments from participants in the comparison groups.
In this ICR, we are seeking clearance for the two instruments associated with the partnerships analysis. Descriptions of the data collection procedures follow:
Grantee and partner staff topic guide. Two members of the RPG cross-site evaluation team will conduct one site visit to 21 grantees (in Year 4 of the grant program). While on-site, they will conduct in-person, individual interviews with grantee, partner, and frontline staff. Evaluators will obtain verbal consent from each interviewee, including permission to audio record the interviews for later transcription. One team member will moderate the interview. If interviewees do not consent to audio recording, the second team member will use a laptop computer to take detailed notes.
Partner survey. This survey is web-based. The cross-site evaluation team will obtain from the grantees contact information for desired respondents and notify respondents in advance about the survey with an email. Personalized links to the survey (along with both an email address and telephone number where respondents can ask any questions about the survey) will then be distributed to each respondent via email. By clicking on the link or pasting it into their browser, respondents will go directly to the 25-minute survey. If they are unable to complete the survey in one sitting, they can return to it as needed.
Information for the partnerships analysis will be descriptive. In general, it will not involve formal hypothesis testing.
In this ICR, we are seeking clearance for the two instruments associated with the enrollment and services analysis. Descriptions of the data collection procedures follow:
Semiannual progress reports. CB will provide a template for the semiannual progress reports, which grantees will submit every six months. Grantees can enter narrative information directly into the template, or they can respond to the questions in other electronic file formats of their choosing. Grantee project directors will submit their reports to www.GrantSolutions.com.
Enrollment and services data. Intake workers will enter demographic characteristics and RPG program enrollment and exit dates for each RPG case into the case management system known as the RPG-Evaluation Data System (RPG-EDS). Staff delivering services will enter individual service contact information on a rolling basis for the duration of participation in RPG services.
Information for the enrollment and services analysis will be descriptive. In general, it will not involve formal hypothesis testing.
Outcomes analysis data. Each grantee is expected to maintain outcomes data from the case-specific standardized instruments and administrative records for all RPG participants in its project or agency database(s). Grantees will upload these data to the RPG-EDS every six months, using file formats specified or provided by the cross-site evaluation. To maximize data quality, automatic data validation checks will occur during the upload, and error messages will indicate any corrections needed before the submission can be accepted.
Information for the outcomes analysis will be descriptive. In general, it will not involve formal hypothesis testing.
Impacts analysis
Impacts analysis data. Each of the 22 grantees participating in the impacts study is expected to maintain the case-specific outcomes data for comparison group members from standardized instruments and administrative records in its project or agency database(s). Grantees will upload these data to RPG-EDS every six months using file formats specified or provided by the cross-site evaluation. To maximize data quality, automatic data validation checks will occur during the upload, and error messages will indicate any corrections needed before the submission can be accepted.
Based on experience with similar grant projects and data collection efforts, the RPG cross-site evaluation expects to obtain a high response rate of 80 percent or more for all instruments used. A grantee liaison will be assigned to each site and serve as a link to work with grantees, if needed, to address nonresponse. Descriptions of the strategies for maximizing response in the data collection efforts follow.
Conduct interviews with key grantee staff, supervisors/managers, partners, and frontline staff during site visits. All interviews with key grantee staff, supervisors/managers, partners, and staff will occur during site visits. We anticipate that all grantees will agree to participate in these visits. Our experience on many similar studies involving grantees indicates that participation rates of the desired interviewees are typically close to 100 percent. To help ensure high participation, we will coordinate with the grantees, supervisors/managers, partners, and staff to determine convenient dates and schedules for these visits.
Design partner survey in a manner that minimizes respondent burden. To minimize burden on respondents, the surveys are brief, web-based, and structured such that respondents do not have to pay attention to routing and skip logic or view questions that do not apply to them.
Send advance and reminder emails to respondents (Appendix H and I). We will send advance emails and an FAQ document to grantee and partner staff requesting their participation. If respondents have not completed the survey within a certain amount of time, we will send reminder emails requesting them to complete the surveys.
Solicit the help of grantees to encourage completion of the partner surveys. If response rates for individual grantees lag, the cross-site evaluation team will work with lead grantee staff to identify additional strategies for increasing completed surveys without compromising respondent confidentiality. For instance, lead grantee staff may be asked to send an email to all the survey participants they had identified in their site, encouraging everyone’s response. In past rounds, this approach helped boost response rates, because lead grantee staff had personal relationships with their partners and used their proximity to encourage responses. This approach of combining follow-up requests from the evaluator to people who have not completed the survey with general requests from the grantee to all desired respondents has proved effective in multiple Mathematica projects that involved collecting similar data through web-based surveys, including in prior surveys for CB.
Conduct telephone follow-up with nonrespondents on the partner survey. If email reminders and requests from the grantee prove ineffective, the cross-site evaluation team will deploy survey staff with expertise in obtaining responses to conduct one round of telephone follow-up with nonrespondents. This approach of following up via telephone when email requests have not been effective has increased response rates in multiple Mathematica projects that involved collecting similar data through web-based surveys, including in prior surveys for CB with former RPG grantees.
Provide an easy-to-use data entry system for enrollment and services data. The design of the enrollment and service data application of RPG-EDS is based on web-based case management systems that Mathematica has developed and successfully implemented for multiple projects that collect these data from similar types of providers. RPG-EDS can be accessed from any computer, allowing for ease of entry, while the data are encrypted in transit and at rest and reside behind firewalls, thereby maintaining data security.
Use multiple sources to check enrollment activity and completion of the enrollment and services data entry. Information on the number of people enrolled in the RPG program every six months will be obtained in the semiannual progress reports. If the number does not match the number of new entries to the enrollment and services data, the cross-site evaluation team will contact the grantee to reconcile the numbers and request they add any missing enrollees to RPG-EDS.
Conduct regular data completion and quality checks. The cross-site evaluation contractor will examine each grantee’s enrollment and services data at regular intervals to identify any potential problems. If problems are identified, contractor staff will notify the grantee and work with the grantee and providers as needed to obtain missing data or remedy other potential problems quickly.
Design the outcomes and impacts instruments in a manner that reduces burden. The outcomes data that grantees must report comprise standardized instruments that often ask for similar information, such as demographic information about the respondent. To avoid such duplication, the outcomes instruments exclude redundant items to prevent duplication. This reduces burden on grantee staff responsible for uploading these data to RPG-EDS.
Develop a user-friendly, flexible upload process that has already proven successful. RPG-EDS, to which grantees will upload data, will provide easy access while maintaining the security of outcomes data. The system, designed with access by grantee staff in mind, is based on successful experiences in prior studies collecting similar types of data from similar types of service providers. The component of the system for managing outcomes data, to which grantees will upload data from the outcomes instruments, is modeled on the data reporting system that was used in RPG projects from 2012 through present. Compared with the former RPG systems, RPG-EDS includes updated features and improved technology to simplify the upload process.
Provide training and technical assistance to grantee staff. We will provide to grantees documentation on, training in, and technical assistance in collecting data from participants, uploading outcomes data to RPG-EDS, and using the web-based enrollment and services data entry application in RPG-EDS.
Include data quality checks in the data system. RPG-EDS will also improve data reliability with automatic data quality checks. For example, if grantee staff enter out-of-range values in a particular field, the system will prompt users to check the value. For some fields, response values will be restricted; for others, grantee site staff will be able to override the check. We will also monitor the data entered by grantee sites and provide feedback to grantees on their data quality.
Optimize the frequency of data collection. Grantees will upload outcomes data once every six months, rather than waiting until their evaluation data collection is complete. This will enable the cross-site evaluation team to regularly identify and troubleshoot problems grantees experience in collecting data from respondents or uploading data.
Most of the instruments to be used in the RPG cross-site evaluation build on existing measures and experience from other studies completed by the cross-site evaluation team.
Grantee and partner staff topic guide. The interview topic guide for RPG grantee and partner staff has been modeled after interview guides used in similar studies such as the Early Head Start Enhanced Home Visiting Pilot Evaluation and the Evidence-Based Home Visiting and previous RPG cross-site evaluations. All site visitors will receive training on the topic guides. After the first site visit, the cross-site evaluation team will meet to discuss the instruments and whether they require any modifications to enhance data quality and reduce burden, such as eliminating or refining any questions that were unnecessary or redundant.
Partner surveys. The partner survey will be identical to the one developed and pre-tested for RPG2 and RPG3. No pretest of the partner survey will occur prior to fielding the instrument. A pretest of the instrument occurred under RPG2 and was covered under OMB #4170-0444.
Before deploying the partner survey on the web, Mathematica staff will rigorously compare the web surveys to the hard-copy instrument for accuracy. Web testers will check question wording and response option formats, and will use different scenarios to test the skip patterns.
RPG-EDS. The component of RPG-EDS for managing outcomes data is modeled after the data systems used with previous cohorts of RPG projects. The development team will rigorously test and evaluate the functionality of RPG-EDS before it goes live. Additionally, prior to OMB clearance we will consult with no more than nine respondents from the grantee projects on the usability of the system and in the testing phase.
We received preliminary input on statistical methods from Mathematica staff, including:
Dr. Sarah Avellar
Mathematica Policy Research
1100 1st Street, NE, 12th Floor
Washington, DC 20002
Dr. Russell Cole
Mathematica Policy Research
P.O. Box 2393
Princeton, NJ 08543
Dr. Yange Xue
Mathematica Policy Research
P.O. Box 2393
Princeton, NJ 08543
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | RPG OMB Supporting statement B |
| Subject | OMB |
| Author | VARIOUS |
| File Modified | 0000-00-00 |
| File Created | 2021-01-15 |