Regional Partnerships Grants (RPG) National Cross- Site Evaluation and
Evaluation Technical Assistance
Supporting Statement, Part B
For OMB Approval
August 2016
In response to federal legislation (Child and Family Services Improvement and Innovation Act of 2011; Pub. L. 112-34), under the Regional Partnership Grant (RPG) program, the Children’s Bureau (CB) within the Administration for Children and Families issued 21 grants to develop interagency collaborations and provide services designed to increase well-being, improve permanency, and enhance the safety of children who are in an out-of-home placement or are at risk of being placed in out-of-home care as a result of a parent's or caretaker's substance use disorder (17 RPG2 grantees funded in 2012 and 4 RPG3 grantees funded in 2014). The overall objective of the RPG Cross-Site Evaluation is to describe and document the performance of the RPG grantees, the outcomes for families enrolled in RPG, and the effectiveness of approaches taken by the grantees, as stated in the legislation. To meet these evaluation goals, the RPG Cross-Site Evaluation includes three study components: (1) an implementation and partnership study; (2) an outcomes study; and (3) an impact study. The implementation and partnership study will collect data on RPG programs—with a special focus on 10 selected evidence-based programs—and on the collaborative partnerships. The outcomes study will describe the characteristics of and changes over time in children, adults, and families who participate in the RPG programs. The impact study will include a subset of five RPG grantees that are implementing rigorous comparison local evaluation designs and can provide outcome data on both treatment and comparison group members.
This ICR requests a renewal of clearance for the OMB package #0970-0444 which was originally approved in March 2014. Four RPG3 grantees were awarded grants from CB in September 2014 and were added to the existing OMB package through a non-substantive change request approved by OMB in June 2015. This ICR focuses on the remaining data collection across all 21 RPG projects. A subset of some of the data collection activities were completed with RPG2 grantees during the first OMB clearance period and were removed from this ICR. Thus, some of the remaining data collection involves only the four RPG3 projects, while other data collection activities include all 21 grantees. However, all data collection activities described below began as a part of the previous clearance period and grantees are familiar with the data systems and data collection processes associated with the cross-site evaluation.
All 21 RPG grantees will participate in the implementation and partnership study. In this ICR, clearance is sought for five instruments associated with this study. Before describing the instruments and their respondents, we describe the sample of RPG program services on which some of these instruments will focus.
Focal EBPs. To meet the needs of adult and child family members and ensure the comprehensive assistance needed to support adult recovery from substance use disorder, ensure child safety, and stabilize families, each grantee will provide multiple services to participants. In its grant announcement, CB emphasized the importance of basing these services on evidence-based or evidence-informed program and practice models (EBPs). Across all grantees combined, 51 different EBPs will be used, with some grantees implementing a dozen or more. To reduce the burden and complexity of data collection that would be necessary if detailed service delivery data were collected on all 51 EBPs, the RPG Cross-Site Evaluation will gather service delivery data on a subset of 10 EBPs (referred to as “focal” EBPs).
CB selected these focal EBPs based on four criteria: (1) the EBP is being implemented as a primary service of at least two RPG grantees; (2) the EBP is a session-based program for which service use data can be collected; (3) each of the 21 grantees is implementing at least one of the focal EBPs; and (4) the set of focal EBPs includes at least one model from each of the five types of interventions RPG grantees are implementing (child-caregiver therapy, counseling, family strengthening, response to trauma, and substance use disorder treatment). On average, each RPG grantee is implementing and will report on 3 of the 10 focal EBPs.
Clearance is requested for the following five instruments associated with this component of the evaluation:
Grantee and partner staff topic guide. During site visits to each of the 4 RPG3 grantees, semi-structured interviews will be conducted with the RPG grantee director. Small group interviews will be conducted with managers and supervisors for each of the focal EBPs, some of which may be implemented by the grantee and others by partner organizations. Individual interviews will be conducted with two frontline staff and two supervisors or managers of focal EBPs. If an EPB is delivered by more than two staff of each type, we will randomly select two to participate in the individual interviews. Note: Site visits with RPG2 grantees were completed under the prior clearance period.
Semi-annual progress reports. All RPG grantee project directors or their designees will submit written semi-annual progress reports, each describing implementation and enrollment progress, program changes, and adherence to focal EBP models during the prior six months.
Enrollment and service log. All RPG grantees will record program enrollment and exit dates, characteristics, and service enrollment and exit dates for all RPG participants in a web-based log. Frontline staff or their designees will document actual service delivery (such as length and content of sessions) for all RPG participants who are enrolled in any of the 10 focal EBPs.
Staff survey. All frontline staff who provide direct services to children, adults, and families for the 10 focal EBPs will be asked to complete a web-based survey. “Frontline staff” are those who work directly with any member(s) of the RPG case, and may include therapists, social workers, home visitors, or similar staff. We estimate that an average of 20 staff members will fit this criterion for each of the 4 RPG3 grantees. Note: Staff surveys with RPG2 grantees were completed under the prior clearance period.
Partner survey. Lead staff of the RPG grantee and RPG partner organizations will be asked to complete a web-based survey. Partner organizations are defined as organizations other than the grantee that provide RPG services to families enrolled in the program and/or coordinate their services with the grantee. We estimate that up to 20 individuals will fit this criterion for each of the 4 RPG3 grantees. Note: Partner surveys with RPG2 grantees were completed under the prior clearance period.
The outcomes study will include all 21 grantees. In this ICR, renewal of clearance is sought for one instrument associated with the outcomes study component of the cross-site evaluation:
Outcomes study master instrument. This includes standardized instruments CB has selected for use by the grantees to measure selected outcomes, and specifications for administrative data elements grantees will obtain for additional outcome measures. Outcome data on child well-being, safety, and permanency will be collected on one focal child in each participating family, from the child’s primary caregiver. Each grantee will select a focal child at enrollment based on their target populations and planned services (for example, some grantees plan to serve families with infants or toddlers, while others plan to serve adolescents or teens.) Grantees will administer instruments that collect data on family functioning/stability to the focal child’s primary caregiver, if the child has not been removed from the home at RPG enrollment. If the child has been removed from the home, the instruments will be administered to the adult in the child’s original home who served as the child’s primary caregiver prior to removal. Instruments that collect data on adult recovery from substance use dependence will also be administered to that adult, unless she or he is not receiving RPG services but a separate adult is receiving RPG services; in that situation, recovery data will be collected for that separate adult who is receiving RPG services.
The impact study will include five grantees that are implementing rigorous comparison local evaluation designs and can provide outcome data on both treatment and comparison group members. In this ICR, clearance is sought to collect a subset of elements contained in the outcome master study instrument from families in the comparison groups.
Impact study master instrument. This includes 4 of the 10 standardized instruments from the outcomes study master outcome instrument and specifications for all of the administrative data elements contained in the outcomes study master outcome instrument. These data will be collected on individuals in the comparison groups of the five grantees that will participate in the impact study.
In this ICR, renewal of clearance is sought for the five instruments associated with the implementation and partnership study. The data collection procedures are described below:
Grantee and partner staff interview topic guide (Instrument #1). Two members of the RPG cross-site evaluation team will conduct one site visit to each of the four RPG3 grantees. While on-site, they will conduct in-person interviews with grantee and partner staff. The interviews will be with individuals or small groups, depending on staff role. Evaluators will obtain written consent from each interviewee, including permission to audio record the interviews for later transcription. One team member will moderate the interview. If interviewees do not consent to audio taping, the second team member will use a laptop computer to take detailed notes.
Semi-annual progress reports (Instrument #2). CB will provide a template for the semi-annual progress reports, which will be submitted every six months. Narrative information can be entered directly into the template, or grantees can respond to the questions in other electronic file formats of their choosing. Grantee project directors will submit their reports to www.GrantSolutions.com.
Enrollment and service logs (Instrument #3). Intake workers will enter demographic characteristics, RPG enrollment and exit dates, and EBP enrollment and exit dates for each RPG case into this web-based log. Staff working on focal EBPs will enter individual service contact information on a weekly basis for the duration of participation in each specific focal EBP.
Staff survey (Instrument #4). This survey is web-based. The cross-site evaluation will obtain contact information for desired respondents from the four RPG3 grantees and send advance notification of the survey with an email. Personalized links to the survey (along with both an email address and telephone number where any questions about the survey can be addressed) will then be distributed to each respondent via email. By clicking on the link or pasting it into their browser, they will be taken to the 30-minute survey. If they are unable to complete the survey in one sitting, they can return to the link as needed.
Partner survey (Instrument #5). This survey is web-based. The cross-site evaluation will obtain contact information for desired respondents from the four RPG3 grantees and send advance notification of the survey with an email. Personalized links to the survey (along with both an email address and telephone number where any questions about the survey can be addressed) will then be distributed to each respondent via email. By clicking on the link or pasting it into their browser, they will be taken to the 30-minute survey. If they are unable to complete the survey in one sitting, they can return to the link as needed.
Information for the implementation and partnership study will be descriptive. In general, it will not involve formal hypothesis testing.
Outcomes study master instrument (Instrument #6). Each grantee is expected to maintain outcome data from the case-specific standardized instruments and administrative records for all RPG participants in their project or agency database(s). Grantees will upload these data to the RPG Data Portal every six months, using file formats specified or provided by the cross-site evaluation. To maximize data quality, automatic data validation checks will occur during the upload and error messages will indicate any corrections needed before the submission can be accepted.
Information for the outcomes study will be descriptive. In general, it will not involve formal hypothesis testing.
Impact study master instrument (Instrument #7). Each of the five grantees participating in the impact study is expected to maintain the case-specific outcome data for comparison group members from standardized instruments and administrative records in their project or agency database(s). They will upload these data to a data portal developed and maintained by the cross-site evaluation every six months using file formats specified or provided by the cross-site evaluation. To maximize data quality, automatic data validation checks will occur during the upload and error messages will indicate any corrections needed before the submission can be accepted.
Based on prior experience with similar grant projects and data collection efforts, the RPG Cross-Site Evaluation expects to obtain a high response rate of 80 percent or more for all instruments used. The grantee liaison will serve as a link to work with RPG grantees, if needed, to address nonresponse. Strategies for maximizing response in the data collection efforts are described below.
Conduct interviews with key grantee staff, administrators, supervisors, and staff during site visits. All interviews conducted with key grantee administrators, supervisors, and staff will occur during site visits to RPG3 grantees. It is anticipated that all grantees will agree to participate in these visits. Our experience conducting site visits to the RPG2 grantees, which were completed during the prior clearance period, was that all grantees agreed to participate and participation rates of the desired interviewees were close to 100 percent. To help ensure high participation, we will coordinate with the RPG3 grantees, supervisors, and staff to determine convenient dates and schedules for these visits.
Staff and partner surveys are designed in a manner that minimizes respondent burden. To minimize burden on respondents, the surveys are brief, web-based, and structured such that respondents do not have to pay attention to routing and skip logic or view questions that do not apply to them. These will be administered to the RPG3 grantees only as the RPG2 grantee staff and partner surveys were completed under the prior clearance period. Response rates for RPG2 grantees were close to 80 percent on both the staff and partner surveys.
Send advance and reminder emails to respondents. We will send advance emails to RPG3 grantee and partner staff requesting their participation. If respondents have not completed the survey within a certain amount of time, we will send reminder emails requesting them to complete the surveys.
Solicit the help of grantees to encourage completion of the staff and partner surveys. If response rates for individual RPG3 grantees lag, the cross-site evaluation team will work with lead grantee staff to identify additional strategies for increasing receipt of completed surveys without compromising respondent confidentiality. For instance, lead grantee staff may be asked to send an email to all the survey participants they had identified in their site, encouraging everyone’s response. This should assist with response rates, since lead grantee staff will have personal relationships with their staff and partners and can use their close proximity to encourage responses. This approach of combining follow-up requests from the evaluator to individuals who have not completed the survey with general requests from the grantee to all desired respondents has proved effective in multiple Mathematica projects that involved collecting similar data through web-based surveys, including in prior surveys performed for the CB as well as the staff and partner surveys conducted for RPG2 grantees under the prior clearance period.
Conduct telephone follow-up with non-respondents on the staff and partner survey. If email reminders and requests from the grantee prove ineffective, the cross-site evaluation team will deploy survey staff with expertise in obtaining responses to conduct one round of telephone follow-up with non-respondents from RPG3 grantee sites. This approach of following up via telephone when email requests have not been effective has increased response rates in multiple Mathematica projects that involved collecting similar data through web-based surveys, including in prior surveys performed for the CB as well as the staff and partner surveys conducted for RPG2 grantees under the prior clearance period.
Provide an easy-to-use enrollment and services log. The design of the enrollment and service log component of the portal is based on web-based case management systems that Mathematica has developed and successfully implemented for multiple projects that collect these data from similar types of providers. The log can be accessed from any computer, allowing for ease of entry, while the data are housed on secure servers behind the cross-site evaluation contractor’s firewall, thereby maintaining data security. All 21 grantees have used the enrollment and service log system during the previous OMB clearance period.
Use multiple sources to check enrollment activity and completion of the service log. Information on the number of people enrolled in the RPG program every six months will be obtained in the semi-annual progress reports, and if the number does not reconcile with the number of new entries to the enrollment and service log, the cross-site evaluation will contact the grantee to reconcile the numbers and request that they add any missing enrollees to the log.
Use automatic prompts to enter service data. Once an RPG participant is enrolled in a focal EBP, the system will automatically prompt provider staff responsible for entering service data each week. If no services were provided during the week, provider staff members must so indicate.
Conduct regular data completion and quality checks. The cross-site evaluation contractor will examine each grantee’s enrollment and service log data at monthly intervals to identify any potential problems. If problems appear, contractor staff will notify the grantee and work with the grantee and providers as needed to obtain missing data or remedy other potential problems on a timely basis.
Design the master outcome and impact instruments in a manner that reduces burden. The outcome data that grantees are required to report comprises standardized instruments that often ask for similar information, such as demographic information about the respondent. To avoid such duplication, the master outcome instrument excludes redundant items to ensure there is no duplication of items. This reduces burden on grantee staff responsible for uploading this data to the RPG portal.
Develop a user-friendly, flexible upload process that has already been used successfully. The RPG Data Portal, which grantees use to upload data, provides easy access while maintaining the security of outcome data. The data portal has been designed with access by grantee staff in mind and is based on successful experience during prior studies collecting similar types of data from similar types of service providers. The outcome data management portion of the system, where grantees will upload data from the outcome instrument, is modeled on the data reporting system that was used for the first round of RPG grants from 2007 through 2011 (RPG1). The current outcome data management system uses updated features and improved technology to simplify the upload process compared to the former RPG1 system. All 21 grantees have been trained on the system and have used the data management system during the previous OMB clearance period.
Provide training and technical assistance to grantee staff. We have provided documentation, training and technical assistance to grantees in collecting data from participants, uploading data to the RPG data portal, and using the web-based enrollment and service log.
Include data quality checks in the data portal. The data portal also ensures data reliability by instituting automatic data quality checks. For example, if grantee staff enter out-of-range values in a particular field, the system will prompt users to check the value. For some fields, response values are restricted; for others, grantee site staff are able to override the check. We will also monitor the data entered by grantee sites and provide feedback to grantees on their data quality.
Optimizing the frequency of data collection. Our data collection strategy is designed to ensure the collection of accurate data. Grantees upload outcome data once every six months, rather than waiting until their evaluation data collection is complete. This enables the cross-site evaluation to identify and troubleshoot problems grantees experience in collecting data from respondents or uploading data.
Most of the instruments to be used in the RPG Cross-Site Evaluation build on existing measures and previous experience from other studies completed by the cross-site evaluation team.
Grantee and partner staff interview topic guide. The RPG grantee and partner staff interview topic guide has been modeled after interview guides used in similar studies such as the Early Head Start Enhanced Home Visiting Pilot Evaluation and the Evidence-Based Home Visiting (EBHV) cross-site evaluation. All site visitors will receive training in use of the topic guides. Site visits to RPG2 grantees have been successfully completed and site visitors refined the site visit process and interview questions. RPG3 site visitors will implement any efficiencies learned as a result of conducting the site visits to RPG2 grantees conducted under the previous clearance period.
Staff and partner surveys. The staff survey adapted items used previously in other Mathematica projects including the EBHV cross-site evaluation, the Child Support Noncustodial Parent Employment Demonstration (CSPED), and Parents and Children Together (PACT). It also included several standardized scales. The partner survey was modeled after the ones used on the EBHV cross-site evaluation, the Integration Initiative cross-site survey, and the survey instrument (Collaborative Capacity Inventory) used in RPG1.
Pretests of the surveys were conducted in September and October 2013, and were discussed as part of the submission of the original OMB package (more information can be found in the original package). The adjustments made as a part of the pretests were implemented when the staff and partner surveys were fielded with the 17 RPG2 grantees. Grantees were able to access the web-based surveys with ease and all aspects of the surveys, including response options, question wording, and skip patterns functioned as expected. No changes will be made before the surveys are deployed with the four RPG3 grantees.
RPG data portal. The outcome data management component of the RPG data portal is modeled after RPG1 and the service log component is modeled after service logs that we have built for other projects such as CSPED and PACT. All functions of the RPG data portal have been rigorously tested and evaluated by the development team to ensure proper functionality. Additionally, the system is now in use and all 21 grantees have used it during the previous clearance period.
Preliminary input on statistical methods was received from staff at Mathematica and WRMA, Inc., and experts at other organizations, including:
Dr. Sarah Avellar
Mathematica Policy Research
1100 1st Street, NE, 12th Floor
Washington, DC 20002
Dr. Russell Cole
Mathematica Policy Research
P.O. Box 2393
Princeton, NJ 08543
Dr. John Deke
Mathematica Policy Research
P.O. Box 2393
Princeton, NJ 08543
Dr. Allison Metz
University of North Carolina
Sheryl-Mar South, Room 142
Campus Box 8185
Chapel Hill, NC 27599-8185
Dr. Joseph Ryan
University of Michigan
Center for Political Studies
1080 South University Avenue
Ann Arbor, MI 48109-1106
Dr. Cheryl Smithgall
Chapin Hall at the University of Chicago
1313 East 60th Street
Chicago, IL 60637
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | RPG cross site evaluation supporting statement part B |
| Author | Autumn Parker |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |