REGIONAL PARTNERSHIP GRANTS (RPG) NATIONAL CROSS-SITE EVALUATION AND EVALUATION TECHNICAL ASSISTANCE
OMB Information Collection Request
0970 - 0527
Supporting Statement Part B –
Statistical Methods
April 2025
Type of Request: Revision
Submitted By:
Children’s Bureau
Administration for Children and Families
U.S. Department of Health and Human Services
1. Respondent Universe and Sampling Methods
The overall objective of the Regional Partnership Grants (RPG) cross-site evaluation is to describe and document the performance of the RPG projects, the outcomes for participants enrolled in RPG, and the effectiveness of the grantees’ approaches, as stated in the legislation. To meet these evaluation goals, the RPG cross-site evaluation includes six study components:
(1) A partnerships analysis to assess the coordination and collaboration of partners’ service systems;
(2) A participant experiences analysis to assess how participants enrolled in RPG services became involved in RPG services and their perspectives on RPG program;
(3) A sustainability analysis to assess projects’ use of data for continuous improvement and their activities to sustain RPG programs once their grants end
(4) An enrollment and services analysis to assess data on RPG participant characteristics and the types of, dosage of, and engagement with services
(5) An outcomes analysis, to describe the characteristics of and changes over time in children, adults, and families who participate in the RPG programs.
(6) An impacts analysis with a subset of 14 of the 18 RPG7 grantees that are implementing rigorous local evaluation designs and may provide outcomes data on both treatment and comparison group members.
The partnerships analysis includes the following instrument and respondents:
Grantee and partner staff topic guide (Appendix B). During site visits to the 18 RPG7 grantees, semi-structured individual interviews will be conducted with the RPG project director, two managers or supervisors for the RPG project, and two frontline staff who are providing services to participants. Individual interviews will take place with three grantee partners who may represent the child welfare agency or substance use treatment provider. We will work with the project director of each RPG7 program to identify program staff in these roles and are knowledgeable about the program operations and partnerships.
Participant experiences analysis
The participant experiences analysis includes the following instruments and respondents:
Individual interviews with participants enrolled in RPG services (Appendix C). During site visits to grantee organizations, individual interviews will be conducted with 16 participants enrolled in RPG services. During the scheduling of the site visits, we will identify up to 4 grantees to work with who will collaborate with us to identify adults enrolled in the grantee’s RPG program for the cross-site evaluation team to interview.
Focus groups with participants enrolled in RPG services (Appendix D). During site visits to grantee organizations, 8 focus groups will be conducted with up to 48 adult participants (6 respondents per group) enrolled in RPG services. The grantee will provide information to the cross-site evaluation team about potential individuals who might be a good fit for the focus groups. We will identify up to 4 grantee oroganizations at which to conduct the focus groups with participants, and will work closely with the grantee staff to recruit potential respondents.
Sustainability analysis
The sustainability analysis includes the following instruments and respondents:
Sustainability survey (Appendix E). Lead staff of grantee and partner organizations with knowledge about sustainability planning will be asked to complete a web-based survey on this topic. We expect to administer the web-based survey once to 126 grantee key staff and partners (seven per project across 18 projects). We will work closely with the grant receipient project director to identify the respondents from partner organizations.
Semi-annual progress reports. The cross-site evaluation will also use the semi-annual progress reports as part of this analysis, which include grantees’ descriptions of sustainability efforts. Al 18 grantees complete the semi-annaul progress reports under a separate OMB clearance (OMB No. 0970-0490, Expiration 3/31/2026).
This component of the evaluation includes the following instruments and respondents:
Enrollment and services data (Appendix F). These data describe participants’ characteristics at enrollment and the services they receive. Grantees record the enrollment date for each RPG family or household and demographic information on each family member including date of birth, ethnicity, race, primary language spoken at home, type of current residence (children only), income (adults only), highest education level attained (adults only), and relationship to a focal child in each family on whom data will be collected. Grantees also record the enrollment date for families or individual family members into RPG services, service contact information for core services, and exit dates for RPG.
Semi-annual progress reports. The cross-site evaluation will also use the semi-annual progress reports as part of this analysis which includes grantees’ reports of enrollment progress into the program and evaluation across the life of the grant. All 18 grantees complete the semi-annaul progress reports under a separate OMB clearance (OMB No. 0970-0490, Expiration 3/31/2026).
The ongoing outcomes analysis includes all 18 grantees.
This component of the evaluation includes five measures and administrative data elements associated with the cross-site evaluation’s outcomes analysis (Appendices G and H). From the child’s primary caregiver, grantees collect outcomes data on child well-being, functioning and stability, safety, and permanency on one focal child in each participating family. However, if the child is in out-of-home placement, then grantees collect child well-being data from the current caregiver. Each grantee selects a focal child at enrollment based on their target populations and planned services (for example, some grantees plan to serve families with infants or toddlers, and others plan to serve adolescents or teens.) Grantees administer the instruments collecting data on adult substance use disorder (SUD) recovery from the same adult, unless he or she is not the adult with an SUD or in recovery from an SUD. In those cases, grantees collect recovery data from a separate adult receiving RPG services who has or had an SUD.
The impacts analysis includes 14 grantees that are implementing rigorous comparison local evaluation designs and may provide outcomes data on both treatment and comparison group members. This component of the evaluation includes the same elements contained in the outcomes analysis instruments from participants in the comparison groups (Appendices G and H).
2. Procedures for the Collection of Information
Grantee and partner staff topic guide. Two members of the RPG cross-site evaluation team will conduct one site visit to 18 grantees (in Year 4 of the grant program). While on-site, they will conduct in-person, individual interviews with grantee, partner, and frontline staff. Evaluators will obtain verbal consent from each interviewee, including permission to audio record the interviews for later transcription. One team member will moderate the interview. If interviewees do not consent to audio recording, the second team member will use a laptop computer to take detailed notes.
Information for the partnerships analysis will be descriptive. In general, it will not involve formal hypothesis testing.
Participant experiences analysis
As a part of the site visits conducted for the partnerships analysis, two additional members of the RPG cross-site team will join the site visit team for up to 8 grantees (in Year 4 of the grant program). Descriptions of the data collection procedures for the two instrument associated with the participant experiences analysis follow:
Individual interviews with participants enrolled in RPG services. One cross-site team member will conducts in-person, individual interviews with participants enrolled in RPG services. Interviewers will obtain verbal consent from each interview respondent, including permission to audio record the interview for later transcription. If respondents do not consent to audio recording, the interviewer will use a laptop computer to take detailed notes.
Focus groups with participants enrolled in RPG services. Two members of the RPG cross-site team conduct focus groups with up to 6 participants enrolled in RPG services. As part of the focus group, we will also administer a brief questionnaire to obtain participant demographic data. Interviewers will obtain verbal consent from each focus group respondent, including permission to audio record the focus group for later transcription. One team member will moderate the focus group. If respondents do not consent to audio recording, the second team member will use a laptop computer to take detailed notes.
Information for the participant experiences analysis will be descriptive. In general, it will not involve formal hypothesis testing.
Sustainability analysis
Sustainability survey. This survey is web-based. The cross-site evaluation team will obtain from the grantees contact information for desired respondents and notify respondents in advance about the survey with an email. Personalized links to the survey (along with both an email address and telephone number where respondents can ask any questions about the survey) will then be distributed to each respondent via email. By clicking on the link or pasting it into their browser, respondents will go directly to the 20-minute survey. If they are unable to complete the survey in one sitting, they can return to it as needed.
Information for the sustainability analysis will be descriptive. In general, it does not involve formal hypothesis testing.
Enrollment and services data. Intake workers enter demographic characteristics and RPG program enrollment and exit dates for each RPG case into the case management system known as the RPG-Evaluation Data System (RPG-EDS). Staff delivering services enter individual service contact information on a rolling basis for the duration of participation in RPG services.
Information for the enrollment and services analysis will be descriptive. In general, it does not involve formal hypothesis testing.
Outcomes analysis data. Each grantee is expected to maintain outcomes data from the case-specific standardized instruments and administrative records for all RPG participants in its project or agency database(s). Grantees upload these data to the RPG-EDS every six months, using file formats specified or provided by the cross-site evaluation. To maximize data quality, automatic data validation checks occur during the upload, and error messages will indicate any corrections needed before the submission can be accepted.
Information for the outcomes analysis will be descriptive. In general, it does not involve formal hypothesis testing.
Impacts analysis
Impacts analysis data. Each of the 14 grantees participating in the impacts study is expected to maintain the case-specific outcomes data for comparison group members from standardized instruments and administrative records in its project or agency database(s). Grantees upload these data to RPG-EDS every six months using file formats specified or provided by the cross-site evaluation. To maximize data quality, automatic data validation checks occur during the upload, and error messages will indicate any corrections needed before the submission can be accepted.
3. Methods to Maximize Response Rates and Deal with Nonresponse
Based on our experience with the second through sixth RPG cohorts, the cross-site evaluation expects to obtain a high response rate of 80 percent or more for surveys and close to total participation in other data collection activities, such as site visits, interviews and focus groups of participants, and enrollment, services, and outcomes data submissions. A grantee liaison is assigned to each site and serve as a link to work with grantees, if needed, to address nonresponse. Descriptions of the strategies for maximizing response in the data collection efforts follow.
Conduct interviews with key grantee staff, supervisors/managers, partners, and frontline staff during site visits. All interviews with key grantee staff, supervisors/managers, partners, and staff will occur during site visits. We anticipate that all grantees will agree to participate in these visits. Our experience with previous RPG cohorts indicates that participation rates of the desired interviewees are typically close to 100 percent. To help ensure high participation, we coordinate with the grantees, supervisors/managers, partners, and staff to determine convenient dates and schedules for these visits.
Participant experiences data
Conduct individual interviews and focus groups with participants enrolled in RPG services during site visits. Individual interviews and focus groups with participants enrolled in RPG services will occur during site visits. Given experiences on a pretest of these data collection activites on the prior sixth RPG cohort, we expect that grantee staff will help identify individuals for the interviews and focus groups, and the cross-site evaluation team will sign participants up and send reminders to individuals about the data collection activities (Appendix I has recruitment materials). We will schedule the individual interviews and focus groups at times convenient for the participants and in a central location that is accessible to the participants, such as the location where they received services. Based on the pretest experience, we expect at least 80 percent of respondents recruited to participate in an interview or focus goup to attend.
Design sustainability survey in a manner that minimizes respondent burden. To minimize burden on respondents, the survey is brief, web-based, and structured such that respondents do not have to pay attention to routing and skip logic or view questions that do not apply to them.
Send advance and reminder emails to respondents (Appendix J and K). We send advance emails and an FAQ document to grantee and partner sample members requesting their participation. If respondents have not completed the survey within a certain amount of time, we send reminder emails requesting them to complete the surveys.
Solicit the help of grantees to encourage completion of the sustainability surveys. If response rates for individual grantees lag, the cross-site evaluation team works with lead grantee staff to identify additional strategies for increasing completed surveys without compromising respondent confidentiality. For instance, lead grantee staff may be asked to send an email to all the survey participants they had identified in their site, encouraging everyone’s response. In past rounds of staff and partner surveys, this approach helped boost response rates, because lead grantee staff had personal relationships with their partners. This approach of combining follow-up requests from the evaluator to people who have not completed the survey with general requests from the grantee to all desired respondents has proven effective with previous RPG cohorts.
Conduct telephone follow-up with nonrespondents on the sustainbility survey. If email reminders and requests from the grantee prove ineffective, the cross-site evaluation team deploys survey staff with expertise in obtaining responses to conduct one round of telephone follow-up with nonrespondents. This approach of following up via telephone when email requests has increased response rates with previous RPG cohorts.
Provide an easy-to-use data entry system for enrollment and services data. The design of the enrollment and service data application of RPG-EDS is based on web-based case management systems that Mathematica has developed and successfully implemented for multiple projects that collect these data from similar types of providers. RPG-EDS can be accessed from any computer, allowing for ease of entry, while the data are encrypted in transit and at rest and reside behind firewalls, thereby maintaining data security.
Use multiple sources to check enrollment activity and completion of the enrollment and services data entry. Information on the number of people enrolled in the RPG program every six months will be obtained in the semiannual progress reports. If the number does not match the number of new entries to the enrollment and services data, the cross-site evaluation team contacts the grantee to reconcile the numbers and request they add any missing enrollees to RPG-EDS.
Conduct regular data completion and quality checks. The cross-site evaluation contractor examines each grantee’s enrollment and services data at regular intervals to identify any potential problems. If problems are identified, contractor staff notifies the grantee and works with the grantee and providers as needed to obtain missing data or remedy other potential problems quickly.
Design the outcomes and impacts instruments in a manner that reduces burden. The outcomes data that grantees must report comprise standardized instruments that often ask for similar information, such as demographic information about the respondent. To avoid such duplication, the outcomes instruments exclude redundant items to prevent duplication. This reduces burden on grantee staff responsible for uploading these data to RPG-EDS.
Develop a user-friendly, flexible upload process that has already proven successful. RPG-EDS, to which grantees upload data, provides easy access while maintaining the security of outcomes data. The system, designed with access by grantee staff in mind, is based on successful experiences in prior studies collecting similar types of data from similar types of service providers. The component of the system for managing outcomes data, to which grantees upload data from the outcomes instruments, is modeled on the data reporting system that was used in RPG projects from 2012 through 2017. Compared with the former RPG systems, RPG-EDS includes updated features and improved technology to simplify the upload process.
Provide training and technical assistance to grantee staff. We provide to grantees documentation on, training in, and technical assistance in collecting data from participants, uploading outcomes data to RPG-EDS, and using the web-based enrollment and services data entry application in RPG-EDS.
Include data quality checks in the data system. RPG-EDS also improves data reliability with automatic data quality checks. For example, if grantee staff enter out-of-range values in a particular field, the system prompts users to check the value. For some fields, response values are restricted; for others, grantee site staff can override the check. We also monitor the data entered by grantee sites and provide feedback to grantees on their data quality.
Optimize the frequency of data collection. Grantees upload outcomes data once every six months, rather than waiting until their evaluation data collection is complete. This enables the cross-site evaluation team to regularly identify and troubleshoot problems grantees experience in collecting data from respondents or uploading data.
4. Test of Procedures or Methods to be Undertaken
Most of the instruments used in the RPG cross-site evaluation build on existing measures and experience from other studies completed by the cross-site evaluation team.
Grantee and partner staff topic guide. The interview topic guide for RPG grantee and partner staff has been modeled after interview guides used in similar studies such as the Early Head Start Enhanced Home Visiting Pilot Evaluation and the Evidence-Based Home Visiting (EBHV) and previous RPG cross-site evaluations. All site visitors receive training on the topic guides. After the first site visit, the cross-site evaluation team meets to discuss the instruments and whether they require any modifications to enhance data quality and reduce burden, such as eliminating or refining any questions that were unnecessary or redundant.
Individual interview and focus groups with participants enrolled in RPG services protocols. In winter/spring of 2023, Mathematica conducted a pretest of the individual interviews and focus groups with participants enrolled in RPG services at two grantee sites in the RPG6 cohort. Staff at the sites helped recruit eligible participants for the pretest, which also allowed us to pretest the recruitment materials (Appendix I) and processes to preparing sites for recruitment. Eight parents participated in the individual interviews, which took about two hours and eight parents participated in the focus groups, which lasted up to 90 minutes. At the end of each interview and focus group, participants answered a few questions about the recruitment and consent process, the methods and instruments used, and their ability to participate in the data collection activity. This part of data collection took up to 15 minutes to complete.
Overall, participants responded positively to the individual interviews and the use of the paper lifeline used during the interview to document visually each participant’s experiences. Several participants said they enjoyed the opportunity to share their life story and most said they wanted to participate to help other people. A few participants remarked that the interview was very in-depth, but they viewed the interview’s thoroughness favorably. Even though the interview raised sensitive topics and experiences, participants said the interviewers put them at ease throughout the process. Based on the pretest, we reordered some of the interview questions, reduced the number of probes, added questions to understand participants’ strengths and positive experiences, and added questions about physical and mental health.
Participants reacted positively to the focus group format. A few participants said they were initially hesitant about the group format, but felt more comfortable once the conversation was underway. They thought the questions were reasonable and not too personal. Based on participants’ feedback, we did not revise the focus group questions. However, we added a questionnaire to administer during focus groups to systematically document participants’ characteristics to help with reporting our findings.
Sustainability survey. The sustainability survey includes adapted items from the partner and staff surveys that were used in the RPG2 and RPG3 projects. The staff survey had adapted items used previously in other Mathematica projects including the Evidence-Based Home Visiting (EBHV) cross-site evaluation, the Child Support Noncustodial Parent Employment Demonstration (CSPED), and Parents and Children Together (PACT). It also included several standardized scales. The partner survey was modeled after the ones used on the EBHV cross-site evaluation, the Integration Initiative cross-site survey, and the survey instrument (Collaborative Capacity Inventory) used in RPG1. In addition to the previous staff and partner surveys, the sustainability survey also adapted items from the RPG-EDS service logs and semi-annual progress reports.
In the summer of 2019, Mathematica conducted a pretest of the survey, as describe under previous OMB packages. Mathematica fielded the survey with the RPG6 grantees in 2023.
RPG-EDS. The component of RPG-EDS for managing outcomes data is modeled after the data systems used with previous cohorts of RPG projects. The development team has rigorously tested and evaluated the functionality of RPG-EDS.
B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
We received preliminary input on statistical methods from Mathematica staff, including:
Dr. Angela D’Angelo
Mathematica
111 East Wacker Drive, Suite 3000
Chicago,
IL 60601
Dr. Yange Xue
Mathematica
P.O. Box 2393
Princeton, NJ 08543
Dr. Sarah Avellar
Mathematica
1100 1st Street, NE, 12th Floor
Washington, DC 20002
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Claire Smither Wulsin |
| File Modified | 0000-00-00 |
| File Created | 2025-06-15 |