1. Program Confirmation Protocol

NS2G Program Confirmation Call Protocol Mathematica

Program confirmation Call Protocol for Next Steps for Rigorous Research on Two-Generation Approaches Program

1. Background

As we have previously discussed, we are contacting you on behalf of the Next Steps for Rigorous Research on Two-Generation Approaches (NS2G) project. NS2G is a study sponsored by the Office of Planning, Research, and Evaluation (otherwise known as OPRE) in the Administration for Children and Families. Through this project, OPRE aims to better understand the implementation of programs that intentionally combine early child development and family economic security services, known as two-generation programs. OPRE is also interested in helping strengthen a small group of these programs in order to better prepare them for evaluations of effectiveness in the future. In addition, the project has two broader goals. The first is building capacity of programs and researchers to conduct rigorous evaluations. The second is to address measurement issues to promote learning across evaluations and understanding of the outcomes of two-generation programs. OPRE contracted with Mathematica to conduct this project.

Back in January 2020, we asked a group of stakeholders, including researchers, two-generation program practitioners, and federal staff, to nominate programs that provide two-generational services. Your program was one of the few that was selected. Today, we are calling to gather additional information about how your program collects data about families, how you use data to inform decision making, [TWO-GEN PROGRAM]’s previous experience with program improvement, your partnerships, and your interest and availability to participate in NS2G. If you decide to proceed with the project, we will use this information to further our understanding of the topics and issues we could work on together in NS2G.

Your responses and participation are completely voluntary. All individual responses that are collected are kept private to the extent permitted by law. There are no repercussions if you choose not to participate. If you do not know the answer to a question asked, or you prefer not to answer a question for any reason, please let us know that and we will simply move on. We expect this call will take 60 to 90 minutes. Do you have any questions?

Paperwork Reduction Act Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0356 and it expires 5/31/2021.

2. Interest

Before we dive into our discussion about your program, we would first like to talk about your interest in NS2G. As a reminder, we’d like for you to think about designating a staff member as a main point of contact for the study. This person would help coordinate participation in various calls and the site visit, be the designated recipient for electronic materials that we send, participate in monthly calls, and transfer key messages from this effort back to program team members who are not regularly participating in the project. We anticipate that this would be a few hours a month. The ideal person might be someone like a program coordinator or supervisor who is involved in the day-to-day operations and who works closely with program management and direct services staff.

1. How interested are you in participating in NS2G?

3. Available data

Next, let’s talk about the data that [TWO-GEN PROGRAM] collects and how that information is stored.

1. What data are collected about the services parents receive and their outcomes? [Probes: What type of data are collected? How are data collected? How often are data collected?]

2. What data are collected about the progress children have made? [Probes: What type of data are collected? How are data collected? How often are data collected?]

1. What measures are used to track their progress?

3. What data are collected about the progress families have made? [Probes: What type of data are collected? How are data collected? How often are data collected?]

1. What measures are used to track their progress?

4. What data system(s) are used to track progress by parents, children, and families?

1. To what extent, do(es) the data system(s) allow for tracking which parents and children are in the same family?

2. [If the program uses multiple systems] How do the different data systems link to one another to combine data for parents, children, and families?

5. What processes are in place to monitor the accuracy and completeness of data? [Probes: What processes check the accuracy of data? What processes check the completeness of data?]

1. At [TWO-GEN PROGRAM], how many staff members are responsible for data processing? What proportion of these staff members’ full-time employment is dedicated to data processing?

2. At [TWO-GEN PROGRAM], how many staff members are responsible for using data for improvement? What proportion of these staff members’ full-time employment is dedicated to using data for improvement?

6. What additional data would [TWO-GEN PROGRAM] like to collect? How would that data be useful to the program?

4. Data use

Now that we know more about the data [TWO-GEN PROGRAM] collects, let’s talk about how data are used in the program.

1. How does the program use data collected about parents, children, and families to make data-informed decisions? [Probe: How does the program use data for program improvement? How does the program use data for measuring program success? How does the program use data to inform what services parents, children, and families receive?]

1. What kinds of conversations do staff have about the data collected?

2. Does the program report data to funders or other stakeholders? What data, and how frequently? Are the data the program reports to funders and others required by those entities, or does the program report the data voluntarily?

3. To what extent are the data that are collected aligned to the program’s intended outcomes for parents, children, and families?

4. What challenges does [TWO-GEN PROGRAM] experience with collecting and using data?

5. Previous experiences with program improvement

Next, we would like to learn more about your previous experiences with program improvement.

1. Has [TWO-GEN PROGRAM] previously used a participated in continuous quality improvement or used data to improve program services? If so:

1. To what extent did supervisors and frontline staff participate in continuous quality improvement and provide input?

2. What was the most challenging aspect of the continuous quality improvement?

3. What changes were implemented after using the continuous quality improvement?

2. Has [TWO-GEN PROGRAM] previously participated in any learning communities? (These might be groups of programs or staff that share their experiences about lessons learned or collaboratively problem solve.) If so:

1. To what extent did supervisors and frontline staff participate in the learning community and provide input?

2. What was the most challenging aspect of participating in the learning community?

3. What was the most rewarding aspect of participating in the learning community?

3. Have you ever worked with an external evaluator? If so, what did you work on?

1. To what extent did supervisors and frontline staff participate in working with the external evaluator and provide input?

2. What did you learn from the process of working with an external evaluator?

6. Established partnerships

Partners can be very important to the implementation of two-generation services. Let’s talk about the programs [TWO-GEN PROGRAM] works with.

1. What are the programs that [TWO-GEN PROGRAM] partners with to deliver two-generation services?

1. What services do partners provide to [TWO-GEN PROGRAM]?

2. How central are these partnerships to your program? How long has the partnerships been in place?

3. How have you formalized these partnerships? [Probes: Have you established partnership agreements or MOUs?

2. How frequently do you communicate with partners that help deliver two-generation services? What do you communicate about?

3. What are some challenges [TWO-GEN PROGRAM] has encountered in working with partners to provide two-generation services?

4. What are some examples of successes [TWO-GEN PROGRAM] has experienced in working with partners to provide two-generation services?

7. Feasibility of participation

Thank you for sharing all of this information with us. Now we would like to talk about [TWO-GEN PROGRAM] potentially participating in NS2G.

5. Based on your experience, what kind of availability would you have to contribute to work involved with NS2G?

6. Based on your experience, what kind of availability would supervisors have to contribute to work involved with NS2G?

7. Based on your experience, what kind of availability would frontline staff have to contribute to work involved with NS2G?

8. Wrap-up and next steps

This wraps up the topics we wanted to discuss with you today.

1. Is there anything we haven’t discussed today that you would like to talk about?

Thank you very much for your time. Everything you have shared with us is going to be very helpful for project planning. Our next steps include taking this information and information other programs have shared with us back to our federal partners to discuss the programs that might be a good fit for this project. We anticipate this process could take three to four weeks.

After OPRE approves the programs to participate in this project, we’ll reach back out to you to confirm your interest in and availability for participation and discuss the next steps. This will likely include developing a memorandum of understanding, or MOU. This MOU would define expectations for the program and Mathematica, such as the expected effort from programs and the technical assistance Mathematica will provide. We expect that process to take another three to four weeks.

DRAFT 1