Instrument 2 Updated Site Visit Topic Guide (clean)

NS2G Site Visit Topic Guide

Site Visit Topic Guide for Next Steps for Rigorous Research on Two-Generation Approaches Program

This guide walks through all of the Next Steps for Rigorous Research on Two-Generation (NS2G) site visit activities. In Part 1 of the virtual site visit, Mathematica will conduct interviews with program leaders and managers, supervisors, frontline staff, and partner directors by videoconference to learn more about the two-generation program model. Specifically, Mathematica will learn about the experience of providing two-generation services to parents, children, and families. Following these interviews, Mathematica will facilitate Part 2 of the virtual site visit, a series of program design sessions that include activities that draw on human-centered design principles for program and partner staff to collaboratively define a theory of change, identify challenges, and brainstorm potential solutions.

A. Part 1 (Interviews)

1. Introduction (for all interviews)

a. Moderator and co-facilitator introductions

Thank you for taking the time to speak with us today. We are from Mathematica, an independent research firm, and we are here to learn about your experiences with the [TWO-GEN PROGRAM]. My name is [NAME] and my colleague is [NAME].

b. Explanation of project and purpose of discussion

INTERVIEWER NOTE: Tailor this portion as needed if the interviewee was also the participant in the earlier phone interview.

We are speaking today on behalf of the NS2G project. NS2G is a study sponsored by the Office of Planning, Research, and Evaluation (otherwise known as OPRE) in the Administration for Children and Families. Through this project, OPRE is interested in strengthening a small group of these programs in order to better prepare them for evaluations of effectiveness in the future. OPRE contracted with Mathematica to conduct this project. In addition, the project has two broader goals. One is building capacity of programs and researchers to conduct rigorous evaluations. The other is to address measurement issues to promote learning across evaluations and understanding of the outcomes of two-generation programs. OPRE contracted with Mathematica to conduct this project.

Today, we want to learn about your experiences in the program to better understand how [TWO-GEN PROGRAM] operates and to hear your perspective on how we could work together to make the program even stronger. The information you share today will help us understand how you deliver two-generation services and identify ways we can continue to support your work, and will help us to prepare for the group activities we have planned in the near future. Providing information is voluntary, and all individual responses that are collected will be kept private to the extent permitted by law. We expect this discussion to take about 90 minutes. Before we start, I want to let you know that your participation in this interview is voluntary. There are no right or wrong answers. We value the information you will share with us, and want to make sure we capture it all by recording it. Do we have your permission to record the discussion?

Do you have any questions before we get started?

Paperwork Reduction Act Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0356 and it expires 2/29/2024.

INTERVIEWER NOTE: Based on your current understanding of the program and previous conversations with the program, tailor this topic guide before the meeting to retain the information we have not already collected and develop program-specific questions tailored to the constructs in the guide.

Not all programs will have staff in the roles identified in the table. Tailor the topic guide based on the following definitions:

• Program leaders and managers- Individuals responsible for the overall direction and management of the program with a high-level understanding of the program’s mission.

• Program supervisors- Those who oversee program implementation, provide support to the frontline staff, and provide information to program leaders and managers.

• Frontline staff- Individuals in the program model responsible for serving parents, children, and families in the program; this can include frontline staff at a partner agency, if appropriate. Separate interviews will be conducted for staff (or small groups of staff) who work with different populations in the program.

• Partner agency director- Individuals responsible for the direction and management of partner programs.

Number of topics covered, by staff role

Topic	Program leaders and managers	Program supervisors	Frontline staff	Partner agency director
A. Community Context	6	7	5	8
B. Vision and goals	4	3	3	4
C. Partners	4	3	3	3
D. Intake	1	2	2	1
E. Service delivery and case flow	2	8	9	3
F. Program staffing	4	9	8	2
G. Data use	5	6	5	4
H. Program improvement and monitoring	5	7	6	4
Total	31	45	41	29

Topics to cover in interviews

Topic	Program leaders and managers	Program supervisors	Frontline staff	Partner agency director
A. Community context
Characteristics of families in the local community				
Characteristics of families that often participate in program				
Key needs of service population				
Federal, state, and local policies that affect program				
Additional context that could affect the program implementation (for example, community economic conditions or social environment)				
Additional program requirements that affect the program’s implementation (for example, reporting or eligibility requirements stemming from program funding sources)				
Gaps in available community services that are filled by the program				
B. Vision and goals
Description of program’s overall mission for providing two-generation services				
Goals of the program				
How participation in two-generation services helps families meet their goals				
Vision for the program in next few years				
C. Partners
Sharing of information across partners about participants, the parties involved, and the frequency				
Lessons learned or advice to share about partnering to deliver two-generation services from the perspective of frontline staff			
How co-designing and delivering services to parents and children together (as opposed to providing services to them separately) has added value for involved service providers.				
How co-designing and delivering services to parents and children together has the improved services provided				
D. Intake
Eligibility requirements for the program				
How a family is identified and recruited into the program			
E. Service delivery and case flow
Types of services offered for parents, children, and families				
Expected and actual duration, frequency, and dosage of services offered			
How and when the program assesses a family’s needs			
Normal case flow of services for parents, children, and families			
Whether some services require completion before accessing additional services			
Whether and how staff work to ensure the services provided to families align with the family’s goals				
Communication between staff about families receiving different services through the program				
Common challenges families encounter when participating in services			
Program procedures for when one family member completes services, but the other member(s) require(s) additional time to complete services			
F. Program staffing
Motivation for working in this position			
Skills necessary for frontline staff to be successful in their role, required qualifications for frontline positions, and (for leadership and supervisors only: the proportion of current staff who meet qualifications).				
Main activities of frontline staff working with program participants (such as recruitment, service delivery, administrative tasks, staff development, and any other activities)			
Proportion of time dedicated to administrative tasks (e.g. data entry)			
Proportion of time dedicated to working directly with families			
Average workload/caseload of frontline staff (by individuals or number of families)			
Frequency, duration, and subjects of meetings with supervisor			
How staff performance is assessed				
Frequency of staff turnover		
G. Data use
How the program tracks families’ goals, service receipt, and progress in program services, and whether it tracks families together, or whether separate systems exist for parents and children				
Whether and how the program tracks progress toward addressing identified family needs				
Measures used to track parents’ progress			
Measures used to track child[ren]’s progress			
Challenges and benefits associated with collecting and using data about the families’ progress				
Reporting requirements	
Whether and how the program monitors quality (completeness and accuracy) of data entered into data system				
H. Program improvement and monitoring
How the program assesses fidelity and whether/how staff at each level are involved				
Whether and how data are collected on program operations and service delivery			
Whether and how the program collects feedback about program services				
Whether and how data are used to identify opportunities for program improvement				
Additional data program staff would like to collect or analyze				
Recommended changes to the program				
Process for changing program policies or procedures		

B. Part 2

The following guide walks through purpose, vision, objectives and activities of Part 2 of the virtual site visit. Mathematica will use the information learned during prior calls to tailor the activities conducted virtually with programs.

The number of attendees and types of staff and leadership in attendance will also shape the activities described in this meeting guide. Mathematica will work with the program to ensure participation from program leaders and managers, program supervisors, frontline staff (which can include partner frontline staff), and program partner directors. Ideally, activities will be conducted with one small group of up to six staff in various roles participating in each group. Mathematica staff will take notes and screen captures of working documents from the activities. After the visit, Mathematica will provide a short write-up based on the activities conducted on site.

a. Purpose and vision of Part 2 activities

After meeting with program leaders and managers, supervisors, frontline staff, and partner staff during Part 1 to learn about the two-generation program model, Part 2 of the virtual site visit aims to identify opportunities to develop and improve the program's two-generation approach and generate creative solutions to challenges the program is facing. Mathematica will use human-centered design activities to guide the discussion and demonstrate how to implement the LI2 framework to the program. This will help program staff use the framework and similar activities for program improvement after the completion of NS2G. In between sessions, Mathematica will use the information gathered to develop a program’s (or refine a program’s existing) logic model and seek feedback from program staff.

b. Objectives

By the end of the series of design sessions, participants will:

• Have a shared understanding of the program's goals and current services

• Identify specific challenges and clarify their motivations for change

• Explore creative ideas for program improvement

• Prioritize and plan next steps for programs

• Learn the phases and activities used in the LI² process

c. Number of sessions

There will be four design sessions, totaling 6.5 hours (including 1 hour to review materials between Sessions 3 and 4):

• Session 1 (60 minutes): Introduction to the NS2G project and technology

• Session 2 (90 minutes): Identifying key outcomes to focus on and thinking about what the program does well and what are areas for improvements

• Session 3 (90 minutes): Exploring and unpacking problems and challenges

• Session 4 (90 minutes): Generating and prioritizing potential improvement strategies

Session 1 (60 minutes)

Objectives

• Understand the NS2G project and LI² framework

• Have a shared understanding of the purpose of this meeting series

• Become familiar with technology that will be used to facilitate sessions

Welcome and introductions

1. Introduction to information collection: Mathematica staff will share the following to begin the day.

• After meeting with program leaders and managers, supervisors, frontline staff, and partner staff to learn about the two-generation program model, the activities we will conduct together aim to identify opportunities to develop and improve [the program]'s two-generation approach and generate creative solutions to challenges the program is facing.

• Providing information is voluntary, and all responses that are collected are kept private to the extent permitted by law.

Paperwork Reduction Act Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0356 and it expires 2/29/2024.

1. Program lead welcome: if interested, program leadership will welcome staff to the meeting.

2. Introductions: Staff will introduce themselves and their role with the two-generation program.

3. Overview of project: Mathematica will provide an overview of the NS2G to program leadership and staff. During this time, Mathematica will describe the motivation for the project, planned activities, and the timeline for the project.

4. Overview of Learn, Innovate, Improve (LI²): Mathematica will describe the LI² framework, the objectives of each phase of the framework, and how Mathematica and the program will use the framework in the formative evaluation. Mathematica will explain to meeting participants that one goal of the session is to teach them LI² activities to build their capacity for continuous quality improvement.

Overview of MURAL functionality

1. Mathematica staff will introduce program staff to MURAL, a virtual whiteboard tool that Mathematica will use to facilitate the design sessions. Mathematica will provide an overview of the whiteboard functionality and then have staff participate in a practice exercise.

Reflecting on experiences working for [Program]

1. Program staff will reflect on the work they do, brainstorming responses to: (1) something they are passionate about that drives them to do the work they do; (2) something they find frustrating or difficult about their work; and (3) an opportunity to improve the work they do, or to do it better. This exercise is intended to give staff more opportunity to practice using MURAL and get into the mindset of thinking critically about their work.

2. After brainstorming, staff will share and discuss.

Closing

1. Mathematica will review the scope and sequence and topics to be discussed in the remaining three sessions.

Session 2 (90 minutes)

Objectives

• Identify priority intended outcomes (longer term) and behavioral changes (shorter term) of the program

• Target areas of program operations and/or services for improvement

Welcome and introductions

1. Mathematica will remind participants of the purpose of the meetings and activities completed in the first session

2. Program staff will ask questions and share reflections from the first meeting

Understanding intended outcomes

1. This exercise is designed to identify longer-term outcomes that would appear in a program logic model

2. Program staff will individually brainstorm four types of intended outcomes: (1) children; (2) parents; (3) families; (4) other outcomes

3. Program staff will share and discuss the intended outcomes they brainstormed and prioritize those that most important. Discussion questions include:

   1. What are the outcomes that you wrote for each quadrant? What common themes were there? What outcomes are the highest priority for the program to focus on (things that the program can have the most direct influence on, or areas where they feel like their program needs the most work)? What are second- and third-order priorities? How many, and what types, of different outcomes are there in each quadrant? What overlaps are there across quadrants? Are there quadrants that you have focused on more heavily? What questions do people have?

Identifying observable changes necessary to achieve intended outcomes

1. This exercise is designed to identify shorter-term outcomes and program outputs that would appear in a program logic model.

2. Program staff will choose the two highest-priority intended outcomes from the activity above and will individually brainstorm responses to the following question: What short-term, observable changes should you see in program participants in order to know that you are on the right track to achieve intended outcomes? Participants will brainstorm responses in three categories:

   1. Adult-focused changes

   2. Child-focused changes

   3. Parents or family-focused changes

3. After the brainstorm, staff will share and group their responses by theme.

Identifying opportunities for change

1. This exercise is to get meeting participants to think critically about how their program is doing, and whether program services are adequate to achieve intended outcomes. This exercise will help participants start to identify challenges that they want to address through rapid-cycle learning.

2. Participants will individually brainstorm responses to three prompts:

   1. What do we do well as a program?

   2. What challenges do we have, or where do we currently fall short, with what we are currently doing?

   3. What opportunities are there for growth?

3. Participants will share their brainstorms and group similar responses and themes together.

Closing

1. Mathematica will explain the goals of the next meeting and explain that we will use the information from Meeting 2, combined with other information about inputs and services gathered in the interviews, to start building a program logic model.

Session 3 (90 minutes)

Objectives

• Flesh out the opportunities and challenges identified in Meeting 1 into a set of concrete problem statements

• Explore the root causes and effects of these problems.

Welcome and review

1. Mathematica will recap the progress made and items discussed during Meeting 2.

Narrowing in on a challenge to address

1. This exercise is designed to focus participants on one challenge to address.

2. Mathematica will present on the computer screen the problems and challenges that staff identified in the last exercise of Meeting 2. Participants will discuss these problems and challenges and come to agreement on one challenge that they would like to focus on for the meeting.

Developing a problem statement

1. This exercise is designed to help program staff more clearly articulate the challenge that they have identified.

2. Participants will discuss the challenge they have just selected. The focus of the discussion will be to agree on responses to three questions:

   1. What is the challenge, specifically?

   2. How does this challenge affect the people involved?

   3. Why is it important?

3. After participants discuss answers to the “what,” “how,” and “why” questions, they will craft a problem statement that incorporates all answers.

Exploring root causes and effects

1. These exercises are designed to get participants to unpack and understand in detail the challenge that they have specified.

2. Participants will brainstorm root causes and effects of their problem statement.

3. After brainstorming root causes and effects, participants will vote on two causes or effects that are most salient to the program, or within its scope of influence.

4. Participants will further unpack each cause or effect they selected by answering five successive “whys,” exploring the underlying “why” for each topic. For example:

   1. Topic: I was late for work

      1. Why: I overslept

      2. Why: I was stressed and had a hard time falling asleep

      3. Why: My child was acting out in school

      4. Why: They have had trouble controlling their emotions lately

      5. Why: Their routines are off because we just had to move

5. At the end of this exercise, participants will have specified and unpacked a specific challenge that their strategy can address.

Closing

1. Mathematica will recap the challenges that participants discussed and the deeper problems that they uncovered connected to those challenges. Mathematica will preview the focus of the final session, to brainstorm creative strategies to address the challenges that participants have identified.

2. After Session 3, Mathematica will draft a program logic model. This model will incorporate information gathered through Part 1 of the virtual site visit (inputs, activities, contextual factors) and Part 2 of the virtual site visit (activities, outputs, outcomes, contextual factors). Mathematica will input the model into MURAL between Sessions 3 and 4, and send it to the participants to request that they provide feedback before Session 4 begins, with instructions to spend no more than an hour on it.

Session 4 (90 minutes)

Objectives

• Identify and prioritize strategies to address challenges and strengthen the program model.

Welcome and recap

1. Mathematica will review feedback that participants provided on the program logic model.

2. Mathematica will summarize the challenges that participants identified in Meeting 3 and ask for additional input, particularly around opportunities participants saw in the logic model to address challenges.

3. Mathematica will refresh participant understanding of the LI² framework and explain that the group is now moving onto the “Innovate” phase to develop creative, science-informed solutions to an identified challenge

Brainstorming creative solutions

1. This activity is designed for participants to start to develop innovative solutions to the root causes and challenges they identified in Session 3.

2. Participants will individually brainstorm as many creative solutions as they can to address the challenge. Then, they will share the creative solutions they generated, grouping similar solutions together by theme.

Prioritizing creative solutions

1. This activity, is designed to help program staff strategically prioritize their strategies.

2. First, participants will be asked to each pick two strategies from the brainstorming activity that were most interesting to them, or to come up with a new strategy that included multiple ideas from the brainstorm. The group should then have about 10 strategies.

3. Next, Mathematica will instruct participants to rank-order the strategies according to how much of a difference they think the strategy will make to program operations, services, or participant outcomes.

4. Once they have ranked the strategies according to importance, Mathematica will instruct participants to rank-order the strategies according to “difficulty,” without shifting their horizontal position.

5. Once the stickies have been ordered according to “importance” and then according to “difficulty,” Mathematica will help participants interpret the graph by showing four quadrants on the screen:

Difficulty	Luxuries	Strategic investments
	Can do	Quick wins
	Impact

This will enable the program to categorize the solutions they brainstormed:

Quick wins: solutions that have a relatively high impact and can be implemented easily

Can dos: solutions that might not make a huge difference, but could still be worth doing since it is relatively easy to do them

Strategic investments: difficult, but high-impact solutions that may be long-term priorities to work towards

Luxuries: low-priority solutions that may not yield results commensurate with the effort it takes to implement them, and may not be worth pursuing

6. Participants will discuss the strategies in the matrix in order from “quick wins” to “luxuries.” Mathematica will ask probing questions to understand how and why the group came up with their rankings, whether the results surprised them, whether they feel that there any priorities coming out of the exercise that they’d like to work on, and which solutions can be achieved during the NS2G project period.

Closing and next steps

1. Participants will individually brainstorm and share their top-of-mind ideas, questions, and considerations related to the activities they have completed over the four sessions and the strategy they have begun to develop. The brainstorm will be divided into four categories: (1) hopes; (2) fears; (3) needs; and (4) wants.

2. After the individual brainstorm, Mathematica will facilitate a discussion about participants’ most important hopes, fears, wants, and needs for the program improvement strategy. Mathematica will ask for volunteers to share and encourage others to build on the discussion, either by adding to a volunteer’s thought or sharing one of their own hopes, fears, wants, or needs. For example, one person’s hope may be another’s need. The goal of the discussion is to identify places where the strategy may need more thought after the call, and/or what guidance staff may need to implement it.

Feedback survey

[This survey will be administered after Meeting 4 using a low-burden web survey tool.]

Paperwork Reduction Act Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0356 and it expires 2/29/2024.

This satisfaction survey will assess how well we met our objectives over our series of meetings (design sessions) with you. Responses will help us improve, so please be candid.

	Strongly agree	Agree	Disagree	Strongly disagree		N/A
After the design sessions, my colleagues and I have a shared understanding of the goals of my program and the current services we provide.	q	q	q	q	q
After the design sessions, we identified specific challenges facing the program.	q	q	q	q	q
The design sessions helped us identify improvements we can make to our program.	q	q	q	q	q
I learned about the LI2 framework and how I can use it in my program.	q	q	q	q	q
The design sessions helped me learn something new about my program.	q	q	q	q	q
The design sessions were engaging and gave everyone an opportunity to be heard.	q	q	q	q	q
These design sessions were a good use of my time.	q	q	q	q	q
We have clear, actionable next steps after the design sessions.	q	q	q	q	q
I left the design sessions feeling inspired, empowered, and committed to work together to strengthen my program.	q	q	q	q	q

Free response

1. Please use the space below to comment on what you thought was best about the design sessions. What was the most interesting or useful thing you took away from the activities?

2. Please use the space below to comment on what you thought could most use improvement about the design sessions.

DRAFT 11