Next Steps for Rigorous Research on Two-Generation Programs (NS2G)

NS2G Site Visit Topic Guide

Site Visit Topic Guide for Next Steps for Rigorous Research on Two-Generation Approaches Program

This guide walks through all of the Next Steps for Rigorous Research on Two-Generation (NS2G) site visit activities. During Day 1, Mathematica will meet with program leaders and managers, supervisors, frontline staff, and partner directors to learn more about the two-generation program model. Specifically, Mathematica will learn about the experience of providing two-generation services to parents, children, and families. Mathematica will use this information during Day 2 to facilitate activities where program and partner staff collaboratively define a theory of change, identify challenges, and brainstorm potential solutions.

A. Day 1

1. Introduction

a. Moderator and co-facilitator introductions

Thank you for taking the time to speak with us today. We are from Mathematica, an independent research firm, and we are here to learn about your experiences with the [TWO-GEN PROGRAM]. My name is [NAME] and my colleague is [NAME].

b. Explanation of project and purpose of discussion

INTERVIEWER NOTE: Tailor this portion as needed if the interviewee was also the participant in the earlier phone interview.

We are here today on behalf of the NS2G project. NS2G is a study sponsored by the Office of Planning, Research, and Evaluation (otherwise known as OPRE) in the Administration for Children and Families. Through this project, OPRE is interested in strengthening a small group of these programs in order to better prepare them for evaluations of effectiveness in the future. OPRE contracted with Mathematica to conduct this project. In addition, the project has two broader goals. One is building capacity of programs and researchers to conduct rigorous evaluations. The other is to address measurement issues to promote learning across evaluations and understanding of the outcomes of two-generation programs. OPRE contracted with Mathematica to conduct this project.

Today, we want to learn about your experiences in the program to better understand how [TWO-GEN PROGRAM] operates and to hear your perspective on how we could work together to make the program even stronger. The information you share today will help us understand how you deliver two-generation services and identify ways we can continue to support your work, and will help us to prepare for the group activities we have planned for tomorrow. Providing information is voluntary, and all individual responses that are collected will be kept private to the extent permitted by law. We expect this discussion to take about 90 minutes. Before we start, I want to let you know that your participation in this interview is voluntary. There are no right or wrong answers. We value the information you will share with us, and want to make sure we capture it all by recording it. Do we have your permission to record the discussion?

Do you have any questions before we get started?

Paperwork Reduction Act Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0356 and it expires 5/31/2021.

INTERVIEWER NOTE: Before going on site and based on your current understanding of the program and previous conversations with the program, tailor this topic guide to retain the information we have not already collected and develop program-specific questions tailored to the constructs in the guide.

Not all programs will have staff in the roles identified in the table. Tailor the topic guide based on the following definitions:

• Program leaders and managers- Individuals responsible for the overall direction and management of the program with a high-level understanding of the program’s mission.

• Program supervisors- Those who oversee program implementation, provide support to the frontline staff, and provide information to program leaders and managers.

• Frontline staff- Individuals in the program model responsible for serving parents, children, and families in the program; this can include frontline staff at a partner agency, if appropriate. Separate interviews will be conducted for staff (or small groups of staff) who work with different populations in the program.

• Partner agency director- Individuals responsible for the direction and management of partner programs.

Number of topics covered, by staff role

Topic	Program leaders and managers	Program supervisors	Frontline staff	Partner agency director
A. Community Context	6	7	5	8
B. Vision and goals	4	3	3	4
C. Partners	4	3	3	3
D. Intake	1	2	2	1
E. Service delivery and case flow	2	8	9	3
F. Program staffing	4	9	8	2
G. Data use	5	6	5	4
H. Program improvement and monitoring	5	7	6	4
Total	31	45	41	29

Topics to cover in interviews

Topic	Program leaders and managers	Program supervisors	Frontline staff	Partner agency director
A. Community context
Characteristics of families in the local community				
Characteristics of families that often participate in program				
Key needs of service population				
				
Federal, state, and local policies that affect program				
Additional context that could affect the program implementation (for example, community economic conditions or social environment)				
Additional program requirements that affect the program’s implementation (for example, reporting or eligibility requirements stemming from program funding sources)				
Gaps in available community services that are filled by the program				
B. Vision and goals
Description of program’s overall mission for providing two-generation services				
Goals of the program				
How participation in two-generation services helps families meet their goals				
Vision for the program in next few years				
C. Partners
Sharing of information across partners about participants, the parties involved, and the frequency				
Lessons learned or advice to share about partnering to deliver two-generation services from the perspective of frontline staff			
How co-designing and delivering services to parents and children together (as opposed to providing services to them separately) has added value for involved service providers.				
How co-designing and delivering services to parents and children together has the improved services provided				
D. Intake
Eligibility requirements for the program				
How a family is identified and recruited into the program			
E. Service delivery and case flow
Types of services offered for parents, children, and families				
Expected and actual duration, frequency, and dosage of services offered			
How and when the program assesses a family’s needs			
Normal case flow of services for parents, children, and families			
Whether some services require completion before accessing additional services			
Whether and how staff work to ensure the services provided to families align with the family’s goals				
Communication between staff about families receiving different services through the program				
Common challenges families encounter when participating in services			
Program procedures for when one family member completes services, but the other member(s) require(s) additional time to complete services			
F. Program staffing
Motivation for working in this position			
Skills necessary for frontline staff to be successful in their role, required qualifications for frontline positions, and (for leadership and supervisors only: the proportion of current staff who meet qualifications).				
Main activities of frontline staff working with program participants (such as recruitment, service delivery, administrative tasks, staff development, and any other activities)			
Proportion of time dedicated to administrative tasks (e.g. data entry)			
Proportion of time dedicated to working directly with families			
Average workload/caseload of frontline staff (by individuals or number of families)			
Frequency, duration, and subjects of meetings with supervisor			
How staff performance is assessed				
Frequency of staff turnover		
G. Data use
How the program tracks families’ goals, service receipt, and progress in program services, and whether it tracks families together, or whether separate systems exist for parents and children				
Whether and how the program tracks progress toward addressing identified family needs				
Measures used to track parents’ progress			
Measures used to track child[ren]’s progress			
Challenges and benefits associated with collecting and using data about the families’ progress				
Reporting requirements	
Whether and how the program monitors quality (completeness and accuracy) of data entered into data system				
H. Program improvement and monitoring
How the program assesses fidelity and whether/how staff at each level are involved				
Whether and how data are collected on program operations and service delivery			
Whether and how the program collects feedback about program services				
Whether and how data are used to identify opportunities for program improvement				
Additional data program staff would like to collect or analyze				
Recommended changes to the program				
Process for changing program policies or procedures		

2. Observation of program activity

Instructions for site visitors. When observing a program activity, such as a group meeting, training, workshop, or even a one-on-one participant-staff interaction, pay careful attention to the following:

• Who is in the room? How many participants are in the room?

• What is the purpose or objective of the activity, as articulated by the staff? Do participants appear to understand this purpose or objective?

• What is the dynamic between staff and participant(s)? Who seems to be leading, initiative, or driving the discussion?

• To what extent are the participant’s views, contributions, and opinions honored and incorporated into the discussion and any decisions?

• To what extent are the participants engaged in the activity? What does body language suggest about their level of engagement?

• To what extent is the activity structured for participants who have children with them?

• How would you describe the staff’s level of energy and engagement?

• What is your assessment of staff’s command and knowledge of the content?

• How is the activity structured? Is it interactive, instructional, or a combination of both?

• What does the physical space look like? Is the environment conducive to the activity? Is the environment warm, inviting, safe, and/or secure?

• To what extent are there distractions present in the room, as a result of staff, participants, or other factors?

• How long does the activity last?

• What information (generally speaking) is collected or shared during the meeting? Here, describe categories of information, but do not record personally identifiable or private information about participant.

B. Day 2

The following guide walks through purpose, vision, objectives and activities of Day 2 of the site visit. Mathematica will use the information learned during prior calls and the Day 1 activities to tailor the activities conducted on site with programs.

The number of attendees and types of staff and leadership in attendance will also shape the activities described in this meeting guide. Mathematica will work with the program to ensure participation from program leaders and managers, program supervisors, frontline staff (which can include partner frontline staff), and program partner directors. Ideally, activities will be conducted with two or more small (three to six person) groups with staff in various roles participating in each group. Mathematica will capture the work leadership and staff engage in during Day 2 (and not individuals) using photographs of the completed activities. Mathematica will reference these photographs for ongoing technical assistance and these photos will inform findings in the final summative report. Mathematica will not capture any staff names or likenesses in the photos. Mathematica staff will retain digital files either on their persons or in a locked cabinet until such time that they can be safely transferred to Mathematica’s network. Once they are transferred to the encrypted project folder, physical and digital files are destroyed. After the visit, Mathematica will provide a short write-up based on the activities conducted on site.

a. Purpose and vision of Day 2 activities

After meeting with program leaders and managers, supervisors, frontline staff, and partner staff during Day 1 to learn about the two-generation program model, Day 2 of the site visit aims to identify opportunities to develop and improve the program's two-generation approach and generate creative solutions to challenges the program is facing. Mathematica will use human-centered design activities to guide the discussion and demonstrate how to implement the LI2 framework to the program. This will help program staff use the framework and similar activities for program improvement after the completion of NS2G.

This meeting will kick off a partnership between the program and Mathematica, through which program staff will design and test a new or refined two-generation approach to serving low-income families and children.

b. Objectives

By the end of the meeting, participants will:

• Have a shared understanding of the program's goals and current services

• Identify specific challenges and clarify their motivations for change

• Explore creative ideas for program improvement

• Prioritize and plan next steps for programs

• Learn the phases and activities used in the LI² process

1. Brainstorming meeting guide

1. Welcome and introductions (30 mins)

1. Introduction to information collection: Mathematica staff will share the following to begin the day.

• After meeting with program leaders and managers, supervisors, frontline staff, and partner staff yesterday to learn about the two-generation program model, the activities today aim to identify opportunities to develop and improve [the program]'s two-generation approach and generate creative solutions to challenges the program is facing.

• We estimate the activities planned for today will take the full day.

• Providing information is voluntary, and all responses that are collected are kept private to the extent permitted by law.

Paperwork Reduction Act Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0356 and it expires 5/31/2021.

2. Program lead welcome: if interested, program leadership will welcome staff to the meeting.

3. Icebreaker activity: Rose, Bud, and Thorn: Participants will take three minutes, and use the red (rose), green (bud), and blue (thorn) post-it note in front of them to write down: Use one of the following based on the first day of the site visit. [Generally, use the first option, unless the site visitor sensed some apprehension from program staff.]:

• Focus on program: 1) something you think is a strength of [TWO-GEN PROGRAM] on the red post-it note, 2) an opportunity for [TWO-GEN PROGRAM] to grow or improve (this can be something discussed yesterday or something new) on a green post-it, and 3) a challenge you have faced with [TWO-GEN PROGRAM] on the blue post-it note.

• Focus on formative evaluation activities: 1) something you are looking forward to today related to this program on a red post-it, 2) an opportunity you’d like to talk more about today (this can be something we discussed yesterday or something new) on a green post-it, and 3) something that gives you pause on the blue post-it note.

After the time is up, Mathematica will ask for volunteers to introduce themselves and share their roses. Mathematica will then ask for volunteers to share their buds, and finally their thorns (participants are not required to share). At the end of the icebreaker, the group will have an understanding about what’s top of mind for staff and leadership.

4. Overview of project: Mathematica will provide an overview of the NS2G to program leadership and staff. During this time, Mathematica will describe the motivation for the project, planned activities, and the timeline for the project.

5. Overview of Learn, Innovate, Improve (LI²): Mathematica will describe the LI² framework, the objectives of each phase of the framework, and how Mathematica and the program will use the framework in the formative evaluation. Mathematica will explain to meeting participants that one goal of the session is to teach them LI² activities to build their capacity for continuous quality improvement.

2. Theory of change (95 minutes)

Mathematica will lead the group through a discussion to describe the program’s theory of change. Though programs may already have a developed theory of change, this activity can still be useful for programs to revisit the components of their program. Mathematica will break the full group into small groups with a minimum of three people in each group. Each group will develop a theory of change.

1. Documenting inputs, activities, outputs, and outcomes: Poster paper will show four empty boxes each labeled “inputs,” “activities,” “outputs,” and “outcomes.” Between each box is an arrow indicating that the first box will affect the subsequent box. Mathematica will lead the staff and leadership through a discussion to complete the boxes on the poster paper.

A theory of change visually maps out the resources required for a program to be implemented, the services provided by the program, the intentional pieces of a program that parents, children, and families participate in, and finally, the expected outcomes. A well-documented theory of change that clearly articulates the links between what the program does and how participant outcomes should change as a result, is important to have in place. The theory of change will help Mathematica better understand the program, and help staff clarify what success looks like for the program.

To build their theory of change, Mathematica will explain that the groups should start with what they want to achieve. This means that the groups will move backwards, from outcomes to inputs.

Small groups will begin by brainstorming expected outcomes for parents, children, and families who participate in their program. Outcomes can also focus on short-, medium-, and long-term outcomes for families who participate in the program. A short-term outcome may sound like “parents obtain full-time employment,” while long-term outcomes may sound like “families move above the federal poverty level.” Staff and leadership will spend three minutes individually brainstorming long-term outcomes on post-it notes. Next, they will spend seven minutes discussing the identified outcomes with their small group and select outcomes to write in their theory of change.

Staff and leadership will then spend another three minutes individually brainstorming short- and medium-term outcomes on post-it notes and seven minutes discussing the identified outcomes with their small group in order to select outcomes to write in the theory of change.

Next, small groups will identify the activities parents, children, and families participate in to achieve the described outcomes. Activities are the intentional pieces of the program model; they can include any tools or technology used to support families, processes and programming families might participate in, or actions staff might take to support families. Each individual will spend three minutes brainstorming activities on post-it notes. After time has expired, they will place post-it notes on their poster paper in the activities box. As a small group, they will talk through the identified activities. After discussing for seven minutes, the small group will select and write activities to include in their theory of change. Mathematica will ask for volunteers to share their written activities as examples.

Now that the groups have identified outcomes the program aims to achieve and the activities parents, children and families will participate in, small groups will connect the two by identifying outputs in their theory of change. Outputs are the direct results of participating in [TWO-GEN PROGRAM] activities and indicate the extent to which parents, children, and families participate in services. For example, a parent might participate in six education classes in order to obtain a credential through the program. Thinking about [TWO-GEN PROGRAM], staff and leadership will brainstorm outputs on post-it notes and place them on the theory of change. After seven minutes, they will discuss the identified outputs and select those to include in their theory of change. Mathematica will ask the groups to share the outputs included in their theory of change.

Finally, small groups will identify the inputs for their program. These are the resources required for a program to be implemented and can include any financial resources, staff, organizational resources, or community resources. Each individual will spend three minutes brainstorming inputs on post-it notes. After time has expired, they will place post-it notes on their poster paper in the inputs box. As a small group, they will discuss their identified inputs. After discussing for seven minutes, the small group will select and write the inputs for their theory of change on the poster paper. Mathematica will then invite groups to share their written inputs.

2. Brainstorming contextual factors: Mathematica will start by defining contextual factors for the group. Contextual factors are things that might enable or inhibit program implementation or influence the outcomes the program is trying to achieve. Program staff may have control over some contextual factors, but not others. Mathematica will explain how contextual factors might exist at different levels. These levels will be tailored to the program but can include participant, program, community, and policy levels. Identifying contextual factors will help staff and leadership identify challenges early on and begin working to address them.

Staff and leadership will spend three minutes individually brainstorming contextual factors on post-it notes. Next Mathematica will ask staff and leadership to brainstorm additional contextual factors at the participant level and then spend time focusing on each of the subsequent levels identified for the program.

After time has expired, the small groups will spend seven minutes discussing the contextual factors each individual identified and select three to four contextual factors to include in their theory of change. After the group has selected and written contextual factors on their theory of change, Mathematica will invite small groups to describe the contextual factors to the full group.

3. Theory of change model narrative: A theory of change narrative consists a succinct description of what the two-generation model aims to achieve, staff’s beliefs about how and why they are trying to achieve it (including any assumptions made about the population served), and the short path by which the program model will achieve its goal(s), including any expected short- and long-term outcomes.

Thinking of each component in the narrative and their theory of change, individuals will spend five minutes brainstorming a narrative. After five minutes have passed, Mathematica will invite staff and leadership to share their statements or components of their statements, if not finished, with their small group for three minutes. After small group members have shared, each small group will spend seven minutes developing a short narrative for their theory of change.

After small groups have written their narratives, they will share them with another group. Each pair of small groups will spend five minutes each reading and discussing each other’s theory of change, moderators, and narrative. Then, the groups will all come back together for a full group discussion (ten minutes). Each small group will present their theory of change and the full group will discuss the alignment and differences in the theories of change.

15-minute Break

3. Challenges and problems (50 minutes)

The small groups will now use their understanding of the program model and newly developed theories of change to identify challenges that staff and leadership have faced in achieving the desired outcomes articulated in the theory of change. These challenges may result in the program implementing in a way that differs from the theories of change or serves a population different from its target population. Grounded in staff’s understanding of the program model, the following activities will help staff clarify and deeply understand their motivations for change.

1. Individual brainstorming: Mathematica staff will instruct staff and leadership to spend three minutes brainstorming challenges (one challenge per post-it note) they have encountered in achieving [TWO-GEN PROGRAM]’s desired outcomes. The types of challenges identified can include but are not limited to: recruitment and enrollment; quality and consistency of service delivery; data collection; availability of services to address participant needs; relationships with partners or other contracted service providers; and data-informed decision-making.

After time has expired, staff and leadership will discuss the challenges they identified in small groups for seven minutes.

2. Affinity clustering: Next small groups will categorize the challenges they identified into groups, referred to as affinity clusters. The small groups define the clusters and what challenges are included in each cluster.

Small groups spend seven minutes discussing and moving challenges into clusters. As the groups discuss, members of each group will circle and label the different clusters. Mathematica will then lead a large group discussion about the challenges and clusters identified, similarities in challenges, and differences in the challenges identified. The group will spend ten minutes talking through which challenges are the highest priority currently. Each small group will select two challenges to address in the next activities.

3. Problem tree analysis: This activity will help staff dig into the causes and effects of the challenges selected in the previous activity. A sheet of poster paper will show a box with “challenge” labeled in the middle. Above the challenge text box is a row of boxes all labeled “effect” and below is a row of boxes labeled “causes.”

First, Mathematica will ask the groups to start with one of their challenges. The small groups will write the challenge into the center box. Then, they will spend three minutes individually brainstorming all the “causes” that create the challenge; these form the roots of the problem tree. Staff and leadership will post their post-it notes on poster paper and Mathematica will then invite staff and leadership to share their identified roots in their small groups for five minutes.

Small groups will then repeat the process for the “effects” of the challenges, forming the branches of the problem tree. Mathematica will, again, invite staff and leadership to share branches in their small groups for seven minutes.

Groups will repeat this process for the second challenge that they identified in the previous activity.

Mathematica will bring the small groups together and invite individuals from each small group to summarize the roots and branches of each problem tree for ten minutes.

4. Problem statements (15 minutes)

After staff and leadership have developed problem trees, Mathematica will lead the small groups through developing problem statements for each challenge in their trees. A problem statement is a clear and succinct way to phrase a challenge, using the five W’s (who, what, when, where, and why). Problem statements describe the challenge without describing any solutions, which will be an activity later in the day.

Small groups will select one problem tree to write a problem statement and will spend seven minutes workshopping a problem statement as a group. Mathematica will invite individuals from the small groups to share their problem statement before breaking for lunch.

60-minute Lunch

5. Stakeholder mapping (15 minutes)

In this activity, the group will move from thinking about problems to identifying enabling solutions. Stakeholders can be part of the solution so staff and leadership will brainstorm the people or groups of people who have a vested interest in [TWO-GEN PROGRAM] to begin thinking creatively about how partnerships could work. This activity will help participants brainstorm a comprehensive list of stakeholders, how they relate to each other, and how they relate to the brainstormed problem statements.

Stakeholders include the program’s target population, existing community partners, funders, policymakers, and other community resources. Staff and leadership will spend seven minutes brainstorming stakeholders. After that time, Mathematica will ask small groups to think about their problem statements and add any stakeholders who might be able to help [TWO-GEN PROGRAM] them address their problem.

After brainstorming stakeholders, the groups will add arrows indicating the relationships among the stakeholders. Mathematica will spend three minutes facilitating a discussion between groups where they verbally summarize the stakeholders identified, how their group defined stakeholders’ roles, and how the group defined the stakeholders’ relationship to the problem statements.

6. Brainstorming creative solutions (15 minutes)

Focusing on the challenge the groups selected for their problem trees, program staff and leadership will begin developing innovative solutions to the root causes that are identified in their problem tree. This activity will help staff creatively explore solutions.

The groups will create “How might we” statements to frame the process of brainstorming solutions. These statements should be aspirational and build on the problem statements created earlier in the day. They will help ensure all staff and leadership start on the same page in understanding the challenge when brainstorming solutions. Examples of “How might we statements” include:

• How might we streamline program enrollment so that parents and their children only have to give their information once?

• How might we get every eligible child of every parent we serve enrolled in services?

• How might we be so integrated with our partners that participants don’t realize we’re separate organizations?

1. Brainstorming: Using a pre-structured matrix on poster paper, each column will be labeled with a “How might we” statement and each row will be labeled with a category to identify enabling solutions. Creative matrices will vary by program, for example:

	How might we streamline program enrollment so that parents and their children only have to give their information once?	How might we get every child of every parent we serve enrolled in services?	How might we be so integrated with our partners that participants don’t realize we’re separate organizations?
Data and data systems
Staffing
Services for parents
Services for children
Services for the whole family
Partnerships
Wildcard

Assuming they have unlimited resources, staff and leadership will spend three minutes individually brainstorming solutions to the “How might we statements” with the goal of brainstorming at least one solution for each cell in the matrix. After time has expired, staff will then discuss the ideas generated in their small groups.

7. Prioritizing creative solutions (30 minutes)

1. Impact-difficulty matrix: For the next activity, each small group will be given a blank sheet of poster paper and instructed to write a horizontal right-facing arrow along the bottom of the sheet. They will label this line “impact.” Next, Mathematica will instruct the small groups to take the creative solution sticky notes from the creative matrix and rank-order them along the horizontal “impact” line according to how much of a difference they think the creative solution will make to program operations, services, or participant outcomes. They will be given 10 minutes to complete this activity.

Once they have ranked the creative solutions, Mathematica will instruct the small groups to draw a vertical arrow pointing upwards along the left side of the poster paper, bisecting the impact arrow in the bottom left corner. They will label this line “difficulty.” The small groups will then be given 10 minutes to rank the creative solutions according to their perception of how difficult they would be to implement. Mathematica will tell them that they should rank-order the solutions’ difficulty by moving them vertically, without shifting their position horizontally.

Once the stickies have been ordered according to “impact” and then according to “difficulty,” Mathematica will ask the small groups to draw lines splitting the paper into four quadrants, as shown:

Difficulty	Luxuries	Strategic investments
	Can do	Quick wins
	Impact

This will enable the program to categorize the solutions they brainstormed:

Quick wins: solutions that have a relatively high impact and can be implemented easily

Can dos: solutions that might not make a huge difference, but could still be worth doing since it is relatively easy to do them

Strategic investments: difficult, but high-impact solutions that may be long-term priorities to work towards

Luxuries: low-priority solutions that may not yield results commensurate with the effort it takes to implement them, and may not be worth pursuing

2. Discuss the impact-difficulty matrix: In the large group, Mathematica will lead a discussion about the solutions in each group’s matrix in order from “quick wins” to “luxuries.” Mathematica will ask probing questions to understand how and why the group came up with their rankings, whether the results surprised them, whether they feel that there any priorities coming out of the exercise that they’d like to work on, and which solutions can be achieved during the NS2G project period.

8. Closing and next steps (15 minutes)

1. Discussion: The full group will discuss the solutions brainstormed and how they relate to the theories of change drafted today. Mathematica will guide the discussion to include any changes the program may need in order to implement the solutions and edits to their theories of change now that they have brainstormed solutions. Mathematica will summarize the day’s activities and highlight what the group seemed to identify as potential quick wins and can-dos that they can work on immediately, and strategic priorities that they may want to plan for in the longer-term.

2. Reactions: To close out the meeting, Mathematica will ask if anyone would like to share any reflections about the solutions brainstormed, the activities conducted, the site visit, and NS2G more broadly.

3. Next steps: Mathematica will summarize concrete next steps with the program in a write up, detailing the support the program can expect from Mathematica, and how both the program and Mathematica will build on the theories of change and solutions brainstormed during upcoming monthly calls. The program will identify a core team of staff who will be responsible for carrying forward the formative evaluation work, including attending monthly phone calls and completing immediate follow-up items that emerge from the site visit.

Feedback survey

[This survey will be administered during the site visit.]

This satisfaction survey will assess how well we met our objectives today. Responses will help us improve, so please be candid.

	Strongly agree	Agree	Disagree	Strongly disagree		N/A
After today’s meeting, my colleagues and I have a shared understanding of the goals of my program and the current services we provide.	q	q	q	q	q
After today’s meeting, we identified specific challenges facing the program.	q	q	q	q	q
The meeting helped us identify improvements we can make to our program.	q	q	q	q	q
I learned about the LI2 framework and how I can use it in my program.	q	q	q	q	q
The meeting helped me learn something new about my program.	q	q	q	q	q
The meeting was engaging and gave everyone an opportunity to be heard.	q	q	q	q	q
This meeting was a good use of my time.	q	q	q	q	q
We have clear, actionable next steps after the meeting.	q	q	q	q	q
I left the meeting feeling inspired, empowered, and committed to work together to strengthen my program.	q	q	q	q	q

Free response

1. Please use the space below to comment on what you thought was best about the meeting. What was the most interesting or useful thing you took away from today’s activities?

2. Please use the space below to comment on what you thought could most use improvement about today’s activities.

DRAFT 10