2021_11_24 Instrument 1. Project Impact RCL Plan Template_v2_clean

Project Impact: CSBG Rapid-Cycle Impact Project – Rapid-Cycle Learning Plan Template

Project Impact Rapid-Cycle Learning Plan Template

The Rapid-Cycle Learning Plan template is a tool to help your team prepare for rapid-cycle learning and inform refinements to your Rapid-Cycle Impact project. Rapid-cycle learning can help your team to assess project improvements or new service strategies quickly and systematically. Results from rapid-cycle learn can support continuous improvement of service delivery, enhancing the likelihood of positive outcomes for clients. Rapid-cycle learning typically involves four steps: (1) identify a problem or area of challenge; (2) propose specific changes or service components that directly address the challenge and intended improvements; (3) identify relevant, easily accessible measures of outcomes; and (4) assess whether the change led to better short-term outcomes (Cody and Arbour 2019, Derr et al 2017, Atukpawu-Tipton and Poes 2020). Rapid-cycle learning is often iterative and may involve repeating these steps multiple times to continuously improve service delivery and outcomes.

The Rapid-Cycle Learning Plan template for Project Impact grant recipients includes three sections:

1. Project road map for change. In this section, you will specify your project goals, implementation plan, factors that may help or hinder your project, and expected outcomes. The road map for change is a streamlined logic model that will help you identify problems or challenges you want to address or components of your project to explore through rapid-cycle learning.

2. Rapid-cycle learning capacity assessment. This assessment covers key areas that facilitate engaging in rapid-cycle learning. Completing the assessment will help your team to understand what you already have in place and where you might consider building capacity over the course of the project.

3. Initial learning questions. This section guides your team to develop an initial set of questions about your project and priorities for rapid-cycle learning. Though additional questions will emerge during the project, identifying a set of initial questions will allow your team to begin to identify (1) what changes might you make to address a known challenge, (2) what you want to learn about those changes, (3) how you will assess whether a change led to better short-term outcomes, and (4) what information you will use to measure those outcomes.

All Project Impact grant recipients are asked to complete this learning plan template. Please complete the template no later than [MONTH DATE], 2022. Share the completed template with [FILL IN].

THE PAPERWORK REDUCTION ACT OF 1995

This collection of information is voluntary and will be used to provide the Administration for Children and Families with information to help refine and guide program development for Community Services Block Grant recipients. Public reporting burden for the collection of information is estimated to average 4 hours (2 hours for the initial response and about 1 hour for each update), including the time for reviewing instructions and completing the form. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #: 0970-0531, Exp: 07/31/2022. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Roneika Carr at [email protected].

1. Project road map for change

A road map for change is a tool that guides learning during the project. It serves three purposes:

1. Helps your team to reflect on whether what you are doing will plausibly lead to your expected outcomes, and if not, what needs to change.

2. Guides rapid-cycle learning by specifying the components of your project, how you think they will be implemented, what might get in the way, and expected outcomes.

3. Anchors learning over the course of the project and supports sharing of insights about how to implement your project well.

We encourage you to use the road map for change as a living document. As you learn about implementing your project, update the road map to reflect new insights about how your project works.

The road map for change template is in Exhibit 1. The following guidance will help you to complete it.

Goals and outcomes: What do you hope to achieve?

• What are your goals? What will it look like you are successful? (Add these in section C.)

• Looking at your project’s logic model, what long-term outcomes did you identify for clients and staff? Which of these are your priority outcomes?

• How will you know if you are successful? What short-term outcomes for clients and staff will show you are on the right track?

Strategies: What will you do and how will you do it?

• Identify the components of your project. (Add these in section A.)

• It is important to be detailed when describing the components. For example, if your logic model included “provide case management” as a key activity, your road map would describe what case management looks like. This would include who will receive it, the frequency, and the content. Specify all steps required for a client to complete the program or service.

• When specifying components, include not just services or processes, but what you need to accomplish these and your approach. This could include processes or policies, people, tools or technology, and guiding principles. Offer enough detail to explain the project to someone new.

• What influencers may help or hinder implementation of your project? (Add these in section D.)

• What may help your team to implement the project’s components well? Identifying help factors will support your team in thinking about how to build on them and use existing supports.

• Now think about what might get in the way of implementing the components well. Considering hinder factors helps your team be proactive in developing strategies to get ahead of these barriers.

• How will you implement the components of your strategies and integrate them into your organization? (Add these in section B.)

• How you will implement is referred to as your integration strategy. Explain how you will implement the project components, including how you will support staff to implement as intended.

• As part of the integration strategy, describe how you will get ahead of hinder factors and build on helpful factors.

Exhibit 1. Road map for change template

Road Map for Change for:

Project goal:

A. Project components (What)	B. Integration strategies (How)		C. Indicators of success (Outcomes)
			Clients
			Short-term:	Long-term:
			Staff
			Short-term:	Long-term:
D. Key influencers
Helps:		Hinders:

2. Rapid-cycle learning capacity assessment

Rapid cycle learning is a process that can be taught and applied. Throughout the duration of your Project Impact grant, the Project Impact TA Support (PITAS) team will offer foundational knowledge and information on skills to build the capacity of your team to plan and carry out rapid-cycle learning. In addition to learning the process, your team will benefit from establishing conditions that support learning and improvement – including a culture that supports innovation and access to high-quality data.

Exhibit 2 provides a self-assessment related to rapid-cycle learning. The assessment identifies relevant areas, such as organizational leadership, culture, and infrastructure, human capital, and stakeholder engagement. Use the self-assessment to gauge the extent to which your Project Impact team uses the practices in your project. Use the following categories to assess where your team is with each area:

• Mark Early if you are considering using this practice as part of your project, but have not yet taken steps to put it into place. Also mark Early if you do not use the practice.

• Mark Transitional if you have begun taking steps to incorporate the practice into your project.

• Mark Advanced if you make this practice an integral and ongoing part of your project.

Over time, you can use this assessment to gauge your team’s growth, appreciating that growth is always possible. Work to develop and grow the areas identified as Early. Consider how to leverage and further grow in areas identified as Advanced.

Exhibit 2. Project Impact – Rapid-Cycle Learning Capacity Assessment

Area	What this looks like	Early	Transitional	Advanced
Organizational leadership and culture
Commitment	We dedicate time and resources to support learning and improvement, such as soliciting feedback for improvement and communicating about the importance of collecting and using data for learning.
Learning culture	We establish a process for engaging in learning and reflection, including prioritizing continuous improvement.
Equity and inclusion	We seek perspectives from a range of staff and stakeholders and value lived experiences as a key source of knowledge.
Decision making	We model data-driven decision-making, using evidence to inform key decisions.
Innovation	We value and encourage innovative ideas; promote a space where it is safe to fail and reframe failed ideas as opportunities for learning.
Organizational Infrastructure
Program monitoring	We have identified priority outcomes, have systems to track outcomes, and regularly monitor data to understand and reflect on performance.
Technology	We collect, store, and analyze data electronically.
Data infrastructure	We have processes and protocols in place for collecting, storing, accessing, and managing data.
	We have identified existing or easily collected information to support learning.
	We make data accessible to staff to support learning.
Improvement infrastructure	We have processes for identifying challenges, testing changes, and assessing improvement.
	We have a team responsible for overseeing rapid-cycle learning efforts.
	We ask staff to contribute to discussions on improvement and learning.
	We use learning to update our approach, materials, and processes.
	We have criteria to determine when to sustain, scale, or abandon a new project.
Human Capital
Learning capacity	We use systematic quantitative and/or quantitative information for learning and feel comfortable using systematic information and evidence to inform improvements.
Data collection capacity	We have access to internal or external staff with skills to develop data collection instruments and tools and to collect high-quality data.
Analysis skills	We have access to internal or external staff with necessary analysis skills to carry out rapid-cycle learning plans.
Stakeholder engagement
Engage stakeholders	We have identified staff, partners, and clients to engage in improvement efforts.
	We regularly share program successes, challenges, and updates with staff, partners, and clients.
Feedback gathering	We provide ongoing opportunities for staff, partners, and clients to provide feedback.
Feedback use	We value and use feedback from staff, partners, and clients to inform decisions.

After you complete the assessment:

• Consider the areas rated as Early. What are some immediate steps you can take to move from Early to Transitional?

• What supports would help your team to build capacity or strengthen practices in this area?

3. Initial learning questions

Developing strong learning questions is a first step to engaging in rapid-cycle learning. Complete Exhibit 3, which provides a structure for identifying initial questions.

Questions should be driven by gaps in knowledge. While new questions will emerge over the course of this project, this is an opportunity to begin identifying and prioritizing learning needs and thinking about the data and information you will need to answer those questions. Remember, these initial questions are not a comprehensive plan for all of your rapid-cycle learning.

Learning questions may focus on the feasibility of implementing a component; how clients experience a service or activity; or how a help or hinder factor influences the project. Consider the following to help your team generate early learning questions:

• What components of the road map for change are you most uncertain or worried about implementing? For example, maybe you’re not sure about the best way to raise awareness about the new or different services that you’re offering and are concerned about recruiting clients to your project. What information can you collect to learn about how your recruitment efforts are going and how you might tailor them to enhance effectiveness?

• Which influencers have the most potential to hinder project implementation? For instance, maybe your project is partnering with child care centers to offer services, like job training and GED classes, to parents, and you’re worried that center directors may be too distracted to partner in building out this new component. How will you learn about what is driving this potential barrier and what you can try to address it?

• What do your stakeholders want to know about your project and how it’s working? Are your partners, clients, or other community members interested in a specific aspect of your program? Do you need to demonstrate the promise of your approach to support long-term sustainability?

As you develop initial questions, consider the following:

• Focus on implementation and short-term outcomes, not effectiveness. Rapid-cycle learning often starts with piloting a strategy with a small number of staff or clients, sometimes called a “road test”. Early road tests often provide information about refining practices to support positive results, not causal evidence about effectiveness. Over time, some teams may engage in road tests that use advanced research designs or analytic approaches.

  • For example, ask: What strategies do case managers use to build relationships in a virtual space? Which are most promising?

  • Avoid asking: Is virtual case management effective?

• Ask about possible solutions. The intent of rapid-cycle learning is to learn, not to offer pre-conceived ideas about answers. You can miss important context if you prematurely presume that you know the answer to the best solution.

  • For example, ask: What are promising strategies for addressing limited Internet access of clients and staff?

  • Avoid: Do we need Internet hotspots to help clients access virtual services?

• Focus on what to do differently to support clients.

  • For example, ask: How can we improve access to virtual case management for clients?

  • Avoid: Why don’t clients show up for case management sessions?

In addition to specifying learning questions, Exhibit 3 prompts you to consider how you might answer these questions. As you complete the rest of the table, consider the following:

• Why is this a priority question? How might answering this question address a known challenge or fill a gap in knowledge? Does how we answer this question affect other parts of the project?

• How will we answer the learning question? What strengths identified in the capacity assessment can we draw on to answer this question? Do we have dedicated staff to help answer this question? Do we want to answer this question through data and quantitative analysis or do we want to gather qualitative feedback from staff or clients? Do we have staff who can design surveys or interview clients? Do we have staff who can analyze descriptive statistics?

• What information do we need to answer the learning question? Do we want to collect information from administrative data, surveys, or interviews and focus groups? Do we already collect any relevant information? If not, how would we collect it?

Exhibit 3. Initial learning questions

Learning question What do you want to know?	Rationale for your question Why is this a priority question?	Rapid-cycle learning methods How could we answer this question?	Data and information What information do you need to answer the question? Are you tracking this information already or will you need to collect it?
Example: Do new virtual case management strategies identified in our roadmap for change help case managers build quality relationships with clients?	Case managers have been delivering services virtually and we are considering sustaining this option post-pandemic to enhance accessibility. Relationship building has been identified as a challenge area and we are interested in testing new virtual strategies.	Talk to case managers about what works well and what doesn’t with the new strategies. Talk to clients about their experiences in sessions. Administer a client survey about satisfaction, trust, and relationship quality after experiencing the new strategies.	Interviews with staff and clients (new information to collect) Client survey data (new information to collect) Program administrative data to assess client engagement with sessions (currently available through MIS records)

References

Atukpawu-Tipton, G., and M. Poes. “Rapid Cycle Evaluation at a Glance.” OPRE Report #2020-152,produced by James Bell Associate. Washington, DC: U.S. Department of Health and Human Services, Office of Planning, Research, and Evaluation, 2020.

Cody, S., and M. Arbour. “Rapid Learning: Methods to Examine and Improve Social Programs.” OPRE Report #2019-86. Washington, DC: U.S. Department of Health and Human Services, Office of Planning, Research, and Evaluation, Administration for Children and Families, 2019.

Derr, M., A. Person, and J. McCay. “Learn, Innovate, Improve (LI²): Enhancing Programs and Improving Lives.” OPRE Report 2017-108. Washington, DC: U.S. Department of Health and Human Services, Office of Planning, Research, and Evaluation, Administration for Children and Families, 2017.

09/12/23 Mathematica^® Inc. 2