7: CQI Plan

OMB Control No.: 0970-0XXX

Expiration Date: XX/XX/XXXX

CONTINUOUS QUALITY IMPROVEMENT PLAN TEMPLATE

Instructions

Please use this template to describe your continuous quality improvement (CQI) plan for the Office of Family Assistance (OFA). You can either (1) add your text within each section and submit this document as your written plan for CQI (note that each table is fillable), or (2) use these headers and describe your CQI plan in another document. This document is meant to be updated and changed over time as you work on CQI.

1. Grantee and CQI plan information

   Grantee name
   Type of grant (HM, RF-New Pathways, RF-ReFORM)
   Date of CQI plan
   Changes to this version of the CQI plan

2. Summary of CQI work to date (if any)

PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) STATEMENT OF PUBLIC BURDEN: The purpose of this information collection is to support program performance monitoring and program improvement activities for Healthy Marriage and Responsible Fatherhood programs. Public reporting burden for this collection of information is estimated to average 4 hours per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. This is a voluntary collection of information. The answers you give will be kept private. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number. The OMB # is 0970-0XXX and the expiration date is XX/XX/XXXX. If you have any comments on this collection of information, please contact Dr. Mathew Stange at [email protected].

Please summarize past CQI issues that your team has worked on and the current status of those issues in Table B1. If you have not engaged in any CQI, leave this blank. Strategies for improvement should be monitored over time to check if they are still working as intended or need to be revisited.

Table B1. Summary of past CQI issues addressed

Past CQI work	Description
Issue 1	-
Issue addressed
Goal
Data source for monitoring goal
Road test status and results
Progress toward goal
Frequency of monitoring progress
Next steps/Notes
Issue 2	-
Issue addressed
Goal
Data source for monitoring goal
Road test status and results
Progress toward goal
Frequency of monitoring progress
Next steps/Notes
Issue 3	-
Issue addressed
Goal
Data source for monitoring goal
Road test status and results
Progress toward goal
Frequency of monitoring progress
Next steps/Notes
Issue 4	-
Issue addressed
Goal
Data source for monitoring goal
Road test status and results
Progress toward goal
Frequency of monitoring progress
Next steps/Notes
Issue 5	-
Issue addressed
Goal
Data source for monitoring goal
Road test status and results
Progress toward goal
Frequency of monitoring progress
Next steps/Notes

3. Your implementation team

   1. Who is on your implementation team?

Although all staff might be involved in CQI efforts, the implementation team oversees and manages the process. Complete the table below listing implementation team members and their CQI responsibilities. The table includes suggested roles and responsibilities, but please change as needed to reflect your implementation team. Add rows as needed. Once you have filled out the table, this is a great opportunity to reflect on who is on your team, whether you are missing important perspectives, whether responsibilities are well distributed amongst team members, and other factors.

What is an implementation team?

Implementation teams are responsible for actively supporting implementation of the grantee’s HMRF program. An implementation team should include a core group of staff—at least three to five people—who have adequate time dedicated to this role. Team members should have detailed knowledge of the program. An implementation team is not an advisory group, but rather a team that is actively and regularly involved in program implementation.

What are its functions?

The team is comprised of “doers” and “barrier busters” to program implementation challenges. Typically, they:

• Meet regularly (and frequently) to discuss staff and organizational needs of HMRF programs;

• Identify challenges to program implementation using data;

• Develop strategies to address the identified barriers/challenges to program implementation;

• Put into practice a system for CQI;

• Engage the community or communities in support of program implementation.

Table C1. Members of your implementation team¹

Staff name and title	Organization	CQI responsibilities
		Team leader: Organizes and oversees the CQI process
		CQI plan lead: Documents and updates CQI plan (this template) with input from implementation team and others
		Key program staff: Identifies targets, develops improvement strategies, and monitors their implementation and testing
		Data manager: Oversees data collection, analyzes data to measure progress on goals, presents results to implementation team and others
		Training/technical assistance supervisor: Supports staff in implementing new strategies

2. Planning CQI communications

   1. How often does your implementation team meet to discuss CQI? The Administration for Children and Families (ACF) recommends meeting every 2 weeks or monthly. It is also helpful to use a set agenda and a formal meeting structure for team engagement and efficiency.

2. Are meetings usually in person or by phone? If by phone, how do you encourage engagement and full participation?

3. What communications, if any, are expected to occur between meetings? Who is in charge of these communications?

4. Working through an issue

1. Identifying an area for improvement and setting goals

Please identify an area in which your program could improve and goals for improvement. Goals should be SMART: specific, measurable, achievable, relevant, and time bound. Your logic model or other program documents can be useful for developing realistic, meaningful goals. If your program is meeting all goals, CQI can focus on enhancing program implementation.

Please select one area of improvement and then complete the table below describing the specific issue you are trying to solve in that area, one SMART goal, and possible data sources to use. If your program is working on a second issue at the same time, please describe the second issue in section E. Please see the Appendix of “Ideas for using nFORM data for CQI” for ways in which nFORM data could be used to measure progress towards goals in these areas.

1. Please select one area of improvement for your program:

☐ Enrollment

☐ Client completion of services

☐ Quality of services

☐ Data collection

☐ Other

(Please specify, such as staff morale, organizational leadership, teamwork or other important areas related to the success of your program)

2. For the selected area of improvement, please describe the specific problem that you are trying to solve, a related SMART goal, and possible data sources. Examples, shown in blue, are only intended to show the type of relevant information. Grantees are not required to use them.

Table D1. Your program’s issue(s), SMART goal(s), and data source(s)

For the area of improvement your team selected above, please describe the issue you are trying to solve, a related SMART goal, and possible data sources you could use.
What is the specific problem or issue you are trying to solve?
What is the SMART goal that would show an improvement on this issue?
What data source could you use to measure progress towards that goal?

Examples: Issues, SMART goals, and data sources

Example: Enrollment
What is the specific problem or issue you are trying to solve?	Enrollment is below OFA-approved target.
What is the SMART goal that would show an improvement on this issue?	Increase referrals by 25% within the next quarter.
What data source could you use to measure progress towards that goal?	Program-developed spreadsheet to record new partners and referred clients.
Example: Client participation in services
What is the specific problem or issue you are trying to solve?	Workshop participation is low.
What is the SMART goal that would show an improvement on this issue?	Increase attendance at group parenting workshops from 60% to 80% within the next 3 months.
What data source could you use to measure progress towards that goal?	Series Session Attendance operational report from nFORM
Example: Quality of services
What is the specific problem or issue you are trying to solve?	Program is using a new curriculum and fidelity should be assessed.
What is the SMART goal that would show an improvement on this issue?	Conduct three observations of each facilitator and provide one-on-one feedback within the next 6 months.
What data source could you use to measure progress towards that goal?	Fidelity checklist from curriculum developer
Example: Data collection
What is the specific problem or issue you are trying to solve?	Workshop participation data in nFORM are not up-to-date.
What is the SMART goal that would show an improvement on this issue?	Facilitators enter workshop participation data in nFORM within 24 hours of an occurrence.
What data source could you use to measure progress towards that goal?	Number of sessions with a status of “pending attendance” on nFORM’s Sessions screen (screen W7)
Example: Other (staff morale)
What is the specific problem or issue you are trying to solve?	Staff morale is low.
What is the SMART goal that would show an improvement on this issue?	In 3 months, staff survey results show that most staff think the work environment is improving.
What data source could you use to measure progress towards that goal?	Program-developed survey of staff satisfaction with work environment and program leadership

2. Identify improvement strategy

Describe strategies the program will use or is using to improve the area identified in Section D1. You might have more than one strategy for an area of improvement. Also describe how your team developed the strategy and the rationale for the approach. This will help your team and ACF better understand the context for these decisions (and help you create a decision trail for why a decision was made). Please complete the table below.

Table D2. Proposed improvement strategies

Area for improvement (from Section D1)	Strategy for improvement	Process for how strategy was developed	Rationale for the strategy: Why might this strategy lead to improvements?
Example: Low attendance at workshops	Call each family the day before the workshop	Facilitators, case managers, and supervisors brainstormed ways to improve attendance. The team agreed this strategy was relatively easy to implement with high potential for improving attendance.	Checking in will remind families about the upcoming workshop, encourage staff and the client to build rapport, and allow the staff to help troubleshoot if the family has barriers to attendance (such as child care or transportation issues).

3. Conducting a road test: Assessing how well a strategy fits with the program

A road test allows you to examine how well a strategy fits in your program. In a road test, small numbers of staff and clients participate over a short period (about 4 to 6 weeks) and provide feedback about their experiences using the new approach or strategy. Afterward, the team analyzes the data and feedback to develop concrete recommendations for refining or revising the strategy. Road tests often include two or more of these feedback periods.²

Why take the extra steps for a road test?

Change can be difficult and some changes have unintended results. For these reasons, it is important to test the proposed strategy on a small scale and gather feedback on it. This gives the team practical information, such as how staff and clients responded to the change, and whether and how you could improve the strategy.

If a strategy is rolled out without using an incremental process, the program change might ultimately fail because of complications or perceived ineffectiveness, inefficient use of resources, and the potential of contributing to change fatigue among program staff. For more information about testing, see www.mathematica-mpr.com/our-publications-and-findings/publications/using-a-road-test-to-improve-human-services-programs-practice-brief.

Please answer the following questions on how you will test the strategy.³

1. What is the strategy being tested?

2. What are your learning questions for the road test?⁴ For example, if testing calls before each workshop: How do families respond to calls before each workshop? How well are staff able to fit this task into their existing responsibilities?

3. When and how will you implement the strategy?

4. When will you start the test?

5. When will you collect feedback?

6. What type of feedback and information will you collect during the test?
   (Check all that apply)

   Staff feedback through: ☐ Questionnaire ☐ Focus group ☐ Interview ☐ nFORM ☐ Other (please specify)

   Client feedback through: ☐ Questionnaire ☐ Focus group ☐ Interview ☐ nFORM ☐ Other (please specify)

   Others’ feedback: Please specify who and how

7. When will you analyze the data and discuss the results within the implementation team?

8. When will you recommend adjustments to the strategy and/or implementation process?

9. When will you implement adjustments to the strategy?

10. When will you revise the road test strategies?

4. Analyzing the road test results: What worked well, what should change?⁵

After a program has collected feedback from relevant stakeholders such as supervisors, staff, and clients, it is time to analyze and interpret the information. Analysis should seek to identify strengths and challenges of the implementation process as well as opportuni­ties and concrete suggestions for improvement. Please use your data to answer the following questions.

1. What seems to have worked consistently well and not so well? What was inconsistent?

2. What, if anything, was surprising, given expectations about how the new strategy would work?

3. Does the feedback suggest that staff or client attitudes, behaviors, or skills are changing?

4. How might your team build on and use your strengths (aspects that have worked well)?

5. How might we address or resolve the challenges that arose (aspects that have not worked well)?

5. Monitoring improvement over time

In addition to understanding whether a strategy is feasible, you must assess whether it seems to be improving the underlying issue and helping your program achieve its goals. After road testing shows the strategy is a good fit, assess progress toward SMART goals with a small number of staff and clients.

Please note that this does not allow you to determine whether the strategy caused the improvement. Other factors might have led to changes. But this preliminary evidence is useful for determining whether the strategy is promising.

For your SMART goal (Section D1), please describe how you will monitor or are monitoring progress toward the SMART goals.

Table D5. Monitoring progress toward SMART goals

SMART goal (from Section D1)	Strategy for improvement (from Table D2)	Data source(s) to assess progress toward goal	Frequency of monitoring	Staff responsibilities
Example: Increase attendance at group workshops from 60% to 80% within the next 3 months	Call each family the day before the workshop	nFORM service data	Monthly for 3 months before testing another strategy	- Facilitators enter workshop participation data in nFORM - Site administrator will analyze participation rates and inform rest of CQI team
Example: Facilitators enter workshop participation data in nFORM within 24 hours of an occurrence	Create colorful flyers reminding staff to enter data within 24 hours of an event; post different versions in break room and hallways	nFORM workshop participation data	Daily until 100% compliance is achieved	- Facilitators enter workshop participation data in nFORM - Supervisor checks whether data are entered during the next business day - Results are discussed with staff during weekly meetings
Example: In 3 months, staff survey results show that most staff think the work environment is improving	Have monthly staff appreciation lunch: supervisors select one staff person for their outstanding work; at staff lunch, highlight his or her achievements to all staff	Staff survey (to be created by program)	Informal check-ins during weekly staff meetings; staff survey administered in 3 months	-Supervisors solicit feedback during staff meetings -Site administrator works with program director to develop a very short staff survey -Site administrator distributes staff survey; sets up collection box for anonymous responses; analyzes results

6. Analyzing improvement over time: Did we make progress toward our SMART goal?
   

After you have collected data as planned, the next step is analyzing the results. Please answer the questions below.

1. What data source did your team use to assess improvement? If the data source is a survey (either from nFORM or developed by the program), please describe the questions used in the analysis. Generally, the data source will be the same as the source listed in Table 3 above, but if a change was needed, please explain why.

2. When did your team collect data? For example, you might have measured enrollment for 3 months: January 2019 to March 2019.

3. How many people are included in your data? Please specify if the people included are clients, staff, or other stakeholders. For example, this could be the number of clients or staff who answered questions in a survey or number of stakeholders interviewed.

4. How did your team measure improvement or change over time? For example, your team might have measured participation in at least one workshop occurrence for clients who enrolled the month before the strategy was implemented and again 6 months later.

5. What did your results show?

7. Interpreting Data

Discuss your data with your Implementation Team, and, as a team, decide on key summary findings. Begin each statement with the phrase: We learned that…

We learned that

We learned that:

We learned that:

8. Communication

Please describe how you will communicate the CQI efforts to staff. Such efforts include the general use of CQI, soliciting feedback on issues to address, informing staff of changes being tested, and results.

9. Action(s)

With your Implementation Team, decide on next steps based on what you learned. If the strategy is a good fit and has shown intended improvements, you may decide to extend this strategy to other parts of your program, or to the whole program: this is sometimes called “scale-up.” But programs should still continue assessing improvement after scale-up to determine if the strategy is working as intended.

Once a road test leads you to decide to continue to implement the strategy, either with a small portion of the program or to some scaled-up level, make sure the strategy is running smoothly before turning to road test another strategy that affects the same area(s) of the program. Smooth, consistent implementation of the strategy will help you avoid overwhelming staff with changes, especially if you road test another strategy.

Please describe whether you intend to scale-up the strategy, and if so, how and when. If you are not ready to scale-up, what next steps will your program take towards improvement?

Generally, it is best to work on one issue at a time to avoid spreading program resources—including staff time—too thin, and to understand the results of the changes. If you work on two issues at the same time, for example, it can be difficult to figure out which strategies likely led to which improvements. However, your program might need to address a few issues simultaneously. If your program is addressing one issue, as described above, please stop here. You do not need to complete the rest of the template. If you are working on two issues, please complete the rest of the template.

5. Working through a second issue

Although it is usually better to work through one issue at a time, this is not always possible. If your program is working on two issues simultaneously, please describe the second issue in this section. It is best in this situation to think about whether the first issue you are working on overlaps with the second issue, so you can monitor whether changes to Issue #1 might also be changing Issue #2.

1. Identifying an area for improvement and setting goals

   1. Please select one area of improvement for your program:

☐ Enrollment

☐ Client completion of services

☐ Quality of services

☐ Data collection

☐ Other

(Please specify, such as staff morale, organizational leadership, teamwork or other important areas related to the success of your program.)

2. For the selected area of improvement, please describe the specific problem that you are trying to solve, a related SMART goal, and possible data sources.

Table E1. Your program’s issue(s), SMART goal(s), and data source(s)

For the area of improvement your team selected above, please describe the issue you are trying to solve, a related SMART goal, and possible data sources you could use.
What is the specific problem or issue you are trying to solve?
What is the SMART goal that would show an improvement on this issue?
What data source could you use to measure progress towards that goal?

3. Please describe how efforts to improve this challenge might overlap (intentionally or not) with work on the first issue your program is addressing, described in section D. For example, the improvement strategy you use for the second issue might also improve the first issue; then, it will be difficult to determine which change likely led to the improvements.

2. Developing improvement strategies

Describe strategies the program will use or is using to improve the area identified in Section E1.

Table E2. Proposed improvement strategies

Area for improvement (from Section E1)	Strategy for improvement	Process for how strategy was developed	Rationale for the strategy: Why might this strategy lead to improvements?

3. Conducting a road test

   1. What is the strategy being tested?

2. What are your learning questions for the road test?⁶ For example, if testing calls before each workshop: How do families respond to calls before each workshop? How well are staff able to fit this task into their existing responsibilities?

3. When and how will you implement the strategy?

4. When will you start the test?

5. When will you collect feedback?

6. What type of feedback and information will you collect during the test?
   (Check all that apply)

   Staff feedback through: ☐ Questionnaire ☐ Focus group ☐ Interview ☐ nFORM ☐ Other (please specify)

   Client feedback through: ☐ Questionnaire ☐ Focus group ☐ Interview ☐ nFORM ☐ Other (please specify)

   Others’ feedback: Please specify who and how

7. When will you analyze the data and discuss the results within the implementation team?

8. When will you recommend adjustments to the strategy and/or implementation process?

9. When will you implement adjustments to the strategy?

10. When will you revise the road test strategies?

4. Analyzing the road test results: What worked well, what should change?⁷
   

Please use your data to answer the following questions.

1. What seems to have worked consistently well and not so well? What was inconsistent?

2. What, if anything, was surprising, given expectations about how the new strategy would work?

3. Does the feedback suggest that staff or client attitudes, behaviors, or skills are changing?

4. How might your team build on and use your strengths (aspects that have worked well)?

5. How might we address or resolve the challenges that arose (aspects that have not worked well)?

5. Monitoring improvement over time

For your SMART goal (Section E1), please describe how you will monitor or are monitoring progress toward the SMART goals.

Table E5. Monitoring progress toward SMART goals

SMART goal (from Section E1)	Strategy for improvement (from Table E2)	Data source(s) to assess progress toward goal	Frequency of monitoring	Staff responsibilities

6. Analyzing improvement over time: Did we make progress toward our SMART goal?

   1. What data source did your team use to assess improvement?

2. When did your team collect data? For example, you might have measured enrollment for 3 months: January 2019 to March 2019.

3. How many people are included in your data? Please specify if the people included are clients, staff, or other stakeholders. For example, this could be the number of clients or staff who answered questions in a survey or number of stakeholders interviewed.

4. How did your team measure improvement or change over time?

5. What did your results show?

7. Interpreting Data

Discuss your data with your Implementation Team, and, as a team, decide on key summary findings. Begin each statement with the phrase: We learned that …

We learned that

We learned that:

We learned that:

8. Communication

Please describe how you will communicate the CQI efforts to staff. Such efforts include the general use of CQI, soliciting feedback on issues to address, informing staff of changes being tested, and results.

9. Action(s)

Please describe whether you intend to scale-up the strategy, and if so, how and when. If you are not ready to scale-up, what next steps will your program take towards improvement?

Great job describing your CQI plan and progress! CQI is a continuous process, so be sure to update your plan, and consider whether and how to change and re-evaluate your overall approach to CQI over time. Remember to stay curious and keep learning!

1 The team might include more than one key program staff person.

2 The road test is based on the Learn, Innovate, and Improve (LI²) approach. For more information on LI², see www.mathematica-mpr.com/our-publications-and-findings/publications/learn-innovate-improve-li2-enhancing-programs-and-improving-lives.

3 See Learn, Innovate, and Improve (LI²) approach.

4 For more information on learning questions, see www.mathematica-mpr.com/our-publications-and-findings/publications/using-a-road-test-to-improve-human-services-programs-practice-brief.

5 Text and questions from LI² road test brief. For more information, see www.mathematica-mpr.com/our-publications-and-findings/publications/using-a-road-test-to-improve-human-services-programs-practice-brief.

6 For more information on learning questions, see www.mathematica-mpr.com/our-publications-and-findings/publications/using-a-road-test-to-improve-human-services-programs-practice-brief.

7 Text and questions from LI² road test brief. For more information, see www.mathematica-mpr.com/our-publications-and-findings/publications/using-a-road-test-to-improve-human-services-programs-practice-brief.

18