Five Principles Program Review Task Research Plan (1B)

Generic Information Collection Plan for Qualitative Consumer Education, Engagement and Experience Information Collections

Five Principles Program Review Task Research Plan (1B)

OMB: 3170-0036

Document [docx]

Download: docx | pdf

Update to the Five Principles of Financial Education

Subtask 1(b) Program Review

Program Review Plan-Draft

Contract #9531CB20A0005/9531CB21F0090

December 19, 2021

Prepared for:

Consumer Financial

Protection Bureau

Washington, DC

Submitted by:

Abt Associates

Paperwork Reduction Act Statement

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor a collection of information, nor is a person required to respond to a collection of information unless it displays a valid OMB control number. The OMB control number for this collection is 3170-0036. The time burden required to complete this collection of information is estimated to be 10 minutes (survey) and 4 hours (online) per response. Comments regarding this collection of information (e.g comments regarding the time burden per response, suggestions for reducing the time burden per response, and/or suggestions for maximizing the utility of the collected information) should be submitted to the Consumer Financial Protection Bureau at [email protected].

Introduction

Since its inception, the Consumer Financial Protection Bureau (CFPB) has been committed to conducting rigorous research to better understand how financial education can improve the lives of consumers. As part of this effort, in 2017, CFPB published its Five Principles of Effective Financial Education report. to guide financial educators, policymakers, and program developers as they seek to enhance the financial well-being of those they serve. These principles are:

Know the individuals and families to be served;
Provide actionable, relevant, and timely information;
Improve financial skills;
Build on motivation; and
Make it easy to make good decisions and follow-through.¹

Now CFPB seeks to update these five principles to reflect the latest research and knowledge of best practices in the financial education field. In addition, the Bureau seeks to apply an equity lens to the principles, recognizing the implications that social, economic, and cultural factors may have in influencing pathways to financial well-being for different demographic groups.

Under Contract #9531CB20A0005, Abt Associates is conducting a multi-method research effort to inform this update. This work includes: 1) a literature review (Task 1a); 2) a program review (Task 1b); and 3) qualitative input with financial education practitioners (Task 1c). The following two research questions will frame our approach to this work:

Since 2017, what has the field of financial education learned about effective practices and approaches?
Do CFPB’s current principles reflect effective practices across an inclusive universe of programs, practitioners, and client populations?
What principles reflect the updated research and more inclusive program base?

This document describes the plan for Task 1b, conducting a review of financial education programs. The plan provides a roadmap for each of three stages of the program review, including how programs will be identified, the information to be collected, the processes for documenting information collected, and (for stage three) our approach to recruiting and interviewing.²

Approach

The program review (Task 2a) will support the project’s two guiding research questions by: 1) examining effective financial education programs; and 2) exploring specific programs, strategies and interventions that may be particularly appropriate for target populations (e.g., defined by race, ethnicity, gender). The program review will be conducted in three stages.

Stage 1 Initial Program Identification

In the first stage, we will identify 75-100 programs that may provide important insights into effective practices in financial education. These programs will include both both programs that serve the general population and programs targeted at specific populations of interest to CFPB. These programs will include those that serve and target African-Americans, Hispanics, Asian American Pacific Islanders, Native Americans.³ We will also look for programs that are predominantly serving women, either by design or circumstance, though we will also explore gender through its intersection with other characteristics.⁴ We will will catalog high-level information on each program into an excel database.

We will use the following five sources to identify programs in this first stage:

Recommendations from and materials provided by CFPB. We will review CFPB’s 2017 report Effective financial education: Five principles and how to use them⁵ (as well as other reports or materials provided by CFPB) for citations of financial education programs. For example, one citation from Effective financial education: Five principles and how to use them identifies four programs which piloted different methods of assessing financial health for participants.⁶
Referrals from our practitioner advisory board. Members of Abt’s advisory board are aware of many financial education programs, especially those serving Latinx, Native, African-American, Asian Pacific Islander, and low-income families. We will ask this group for suggestions of effective programs.
Programs identified in the literature review. As we review reports and papers as part of the literature review under Task 1a of this contract, we will make a note of particularly interesting or compelling financial education programs that are highlighted for inclusion in this initial program scan.⁷
Abt’s prior environmental scans. Over the past several years, Abt has conducted mutliple projects that included enviromental scans on finanical education resources.⁸ We will review these environmental scans and include relevant financial education programs included the scans.
A web-based search. If the above approaches are insuffient to build a preliminary list of 75-100 diverse programs with at least some level of evidence of effectiveness, we will conduct a targeted internet search to identify additional programs.⁹ We plan to combine key words to create search terms. For example: “financial education” OR “financial coaching” AND [subpopulation of interest]

Development of Database

In the first stage we will collect basic identifying information about each program. Identifying information will include:

Program name
Website or link
Source of identifying the program (e.g., referral, citation, web search)
Geography (e.g., city, region, national, urban, rural)
Population served
- Racial / ethnic groups (e.g., African-Americans, Hispanic, Asian American Pacific Islanders, Native Americans)
- Income groups (e.g., low, moderate, high, all)
- Other groups (e.g., immigrants, women, formerly incarcerated individuals, military)
Targeted services to a particular subpopulation (Yes, no, or unclear).¹⁰
- If yes, which subpopulation(s) are targeted (same options as populations served as above)
Types of services offered (e.g., credit counseling, budgeting, homeownership counseling)
Mode (e.g., in-person, online, group)
Structure (e.g., standalone, embedded, one-on-one, group)
Documented outcomes (e.g., change in credit score, change in savings, reduction in debt)

While it is not feasible to assess the effectiveness of programs (either normatively or comparatively) within the context of this study, we will asssess the level of evidence on effectiveness that is available and assign an “evidence” rating – high, moderate, or low. This rating will aid in selecting effective programs for later stages of review (see Exhibit 1 for evidence level assessment rubric).¹¹

Exhibit 1. Level of Evidence Assessment Rubric:

Type of Evidence	Points Assigned
Expert referral	1
Descriptive evidence of effectiveness (e.g., pre-/post- analysis of outcomes, qualitative evaluations (e.g. client satisfaction surveys)	1
Quasi-experiment/Experiment	3
Sum total number of points for a given program and assign rating: 1 = low 2 = medium 3+ = high

Identifying data and the effectiveness rating will be entered into an excel database, which will be used throughout the program review to build a catalog of programs.

Stage 2 Program Selection

In the second stage, we will conduct a more in-depth review of a subset of programs identified in Stage 1. In collaboration with CFPB, we will select 50-60 diverse programs from the larger Stage 1 program list for further review, using the information collected in Stage 1. We will select programs to meet two primary goals: 1) to identify programs serving or targeting specific subpopulations, and 2) to identify effective programs.We will first group programs serving or targeting similar subpopulations and select approximately half programs based on the populations served or targeted. We anticipate the majority of these programs will serve or target African-American, Hispanic, Asian American Pacific Islander, and Native American communities.¹² To the extent that the programs identified in Stage 1 serve or target more specific populations (such as Mexican immigrants, victims of domestic violence, or military families) we may include additional groups. For the remaining programs, we will select those programs with higher levels of evidence of effectiveness.¹³

Stage 3 Data Collection

For the 50-60 selected programs, we will conduct a web scan to identify published reports and other online materials. After reviewing these materials, we will develop short qualitative narratives that summarize the program’s approach and evidence of its effectiveness. In addition, we will highlight, where relevant, any information that describes how the program is aligned with each of the five principles.These summaries will be added to the existing entries for each program in the excel database.

We will also interview conduct interviews with key program staff (such as program managers or directors) to gain a deeper understanding of the program’s approach and effectiveness.¹⁴ Through interviews with program managers or directors of the selected program, we will collect more detailed information about each program’s approach, overall strategy, and methods of assessing effectiveness. We will also ask about each of CFPB’s five existing principles of effective financial education to get specific feedback on how these principles relate to their programs and any recommendations they have for adjusting or modifying the principles.¹⁵

(See Appendix 1. Program Director/Manager Interview Guide for more details).

Analysis

We will use an approach called “applied thematic analysis” to analyze our interview notes.¹⁶ Following this approach, researchers will review the interview notes in detail and summarize and describe the information collected. Researchers will code the interview notes to identify passages and ideas that align with pre-determined themes as well as develop new codes to identify ideas that emerge from the analysis, such as a new principle suggested by the interviewee. We anticipate organizing our analysis around the following topics, in alignment with the topics and questions in the interview guide:

Topic	Questions	Potential Codes	Potential Themes
Program summary	A.1, A.2, A.3	Population(s) targeted, developed curriculum, modify materials, partner with complementary program	Programs develop specific curriculum to meet the needs of those they target.
Program effectiveness	A.4	Measures, quantitative, qualitative, comparisons	Programs primarily assess effectiveness with respect to quantitative outcomes (e.g., credit scores, savings balances, etc.)
Principles overall	B.1, B.7	Clear, confusing, comprehensive, lacking, broad, applicable	The existing five principles of effective financial education are broadly applicable.
Principle 1	B.2	Important, not important, applicable, not applicable	Principle 1 is particularly important for programs targeting immigrants. Principle 2 should be modified to include “personalized” information since everyone’s needs differ. (etc.)
Principle 2	B.3
Principle 3	B.4
Principle 4	B.5
Principle 5	B.6
Dissemination	C.1, C.2	FinEx, Conference, Colleague	Practitioners primarily learn about CFPB’s financial education materials from being on the FinEx listserv.

We will use NVivo, a qualitative analysis software to aid the coding process. The Abt team will also meet periodically to discuss emerging themes and bringing multiple perspectives to bear on the data in developing the summary of findings.

Report

The team will summarize findings from all three Stages in a written report for CFPB. We propose submitting this report as an annotated outline organized around key findings or themes. We anticipate there will be a section on each of the principles, each of the subpopulations, and any additional themes that emerge. In addition, the Abt team will provide CFPB with the excel database of programs, which will allow CFPB to easily search and sort programs by key programmatic elements in its future work.

Program Director / Manager Recruitment email

Dear [practitioner name],

I’m writing to ask for your help as we are working with the Consumer Financial Protection Bureau (CFPB) to update their 2017 report Effective financial education: Five principles and how to use them. We identified your program through [add how we identified this program] and are particularly interested in talking to you because [add why we selected this program].

We’d like to set up a 60-minute virtual interview with you to discuss your program and CFPB’s five principles. The CFPB has partnered with Abt Associates, an independent research organization, to conduct the interviews. The session will be done over WebEx (an online video call system).

Your insights will help shape CFPB’s understanding of effective financial education practices. Participants typically reflect that they find the process engaging and informative. We hope that you will be able to participate. If you can help us, please email me and share a few times you are available between [dates] for this interview.

Sincerely,

[Contact]

Program Director / Manager Semi-Structured Interview Protocol

Privacy Notice

The information you provide to Abt Associates during this interview will be used by the Consumer Financial Protection Bureau (CFPB) to inform and improve the CFPB Five Principles of Effective Financial Education.

Abt Associates may capture video and audio of your responses as you participate in the interview. Recordings will only be used by the Abt Associates project team for research and will not be shared with the CFPB.

Your feedback will be kept private. Only de-identified and aggregated information will be included in any final or published reports shared with the CFPB. None of your identifying information will be included in the reports.

This collection of information is authorized by Pub. L. No. 111-203, Title X, Sections 1013 and 1022, codified at 12 U.S.C. §§ 5493 and 5512.

Participation is voluntary, and you may withdraw participation at any time.

Hi, I’m [NAME], a researcher from Abt Associates. As you know, Abt is conducting research for the Consumer Financial Protection Bureau (CFPB) to learn more about what makes financial education programs effective, particularly programs serving [insert target population]. We plan to use the information we gather through these interviews to help the Bureau update their Five Principles of Effective Financial Education report and to identify best practices that can be shared with other educators.

This interview should last around an hour. Your participation in the study is voluntary. You don’t have to answer all of the questions, and you can end the interview at any time for any reason. Choosing not to participate will not affect you in any way.

We’ll combine the information you provide us with the information provided by other people we interview. We’ll keep any personal information that we collect—such as your name and affiliation – confidential to the best of our ability. Only authorized researchers will have access to this information.

If you have any questions or concerns about your participation in this study, you may contact Sarah Wolff at Abt Associates. I’ll give you her contact information to write down at the end of the interview.

Before we begin, can you formally confirm that you consent to this interview? Thank you. Also, we’d like to record today’s conversation. The recording will strictly be used as a reference as we develop our report. Is it alright with you that I record this interview? My colleague [ ] is also on the line taking detailed notes. He/She may chime in from time to time with questions.

Do you have any questions before we begin?

Overview of program

I’d like to start our conversation by learning a bit more about your program.

A.1. Can you describe your financial education program?

- Ask interviewee to describe their clients/target population

A.2. One reason we wanted to talk to you want [describe what we found during stage 1 and 2 reviews that was particularly interesting about the program]. Can you tell me more about that program or aspect of your program?

Ask any clarifying questions about the information collected in stages 1 and 2

A.3. Are there any elements about your program or approach that are unique or different than most financial education programs?

Why do you use these elements/approaches?
How, if you at all, do adjust your program to serve different population (i.e., different genders)?

[probe re: gender, race, intersectionality]

A.4. Do you think your program is particularly effective at serving [target population]?

Why do you think its effective? How do you assess your effectiveness?
What elements about your program or approach you think are particularly effective for [target population]?
What do you think you could do to be more effective in serving [target population]?

Thoughts on CFPB’s existing 5 principles

In 2017, CFPB conducted research and published a report on the five principles of effective financial education. They are:

Know the individuals and families to be served
Provide actionable, relevant, and timely information
Improve key financial skills
Build on motivation
Make it easy to make good decisions and follow-through

B.1. Thinking about your work, do these principles resonate with you?

Which ones?
Why or why not?

B.2. Let’s talk more about the first principle, Know the individuals and families to be served.