Financial Education Program Evaluation Supporting Statement B FINAL

CONSUMER FINANCIAL PROTECTION BUREAU
INFORMATION COLLECTION REQUEST – SUPPORTING STATEMENT
CLEARANCE FOR FINANCIAL EDUCATION PROGRAM EVALUATION
(OMB CONTROL NUMBER: 3170-XXXX)

TERMS OF CLEARANCE: None.
2. Information Collection Procedures
2.A. Protocols for data collection
The plans for each data collection component are discussed below.

Application survey. The application survey will be uniformly administered to all 1,000
individuals who agree to participate in the evaluation, before individuals are randomly assigned
to the treatment or control group. Program staff will administer the survey in English or Spanish,
depending on the participant’s preference, and it should require an estimated 10 minutes for the
client to complete on his or her own. The application is a condensed version of the post-program
outcome survey that collects data on roster items and client demographic information, as well as
information on income, accounts held, goals, and financial well-being. The topics covered are
necessarily small so as to minimize the burden on program applicants. The principal purpose of
the application survey is to provide baseline information on both the control group (which will
not be in the programs’ administrative database) and the treatment group. It will also allow UI to
confirm that the randomization process was successful—i.e. that the observable characteristics of
the treatment and control groups are statistically equivalent.
Outcome survey. UI and its survey subcontractor will field an outcome survey to all
program applicants, both treated and controls. The survey will be administered over the
telephone and will take about 40 minutes. Depending on the respondent’s preference, the survey
will be administered in English or Spanish. The one-time outcome survey is a compilation of
field-tested and custom-designed questions that target the programs’ goals of improving the
household balance sheet, reducing financial stress, improving financial behavior and decisionmaking, increasing financial knowledge, and assisting individuals in achieving their financial
goals. The survey uses a range of questions that approach each of these goals from a variety of
angles so that the research team is able to have as thorough of an assessment as possible of each
applicant’s status and progress. Credit reports. UI plans to obtain credit reports and credit
scores from a commercial provider at the same time that the outcome survey is administered,
using social security numbers obtained through research consent. UI will take all needed
precautions to ensure data security as the research team collects and uses this sensitive
information, including storing the information on a secure drive at UI. The commercial provider
will also make a site visit to UI to ensure that the data are being secured appropriately.

2.B. Statistical Methodology for Sample Selection

UI will include the first 500 study applicants to the Clinic and SFLUM programs in the
evaluation. Study enrollment is expected to last from December 2012 through August 2013. As
the research team is not sampling applicants, the research team does not anticipate the need to
rely on stratification. The research team will consider the merits of clustering applicants for
analysis after investigating the pool of applicants and their characteristics.
2.C. Estimation Procedures

The research design for evaluating the effects of the financial coaching programs on
participant outcomes is a RCT experiment. In randomized controlled trials, potential program
enrollees are randomly assigned either to participate in the program, or not to participate in the
program, for the duration of the treatment period. The impact of financial education that the
Urban Institute is able to measure in this type of RCT experiment is the effect of the availability
of financial education (in this case financial coaching) for individuals who are interested in
pursuing it, compared to individuals who are also interested in financial education but do not
receive it. If the randomization process is carried out correctly, the baseline financial
characteristics and outcomes of the treatment and control groups should be identical, and the
estimated effects of the program interventions on the outcomes of interest are simply differences
between outcomes for the treatment and control groups.
2.D. Degree of accuracy needed for the purposes described in the justification

Table 1 below presents the minimum detectable effects of financial coaching on financial
outcomes, choosing an example financial outcome measure for each of the research questions,
and where available, in the programs’ existing administrative data. To place these effects in
context, pre-treatment values among currently-enrolled program participants at each of the
evaluation sites are also reported, using existing program administrative data.
Table 1: Minimum Detectable Effects for Key Financial Outcomes
The Financial Clinic
Average

Minimum
Detectable Effect

South Florida Urban
Ministries
Minimum
Average
Detectable
Effect

Research Question 1:
Household Balance Sheet
How much owed on credit cards

$8,032

-$4,412

2.0%

-2.0%

$2,752

-$1,022

14.0%

-7.4%

Research Question 2:
Financial Stress
Use payday loans regularly
Use payday loans at least once
in past 3 months

Less than $500 in emergency
savings
Less than $300 in emergency
savings

78%

-11.1%
72.5%

-11.7%

Research Question 3:
Financial Behaviors
Participants with checking
account
Participants with savings
account

47%

12.4%

37.2%

12.3%

30%

11.9%

16.1%

10.1%

Note: One-sided Test where p=.05 and power=.8.

To understand these results, it is helpful to describe the results for one of the outcomes in
detail, as an example. Among the Financial Clinic’s existing clients, 78 percent have less than
$500 in emergency savings when they initially enroll in financial coaching. In order to
statistically detect the effect of financial coaching on the share of participants with less than $500
in emergency savings in this population, with treatment and control groups of 250 participants
each, the financial coaching must reduce the share of participants with less than $500 in
emergency savings by at least 11.1 percentage points (or 14 percent of the participants with low
emergency savings). In other words, in a sample of 200 participants receiving financial coaching,
at least 22 of the 156 participants with low emergency savings must increase their emergency
saving to at least $500 as a result of the coaching, for the research team to be able to statistically
detect an effect of coaching on this level of emergency savings (given the baseline level of
savings among people in this program). Similarly, at SFLUM, 72.5 percent of clients have less
than $300 in emergency savings at baseline, and given the sample design, the minimum
statistically detectable effect of financial coaching on the share of clients with low emergency
savings is 11.7 percentage points.
2.E. Unusual problems requiring specialized sampling procedures/Frequency of data
collection

The research team does not anticipate any unusual problems with collecting these data that
would require specialized sampling procedures. Currently, this evaluation requires three data
collections: the application survey, follow-up survey, and obtaining credit reports. As this is not
an annual data collection, the research team does not plan to develop or modify further data
collection cycles at this point.
3. Methods to Maximize Response Rates and to Deal with Issues of Non-Response
Response rate and follow up.
The application survey will be fielded during study enrollment by the financial coaches at
each site, therefore issues of non-response for the initial application are not likely to be a
concern. Study participants who participate in the telephone outcome survey will receive a
monetary token of appreciation, which is likely to facilitate a high response rate. The survey

administrator UI selects will have an excellent record of high response rates, due mainly to
regular follow-up calls to non-respondents. Furthermore, interviewers at the chosen survey
organization will strategically place multiple calls at varying times of the day and days of the
week to non-respondents. If interviewers consistently reach voice mail when calling a number,
they will leave the survey organization’s toll-free number for the study participant to call back. If
necessary, survey interviewers will also follow-up with the personal contacts listed by the client
in the application form to maximize the likelihood of successfully contacting the client and
administering the survey.
The telephone interviewers at the survey organization will have extensive experience
conducting telephone surveys. All interviewers will have in-depth experience using the survey
organization’s CATI system and have undergone training that covers interviewing skills,
techniques, etiquette, instilling confidence in the respondent, and converting refusals to
responses, which allow the research team to ensure that the highest standards are maintained.
Issues of Non-Response
To test for the possibility of non-response bias, non-respondents will be compared to
respondents on data pulled from the application survey with respect to demographic
characteristics such as age, gender, minority status, and income. If differences beyond the
standard error are observed, consideration will be given to applying a non-response weighting
adjustment.
4. Testing of Procedures or Methods
UI researchers have pre-tested the application and outcome surveys, by administering the
surveys to four clients currently receiving financial coaching services. The objectives of the pretests were to: (a) test the application for wording, flow, and meaning; (b) determine the average
time to complete the survey; and (c) allow staff to discuss the application with respondents to
better understand their interpretation of the questions. The pre-test experience was used to
modify the instruments as well as the procedures related to contacting potential respondents,
scheduling interview time, explaining the survey purpose, and encouraging participation.
5. Contact Information for Statistical Aspects of Design


The agency responsible for receiving and approving contract deliverables is:
Office of Financial Education
Consumer Financial Protection Bureau
1700 G Street, NW
Washington, DC 20552
Persons Responsible:
Irene Skricki, Project Officer, [email protected]
Cassandra McConnell, Contracts Officer, [email protected]



The organization responsible for statistical design of data to be collected is:
The Urban Institute
2100 M Street, NW
Washington, DC 20037
Persons Responsible:
Dr. Margaret Simms, Principal Investigator, (202) 261-5699, [email protected]
Mr. Brett Theodos, Project Manager, 202-261-5685, [email protected]



The organization responsible for analyzing all data to be collected is:
The Urban Institute
2100 M Street, NW
Washington, DC 20037

Persons Responsible:
Dr. Margaret Simms, Principal Investigator, (202) 261-5699, [email protected]
Mr. Brett Theodos, Project Manager, 202-261-5685, [email protected]