Financial Education Program Evaluation Supporting Statements A & B with appendices FINAL

CONSUMER FINANCIAL PROTECTION BUREAU
INFORMATION COLLECTION REQUEST – SUPPORTING STATEMENT
CLEARANCE FOR FINANCIAL EDUCATION PROGRAM EVALUATION
(OMB CONTROL NUMBER: 3170-XXXX)

TERMS OF CLEARANCE: None.

A. JUSTIFICATION
1. Circumstances Necessitating the Data Collection
The Dodd-Frank Wall Street Reform and Consumer Protection Act of 2010 (Pub. L. 111203) (the Dodd-Frank Act) established the Consumer Financial Protection Bureau (the CFPB) to
regulate the offering and provision of consumer financial products or services under federal
consumer financial laws. The Dodd-Frank Act also established the Office of Financial Education
(OFE) within the CFPB. The Dodd-Frank Act mandates that “the Office of Financial Education
shall develop and implement a strategy to improve the financial literacy of consumers that
includes measurable goals and objectives, in consultation with the Financial Literacy and
Education Commission, consistent with the National Strategy for Financial Literacy.” It is also
responsible, with the CFPB’s Office of Research, for conducting “research related to consumer
financial education and counseling.” Focusing on Goal 4 of the National Strategy for Financial
Literacy 2011, which encourages “research on financial literacy and education strategies that
affect consumer behavior,” and identification of “customized programs that address local and
other needs,” OFE is responsible for developing and implementing initiatives intended to educate
and empower consumers to make better informed financial decisions.
To develop and implement initiatives that encourage and enable informed financial decision
making, it is first important to understand which elements of financial education programs
contribute effectively and efficiently to improved financial decision-making. There is a growing
realization among experts in the field of financial education that there is not enough evaluationbased evidence to indicate which financial education strategies are most effective. According to a
recent Government Accountability Office (GAO) report on financial literacy: “Relatively few
evidence-based evaluations of financial literacy programs have been conducted, limiting what is
known about which specific methods and strategies are most effective.”1 Without adequate
evidence of what works, it is difficult to improve the effectiveness and quality of financial
education, and therefore improve consumer outcomes.
To address this need, the CFPB has contracted with the Urban Institute (UI) to engage in a
rigorous quantitative evaluation of two financial coaching programs to increase its understanding
of which interventions can improve consumers’ financial decision-making skills. The CFPB has
selected the Financial Clinic (the Clinic) and South Florida Urban Ministries (SFLUM), two
community-based financial coaching providers, for the evaluation.2 The evaluation will focus on
1

United States Government Accountable Office. 2011. Financial Literacy: A Federal Certification Process for
Providers Would Pose Challenges. GAO-11-614.
2

The CFPB selected the two test sites based on: (1) their preliminary evidence of program effectiveness, including
length of history and consistency of programming; (2) program staff capacity and willingness to participate in a
rigorous evaluation; (3) program size, demand, and suitability for impact evaluation; (4) the relative similarity of the
two programs’ approaches being useful for meta-analysis; and, (5) the relative diversity of the two programs’
geographic context and target population.

1

the extent to which the selected sites’ financial coaching interventions increase household nonretirement savings and reduce financial distress among program participants. The costs of
providing coaching services are part of the two organizations’ normal business and are not
funded through this evaluation. The two programs are currently active providers of financial
coaching and the proposed data collection does not incentivize them to modify their typical
service model.
The proposed data collection described in this document will provide necessary information
for an independent, statistically unbiased evaluation of the financial coaching programs provided
by each site. Currently, each of these programs tracks a limited set of participant outcomes using
data collected by program staff. Such data do not permit evaluators to determine the independent
impact of financial coaching on participant outcomes, however, because these self-selected
participants may have been motivated to improve their financial outcomes even without
receiving financial coaching.
The primary method for evaluating the programs’ effectiveness at improving financial
decision making will be a randomized control trial (RCT) experiment, where evaluation
participants will be randomly assigned to treatment and control groups, and the impact of the
financial coaching will be measured by differences in financial outcomes between the two
groups. By also collecting data on individuals who do not receive financial coaching services
during the evaluation, the UI research team can be confident that any improvement in the
treatment group’s outcomes beyond that of the control group can be directly attributed to
participation in the specific interventions, providing the CFPB with an empirical foundation for
what strategies are effective for improving outcomes.
2. Use of the Information
As discussed above, the overall purpose of UI’s data collection activities described in this
document is to inform financial education programs by statistically analyzing, via a RCT
experiment, the effectiveness of the financial coaching services provided by two communitybased organizations selected by the CFPB for evaluation. This evaluation will help to fill the gap
between theoretical research and programs that work in practice by providing solid statistical
evidence of which interventions can improve consumers’ financial decision-making skills. While
impacts from two programs cannot be assumed to generalize to the universe of financial
coaching interventions, this evaluation will provide much needed information about the impacts
generated by the selected interventions.
The interventions UI will evaluate are the financial coaching programs offered by the Clinic
in New York City, New York, and SFLUM in Miami, Florida. The Clinic serves a primarily lowincome population at 24 sites, most of which are housed by partner nonprofit organizations,
across New York City and Newark, New Jersey. SFLUM serves low-income clients through its

2

direct recruitment efforts, but also targets the broader Miami-area population through its
workplace partnerships, including the Miami-Dade County government and the Miami-Dade
County Public School district. The Clinic and SFLUM are headquartered in New York City and
Miami respectively, but their service areas, and the service areas that will be used for the study,
cover the broader New York City and Miami Dade county regions that extend beyond each city’s
limit.
The information gathered from these interventions will be used by UI to conduct an analysis
of the effectiveness of the financial coaching services provided by the Clinic and SFLUM.
Specifically, UI will test whether the programs:






Improve households’ balance sheets;
Reduce household financial stress;
Improve household financial behavior and decision-making;
Facilitate progress towards individual financial goals; and
Increase individual financial knowledge.

As discussed above, the primary method of evaluating the programs’ effectiveness at
improving financial decision making will be a RCT experiment, where evaluation participants
will be randomly assigned to treatment and control groups. Subjects in the treatment group will
immediately receive financial coaching, and subjects in the control group will be delayed access
to financial coaching services for the duration of the evaluation. The impact of the financial
coaching will be measured by differences in outcomes between the two groups.
UI will use administrative data collected by the program sites to assess whether the
randomization process has occurred properly, and to identify any problems that may initially
arise in implementing the randomization. However, it will still be necessary for UI to collect
additional data on evaluation treatment and control participants, as the programs do not already
conduct RCT analyses and have not done so in the past. Specifically, clearance is requested for
the following data collection efforts:


A one-time closed-ended application survey of 1,000 evaluation participants (500 at each
site – 250 each in the treatment and control groups) administered in writing at the
beginning of the evaluation; and



A one-time closed-ended outcome survey of the same 1000 evaluation participants,
administered over the phone by interviewers using computer-assisted personal
interviewing software.

UI will also obtain credit reports for the evaluation participants via a third-party provider,
Credit Builders Alliance, and rely on the two programs’ administrative data about clients who
receive coaching services. The credit reports will be obtained electronically by Credit Builders
Alliance directly from TransUnion, a commercial credit reporting agency, and the provider will
be compensated by UI for this service. Obtaining credit reports will not impose additional data
collection burden on evaluation participants.
3

The application survey and outcome survey are included as appendices to this document. The
application survey will be administered by the financial coaches and program staff on-site, while
the outcome survey will be administered by a survey firm on behalf of UI. Table 1 provides a
summary of the questions on each one-time survey that address each of the outcome areas of
interest—household balance sheet, financial stress, financial behavior, financial goals, and
financial knowledge. The remainder of this section provides more detail on how each of these
data sources will be collected, analyzed, and shared as a part of the final report.
Table 1: Evaluation Data – Outcome Measures
Data Measure/Outcome
Contact information for client and
three contacts
Social security number*
Gender, race*, citizenship*,
birthdate, household composition,
marital status, employment status
Financial dependents
History with organization
Reason for seeking service*
Past use of financial services

Source
Purpose / Uses
Application survey Contact participants

Income sources and amount

Application,
Outcome survey

Recent change in
income/expenses*
Bank accounts and balance

Outcome survey

Use of direct deposit

Use of automatic savings

Retirement savings (type of
account(s) and balance)
Financial assets (type of accounts
and balance)
Number of and balance on credit
cards

Application survey
Application survey
and/or outcome
survey
Outcome survey
Application survey
Application survey
Application survey

Application
survey, Outcome
survey
Application
survey, Outcome
survey
Application
survey, Outcome
survey
Outcome survey

Credit report retrieval
Demographics, Check
randomization
Demographics, Context
Context
Context, Check randomization
Research question on financial
behavior, Context, Check
randomization
Research question on household
balance sheet, Check
randomization
Research question on financial
stress, Context
Research question on household
balance sheet, Check
randomization
Research question on financial
behavior
Research question on financial
behavior

Research question on financial
behavior, RQ on household
balance sheet
Outcome survey
Research question on household
balance sheet
Application survey RQ on financial behavior, RQ on
(number only)
household balance sheet
Outcome survey,
Credit Report
4

Household debts

Outcome survey

Mortgage information

Outcome survey

Foreclosure information*

Outcome survey

Information on who handles
household‘s finances
Timeliness of bill paying

Outcome survey

Method of bill paying

Application
survey, Outcome
survey, Credit
Report
Outcome survey

Use of budget

Outcome survey

Relationship of spending to
income

Outcome survey

Frequency of withdrawal from
accounts
Information on bankruptcy*

Outcome survey

Use of debit card

Outcome survey

Frequency of overdraws

Outcome survey

Frequency of saving

Application
survey, Outcome
survey
Application
survey, Outcome
survey
Outcome survey

Accounts used for savings

Reasons for not using bank
accounts
Bank fees (type and amount)
Use and knowledge of credit
reports and scores
Use of alternative financial
services
Use/existence of emergency fund

Outcome survey

Outcome survey
Application
survey, Outcome
survey
Application
survey, Outcome
survey
Application
survey, Outcome
survey
5

Research question on household
balance sheet
Research question on household
balance sheet
Research question on financial
stress, Context
Research question on financial
behavior
Research question on financial
behavior

Research question on financial
behavior, Context
Research question on financial
behavior
Research question on household
balance sheet
Research question on financial
stress
Research question on financial
stress, Context
Research question on financial
behavior, Context
Research question on financial
behavior
Research question on financial
behavior
Research question on financial
behavior
Research question on financial
behavior
Research question on financial
behavior, Context
Research question in financial
behavior, Research question on
financial knowledge, Context
Research question on financial
behavior
Research question on financial
behavior, RQ on financial stress

ICEF COPE Scale3

Identification of financial goals
and measure of progress toward
goal attainment
Understanding of costs/benefits of
savings options
Understanding of the effect of
timely and in-full credit card
payments
Understanding of the effect of late
payments on credit history and the
ability to obtain future credit
Number of delinquent/derogatory
credit accounts
Total amount past due on credit
accounts
FICO score*

Application
survey, Outcome
survey
Application
survey, Outcome
survey
Outcome survey
Outcome survey

Outcome survey

Research question on financial
stress
Research question on financial
goals
Research question on financial
knowledge
Research question on financial
knowledge
Research question on financial
knowledge

Credit report

Research question on financial
behavior
Credit report
Research question on household
balance sheet
Credit report
Research question on financial
behavior
*Indicates questions/information of a sensitive nature. Additional justification for collecting this
information is provided in Section 11.
Application Survey
The application survey will be administered once to every individual who agrees to
participate in the evaluation, before individuals are randomly assigned to the treatment or control
group. Each program’s financial coaches and program staff will recruit evaluation participants
and administer the application survey on-site. UI and the program directors at each of the sites
will be jointly responsible for training coaching staff in the study recruitment and randomization
procedures. The coaches will introduce the evaluation study using predetermined, programspecific scripts (included as Appendix A) developed by UI with input from each of the sites and
tailored to the specific programs and randomization strategies. These scripts will describe the
evaluation in detail as well as any possible participation risks (such as negative feelings resulting
from discussions of financial difficulties) as determined by the research staff (with input from the
UI Institutional Review Board (IRB)). In addition, the scripts will describe the program
application survey and obtain informed consent to collect information from those who opt in to
the study. The informed consent documents were jointly developed by UI and the programs and
are attached as Appendix A. The informed consent documents will give the research team
3

The InCharge Education Foundation COPE ScaleTM is an instrument constructed to help assess the changes,
advances and progress people make with respect to their financial condition over time. (Information taken from
InCharge website: "InCharge Education Foundation COPE Scale." InCharge Education Foundation. N.p., n.d. Web.
22 June 2012. .)

6

permission to collect and analyze participants’ demographic and financial information
throughout the duration of the study.
The application survey, a condensed version of the outcome survey, collects data on roster
items and clients’ demographic information, as well as information on income, financial account
holdings, financial goals, and financial well-being. The topics covered are necessarily small so as
to minimize the burden on program applicants. Participants in both programs include speakers of
English and Spanish, and the population served by SFLUM also includes speakers of Creole.
Based on information provided by the program staff, UI plans to translate both the application
survey and outcome survey into Spanish to better serve the needs of their target populations.
Creole translation for SFLUM clients will be unnecessary, as the program director informed UI
that their Creole-speaking clients generally speak and read English well. The survey will be
administered via filling out paper survey forms by hand.
The principal purpose of the application survey is to provide baseline information on both the
control group (which will not be in the programs’ administrative database) and the treatment
group. It will also allow UI to confirm that the randomization process was successful—i.e. that
the observable characteristics of the treatment and control groups are statistically equivalent. UI
has also included a few key financial indicators for which the research team wants to observe
how much of an impact the financial coaching has on outcomes relative to the baseline measure,
not only relative to the control group.
Outcome Survey
The outcome survey will be conducted in March 2014 (six months after the final evaluation
participants are enrolled) for all evaluation participants, both treated and controls. Each
participant will be asked to fill out the survey one time. UI is in the process of selecting a survey
firm. UI is considering several highly qualified groups, all of which have successfully collected
survey data via telephone for UI in the past.
The outcome survey is divided into three sections that look at: (1) household
composition and finances, such as financial dependents, income, benefit receipt, financial
account holdings, and assets and liabilities; (2) household financial practices, such as
financial decision making practices in the household, financial habits, bank account usage
patterns, savings behavior, alternative financial services usage patterns, and access to
emergency funds; and (3) financial well-being and capability, using an assessment tool to
gauge attitudes about financial well-being and the progress towards financial goals, as well as
a multiple choice quiz section that gauges financial knowledge. UI will also ask whether
applicants obtained financial coaching or education services from another source during the
evaluation observation period, and if necessary, will adjust for the receipt of other financial
education services in the analysis.Credit Reports and Credit Scores
UI also plans to obtain credit reports and credit scores from TransUnion, a commercial credit
reporting agency, at the same time that the outcome survey is administered. UI will use a
7

nonprofit organization, Credit Builders Alliance, to facilitate its interactions with TransUnion.4
In order to obtain the credit reports, UI will need to ask subjects for their Social Security or Tax
Identification numbers. UI will take all needed precautions with this sensitive information;
please refer to section 11 for a detailed discussion of the security precautions and protections to
be implemented.
Use of Data
UI will produce a report on the findings and conclusions of the evaluation, particularly
focusing on differences between the treatment and control groups on measured outcomes related
to the main research questions of the effect of financial coaching on households’ balance sheets,
financial stress, financial behavior and decision-making; and individual progress towards
financial goals and increased financial knowledge. The report will document the programmatic
elements of each selected intervention and, where possible, link program elements with
participant outcomes. In particular, the report will highlight elements found to be important to
participant success that are common across the selected programs.
The CFPB will have unlimited rights to the data collected as a result of this evaluation and
may, in its sole discretion, disseminate such information to financial education practitioners and
providers, researchers, policymakers, and relevant federal agencies. While impacts from two
programs cannot be assumed to generalize to the universe of financial coaching interventions,
this evaluation will provide much needed information about the impacts generated by the
selected interventions. The CFPB will use UI’s data analysis, findings and conclusions to inform
the CFPB’s work on promoting effective financial education. The research findings will be
shared with other financial education practitioners, policymakers, and other stakeholders to
improve practice and policy around financial education. The Clinic and SFLUM will also use the
study’s findings to inform operations and program design.
3. Use of Information Technology
Outcome survey data will be recorded by subcontractor staff via the survey organization’s
Computer Assisted Telephone Interviewing (CATI) system. This system allows interviewers to
put pre-coded responses directly into an electronic database which reduces errors. It
automatically guides interviewers through skip patterns5, which saves time. This system will

4

Credit Builders Alliance helps institutions serving low and moderate income individuals build client credit and
financial access to grow their businesses and assets. The credit reports will be pulled from Transunion through the
pre-established relationship between TransUnion and Credit Builders Alliance. TransUnion, the third-largest credit
bureau in the United States, offers credit-related information to interested parties.
5

Skip patterns are the order in which survey questions are asked and skipped, and the sequence is used to guide
respondents to certain questions depending on answers given. For example, if a survey respondent indicates that she
is unemployed, the skip pattern will prompt the interviewer to skip questions regarding what kind of job the
respondent has. Having skip patterns ensures that respondents are only asked questions relevant to their situations,

8

ensure that the interview data are accurately recorded and that the outcome survey imposes a
reduced burden on respondents. All interviewers will receive explicit training on the
administration procedures for surveys and interviews to ensure accurate data collection and
recordation with the least possible burden on respondents. The application survey will be paper
and pencil interview [PAPI] administered, which is appropriate given its short length and the
need for the programs to simultaneously administer the survey to a group of applicants in various
sites.
4. Efforts to Identify Duplication
To the CFPB and UI’s knowledge, this will be the first ever evaluation of financial coaching
interventions using an RCT approach to evaluate financial outcomes. UI researchers extensively
reviewed the existing literature on the effects of financial coaching on financial outcomes and
found no studies that were able to implement randomized controlled trials. This finding is in line
with that of the recent GAO report on financial literacy: “Relatively few evidence-based
evaluations of financial literacy programs have been conducted, limiting what is known about
which specific methods and strategies are most effective.”6
UI researchers have worked in tandem with staff from both financial coaching programs to
ensure that the evaluation’s data collection instruments do not duplicate information solicited by
the programs themselves. Each program already administers an intake survey and follow-up
assessments as a part of its coaching services, and this evaluation will use these data in analysis
to the fullest extent possible. The two programs collect this data regardless of participating in this
evaluation. However, there is still a need to conduct application and outcome surveys as part of
the evaluation for two reasons. First, the additional information collected (i.e. the questions
asked, at what point in the evaluation the questions are asked, and the order in which they are
asked) needs to be consistent across the two program sites in order to make the results of the
analysis comparable across sites. Second, in order to implement the RCT analysis, data needs to
be collected from both treatment participants and control participants, and the financial coaching
programs do not have the institutional capacity to collect information from individuals who do
not receive financial coaching services.
5. Efforts to Minimize Burdens on Small Entities
This proposed information collection does not survey small businesses or other entities.

and also increases efficiency by ensuring that no additional time burden is added to respondents by asking them
questions irrelevant to their circumstances.

6

United States Government Accountability Office. 2011. Financial Literacy: A Federal Certification Process for
Providers Would Pose Challenges. GAO-11-614.

9

6. Consequences of Less Frequent Collection and Obstacles to Burden Reduction
Section 1013(d)(2) of the Dodd-Frank Act mandates that the CFPB “develop and implement
a strategy to improve the financial literacy of consumers that includes measurable goals and
objectives…” 12 U.S.C. 5693(d)(2). This project represents the first evaluation performed by
the CFPB of financial coaching programs, and the first ever known evaluation of financial
coaching interventions using an RCT approach to evaluate financial outcomes. The RCT
approach is the most defensible and definitive method of showing whether financial coaching
can change financial outcomes. While the programs collect administrative data from their clients,
they do not collect similar data for non-participants. A failure to collect these data will result in
insufficient information to evaluate the performance of these programs and participants’
resulting financial outcomes using an RCT approach. .
The CFPB will be unable to carry out its mandate if it does not have solid evidence about
what strategies can in fact improve financial literacy. If the CFPB does not have a clear
understanding of what financial education interventions work, its strategies, goals, and objectives
will be without basis. Additionally, section 1013(d)(3)(B) provides that the CFPB “conduct
research related to consumer financial education and counseling.” 12 USC 5493(d)(3)(B).
Identifying and supporting the most rigorous research possible will help the CFPB fulfill its
mandate.
7. Circumstances Requiring Special Information Collection
The proposed data collection activities are consistent with the guidelines set forth in 5 CFR
1320.6 (Controlling Paperwork Burden on the Public—General Information Collection
Guidelines). There are no special circumstances that require deviation from these guidelines.
8. Consultation Outside the Agency
In accordance with the Paperwork Reduction Act of 1995, a 60-day Notice was published in
the Federal Register on January 19, 2012 (77 FR 2684) announcing the CFPB’s intention to
conduct a potential collection of information on this effort and solicit public comment on such
collection. . No public comments were received in response to the Notice. A copy of the Notice
is attached as Appendix D of this document.
UI solicited comments from the two evaluation sites on the research plan and survey
instruments. Both sites provided detailed feedback, which was incorporated into the research
design, application form, and follow-up survey. UI also conducted pre-tests of the application
and outcome surveys, by administering the surveys to four clients currently receiving financial
coaching services. The objectives of the pre-tests were to: (a) test the application for wording,
flow, and meaning; (b) determine the average time to complete the survey; and (c) allow staff to
discuss the application with respondents to better understand their interpretation of the questions.
The pre-test experience was used to modify the instruments as well as the procedures related to
10

contacting potential respondents, scheduling interview time, explaining the survey purpose, and
encouraging participation.
9. Payments or Gifts to Respondents
Neither UI nor the programs will provide any incentives to participants for enrolling in the
study or completing the application survey. Respondents for the outcomes survey under this
clearance will receive a small token of appreciation This practice has proven necessary and
effective in recruiting respondents to participate in this small-scale research and is also employed
by other Federal cognitive laboratories. The incentive for participating in the outcome survey
will be a $30 gift card. UI and the programs may provide smaller incentives than these amounts
at its discretion; however, any requests for larger amounts will be justified in writing to the
Office of Management and Budget (OMB).
10. Assurances of Privacy
Per Section 60 of OMB’s January 20, 2006 publication “Guidance on Agency Survey and
Statistical Information Collections,” and the Office of Personnel Management’s System of
Records Notice (SORN) Guide published April 22, 20107, the CFPB has specified in its contract
with the data collection contractor UI that the CFPB does not own and cannot receive personal
identifying information8 in order to protect the privacy of respondent data and ensure the pledges
of privacy made to clients by the UI research team.
UI is committed to securely collecting and using the private collected under this evaluation
(i.e. application data, outcomes survey data, and administrative data). UI maintains a permanent
Institutional Review Board (IRB), which reviews proposed collections of potentially sensitive or
private data and associated data security measures.9 To receive IRB approval for a study, the data
collection effort must adhere to the following principles:
7

Document available at http://www.opm.gov/privacy/SORNGuide.pdf.
The contract language includes the following: “Contractor shall not provide CFPB any information that could
reasonably be used to identify individuals participating in the project. “
9
The IRB requires that the project’s data security measures ensure that: (1) Data will only be collected about
individuals for whom a signed active informed consent has been obtained. The consent procedures include
assurances that requests for information are not mandatory and that the data will be used only for research purposes;
(2) Access to privacy data will be limited to research team members who need access for evaluation purposes and
who have agreed in writing to maintain the privacy of all data; (3) Adequate precautions will be taken to ensure
administrative and physical security of personally identifiable information; (4) Project findings and reports prepared
for dissemination will not contain information that could reasonably be used to identify an individual. Both
electronic files and hard copy data materials will be secured at UI. De-identified privacy electronic files will be
stored on a confidential network drive. Access to the confidential drive will be limited to researchers who have
signed the project’s pledge of privacy and who need access to conduct analyses. Upon receipt of identifying data
(name, date of birth, and social security number), it will be kept on an encrypted local drive. Hard copy materials
(completed application interview instruments, contact locator information forms, program records, and electronic
CDs which link this information) containing identifying information will be secured on an encrypted medium in
8

11



Subjects are informed of the nature of the research and how it will be used, and their
consent is either obtained or explicitly waived, where risks to them are determined to be
minimal.



Adequate provision is made to protect the privacy of subjects and to maintain the privacy
of data that are collected, where promised and as appropriate.



Risks to subjects from the research are reasonable in relation to anticipated benefits, and
are minimized to the extent possible.



The selection of research subjects is as equitable as possible (the burdens and benefits of
the research are fairly distributed), with particular attention paid to research involving
vulnerable populations and protected health information.

For this evaluation, procedures to secure private data are in place for all data to be collected
(see Footnote 9 for details). Any reports produced under this evaluation will not identify
individual respondents by name or with any personal identifying information from the
application survey or program administrative data, such as date of birth or Social Security
number. For all report and presentation purposes (e.g. draft reports, final reports, and any
presentations of the report findings), primary data that are collected for the evaluation will be
aggregated to a level that will not result in the disclosure of the identities of those who provided
the information. As specified in the contract terms described above, UI will not share personal
identifying information with the CFPB.
The introductory scripts and survey instruments for telephone discussions will assure
potential respondents that the information gathered in the application and outcomes survey will
be combined with information from all other participants, will not be reported or shared in a way
that would allow anyone to link what they tell the research team with who they are, and that no
personal identifying information will be shared with anyone outside of the research team or at
the CFPB.10 Within UI, identifying information for respondents will only be shared with
evaluation staff who need the information for research purposes and who have signed a “Data
Privacy Pledge” (Attachment D). The Data Privacy Pledge is a document developed by UI which
locked cabinets in the project Research Assistant’s office. Materials containing respondent names will always be
stored separately from completed survey and interview instruments and separately still from clients’ social security
numbers to preclude the inadvertent linking of private or sensitive information to an individual research subject, and
ensure that data cannot be re-identified. To protect the privacy of research subjects, the data files submitted to the
CFPB at the conclusion of the project will be “sanitized” to remove any information that could be used to identify a
particular individual. All hard copy data files and materials containing personal identifiers will be destroyed within
six months of project completion.

10

All data will first undergo standard non-disclosure testing to ensure that individuals cannot be identified. This may
include suppressing certain fields altogether if necessary, or modifying data (for example through top-coding
responses) to maintain privacy.

12

stipulates the data security and privacy procedures which must be followed by all those
participating in the evaluation. All evaluation staff and consultants have signed this pledge.
Hard-copy materials containing personal identifying information will be locked up when not in
use, and electronic materials will be stored on a secured server in password-protected computers
and/or encrypted.
11. Justification for Sensitive Questions
Some of the information collected under this evaluation is of a sensitive nature because it
addresses matters commonly considered private, whose answers, if made public, could cause
physical, mental, emotional, economic, or other harm to an individual. The questions pertain to
study participants’ race, citizenship status, Social Security number, FICO score, and questions
related to recent household financial distress including the individual’s reason for seeking
financial coaching, recent changes in income and expenses, foreclosure, and bankruptcy. It is
necessary to obtain information on respondent race and citizenship status to ensure that the
randomization process was correctly executed and that the treatment and control groups are
demographically similar with regard to factors known to impact financial outcomes. UI must
request participants’ Social Security numbers in order to obtain important outcome data from
their credit reports. Similarly, the questions on financial distress and on personal FICO score are
essential to understanding and assessing the impact of the financial coaching interventions
provided by the two study sites on key financial outcomes.
Because of the sensitive nature of some of the data collected in these three efforts, UI will
obtain informed consent to collect information only from those who choose to participate in the
study. Informed consent documents (included in Appendix A) will be provided before
participants fill out the application survey and again before they participate in the outcome
survey. The informed consent documents were jointly developed by UI and the programs and
will give the research team permission to collect and analyze participants’ demographic and
financial information (including some personal identifying and sensitive information, such as
race/ethnicity, citizenship, Social Security number, credit report, and credit score) throughout the
duration of the study. Using scripts developed jointly by UI and the programs, interviewers will
stress that participation is voluntary, that participants have the right to refuse to answer any
question without consequence, and that their identity will be kept private, with answers only
reported in the aggregate. UI and the survey firm administering the outcome survey will take all
necessary precautions to ensure that the information provided by individuals will remain secure,
as described in Section 10 and Footnote 9 above.
12. Estimated Burden of Information Collection
UI anticipates 1,000 participants will participate in the one-time application survey, which
will be administered by the two program sites. The survey firm will reach out to all 1,000
participants in the study for the one-time outcome survey, with an expected response rate of 80
13

percent. Therefore, UI anticipates that only 800 will complete the outcome survey, which
coincides with the estimate the research team used in its minimum detectable effects
calculations. The average response time for the application survey is estimated at 10 minutes.
This will result in an estimated response burden of 166.7 hours. The average response time for
the outcome survey is 40 minutes, which will result in an estimated total response burden of
533.3 hours (see Table 2). The instruments have been pre-tested in accordance with the PRA to
make sure the wording is clear, and to confirm the length of time it takes to complete the survey.
The total estimated respondent burden is 700 hours.
Table 2: Estimates of Burden Hours

Description
Application Survey
Outcome Survey

No. of
Respondents

No. of
Responses
per
Respondent

Estimated
Total
Responses

Estimated
Hours per
Response

Total
Burden
Hours

1,000

1

1,000

0.167

166.7

800

1

800

0.67

533.3

TOTALS

1600

700 hours

Applicants for the financial coaching programs across the sites are likely to be low-income,
and from similar socioeconomic backgrounds. Table 3 shows median income for clients at each
program location.
Table 3: Median Income for Each Program Site
Average Client Annual
Income
$12,018*

Organization
The Financial Clinic
United Way Center for Financial Stability at South Florida
Urban Ministries

$31,250**

*Based on data collected in 2010, reported in J. Michael Collins, “MoneyUp: A Field Study” Final Report. 2010.
** The median income of the SFLUM applicants (classified by Standard Occupational Classification, SOC, codes)
was estimated using May 2011 Occupational Employment Statistics from the U.S. Department of Labor’s Bureau of
Labor Statistics. For the employer applicants, median income was based on Office and Administrative Support
Positions (major group 43-0000).

The median hourly wages for program applicants was calculated based on annual wages that
were taken from site-specific surveys, or reported by the program. Below are the estimated cost
calculations for respondents:
Financial Clinic:
35hr week x 46 weeks/year = 1,610 hours/year
$12,018/1,610 = $7.46/hour
SFLUM
14

35hr week x 46 weeks/year = 1,610 hours/year
31,250/1,610 = $19.41/hour
Application Survey:
Financial Clinic: $7.46 x 166.7 = $1,243.33
SFLUM: $19.41 x 166.7 = $3,235.00
Outcome Survey:
FC: $7.46 x 533.3 = $3,978.67
SFLUM: $19.41 x 533.3 = $ 10,352.00
$1,243.33 + $3,235.00 + $3,978.67 + $10,352.00 = Total Respondent Costs of $18,809.00

13. Estimated Total Annual Cost Burden to Respondents or Recordkeepers
There are no capital/start-up or ongoing operation/maintenance costs associated with this
information collection.
14. Estimated Cost to the Federal Government
The cost to the CFPB of the contract with UI is $1,102,700 over a 29-month period. There
are no additional costs to the Federal Government.
15. Program Changes or Adjustments
This submission to OMB is a new request for approval; there is no change in burden.
16. Plans for Tabulation, Statistical Analysis, and Publication
Data Analysis and Reporting
UI will use treatment and control subjects’ characteristics at baseline to evaluate the quality
of the randomization process. If the assignment to treatment and control groups has truly
occurred randomly, without any consideration to factors that might be correlated with later
financial outcomes, then the descriptive characteristics should be statistically indistinguishable
between the treatment and control groups. It may prove useful to calculate these statistics in the
interim period after the first month of the participant application and assignment period (before
the end of the enrollment period), to identify any subsamples where assignment may be
nonrandom. The initial application survey will include basic information on demographic
characteristics; current financial situation, including income, assets, and liabilities; alternative
financial services usage patterns, financial goals, and savings behavior. Using these data, the
research team will compare the characteristics of the treatment and control groups to determine
whether statistically significant differences exist between the two groups.
15

The research team will also calculate basic descriptive statistics for evaluation participants as
a whole, for participants by program, and for participants by program partner site. UI will report
average measures, as well as the share of survey respondents answering each question. Using
these descriptive statistics in conjunction with secondary data (e.g. the American Community
Survey), the research team can get a sense of how similar the evaluation participants at each site
are to the population in the programs’ service areas, and to the national population of interest to
the CFPB.
While UI expects that the vast majority of program enrollees will attend at least one financial
coaching session, there may be some who are unable to participate or who lose interest. Focusing
on evaluation participants who were assigned to treatment, UI will calculate the share of program
enrollees who schedule financial coaching and/or workshop sessions, and the share of these
participants who attend any of the sessions that they sign up for. Client attendance data will be
provided by the programs, which collect and record data on the meetings or calls financial
coaches hold with clients as a regular part of their operations.
Since program participants can schedule and attend as many coaching sessions as they
choose, UI will also report the average number of financial coaching sessions (workshops,
coaching, and both) attended, their average length, how many participants attend each number of
sessions, and what the purposes of those meetings were. While the primary research question is
whether the availability of financial coaching has an impact on financial outcomes, the size of
the impact for individual subjects will be affected by the share of the treatment population that
actually receives the services, and the intensity of their participation.
UI also will compare participant characteristics across these groups (participants who never
schedule a session, participants who schedule a session but never attend, and participants who
attend a single, or multiple, sessions). Systematic differences in certain characteristics between
groups may highlight individual factors that make it more or less difficult to follow through and
obtain financial coaching even when the opportunity is provided, or that are correlated with
individual motivation to pursue financial coaching.
Impact Analysis
The primary method for estimating the impact of the availability of financial coaching
services on financial outcomes among interested participants is to compare the average outcomes
for participants who were provided access to these services to the outcomes for those who were
deferred. This is referred to as the intent-to-treat model, because it estimates the effect of the
“intent to treat” – which is the opportunity for subjects in the control group to get the financial
coaching treatment. Assuming that the randomization process is carried out properly, the causal
effects of access to financial coaching services will simply be the differences in each financial
outcome between the treatment and control subjects. The research team will calculate these
differences using data from the outcome survey, and also calculate statistical confidence
intervals for these differences using standard distributional assumptions. Impact will be
16

measured separately for each program. The impact analysis will rely on information collected
through the outcome survey and via credit reports, and will focus on outcome measures detailed
in Table 1.
For a limited set of outcomes, the research team is interested in how program enrollees were
able to make changes in their outcomes relative to the baseline values for these outcomes. For
example, the research team may want to see whether financial coaching doubled the share of
participants with savings accounts, or increased it by a more modest but still statistically
significant amount. UI will measure this subset of outcomes in both the application and outcome
surveys, and compare changes in outcomes in the treatment group to changes in outcomes in the
control group.
Even in an RCT, sampling variation can lead to differences in the average characteristics of
treatment and control participants, particularly in small samples. These differences may then lead
to different outcomes between the two groups that should not be attributed to the effects of
program availability. The intent-to-treat model can be augmented to control for these measured
differences using a regression-based approach. By controlling for these differences, the research
team can be more certain that the measured effects are caused by the availability of financial
coaching services.
Among treatment subjects, some may be unable to or choose not to attend any financial
coaching or workshop sessions. The research team may be interested in the effects of actually
participating in financial coaching, rather than just the effects of being offered access to financial
coaching. In order to estimate this effect, the research team will apply what is known as a
“Bloom Adjustment.”11 In this model, one must assume that being assigned to the treatment
group has no effect for treatment subjects who do not go to any financial coaching sessions.
Under this assumption, the Bloom statistical adjustment can be applied to estimate what is
known as the treatment-on-the-treated effect – the causal effect of attending financial counseling
or workshop sessions on financial outcomes.
Publication
The research team will analyze, integrate, and summarize data and findings from the process
and outcome evaluations in interim and final reports. The final report, which will be made
publically available by the CFPB, will describe the evaluation design and implementation, and
then devote itself largely to the results of the outcome evaluation.
Table 5: Timeline for Evaluation Design, Implementation, and Analysis
Task
Receive OMB approval (allowing 9 weeks for OMB approval)

11

Date
12/30/2012*

Bloom, Howard. “Accounting for No-Shows in Experimental Evaluation Designs.” Evaluation Review, 8 (1984):
225-246.

17

Random assignment and program enrollment (on a rolling
basis), including application survey data collection
Allow for at least three-month treatment for all clients
Allow for at least three-month post-treatment period for
participant outcomes to develop
Complete collection of administrative data and post-treatment
outcome survey
Complete analysis of impact data for surveyed programs

1/3/2013 – 9/30/2013

Complete draft report

8/17/2014

Complete final report

9/20/2014

10/1/2013 – 12/29/2013
12/2/2013 – 3/28/2014
4/1/2014 – 4/30/2014
5/1/2014 – 6/30/2014

* Estimated date of OMB approval. All subsequent dates are based on this estimate.
17. Display of Expiration Date
Any reproduction of the data collection instruments approved by OMB for this collection
will prominently display the OMB control number and expiration date for OMB approval.
Data collection materials will indicate the voluntary nature of the survey, procedures to
ensure the protection of data provided, the OMB approval number for the data collection, and the
date of expiration of the control number.
18. Exceptions to the Certification Requirement
This submission, describing data collection, requests no exceptions to the Certificate for
Paperwork Reduction Act (5 CFR 1320.9).

18

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
1. Respondent Universe and Selection Methods
1.A Respondent Universe
For each of the two evaluation sites, The Financial Clinic in New York, NY, and South
Florida Urban Ministries in Miami, FL, three separate data collection efforts employing
statistical methods will be conducted. The respondent universe for each is as follows:
1. For the application survey, the respondent universe will be 1,000 program applicants
(who consent to participate in the research study and complete the application form), i.e.
those individuals assigned to the treatment and control groups. The application survey
will be administered to evaluation applicants when they sign up to participate in the
study, and it will be administered by the financial coaches and program staff on-site. The
application survey will be translated into Spanish to better serve the needs of their target
populations. Creole translation for SFLUM clients will be unnecessary, as the program
director informed UI that their Creole-speaking clients generally speak and read English
well.
2. For the follow-up/outcome survey, the respondent universe will be 1,000 program
applicants (who consented to participate in the research study and completed the
application form and who consent to complete the outcome survey). An expected 800
applicants will complete the follow-up survey from, an 80 percent response rate. This
survey will be administered at least three months after treatment has ended for the
participants enrolling at the end of the enrollment period. The outcome survey will be
administered by UI’s survey subcontracting partner. The outcome survey will be
translated into Spanish to better serve the needs of their target populations.
3. For credit bureau records, the respondent universe will be all 1,000 program applicants
who consented to participate in the research study, consented to and completed the
application form, and consented to allow the research team to use their credit bureau
records.
As described in more detail below, the data from these groups will allow the research team
to examine the efficacy of (the offer of) financial coaching interventions in improving financial
outcomes and well-being for applicants. The statistical methods will include an impact analysis
that examines differences in outcomes between study applicants offered financial coaching and
those who are not offered coaching for the duration of the study period. The research team will
report the findings from the implementation and outcomes analysis in a final report that will
summarize findings and draw implications for programmatic practice.

19

1.B. Sample Selection
The impact evaluation will not rely on sampling. UI will include the first 500 applicants to
the Clinic and SFLUM programs in the evaluation, which is expected to last from December
2012 through August 2013. At both programs, financial coaching services will be offered as an
“extra” service that potential clients can take up if they so choose. The Clinic participants will be
generated from clients who received assistance with their taxes or a similar financial workshop.
Therefore, financial coaching will be an "extra" service that the tax-time clients can choose to
take-up if interested and all participants in the treatment and the control group will have received
the services that initially drew them to the Clinic. Coaching will be introduced to clients after
their tax or workshop session is finished and if a person is interested in coaching, clients will be
told that there are limited spaces. They will be told that all control group members will be able to
get access to coaching in a year’s time.
SFLUM will be travelling to the clients and presenting program offerings at workplaces.
Financial coaching will be highlighted as an additional service that applicants can take up, but all
employees in these workplaces will be able to receive an introductory workshop hosted by
SFLUM. After the presentation, individuals interested in receiving SFLUM coaching services
will be asked to sign up at that time and provide SFLUM with their contact information. SFLUM
coaches will then follow-up with clients to schedule appointments. Those appointments can be
held at any number of locations based on what is most convenient for the coach and client,
including the client's office, the SFLUM office, or some other available meeting space. The
office-based recruitment strategy ensures that none of the clients in the sample will be those who
sought out SFLUM specifically for financial coaching services by approaching their offices apart
from the workplace recruitment. Due to the large number of employees in workplaces partnering
with SFLUM, it would be impossible for SFLUM to provide coaching to all interested clients.
As such, a comparable number of potential SFLUM clients would need to wait to receive
financial coaching services regardless of the study.
The Clinic research participants recruited from their tax time clients are not expected to differ
significantly from their overall client base. SFLUM has many programs that are targeted towards
different client groups, including employer-based groups. The clients in this study (i.e.
participants recruited from places of employment) will not differ significantly from those in
SFLUM’s other employer-based programs.
As the research team is not sampling applicants, the research team does not anticipate the
need to rely on stratification. To the extent that the research team relies on clustering, it will be
as an analytic tool. The research team will consider the merits of clustering applicants for
analysis after investigating the pool of applicants and their characteristics. (One factor that may
be useful to cluster by, for example, is each individual program office location, as The Financial
Clinic and SFLUM both solicit applicants from a set of different program offices.)
UI has based its target of 250 applicants in each group, treatment and control, for each of the
two evaluation sites on estimates of the potential effects of financial coaching on key financial
20

outcomes and expect that only 80 percent of the applicants (200 clients in each group) will
participate in the outcomes survey due to nonresponse. Generating expected effects of the
financial education relies on knowledge of financial coaching programs and what can reasonably
be expected of them, as well as estimates of what these financial outcomes will be for treatment
and control individuals before the evaluation. There have been few evaluations of comparable
programs to provide good estimates of program effects. Studies of financial education programs,
and education programs linked to incentive products such as Individual Development Accounts
(IDAs), have documented positive effects on financial outcomes.12 However, the majority of
these studies were not RCTs, and the research team might think that the true effect sizes were
smaller than those reported in these studies.
A detailed discussion of minimum detectable effects expected under this evaluation is
presented below in section 2c.
1.C. Expected Response Rates
There are many factors which can influence expected response rates in a given low-income
population. The primary factors are discussed below:
1. Mode. The application survey will be administered universally to all applicants who
consent to participate in the research study. The follow-up outcome survey will be administered
via telephone, a method that is more conducive to response than surveys administered through
mail.
2. Good contact information. Non-response is often linked to non-contact rather than
refusal of the client to participate. This issue is addressed by asking in the required application
survey for the contact information of three individuals who do not live in the same household as
the client. Therefore, should the research team lose contact with a client, the survey firm will
have three independent resources to help locate the participant.
3. Experience of the survey administrator. The survey firm that will administer the
outcome survey will have extensive experience fielding telephone surveys, and will be
recognized for the high response rates it achieves.
4. Survey design. A well-designed survey encourages response—and completion. The
outcome survey has been carefully designed to collect the necessary data needed for the analysis,
12

See Mills, Gregory, William G. Gale, Rhiannon Patterson, Gary Engelhardt, Michael D. Eriksen, and Emil
Apostolov. 2008. “Effects of individual development accounts on asset purchases and saving behavior: Evidence
from a controlled experiment”. Journal of Public Economics, 92(5-6): 1509-1530. For a review of many evaluations,
see Lusardi, Annamaria. 2008. “Household Saving Behavior: The Role of Financial Literacy, Information, and
Financial Education Programs.” Cambridge, MA: National Bureau of Economic Research, NBER Working Paper
13824.

21

while not overburdening the participant. The research team drew from previous surveys which
have been fielded among comparable populations to ensure that the survey will be appropriate
for study participants. The surveys have been pre-tested with current program clients and then
revised on the basis of those tests.
5. Incentives. Providing incentives is helpful in increasing response rates, and the research
team will be providing a $30 incentive in the form of a gift card to study participants taking the
outcome survey as a token of appreciation.
6. Follow-up procedures. UI will subcontract with a survey organization that has extensive
experience generating high response rates, and one that will be diligent in following up with
applicants who agreed to participate in the research study and completed the application survey.
In addition, program staff at both sites will assist with contacting study participants who they
have had contact with for the outcome survey. The survey firm will use the script for the
outcomes survey included in Appendix A.
7. Timeframe for data collection. The timeframe for survey data collection is one month,
which is ambitious, but with sufficient staff at the survey administrator, should be adequate. The
outcome survey will be collected six to 15 months after completing the application survey, so it
is likely that even among applicants assigned to the control group, many will recall the
evaluation study. The outcome survey will be conducted at one point in time for all members of
the treatment and control groups. Clients’ exit from services at both SFLUM and the Clinic is
open-ended. As such, it is not possible to create a uniform time post-treatment to survey clients
(with a comparable time period for the control group). While it would be theoretically possible to
survey both the treatment and control group a set amount of time after their application to the
program in question—i.e. to field an outcome survey on a rolling basis, a survey fielded only
once has two key benefits. One is that it is more cost-effective, and will be logistically simpler to
execute. The different response periods that come from the single collection wave may make it
slightly more difficult to argue a pure link between receipt of financial coaching and reported
outcomes. But analysis of those different response periods may become an interesting research
question in its own right; UI may be able to detect variations in outcomes by the length of the
follow-up period, which is especially interesting because a key research question involves the
extent of detectable behavioral changes.
Evidence from prior field studies show that response rates among low-income populations
often range from below 50 percent to 80 percent.13 Because the data collection strategy has been

13

Mitchell, Susan, Colette Lamothe-Galette, and Frank Potter. “Survey Response Incentives for a Low-Income
Population: What Works?” Mathematica November, 2003. http://www.mathematicampr.com/publications/PDFs/surveyresponse.pdf.
Cantor, David and Patricia Cunningham (2002) "Methods for Obtaining High Response Rates in Telephone
Surveys." in Citro, C., Moffitt, R.A. and M. Ver Ploeg (eds) Studies of Welfare Populations: Data Collection and
Research Issues. National Academy Press: Washington D.C. at http://aspe.dhhs.gov/hsp/welf-res-dataissues02/pdf/02.pdf.

22

designed to preempt many of the challenges outlined above, as well as the additional incentives
that have been included to encourage survey participation, the research team expects the
response rate to be approximately 80 percent.
2. Information Collection Procedures
2.A. Protocols for data collection
The plans for each data collection component are discussed below.

Application survey. The application survey will be uniformly administered to all 1,000
individuals who agree to participate in the evaluation, before individuals are randomly assigned
to the treatment or control group. Program staff will administer the survey in English or Spanish,
depending on the participant’s preference, and it should require an estimated 10 minutes for the
client to complete on his or her own. The application is a condensed version of the post-program
outcome survey that collects data on roster items and client demographic information, as well as
information on income, accounts held, goals, and financial well-being. The topics covered are
necessarily small so as to minimize the burden on program applicants. The principal purpose of
the application survey is to provide baseline information on both the control group (which will
not be in the programs’ administrative database) and the treatment group. It will also allow UI to
confirm that the randomization process was successful—i.e. that the observable characteristics of
the treatment and control groups are statistically equivalent.
Outcome survey. UI and its survey subcontractor will field an outcome survey to all
program applicants, both treated and controls. The survey will be administered over the
telephone and will take about 40 minutes. Depending on the respondent’s preference, the survey
will be administered in English or Spanish. The one-time outcome survey is a compilation of
field-tested and custom-designed questions that target the programs’ goals of improving the
household balance sheet, reducing financial stress, improving financial behavior and decisionmaking, increasing financial knowledge, and assisting individuals in achieving their financial
goals. The survey uses a range of questions that approach each of these goals from a variety of
angles so that the research team is able to have as thorough of an assessment as possible of each
applicant’s status and progress. Credit reports. UI plans to obtain credit reports and credit
scores from a commercial provider at the same time that the outcome survey is administered,
using social security numbers obtained through research consent. UI will take all needed
precautions to ensure data security as the research team collects and uses this sensitive
information, including storing the information on a secure drive at UI. The commercial provider
will also make a site visit to UI to ensure that the data are being secured appropriately.

Beebe, Timothy J., Michael E. Davern, Donna D. McAlpine, Kathleen Thiede Call and Todd H. Rockwood
(2005). “Increasing Response Rates in a Survey of Medicaid Enrollees: The Effect of a Prepaid Monetary Incentive
and Mixed Modes (Mail and Telephone)”Medical Care , Vol. 43, No. 4 (Apr., 2005), pp. 411-414 at
http://www.jstor.org/stable/3768443.

23

2.B. Statistical Methodology for Sample Selection

UI will include the first 500 study applicants to the Clinic and SFLUM programs in the
evaluation. Study enrollment is expected to last from December 2012 through August 2013. As
the research team is not sampling applicants, the research team does not anticipate the need to
rely on stratification. The research team will consider the merits of clustering applicants for
analysis after investigating the pool of applicants and their characteristics.
2.C. Estimation Procedures

The research design for evaluating the effects of the financial coaching programs on
participant outcomes is a RCT experiment. In randomized controlled trials, potential program
enrollees are randomly assigned either to participate in the program, or not to participate in the
program, for the duration of the treatment period. The impact of financial education that the
Urban Institute is able to measure in this type of RCT experiment is the effect of the availability
of financial education (in this case financial coaching) for individuals who are interested in
pursuing it, compared to individuals who are also interested in financial education but do not
receive it. If the randomization process is carried out correctly, the baseline financial
characteristics and outcomes of the treatment and control groups should be identical, and the
estimated effects of the program interventions on the outcomes of interest are simply differences
between outcomes for the treatment and control groups.
2.D. Degree of accuracy needed for the purposes described in the justification

Table 1 below presents the minimum detectable effects of financial coaching on financial
outcomes, choosing an example financial outcome measure for each of the research questions,
and where available, in the programs’ existing administrative data. To place these effects in
context, pre-treatment values among currently-enrolled program participants at each of the
evaluation sites are also reported, using existing program administrative data.
Table 1: Minimum Detectable Effects for Key Financial Outcomes
The Financial Clinic
Average
Research Question 1:
Household Balance Sheet
How much owed on credit cards
Research Question 2:
Financial Stress
Use payday loans regularly

Minimum
Detectable Effect

$8,032

-$4,412

2.0%

-2.0%

Use payday loans at least once
in past 3 months
Less than $500 in emergency
savings

78%

-11.1%

24

South Florida Urban
Ministries
Minimum
Average
Detectable
Effect

$2,752

-$1,022

14.0%

-7.4%

Less than $300 in emergency
savings

72.5%

-11.7%

Research Question 3:
Financial Behaviors
Participants with checking
account
Participants with savings
account

47%

12.4%

37.2%

12.3%

30%

11.9%

16.1%

10.1%

Note: One-sided Test where p=.05 and power=.8.

To understand these results, it is helpful to describe the results for one of the outcomes in
detail, as an example. Among the Financial Clinic’s existing clients, 78 percent have less than
$500 in emergency savings when they initially enroll in financial coaching. In order to
statistically detect the effect of financial coaching on the share of participants with less than $500
in emergency savings in this population, with treatment and control groups of 250 participants
each, the financial coaching must reduce the share of participants with less than $500 in
emergency savings by at least 11.1 percentage points (or 14 percent of the participants with low
emergency savings). In other words, in a sample of 200 participants receiving financial coaching,
at least 22 of the 156 participants with low emergency savings must increase their emergency
saving to at least $500 as a result of the coaching, for the research team to be able to statistically
detect an effect of coaching on this level of emergency savings (given the baseline level of
savings among people in this program). Similarly, at SFLUM, 72.5 percent of clients have less
than $300 in emergency savings at baseline, and given the sample design, the minimum
statistically detectable effect of financial coaching on the share of clients with low emergency
savings is 11.7 percentage points.
2.E. Unusual problems requiring specialized sampling procedures/Frequency of data
collection

The research team does not anticipate any unusual problems with collecting these data that
would require specialized sampling procedures. Currently, this evaluation requires three data
collections: the application survey, follow-up survey, and obtaining credit reports. As this is not
an annual data collection, the research team does not plan to develop or modify further data
collection cycles at this point.
3. Methods to Maximize Response Rates and to Deal with Issues of Non-Response
Response rate and follow up.
The application survey will be fielded during study enrollment by the financial coaches at
each site, therefore issues of non-response for the initial application are not likely to be a
concern. Study participants who participate in the telephone outcome survey will receive a
monetary token of appreciation, which is likely to facilitate a high response rate. The survey
administrator UI selects will have an excellent record of high response rates, due mainly to
regular follow-up calls to non-respondents. Furthermore, interviewers at the chosen survey
25

organization will strategically place multiple calls at varying times of the day and days of the
week to non-respondents. If interviewers consistently reach voice mail when calling a number,
they will leave the survey organization’s toll-free number for the study participant to call back. If
necessary, survey interviewers will also follow-up with the personal contacts listed by the client
in the application form to maximize the likelihood of successfully contacting the client and
administering the survey.
The telephone interviewers at the survey organization will have extensive experience
conducting telephone surveys. All interviewers will have in-depth experience using the survey
organization’s CATI system and have undergone training that covers interviewing skills,
techniques, etiquette, instilling confidence in the respondent, and converting refusals to
responses, which allow the research team to ensure that the highest standards are maintained.
Issues of Non-Response
To test for the possibility of non-response bias, non-respondents will be compared to
respondents on data pulled from the application survey with respect to demographic
characteristics such as age, gender, minority status, and income. If differences beyond the
standard error are observed, consideration will be given to applying a non-response weighting
adjustment.
4. Testing of Procedures or Methods
UI researchers have pre-tested the application and outcome surveys, by administering the
surveys to four clients currently receiving financial coaching services. The objectives of the pretests were to: (a) test the application for wording, flow, and meaning; (b) determine the average
time to complete the survey; and (c) allow staff to discuss the application with respondents to
better understand their interpretation of the questions. The pre-test experience was used to
modify the instruments as well as the procedures related to contacting potential respondents,
scheduling interview time, explaining the survey purpose, and encouraging participation.
5. Contact Information for Statistical Aspects of Design


The agency responsible for receiving and approving contract deliverables is:
Office of Financial Education
Consumer Financial Protection Bureau
1700 G Street, NW
Washington, DC 20552
Persons Responsible:
Irene Skricki, Project Officer, [email protected]
Cassandra McConnell, Contracts Officer, [email protected]



The organization responsible for statistical design of data to be collected is:
26

The Urban Institute
2100 M Street, NW
Washington, DC 20037
Persons Responsible:
Dr. Margaret Simms, Principal Investigator, (202) 261-5699, [email protected]
Mr. Brett Theodos, Project Manager, 202-261-5685, [email protected]


The organization responsible for analyzing all data to be collected is:
The Urban Institute
2100 M Street, NW
Washington, DC 20037

Persons Responsible:
Dr. Margaret Simms, Principal Investigator, (202) 261-5699, [email protected]
Mr. Brett Theodos, Project Manager, 202-261-5685, [email protected]

27

APPENDIX A: DATA COLLECTION PROTOCOL SCRIPTS

28

Script for Introducing the Research Study
[The Financial Clinic/SFLUM] is working with the Urban Institute in Washington, DC and
the Consumer Financial Protection Bureau, a federal agency, on a study on the effects of
financial coaching programs on our customers. Your participation is voluntary but is muchappreciated. The report that results from this study will help researchers and policymakers
understand how these programs can better serve individuals like you and communities like yours,
so we hope that you will be willing to participate.
The goal of this study is to understand how financial coaching programs affect financial
outcomes, so we are going to be looking at things such as: what your household income and
expenses look like, what kinds of bank accounts you have, and how you feel about your finances
in general.
Because we have a limited number of financial coaches, we will only be able to provide
services to half of the study participants right now, and the other half will be able to receive
services once the study period has ended in about a year. To make it fair, we will randomly
decide who receives treatment now, and who will receive it later. In other words, you have about
an equal chance of getting services now or getting them later.
Those receiving services now will meet with the [The Financial Clinic/SFLUM]’s financial
coaches and work together to build your savings, improve banking and credit files, reduce debt
and discuss taxes. We will also be pulling credit reports of study participants to help us better
understand the financial situation of individuals who are receiving services at [The Financial
Clinic/SFLUM].
Everyone who is in the study, whether you get services now or not, will be asked to complete
our application form today, and a more in-depth outcomes survey over the phone at the end of
2013. We cannot stress how important this follow-up survey is to the study. We will also be
providing a $30 dollar gift card to you after you complete the outcomes survey in 2014.
We want to assure you that the information that we collect in the application form, from the
credit report, and in the later outcomes survey will be kept private. This information will be
combined with information from all other study participants and will not be reported or shared in
a way that would allow anyone to link what you tell us with who you are. We will not publish
your name or other personal identifying information in anything we write or talk about. We also
will not share information that identifies you, including your name or information from your
credit report, with anyone outside of the research team, including the Consumer Financial
Protection Bureau. All of your answers will be stored in databases with secured password
protection and accessed only by research staff who are committed to ensuring your privacy, and
who have signed data privacy pledges.
An agency may not conduct or sponsor, and a person is not required to respond to, a
collection of information unless the collection of information displays a valid control number
29

assigned by the Office of Management and Budget (OMB). The OMB control number for this
collection is 3170–XXXX. The collection expires on XX/XX/XXXX. Also, a federal law called
the Privacy Act directs how the federal government treats personally identifiable information
contained in your answers to these questions. To understand how and when your personally
identifiable information may be shared, you can read the Privacy Act Statement on the CFPB’s
website at www.consumerfinance.gov and search for CFPB.021 Consumer Education and
Engagement Records. Additionally, the CFPB will treat the information received from you
consistent with its privacy regulations at 12 C.F.R. Part 1070, et seq. We anticipate the
application form taking about 10 minutes of your time, and the follow-up survey taking about 40
minutes of your time.
We have an informed consent document that we would like to review with you and have you
sign to agree to participate in the research study. This document explains the study and your role
as a participant. This also gives you contact information if you have any questions about the
study, and we will make you a copy that you can keep for your own records.
[REVIEW CONSENT FORM, AND SIGN]
Do you have any questions before we begin?
An agency may not conduct or sponsor, and a person is not required to respond to, a collection
of information unless the collection of information displays a valid control number assigned by
the Office of Management and Budget (OMB). The OMB control number for this collection is
3170–XXXX. The collection expires on XX/XX/XXXX.

30

Consent for Participation in Research
Evaluation of [The Financial Clinic/SFLUM]
Introduction/Purpose
[The Financial Clinic/SFLUM] is participating in a national study to understand how
financial coaching programs such as the one provided by this organization can help improve the
financial outcomes for coaching participants. The study is funded by the Consumer Financial
Protection Bureau, the federal agency established in 2010 to protect consumers’ interests. We
would like to have you complete our application form today, and we would like to interview you
by telephone in early 2014.
We want to assure you that the information that we collect in the application form and in the
later outcomes survey will be kept private. This information will be combined with information
from all other study participants and will not be reported or shared in a way that would allow
anyone to link what you tell us with who you are. We will not publish your name or other
personal identifying information in anything we write or talk about. We also will not share
information that identifies you, including your name or information from your credit report, with
anyone outside of the research team, including the Consumer Financial Protection Bureau. All of
your answers will be stored in databases with secured password protection and accessed only by
research staff who are committed to ensuring your privacy, and who have signed data privacy
pledges. Your participation in this study is voluntary.
Benefits of Participating in the Study
We want to know from people like you, who are seeking out financial coaching programs
like [The Financial Clinic/SFLUM]’s coaching program, what your relationship with your
finances is like. Your participation in the study is incredibly important because you can
contribute information from first-hand experience that can help to improve programs for others
in financial distress throughout the country. The report that results from this study will help
researchers and policymakers understand how these programs can better serve individuals like
you and communities like yours. Again, please remember that the information you provide will
be combined with information from all other study participants and will not be reported or shared
in a way that would allow anyone to link what you tell us with who you are. Without the
information we gather through this study, no one can be sure if the services help individuals like
you or how current programs can be improved.
Possible Risks to You of Participating
You should know that we will ask you to talk about your personal financial situation, as well
as circumstances that may have contributed to your desire to seek out financial coaching. This
may remind you of difficult situations or cause negative feelings. If this happens, you should
know that the interviewer will be able to refer you to resources in the community to help you, if
you would like it.
Who Is Conducting this Study?
31

The Urban Institute, a research organization based in Washington, D.C., [The Financial
Clinic/SFLUM], and [Survey Organization] are conducting this study. The principal investigator
is Dr. Margaret Simms of the Urban Institute. If you are willing to be part of the study, you will
complete an application administered by your financial coach, and an interviewer from [Survey
Organization] will call you for an interview in early 2014 for a final telephone survey.
What Will I Be Asked About?
You will be asked about your:







Family members, including your education level and employment status;
Income, including any public benefits ;
Expenses, bills, as well as how you pay your bills;
Kinds of bank accounts you have, and other alternative financial providers you may rely
on for services;
Financial goals you have, and plans and progress you made towards achieving them; and
Kinds of financial stress that you may experience.

Contact Information
You will be asked to provide the study with safe ways to contact you by telephone, either at
your home or through trusted friends and relatives who will know where and how to reach you.
[Survey Organization] will receive this contact information, and will use it only to contact you
for the interviews.
What Other Information Will the Researchers Collect?
If you agree to participate, the research staff will collect information that you give to [The
Financial Clinic/SFLUM] when you apply for its financial coaching program, and that [The
Financial Clinic/SFLUM] stores in its computer databases. We will also ask for your social
security or tax identification number so that we can retrieve your credit report as a part of the
research study. If the interviewer loses touch with you while we are conducting the outcomes
survey, he or she may try and find you by reaching out to the contact individuals you list on the
application form, or the employees at [The Financial Clinic/SFLUM].
Procedure
If you agree to participate, a [The Financial Clinic/SFLUM] financial coach will ask you to
fill out a study application form and will give [Survey Organization] your contact information. In
late 2013, a [Survey Organization] staff member will contact you to schedule a telephone
interview for the final telephone survey. They will not leave messages for you if you ask them
not to, and will follow your instructions for how to reach you safely by telephone. Each
interview should each take about 40 minutes of your time to complete.
Privacy
You should understand that no researcher, financial coach, or interviewer will reveal the fact
that you are participating in this study—no one will know unless you tell them yourself. The
information that we collect in the application form and in the later outcomes survey will be kept
32

private. This information will be combined with information from all other study participants and
will not be reported or shared in a way that would allow anyone to link what you tell us with who
you are. We will not publish your name or other personal identifying information in anything we
write or talk about. We also will not share information that identifies you, including your name
or information from your credit report, with anyone outside of the research team, including the
Consumer Financial Protection Bureau. All of your answers will be stored in databases with
secured password protection and accessed only by research staff who are committed to ensuring
your privacy, and who have signed data privacy pledges.
An agency may not conduct or sponsor, and a person is not required to respond to, a collection of
information unless the collection of information displays a valid control number assigned by the
Office of Management and Budget (OMB).. The OMB control number for this collection is
3170–XXXX. The collection expires on XX/XX/XXXX. Also, a federal law called the Privacy
Act directs how the federal government treats the personally identifiable information contained
in your answers to these questions. To understand how and when your personally identifiable
information may be shared, you can read the Privacy Act Statement on the CFPB’s website at
www.consumerfinance.gov and search for CFPB.021 Consumer Education and Engagement
Records. Additionally, the CFPB will treat the information received from you consistent with its
privacy regulations at 12 C.F.R. Part 1070, et seq. We anticipate the application form taking
about 10 minutes of your time, and the follow-up survey taking about 40 minutes of your time.
Compensation
You will receive a $30.00 gift card as a token of appreciation for participating in the followup survey, which will be given to you after completing the final telephone survey. These funds
are offered as a way to thank you for completing the outcome survey.
Questions
If you have any questions about the study you can contact the Urban Institute research staff at
(202) 261-5574.
Signature of Subject or Legally Authorized Representative
My rights as a study participant have been explained to me and my signature below indicated
that I understand my rights. I am willing to participate in this study of how financial coaching
programs help participants. I have been given a copy of this form. I understand that by agreeing
to participate in this study I grant the research team of [Survey Organization] and the Urban
Institute the following permissions:
Permission to use the contact information I provide only for the purposes of reaching me
for an interview;
Permission to obtain access to my credit report from a third party credit reporting agency;
Permission for [The Financial Clinic/SFLUM] to share information that I give to [The
Financial Clinic/SFLUM] when I apply for the financial coaching program and
information that [The Financial Clinic/SFLUM] uses to keep track of participants using
their services with the Urban Institute and Consumer Financial Protection Bureau;
33

Permission to contact [The Financial Clinic/SFLUM] and the individuals I list as contacts
on the application form, in case the interviewer and I lose touch.

Signature

Date

Printed Name

Signature of Staff

Date (must be same as respondent’s)

Printed name of Staff

34

Script for Completing the Application Survey
At this point, we would like you to complete our program application survey that will help us
understand the characteristics of people who are interested in financial coaching. The survey will
take approximately 10 minutes. Completing the application form is voluntary. It is possible that
you may find some of the survey questions to be personal; however, your information will be
kept private and will help us to understand your financial background.
Please remember that the information that we collect in the application form and in the later
outcomes survey will be kept private. This information will be combined with information from
all other study participants and will not be reported or shared in a way that would allow anyone
to link what you tell us with who you are. We will not publish your name or other personal
identifying information in anything we write or talk about. We also will not share information
that identifies you, including your name or information from your credit report, with anyone
outside of the research team, including the Consumer Financial Protection Bureau. All of your
answers will be stored in databases with secured password protection and accessed only by
research staff who are committed to ensuring your privacy, and who have signed data privacy
pledges.
An agency may not conduct or sponsor, and a person is not required to respond to, a collection of
information unless the collection of information displays a valid control number assigned by the
Office of Management and Budget (OMB). The OMB control number for this collection is
3170–XXXX. The collection expires on XX/XX/XXXX. Also, a federal law called the Privacy
Act directs how the federal government treats the personally identifiable information contained
in your answers to these questions. To understand how and when your personally identifiable
information may be shared, you can read the Privacy Act Statement on the CFPB’s website at
www.consumerfinance.gov and search for CFPB.021 Consumer Education and Engagement
Records. Additionally, the CFPB will treat the information received from you consistent with its
privacy regulations at 12 C.F.R. Part 1070, et seq. We anticipate this application form taking
about 10 minutes of your time.
Also please remember that completing this form is voluntary; you can choose not to answer
any question, and you can stop at any time.
Do you agree to complete the application survey?
[If Yes then continue]
Great, let’s get started. Please fill it out the application survey as accurately as possible, and
feel free to ask me any questions you have while completing it.

35

Script for Completing the Outcome Survey
Hello, my name is [ ______________]. We are calling you to complete a follow-up survey
on behalf of the Urban Institute and [The Financial Clinic/SFLUM], and to arrange for you to
receive your $30gift card that we mentioned when you agreed to participate in this study. I’m
calling from [Survey Organization], a research company in [STATE] that is working for the
research team.
During an initial meeting with [The Financial Clinic/SFLUM] in [Month and Year of
Application], you completed an application survey for a study on the financial coaching
program. You may recall that the application asked a few questions about your financial situation
and your interest in financial services. You may also recall the informed consent document that
you signed at the beginning of the study. We would like to take this opportunity to review this
document, and to answer any questions you may have at this point.
This telephone interview will last about 40 minutes and include questions about your income,
employment, banking history, and financial experiences. The Urban Institute and [The Financial
Clinic/SFLUM] are committed to protecting the privacy of the personal and financial
information that we collect in this survey.
The information that we collect in this outcomes survey will be kept private. This
information will be combined with information from all other study participants and will not be
reported or shared in a way that would allow anyone to link what you tell us with who you are.
We will not publish your name or other personal identifying information in anything we write or
talk about. We also will not share information that identifies you with anyone outside of the
research team, including the Consumer Financial Protection Bureau. All of your answers will be
stored in databases with secured password protection and accessed only by research staff who are
committed to ensuring your privacy, and who have signed data privacy pledges.
As required by federal law, the Office of Management of Budget has approved these
questions under the Paperwork Reduction Act. The OMB control number for this collection is
3170–XXXX. The collection expires on XX/XX/XXXX. Also, a federal law called the Privacy
Act directs how the federal government treats the personally identifiable information contained
in your answers to these questions. To understand how and when your personally identifiable
information may be shared, you can read the Privacy Act Statement on the CFPB’s website at
www.consumerfinance.gov and search for CFPB.021 Consumer Education and Engagement
Records. Additionally, the CFPB will treat the information received from you consistent with its
privacy regulations at 12 C.F.R. Part 1070, et seq. We anticipate this application form taking
about 10 minutes of your time.
Also please remember that completing this form is voluntary; you can choose not to answer
any question, and you can stop at any time.
Is this a good time and place to answer our questions?
36

[If Yes: Continue]
[If No: When would be a good time to reach you when you will be in a setting where you
will feel comfortable answering these questions?]
Do you have any questions before we start?
[If Yes then attempt to answer them]
[If No then continue]
Do you consent to complete the survey?
[If Yes then continue]

37

APPENDIX B: DATA COLLECTION INSTRUMENTS
(included as separate documents: Application Form and Outcome Survey)

38

APPENDIX C: FEDERAL REGISTER NOTICE
(included as separate document)

39