Metrics Supporting Statement A - 6.3.14 revisions

CONSUMER FINANCIAL PROTECTION BUREAU
INFORMATION COLLECTION REQUEST – SUPPORTING STATEMENT
PART A
DEVELOPMENT OF METRICS TO MEASURE FINANCIAL WELL-BEING OF
WORKING-AGE AND OLDER AMERICAN CONSUMERS
(OMB CONTROL NUMBER: 3170-XXXX)

TERMS OF CLEARANCE: Not applicable. This is a request for a new collection of
information.
ABSTRACT: Under the Dodd-Frank Wall Street Reform and Consumer Protection Act, Public
Law No. 111-203, the Bureau’s Office of Financial Education (“OFE”) is responsible for
developing and implementing a strategy to improve the financial literacy of consumers that
includes measurable goals and initiatives, in consultation with the Financial Literacy and
Education Commission, consistent with the National Strategy for Financial Literacy. In addition,
the Office of Financial Protection for Older Americans (OA) within the CFPB is charged with
conducting research to identify methods and strategies to educate and counsel seniors, and
developing goals for programs that provide seniors with financial literacy and counseling.
The CFPB intends to collect quantitative data through questionnaires with working-age (age 1861) and older American (age 62 and older) consumers in order to develop and refine instruments
that will enable the CFPB to reliably and accurately measure adult consumers’ financial wellbeing and financial ability. The primary anticipated data collection strategy is through internetbased questionnaires. The core objective of the data collection is to iteratively test, refine, and
produce valid and reliable measures of consumer financial well-being and financial ability that
will create a strong, standardized basis for setting measurable goals, and evaluating financial
education strategies and programs. This project focuses on scale development and is not intended
to evaluate substantitive hypotheses, inform policy, or any other activity that falls outside
standard scale (also called metric, item bank, questionnaire, or instrument) development
protocols.

A. JUSTIFICATION
1. Circumstances Necessitating the Data Collection
The Dodd-Frank Wall Street Reform and Consumer Protection Act of 2010 (Pub. L. 111-203)
(“the Dodd-Frank Act” or “the Act”) requires the Consumer Financial Protection Bureau (CFPB)
to regulate the offering and provision of consumer products or services under Federal consumer
financial law. The Act also established the Office of Financial Education (OFE) within the
CFPB, which is responsible for developing and implementing a strategy to improve the financial
literacy of consumers that includes measurable goals and objectives, in consultation with the
Page 1 of 16

Financial Literacy and Education Commission (FLEC) 1, and, together with the CFPB’s Office of
Research, for conducting research related to financial education and counseling. In addition, the
Act established the Office of Financial Protection for Older Americans (OA) within the CFPB,
which is charged with conducting research to identify methods and strategies to educate and
counsel seniors, and developing goals for programs that provide seniors with financial literacy
and counseling.
There are at least three challenges in formulating and setting measurable goals for effective
financial education initiatives. First, there is inadequate evidence regarding which financial
education strategies are most effective. According to a June 2011 Government Accountability
Office (GAO) report on financial literacy, “relatively few evidence-based evaluations of
financial literacy programs have been conducted, limiting what is known about which specific
methods and strategies are most effective. 2” Second, there are no standard or rigorously tested
measures of consumer financial knowledge, behavior, or well-being outcomes that can provide a
gauge to measure the effectiveness of financial education or capability programs, products, or
services. FLEC’s Research & Evaluation Working Group has identified the development of
“‘key metrics’ for financial education/capability, including measures of knowledge, behaviors,
and well-being” as one of its top priority areas for research. 3 Third, there is limited evidence of
the precise role financial knowledge plays in improving financial decisions and financial wellbeing and whether it is relatively more important in this respect than, for example, numeracy,
patience, cognitive abilities or social aspects of financial choices and financial behavior.
In order to help the CFPB, other FLEC agencies, and the broader financial education field
develop and support strategies and programs that lead to better financial outcomes for American
consumers, the CFPB is engaging in a major research project (“this Project” or “the Project”)
focused on making significant progress toward learning:
1. What knowledge and behavior predict financial well-being;
2. What the importance of financial knowledge is relative to other factors (personal
traits and social context) in improving financial well-being; and,
3. How to effectively measure key financial knowledge, behavior, and well-being
concepts.
The products of this Project should allow the CFPB, other FLEC agencies, and the broader
financial education field to develop well-informed approaches to improving consumer financial
1

The Financial Literacy and Education Commission was established under the Fair and Accurate Credit
Transactions Act of 2003. The Commission was tasked to develop a national strategy on financial education. It is
chaired by the Secretary of the Treasury, vice-chaired by the Director of the CFPB, and made up of the heads of 20
additional federal agencies.
2
U.S. Government Accountability Office, GAO-11-614, Financial Literacy: A Federal Certification Process for
Providers Would Pose Challenges (June 28, 2011), available at http://www.gao.gov/assets/330/320203.pdf.
3
Financial Literacy and Education Commission, 2012 Research Priorities and Research Question, available at
http://www.treasury.gov/resource-center/financial-education/Documents/2012%20Research%20Priorities%20%20May%2012.pdf

Page 2 of 16

well-being. By creating psychometrically sound metrics for outcomes of primary interest in
consumer financial research (i.e., financial well-being and financial ability), this project will
facilitate comparisons between programs and interventions and help toconsolidatie the rapidly
expanding and piecemeal consumer financeliterature. The use of these metrics in future projects
will help eliminate idiosyncratic definitions of the constructs and reduce variation due to the use
of different metrics to measure the noted constructs. These metrics will provide a solid
foundation for the development of a body of relevant and actionable knowledge in the field of
consumer finance. Further, by creating and vetting rigorously developed metrics (measures) of
consumer financial well-being and financial ability, the Project will create a strong basis for
evaluating financial education policies and programs, including the ability for direct cross-study
comparisons among research projects that utilize the metrics in the future. More specifically,
these metrics should significantly increase the ability of the CFPB, other FLEC agencies, and
private researchers to undertake a wide range of program and policy effectiveness testing to
determine which approaches make the biggest difference in consumer outcomes.
The CFPB’s Office for Older Americans (OA) will use these metrics to inform the development of
evidence-based goals for financial education and counseling programs that serve seniors, as well as to
inform and/or be used to assess current OA projects and programs, including:
•
Money Smart for Older Adults: A training and educational program with the FDIC to help
seniors protect themselves and others from financial exploitation and fraud, which currently includes an
evaluation questionnaire that assesses the participants’ knowledge and outcomes. The validated metrics
will be used to enhance this questionnaire.
•
Financial Advisers/Financial Caregiver Work: Many Americans rely on someone else – from
professionals to family members- to manage their money. OA has developed a number of reports and
products aimed at this population. Yet, there is very limited research on the financial well-being of those
who rely on someone else to manage their money. A set of validated financial well-being metrics will be
used for future research –internal or by outside partners- to study the relationship between a consumers’
financial well-being and their reliance on someone else to manage their money.
•
Older Consumers and Debt: Research shows that more seniors are carrying debt into their
retirement years, and in larger amounts, than a decade ago. The availability of an alternative and validated
way to measure financial well-being will help us explore whether debt is inherently a barrier to financial
well-being or whether for some seniors holding debt or taking on debt is a way to maintain their financial
well-being.

This Project is anticipated to encompass three phases:
(1) Qualitative research to inform definitions of financial well-being 4 and financial ability for
4

Financial well-being has been defined in various different ways in the literature, mostly depending on the purpose
of the study and the data sources used. The initial phase of the project involved devoping a definition of financial
well-being grounded in consumer insights. By proposing a definition of financial well-being based on input from a
large, diverse sample of consumers, this work is intended to provide a definition and measurement approach that can
facilitate comparisons between financial capability interventions.
Page 3 of 16

both working-age and older Americans, and the development of clear hypotheses
regarding the drivers of financial well-being. This phase has been completed. In addition,
the qualitative research informed the wording of candidate items.
(2) Rigorous development, testing, and refinement of items to measure the constructs
(variables) of financial well-being and financial ability, including preliminary validation
of the developing scales.
(3) A quantitative test of the hypotheses developed in phase one, using instruments
developed during phase two. This phase is not being proposed here and will be examined
in future studies using new rounds of data collection.
The information collected through these processes will increase the CFPB’s understanding of
consumers’ financial experiences and outcomes and therefore what type of financial education
and empowerment programs and practices may improve financial decision-making skills and
outcomes for consumers.
The Office of Management and Budget (OMB) has previously approved information collection
for phase one, the qualitative research, to inform hypotheses about financial well-being and the
wording of items to assess the noted constructs, which has been completed. Through this new
Information Collection Request, “Development of Metrics to Measure Financial Well-being of
Working-Age and Older American Consumers,” the CFPB seeks approval to conduct the second
phase of this project (#2 above): rigorous development, testing, and refinement of items to
measure the constructs (variables) hypothesized in phase one to relate to financial well-being.
2. Use of the Information
In order to support the development of financial well-being metrics, the CFPB has contracted
with the Corporation for Enterprise Development (CFED). CFED, in turn, has subcontracted
with the Center for Financial Security at the University of Wisconsin-Madison (CFS) and Vector
Psychometric Group (VPG), to conduct cognitive interviews and field three rounds of items.
CFED and its subcontractors will work with Survey Sampling International (SSI) to coordinate
fielding of the item pools and item refinements.
Of the approximately 14,800 questionnaires that will be completed by a sample of Americans
ages 18 and over:
 11,300 questionnaires will be fielded online in two rounds of increasing size by Survey
Sampling International. SSI together with another subcontractor will then field a third
round of 3,000 questionnaires using mixed presentation modes (online, telephone and
pencil-paper) in order to compare responses of individual questions across methods of
administration;
 7,800 questionnaires will be completed by working-age Americans (ages 18-61) and
6,500 will be completed by older Americans (age 62 and older).
Page 4 of 16

In addition, before fielding the items (question and answer sets), the team will complete 12
cognitive interviews; 6 with Americans between the ages of 18-61, and 6 with Americans over
the age of 61.

Cognitive
interviews

Round 1
preliminary

Round 1 main

Round 2

12

700

3800

6800
WA
OA
3800 3000

Total
WA
6

OA
6

WA
400

OA
300

WA
2100

OA
1700

Round 3
(mode testing)
3000
WA
1500

OA
1500

Survey Sampling International (SSI) will recruit two online samples: one that includes workingage Americans (ages of 18-61) and another that includes older Americans (age 62 and older).
The sampling design is not intended to result in statistically representative samples of the
American population, however, both the working-age and older American samples will conform
to quotas on key demographic variables to achieve a dataset that is diverse in terms of income,
education, employment status, marital status, age, gender, race/ethnicity, presence/ages of
children, and geography of the participants. While statistically representative samples are not
required for the planned scale refinement, for the analyses to have validity, large samples with
demographic diversity will provide for more stable parameter estimation.
Working-Age American Samples
CFED will conduct 6 one-on-one “think aloud” cognitive interviews in order to guage
respondents’ comprehension of terms included in the items (e.g. do resondents understand the
term “financial goal”?), item intent (e.g. are respondents thinking of “large purchases” from
their perspective, as we’ve intended, or from the perspective of what they believe society at large
feels is a “large purchase”) and appropriate response mapping (e.g. have we included the “does
not apply” option everywhere it is applicable?). Respondents of a variety of ages, income levels
and ethnicities will be recruited in public locations such as laundromats and parks to participate
in the interviews. The interviews will be one-on-one because people may not be willing to share
information regarding their personal finances with a larger group. CFED will pay respondents
$25 for their participation, require informed consent and have the cognitive interview protocol
approved by an IRB. Results of these interviews will be summarized in a memo and they will
inform item development.
Following the cognitive interviews, SSI will administer three rounds of items to working-age
Americans. No respondent will be asked to complete more than one questionnaire. The first
round will consist of a preliminary round of 400 completed questionnaires followed by a main
round of 2,100 completed online questionnaires. The preliminary round of questionnaires will
test whether or not there are any large problems with our item bank. It gives the team the option
to “pull the plug” and undertake significant revisions before proceeding with testing with larger
samples.
Page 5 of 16

The second round will consist of 3,800 completed online questionnaires.
The third round (mode testing) will consist of 1,500 questionnaires: approximately 500
completed online, 500 completed over the phone and 500 completed using a pencil-paper
questionnaire. Survey Sampling International will field both the online and phone questionnaires
during this third round. 350 questionnaires will be administered to persons with landline phones
and 150 questionnaires will be administered to persons with cell phones. The online and phone
samples will have the same general socio-demographic profile as the respondents in the first two
rounds of item testing.
CFED has received bids to conduct the mail (paper-pencil) questionnaires from a mail house that
regularly partners with both SSI clients and the Wyoming Survey and Analysis Center
(WYSAC). Surveys will be mailed to 2,500 addresses of persons who have the same general
socio-demographic profile as the respondents in the first two rounds of item testing; we
anticipate a 20% return rate netting us two samples of roughly 500.
The objective of the third round of data collection is to compare responses of individual
questions across methods of administration and to further refine questions for the final
instruments. In this stage, IRT-based psychometric analysis will focus on response differences by
delivery mode as opposed to any socio-demographic characteristics.
Older Americans Samples
CFED will also conduct 6 cognitive interviews with older Americans. Cognitive interviews with
older Americans will follow the same protocol as the one we described above for the working
age group.
SSI will administer three rounds of questionnaires to older Americans. No respondent will be
asked to complete more than one questionnaire. The first round (which includes the preliminary
round describe ablove) will consist of 2,000 completed online questionnaires (300 of which will
take place during the preliminary round). The second round will consist of 3,000 completed
online questionnaires.
The third round (mode testing) will consist of 1,500 questionnaires: approximately 500
completed online, 500 completed over the phone and 500 completed using a pencil-paper
questionnaire. Survey Sampling International will field both the online and phone questionnaires
during this third round. 350 questionnaires will be administered to persons with landline phones
and 150 questionnaires will be administered to persons with cell phones. The online and phone
samples will have the same general socio-demographic profile as the respondents in the first two
rounds of item testing.
CFED has received bids to conduct the mail (paper-pencil) questionnaires from a mail house that
regularly partners with both SSI clients and the Wyoming Survey and Analysis Center
(WYSAC). Surveys will be mailed to 2,500 addresses of persons who have the same general
Page 6 of 16

socio-demographic profile as the older American respondents in the first two rounds of item
testing; we anticipate a 20% return rate netting us two samples of roughly 500.
The objective of the third round of data collection is to compare responses of individual
questions across methods of administration and to further refine questions for the final
instruments. In this stage, IRT-based psychometric analysis will focus on response differences by
delivery mode as opposed to any socio-demographic characteristics.
Information to be Collected
SSI collects a range of socio-demographic information from panelists when they sign up with
SSI that they makes available to organizations that contract with them. Each iteration of the
questionnaire will ask questions regarding:
• financial well-being; and
• financial ability.
And be followed by items addressing:
•
•
•
•

financial knowledge;
financial behaviors and decisions;
personality traits; and
social context

which will be used for preliminary analyses examining construct validity.
The goal of these questionnaires is to develop a final set of items to measure individually,
financial well-being and financial ability. The final items included in each metric should be
easily comprehended by the average American consumer and should provide a statistically
reliable measure of each concept. To achieve this goal, cognitive interviewing and an iterative
process is used with continuous refinement of the wording of each item, as well as a weeding out
of poor or irrelevant items. Individual question and answer sets may change each time the item
pool is fielded based on the analysis of the results of the previous questionnaire, such as the
elimination of poorly performing items or a reduced number of response options for items in
consideration of the distribution of previously observed responses.
In its simplest form, the process can be described as follows:
Step 1: Generate a large pool of candidate items (see Item Bank, attached as Appendix A) based
on a literature review and qualitative research undertaken in a previous phase of the project.
Step 2a: Revise the candidate items with the panel of experts associated with the project in order
to refine item wording and weed out poor questions.
Step 2b: Revise the candidate items using results from cognitive interviews in order to futher
Page 7 of 16

refine item wording, response options, and weed out poor questions.
Step 3: Field the first round of the item pool online to test the items and use psychometric
techniques (e.g., exploratory and confirmatory factor analysis and item response theory-based
methods) to analyze the items’ performance. Promising questions will be retained and poor
performing questions will be weeded out. There will be a preliminary phase to this first round
that will allow the team to “pull the plug” on the data collection should there be any large
problems with the items.
Step 4: Field the second round of the item pool online with the revised items to refine and
validate the work from round one . The results of this step lead to recommended measures for the
selected concepts (i.e., financial well-being, financial ability).
Step 5: Field a third, final round in order to test the items’ performance when administered
online, over-the-phone and as a pencil-and-paper test. Psychometric analyses will focus on
eliminating items that peform inconsistently across the three modes (online, P&P, and phone).
Additionally, preliminary analyses related to the construct validity (specifically, discriminant and
convergent validity) of the new metrics will also be conducted.
No population inferences will be made using these data.
3. Use of Information Technology
The contractor will collect data electronically through the use of online electronic data collection
(EDC) software. Such software will help reduce errors by:
•
•

•
•

automatically skipping questions, where appropriate, based on prior answers to questions;
randomizing how certain questions, or groups of questions, are presented (e.g., asking
certain consumers about financial behaviors before asking about financial well-being and
asking about financial well-being before asking about financial behaviors) to mitigate the
effects of potential biases caused by the order the questions are presented;
ensuring that some questions cannot be skipped; and.
rejecting invalid responses or data entries.

Additionally, the contractor can collect data on the length of time respondents spent answering
the questionnaire and unit and item non-response rates. This type of information can be used to
improve the question and answer sets.
4. Efforts to Identify Duplication
The CFPB is not aware of any similar data collection efforts aimed at testing the accuracy,
statistical reliability, and respondent comprehension of candidate items and response sets
intended to measure financial well-being and financial ability. While existing data collection
efforts on financial topics are on-going (e.g., ALP, HRS), it is emphasized that these surveys do
not contain psychometrically rigorous and validated metrics for the measurement of financial
Page 8 of 16

well-being or financial ability, as such metrics do not currently exist per the literature review
conducted in the qualitative phase of this project. See also the World Bank’s 2013 publication
“Making Sense of Financial Capability Surveys around the World: A Review of Existing
Financial Capability and Literacy Measurement Instruments 5” for an extremely thorough review
of measurement tools in this space. It describes no existing tools for either the direct concept of
financial well-being, or for the type of financial skill described in our proposed concept of
financial ability.
Prior to initial development of items for these instruments, the contractor conducted an extensive
review of academic literature and available instruments intending to measure individuals’
financial knowledge 6, behavior 7, and overall financial capability 8. While these prior studies refer
to the concept of financial well-being, they neither rigorously defined the concept nor provided
metrics for its measurement. Many studies in the consumer science literature have focused
heavily on explicit financial knowledge or numeracy: e.g. asking respondents to identify the
correct answer when presented with a mathematical word problem on a financial topic such as
calculating compound interest or inflation. 9 A review of the broader literature on knowledge and
5

Available at: http://responsiblefinance.worldbank.org/~/media/GIAWB/FL/Documents/Misc/Financial-CapabilityReview.pdf
6
See Allgood, Sam, and William Walstad. 2011. “The Effects of Perceived and Actual Financial Knowledge on
Credit Card Behavior.” Networks Financial Institute Working Paper 15. Indianapolis, Indiana: Networks
Financial Institute.; Atkinson, Adele, and Flore-Anne Messy. 2012. “Measuring Financial Literacy: Results of
the OECD / International Network on Financial Education (INFE) Pilot Study.” Finance, Insurance and Private
Pensions Working Paper 15. Washington, D.C.: Organization for Economic Cooperation and Development.;
Gerardi, Kristopher, Lorenz Goett, and Stephen Meier. 2010. “Financial Literacy and Subprime Mortgage
Delinquency: Evidence from a Survey Matched to Administrative Data.” Atlanta Federal Reserve Bank
Working Paper 2010-10. Atlanta, GA: Federal Reserve Bank of Atlanta.; Knoll, Melissa A.Z., and Carrie R.
Houts. 2012. “The Financial Knowledge Scale: An Application of Item Response Theory to the Assessment of
Financial Literacy.” Journal of Consumer Affairs 46(3), 381- 410.; Hilgert, Marianne A., Jeanne M. Hogarth,
and Beverly, Sondra G,. 2003. “Household Financial Management: The Connection between Knowledge and
Behavior.” Federal Reserve Bulletin 89: 309-322.; Lusardi, Annamaria, and Olivia S. Mitchell. 2009. “How
ordinary consumers make complex economic decisions: Financial literacy and retirement readiness.” National
Bureau of Economic Research Working Paper 15350. Cambridge, MA: National Bureau of Economic
Research.; Van Rooij, Maarten, Annamaria Lusardi and Rob Alessie. 2011. “Financial literacy and stock market
participation.” Journal of Financial Economics 101(1): 449-472.
7

See Dew, Jeffery, and Jing Jian Xia. 2011. “The Financial Management Behavior Scale: Development and
Validation.” Journal of Financial Counseling and Planning 22(1): 43-59.; Hilgert, Marianne A., Jeanne M.
Hogarth, and Beverly, Sondra G,. 2003. “Household Financial Management: The Connection between
Knowledge and Behavior.” Federal Reserve Bulletin 89: 309-322.; Lynch, John G., Richard G. Netemeyer,
Stephen A. Spiller, and Alessandra Zammit. 2010. “A Generalizable Scale of Propensity to Plan: The Long and
the Short of Planning for Time and Money.” Journal of Consumer Research 37(1): 108-128.

8

See Taylor, Mark. 2011. The Long Term Impacts of Financial Capability: Evidence from the BHPS. Colchester,
Essex: The University of Essex. Consumer Financial Education Body Research Report 03.; Shim, Soyeon, and
Joyce Serido. 2011. Young Adults’ Financial Capability. Tuscan, AZ: University of Arizona. A Pathway to Life
Success for University Students Research Brief Wave 2.

9

The best example of questions commonly used in other studies to measure financial literacy comes from Lusardi,
Annamaria, and Olivia S. Mitchell. 2007. “Financial Literacy and Retirement Preparedness: Evidence and
Page 9 of 16

well-being related behaviors in the education, psychology and health fields conducted as a part
of this study suggests, however, that consumers’ knowledge of financial facts likely is not the
sole relevant predictor of their financial well-being.
5. Efforts to Minimize Burdens on Small Entities
The data collection is not anticipated to burden small entities because the questionnaires will
only collect information from individuals.
6. Consequences of Less Frequent Collection and Obstacles to Burden Reduction
The CFPB would be less able to meet its obligation to develop measurable goals and objectives
that support increasing the financial literacy of consumers, or to develop goals for programs that
provide seniors with financial literacy and counseling, as specified by the Dodd-Frank Act,
without undertaking the proposed questionnaires to develop measures of financial well-being.
7. Circumstances Requiring Special Information Collection
There are no special circumstances. The collection of information is conducted in a manner
consistent with the guidelines in 5 CFR 1320.5(d)(2).
8. Consultation Outside the Agency
In accordance with 5 CFR 1320.8(d)(1), on August 8, 2013, the Bureau published a notice in the
Federal Register allowing the public 60 days to comment on this proposed new collection of
information (78 FR 48422). Further and in accordance with 5 CFR 1320.5(a)(1)(iv), the Bureau
has published a notice in the Federal Register allowing the public 30 days to comment on the
submission of this information collection request to the Office of Management and Budget. No
comments were received.

9. Payments or Gifts to Respondents
The incentives respondents will receive are based on their pre-existing agreements with Survey
Sampling International (SSI) and are not specific to this project. Respondents will come from
existing panels managed by SSI. Agreements regarding incentive payments were established
between SSI and the panelist at the time s/he was recruited. For example, when signing up for
OpinionWorld, a panel administered by SSI, respondents can choose from a variety of incentives
including donations to a preferred charity, entries into a drawing for a monetary prize, etc. SSI
offers a wide variety of incentives in order to increase diversity of its sample frames because
different types of rewards will motivate different respondents to participate in a project.
Rewards offered vary by questionnaire length and the characteristics of the population being
Implications for Financial Education.” Business Economics 42(1): 35-44.

Page 10 of 16

targeted. SSI uses a reasonable level of reward based on the amount of effort required and
subpopulation interests (iTunes credits are popular among younger panelists, for example).
These levels in whatever currency was agreed for that particular respondent are set based on
extensive research to balance motivation to respond and impact on the content of that response.
For the length of questionnaires proposed in this collection, the typical dollar value of the
incentive ranges from $1.50 to $2.
10. Assurances of Confidentiality
An email or online invitation to participate in the project (attached as Appendix B) will direct
respondents to an informed consent statement (attached as Appendix C). This statement will tell
respondents about the study’s purpose, that cooperation is voluntary, and that directly identifying
information, such as name and email address, will not be provided to the CFPB or to any third
party outside of SSI10 (SSI’s privacy policy is attached as Appendix D). If they agree to the
terms described in the informed consent statement, respondents will select the link provided to
enter the project.
The CFPB has specified in its contract with the data collection contractor CFED that the CFPB
does not own and cannot receive personally identifiable information 11 in order to protect the
privacy of respondent data and ensure the pledges of privacy made to respondents by the CFED
research team.
In addition, no personally identifiable information will be stored in the data file or provided to
the contractor. Survey Sampling International protects the identifying information on its
panelists and does not share it with any agency or company conducting research with its panel.
This means that neither the contractor nor the CFPB will be able to reference or disclose such
personally identifiable information about the participants. The contractor and the CFPB will
analyze data and report results only in the aggregate.
11. Justification for Sensitive Questions
Respondents will be asked to provide basic demographic and socio-economic information
including their race/ethnicity, income level, employment status, gender and age. This
information will be used to ensure that a socio-demographically diverse data set is collected, per
the requirements for stable IRT parameter estimation . Respondents will also be asked questions
about their current objective financial circumstance (e.g., liquid savings, credit scores, bill
payment behavior,employment status, retirement savings, etc.). This information will be used to
provide preliminary construct validation of the developing financial well-being and financial
ability scales.
10

SSI will not share any personally identifiable information with the research team. Rather, the research team
(CFED and its subcontractors) will only receive information on respondent demographics, along with their
responses to the questions.
11
The contract language includes the following: “Contractor shall not provide CFPB any personally identifiable
information that could reasonably be used to identify individuals participating in the project.”
Page 11 of 16

12. Estimated Burden of Information Collection
The table below presents the burden hours for the proposed questionnaire, split up by type of
respondent. The last round of data collection, the implementation assessment, is expected to take
less time than the first two rounds, so the table splits these respondents out.

Data Collection
Working-Age Americans
Cognitive interviews
Rounds 1 & 2: Metric Development &
Validation - online
Round 3: Implementation Assessment –
online/telephone/paper
Older Americans
Cognitive interviews
Round 1 & 2: Metric Development &
Validation - online
Round 3: Implementation Assessment –
online/telephone/paper

Hours per
Respondent

Number of
Respondents

Burden
hours per
task

40 minutes
17 minutes

6
6,300

4
1,785

15 minutes

1,500

375

40 minutes
17 minutes

6
5,000

4
1,417

15 minutes

1,500

375

Total Burden Hours:

3,960

13. Estimated Total Annual Cost Burden to Respondents or Recordkeepers
There are no capital/start-up or ongoing operation/maintenance costs associated with this
information collection.
14. Estimated Cost to the Federal Government
There will be no annualized capital/start-up costs for the government to receive the questionnaire
information. The testing is funded with non-appropriated funds. It will cost approximately
$860,000 to collect and analyze these data.
The Blanket Purchase Agreement, under which the questionnaire will be conducted, was
awarded through a competitive bidding process.
15. Program Changes or Adjustments
This is the first request for a new collection.
16. Plans for Tabulation, Statistical Analysis, and Publication
Using responses from three distinct rounds of data collection, the contractor will develop metrics
to assess two constructs of interest (i.e., Financial Well- and Financial Ability) using modern
Page 12 of 16

psychometric methods (e.g., Netemeyer et al. 2003; Thissen and Wainer 2001; Wirth and
Edwards 2007). A large pool of candidate items (questions and response sets) based on the
finding from qualitative research, the project literature review, and a review of other financially
related questionnaires was developed (see attached Item Bank document for an initial pool of
questions on all anticipated topics). A panel of psychometric and content experts reviewed and
refined the pool of candidate items so that the remaining items are targeted and wellworded. Cognitive interviewing of persons who would normally be good candidates for inclusion
in our samples will further serve to refine the item pool and items. No more than 120 items
(defined as a statement, typically one sentence, to which the participants will express their level
of agreement) will be included in the refined item pool.This number of items is specified to
ensure that respondents are not overly burdened when responding to the item pool.
In round I, SSI will field the candidate items online to approximately 2,500 working-age and
2,000 older Americans. Following any necessary data cleaning, preliminary data analyses, by age
group, will then be conducted to examine the variables individually and in pairs to see if any
response patterns need to be addressed before the next steps in data analysis. These will include
one and two-way frequency tables for items and item pairs as well as a polychoric correlation
matrix. Classical test theory analyses (e.g., Cronbach’s alpha, item-total correlations, alpha if
removed) for both individual items and the preliminary scales as a whole will also be conducted
to see how well these items, individually and in groups, measure what we expect them to
measure. Structural assessments (exploratory and confirmatory factor analysis) will be conducted
to ensure that an appropriate number of dimensions are employed to model the relationships
among the items for each construct, using analysis methods appropriate for the categorical nature
of the data (Wirth and Edwards 2007). To ensure that preferred models are not over-fit to the
data, each sample will be randomly divided into two sub-samples: one for exploratory modeling
and the other for confirmatory modeling. Finally, item response theory (IRT) calibration analyses
will be conducted within each age group for each construct individually using the preferred
factor structure from the structural assessment results. Item parameter results from the IRT
calibration of each construct will be presented in tables and in the form of trace line plots/item
characteristic curves (ICCs) for each item. An ICC for a given item represents the probability of
endorsing an item as a function of an individual’s level on the underlying construct.
Using the IRT calibration results and expert opinion, the candidate items used in round I will be
re-evaluated. Items that are performing poorly will be eliminated from further use. An
examination of the remaining items will be conducted to determine any content areas that are not
well-represented or areas on the construct continuum (e.g., ranges along the spectrum from low
financial well-being to high financial well-being) that are not well-measured. Item development
will be conducted to create items intended to fill any identified gaps.
Using the items that performed well in round I and the newly developed items, new iterations of
the 2 financially- related metrics (well-being and ability) will be created. The second round of
online questionnaires (round II) will be fielded to 3,800 working-age and 3,000 older Americans.
Analyses will follow the same structure as outlined previously. As a final step, it is currently
planned that analyses will be conducted to assess if the performance of items differs across the
two age groups, working-age and older Americans. However, for these differential item
Page 13 of 16

functioning (DIF) analyses to be conducted, it is necessary that there is a sufficient amount of
overlap across age-groups, with respect to the well-performing items. A feasibility review will be
conducted to determine, for each construct, if the overlap of items across age groups is sufficient
to allow for comprehensive DIF testing across the age groups. At the end of round II, the IRT
calibration and DIF results, if available, as well as expert opinion will inform the selection of
items to the final versions of the metrics for each construct.
The final set of analyses is planned for the third round of data collection, which will contain
responses from 1500 working-age and 1500 older Americans and use three different modes of
item presentation: online, phone, and paper-and-pencil. Structural and IRT analyses will be
performed, individually for each presentation mode. Next, DIF analyses will be conducted to
identify any item performance changes across the three presentation modes. Items with poor or
inconsistent performance across modes are candidates for removal. Finally, targeted analyses
will be conducted to provide preliminary indications of the construct validity (e.g., convergent
and discriminant validity) for the newly finalized metrics of financial well-being and financial
ability. Appendix E provides a table of the validation measures we will include in the initial data
collection, as well as the expected direction of the relationship for each variable/scale with the
newly developed metrics. Analyses (e.g., correlational, ANOVA) will be conducted as
appropriate for each combination of variables, in consideration of the measurement properties
and response distributions of the variables in question.
The CFPB anticipates that data collection (all three rounds for working age and older Americans)
will be completed within five months of OMB approval. The contractor will develop the initial
analysis and summary of the study findings after data collection has been completed. This is to
be completed within the four months after data has been collected. The CFPB plans to publish
the research findings.
17. Display of Expiration Date
The CFPB plans to display the expiration date for OMB approval of the information collection
on all instruments.
18. Exceptions to the Certification Requirement
The Bureau certifies that this collection of information is consistent with the requirements of 5
CFR 1320.9, and the related provisions of 5 CFR 1320.8(b)(3) and is not seeking an exemption
to these certification requirements.
Bibliography
Allgood, Sam, and William Walstad. 2011. “The Effects of Perceived and Actual Financial
Knowledge on Credit Card Behavior.” Networks Financial Institute Working Paper 15.
Indianapolis, Indiana: Networks Financial Institute.
Atkinson, Adele, and Flore-Anne Messy. 2012. “Measuring Financial Literacy: Results of the
OECD / International Network on Financial Education (INFE) Pilot Study.” Finance, Insurance
Page 14 of 16

and Private Pensions Working Paper 15. Washington, D.C.: Organization for Economic
Cooperation and Development.
Cortina, Jose M. 1993. “Interaction, Nonlinearity, and Multicollinearity: Implications for
Multiple Regression.” Journal of Management 19 (4): 915-922.
Dew, Jeffery, and Jing Jian Xia. 2011. “The Financial Management Behavior Scale:
Development and Validation.” Journal of Financial Counseling and Planning 22(1): 43-59.
FLEC. Financial Literacy and Education Commission Research and Evaluation Working Group.
2012. “2012 Research Priorities and Research Questions.” http://www.treasury.gov/resourcecenter/financial-education/Documents/2012%20Research%20Priorities%20-%20May%2012.pdf
GAO Highlights. 2011. “Financial Literacy: A Federal Certification Process for Providers Would
Pose Challenges.” GAO-11-614 http://www.gao.gov/assets/330/320203.pdf
Gerardi, Kristopher, Lorenz Goett, and Stephen Meier. 2010. “Financial Literacy and Subprime
Mortgage Delinquency: Evidence from a Survey Matched to Administrative Data.” Atlanta
Federal Reserve Bank Working Paper 2010-10. Atlanta, GA: Federal Reserve Bank of Atlanta.
Knoll, Melissa A.Z., and Carrie R. Houts. 2012. “The Financial Knowledge Scale: An
Application of Item Response Theory to the Assessment of Financial Literacy.” Journal of
Consumer Affairs 46(3): 381- 410.
Hilgert, Marianne A., Jeanne M. Hogarth, and Beverly, Sondra G,. 2003. “Household Financial
Management: The Connection between Knowledge and Behavior.” Federal Reserve Bulletin 89:
309-322.
Lusardi, Annamaria, and Olivia S. Mitchell. 2009. “How ordinary consumers make complex
economic decisions: Financial literacy and retirement readiness.” National Bureau of Economic
Research Working Paper 15350. Cambridge, MA: National Bureau of Economic Research.
Lusardi, Annamaria, and Olivia S. Mitchell. 2007. “Financial Literacy and Retirement
Preparedness: Evidence and Implications for Financial Education.” Business Economics 42(1):
35-44.
Lynch, John G., Richard G. Netemeyer, Stephen A. Spiller, and Alessandra Zammit. 2010. “A
Generalizable Scale of Propensity to Plan: The Long and the Short of Planning for Time and
Money.” Journal of Consumer Research 37(1): 108-128.
Netemeyer, Richard G., William O. Bearden, and Subhash Sharma. 2003. Scaling Procedures:
Issues and Applications. Thousand Oaks, CA: Sage Publications.
Shim, Soyeon, and Joyce Serido. 2011. Young Adults’ Financial Capability. Tuscan, AZ:
Page 15 of 16

University of Arizona. A Pathway to Life Success for University Students Research Brief Wave
2.
Taylor, Mark. 2011. The Long Term Impacts of Financial Capability: Evidence from the BHPS.
Colchester, Essex: The University of Essex. Consumer Financial Education Body Research
Report 03.
Thissen, David, and Howard Wainer (Eds.). 2001. Test Scoring. Mahwah, NJ: Lawrence
Erlbaum Associates.
Van Rooij, Maarten, Annamaria Lusardi and Rob Alessie. 2011. “Financial literacy and stock
market participation.” Journal of Financial Economics 101(1): 449-472.
Wirth, R. J., & Michael C. Edwards. 2007.”Item factor analysis: Current approaches and future
directions.” Psychological Methods 12: 58-79.
Attachments
•
•
•
•
•
•

Supporting Statement Part B
Appendix A – Survey Question Item Bank
Appendix B – Survey Invitations
Appendix C– Informed Consent and Link to Enter Survey
Appendix D – SSI Privacy Policy
Appendix E – Preliminary Validation Items

Page 16 of 16