Generic PRA Cognitive Research and Pilot Testing Supporting Statement A_OMB edits CLEAN

BUREAU OF CONSUMER FINANCIAL PROTECTION
PAPERWORK REDUCTION ACT SUBMISSION
INFORMATION COLLECTION REQUEST

SUPPORTING STATEMENT PART A
Generic Information Collection Plan to Conduct Cognitive Research and Pilot Testing
(OMB CONTROL NUMBER: 3170-XXXX)

OMB TERMS OF CLEARANCE: None
ABSTRACT: Under the Dodd-Frank Wall Street Reform and Consumer Protection Act, the
Consumer Financial Protection Bureau is charged with researching, analyzing, and reporting on
topics relating to the Bureau’s mission, including developments in markets for consumer
financial products and services, consumer awareness, and consumer behavior. In order to
improve its understanding of how consumers engage with financial markets, the CFPB seeks to
obtain approval for a generic information collection plan to conduct research to improve the
quality of data collection by examining the effectiveness of data-collection procedures and
processes, including potential psychological and cognitive issues.
A. JUSTIFICATION
1. Circumstances Necessitating the Data Collection
The Consumer Financial Protection Bureau (CFPB, Bureau)’s mission focuses on
understanding how consumer financial markets work, highlighting avenues for innovation in
financial products and services, and identifying and mitigating potential risks to consumers.1 The
Dodd-Frank Wall Street Reform and Consumer Protection Act (Dodd-Frank Act) authorizes the
CFPB to engage in research and market monitoring activities to assess trends and to identify
emerging risks in consumer financial markets.2 Section 1013(b)(1) of the Dodd-Frank Act
1

CFPB FY 2013 – FY 2017 Strategic Plan, goal 3, http://www.consumerfinance.gov/strategic-plan/
DFA Section 1013(b)(1) mandates that the Bureau have research staff whose responsibilities include
“[r]esearching, analyzing, and reporting on (A) developments in markets for consumer financial products or
services, including market areas of alternative consumer financial products or services with high growth rates and
areas of risk to consumers; (B) access to fair and affordable credit for traditionally underserved communities; (C)

2

Page 1 of 15

establishes the Office of Research for the purpose of researching, analyzing, and reporting on
topics related to the Bureau’s mission, including consumer behavior, consumer awareness, and
developments in markets for consumer financial products and services. The Act also established
the Office of Financial Education (OFE) within the CFPB, which is responsible for developing
and implementing initiatives intended to educate and empower consumers to make betterinformed financial decisions, and, together with the CFPB’s Office of Research, for conducting
research related to financial education and counseling. Using empirical evidence and rigorous
research, the Bureau improves its understanding of consumer financial markets and is able to
provide accurate and comprehensive information to consumers, firms, researchers, and
policymakers.
The CFPB supports this mission by building an understanding of the markets that it regulates
and the nature of consumer behavior within these markets. For this reason, the Bureau plans to
seek several generic clearances for “research” functions (outside of rulemaking), to improve our
scientific understanding. To this end, the Office of Research and the Office of Financial
Education as well as other relevant offices at the CFPB, plan to use this generic information
collection plan for cognitive testing or piloting studies pursuing related research projects.
In general, examples of topics that the Bureau is interested in studying are noted below
and will be limited to the cognitive testing of study materials and limited piloting of study
methodologies:
•
•
•
•

consumer awareness and decision-making, for example, understanding what financial
products or features people are aware of and why they choose one product over another;
consumer experiences and potential risks to consumers, including traditionally underserved
consumers;
information disclosure and how disclosure may affect behavior;
the effect of educational or other interventions on consumer financial behavior or outcomes;

These topics will usually be in the context of consumer financial product markets,
including mortgages, car loans, student loans, installment loans, small dollar loans and credit,
debit, and prepaid cards. In addition, research may be related to the CFPB’s mission regarding
financial education, including evaluating the effectiveness of financial education programs and
understanding financial planning behaviors, including savings and spending behavior. The
CFPB envisions that the research covered under this generic clearance will be to help inform
Page 2 of 15

larger primary data collection research project about consumer credit markets and household
finance, for example, surveys or controlled trials in field and economic laboratory settings.

The purpose of this generic clearance is to improve and- refine data collection procedures by
testing specific aspects of methods and procedures such as questionnaire design. For research
projects like surveys and controlled trials in field and economic laboratory settings, it is
generally best practice in the social sciences to rigorously pretest instruments or interventions
prior to the full data collection. Pretesting methods will allow the CFPB to develop, test, and
improve its survey and data collection instruments and methodologies, for example: evaluating
questionnaires/assessments and various data collection procedures; developing survey scales;
identifying questionnaire/assessment and procedural problems; suggesting solutions; and
measuring the relative effectiveness of alternative solutions.
Common pretesting methods include: pilot testing, behavior coding, interviewer debriefing,
exploratory interviews, respondent debriefing questionnaire, follow-up interviews or reinterviews, split sample experiments, cognitive and usability interviews, and focus groups.

consumer awareness, understanding, and use of disclosures and communications regarding consumer financial
products or services; (D) consumer awareness and understanding of costs, risks, and benefits of consumer financial
products or services; (E) consumer behavior with respect to consumer financial products or services, including
performance
on mortgage
loans; and (F) experiences of traditionally underserved consumers, including un‐banked
and under‐banked
consumers.”

Page 3 of 15

Below are detailed explanations of each method, which may be used in combination for a single
research project.
Pilot testing. For the purposes of this clearance, we are defining pilot tests as data
collection efforts to evaluate, for example, data collection-instruments, materials, and
procedures on a reduced scale before conducting a full-scale data collection. Pilot tests
are an essential component of this clearance package because they serve as the vehicle for
investigating basic item properties for new or redesigned data collection efforts, such as
reliability, validity, and difficulty, as well as feasibility of methods for standardized
administration of forms.

Behavior coding. This method involves applying a standardized coding scheme to the
completion of an interview or questionnaire, either by a coder using a tape-recording of
the interview or by an in-person observer at the time of the interview. The coding scheme
is designed to identify situations that occur during the interview that reflect problems
with the questionnaire. For example, if respondents frequently interrupt the interviewer
before the question is completed, the question may be too long. If respondents frequently
give inadequate answers, this suggests the question is unclear. Quantitative data derived
from this type of standardized coding scheme can provide valuable information to
identify problem areas in a questionnaire, and can be used as a substitute for or as a
complement to the traditional interviewer debriefing.

Interviewer debriefing. This method employs the knowledge of the employees who have
the closest contact with the potential respondents. In conjunction with other methods, we
plan to use this method to collect information about how interviewers react to the survey
instruments and laboratory or field experimental procedures; to understand problems and

Page 4 of 15

pitfalls encountered by interviewers during the interview; and to learn from interviewers
about respondents’ reactions in the field.

Exploratory interviews. These may be conducted with individuals who are experts to
understand a topical area and may be used in the very early stages of developing a new
procedure. For example, experts may be from relevant industries or leading researchers
who study the issues at hand. It may cover discussions related to administrative records
(e.g. what types of records, where, and in what format), subject matter, definitions, etc.
Exploratory interviews may also be used to investigate whether sufficient issues are
present related to an existing data collection to consider a redesign.

Respondent debriefing questionnaire. In this method, standardized debriefing
questionnaires are administered to respondents who have participated in a field test,
laboratory study, or completed a survey that is being tested. The debriefing form is
administered at the end of the questionnaire being tested, and contains questions that
probe to determine how respondents interpret the questions and whether they have
problems completing the questionnaire or procedure. For example, this structured
approach to debriefing enables quantitative analysis of data from a sample of respondents
in order to learn whether respondents can answer the survey questions, and whether they
interpret them in the manner intended by the questionnaire designers.

Follow-up interviews or re-interviews. This involves re-interviewing or re-assessing a
sample of respondents after the completion of a survey or assessment. Responses given in
the re-interview are compared with the respondents’ initial responses to gauge
consistency between responses. In this way, re-interviews provide data for studies of test–
retest reliability and other measures of data quality. In turn, this information aids in the
development of more reliable measures.

Split sample experiments. This involves testing alternative versions of questionnaires,
and other collection methods. The alternatives may be designed to address problems
identified in draft questionnaires or questionnaires from previous studies or to investigate
Page 5 of 15

methodological research questions. In the case of surveys, multiple questionnaires are
developed and then randomly assigned to potential respondents, to permit statistical
comparisons; data collection can include mail, telephone, Internet, or in-person
interviews or group sessions at which self-administered questionnaires are completed.
Comparison of revised questionnaires against a control version or against each other
facilitates statistical evaluation of the performance of alternative versions of the
questionnaire. Split sample tests that incorporate questionnaire design experiments are
likely to have a larger maximum sample size than field tests using other methodologies.
Larger sample sizes will enable the detection of statistically significant differences and
facilitate methodological experiments that can extend questionnaire design knowledge
more generally for use in a variety of CFPB data collection instruments.

Cognitive and usability interviews. This method involves intensive, one-on-one
interviews in which the respondent is typically asked to "think aloud" as he or she
answers survey questions or participates in the experimental procedure and to answer
follow-up questions about the survey or experimental procedure. A number of different
techniques may be involved, including asking respondents to paraphrase questions or
instructions, asking probing questions to determine how respondents came up with their
answers or interpreted directions, and so on. The objective is to identify problems of
ambiguity or misunderstanding, or other difficulties respondents have. This is frequently
one of the early stages of revising study materials.

Focus groups. This method involves group sessions guided by a moderator, who follows
a topical outline containing questions or topics focused on a particular issue, rather than
adhering to a standardized questionnaire. Focus groups are useful for surfacing and
exploring issues (e.g., confidentiality concerns).

This clearance is similar in intent and scope to the testing clearances held by the Census
Bureau, the Bureau of Labor Statistics, the National Center for Education Statistics, and the
National Science Foundation, that allowing these agencies to develop, redesign, and test data
collection instruments and procedures in a timely manner.
Page 6 of 15

2. Use of the Information
The CFPB requests approval from the Office of Management and Budget (OMB) for a
generic information collection plan which would allow the Bureau to collect data through
pretesting, for example, by testing various instruments or methodologies prior to the full
collection. The aim of these pretests is to improve the Bureau’s data collection processes and
improve the quality of the data in the studies that are ultimately conducted.

The CFPB generally plans to test survey instruments, laboratory methods, and field trials
on a small scale prior to their use in full-scale studies, e.g. through cognitive pretesting or by
conducting a subset of the surveys initially and reviewing the responses for evidence of problems
before fielding the survey more widely. These methods can provide tools to improve
questionnaires and other research instruments, for example, by testing comprehension of survey
and lab questions. Through this testing, questions can be tailored to increase the accuracy of the
collected information and to reduce respondent burden by ensuring that large-scale information
collection is optimized.

Research under this generic clearance is important in order to 1) improve the data
collection instruments employed by the Bureau, 2) increase the accuracy of the data and, in turn,
of the research studies produced by the CFPB, including studies that may be factored into some
policy decisions, 3) reduce the burden of completing or administering survey instruments for
respondents and interviewers, 4) increase response rates in surveys, 5) increase the ease of use of
the CFPB programs and products, and 6) improve the scientific accuracy of CFPB studies.

The information collected in this generic clearance will be used by the CFPB to evaluate
and improve the quality of instruments or interventions before full-scale research projects are
conducted. In general, data collected under this clearance will not alone be used as official
federal agency statistics. Data may be made public, when appropriate, in research reports,

Page 6 of 15

presentations related to study methodology, at professional meetings, in publications on the
CFPB website, and in professional journals.

3. Use of Information Technology
The CFPB will employ information technology as appropriate to reduce the burden of
respondents who agree to participate in its research. Computer-assisted participation will be used
when feasible and appropriate, including online studies. In general, the Bureau intends to
incorporate electronic data-collection techniques whenever possible, particularly since in many
instances they are likely to impose the lowest burden on respondents and may be the most costeffective. However, the extent to which the Bureau will use electronic data collection for a given
project will be determined by weighing several factors, including response rates and costs. For
example, evidence to date suggests that web-only data collections tend to have lower response
rates than surveys conducted by mail or phone.3 The Bureau expects that the most effective
mode, including mixed-mode combinations, may also vary with the complexity and sensitivity of
the information to be collected.

4. Efforts to Identify Duplication
The Bureau will explore methodological, psychological and cognitive methods-related
issues in order to answer questions that are not satisfactorily addressed in the relevant research
literature. A review of the existing relevant literature will be undertaken before proceeding with
projects. To the extent possible, the CFPB will make use of existing information, including
information from previous studies. However, this information often is not sufficient to refine our
study questionnaires and assessments without conducting additional research.

5. Efforts to Minimize Burdens on Small Entities
All of our research projects will be limited to individual consumers, not businesses, so
there will be no impact on small businesses or other small entities.
3

Roger Tourangeau, Fredrick Conrad, and Mick Couper (2013), “The Science of Web Surveys.” USA: Oxford
University Press.

Page 7 of 15

6. Consequences of Less Frequent Collection and Obstacles to Burden Reduction
The planned collection of data under this generic clearance will allow the CFPB to
improve larger scaled data collections before they are administered, reducing burden by
increasing the effectiveness of the larger information collection. Failing to scientifically
investigate the data collection process can lead to significant costs in terms of data quality and
burden to respondents, interviewers, and data users alike during the final data collection and
analysis of results. For example, poorly or confusing question that is not written clearly could
unintentionally engender an incorrect response the lead to biased responses or to high rates of
item nonresponse.

7. Circumstances Requiring Special Information Collection
There are no special circumstances anticipated for the information collections under this
generic clearance, they will be consistent with the applicable guidelines contained in 5 CFR
1320.5(d)(2).

8. Consultation Outside the Agency
In accordance with 5 CFR §1320.8(d)(1), the Bureau has published a notice Federal
Register allowing the public 60 days to comment on this proposed new generic information
collection plan. No comments were received in response to this notice.. Further and in
accordance with 5 CFR §1320.5(a)(1)(iv), the Bureau has also published a notice in the Federal
Register allowing the public 30 days to comment on the submission of this information
collection request to the Office of Management and Budget.
9. Payment or Gifts to Respondents
Respondents for activities conducted in the laboratory (for example, cognitive interviews
and focus groups) under this clearance will receive a small stipend. This practice has proven
necessary and effective in recruiting subjects to participate in this small-scale research, and is

Page 8 of 15

also employed by the other Federal cognitive laboratories. The incentive for participation in a
cognitive interview is $40, and for participation in a focus group is $50-$75. The CFPB may
provide smaller incentives than these amounts at its discretion; however, any requests for nonstandard amounts must be justified in writing to OMB.

10. Assurances of Confidentiality
CFPB shall treat the information collected in accordance with applicable federal law,
including, but not limited to the Bureau’s confidentiality rules, 12 C.F.R. Part 1070, and the
federal laws and regulations that apply to federal agencies for the protection of privacy,
confidentiality, security and integrity.
The CFPB uses best practices of social science research design to inform the notice
and consent vehicles. When required and when practicable, the CFPB provides notice to
individuals that explain how their information will be used through appropriate vehicles,
such as Privacy Notices, Privacy Act Statements or Informed Consent forms. Such notice is
typically made available prior to the collection of information and explains whether the
information is mandatory or voluntary; whether there are any opportunities to consent to
sharing and submission of information; how the information will be secured and when a
System of Records is created under the Privacy Act. In some instances, research may be
based on observations or on information collected directly from a third party, and notice and
consent may not be applicable.

When the CFPB collects information from third parties, commercial sources, and
public databases, individuals to whom information may pertain do not have the opportunity
to consent to uses, decline to provide information, or opt out directly to the Bureau. In such
cases, it is those third parties’ responsibility to provide any opportunity that may be required
to consent, decline, or opt out of how their information may be used. As part of the due
diligence process, the CFPB reviews the privacy policies or other public disclosures from
the third-parties regarding their use of the information to verify that there are no
contradictions with the research, and the CFPB encourages the development of notice and
Page 9 of 15

consent opportunities. In these instances, the Bureau will ensure that appropriate privacy
provisions are included in contract language, Memorandum of Understanding (MOUs) and
Statements of Work (SOW), as applicable. The Bureau also evaluates the potential privacy
risk and harm to individuals of specific research relative to that authorized purpose, and vets
research proposals to ensure that they serve an authorized purpose.

The information collected under this generic clearance may include direct identifying
personally identifiable information (PII), depending on the type of project. Typically direct
identifying PII consists of contact information used to facilitate administration of research,
and is separated from response data generated from the research. When the CFPB does use
direct identifying PII in analyses, it does so in order to a) match across datasets; b) update
data sets; c) weight datasets in order to make generalizations regarding a given population;
and d) contact potential recipients of our studies to solicit information on consumer
experiences with consumer financial products and services.

Conducting these research studies implicates privacy concerns because a breach of
confidentiality, or re-identification, could result in an individual suffering harm. To reduce
the risk of breaches of confidentiality, the CFPB designs recruitment materials so as not to
disclose sensitive information about those it seeks to recruit, and uses appropriate security
controls to protect information used in research. There is also risk related to misuse of
information collected for research. Misuse might involve secondary types of research that
are incompatible with the purposes of the initial collection, or a use of the information that
individuals do not understand or to which they have not provided consent.

To reduce the risk of misuse, the CFPB minimizes access to PII based on need-toknow and stipulates, in most instances, that contractors that collect data on behalf of the
Bureau remove or redact all direct identifying PII, as defined by the Bureau’s privacy office,
before transmitting data to the Bureau. In general, the CFPB will only keep response data
stripped of direct identifying PII. Any contractor staff assigned to the project also sign
Page 10 of 15

confidentiality agreements. When appropriate, research results will be presented in
aggregated form to protect the confidentiality of firms or consumers, and any publicly
released version of data will use disclosure protection techniques (e.g., rounding,
imputation, exclusion of some variables, aggregation of categorical responses) to minimize
the risk of releasing personally identifiable or otherwise sensitive information (12 CFR
1070.40 et seq.). The Bureau treats the information collected from participating persons and
institutions in a manner consistent with our confidentiality regulations, and all data and
analyses are subject to legal and privacy review prior to their release.

The Bureau also evaluates the potential privacy risk and harm to individuals of
specific research relative to that authorized purpose, and vets research proposals to ensure
that they serve an authorized purpose. Pretesting research will be conducted consistent with
the Privacy Act and the E-Government Act. The requisite SORNs and PIAs will document
the collection, use, disclosure, and retention of PII; the technical, administrative, and
physical controls used to minimize privacy risks. The SORNs and PIAs will so be updated
to reflect any changes. The CFPB has two SORNs that will cover pretesting research:
CFPB.021 Consumer Education and Engagement Records, 77 FR 60382; and CFPB.022
Market and Consumer Research Records, 77 FR 67802. Specific details regarding
information handling will be specified in individual submissions under this generic
clearance, but will conform to these broad guidelines.

11. Justification for Sensitive Questions
Questions about an individual’s finances, for example, what their income is or how
much their mortgage costs each month, are commonly considered sensitive. Nonetheless,
the CFPB often must ask these questions in order to understand consumer behavior and to
recognize financial trends and emergent risks relevant to consumers. The Bureau believes
that these questions are justified by their central importance to the Bureau’s mission.

In addition, some people may consider questions about race and other demographic
information to be sensitive. The CFPB is, however, mandated to enforce fair lending laws
Page 11 of 15

and to identify risks to vulnerable populations, including service members, older
Americans, and lower-income consumers. For this reason, the Bureau often needs to ask
these types of questions. For information collections involving questions of race or
ethnicity, the CFPB will follow the OMB standards for Classification of Federal Data on
Race and Ethnicity (Federal Register, October 30, 1997, Volume 62, Number 210, pages
58781-59790).

Respondent participation will be voluntary for each survey, economic lab experiment,
or other information collection distributed under this generic clearance, and participants
will be made aware of this fact. All respondents shall be free to opt-out of a data
collection at any time and for any reason. When possible, administrative data for the
respondents and nonrespondents will be used to analyze nonresponse bias for information
collections.

The CFPB will ensure that a citation is made to any applicable SORN and PIAs are
published as appropriate.
12. Estimated Burden of Information Collection
For research projects like surveys and controlled trials in field and economic laboratory
settings, the CFPB plans to rigorously pretest instruments or interventions prior to the full data
collection. Currently the Bureau plans to conduct up to four surveys, three controlled trials in the
field, and four controlled trials in the lab each year, and in many cases the Bureau will want to
pretest components of each of these before conducting the full data collection. On average,
pretesting of surveys will require 600 participants each year for 60 minutes each, while studies in
field and lab contexts will require 50 and 20 participants, respectively, for an hour and a half
each. Therefore, estimated total burden hours under this generic clearance total 8,325.

Burden Hour Estimate for All Data Collections, 3-yr period
Types of Projects
Subtotal

Page 12 of 15

Pretesting for Surveys
Pretesting for Field Studies
Pretesting for Lab Studies

7,200
360
675
8,235

Burden Hour Estimate for Pretesting of Surveys, 3-yr period
Number of
Hours per
Number of Projects Participants
Subtotal
Participant
in a Project
12
600
1
7,200
7,200
Burden Hour Estimate for Pretesting of Economic Studies in the Field,
3-yr period
Number of
Hours per
Number of Projects Participants
Subtotal
Participant
in a Project
9
50
1.5
675
675
Burden Hour Estimate for Pretesting of Economic Studies in the Lab,
3-yr period
Number of
Hours per
Number of Projects Participants
Subtotal
Participant
in a Project
12
20
1.5
360
360

There will be no financial burden on the respondents. For each specific collection request
submitted under this generic plan, a detailed estimate of the burdens of that particular request
will be submitted.

13. Estimated Total Annual Cost Burden to Respondents or Recordkeepers
There are no capital/start-up or ongoing operation/maintenance costs associated with this
information collection. If there are postage costs to return a survey questionnaire, it will be prepaid by the contractor. For other methods of participation, including phone, online, or in-person,

Page 13 of 15

respondents should not bear costs. In tests of instruments or procedures of controlled trials in
laboratory or field settings, the CFPB does not anticipate any cost burden to respondents.

14. Estimated Cost to the Federal Government
The costs of pretesting and cognitive testing research projects under this generic
clearance will vary based on the number of individuals in the study, the method of pretesting
used, and the size of any payments to incentivize participation. We expect the cost of each
project to range from $15,000 to $75,000

15. Program Changes or Adjustments
This is a new information collection request; therefore, all the burden (see item 12) is
considered a program change.
16. Plans for Tabulation, Statistical Analysis, and Publication
This generic clearance will be used to pretest instruments or interventions prior to the full
data collection for surveys and controlled trials in field and laboratory settings. The purpose of
data collected under this generic clearance is to improve the quality of data collection by
examining the psychological and cognitive aspects of collection methods and procedures. For
this reason, results from data collected under this generic clearance may or may not be published,
depending on the project. For example, if the methodology appears sound, pilot data may be used
to augment data collected during the full research project. When results are published, the data
may also be used for research and development reports or be included as a methodological
appendix or footnote in a report containing data from a larger data collection effort. In addition,
the methodological results of this research may be prepared for presentation at professional
meetings or for publication in professional journals.
The Bureau’s research projects are intended to expand scientific understanding of
consumer decision making in financial domains. Research findings of full research projects
generally will be published as technical articles that are publicly disseminated as working papers,
peer-reviewed journal articles, CFPB white papers, or CFPB research briefs. Due to the nature
Page 14 of 15

of this clearance, there currently is no definitive or tentative schedule. We expect work to be
conducted more or less continuously throughout the duration of the clearance. The Office of
Research plans to conduct an ongoing program of research and therefore plans to apply for an
extension to the anticipated generic clearance when it expires.
17. Display of Expiration Date
The OMB control number and expiration date associated with this PRA submission will
be displayed on the Federal government’s electronic PRA docket at www.reginfo.gov and on all
instruments associated with information collections approved under this collection plan.
18. Exceptions to the Certification Requirement
The Bureau certifies that this collection of information is consistent with the requirements of
5 CFR 1320.9, and the related provisions of 5 CFR 1320.8(b)(3) and is not seeking an exemption
to these certification requirements.

Page 15 of 15