3170-0055_Cognitive testing generic SS A_2-30-day OMB

30-day Federal Register Notice

CONSUMER FINANCIAL PROTECTION BUREAU PAPERWORK REDUCTION ACT SUBMISSION

INFORMATION COLLECTION REQUEST

SUPPORTING STATEMENT PART A

Generic Information Collection Plan to Conduct Cognitive

and Pilot Testing of Research Methods, Instruments, and

Forms

(OMB CONTROL NUMBER: 3170-0055)

OMB TERMS OF CLEARANCE: The Office of Management and Budget (OMB) did not provide any Terms of Clearance when it last approved this generic information collection plan in November 2015.

ABSTRACT: Under the Dodd-Frank Wall Street Reform and Consumer Protection Act, the Consumer Financial Protection Bureau (Bureau) is charged with researching, analyzing, and reporting on topics relating to the Bureau’s mission, including developments in markets for consumer financial products and services, consumer awareness, and consumer behavior. In order to improve its understanding of how consumers engage with financial markets, the Bureau seeks to renew approval for a generic information collection plan to conduct research to improve the quality of data collection by examining the effectiveness of data-collection procedures and processes, including potential psychological and cognitive issues.

JUSTIFICATION

1. Circumstances Necessitating the Data Collection

The Dodd-Frank Wall Street Reform and Consumer Protection Act (Dodd-Frank Act) authorizes the Bureau to engage in research and market monitoring activities to assess trends and to identify emerging risks in consumer financial markets.¹ Section 1013(b)(1) of the Dodd-Frank Act establishes the Office of Research for the purpose of researching, analyzing, and reporting on topics related to the Bureau’s mission, including consumer behavior, consumer awareness, and developments in markets for consumer financial products and services. ² The Act also established the Office of Financial Education (OFE) within the Bureau, which is responsible for developing and implementing initiatives intended to educate and empower consumers to make better- informed financial decisions, and, together with the Bureau’s Office of Research, for conducting research related to financial education and counseling. Using empirical evidence and rigorous research, the Bureau improves its understanding of consumer financial markets and is able to provide accurate and comprehensive information to consumers, firms, researchers, and policymakers.

In general, examples of topics that the Bureau is interested in studying are noted below and will be limited to the cognitive testing of materials and limited piloting of study methodologies:

• consumer awareness and decision-making, for example, understanding what financial products or features people are aware of and why they choose one product over another;

• consumer experiences and potential risks to consumers, including traditionally underserved consumers;

• information disclosure and how disclosure may affect behavior; and

• the effect of educational or other interventions on consumer financial behavior or outcomes.

These topics will usually be in the context of consumer financial product markets, including mortgages, car loans, student loans, installment loans, small dollar loans and credit, debit, and prepaid cards. In addition, research may be related to the Bureau’s mission regarding financial education, including evaluating the effectiveness of financial education programs and understanding financial planning behaviors such as savings and spending behavior. The Bureau envisions that the research covered under this generic clearance will be to help inform larger primary data collection research project about consumer credit markets and household finance, for example, surveys or controlled trials in field and economic laboratory settings.

The purpose of this generic clearance is to support the Bureau’s research mission by providing a vehicle that will help the Bureau to develop disclosure forms or to refine its data collection instruments and procedures through iterative cognitive research. For research projects like surveys and controlled trials in field and economic laboratory settings, it is generally a best practice in the social sciences to rigorously test instruments or interventions prior to the full data collection. Testing allows the Bureau to develop and improve its survey and data collection instruments and methodologies. Similarly, cognitive research methods allow the Bureau to develop and improve disclosures that it develops through direct feedback from consumers. Cognitive research conducted through this clearance may include evaluating questionnaires/disclosures and various data collection procedures; developing survey scales; identifying problems of consumer understanding of questionnaires/disclosures; identifying procedural problems of research methods; suggesting solutions; and measuring the relative effectiveness of alternative solutions.

Common testing methods include: pilot testing, behavior coding, interviewer debriefing, exploratory interviews, respondent debriefing questionnaire, follow-up interviews or re- interviews, split sample experiments, cognitive and usability interviews, and focus groups. Below are detailed explanations of each method, which may be used in combination for a single research project.

Pilot testing. For the purposes of this clearance, we are defining pilot tests as data collection efforts to evaluate data collection instruments, materials, and procedures on a reduced scale before conducting a full-scale data collection. Pilot tests are an essential component of this clearance package because they serve as the vehicle for investigating basic item properties for data collection efforts, such as reliability, validity, and difficulty, as well as feasibility of methods for standardized administration of forms.

Behavior coding. This method involves applying a standardized coding scheme to the completion of an interview or questionnaire, either by a coder using a tape-recording of the interview or by an in-person observer at the time of the interview. The coding scheme is designed to identify situations that occur during the interview that reflect problems with the questionnaire or disclosure. For example, if many respondents interrupt the interviewer before the question is completed, the question may be too long. If many respondents give inadequate answers, this suggests the question is unclear. Quantitative data derived from this type of standardized coding scheme can provide valuable information to identify problem areas for a questionnaire or disclosure, and can be used as a substitute for or as a complement to the traditional interviewer debriefing.

Interviewer debriefing. This method employs the knowledge of the employees who have the closest contact with the potential respondents. In conjunction with other methods, we plan to use this method to collect information about how interviewers react to the survey instruments and laboratory or field experimental procedures; to understand problems and pitfalls encountered by interviewers during the interview; and to learn from interviewers about respondents’ reactions in the field.

Exploratory interviews. These may be conducted with individuals who are experts to understand a topical area and may be used in the very early stages of developing a new procedure. For example, experts may be from relevant industries or leading researchers who study the issues at hand. Experts may also include consumers who have experience with financial products that will be covered by disclosures that the Bureau may develop. It may cover discussions related to administrative records (e.g. what types of records, where, and in what format), subject matter, definitions, etc. Exploratory interviews may also be used to investigate whether sufficient issues are present related to an existing data collection or disclosure to consider a redesign.

Respondent debriefing questionnaire. In this method, standardized debriefing questionnaires are administered to respondents who have participated in a field test, laboratory study, or completed a survey or reviewed a disclosure that is being tested. The debriefing form is administered after the participant reviews the disclosure or questionnaire being tested, and contains questions that probe to determine how respondents interpret the questions and whether they have problems completing the questionnaire or procedure or reviewing the disclosure. For example, this structured approach to debriefing enables quantitative analysis of data from a sample of respondents in order to learn whether respondents can answer the survey questions, and whether they interpret them in the manner intended by the questionnaire designers.

Follow-up interviews or re-interviews. This involves re-interviewing or re-assessing a sample of respondents after the completion of a survey or assessment. Responses given in the re-interview are compared with the respondents’ initial responses to gauge consistency between responses. In this way, re-interviews provide data for studies of test– retest reliability and other measures of data quality. In turn, this information aids in the development of more reliable measures.

Split sample experiments. This involves testing alternative versions of questionnaires and other collection methods. The alternatives may be designed to address problems identified in draft questionnaires or questionnaires from previous studies or to investigate methodological research questions. In the case of surveys, multiple questionnaires are developed and then randomly assigned to potential respondents, to permit statistical comparisons; data collection can include mail, telephone, Internet, or in-person interviews or group sessions at which self-administered questionnaires are completed. Comparison of revised questionnaires against a control version or against each other facilitates statistical evaluation of the performance of alternative versions of the questionnaire. Split sample tests that incorporate questionnaire design experiments are likely to have a larger maximum sample size than field tests using other methodologies. Larger sample sizes will enable the detection of statistically significant differences and facilitate methodological experiments that can extend questionnaire design knowledge more generally for use in a variety of Bureau data collection instruments.

Cognitive and usability interviews. This method involves intensive, one-on-one interviews in which the respondent is typically asked to "think aloud" as he or she answers survey questions, reviews a disclosure, or participates in the experimental procedure and to answer follow-up questions. A number of different techniques may be involved, including asking respondents to paraphrase the materials they are reviewing, asking probing questions to determine how respondents came up with their answers or interpreted directions, and so on. The objective is to identify problems of ambiguity or misunderstanding, or other difficulties respondents have. Cognitive and usability interviews may be conducted iteratively, with questionnaires, disclosures, or other materials improved between rounds of data collection to respond to issues that arise during the interviews.

Focus groups. This method involves group sessions guided by a moderator, who follows a topical outline containing questions or topics focused on a particular issue, rather than adhering to a standardized questionnaire. Focus groups are useful for surfacing and exploring issues (e.g., confidentiality concerns), and for learning about consumer perspectives that can help to inform the development of materials such as questionnaires or disclosures that communicate information in a way that is clear and understandable for consumers.

This clearance is similar in intent and scope to the testing clearances held by the Census Bureau, the Bureau of Labor Statistics, the National Center for Education Statistics, and the National Science Foundation, that allowing these agencies to develop, redesign, and test data collection instruments and procedures in a timely manner.

2. Use of the Information

The Bureau requests approval from OMB for a generic information collection plan which would allow the Bureau to conduct cognitive testing. The aim of this cognitive testing is to improve the Bureau’s data collection processes and data quality before launching larger-scale studies, and to develop information for consumers (such as disclosures) that is clear and understandable.

The Bureau generally plans to test survey instruments, laboratory methods, disclosure forms, and field trials on a small scale prior to their use in full-scale studies, e.g. through cognitive testing or by conducting a subset of the surveys initially and reviewing the responses for evidence of problems before fielding the survey more widely. These methods can provide tools to improve questionnaires and other research instruments, for example, by testing comprehension of survey and lab questions. Through this testing, questions can be tailored to increase the accuracy of the collected information and to reduce respondent burden by ensuring that large-scale information collection is optimized.

Research under this generic clearance is important in order to 1) improve the data collection instruments and other information (such as disclosures) created by the Bureau, 2) increase the accuracy of the data and, in turn, of the research studies produced by the Bureau, including studies that may be factored into some policy decisions, 3) reduce the burden of completing or administering survey instruments for respondents and interviewers, 4) increase response rates in surveys, 5) increase the ease of use of the Bureau’s programs and products, and 6) improve the scientific accuracy of Bureau studies.

The information collected in this generic clearance will be used by the Bureau to evaluate and improve the quality of instruments and disclosure forms. In general, data collected under this clearance will not alone be used as official federal agency statistics. Data may be made public, when appropriate, in research reports, presentations related to study methodology, at professional meetings, in publications on the Bureau’s website, and in professional journals.

3. Use of Information Technology

The Bureau will employ information technology as appropriate to reduce the burden of respondents who agree to participate in its research. Computer-assisted participation will be used when feasible and appropriate, including online studies. In general, the Bureau intends to incorporate electronic data-collection techniques whenever possible, particularly since in many instances they are likely to impose the lowest burden on respondents and may be the most cost- effective. However, the extent to which the Bureau will use electronic data collection for a given project will be determined by weighing several factors, including response rates and costs. For example, evidence to date suggests that web-only data collections tend to have lower response rates than surveys conducted by mail or phone.³ The Bureau expects that the most effective mode, including mixed-mode combinations, may also vary with the complexity and sensitivity of the information to be collected.

4. Efforts to Identify Duplication

The Bureau will explore methodological, psychological and cognitive methods-related issues to answer questions that are not satisfactorily addressed in the relevant research literature. A review of the existing relevant literature will be undertaken before proceeding with projects. To the extent possible, the Bureau will make use of existing information, including information from previous studies. However, this information often is not sufficient to refine our study questionnaires and disclosures without conducting additional research.

5. Efforts to Minimize Burdens on Small Entities

All of our research projects will be limited to individual consumers, not businesses, so there will be no impact on small businesses or other small entities.

6. Consequences of Less Frequent Collection and Obstacles to Burden Reduction

The planned collection of data under this generic clearance will allow the Bureau to improve larger scaled data collections before they are administered, reducing burden by increasing the effectiveness of the larger information collection. Failing to scientifically investigate the data collection process can lead to significant costs in terms of data quality and burden to respondents, interviewers, and data users alike during the final data collection and analysis of results. For example, survey questions or disclosures that are not written clearly could lead to biased responses, high rates of item nonresponse, or misunderstanding of the information.

7. Circumstances Requiring Special Information Collection

There are no special circumstances anticipated for the information collections under this generic clearance, they will be consistent with the applicable guidelines contained in 5 CFR 1320.5(d)(2).

8. Consultation Outside the Agency

In accordance with 5 CFR §1320.8(d)(1), the Bureau published a notice in the Federal Register allowing the public 60 days to comment on the proposed reinstatement of this collection of information. No comments were received. Additionally, in accordance with 5 CFR §1320.5(a)(1)(iv) the Bureau has published a notice in the Federal Register allowing the public 30 days to comment on the submission of this information collection request to the Office of Management and Budget.

9. Payment or Gifts to Respondents

Respondents for activities conducted in the laboratory (for example, cognitive interviews and focus groups) under this clearance will receive a small stipend. This practice has proven necessary and effective in recruiting subjects to participate in this small-scale research, and is also employed by other Federal agencies when conducting cognitive research. The incentive for participation in a cognitive interview is $40, and for participation in a focus group is $50-$75. The Bureau may provide smaller incentives than these amounts at its discretion; however, any requests for non-standard amounts must be justified in writing to OMB. The amount of any incentive or remuneration requested and the justification for the amount will be provided in each collection request submitted to OMB under this generic information collection plan.

10. Assurances of Confidentiality

The Bureau shall treat the information collected in accordance with applicable federal law, including, but not limited to the Bureau’s confidentiality rules, 12 CFR Part 1070, and the federal laws and regulations that apply to federal agencies for the protection of privacy, confidentiality, security and integrity.

The Bureau uses best practices of social science research design to inform the notice and consent vehicles. When required and when practicable, the Bureau provides notice to individuals that explain how their information will be used through appropriate vehicles, such as Privacy Notices, Privacy Act Statements or Informed Consent forms. Such notice is typically made available prior to the collection of information and explains whether the information is mandatory or voluntary; whether there are any opportunities to consent to sharing and submission of information; how the information will be secured and when a System of Records is created under the Privacy Act. In some instances, research may be based on observations or on information collected directly from a third party, and notice and consent may not be applicable.

When the Bureau collects information from third parties, commercial sources, and public databases, individuals to whom information may pertain do not have the opportunity to consent to uses, decline to provide information, or opt out directly to the Bureau. In such cases, it is those third parties’ responsibility to provide any opportunity that may be required to consent, decline, or opt out of how their information may be used. As part of the due diligence process, the Bureau reviews the privacy policies or other public disclosures from the third-parties regarding their use of the information to verify that there are no contradictions with the research, and the Bureau encourages the development of notice and consent opportunities. In these instances, the Bureau will ensure that appropriate privacy provisions are included in contract language, Memorandum of Understanding (MOUs) and Statements of Work (SOW), as applicable. The Bureau also evaluates the potential privacy risk and harm to individuals of specific research relative to that authorized purpose, and vets research proposals to ensure that they serve an authorized purpose.

The information collected under this generic clearance may include direct identifying personally identifiable information (PII), depending on the type of project. Typically direct identifying PII consists of contact information used to facilitate administration of research, and is separated from response data generated from the research. When the Bureau does use direct identifying PII in analyses, it does so in order to a) match across datasets; b) update data sets; c) weight datasets in order to make generalizations regarding a given population; and d) contact potential recipients of our studies to solicit information on consumer experiences with consumer financial products and services.

Conducting these research studies implicates privacy concerns because a breach of confidentiality, or re-identification, could result in an individual suffering harm. To reduce the risk of breaches of confidentiality, the Bureau designs recruitment materials so as not to disclose sensitive information about those it seeks to recruit, and uses appropriate security controls to protect information used in research. There is also risk related to misuse of information collected for research. Misuse might involve secondary types of research that are incompatible with the purposes of the initial collection, or a use of the information that individuals do not understand or to which they have not provided consent.

To reduce the risk of misuse, the Bureau minimizes access to PII based on need-to- know and stipulates, in most instances, that contractors that collect data on behalf of the Bureau remove or redact all direct identifying PII, as defined by the Bureau’s privacy office, before transmitting data to the Bureau. In general, the Bureau will only keep response data stripped of direct identifying PII. Any contractor staff assigned to the project also sign confidentiality agreements. When appropriate, research results will be presented in aggregated form to protect the confidentiality of firms or consumers, and any publicly released version of data will use disclosure protection techniques (e.g., rounding, imputation, exclusion of some variables, aggregation of categorical responses) to minimize the risk of releasing personally identifiable or otherwise sensitive information (12 CFR 1070.40 et seq.). The Bureau treats the information collected from participating persons and institutions in a manner consistent with our confidentiality regulations, and all data and analyses are subject to legal and privacy review prior to their release.

The Bureau also evaluates the potential privacy risk and harm to individuals of specific research relative to that authorized purpose, and vets research proposals to ensure that they serve an authorized purpose. Testing research will be conducted consistent with the Privacy Act and the E-Government Act. The requisite SORNs and PIAs will document the collection, use, disclosure, and retention of PII; the technical, administrative, and physical controls used to minimize privacy risks. The SORNs and PIAs will so be updated to reflect any changes. The Bureau has two SORNs that will cover testing research: CFPB.021 Consumer Education and Engagement Records, 77 FR 60382; and CFPB.022 Market and Consumer Research Records, 77 FR 67802. Specific details regarding information handling will be specified in individual submissions under this generic clearance, but will conform to these broad guidelines. Issues of confidentiality and privacy, as applicable, will be included with each request submitted to OMB under this generic information collection plan. The Consumer Experience Research PIA covers the data that will be collected under this clearance.⁴

11. Justification for Sensitive Questions

Questions about an individual’s finances, for example, what their income is or how much their mortgage costs each month, are commonly considered sensitive. Nonetheless, the Bureau often must ask these questions in order to understand consumer behavior and to recognize financial trends and emergent risks relevant to consumers. The Bureau believes that these questions are justified by their central importance to the Bureau’s mission.

In addition, some people may consider questions about race and other demographic information to be sensitive. The Bureau is, however, mandated to enforce fair lending laws and to identify risks to vulnerable populations, including service members, older Americans, and lower-income consumers. For this reason, the Bureau often needs to ask these types of questions. For information collections involving questions of race or ethnicity, the Bureau will follow the OMB standards for Classification of Federal Data on Race and Ethnicity (Federal Register, October 30, 1997, Volume 62, Number 210, pages 58781-59790).

Respondent participation will be voluntary for each information collection distributed under this generic clearance, and participants will be made aware of this fact. All respondents shall be free to opt-out of a data collection at any time and for any reason. When possible, administrative data for the respondents and non-respondents will be used to analyze non-response bias for information collections.

The Bureau will ensure that a citation is made to any applicable SORN and PIAs are published as appropriate.

12. Estimated Burden of Information Collection

For research projects like surveys and controlled trials in field and economic laboratory settings, the Bureau plans to rigorously test instruments or interventions prior to the full data collection. Currently the Bureau plans to conduct up to two surveys, two controlled trials in the field, and four controlled trials in the lab each year, and in many cases the Bureau will want to test components of each of these before conducting the full data collection. On average, testing of surveys will require 600 participants each year for 60 minutes each, while studies in field and lab contexts will require 50 and 20 participants, respectively, for an hour and a half each.

For disclosure development studies, the Bureau generally conducts iterative cognitive testing of disclosure language and design as an initial step in its disclosure development and testing process. Such iterative testing is conducted with a small number of consumers over several rounds of testing, enabling the Bureau to improve the disclosures in response to feedback from consumers. The number of rounds will vary based on factors such as the complexity of the information being communicated through the disclosure and whether any prior research exists related to the disclosure. The Bureau estimates that it will conduct three such testing projects each year, with 10 participants involved in 5 rounds of testing for one hour each, on average. The Bureau will also conduct cognitive testing of quantitative disclosure research under this clearance, approximately one per year, requiring 200 participants for one hour each.

Estimated burden hours under this generic clearance total 5,460.

Process	Number of respondents	Number of responses per respondent	Number of annual responses	Average burden per response (hours)	Total burden (hours)
Testing for Surveys	1,200	1	1,200	1	1,200
Testing for Field Studies	100	1	100	1.5	150
Testing for Lab Studies	80	1	80	1.5	120
Testing for Disclosure Forms	350	1	350	1	350
Annual Total:	1,730	……………	1,730	…………..	1,820
Three Year Total:	5,190	……………	5,190	…………..	5,460

Burden Hour Estimate for Testing of Surveys, 3-yr period

Number of Projects	Number of Participants in a Project	Hours per Participant	Subtotal
6	600	1	3,600
			3,600

Burden Hour Estimate for Testing of Economic Studies in the Field, 3-yr period

Number of Projects	Number of Participants in a Project	Hours per Participant	Subtotal
6	50	1.5	450
			450

Burden Hour Estimate for Testing of Economic Studies in the Lab, 3-yr period

Number of Projects	Number of Participants in a Project	Hours per Participant	Subtotal
12	20	1.5	360
			360

Burden Hour Estimate for Cognitive Testing of Disclosure Forms, 3-yr period

Type of Projects	Number of Projects	Number of Participants in a Project	Hours per Participant	Subtotal
Qualitative	9	50	1	450
Quantitative	3	200	1	600
				1,050

There will be no financial burden on the respondents. For each specific collection request submitted under this generic plan, a detailed estimate of the burdens of that particular request will be submitted.

13. Estimated Total Annual Cost Burden to Respondents or Recordkeepers

There are no capital/start-up or ongoing operation/maintenance costs associated with this information collection. If there are postage costs to return a survey questionnaire, it will be pre- paid by the contractor. For other methods of participation, including phone, online, or in-person, respondents should not bear costs. In tests of instruments or procedures of controlled trials in laboratory or field settings, the Bureau does not anticipate any cost burden to respondents.

14. Estimated Cost to the Federal Government

The costs of pilot testing and cognitive testing research projects under this generic clearance will vary based on the number of individuals in the study, the method of testing used, and the size of any payments to incentivize participation. We expect the cost of each project to range from $15,000 to $200,000. Individual project costs will be provided in each collection request.

15. Program Changes or Adjustments

	Total Respondents	Annual Responses	Burden Hours
Total Burden Requested	5,190	5,190	5,460
Current OMB Inventory	0	0	0
Difference (+/-)	+5,190	+5,190	+5,460
Program Change	+5,190	+5,190	+5,460
Discretionary
New Statute
Violation
Adjustment

Since this is a reinstatement of a previous approved and discontinued collection, there is no current inventory for this collection. When last approved, the inventoried burden for this generic information collection place was 7,890 responses and 8,235 hours. Based on research the Bureau has conducted over the last several years, and its plans for upcoming research studies, the Bureau has better estimates of what projects it will pursue in the next three years.

16. Plans for Tabulation, Statistical Analysis, and Publication

This generic clearance will be used to test instruments or interventions prior to the full data collection for surveys and controlled trials in field and laboratory settings. The purpose of data collected under this generic clearance is to improve the quality of data collection by examining the psychological and cognitive aspects of collection methods and procedures. For

this reason, results from data collected under this generic clearance may or may not be published, depending on the project. For example, if the methodology appears sound, pilot data may be used to augment data collected during the full research project. When results are published, the data may also be used for research and development reports or be included as a methodological appendix or footnote in a report containing data from a larger data collection effort. In addition, the methodological results of this research may be prepared for presentation at professional meetings or for publication in professional journals.

The Bureau’s research projects are intended to expand scientific understanding of consumer decision making in financial domains. Research findings of full research projects generally will be published as technical articles that are publicly disseminated as working papers, peer-reviewed journal articles, Bureau white papers, or Bureau research briefs. Research findings of cognitive testing of disclosures may be published by Bureau contractors. Due to the nature of this clearance, there currently is no definitive or tentative schedule.

17. Display of Expiration Date

The OMB control number and expiration date associated with this PRA submission will be displayed on the Federal government’s electronic PRA docket at www.reginfo.gov and on all instruments associated with information collections approved under this collection plan.

18. Exceptions to the Certification Requirement

The Bureau certifies that this collection of information is consistent with the requirements of 5 CFR 1320.9, and the related provisions of 5 CFR 1320.8(b)(3) and is not seeking an exemption to these certification requirements.

1 One of the “primary functions” of the Bureau of Consumer Financial Protection (“Bureau”) as set forth in section 1021(c) of the Dodd-Frank Act is “collecting, researching, monitoring and publishing information relevant to the functioning of markets for consumer financial products and services.” Effectively performing that function is integral to assuring that the Bureau achieves the purposes and objectives set forth in Sections 1021(a) and 1021(b) of the Act.

2 DFA Section 1013(b)(1) mandates that the Bureau have research staff whose responsibilities include “[r]esearching, analyzing, and reporting on (A) developments in markets for consumer financial products or services, including market areas of alternative consumer financial products or services with high growth rates and areas of risk to consumers; (B) access to fair and affordable credit for traditionally underserved communities; (C) consumer awareness, understanding, and use of disclosures and communications regarding consumer financial products or services; (D) consumer awareness and understanding of costs, risks, and benefits of consumer financial

products or services; (E) consumer behavior with respect to consumer financial products or services, including performance on mortgage loans; and (F) experiences of traditionally underserved consumers, including un‐banked and under-banked consumers.

3 Roger Tourangeau, Fredrick Conrad, and Mick Couper (2013), “The Science of Web Surveys.” USA: Oxford University Press.

4 Consumer Experience Research PIA https://s3.amazonaws.com/files.consumerfinance.gov/f/201406_cfpb_consumer-experience-research_pia.pdf; and Market Analysis of Administrative Data Under Research Authorities PIA https://s3.amazonaws.com/files.consumerfinance.gov/f/201312_cfpb_pia_admin-data-research.pdf.

11