2014_08_28_CLEAN Part B

CONSUMER FINANCIAL PROTECTION BUREAU
INFORMATION COLLECTION REQUEST – SUPPORTING STATEMENT B
COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
TELEPHONE SURVEY EXPLORING CONSUMER AWARENESS OF AND
PERCEPTIONS REGARDING DISPUTE RESOLUTION PROVISIONS IN
CREDIT CARD AGREEMENTS
(OMB CONTROL NUMBER: 3170-XXXX)
1.

Respondent Universe and Selection Methods

A telephone survey (including both landline and cellphones) will be conducted to
explore consumer awareness of and perceptions regarding dispute resolution
provisions in consumer credit card agreements. The respondent universe for this
survey will consist of the approximately 160 million Americans who own credit
cards. The survey will initially reach out to 10,500 potential respondents, in an
effort to ultimately collect information from approximately 1,000 credit card holders
(assuming a response rate of approximately 15%) (and an estimated 500 non-credit
card holders). Interviews will be conducted in English or Spanish, according to the
respondent’s preferred language. 1
The proposed telephone script is attached as Appendix A, revised based on user
testing of the original and comments received as part of the 60-day Federal Register
Notice process; two consultation rounds with interested parties representing both
financial service organizations and consumer groups; focus group testing with two
sets of consumer panels; and extensive user testing. The justification of each
question is attached as Appendix B.
The CFPB expects that the survey sample will be selected through random-digit
dialing of residential and cellular telephones to obtain data from a target sample
size of approximately 1,000 credit card holders across the country. Unlike an
1 In 2010, the U.S. Census released a report on language use in the United States in 2007. Language
Use in the United States: 2007. United States Census Bureau. Released April 2010. The study found

that 80% of the population 5 years and older spoke only English at home. Regarding the remaining
20%, the population that spoke a language other than English at home, three-fifths spoke Spanish or
Spanish Creole (defined as including Spanish, Spanish Creole, and Ladino). The remaining
population was in a group of diffuse categories: “Other Indo-European languages,” “Asian and Pacific
Island languages,” or “Other languages.”
Spanish language interview templates will be produced following OMB approval of this proposal and
will be subsequently provided to OMB.

Internet-based or mailed survey, a telephone call requires real-time interaction,
which minimizes the possibility that respondents will supplement their knowledge
before responding to survey questions by searching the Internet or reviewing their
credit card agreements.
2.

Information Collection Procedures

The sample design for conducting the approximately 1,000 telephone interviews
with credit card holders will be based on a list-assisted random digit dialing
telephone sample. In such case, analyzing the vast majority of the survey questions
would involve calculating ratios based on data from the survey, for example how
many interviewees (out of 1,000), picked a particular option. In general, confidence
intervals are not known without assuming a particular functional form for the
standard error of the population. In the case of ratios, and after making appropriate
assumptions, one can use a Bernoulli distribution. The maximum variance, when
both variables are equally likely, is 0.25. 2 Based on that, a 95% confidence interval
can be computed, with a confidence interval of at most +/- 3.1%. 3 The confidence
interval is actually narrower, given that answers are not equally likely. The exact
bounds of the 95% confidence interval, however, will not be known until after the
data are collected. The confidence interval described above is applicable for testing a
single hypothesis. The CFPB will utilize the Bonferroni correction, or other widely
acceptable and similar statistical methods, to arrive at the correct confidence
intervals while testing many hypotheses simultaneously. 4
Regarding landline telephone numbers, telephone numbers are ten digits long
(AAA-EEE-XXXX), where the first three numbers are the area code, the second
three are the exchange, and the last four digits are the number within the
exchange. The area code, three-digit exchange numbers, and the first two digits of
the four-digit suffix specify a “100-bank” containing 100 telephone numbers. ICF
International will sample telephone numbers (using randomly-generated two-digit
strings to complete the telephone number) from 100-banks that have been identified
2

This calculation is based on a binomial distribution where both outcomes are equally likely.

See Peter Stopher, Collecting, Managing, and Assessing Data Using Sample Surveys, Cambridge
University Press, Cambridge UK (2012). There are 1,000 respondents, the 95% level of confidence is
equivalent to +/- 1.96 standard errors based on the normal distribution of the ratio statistic, and the
maximum variance of the population is 0.25. This results in a confidence interval of the estimated
1.96
≈ .031.
statistic +/- x, where 𝑥 =
3

�1000/0.25

4 If researchers are testing a sufficient number of null hypotheses, then at least one of them is likely
to be rejected based on pure luck of the draw. Bonferroni correction, or a similar method, is utilized
to ensure that a statistically significant result is not simply due to researchers testing so many
hypotheses that some just happened to be true.

as containing at least one listed residential numbers, to ensure a relatively high hitrate.
The cellphone sample of telephone numbers will be drawn separately from
telephone exchanges dedicated to cellphones. 5
ICF International will normalize the collected data by geographic region, where
feasible.
The initial sample sizes would be large enough to yield the required number (1,000)
of interviews with credit card holders nationwide.
The RDD frame will be stratified by census region. Sampling would be conducted
independently within each stratum.
ICF International will seek a 30% to 70% sample split between cellphone-only
households and households with a landline (the latter group including both
landline-only households and ‘dual-users’ – those with both a cell and landline
phone). 6 This ratio reflects a balance between great representativeness of the
sample population and cost, as cell phone interviews are significantly more
expensive than traditional landline interviewing, due to legislative restrictions that
require dialing of cell phone numbers and a higher rate of refusals, vis-à-vis
landlines.
ICF International will vary call attempts across the days of the week and times of
day including weekdays, weeknights, and weekends. Groups of calls will be made
at three separate times over a number of weeks, which should improve the
likelihood that those who are travelling during some portion of the collection
Upon initial contact and introduction with cellphone respondents, the interviewer will immediately
ask questions to determine that the person is not in a situation that could pose a safety risk to that
individual (e.g., driving). If the contacted individual is found to be in a situation that could pose a
risk, the interviewer will terminate the call and call back at another time.

5

6 The most recent data published from the National Health Interview Survey in December 2012
shows that 35.8% of American homes relied solely on cellphones. Further, 15.9% of American homes
received “all or almost all” calls on cellphones despite having a landline telephone.

Demographic differences may alter these numbers, though. For example, 60.1% of adults aged 25-29
lived in households with only cellphone telephones. 58.2% of adults renting their home relied solely
on cellphones – twice the rate of adults owning their homes, 23.2%. 75.9% of adults living only with
unrelated adult roommates were in cellphone-only households. Blumberg, Stephen J. and Julian V.
Luke. Wireless Substitution: Early Release of Estimates From the National Health Interview
Survey, January-June 2012. U.S. Department of Health and Human Services, Centers for Disease
Control and Prevention, National Center for Health Statistics. Released 12/19/12 available at
http://www.cdc.gov/nchs/data/nhis/earlyrelease/wireless201212.PDF.

timeframe are available to participate in the survey and mitigate the risk that an
external event alters consumers’ perceptions on a large scale. ICF International will
also set their telephone dialing system to display a toll-free number that consumers
can call back, in the event that consumers “screen” their calls.
ICF International will attempt to contact sample members seven times (e.g., across
multiple call groups), to ensure that the survey does not capture just those easier to
contact and most likely to be home on the first attempt.
When contact is made with an answering machine or voicemail, ICF International
will leave at least two messages during the field period. The message would explain
that the household has been selected to participate in a national study by the CFPB
and ask that the contractor’s toll-free number be called to schedule an interview.
The message will explain that participation is voluntary. The message will also
reference a notice on the CFPB website, so prospective respondents can verify the
legitimacy of the survey. The website would not collect any data from consumers or
offer any tools for scheduling beyond displaying the contractor’s toll-free telephone
number.
Once contact has been made via landline telephone numbers, one adult in the
selected household will be chosen at random for participation in the survey. ICF
International will make the selection based on the household’s “most recent
birthday.” The “most recent birthday” method asks for the credit-card holding adult
within the sampled household who, at the time of respondent selection, has the
most recent birthday (the callers will not request the date of the actual birthday).
Only that randomly-selected person will be eligible to participate in the survey.
Randomization processes will not be used with cellphone respondents, where there
is presumably a greater correlation between telephone numbers and specific
individuals.
The CFPB projects it will obtain approximately 1,500 replies to the survey in order
to gather responses from approximately 1,000 credit card holders. Given the unique
nature of the survey, as described above, it is difficult to predict what the
respondent reply rate will be with a reasonable degree of accuracy. The CFPB
estimates that the response rate will be 15% or less, given that dispute resolution
provisions are unlikely to be salient topics for potential respondents. 7 The effect of
7 Certain commentators raised concerns about the original projections regarding response rates
reflected in the CFPB’s prior submission to OMB. See¸e.g., (AFSA Aug. 6, 2013 pp. 5-6; Shestowsky
and Welsh, Aug. 8, 2013).

The CFPB’s projections are within the range of other estimates of expected response rates. See, e.g.,
Floyd J. Fowler, Jr., Survey Research Methods p.58 (4th Ed. 2009) (“[E]ven conscientious
organizations sometimes achieve response rates in the 30% to 50% range – particularly when the
subject of the survey is not immediately engaging. The difficulty of achieving high response rates is

the reply rate could be minimized, however, by careful study of the demographics of
participants to assure representative weighting across education, race/ethnicity,
age, income, and gender. Where possible, the CFPB will use other metrics to verify
the accuracy of its statistical weighting when testing the representativeness of the
sample. In that regard, the CFPB will be able to use publicly available data sources
describing the general market share of credit card issuers against Question 8, which
asks consumers to identify the “name of the bank, credit union, or company to
which you make payments” regarding the credit card the respondent uses most
often for personal use.
Multiple telephone calls to potential survey respondents, references to the CFPB
website, and the prenotification letter to landline users, should help maximize the
survey reply rate.
The CFPB anticipates that questions about consumer income may result in a
significantly higher non-response rate than any other survey questions. Question
23, which seeks income information, is the second-to-last question, followed only by
a query about the consumer’s gender. Accordingly, data from any “drop outs” may
still be used. Consumers will also be instructed that they may refuse to answer or
otherwise skip any questions that they do not to answer.
Each interviewer will be monitored for quality assurance throughout the course of
the project:
•

Audio monitoring will be used to maintain the quality of interviewer
performance relating to matters such as:
o Initial contact and recruitment;
o Reading the questions as written;
o Reading (or not reading, where appropriate) response categories
according to study specifications;
o Whether open-ended questions are properly probed;

one of the intrinsic limits of [random digit dialing] as a sampling approach.”); Robert M. Groves, et.
al., Survey Methodology pp. 187-88 (2d Ed. 2009) (describing non-response rates for the Survey of
Consumers, conducted by the University of Michigan Survey Research Center, as growing “from
percentages in the 30s in the 1980s to percentages near 60 more recently.” “Again, the lesson is clear:
response rates have declined over time.”). Cf. Panel on a Research Agenda for the Future of Social
Science Data Collection; Committee, et al., Nonresponse in Social Science Surveys: A Research
Agenda p. 1-6 (2013) advanced copy available at http://www.nap.edu/catalog.php?record_id=18293
(“Based on the above evidence, we are able to conclude that non-response rates continue on a longterm upward path, but we are concerned that solid evidence about the reasons for the decline is still
elusive”) (emphasis in original).
As described above, the proposed survey has been simplified and shortened extensively, which
should increase response rates.

•

o Whether ambiguous or unclear answers are properly clarified;
o Avoiding bias, either by comment, rate of speech, or by tone of voice;
and
o General professional conduct.
The interviewers’ data entry will also be monitored to ensure that the
interviewer enters the correct code, number, or verbatim response.

As referenced above, the survey will collect data at three separate points in time
over a series of weeks. In addition to assisting with gathering a wide demographic
sample, this three-part collection also allows for contingency planning, in the event
that design parameters fail to meet expectations in initial data collections. In that
regard, if quality assurance monitoring or initial test results yield extremely
problematic results (such as dramatically inadequate participation), the contractor
may suggest changes to the survey’s questions or format. In such limited
circumstances, additional rounds of data collection may be added to ensure that
consistent standards are used across the entire collection.
The CFPB anticipates that this will otherwise be a one-time survey.
3.

Methods to Maximize Response Rates and Address Issues of Non-Response

As referenced in greater detail above, at least seven attempts to contact potential
respondents will be made before the calls are abandoned. Calls will be made at
different times of day and different days of the week.
A prenotification letter will be sent to those landline sample members for whom an
address can be identified using outside data sources. The letter will emphasize the
voluntary nature of the inquiry and encourage participation by stressing the value
of the survey and how the results will be used. The letter will not, however,
reference credit card agreements or dispute resolution provisions, so that consumers
do not conduct research before the interview. A copy of the proposed advance letter
is attached hereto as Appendix C.
The questionnaire was extensive tested to identify problems that could deter
respondents from completing the survey:
•

It was initially tested internally by the CFPB with a small group of CFPB
employees to identify any problems that could deter respondents from
completing the survey.

•

The Initial Questionnaire was made available with the CFPB’s original
Federal Register Notice and request for comment describing the proposal on
June 7, 2013. (Vol. 78, No. 110, Page 34352.) The CFPB received
approximately 22 comments in response.

•

The CFPB met with two sets of external stakeholders on July 8, 2013 to
discuss their general feedback to the proposal and Initial Questionnaire.

•

In January 2014, the CFPB sought (and subsequently received) OMB
approval to perform consumer focus groups and usability testing relating to a
revised version of the proposed survey.

•

On February 18, 2014, ICF International performed two ninety-minute focus
groups with credit card holders in the Bethesda, Maryland area. The focus
groups explored consumers’ general familiarity with issues raised by the
proposed survey and the vocabulary used by consumers when they discuss
them. The focus group panels were diverse with respect to age, ethnicity, and
education.

•

On March 11 and 12, 2014, the CFPB met with three sets of external
stakeholders to discuss their general feedback on the Initial Questionnaire,
after having revised the document to incorporate information learned from
the focus group testing.

•

From March 24 to March 28, 2014, ICF International user-tested revised
versions of the proposed survey in individual calls with a diverse set of
consumers.

ICF International will emphasize to its interviewers that their manner of approach
at the time of first contact is often the most important factor in securing the
interview. Interviewers will rely on a carefully-designed introduction emphasizing
trust and the importance of the survey, shown in Appendix A. Similarly,
interviewers will be trained on how to answer the most likely questions for the
survey. A hard copy of draft Frequently Asked Questions, shown in Appendix D,
will be provided to each interviewer at their station. Interviewers will be trained to
answer all questions in an open, positive, and confident manner. If a caller is
reluctant, the interviewer could provide a reference to the CFPB website, so
prospective respondents can verify the legitimacy of the survey.
When respondents refuse to participate, the interviewer will seek information
regarding the refusal to participate. The interviewer will enter this answer and any
other details relating to the refusal into the computer-assisted telephone
interviewing software.
We will analyze nonresponse using several different methods:
1. Compare early and late responders to determine whether there are
differences in survey estimates between the two groups. Early responders are

those who respond to the survey on the first few call attempts, while late
responders are those who respond on later call attempts and therefore
required more effort to obtain a completion. The underlying assumption is
that late respondents are similar to nonrespondents with respect to their
distributions on key survey variables. If the late respondents are assumed to
be similar to nonrespondents while early responders are not, then the
difference between early and late respondents can be used to approximate the
difference between respondents and nonrespondents.
2. Review the demographics of the RDD sample in comparison to benchmark
demographics from the American Community Survey and/or Current
Population data. Large differences between the sample and benchmarks
indicate differential nonresponse for population subgroups. If these
subgroups differ with respect to the substantive data, the estimates will be
biased. This analysis is the basis for the calibration weighting (e.g. raking.)
These comparisons will include credit card holders and non-credit card
holders.
3. Evaluate the distribution of credit cards to independent market share data.
The survey collects the type of credit cards from each respondent. The
weighted estimate of credit cards from the sample is an estimate of the total
active credit cards for the household adult population. Large differences
between the sample distribution of credit cards (e.g. Visa, Mastercard,
American Express, Discover) and the independent market share data could
indicate bias.
4.

Testing of Procedures or Methods

As discussed above, ICF International used focus group testing, followed by user
testing to gather information and tailor survey questions.
Pre-test interviews will also be conducted with nine or fewer respondents prior to
formal administration of the survey, in order to test all of the survey systems, the
survey instrument, and the computer-assisted telephone interviewing software. 8
The pre-test will be conducted under conditions identical to the planned conditions
for the administration of the survey. The issues to be addressed in the pre-test will
include: question sequencing and skip patterns; completion time; the randomization
of certain question elements; checking that invalid responses are not entered; and
the full range of procedures.
Following the pre-test interviews, the interviewers will report problems or
observations they have with the project team. That feedback, combined with other

Such testing will include providing all possible responsible categories for each question to identify
embedded errors, as well as skipping issues.
8

data gathered from the pre-test, will be used to revise procedures, the computerassisted telephone interviewing software, and the survey questions, as appropriate.
5.

Contact Information for Statistical Aspects of the Design

The following individuals have reviewed technical and statistical aspects of
procedures that will be used to conduct the proposed survey.
Will Wade-Gery
Division of Research, Markets and Regulations
Consumer Financial Protection Bureau
(202) 435-7700
[email protected]
Alexei Alexandrov
Division of Research, Markets and Regulations
Consumer Financial Protection Bureau
(202) 435-7700
[email protected]
Kelvin D. Chen
Division of Research, Markets and Regulations
Consumer Financial Protection Bureau
(202) 435-7700
[email protected]