3170-xxxx_ss_part_b(3-26-13)

CONSUMER FINANCIAL PROTECTION BUREAU
INFORMATION COLLECTION REQUEST – SUPPORTING STATEMENT
CLEARANCE FOR CONSUMER ATTITUDES, UNDERSTANDING, AND
BEHAVIORS WITH RESPECT TO FINANCIAL SERVICES AND PRODUCTS
(OMB CONTROL NUMBER: 3170-XXXX)
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
1. Respondent Universe and Selection Methods
RECRUITMENT METHODOLOGY
Recruitment to KnowledgePanel® was first conducted in 1999 using random-digit dial (RDD)
methods. At that time, all recruited households were given a WebTV to use for answering
surveys. In August 2002, Knowledge Network (KN) began allowing households to use their own
computers connected to the Internet for taking surveys. Since January 2009, Windows-based
laptop computers, and now netbooks, are provided to non- Internet households instead of
WebTV units.
Until recently, KnowledgePanel’s probability-based recruitment had been based exclusively on a
national RDD frame. In 2009, KN initiated the use of an address-based sample (ABS) frame to
first supplement the RDD frame and subsequently replace it. This was in response to the growing
number of cell phone only households, including many young adults and minorities, who are
outside the traditional RDD landline telephone frame. Also, this switch was motivated by
declining RDD response rates.
ABS involves probability-based sampling of addresses from the U.S. Postal Service’s Delivery
Sequence File. Randomly sampled addresses are invited to join KnowledgePanel through a series
of mailings (English and Spanish materials) and by telephone follow-up to non-responders when
a telephone number can be matched to the sampled address. This ensures that households who do
not have a computer in the home are invited to participate in the panel. Invited households can
join the panel by one of several means: completing and mailing back an acceptance form in a
postage-paid envelope; calling a toll-free hotline staffed by bilingual recruitment agents; or going
to a dedicated KN recruitment Web site and completing the recruitment information online.
The address sampling, conducted throughout the year, is done without replacement. Addresses
with matched telephone numbers from the former RDD recruitment samples (for the last five
years of calling) are also removed to eliminate duplication. In addition, in 2008 KN constructed
KnowledgePanel LatinoSM taking quality of online panel representation to the next level by
providing netbooks and Internet service for the roughly 40% of Latinos who do not have Internet
access at the time of recruitment.
For all new panel members, demographic information such as gender, age, race/ethnicity,
income, education, and for Latino members, language proficiency are collected in an online
“profile” survey. This information is used to determine eligibility for specific studies and
eliminates the need for gathering basic demographic information on each panel survey. After this

survey is completed, the panel member is regarded as active and ready to be sampled for other
surveys. Additionally, for all Hispanic panel members Knowledge Networks ask a series of
questions that can be used by its clients to apply to an acculturation scale. Such questions include
media use, country of birth, number of years in the U.S. and other attitude and values questions.
As of May 2010, all of the active KnowledgePanel households initially provided a MSN® TV
unit had been transitioned to taking KnowledgePanel surveys on a netbook computer provided
by Knowledge Networks. All households who did not have a computer heading into the panel,
are given training and access.
PANEL SURVEY SAMPLING
Once panel members are profiled, they become “active” for selection for specific surveys.
Profiling consists of collecting demographic information such as gender, age, race/ethnicity,
income, education, and for Latino members, language proficiency in an online survey for new
panel members. Samples are drawn from among active members using a probability proportional
to size (PPS) weighted sampling approach. Customized stratified random sampling based on
profile data is also conducted, as required by specific studies.
In September 2007, KN was assigned a patent (U.S. Patent No. 7,269,570) for its unique
methodology for selecting multiple online survey samples from a panel. The selection
methodology, which has been used by KN since 2000, assures that multiple sequential
KnowledgePanel samples from a finite panel membership will each reliably represent the U.S.
population.
This sampling methodology was developed by KN in recognition of the practical issue that
different survey samples may target different panel subpopulations. It is not unusual that only
panel members with certain characteristics are selected for a survey. This selectivity can skew
the remaining panel membership demographics and affect the representativeness of later survey
samples. The patented sampling methodology was developed to correct for this in panel
sampling; see U.S. Patent No. 7,269,570 for more information.
This is the first time this information is being collected, but CFPB anticipates a 60% completion
rate (sometimes called cooperation rate), similar to other surveys conducted using
KnowledgePanel (please see Appendix A “Response Rate and Survey Completion Rate” for an
explanation of response rate and completion rate calculations for a panel). Panel members have
already agreed to take online surveys when they became panel members and therefore are more
likely to respond. A survey invitation will be emailed to the selected sample along with three
email reminders to non-respondents.
The typical response rate calculation required by OMB (as discussed in OMB Statistical
Standards page 14), is similar to the AAPOR standard response rate #3. This response rate
calculation is widely used for surveys,. Appendix A “Response Rate and Survey Completion
Rate), explains how response rates for panels are calculated differently because of the four stages
of recruiting for any one survey, starting from panel recruiting and ending with survey recruiting.
As described in Appendix A, the cumulative response rate for this survey is expected to be about

4%.
2. Information Collection Procedures
See PANEL SURVEY SAMPLING above.
STATISTICAL WEIGHTING
KnowledgePanel sampling begins as an equal probability sample that is self-weighting with
several enhancements incorporated to improve efficiency. Since any alteration in the selection
process is a deviation from a pure equal probability sample design, statistical weighting
adjustments are made to the data to offset known selection deviations. These adjustments are
incorporated in the sample’s base weight.
There are also several sources of survey error that are an inherent part of any survey process,
such as non-coverage and non-response due to panel recruitment methods and to inevitable panel
attrition. These sources of sampling and non- sampling error are addressed using a panel
demographic post-stratification weight as an additional adjustment.
Even with this weighting, it is important to note that the panel includes those people who use the
internet, but does not include those who would refuse even a free internet connection. Therefore,
the results of the survey will be applicable to those who are willing to use the internet, whether at
home (paid or for free) or at any other location, such as at work or a library. To better understand
any potential differences in this group who refuse to use the internet, we will analyze differences
in the responses of those who recently got an internet connection when they joined the panel and
in differences in the responses of those who use the internet less frequently.
3. Methods to Maximize Response Rates and Address Issues of Non-Response
Every effort is made to obtain responses from all invited respondents. However, some degree of
non-response is expected in every survey. In order to minimize non-response, KnowledgePanel
employs the following procedures:
•
•
•
•
•

Ensure that all surveys contain clear language about their intent, use and purpose;
Ensure that respondents receive a survey instrument that is well structured and contains
only those questions that are necessary for the intended purpose;
Support survey respondents with staffed help lines during virtually all daytime hours;
Provide periodic reminder emails to alert and remind respondents that they have survey
invitations and how to access them; and,
Provide an ongoing loyalty program that incentivizes responses to surveys, providing a
“Thank You” for their efforts. The standard incentive has two classes: those who provide
their own computer and ISP and those who use a computer and ISP provided as a part of
their recruitment into KnowledgePanel. Those who use their own computer and ISP
receive 1,000 loyalty points for completing this survey. 1,000 loyalty points is equivalent
to $1.00 and is deposited into their account for future use. When a computer and ISP is
provided to a respondent, access to the internet when not taking a KnowledgePanel

survey represents their standard incentive.
In addition, the demographic information for those who do not complete a survey is available to
researchers to allow for examination of the demographics of non-respondents. A full nonresponse analysis will not be conducted due to the nature of this survey. The core objective of the
survey is to measure consumers’ awareness, understanding, and behaviors with respect to consumer
financial services and products, and to use this knowledge to inform agency consumer engagement
choices. The survey is not at all intended to inform public policy decisions, nor is it intended to be
representative of the American public as a whole, but is only intended to provide insights for the
agency to guide consumer engagement choices.
As discussed in #1 above, the typical response rate calculation required by OMB (as discussed
in OMB Statistical Standards page 14), is widely used for surveys. Appendix A “Response Rate
and Survey Completion Rate), explains how response rates for panels are calculated differently
because of the four stages of recruiting for any one survey, starting from panel recruiting and
ending with survey recruiting.

4. Testing of Procedures or Methods
This survey will be pretested with 25 respondents. The purpose of this pretest is to ensure that
the survey functions as expected and that the survey requires the expected amount of time to
complete.
5. Contact Information for Statistical Aspects of the Design
Individuals consulted on statistical aspects of the design and analyzing information:
•
•
•
•

Dr. Peter Webb, Vice President, Pacific Consulting Group, 650-327-8108
Beruria Novich, Senior Project Manager, Pacific Consulting Group, 650-327-8108
Simone Berkowitz, Senior Analyst/Project Manger, Pacific Consulting Group, 650-3278108
Wei Tang, Senior Analyst/Project Manager, Pacific Consulting Group, 650-327-8108

Individual consulted on statistical aspects of the sample:
•

Dr. Charles DiSogra, Chief Statistician, GfK Customer Research, 650-289-2185

Full biographies and curricula vitae are included as an appendix to this Supporting Statement.

Appendix:
Response Rate and Survey Completion Rate
During last week’s discussion, the difference between Response Rate and Survey Completion
Rate was discussed.
Calculation of a response rate for a freshly recruited sample survey is fairly simple. As published
in Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys, THE
AMERICAN ASSOCIATION FOR PUBLIC OPINION RESEARCH (AAPOR), Revised 2011,
the calculation of the response rate is the number of completed interviews divided by the number
of eligible reporting units in the initial sample.
Response rates - The number of complete interviews with reporting units divided by the
number of eligible reporting units in the sample. The report provides six definitions of
response rates, ranging from the definition that yields the lowest rate to the definition that
yields the highest rate, depending on how partial interviews are considered and how cases
of unknown eligibility are handled.
The response rate for a Panel sample is a bit more complicated. Respondents must first agree to
join a panel in order to participate in an ongoing array of surveys. In the case of
KnowledgePanel®, then must then complete an initial Profile survey. Some will ultimately leave
the Panel, either by choice or when they’ve been Panelists for a number of years. Finally, they
must complete the specific survey for which the calculation is made. Thus, the Response Rate is
a composite of the following items:
household recruitment rate x
household profile rate x
panel retention rate x
survey completion rate
The composite of these items is likely to result in Response Rate calculations in the 6% - 8%
range for panels more than 10 years old (KnowledgePanel was initially recruited in 1999). The
specific values for each stage of the recruitment can only be calculated when the sample is
identified.
As an example, the following calculation was performed for an article entitled Computing
Response Metrics for Online Panels by Mario Callegaro and Charles DiSogra (Public Opinion
Quarterly, Vol. 72, No. 5 2008, pp. 1008–1032).
HOUSEHOLD RECRUITMENT RATE (RECR) = 0.326
HOUSEHOLD PROFILE RATE (PROR) = 0.568
HOUSEHOLD RETENTION RATE (RETR) = 0.390
STUDY COMPLETION RATE (COMR) = 0.845
This example resulted in a Response Rate of 6.1%. If the first three stages of the calculation for
the current survey remained valid and the Study Completion Rate was 60%, the Response rate
would be 4.3%. It is unlikely that any two surveys will have identical values for the first three
stages but the results are likely to be similar.
It should be noted that a recent Pew Research Center report (Assessing the Representativeness of
Public Opinion Surveys, May 15, 2012) noted recent Random Digit Dial telephone typical
response rates of 9% even with the known advantage of social pressure to participate associated
with interaction with a human interviewer.
It should be understood that respondents to a specific Panel survey are also a well-known group,

having completed multiple Profile surveys during their Panel tenure. The characteristics
(demographic, health, financial, etc.) are well known. These same characteristics are known for
those who choose not to respond to the specific survey. Thus survey responders and nonresponders can be evaluated for demographic and other differences quite efficiently. (Note: this
can also be done for any stage of the recruitment effort but the cost and complexity will be
considerable. Prior efforts of this type have not resulted in identifiable non-response bias.)
These Profile data are also valuable in identify potential respondents to the specific survey.
Panel surveys can target the specific set of individuals targeted for the survey, thereby potentially
reducing screening effort and respondent burden of the survey. This is specifically true for
surveys targeting identifiable subpopulations where the Profile information is available to
directly target the sample of interest.
Survey Completion Rates, on the other hand, is fairly simple. It is the number of qualified
responses divided by the number of invitations sent. Again from the AAPOR publication:
Cooperation rates - The proportion of all cases interviewed of all eligible units ever
contacted. The report provides four definitions of cooperation rates, ranging from a
minimum or lowest rate, to a maximum or highest rate.
For KnowledgePanel surveys, this value generally ranges from 60% to more than 80% and is
influenced by a number of items including:
• Survey content and salience
•

Survey length (shorter is better)

•

Field period (longer is better)

•

Survey sponsorship (name recognition generally improves cooperation)

•

Use of survey specific incentives (incentives are in the form of loyalty points or
sweepstake prizes)

Our assumption of a relatively low cooperation rate is based upon the survey content, length, a
relatively unknown sponsor and the lack of survey specific incentives. We believe this is a
conservative estimate. We might expect to achieve greater cooperation if we can remain in the
field for a longer period.