NORC AmeriSpeak Technical Overview

TECHNICAL OVERVIEW OF THE AMERISPEAK® PANEL
NORC’S PROBABILITY-BASED HOUSEHOLD PANEL
Updated February 18, 2019
Prepared by J. Michael Dennis, Ph.D.
Funded and operated by NORC at the University of Chicago, AmeriSpeak® is a probability-based panel
designed to be representative of the US household population. Randomly selected US households are
sampled with a known, non-zero probability of selection from the NORC National Frame and address-based
sample, and then contacted by US mail, telephone interviewers, overnight express mailers, and field
interviewers (face to face). AmeriSpeak panelists participate in NORC studies or studies conducted by
NORC on behalf of NORC’s clients.
In 2018, the AmeriSpeak Panel expanded to approximately 30,000 households and will expand to 35,000
households in 2019. The AmeriSpeak Panel includes sample support for surveys of various segments
through AmeriSpeak Latino, AmeriSpeak Teen, and AmeriSpeak Young Adult (which includes an oversample
of African Americans, Hispanics, and Asians age 18-34). AmeriSpeak is also the probability-sample source
for TrueNorthTM, which combines probability-based AmeriSpeak and non-probability online samples using
calibrating statistical weights derived from AmeriSpeak, the American Community Survey, and other data
sources. 1
Sample Frame
In order to provide a nationally representative sample, AmeriSpeak leverages the NORC National Frame,
which provides sample coverage for over 97 percent of the U.S. households. The 2010 National Frame used
a two-stage probability sample design to select a representative sample of households in the United States.
The first stage—the sampling unit—is a National Frame Area (NFA), which is either an entire metropolitan
area (made up of one or more counties) or a county (some counties were combined so that each NFA
contains a population of at least 10,000). The largest NFAs with a population of at least 1,543,728 (0.5
percent of the 2010 Census U.S. population) were selected with certainty; these areas have a high-population
density, and are dominated by tracts with street-style addresses. These areas contain 56 percent of the
population within 8 percent of the geographic area of the United States. The remaining areas were stratified
into areas where street-style addresses predominate, and the remaining areas, which are less likely to have
street -style addresses. The latter stratum (“rural” areas) comprises 81 percent of the geographic area, but
only 14 percent of the population.
Within the selected NFAs, the second stage sampling unit is a segment, defined in terms of either Census
tracts or block groups, containing at least 300 housing units according to the 2010 Census. A stratified
probability sample of 1,514 segments was selected with probability proportional to size. For most of the
For more information, see “Estimation Methods for Nonprobability Samples with a Companion Probability Sample”
authored by Michael Yang, N. Ganesh, Edward Mulrow, and Vicki Pineau. Published in the 2018 JSM Proceedings,
Survey Research Methods Section. Alexandria, VA: American Statistical Association. 1715-1723. The paper is available at
https://amerispeak.norc.org/research/. Please note that the TrueNorthTM trademarking was in progress when this paper
was published. Hence, the authors do not reference the TrueNorthTM name on this paper.
1

1,514 segments, the U.S. Postal Service Delivery Sequence File (DSF) provided over 90 percent coverage of
the segments in terms of city-style addresses that are geo-codeable. For the 123 segments where the DSF
provided insufficient coverage, we enhanced the DSF address list with in-person listing. The National Frame
contains almost 3 million households, including over 80,000 rural households added through the in-person
listing.
The National Frame involves addresses in almost every state. For the states that are not included in the
National Frame, AmeriSpeak selected an address-based sample (ABS) in 2016 through 2018 from the USPS
DSF to assure AmeriSpeak sample representation for all US States and Washington, DC.
In 2017, a targeted address-based sample was added to AmeriSpeak recruitment in order to develop a new
Latino Panel with adequate representation of Spanish-language-dominant Hispanics. Census tracts with high
incidence (at least 30%) of Spanish-dominant Hispanics were targeted for this recruitment. Furthermore,
within these Census tracts, households that were flagged as Hispanic based on consumer vendor data (that
are typically used for direct-mail marketing) were oversampled. This new AmeriSpeak Latino Panel contains
approximately 5,400 Hispanic panelists with 24% of those panelists being Spanish-language dominant. As of
February 2019, 11% of AmeriSpeak Panel (including the Latino Panel) recruited adults were sourced from the
targeted address-based sample and 89% from the National Frame. Proper weights allow the full use of the
combined sample.
Sample Selection for Panel Recruitment
The 2014-2018 AmeriSpeak Panel sample consists of nationally representative housing units drawn primarily
from the 2010 NORC National Sample Frame. To create AmeriSpeak Latino and provide sample coverage
for states where the National Frame is not operative, a secondary source is address-based sampling, which
accounts for 12% of the 2014-2018 sample. The 2010 NORC National Sample Frame is stratified based on
segment (Census tract or Census block group) characteristics such as age and race/ethnicity composition of
the segment, and then, a stratified simple random sample of housing units is selected. Specifically, based on
Census tract-level data, segments were classified as having a higher concentration of 18-24 year old adults or
not, and a higher concentration of Hispanics, non-Hispanic African Americans, and other. Based on these
strata definitions, 6 strata (2 based on age times 3 based on race/ethnicity) were used to oversample housing
units in segments higher in young adults and/or Hispanics and non-Hispanic African-Americans. This is
referred to as the initial sample or first stage of panel recruitment.
In the second stage of panel recruitment, initially sampled but nonresponding housing units are subsampled
for a nonresponse follow-up (NRFU). At this stage, consumer vendor data are matched to housing units,
and housing units that are flagged (based on consumer vendor data) as having a young adult (18-34 years of
age) or minority (Hispanic and non-Hispanic African American) are oversampled for the NRFU. Overall,
approximately one in five initially nonresponding housing units are subsampled for NRFU. Due to NRFU,
these initially nonresponding housing units have a much higher selection probability compared to the housing
units that were recruited during the first stage of panel recruitment. Note that a small fraction of initially
nonresponding housing units are not eligible for NRFU due to these housing units being classified as “hard
refusals” or having an appointment for a call back from NORC.
In summary, there are mainly two reasons why the sampling design for AmeriSpeak Panel recruitment
deviates from Equal Probability of Selection Method (EPSEM) sampling: (a) oversampling of housing units
in segments with a higher concentration of young adults and minorities results in the sample selection
probabilities being higher for housing units in these segments; and (b) the nonresponse follow-up effort
results in initially nonresponding housing units having a much higher selection probability. Furthermore,
oversampling associated with NRFU results in higher selection probabilities for initially nonresponding
housing units that are flagged (based on consumer vendor data) using demographics that are correlated with
sample member’s propensity to respond. The initial and NRFU sampling procedures are examined and
2

modified each year to more efficiently recruit types of panelists who are less likely to respond based on their
certain demographic characteristics.
AmeriSpeak Panel Recruitment Procedures
Recruitment is a two-stage process: initial recruitment using less expensive methods and then non-response
follow-up using personal interviewers. For the initial recruitment, sample units are invited to join
AmeriSpeak online by visiting the panel website AmeriSpeak.org or by telephone (in-bound/outbound
supported). English and Spanish language are supported for both online and telephone recruitment. Study
invitations are communicated via an over-sized pre-notification postcard, a USPS recruitment package in a
9”x12” envelope (containing a cover letter, a summary of the privacy policy, FAQs, and a study brochure),
two follow-up post cards, and also contact by NORC’s telephone research center for sample units matched to
a telephone number.
The second-stage non-response follow-up targets a stratified random sub-sample of the non-responders from
the initial recruitment. Units sampled for the non-response follow-up are sent by Federal Express a new
recruitment package with an enhanced incentive offer. NORC field interviewers then make personal, face-toface visits to the respondents’ homes to encourage participation. NORC field interviewers administer the
recruitment survey in-person using CAPI or else encourage the respondents to register at AmeriSpeak.org or
call the toll-free AmeriSpeak telephone number to register.
Recruiting Non-Internet and “Net Averse” Households
Under certain conditions, AmeriSpeak gives panelists a choice regarding their preferred mode for future
participation in AmeriSpeak surveys. As of February 2019, 83% of the active panelists were enrolled in
AmeriSpeak to receive online surveys, while 17% of the active panelists agreed to participate in AmeriSpeak
telephone mode surveys. For the 2016 through 2018 recruitment, respondents provided an option of online
or telephone modes include: persons without internet access, persons whose only internet access is via a
smartphone, and persons with internet access but unwilling to share an email address. A recruited household
can consist of both web-mode and phone-mode panelists residing in the same household.
Impact of Non-Response Follow-up
The non-response follow-up (NRFU) is instrumental for producing a credible AAPOR response rate for the
panel, boosting the panel recruitment response rate by a factor of 6.1 (AAPOR RR3, weighted to take into
account selection probabilities). Additionally, NRFU reduces non-response bias significantly by improving
the representativeness of the AmeriSpeak panel sample with respect to certain hard-to-reach segments of the
population underrepresented by recruitment relying only mail and phone. NRFU improves representation
for demographic segments (typically more reluctant to respond to surveys), including lower income
households, cell-phone only households, renters, persons age 18 to 34, African Americans, Hispanics, and
persons without a high school degree or have only a high school degree (no college). Even though NRFU
panelists are more reluctant to complete surveys, the addition of NRFU panelists reduced total absolute bias
on average 5-21% when compared to the initial stage recruits (among examined surveys). 2 Compared to
panelists recruited in the initial stage, panelists recruited via the non-response follow-up campaign overall
report more moderate opinions towards policy issues and are somewhat more conservative. Based on study
specific findings, NRFU panelists report being less knowledgeable about science, report less interest in
current events and topics in the news (such as climate change and energy resources), and are less likely to read
a print newspaper (more likely to read the news online and use social media). 3 They are also more likely to
See “Nonresponse Follow-up Impact on AmeriSpeak Panel Sample Composition and Representativeness” authored by
Ipek Bilgen, J. Michael Dennis, N. Ganesh. The paper is available at https://amerispeak.norc.org/research/;
Total Absolute Error = [Sum(|ACS 2016 Benchmark - Unweighted Variable Percentage Point|)]; see pages 8-13.
3 See “The Undercounted: Measuring the Impact of ‘Nonresponse Follow-up’ on Research Data and Outcome
Measures” authored by Ipek Bilgen, J. Michael Dennis, N. Ganesh. The paper will be soon available at
https://amerispeak.norc.org/research/.
2

3

attend church, less likely to be in favor of gun control policies, and more likely to eat at a fast food restaurant
than the initial stage recruits. Accordingly, our extensive research on this topic illustrates that NRFU
panelists make the substantive estimates in any AmeriSpeak study more inclusive and accurate.
AmeriSpeak Panel Recruitment Response Rate and Other Sample Metrics
The AAPOR RR3 (response rate) for the 2014-2018 panel recruitment 34.2% (weighted to take into account
selection probabilities). 4 The estimated cumulative AAPOR RR3 for client surveys is 10% to 20% (varying
according to study parameters and taking into account all sources of non-response including panel
recruitment, panel household attrition, and survey participation). 5 NORC documented the AAPOR response
rate calculation methodology for 2014-2015 recruitment. 6
Key statistics with respect to the 2014-2018 recruited households are as follows: 52% recruited via the nonresponse follow-up recruitment using overnight Federal Express mailers and face-to-face methodology (with
NORC field staff visiting households); 22% indicated a preference for the telephone mode of data collection
for participating in AmeriSpeak studies; 25% of the recruited households are non-Internet 7; 79% are cellphone only or cell-phone mostly; 17% are African-American and 23% Hispanic; and 35% have household
income below $30,000 (compared to CPS benchmark of 26%). 8
Mixed-Mode Data Collection
Panelists may participate in two to three AmeriSpeak Panel studies per month via online (computer, tablet, or
smartphones) or by CATI phone. CATI phone mode respondents represent a population currently underrepresented in web panels that exclude non-internet households or “net averse” persons. NORC’s telephone
interviewers administer the phone mode of survey questionnaires using a data collection system supporting
both the phone and web modes of data collection, providing an integrated sample management and data
collection platform. For panelists using smartphones for web-mode AmeriSpeak surveys, the NORC survey
system renders an optimized presentation of the survey questions for these mobile users. For general
population client studies, approximately 17% of the completed interviews by the active panelists are
completed via the telephone mode.
Panel Management Policies
NORC maintains strict rules to limit respondent burden and reduce the risk of panel fatigue. On average,
AmeriSpeak panel members typically participate in AmeriSpeak web-based or phone-based studies two to
three times a month.
Because the risk of panel attrition increases with the fielding of poorly constructed survey questionnaires, the
AmeriSpeak team works with NORC clients to create surveys that provide an appropriate user experience for
The response rate calculation incorporates the selection probabilities of the samples for the initial recruitment and nonresponse follow-up stages, as calculated by the US Bureau of the Census for the American Community Survey.
5 A properly calculated cumulative AAPOR response rate for panel-based research takes into account all sources of nonresponse at each stage of the panel recruitment, management, and survey administration process. A common
misapplication of the term “response rate” in online panel surveys is to represent the survey-specific cooperation rate as
the “cumulative survey response rate.”
6 See “Response Rate Calculation Methodology for Recruitment of a Two-Phase Probability-Based Panel: The Case of
AmeriSpeak” authored by Robert Montgomery, J. Michael Dennis, N. Ganesh. The paper is available at
https://amerispeak.norc.org/research/.
7
The non-internet households (HHs) are those that do not select “High-speed, broadband internet at home (such as
cable or DSL)” or “Dial-up internet at home” response options when they are asked “What kind of internet access do
you have? Please select all that apply” item in the recruitment survey. The non-internet HHs include those that only use
internet on a cell connection or mobile phone.
8
For transparency purposes, unweighted percentages are presented in this section. Hence, these results do not take into
account oversampling and selection probabilities. The base weighted distributions that take take into account selection
probabilities can be provided upon request.
4

4

AmeriSpeak panelists. AmeriSpeak will not field surveys that in our professional opinion will result in a poor
user experience for our panelists and in panel attrition.
ABOUT NORC AT THE UNIVERSITY OF CHICAGO
As one of the world’s foremost independent research institutions, NORC at the University of Chicago
delivers objective data and meaningful analysis to help decision-makers and leading organizations make
informed choices and identify new opportunities. Since 1941, NORC has applied sophisticated methods and
tools, innovative and cost-effective solutions, and the highest standards of scientific integrity and quality to
conduct and advance research on critical issues. Today, NORC expands on this tradition by partnering with
government, business, and nonprofit clients to create deep insight across a broad range of topics and to
disseminate useful knowledge throughout society.
Headquartered in downtown Chicago, NORC works in over 40 countries around the world, with additional
offices on the University of Chicago campus, the DC metro area, Atlanta, Boston, and San Francisco.
ADDITIONAL RESOURCES
To learn more about AmeriSpeak or to share an RFP, please contact AmeriSpeak at [email protected]. Information about AmeriSpeak capabilities and research papers are available online at
AmeriSpeak.NORC.org.

5