B-1 AmeriSpeak Overview

TECHNICAL OVERVIEW OF THE AMERISPEAK® PANEL:
NORC’S PROBABILITY-BASED HOUSEHOLD PANEL
Updated January 26, 2021
This technical overview provides the basic information about AmeriSpeak®, a large probability-based panel
funded and operated by NORC at the University of Chicago. AmeriSpeak is designed to be representative of
the U.S. household population, including all 50 states and the District of Columbia. U.S. households are
randomly selected with a known, non-zero probability from the NORC National Frame as well as addressbased sample (ABS) frames, and then recruited by mail, telephone, and by field interviewers face to face.
AmeriSpeak panelists participate in NORC studies or studies conducted by NORC on behalf of
governmental agencies, academic institutions, the media, and commercial organizations.
The construction of the AmeriSpeak panel started in 2014 with pilot samples. In 2015, about 7,000
households were recruited from a sample of around 60,000 addresses. In 2016, about 128,000 addresses were
sampled to expand the panel to around 20,000 recruited households. About 51,000 addresses were selected
for the 2017 recruitment, which led to the expansion of the regular AmeriSpeak panel to 23,000 recruited
households. The AmeriSpeak Panel expanded to approximately 30,000 households in 2018 and 35,000
households in 2019 through further recruitment efforts. The 2020 recruitment is expected to expand the
panel by another 5,000 households. The current panel size is 48,900 panel members age 13 and over residing
in over 40,000 households.
In addition to the regular panel for general population studies, AmeriSpeak also contains sub-panels to
support studies of special populations, including AmeriSpeak Latino, AmeriSpeak Teen, and AmeriSpeak
Young Adult 18-34 (which features an oversample of African Americans, Hispanics, and Asians). AmeriSpeak
is also the probability sample source for TrueNorth®, the NORC calibration solution for combining
probability and non-probability samples for estimation through small area modeling that leverages data from
AmeriSpeak, the American Community Survey, Current Population Survey, and other data sources for
improved statistical efficiency.1
Panel Sample Frame
The primary sampling frame for AmeriSpeak is the 2010 NORC National Frame, a multistage probability
sample that fully represents the U.S. household population. We provide a brief description of how the
National Frame was constructed after the 2010 Census.
The primary sampling units (PSUs) in the first stage sample selection are 1,917 National Frame Areas
(NFAs), each of which is an entire metropolitan area (made up of one or more counties), a county, or a group
of counties with a minimum population of 10,000. A total of 126 NFAs are selected in the first stage,
including 38 certainty NAFs, 60 urban NFAs, and 28 non-urban NFAs. The largest 38 NFAs, those with a
population of at least 1,543,728 (0.5 percent of the 2010 Census U.S. population), were selected into the
National Frame with certainty. The certainty NFAs consist of areas with high population density and are
1

For more information about TrueNorth, see http://amerispeak.norc.org/our-capabilities/Pages/TrueNorth.aspx.

dominated by census tracts with city-style mailing addresses. These areas contain 56 percent of the U.S.
household population while only accounting for about 8 percent of the geographic area of the United States.
The remaining 1,879 NFAs are stratified into urban areas where city-style addresses predominate, and rural
areas that are less likely to have city-style addresses. The latter stratum comprises 81 percent of the
geographic area, but only 14 percent of the population. A sample of 60 and 28 NFAs is selected systematically
from the urban and rural stratum, respectively, with selection probability proportional to size (PPS) where the
measure of size is the number of housing units per NFA.
Within the 126 selected NFAs, the secondary sampling units (SSUs) are segments defined from Census tracts
or block groups, where each segment contains at least 300 housing units according to the 2010 Census.
Within the certainty NFAs, a sample of 896 segments was selected using systematic PPS sampling, where the
size of a segment is the number of housing units. Implicit stratification was achieved by sorting the segments
by location (NFA, state, and county), by principal city indicator, by ethnic and income indicators. From each
urban and rural NFA, a sample of 8 and 5 segments was selected, respectively, using systematic PPS sampling
where the measure of size is the number of housing units per segment. A total of 618 segments are selected
from the non-certainty NFAs2. Overall, a stratified probability sample of 1,514 segments was selected into the
National Frame in the second stage sampling.
Within the selected segments, all housing units are listed using the U.S. Postal Service Delivery Sequence File
(DSF). In the 123 segments where the DSF coverage is deemed inadequate, the DSF address list is enhanced
with in-person field listing to improve coverage. The final National Frame, consisting of all listed households
in the sample segments, is estimated to provide over 97 percent coverage of the U.S. household population. It
contains almost 3 million households, including over 80,000 rural households that are added through inperson listing. In addition to NORC’s National Frame, the DSF is used as a supplemental sample frame in
four states. Although nationally representative, the National Frame does not include households from Alaska,
Iowa, North Dakota, and Wyoming. Since 2016, the annual panel recruitment sample has included a small
address-based sample (ABS) from these four states to assure AmeriSpeak presence in in all U.S. States and
Washington, D.C.
In 2017, an enhanced DSF frame was also used to develop a new Latino Panel with adequate representation
of Spanish-language-dominant Hispanics. Census tracts with high incidence (at least 30%) of Spanishdominant Hispanics were targeted for this recruitment. Furthermore, within these Census tracts, households
that were flagged as Hispanic based on consumer vendor data (that are typically used for direct-mail
marketing) were oversampled. This new AmeriSpeak Latino Panel contains approximately 5,400 Hispanic
panelists with 24% of those panelists being Spanish-language dominant. As of February 2020, 63.4% of the
recruited adults in the AmeriSpeak Panel were sourced from the National Frame, 27.1% from targeted ABS
frames, and 9.5% from voter registration files3. Proper weights allow the full use of the combined sample.
Panel Sample Selection
For panel sample selection between 2014 and 2018, National Frame segments are stratified into six sampling
strata based on the race/ethnicity and age composition of each segment, as below:
•
•
•
•
•

Hispanic, high youth segments
Hispanic, not high youth segments
Non-Hispanic Black, high youth segments
Non-Hispanic Black, not high youth segments
Other, high youth segments

A sample of 5 segments was selected from each of the 28 non-urban NFAs. However, 2 sample segments were later
subsampled out in Montana due to cost.
3
Voter registration files were used as supplemental sample source for the AmeriSpeak Young Adult Panel.
2

2

•

Other, not high youth segments

Hispanic segments are those where Hispanics make up at least a third of the population and the Hispanic
share in the population is greater than that of non-Hispanic Black. Similarly, non-Hispanic Black segments are
those where non-Hispanic Black make up at least a third of the population and the non-Hispanic Black share
in the population is greater than that of Hispanic. Finally, High Youth refers to segments in which 18-24 year
old adults are at least 12% of the total adult population. The above stratification is used to oversample
housing units in areas with higher concentration of young adults, Hispanics, and non-Hispanic AfricanAmericans. The resulting household sample is referred to as the initial AmeriSpeak sample or sample for
initial panel recruitment.
To support the second stage of panel recruitment, initially sampled but nonresponding housing units are
subsampled for a nonresponse follow-up (NRFU)4. At this stage, consumer vendor data are matched to the
pending housing units, and housing units that are flagged as having a young adult5 (18-34 years of age) or
minority (Hispanic6, non-Hispanic Black7) are oversampled for the NRFU sample. Overall, approximately
one in five initially nonresponding housing units are subsampled for NRFU using the same six sampling
strata defined above. Due to NRFU, these initially nonresponding housing units have a much higher selection
probability compared to the housing units that were recruited during the first stage of panel recruitment.
A 2-phase state-based ABS sample design was used for the 2019 AmeriSpeak recruitment. NORC’s National
Frame is designed to represent the U.S. household population nationally. At the state level, however, the
panel may have more significant clustering effects from the use of the National Frame, especially for states
with a small population. The primary objective of the 2019 design is to improve state-level representation by
selecting the recruitment sample mostly from areas that are outside the National Frame. A stratified
systematic sample was selected in the first phase, where each state constitutes a sampling stratum and the
sample was allocated to the strata proportional to the square root of the state population. In the second
phase, young adults, Hispanic, non-Hispanic Black, and conservatives are oversampled based on commercial
data sources to improve their representation in the panel. Because the 2019 design did not use NRFU face-toface recruitment, the 2019 design did not involve geographic clustering.
The overall AmeriSpeak Panel sample design reflects the cumulative design features of the NORC National
Frame and the annual recruitment samples thus far. These design features are captured in the final panel
weight for each recruited household and each panelist. There are mainly two reasons why the sampling design
for AmeriSpeak Panel recruitment deviates from EPSEM (Equal Probability of Selection Method) sampling:
(a) oversampling of housing units in segments with a higher concentration of young adults and minorities
results in the sample selection probabilities being higher for housing units in these segments; and (b) NRFU
subsampling results in initially nonresponding housing units having a much higher selection probability.
Furthermore, within the NRFU samples, selection probabilities vary for housing units depending on the
appended commercial data flags to target specific demographics for improved efficiency. The initial and
NRFU sampling procedures are examined and possibly modified each year to more efficiently recruit
subpopulations who are less likely to respond to the recruitment survey.
A small fraction of initially nonresponding housing units are not eligible for NRFU, including “hard
refusals” and those with an appointment for a call back from NORC.
4

A young adult flagged household refers to a household where MSG or TargetSmart indicated there was an 18-24 year
old adult in the household. In 2016 and 2017, a slightly different definition was used, and a young adult flagged
household was defined as having an 18-34 year old adult in the household by MSG or 18-30 year old adult by
TargetSmart.
6 A Hispanic flagged household refers to a household where MSG or TargetSmart indicated the presence of a Hispanic
adult in the household.
7 A non-Hispanic Black flagged household refers to a household where MSG or TargetSmart indicated the presence of a
non-Hispanic Black adult in the household.
5

3

Panel Recruitment Procedures
The highly efficient recruitment procedures set AmeriSpeak apart from other national panels. AmeriSpeak
Panel recruitment is a two-stage process: (i) an initial recruitment using USPS mailings, telephone contact, and
modest incentives, and (ii) a more elaborate NRFU recruitment using FedEx mailings, enhanced incentives,
and in-person visits by NORC field interviewers.
For the initial recruitment, sample households are invited to join AmeriSpeak online by visiting the panel
website AmeriSpeak.org or by calling a toll-free telephone line (in-bound/outbound supported). Both English
and Spanish languages are supported for online and telephone recruitment. The initial recruitment data
collection protocol features the following: an over-sized pre-notification postcard, a USPS recruitment
package in a 9”x12” envelope (containing a cover letter, a summary of the privacy policy, FAQs, and a study
brochure), two follow-up post cards, and contact by NORC’s telephone research center for sample units with
a matched telephone number.
For the second-stage NRFU recruitment, a stratified random sample is selected from the nonrespondents of
the initial recruitment. Units sampled for NRFU are sent a new recruitment package by Federal Express with
an enhanced incentive offer. Meanwhile, NORC field interviewers make personal, face-to-face visits to the
pending cases to encourage participation. Once the households are located, the field interviewers administer
the recruitment survey in-person using CAPI or else encourage the respondents to register online or by
telephone.
Panel Recruitment Response Rate and Other Panel Statistics
A sample household is considered recruited or responded if at least one adult in the household joins the
panel. The weighted household response rate (AAPOR RR3) is about 6% for initial recruitment and 28% for
NRFU recruitment. We report two recruitment response rates: one for all the panel recruitment years (20142019) and one for the recruitment years with NRFU (2014-2018). For all recruitment years, the cumulative
weighted household response rate is 24.1%; for recruitment years with NRFU, and the cumulative weighted
household response rate is 34.2%. For client studies requiring a panel recruitment response rate exceeding
30%, the sampling frame may be restricted to the panelists recruited in the NRFU years. The panel
recruitment response rate calculation methodology is consistent with AAPOR guidelines and fully
documented.8 The annual panel retention rate is about 85%.
For individual client surveys based on the AmeriSpeak Panel, the AAPOR RR3 response rate is between 10%
to 20% depending on specific study parameters such as target population, survey length, time in the field,
salience of subject, and the like. This response rate takes into account panel recruitment rate, panel retention
rate, and survey participation rate.9
Other important panel statistics with respect to the 2014-2019 recruited households are as follows: 62% are
recruited in the initial stage and 38% are recruited via NRFU; 85% of the active panelists prefer to do web or
online surveys, while 15% prefer to participate in telephone surveys; 21% of the recruited households are
http://amerispeak.norc.org/research/Pages/WhitePaper_ResponseRateCalculation_AmeriSpeak_2016.pdf
A properly calculated cumulative AAPOR response rate for panel-based research takes into account all sources of nonresponse at each stage of the panel recruitment, management, and survey administration process (see
https://www.aapor.org/AAPOR_Main/media/publications/Standard-Definitions20169theditionfinal.pdf, page 48-9). A
common misapplication of the term “response rate” in online panel surveys is to represent the survey-specific
cooperation rate as the “cumulative survey response rate.” See “Response Rate Calculation Methodology for Recruitment
of a Two-Phase Probability-Based Panel: The Case of AmeriSpeak” authored by Robert Montgomery, J. Michael
Dennis, N. Ganesh. The paper is available at https://amerispeak.norc.org/research/.
8 See
9

4

non-Internet10; 82% are cell-phone only or cell-phone mostly; 19% are African-American and 20% Hispanic;
and 33% have household income below $30,000 (compared to CPS benchmark of 26%).11
Impact of Non-Response Follow-Up
NRFU is instrumental in producing the industry-leading response rate for AmeriSpeak Panel recruitment.
Moreover, due to the more intensive effort, NRFU recruitments better represent hard to reach groups and
are therefore more representative of the target population. For example, initial recruitment tends to underrepresent young adults 18-34 years of age. NRFU recruitment corrects for this bias by bringing the age
distribution of the panel closer to ACS benchmarks.
Overall, NRFU recruitment significantly improves the representation of the panel with respect to
demographic segments that are under-represented among the respondents to the initial recruitment, including
young adults (persons 18 to 34 years of age), African Americans, Hispanics, lower income households,
renters, cell-phone only households, and persons with lower educational attainment (e.g., no college degree).
To the extent that these demographic characteristics are correlated with substantive survey variables, NRFU
helps to reduce potential non-response bias in the sample estimates. NORC’s research indicates that NRFU
respondents are indeed somewhat different from initial respondents for many common survey variables. For
example, compared to the panelists recruited during the initial stage, NRFU panelists tend to be more
conservative politically, more likely to attend church, less interested in current events or topics in the news
report, less knowledgeable about science, less likely to be in favor of gun control policies, less likely to read a
print newspaper (more likely to read the news online and use social media), more likely to eat at fast food
restaurants and so on12. These observations illustrate that NRFU recruitment is critical for achieving a more
balanced panel and for making the substantive estimates in AmeriSpeak studies more accurate. Even though
NRFU panelists are more reluctant to complete surveys, the addition of NRFU panelists reduced total
absolute bias on average 5 to21 percentage points when compared to the initial stage recruits (among
examined surveys).13
Mixed-Mode Data Collection
The AmeriSpeak Panel supports mixed-mode data collection to improve response rate and the
representativeness of the complete surveys. During the recruitment survey, AmeriSpeak panelists are offered
an opportunity to choose their preferred mode—web or phone—for future participation in AmeriSpeak
surveys. A recruited household can consist of both web- and phone-mode panelists residing in the same
household. Panelists predominantly prefer web over phone mode. As of February 2020, 85% of the active
panelists prefer to do web or online surveys, while 15% prefer to participate in telephone surveys. The
telephone mode encompasses panelists without internet access, panelists whose only internet access is via a
smartphone, and panelists with internet access but are unwilling to share an email address.
The non-internet households (HHs) are those that do not select “High-speed, broadband internet at home (such as
cable or DSL)” or “Dial-up internet at home” response options when they are asked “What kind of internet access do
you have? Please select all that apply” item in the recruitment survey. The non-internet HHs include those that only use
internet on a cell connection or mobile phone.
11
For transparency purposes, unweighted percentages are presented in this section. Hence, these results do not take into
account selection probabilities. The base weighted distributions that take into account selection probabilities can be
provided upon request.
12 See “The Undercounted: Measuring the Impact of ‘Nonresponse Follow-up’ on Research Data and Outcome
Measures” authored by Ipek Bilgen, J. Michael Dennis, N. Ganesh. The paper will be soon available at
https://amerispeak.norc.org/research/.
13 See “Nonresponse Follow-up Impact on AmeriSpeak Panel Sample Composition and Representativeness” authored by
Ipek Bilgen, J. Michael Dennis, N. Ganesh. The paper is available at https://amerispeak.norc.org/research/.
10

5

To the extent that non-internet households or “net averse” persons are different from the rest of the
population, mixed-mode surveys have better population coverage and produce more accurate population
estimates. NORC’s telephone interviewers administer the telephone surveys using a data collection system
supporting both the phone and web modes, providing an integrated sample management and data collection
platform. For panelists using smartphones for web-mode surveys, the NORC survey system renders an
optimized presentation of the survey questions for these mobile users.
Panel Management and Maintenance
Panel management and maintenance are crucial for panel health and efficiency. NORC maintains strict panel
management rules to limit respondent burden, reduce panel attrition, and minimize the risk of panel fatigue.
On average, AmeriSpeak panelists are invited to participate in client studies two to three times a month.
AmeriSpeak works with NORC clients to create surveys that provide an appropriate user experience for
AmeriSpeak panelists. AmeriSpeak will not field surveys that in our professional judgment will result in a
poor user experience for our panelists. AmeriSpeak also has a designated website and a telephone number for
panelist communications.
Panel maintenance is a dynamic process because the AmeriSpeak Panel is supplemented and refreshed
regularly over time to grow the panel, compensate for panel attrition, and improve panel representation for
specific subpopulations. For example, the Latino Panel and Teen Panel are created to support studies of
Hispanics and teenagers, respectively; the 2019 recruitment is primarily designed to improve sample
representation at the state level. As panelists are added or/and removed from the panel, the panel
refreshment process takes place to ensure that the refreshed panel fully represents the corresponding target
population.
ABOUT NORC AT THE UNIVERSITY OF CHICAGO
As one of the world’s foremost independent research institutions, NORC at the University of Chicago
delivers objective data and meaningful analysis to help decision-makers and leading organizations make
informed choices and identify new opportunities. Since 1941, NORC has applied sophisticated methods and
tools, innovative and cost-effective solutions, and the highest standards of scientific integrity and quality to
conduct and advance research on critical issues. Today, NORC expands on this tradition by partnering with
government, business, and nonprofit clients to create deep insight across a broad range of topics and to
disseminate useful knowledge throughout society.
Headquartered in downtown Chicago, NORC works in over 40 countries around the world, with additional
offices on the University of Chicago campus, the DC metro area, Atlanta, Boston, and San Francisco.
ADDITIONAL RESOURCES
To learn more about AmeriSpeak or to share an RFP, please contact AmeriSpeak at [email protected]. Information about AmeriSpeak capabilities and research papers are available online at
AmeriSpeak.NORC.org.

6