Att A-2 AmeriSpeak Technical Overview 09 26 17

Att A-2. AmeriSpeak Technical Overview

TECHNICAL OVERVIEW OF THE AMERISPEAK® PANEL
NORC’S PROBABILITY-BASED RESEARCH PANEL

Updated August 28, 2017
Prepared by J. Michael Dennis, Ph.D.
Detailed methodological information about AmeriSpeak is located on the “Research” page of
AmeriSpeak.norc.org.
Overview
Funded and operated by NORC at the University of Chicago, AmeriSpeak® is a probability-based panel
designed to be representative of the US household population. Randomly selected US households are
sampled with a known, non-zero probability of selection from the NORC National Frame, and then
contacted by US mail, telephone interviewers, overnight express mailers, and field interviewers (face to face).
AmeriSpeak panelists participate in NORC studies or studies conducted by NORC on behalf of NORC’s
clients.
In 2016, the AmeriSpeak Panel expanded to 20,000 households, with a large oversample of young AfricanAmerican, Hispanic, and Asian adults (age 18 to 30). AmeriSpeak will expand to 23,000 households in 2017
by creating new panels specific to Latino and teen research.
Sample Frame
In order to provide a nationally representative sample, AmeriSpeak leverages the NORC National Sample
Frame, constructed by NORC to cover over 97 percent of U.S. households. The 2010 National Frame used a
two-stage probability sample design to select a representative sample of households in the United States. The
first stage—the sampling unit—is a National Frame Area (NFA), which is either an entire metropolitan area
(made up of one or more counties) or a county (some counties were combined so that each NFA contains a
population of at least 10,000). The largest NFAs with a population of at least 1,543,728 (0.5 percent of the
2010 Census U.S. population) were selected with certainty; these areas have a high-population density, and are
dominated by tracts with street-style addresses. These areas contain 56 percent of the population within 8
percent of the geographic area of the United States. The remaining areas were stratified into areas where
street-style addresses predominate, and the remaining areas, which are less likely to have street -style
addresses. The latter stratum (“rural” areas) comprises 81 percent of the geographic area, but only 14 percent
of the population.
Within the selected NFAs, the second stage sampling unit is a segment, defined either in terms of Census
tracts or block groups, containing at least 300 housing units according to the 2010 Census. A stratified
probability sample of 1,514 segments was selected with probability proportional to size. For most of the
1,514 segments, the USPS DSF provided over 90 percent coverage of the segment in terms of city-style
addresses that are geo-codeable. For the 123 segments where the DSF provided insufficient coverage, we

enhanced the DSF address list with in-person listing. The National Sample Frame contains almost 3 million
households, including over 80,000 rural households added through the in-person listing.
The National Frame involves addresses in almost every state. For the remaining states, AmeriSpeak added
some address-based sampling (ABS) addresses in 2016 and 2017 from the USPS DSF to assure AmeriSpeak
sample representation for all US States. In 2017, a targeted address-based sample was added to AmeriSpeak
recruitment in order to develop a new Latino Panel with adequate representation of Spanish-dominant
Hispanics. Census tracts with high incidence (at least 30%) of Spanish-dominant Hispanics were targeted for
this recruitment. Furthermore, within these Census tracts, households that were flagged as Hispanic based on
consumer vendor data were oversampled. This Latino Panel has 5,500 panelists with around 23% of those
panelists being Spanish-dominant. As of July 2017, 13% of AmeriSpeak Panel (including the Latino Panel)
recruited adults were sourced from the ABS (87% from the National Frame). Proper weights allow the full
use of the combined sample.
Sample Selection for Panel Recruitment
The 2014-2017 AmeriSpeak Panel sample consists of nationally representative housing units drawn from the
2010 NORC National Sample Frame and 14% from address-based sampling. The 2010 NORC National
Sample Frame is stratified based on segment (Census tract or Census block group) characteristics such as age
and race/Hispanic ethnicity composition of the segment, and then, a stratified simple random sample of
housing units is selected. Specifically, based on Census tract-level data, segments were classified as having a
higher concentration of 18-24 year old adults or not, and a higher concentration of Hispanics, non-Hispanic
African Americans, and other. Based on these strata definitions, 6 strata (2 based on age times 3 based on
race/ethnicity) were used to oversample housing units in segments higher in young adults and/or Hispanics
and non-Hispanic African-Americans. This is referred to as the initial sample or first stage of panel
recruitment.
In the second stage of panel recruitment, initially sampled but nonresponding housing units are subsampled
for a nonresponse follow-up (NRFU). At this stage, consumer vendor data are matched to housing units, and
housing units that are flagged (based on consumer vendor data) as having a young adult or minority (Hispanic
and non-Hispanic African American) are oversampled for the nonresponse follow-up. Overall, approximately
one in five initially nonresponding housing units are subsampled for NRFU. However, as mentioned
previously, selection of housing units for NRFU is a stratified simple random sample based on consumer
vendor data. Due to NRFU, these initially nonresponding housing units have a much higher selection
probability compared to the housing units that were recruited during the first stage of panel recruitment.
Note that a small fraction of initially nonresponding housing units are not eligible for NRFU due to these
housing units being classified as “hard refusals” or having an appointment for a call back from NORC.
In summary, there are two reasons why the sampling design for AmeriSpeak Panel recruitment deviates from
Equal Probability of Selection Method Sampling (EPSEM): (a) oversampling of housing units in segments
with a higher concentration of young adults and minorities results in the sample selection probabilities being
higher for housing units in these segments; and (b) the nonresponse follow-up effort results in initially
nonresponding housing units having a much higher selection probability. Furthermore, oversampling
associated with NRFU results in higher selection probabilities for initially nonresponding housing units that
are flagged (based on consumer vendor data) as having a young adult or minority.
AmeriSpeak Panel Recruitment Procedures
Recruitment is a two-stage process: initial recruitment using less expensive methods and then non-response
follow-up using personal interviewers. For the initial recruitment, sample units are invited to join AmeriSpeak
online by visiting the panel website AmeriSpeak.org or by telephone (in-bound/outbound supported).
2

English and Spanish language are supported for both online and telephone recruitment. Study invitations are
communicated via an over-sized pre-notification postcard, a USPS recruitment package in a 9”x12” envelope
(containing a cover letter, a summary of the privacy policy, FAQs, and a study brochure), two follow-up post
cards, and also follow-up by NORC’s telephone research center for matched sample units.
The second-stage non-response follow-up targets a stratified random sub-sample of the non-responders from
the initial recruitment. Stratification is based on consumer vendor data and stratification variables from the
initial recruitment stage in order to increase sample representation of young adults, non-Hispanic African
Americans, and Hispanics. Units sampled for the non-response follow-up are sent by Federal Express a new
recruitment package with an enhanced incentive offer. NORC field interviewers then make personal, face-toface visits to the respondents’ homes to encourage participation. NORC field interviewers administer the
recruitment survey in-person using CAPI or else encourage the respondents to register at AmeriSpeak.org or
call the toll-free AmeriSpeak telephone number to register.
Recruiting Non-Internet and “Net Averse” Households
Under certain conditions, AmeriSpeak gives respondents a choice regarding their preferred mode for future
participation in AmeriSpeak surveys. For the 2014-2017 recruitment, 82% of panelists were enrolled in
AmeriSpeak to receive online surveys, while 18% of panelists agreed to participate in AmeriSpeak telephone
mode surveys. For the 2016 and 2017 recruitment, respondents provided an option of online or telephone
modes include: persons without internet access, persons whose only internet access is via a smartphone, and
persons with internet access but unwilling to share an email address. A recruited household can consist of
both web-mode and phone-mode panelists residing in the same household.
Impact of Non-Response Follow-up
The non-response follow-up improves the representativeness of the AmeriSpeak sample with respect to
certain demographic segments, including but not limited to rural and/or lower income households, cellphone only households, persons age 18 to 34, African Americans, Hispanics, and persons without a high
school degree on have only a high school degree (no college). Compared to panelists recruited in the initial
stage, panelists recruited via the non-response follow-up campaign are more politically conservative, are less
knowledgeable about science, report less interest in current events and topics in the news (such as climate
change), and are less likely to read a print newspaper.
AmeriSpeak Panel Recruitment Response Rate and Other Sample Metrics
Between October 2014 and July 2017, 25,657 households were recruited to the AmeriSpeak Panel. The
AAPOR RR3 (response rate) for the panel recruitment during this time frame is 33.5% (weighted to take into
account selection probabilities).1 The estimated cumulative AAPOR RR3 for client surveys is 10% to 20%
(varying according to study parameters and taking into account all sources of non-response including panel
recruitment, panel household attrition, and survey participation).2 NORC documented the AAPOR response
rate calculation methodology for 2014-2015 recruitment.3

1

The response rate calculation incorporates the selection probabilities of the samples for the initial recruitment and non-response
follow-up stages, as calculated by the US Bureau of the Census for the American Community Survey.
2 A properly calculated AAPOR response rate for panel-based research takes into account all sources of non-response at each
stage of the panel recruitment, management, and survey administration process. A common misapplication of the term “response
rate” in online panel surveys is representing the survey-specific cooperation rate as the “survey response rate.”
3See “Response Rate Calculation Methodology for Recruitment of a Two-Phase Probability-Based Panel: The Case of
AmeriSpeak” authored by Robert Montgomery, J. Michael Dennis, Nada Ganesh. The paper is available at amerispeak.norc.org
on the “research” page.

3

Key statistics with respect to the 2014-2017 recruited households are as follows: 52% recruited via the nonresponse follow-up recruitment using overnight Federal Express mailers and face-to-face methodology (with
NORC field staff visiting households); 19% indicated a preference for the telephone mode of data collection
for participating in AmeriSpeak studies; 23% of the recruited households are non-Internet; 79% are cellphone only or cell-phone mostly; 17% are non-Hispanic African-American and 24% Hispanic; and 36% have
household income below $30,000 (compared to CPS benchmark of 27%).
Mixed-Mode Data Collection
Panelists may participate in 2 to 3 AmeriSpeak Panel studies per month via online (computer, tablet, or
smartphones) or by CATI phone. CATI phone mode respondents represent a population currently underrepresented in web panels that exclude non-internet households or “net averse” persons. NORC’s telephone
interviewers administer the phone mode of survey questionnaires using a data collection system supporting
both the CATI phone and web modes of data collection, providing an integrated sample management and
data collection platform. For panelists using smartphones for web-mode AmeriSpeak surveys, the NORC
survey system renders an optimized presentation of the survey questions for these mobile users. For general
population client studies, approximately 20% of the completed interviews are completed by the telephone
mode.
Panel Management Policies
NORC maintains strict rules to limit respondent burden and reduce the risk of panel fatigue. On average,
AmeriSpeak panel members typically participate in AmeriSpeak web-based or phone-based studies two to
three times a month.
Because the risk of panel attrition increases with the fielding of poorly constructed survey questionnaires, the
AmeriSpeak team works with NORC clients to create surveys that provide an appropriate user experience for
AmeriSpeak panelists. AmeriSpeak will not field surveys that in our professional opinion will result in a poor
user experience for our panelists and in panel attrition.
ABOUT NORC AT THE UNIVERSITY OF CHICAGO
As one of the world’s foremost independent research institutions, NORC at the University of Chicago
delivers objective data and meaningful analysis to help decision-makers and leading organizations make
informed choices and identify new opportunities. Since 1941, NORC has applied sophisticated methods and
tools, innovative and cost-effective solutions, and the highest standards of scientific integrity and quality to
conduct and advance research on critical issues. Today, NORC expands on this tradition by partnering with
government, business, and nonprofit clients to create deep insight across a broad range of topics and to
disseminate useful knowledge throughout society.
Headquartered in downtown Chicago, NORC works in over 40 countries around the world, with additional
offices on the University of Chicago campus, the DC metro area, Atlanta, Boston, and San Francisco.
ADDITIONAL RESOURCES
To learn more about AmeriSpeak or to share an RFP, please contact AmeriSpeak at [email protected]. Information about AmeriSpeak capabilities and research papers are available online at
AmeriSpeak.NORC.org.

4