Supp Statement B_RANDS COVID19 Round 3_01_13_2021

Supporting Statement B for Request for Emergency Clearance:

NATIONAL CENTER FOR HEALTH STATISTICS RESEARCH AND DEVELOPMENT SURVEY

OMB No. 0920-XXXX

Expiration Date: XX/XX/XXXX

Contact Information:

Paul J Scanlon Jr, PhD.

Senior Behavioral Scientist

Collaborating Center for Questionnaire Design and Evaluation Research

Division of Research and Methodology

National Center for Health Statistics/CDC

3311 Toledo Road

Hyattsville, MD 20782

301-458-4649

[email protected]

January 13, 2021

Table of Contents

B. Collections of Information Employing Statistical Methods

B.1. Response Universe and Sampling Methods 3

B.2. Procedures for the Collection of Information 6

B.3. Methods to Maximize Response Rates and Deal with Nonresponse…………………….10

B.4. Tests of Procedures or Methods to be Undertaken………………………………………11

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting

and/or Analyzing Data…………………………………………………………………...12

LIST OF ATTACHMENTS

Attachment A - Pub Health Serv Act

Attachment B - Intro Screen and Questionnaire

Attachment C - NORC CIPESA Protection Plan

Attachment D - NCHS Non-Disclosure Affidavit for NORC Organization Staff working on RANDS

Attachment E - NCHS ERB Approval Notice

Attachment F - Abstracts of Previous RANDS Research on Estimation and Calibration

Attachment G - RANDS during COVID-19 Variable Crosswalk, Uses, and Power Calculation

Attachment H - Health and Health Care Access Questions Across Federal Surveys During COVID-19

Attachment I - NORC Field Report for RANDS during COVID-19, Round 1

Attachment J - NORC Field Report for RANDS during COVID-19, Round 2

B. STATISTICAL METHODS

As explained in detail in Supporting Statement A, this emergency information collection request has two purposes: a) generation of data that can help explain health-related experiences of the US population during the pandemic and b) continuation of developmental survey methods research.

These purposes encompass three distinct, but related, activities:

1. Experimental Estimation Production: The RANDS-COVID-19 survey, which will be conducted by NORC using their commercially-available probability sampled Amerispeak survey panel and whose data will be calibrated to the National Health Interview Survey (NHIS) based on previous NCHS RANDS research findings.

2. Estimation Research: Continued research into the similarities, differences, and potential for calibration between RANDS and other NCHS and federal surveys, including the NHIS, the National Survey of Family Growth (NSFG), and the Census Pulse Survey.

3. Measurement Research: Continued evaluation and validation of COVID-19- and other health-related survey questions via methods such as web probing and experimental design.

The statistical methods for these three activities are largely similar.

In short, the third round of the RANDS during COVID-19 survey will use a statistically-sampled, recruited panel (NORC’s AmeriSpeak). The survey will be administered by NORC in two modes (web for the majority of responses and telephone for the rest) using their web survey and CATI applications respectively, and will use reminder emails and telephone calls to maximize response. In addition, for the measurement research activity, information collected from a set of cognitive interviews and web probing findings from the previous two rounds (all approved in the emergency clearance for the previous two rounds of RANDS during COVID-19, OMB No. 0920-1298, expiration: 11/30/2020) will be integrated with the data from this proposed follow-up round of the survey.

1. Respondent Universe and Sampling Methods

NCHS plans to conduct a third round of data collection for the RANDS during COVID-19 project using NORC’s AmeriSpeak Panel in order to produce experimental estimates informing understanding of the population during the COVID-19 pandemic, and to continue the estimation and measurement research conducted using the previous two rounds. For the estimation research, we will take the opportunity of fielding this study to continue the developmental research comparing NHIS with probability panels available in the commercial sector. As such, we will conduct this round using a similar set of questions as the previous rounds (see Attachments B and G). The survey will be fielded as soon as OMB approval is received. NCHS plans on obtaining 5,200 responses to the survey. Of this sample, 4,000 will respond via web, whereas the other 1,200 will be interviewed over the telephone via Computer Assisted Telephone Interviews (CATI). All of the respondents will come from NORC’s probability-sampled AmeriSpeak panel (the same panel previously approved for use in the initial two rounds of RANDS during COVID-19 under 0920-1298 as well as two previous rounds of RANDS that were approved under CCQDER’s generic clearance OMB No. No. 0920-0222). This third round will not represent a longitudinal sample between the first two round of RANDS during COVID-19 and this proposed follow-up round: while given the panel nature of AmeriSpeak, some panelists may be re-contacted by chance, NORC will not attempt to specifically recontact the exact set of panelists that was previous sampled.

NORC’s AmeriSpeak Panel: NORC recruits panel members using address-based sampling (ABS) to contact U.S. households at random. During recruitment, respondents take a short demographic survey, and are asked if they would be interested in participating in additional surveys as a member of the AmeriSpeak panel. Unlike opt-in panels, the recruitment process for AmeriSpeak’s panel starts with a random sample of addresses from an address frame (NORC has developed an in-house address frame based on USPS’ delivery sequence file and updated through field operations and canvassing), and as a result, it is possible to derive the selection probability and hence the sampling weight for each respondent on the panel. There is no time commitment to membership in AmeriSpeak. Rather, households and individuals are encouraged to remain members as long as they are willing and interested. As with any longitudinal design, AmeriSpeak is affected by panel attrition; NORC makes significant effort to retain panelists for as long as possible.

Most panelists complete AmeriSpeak surveys using NORC’s web interface, though about 10% of the panel has indicated they prefer to complete surveys via telephone calls. As with the previous round of RANDS that was approved (again, under both OMB No. 0920-1298 and OMB No. 0920-0222), the third round of RANDS during COVID-19’s sample will include both web and phone respondents. This is important given that panelists who prefer phone (either by choice or because they do not have ready access to an internet-capable device) may have different health outcomes, behaviors, and attitudes; therefore excluding them from the sample would lead to coverage bias beyond what is already expected from using a commercial survey panel (as compared to a direct ABS sample, such as that used by the National Health Interview Survey, or NHIS). NCHS plans a phone oversample (approximately 23% of the total sample) for the third round of RANDS during COVID-19 in order to further evaluate the effects that survey mode and mode preference have on both survey measurement and estimation.

As with prior rounds of RANDS, a stratified sample of AmeriSpeak panelists will be contacted. The sample design will use the panel information that NORC holds about the AmeriSpeak members to create strata based on age, race/ethnicity, and educational attainment.

Estimation Procedures

For the first purpose of generating health-related pandemic data, given the major differences in sample quality between RANDS and NCHS’ household surveys that produce official statistics (such as the NHIS and National Health and Nutrition Examination Survey, or NHANES), a series of estimation procedures that the NCHS Division of Research and Methodology (DRM) has developed over the past six rounds of RANDS data collection and analysis will be used (see Attachment F). DRM will use both traditional survey estimation procedures, as well as model-assisted methods. Experimental estimates for a limited set of variables (see Attachment G for “estimation” variables) will be publicly released and based on the model-assisted methods applied to the AmeriSpeak panel respondents, and technical notes will accompany the estimates that describe the methods and their limitations. Estimates will explicitly be labeled ‘experimental’.

For the model-assisted methods applied for purpose of generating health-related pandemic data, we will develop calibrated survey weights using the NHIS for calibration. (A number of NHIS questions are included verbatim on the RANDS during COVID-19 questionnaire specifically for this purpose.)

Generally, the calibrated survey weights calculated by NCHS will interact with, or start with, the RANDS sample weights provided by NORC for RANDS. The variables used by NCHS will be those included in RANDS for the purpose of adjusting the weights. More specifically to produce estimates, NCHS will adjust the RANDS sample weights provided for RANDS from NORC for health-related factors (i.e. the chronic conditions collected on both RANDS and NHIS, see Attachment G for “calibration” variables) using raking methods.   This adjustment is intended to control for possible health differences between the NHIS and RANDS samples due to different response propensities, coverage and/or sample variability.  Underlying health may affect participation in RANDS and response to COVID 19 related questions for some participants.  Based on our past and ongoing research, additional calibrations to the weights provided for RANDS can reduce differences between RANDS and NHIS estimates, though the effects depend on the outcomes and calibration variables, as well as these relationships. For RANDS during COVID 19 round 3, calibration methods will follow from those used for the first two rounds. However, unlike our usual data releases for NCHS core surveys, it is possible that some refinements to the weighting calibrations will be identified for round 3. If this is the case, updated estimates for prior rounds may be made available and documented¹.

The effectiveness of the model-assisted approach has been documented in statistical literature². This strategy has also been explored in our research on previous RANDS recruited panel survey data. To illustrate this research, Attachment F presents abstracts for 1) a draft journal article currently in the journal review stage evaluating the need to temporally align RANDS with NHIS for estimates, 2) a journal article to be published in the Statistical Journal of the International Association of Official Statistic, and 3) a report recently released as a NCHS Series 1 Report giving an overview of the estimation methodology research conducted on RANDS 1 and RANDS 2. For the first production estimates, a model based on our experience with prior rounds of RANDS will be used; later estimates may be based on updated models.

AmeriSpeak sample weights provided by NORC. For the purpose of generating estimates, NCHS weighting approaches will use the RANDS sample weights provided by NORC for the Amerispeak panel as primary inputs and adjust them as described above. The RANDS sample weights provided to NCHS by NORC for the AmeriSpeak data are a combination of two processes, the weights used for the AmeriSpeak Panel itself and the weights calculated for RANDS.

NORC has provided the following information about the AmeriSpeak Panel weights, which is similar to the information available for other AmeriSpeak surveys (see, for example, the NORC Documentation for the Data Coalition’s COVID-Impact Survey: https://static1.squarespace.com/static/5e8769b34812765cff8111f7/t/5eb0a6aab2f2aa04386c1032/1588635307466/COVID_Impact_Survey_W1_Field+Report_final_Web_Update.pdf):

• Panel base weights for all sampled housing units are computed as the inverse of the probability of selection from the NORC National Frame (the sampling frame that is used to sample housing units for AmeriSpeak) or other address-based sample frames (supplemental panel samples were selected from frames developed from the USPS Delivery Sequence Files). The sample design and recruitment protocol for the AmeriSpeak Panel involves subsampling of initial non-respondent housing units for an in-person follow-up. The subsample of housing units that are selected for the nonresponse follow-up (NRFU) have their panel base weights inflated by the inverse of the subsampling rate. The base weights are then adjusted to account for unknown eligibility and nonresponse among eligible housing units (see below for nonresponse adjustment variables). To produce the final household panel weights, the household-level nonresponse adjusted weights are post-stratified to external counts for number of households obtained from the Current Population Survey.

• Final household weights are assigned to each eligible adult in the recruited household. These person-level weights are then adjusted to compensate for nonresponding adults within a recruited household.

• The HH nonresponse adjustment cells are defined by crossing partisan score categories, young adult/minority categories, and TargetSmart flag for Republican (obtained from external vendors including TargetSmart and MSG). Additional person level non-response to the panel is adjusted by age and sex. 

Finally, to produce the RANDS-specific weights, the person weights are raked to population benchmarks for age, sex, education, race/Hispanic ethnicity, housing tenure, telephone status, and Census Division.

Research

For the second and third purposes outlined above that encompass NCHS’ ongoing survey methods research, RANDS during COVID-19 will continue to function as a methodological study on how data about an ongoing public health crisis and large shift health and social behaviors can be collected. As RANDS during COVID-19 is a part of the RANDS series, the methodological findings—both on the estimation and measurement error sides—from prior rounds of RANDS will be incorporated in this survey to inform the calculation of substantive estimates. Previous rounds of RANDS have been used to evaluate and develop methods for calibrating external data sources, such as data from commercial panels, with the NHIS and this work will inform the current data collection.

Estimation research

The estimation research activity of this ICR is to continue our developmental research comparing data from a probability-based panel (AmeriSpeak) to data from traditional household surveys.

In the ongoing series of RANDS data, NCHS continues to assess the quality of the probability-recruited panels (probability sampled Gallup Panel and the NORC AmeriSpeak Panel) by comparing the estimates derived from these panels with those from established household face-to face surveys (such as the NHIS), examining these comparisons by subgroup, and examining trends for data consistency over time.

It is worth repeating in this ICR that NCHS’ research with RANDS includes the aims of understanding and assessing the properties of commercial survey panels and their data and developing effective analytic strategies to combine information from multiple data sources and the opportunities for subgroup analysis that a large supplemental sample may afford.

Measurement research

For the measurement research activity of this ICR, the past rounds of RANDS have been used to develop and refine how NCHS can use “web probes” (or set cognitive probes in the case of a multi-mode survey like RANDS-COVID-19) alongside experimental design to determine the extent of patterns of interpretation and the relatively measurement quality of similar items. These methods will be leveraged in the case of RANDS during COVID-19 to provide information on the interpretation of survey items on other ongoing and planned information collections, including the NHIS, the NSFG, the Current Population Survey, the Census Bureau’s Covid-19 Household Pulse Survey, the Medicare Current Beneficiary Survey, and BLS’ National Longitudinal Survey. Furthermore, this information collection will be used to continue the methodological study of web probes and the establishment of their best practices of use.

2. Procedures for the Collection of Information

Questionnaire:

As noted above in B1, one follow-up round of RANDS during COVID-19 will be conducted on the previously approved AmeriSpeak web and telephone samples. The questionnaire for this third round will be very similar to the previous two, and is found in Attachment B. Following the initial analysis of the results of the first two rounds of RANDS during COVID-19, as well as conversations with stakeholders across NCHS and CDC, some changes to questions and question text have been incorporated into this new questionnaire.

As with the first two rounds of RANDS during COVID-19, the questionnaire is composed of items that serve five purposes: estimation, secondary Coronavirus-related variables, alignment, calibration, and measurement research.

• Estimation: NCHS will produce experimental estimates of selected health care access measures from the AmeriSpeak component (the probability-sampled component) of RANDS during COVID-19 for release as tables on the NCHS RANDS web page. Weighted estimates will be calculated using the sample weights calibrated to the NHIS and will be produced for all adults and for population subgroups stratified by age group, sex, and race/Hispanic origin. Additional subgroups will be defined by underlying health characteristics (e.g. diagnosed diabetes, hypertension, asthma), demographic characteristics (e.g. educational attainment, income), health behavior (i.e. smoking), and other health care access variables (e.g. health insurance coverage and usual place of care) depending on available sample. The intention of subgroup calculations is not to infer associations or causation between these factors and the COVID-19 variables, rather to provide information for possible high-risk population subgroups and other subgroups of interest.

The estimation variables are the same as those approved for the first two rounds of the survey, with the exception of a new telemedicine access question. Following analysis of the web probing and the cognitive interviews, it became clear that there is a potential for measurement error with the existing telemedicine access question (TELMED, “In the last two months, has this provider offered you an appointment with a doctor, nurse, or other health professional by video or by phone?”), with a number of respondents appearing to interpret this as a question about their use of telemedicine instead of their access to it. As such, we are including both the original and a new version (ALT_TELMED, “Does this provider offer telephone or video appointments, so that you don't need to physically visit their office or facility?”). These questions will be each administered to half of the RANDS during COVID-19 sample so that estimates of both can be compared.

• Secondary Coronavirus-related variables: A small number of variables—including about health insurance loss and Coronavirus prevention behaviors—are included not to produce estimates, but to serve as covariates and sources of ancillary information for both the estimation and measurement research goals of RANDS during COVID-19.

• Alignment: As noted in A2, to evaluate RANDS during COVID 19 and to permit triangulation, we include some questions on other sources, primarily the NHIS but also the CPS and Census Pulse Survey. Comparisons of estimates from RANDS recruited panels surveys after statistical adjustment to those from the NHIS using questions fielded on both sources continues to be an important tool for NCHS to gauge the effectiveness of its statistical methods. In addition, comparisons of estimates from the RANDS during COVID 19 Amerispeak component (after calibration) to those from the Census Pulse Survey and the CPS will inform inferences from all surveys. One developmental research area for RANDS is evaluation of alignment of key estimates across subgroups given possible different participation and response propensities for subgroups not accounted for in weighting.

• Calibration: For the purpose of generating data that can help explain health-related experiences of the US population during the pandemic, we will calibrate the sample weights provided with the NORC AmeriSpeak data to the NHIS using a set of questions fielded on both surveys (see B1 for details). Questions identified for calibration are those that adjust for possible differences in underlying health between the samples (e.g. diagnosed asthma) and mode (telephone versus web), but that are not considered to be related to the pandemic (e.g. mental health).

• Measurement Research: As with previous rounds of RANDS, NCHS plans to use RANDS during COVID-19 for methodological work related to measurement error and question design. This work will largely rely on the use of set cognitive probes and experimental design. As noted in A2, this measurement research will contribute some of the first data to the corpus of information relating to how survey respondents understand Coronavirus-related questions and concepts. This information will be used to not only inform current and future NCHS surveys (such as the NHANES planned changes due to COVID-19), but also the design and analysis of other Federal and non-Federal Coronavirus surveys and questionnaires.

Additionally, some of these variables will also be used as covariates in the estimates. Attachment G details which purpose category each questionnaire item falls into, and whether or not it will be used as a covariate.

Data Collection Procedures:

The survey will be conducted using either NORC’s in-house web survey platform or CATI application, depending on the chosen mode of the respondent. As with previous rounds, the RANDS survey itself will begin with an introduction screen (or introduction text for telephone respondents) similar to what is seen at the beginning of Attachment B, explaining the general purpose of the survey and providing the confidentiality and Paperwork Reduction Act language. As signed consent is not possible for surveys where the population of respondents is anonymous to NCHS, a waiver of signed informed consent has been requested from the NCHS ERB. The introduction page will require the respondent to manually click through to the first page of questions (or agree to continue and not hang up for telephone respondents); this action therefore implies consent.

Following the end of the field period for this proposed third round of RANDS during COVID-19, NORC will process the survey data and prepare data files. The data files will not include the respondents’ names, addresses, or any other direct personally identifiable information (PII), including any ISP (internet service provider) data NORC has about the computer from which the respondent replied to the survey. All metadata tying the respondents to their inclusion in the RANDS during COVID-19 sample will be eliminated from the NORC servers, including the backups, following final delivery. The data files will be transferred to NCHS via either a secure File Transfer Protocol (FTP) web portal or via some other encrypted electronic mean. Following confirmation that the transfer is complete and successful, NORC will delete the data file from their secured servers and will provide a certificate of destruction certifying that all RANDS during COVID-19-related data and metadata have been removed from their servers and backups.

Respondents will not receive a monetary incentive for participating in RANDS-COVID-19.

3. Methods to Maximize Response Rates and Deal with Nonresponse

NORC employs several approaches to maintain panel participation and to maximize response for the fielded surveys, such as RANDS. NORC employs dedicated staff to maintain panelists’ participation in AmeriSpeak and limits the number of surveys any one panelist may be asked to complete per month. This leads to a relatively high participation rate (for instance, the first two rounds of RANDS during COVID-19 data collection in June and August of 2020 had participation rates of 78.5% and 69.1% respectively among sampled AmeriSpeak panelists, with overall response rates—from panel selection to survey participation and completion and including panel attrition—of 23.0 and 20.0% respectively—see Attachments I and J for the brief field reports NORC provided on these two information collections). There are two levels of non-response that need to be accounted for when using survey panels, such as those proposed for RANDS during COVID-19: non-response (and coverage bias) during the panel recruitment phase and non-participation in the actual survey.

The first of these two levels of non-response and coverage bias is one of the greatest limitations in using commercially-available panels, though NCHS has implemented some processes to mitigate them. The panel provider NCHS has selected for this survey (and previous rounds of RANDS) does internal work to address the problem. First, NORC uses a well-maintained ABS frame that is constantly updated as NORC conducts their field surveys for panel recruitment. Second, for AmeriSpeak in particular, a large non-response follow-up effort (NRFU) is made during the panel recruitment effort, and NORC estimates that over half of its panelists are brought into AmeriSpeak because of this face-to-face effort (as opposed to the initial mail-back recruitment survey). However, NORC has not published a complete non-response bias analysis that takes into account both the NRFU non-response and any coverage issues their ABS frame may have. Because of this, NCHS has developed a series of model-based estimation procedures that use covariates to model the propensity to respond to web surveys (as detailed in B2) in an effort to correct for these inherent biases of representation.

As to the non-participation in AmeriSpeak surveys (which can also be thought of as the survey’s direct non-response), NCHS will conduct a non-response bias analysis of RANDS’ sample. All of NORC’s panel participants have been fully screened and a substantial amount of background data is available for each (e.g., socio-economic and occupational status, age, gender, race, ethnicity, and geographic characteristics). These data will be attached to the final files delivered by NORC to NCHS, which will allow NCHS to examine whether or not the unit (or individual item) non-responses to RANDS-COVID-19 are systematic or appear at random. The results of these analysis will permit NCHS to further refine its model-based estimation techniques and create calibrated survey weights. Both the original (which NORC supplies on the final files) and calibrated survey weights account for the nonresponses in the survey process. Therefore, estimates based on these weights are expected to cover the nonresponse error in the survey data.

4. Tests of Procedures or Methods to be Undertaken

Four separate empirical activities have informed the design of the RANDS during COVID-19 questionnaire. First, CCQDER conducted a series of cognitive interviews administered to NCHS staff in order to obtain an initial sense of the quality of the questionnaire’s items and to begin the analysis of those items’ construct validities. Second, NCHS partnered with the US Census Bureau as it conducted pre-testing for its Covid-19 Household Pulse Survey. NCHS staff (alongside staff from BLS) worked with the Census Bureau to conduct and analyze pre-testing via web probing or debriefing using Census’ Affinity Panel. Third, the first two rounds of RANDS during COVID-19 included a series of web probes on their questionnaires, providing both quantitative and qualitative findings regarding respondents’ interpretations of selected survey items. Finally, CCQDER conducted a full set of cognitive interviews with members of the public in order to validate RANDS during COVID-19’s items. (This cognitive interviewing project was approved under the emergency clearance for the first two rounds of RANDS during COVID-19, OMB No. 0920-1298, expiration: 11/30/2020.)

5. Individual Consulted on Statistical Aspects and Individuals and/or Analyzing Data

The person with overall responsibility for the methodological and technical aspects of the

described activities is:

Jennifer Parker, Ph.D.

Director, Division of Research and Methodology

National Center for Health Statistics

3311 Toledo Road

Hyattsville, Maryland

(301) 458-4419

[email protected]

Kristen Miller, Ph.D.

Director, Collaborating Center for Questionnaire Design and Evaluation Research

National Center for Health Statistics

3311 Toledo Road

Hyattsville, Maryland

(301) 458-4625

[email protected]

1 Any updates will be noted in the technical notes accompanying the experimental estimates released on NCHS’ RANDS during COVID-19 webpage, see https://www.cdc.gov/nchs/covid19/rands/work.htm#limitations for an example.

2 See for instance, Lee S., Valliant R. (2009) Estimation for Volunteer Panel Web Surveys Using Propensity Score Adjustment and Calibration Adjustment. Sociological Methods and Research. 37, 319-343. Doi: https://doi.org/10.1177%2F0049124108329643.

10