RANDS Abstracts

Attachment F – Abstracts of Previous RANDS Research on Estimation and Calibration

This manuscript is under review with Survey Methods: Insights from the Field and should not be disseminated.

Comparison of Quarterly and Yearly Calibration Data for Propensity Score Adjusted Web Survey Estimates

Katherine E. Irimata, Yulei He, Bill Cai, Hee-Choon Shin, Van L. Parsons, Jennifer D. Parker

National Center for Health Statistics, Centers for Disease Control and Prevention, Hyattsville, MD

Abstract

While web surveys have become increasingly popular as a method of data collection, there is concern that estimates obtained from web surveys may not reflect the target population of interest. Web survey estimates can be calibrated to existing national surveys using a propensity score adjustment, although requirements for the size and collection timeline of the reference data set have not been investigated. We evaluate health outcomes estimates from the National Center for Health Statistics’ Research and Development web survey. In our study, the 2016 National Health Interview Survey as well as its quarterly subsets are considered as reference datasets for the web data. It is demonstrated that the adjusted health estimates overall vary little when using the quarterly or yearly data, suggesting that there is flexibility in selecting the reference dataset. This finding has many practical implications for constructing reference data, including the reduced cost and burden of a smaller sample size and a more flexible timeline.

This manuscript is being finalized and will be submitted to the Statistical Journal of the International Association of Official Statistics. Not for citation.

Overview and Initial Results of the National Center for Health Statistics’ Research and Development Survey

Jennifer Parker, Kristen Miller, Yulei He, Paul Scanlon, Bill Cai, Hee-Choon Shin, Van Parsons, Katherine Irimata. National Center for Health Statistics, Centers for Disease Control and Prevention, Hyattsville, MD

Abstract

The National Center for Health Statistics is assessing the usefulness of recruited web panels in multiple research areas. One research area examines the use of close-ended probe questions and split-panel experiments for evaluating question-response patterns. Another research area is the development of statistical methodology to leverage the strength of national survey data to evaluate, and possibly improve, health estimates from recruited panels. Recruited web panels, with their lower cost and faster production cycle, in combination with established population health surveys, may be useful for some purposes for statistical agencies. Our initial results indicate that web survey data from a recruited panel can be used for question evaluation studies without affecting other survey content. However, the success of these data to provide estimates that align with those from large national surveys will depend on many factors, including further understanding of design features of the recruited panel (e.g. coverage and mode effects), the statistical methods and covariates used to obtain the original and adjusted weights, and the health outcomes of interest.

This report is close to completion and will be released as an NCHS Series 1 report. Not for citation.

Title: The National Center for Health Statistics’ 2015 and 2016 Research and Development Surveys

Yulei He, Bill Cai, Hee-Choon Shin, Vladislav Beresovsky, Van Parsons, Katherine Irimata, Paul Scanlon, Jennifer Parker

Abstract:

Objective: The Division of Research and Methodology of the National Center for Health Statistics (NCHS) has been conducting a series of web survey studies, referred to as the Research and Development Survey (RANDS). RANDS are collected via external contractors using existing probability-sampled panels of primarily web users. RANDS are being used in question design evaluations and for investigating statistical methodologies for estimation.

Methods: This report aims to provide a general description of the background and operation of the first two rounds of RANDS (1 and 2). To illustrate the data, some demographic and health estimates of RANDS are presented and compared with estimates from identical questions fielded on the corresponding quarters of the National Health Interview Survey (NHIS).

Results: RANDS 1 and 2 were web surveys collected from recruited probability panels from Gallup, Inc. Overall, 2304 and 2480 respondents were included in the 2015 (RANDS 1) and 2016 (RANDS 2), respectively. Public-use versions of the data can be found at (https://wwwdev.cdc.gov/nchs/rands/index.htm). Some weighted demographic and health estimates from RANDS 1 and RANDS 2 were similar to the corresponding NHIS estimates, while others differed.

Conclusion: RANDS is an ongoing platform for research at NCHS for understanding the properties of probability-sampled panels of primarily web users, investigating and developing statistical methods for using such data in conjunction with large nationally representative health surveys, and for extending question-design evaluation.

Keywords:

Health Survey, National Health Interview Survey, Official Statistics, Probability Panel, Web Survey