10-Day Letter

DEPARTMENT OF HEALTH & HUMAN SERVICES Public Health Service

Centers for Disease Control and Prevention

National Center for Health Statistics

3311 Toledo Road

Hyattsville, Maryland 20782

Margo Schwab, Ph.D.

Office of Management and Budget

725 17th Street, N.W.

Washington, DC 20503

Dear Dr. Schwab:

The staff of the National Center for Health Statistics (NCHS) Collaborating Center for Questionnaire Design and Evaluation Research (CCQDER) (OMB No. 0920-0222, exp. 08/31/2021), in collaboration with the Division of Research and Methodology, plans to conduct a methodological study exploring if and how data from non-opt-in web panels may be linked to existing NCHS datasets and to quantify measurement error. This request is to expand a previously approved round of data collection. In addition to the previously-approved web survey (Generic IC, approved 11/19/2018, n=2,000 web respondents), we are requesting approval to administer a parallel survey to a sample (n=1,200) of National Opinion Research Center (NORC) at the University of Chicago’s Amerispeak phone-only panelists. NORC gives Amerispeak panelists the option of completing surveys on either a web device or via live interviewer phone calls, with about 85% of the total panel opting for the former and 15% for the latter. NCHS’ Division of Research and Methodology (DRM) is interested in exploring whether and how including phone respondents into the sample design affects its ability to evaluate questionnaire items. Both the already-approved web survey and the proposed telephone survey will use the same questionnaire (with minor mode differences) and be administered during the same field period, currently estimated to be February of 2020.

Please note that in this package, we are only requesting approval for the phone survey itself. OMB previously approved both the cognitive evaluation of the Research and Development Survey (RANDS) questionnaire and the web survey component—see “NCHS RANDS” IC under ICR Reference # 201807-0920-004, approved 11/19/2018.

Proposed Project: RANDS 4 Telephone Sample

As the nation’s principal health statistics agency, NCHS is responsible for not only producing high-quality statistics on the health and wellbeing of the American public and the state of the country’s health care system, but also for contributing to the development of survey methodologies that will allow the agency to continue producing these health statistics in the future.

To this effect, DRM proposes a methodological survey that will allow the agency to continue the long process of studying whether or not, and how, commercially-developed, -managed, and -available survey panels may be integrated into NCHS’ existing survey systems. The over-arching goal of this information collection is to discover new, and to improve existing, methods that will increase data quality in the midst of declining response rates and increased costs. It is well documented that the American public are becoming less likely to respond to surveys. While, on a whole, government surveys enjoy a higher response rate than ones conducted by private organizations, this overall trend is undeniable. Methodological research into new ways to capture respondents while minimizing their burden is necessary to ensure that the government is adapting to this changing survey environment in a proactive and efficient manner. The scope of this project involves both measurement error and sampling strategies. In particular, the project will research how measurement error can be examined in order to inform better-constructed questionnaires as well as laying the foundation for how recruited, commercial panel-based samples might be integrated alongside traditional sampling methods.

This current proposal is for a single round of data collection with phone-only sample to be administered alongside an already-approved web-only sample. To date, three separate rounds of RANDS have been conducted—in late 2015 and early 2016 using the Gallup Panel, and in 2019 using Amerispeak. All three previous rounds of RANDS have used web-only samples; the proposed expansion discussed in this amendment will be the first opportunity for NCHS to examine mode differences using RANDS. This dual-mode survey will allow NCHS to explore mode differences in its exploratory question evaluation and estimation strategies that follow the findings from the previous rounds of RANDS.

On the estimation side, DRM staff have studied a number of modelling techniques that attempt to estimate target variables using data from both the National Health Interview Survey (NHIS) and RANDS. The largest successes to this point have come from the estimation of chronic conditions; as such the proposed RANDS questionnaire for these new rounds will include a larger proportion of chronic condition questions in order to aid this effort. The proposed round will allow DRM staff to examine how including the ~15% of the Amerispeak Panel that chooses to respond to surveys via the telephone affects these modelling and estimation techniques.

On the question evaluation front, CCQDER has shown that the use of set cognitive probes (“web probes” since RANDS has used web-only samples to this point) is a viable way to explore not only the distribution of patterns of interpretation across a larger sample, but also explore how sub-groups within that sample think about and answer questions. For example, we have found that web probing can be used to examine whether or not out-of-scope patterns of interpretation are more likely to occur for particular respondent groups (i.e. a method for investigating measurement bias). The proposed round of RANDS will allow CCQDER to expand upon these results and explore how this form of probing can be done in a dual-mode survey environment.

RANDS/NORC Panel

The RANDS 4 (including both the already-approved web sample and the proposed telephone sample) will be conducted for NCHS by NORC (subcontracted through CCQDER’s contractor Swan Solutions), using their standing, proprietary, recruited panel (branded as the “AmeriSpeak Panel”). As in previous rounds, the questionnaire will include current and proposed NHIS questions, as well as an interspersed set of CCQDER-developed structured web probe questions.

Background Information about the AmeriSpeak: NORC recruits panel members using address-based sampling (ABS) to contact U.S. households at random. During recruitment, respondents take a short demographic survey, and are asked if they would be interested in participating in additional surveys as a member of the AmeriSpeak. Unlike opt-in panels, the recruitment process for AmeriSpeak’s panel starts with a random sample of addresses and, as a result, it is possible to derive the selection probability and hence the sampling weight for each respondent on the panel. There is no time commitment to membership in AmeriSpeak. Rather, households and individuals are encouraged to remain members as long as they are willing and interested. As with any longitudinal design, AmeriSpeak is affected by attrition; NORC makes significant effort to retain panelists for as long as possible.

As AmeriSpeak will be used as the sample source for completing the surveys, the sampling frame information will be protected under Section 308(d) of the Public Health Service Act [42 U.S.C. 242m(d)] and the Confidential Information and Statistical Efficiency Act or CIPSEA (Title V of PL 107-347). NORC will create a flag variable or other solution to identify and isolate the RANDS 4 sample from the rest of its panel data, ensuring that CIPSEA data cannot be used or accessed outside of this project.

All Panel participants have been fully screened and a substantial amount of background data have already been collected (e.g., health and well-being, socio-economic and occupational status, media usage, political views, age, gender, race, ethnicity, etc.), which will be attached to the final files delivered by NORC to NCHS, allowing for extensive non-response bias analysis. Following the delivery of the combined web and phone dataset and the final methodological report, NORC will remove all RANDS data from its servers, including backups. This will include not only the responses to the survey itself, but the metadata associated with the RANDS (including response, non-response, participation, and sampling flags identifying the RANDS 4 sample). NORC has extensive cyber and physical security in place, including a CIPSEA Information Protection Plan approved by the NCHS Confidentiality Officer and the NCHS Information Systems Security Officer, in order to protect both the security of the front-end survey interface and the back-end storage of the survey’s data. Additionally, all NORC employees working on RANDS 4 will complete NCHS confidentiality training, sign the NCHS affidavit of nondisclosure (see Attachment 3), and will be NCHS designated agents via the Designated Agent Agreement between Swan Solutions, LLC and NCHS.

Specific Plans for the Proposed Study: RANDS 4 will be conducted on both an already-approved web sample and a proposed telephone sample, using a substantively equivalent questionnaire (with some slight wording or formatting differences due to mode). In addition to current and proposed NHIS questions, the questionnaire will include an interspersed set of CCQDER-designed cognitive probes. These embedded probes are included to better understand and quantify measurement error for certain RANDS questions. The questionnaire can be seen in Attachment 1.

The questions, in the form that they will actually be used on the RANDS, will be pre-tested by NCHS after NORC has programmed the questionnaire into its Computerized Self-Administered Questionnaire (CSAQ) and Computer Assisted Telephone Interview (CATI) software. Please note that this pre-test was already approved by the NCHS ERB as part of the approval for the cognitive evaluation of the RANDS questionnaire—see protocol 2016-16, Amendment 14 approved on 12/18/17 and then revised and re-approved as 2016-16, Amendment 17 on 2/23/18.

 

The telephone portion of RANDS 4 will begin with introduction text read by the telephone interviewer similar to what is shown in Attachment 2, explaining the general purpose of the survey and providing the confidentiality and Paperwork Reduction Act language. A waiver of signed informed consent has been requested from the NCHS ERB as this procedure is not possible for commercial panel surveys where the population of respondents is anonymous to NCHS, as in a commercial panel. For the web mode, the introduction page will require the respondent to manually click through to the first page of questions; this action therefore implies consent; for the telephone mode, the interviewer will confirm that the respondent is willing to participate. In either case, the respondent can simply refuse to participate by either stopping the web survey or ending the phone call.

Following both web and phone data collection, NORC will process the survey data and prepare data files. These files will include the responses for both the already-approved web sample and the proposed telephone sample. The data files will not include the respondents’ names, addresses, or any other primary personally identifiable information (PII), including any Internet Service Provider (ISP) data NORC has about the computer from which the respondent replied to the survey. As stated above, all metadata tying the respondents to their inclusion in the RANDS sample will be eliminated from the NORC servers, including the backups, following final delivery. The data files will be transferred to NCHS via either a secure File Transfer Protocol (FTP) web portal or by loading them directly on an encrypted memory stick. Following confirmation that the transfer is complete and successful, NORC will delete the data file from their secured servers and will provide a certificate of destruction certifying that all RANDS-related data and metadata have been removed from their servers and backups. All NORC staff who will be working on the RANDS data will successfully completed NCHS’ confidentiality training (https://www.cdc.pgov/nchs/training/confidentiality/training/) and sign the NCHS Contractor Non-Disclosure Affidavit (Attachment 3).

NCHS will not provide respondents with an incentive to participate in either the already-approved web survey or the proposed telephone survey. The estimated average burden of each round’s questionnaire is 20 minutes.

Project Burden

A burden table for this project is 400 total burden hours as shown below:

Form Name	Number of Participants	Number of Responses/ Participant	Average hours per response	Response Burden (in hours)
Questionnaire	1,200	1	20/60	400
Total	400

Attachments (3)

cc:

Summer King

Jeff Zirger

DHHS RCO

3