Re-Design Consultations

Attachment 9 - Consultations for the 2019 Questionnaire Redesign
Formal consultations with experts outside of the agency for the 2019 NHIS questionnaire redesign began in 2014
as discussions with CDC partners and Healthy People federal interagency working groups. In 2015, the first
presentations on the NHIS redesign were given at government (National Conference on Health Statistics) and
professional association (Population Association of America) conferences, and audience feedback was obtained.
These groups expressed support for the overarching goal of the redesign to streamline the survey and increase
the relevance of the data collected, and provided valuable suggestions on ways to collect the breadth of content
in a streamlined survey, including matrix sampling, use of administrative or alternative data sources, other
modes of data collection, and rotating content modules. NHIS redesign staff examined multiple possibilities and
elected to pursue rotating content modules.
Following this initial outreach and preliminary decision-making about questionnaire content structure, three
more formal, structured, and in-depth forms of consultation were pursued in the form of Technical Expert
Panels and calls for public feedback (each described in detail below). In addition, meetings were held throughout
the questionnaire development process between survey design staff at NCHS, stakeholders, and subject matter
experts across health care, health economics and demography. These included experts from within the federal
government (CDC, NIH, HHS, Census, and BLS) as well as academic and professional associations. Further
briefings were held throughout the redesign process, including at the National Academy of Sciences, the Federal
Interagency Forum on Child and Family Statistics, the American Economic Association, and the Council of
Professional Associations on Federal Statistics, with audience feedback obtained at each. Information obtained
through these consultation efforts led to changes, improvements, and fine-tuning in both the structure and
content of the 2018 NHIS questionnaire.
a. Technical Expert Panels (TEPs)*
Technical Expert Panels (TEPs) were convened on four topics: child health, income, pain, and injury. These
informal panels were an efficient and effective means for NCHS to collect feedback on specific topics from a
broad range of NHIS data users and stakeholders. These meetings involved multiple conference calls lasting
approximately 2 hours and typically involved discussing content areas, but not specific questions.
i.

Child Health Technical Expert Panel

There were two TEPs for child health. The first TEP, consisting of twelve members, met three times between
February and March in 2016, while the second TEP, consisting of five members met two times in April. Members
of the first TEP were affiliated with a range of academic, government, non-profit, and private organization,
including the University of Illinois College of Medicine at Chicago, Maternal and Child Health Bureau, Children’s
Hospital of Pittsburgh, Duke University School of Medicine, University of San Francisco College of Public Health,
Harvard Medical School, UC Berkeley School of Public Health, RTI International, The Lucile Packard Foundation
for Children’s Health, Boston Public Health Commission, Abt Associates, and the University of North Carolina
Gillings School of Global Public Health. Members from the second Child TEP were affiliated with Population
Reference Bureau, Johns Hopkins Bloomberg, School of Public Health, DHHS Office of Minority Health, and the
American Academy of Pediatrics. The members of the second TEP reviewed feedback from members of the first
TEP, and offered their own ideas for reducing the number of topics covered.
The primary feedback received from members of the child TEPs included increased emphasis on social
determinants of health, expanded focus on children’s mental health, and use of functioning as the framework
for measuring child disability. Additionally, members noted that a redesigned NHIS could decrease focus on rare
health conditions and increase focus on key measures for which benchmarks and long-term trends are needed.
Toward this end, the members of the TEPs identified priority populations who are at elevated risk of poorer
* The complete roster for each TEP is appended at the end of this document.

health or receiving poorer health care. Some TEP members were also interested in the possibility of pairing
parent and child interviews into dyads that could be analyzed as a set.
ii.

Income Technical Expert Panel.

NCHS held two meetings in April 2016 with the income TEP, a panel consisting of eight members from federal
agencies and research organizations. These affiliations included: the Agency for Healthcare Research and
Quality, National Center for Health Statistics’ Division of Research and Methodology, HHS’ Office of the Assistant
Secretary for Planning and Evaluation, University of Michigan’s Survey Research Center, and the U.S. Census
Bureau.
In the 1997-2017 NHIS, the family respondent is asked about various sources of income received by family
members. For the redesign, members of the TEP noted the importance of continuing to ask the questions on
some of the various sources of income. Other ideas included rethinking the income reference period, and
establishing a clear definition for who is included in the definition of a family.
iii.

Pain Technical Expert Panel.

The Pain TEP, consisting of seven members representing governmental agencies, non-profits, and academics,
met twice in May 2016. The panel included members affiliated with the CDC/National Center for Chronic
Disease Prevention and Health Promotion, Group Health Research Institute, National Institute of Neurological
Disorders and Stroke, and Stanford University. The members of the TEP focused on the impact of chronic pain
rather than the area of the body in which the pain occurs, and used the recently published National Pain
Strategy as a framework for their discussions.
iv.

Injury Technical Expert Panel.

The Injury TEP consisted of a panel of 30 members from federal agencies, research and academic
institutions. This group met monthly from June 2015 through February 2016. The Injury TEP included individuals
affiliated with the Agency for Healthcare Research and Quality, CDC/National Center for Chronic Disease
Prevention and Health Promotion, CDC/National Center for Injury Prevention and Control, CDC/National Institute
for Occupational Safety and Health, Center for Injury Research and Policy, the Research Institute at Nationwide
Children's Hospital, Colorado School of Public Health, Colorado State University, Johns Hopkins Center for Injury
Research and Policy, Liberty Mutual, National Safety Council, Pacific Institute for Research and Evaluation, the
Children's Hospital of Philadelphia, University of Iowa College of Public Health, and the University of Maryland
School of Medicine. Members of this panel reviewed the utility of the poisoning data and noted that while
sample sizes were small, the NHIS would continue to be an important data source for injury epidemiology and
surveillance.

The input from these TEPs and other subject matter experts was iteratively integrated into draft versions of the
redesigned questionnaire. The first draft of the proposed overall questionnaire structure and sample adult
questionnaire topic areas was presented to NCHS leadership, the HHS Assistant Secretary for Planning and
Evaluation, and released to the public in late 2015. A first draft of the sample child topic areas was released soon
after (April 2016) and both questionnaire sections were presented to the NCHS Board of Scientific Counselors in
May 2016.

b. Calls for Public Feedback
In addition to convening subject matter experts throughout the TEP process, the general public was also
consulted at multiple key decision points. Specifically, three calls for feedback were put out in which members
of the public were asked to provide their thoughts more informally, directly to NCHS via email. In addition, a
formal 30-day notice for public comment was placed in the Federal Register.
i.

First call for feedback

In October 2015, the first call for public feedback was posted on the NCHS website and disseminated via the
NHIS email listserv. The announcement informed the public that the NHIS would be redesigned in 2018 with the
goals of improving the measurement of health topics, reducing respondent burden by shortening the
questionnaire, harmonizing NHIS data with that on other federal surveys, establishing a long-term structure of
ongoing and periodic topics, and incorporating best practices in survey methodology and measurements. In this
announcement, the proposed plan was presented to keep some content on the survey annually while other
content would rotated on and off with a fixed and pre-established periodicity. Additionally, it was announced
that supplements would be limited to 5 minutes per sponsor, per year.
Commenters were requested to consider the uses and types of products for which they use NHIS data, the
potential consequences on their work of losing NHIS as a data source, content areas no longer relevant or with
limited utility to them, the minimum periodicity and number of consecutive years of survey data they require for
their work, and the number and types of covariates needed to allow for meaningful use of NHIS content areas.
Commenters were asked to support their ideas with evidence-based information when available.
This call for feedback resulted in 60 emails from CDC, agency partners, academia, non-profits, small businesses,
and the general public. These comments echoed serval reoccurring themes. Many stressed the importance of
collecting broad measures of disability. Another key theme was the recommendation to maintain health
insurance and utilization measures in the NHIS.
Information obtained through this first call for feedback was combined with information gleaned from the TEPs
and meetings with key stakeholders to create a tentative content draft of topics to be covered in the sample
adult NHIS redesign, and identify content areas under development.
ii.

Second call for feedback

In February 2016, public feedback was invited on this tentative content draft, again via announcement on the
NCHS website and dissemination to the email listserv. At this time, commenters were invited to weigh in on the
benefits and drawbacks of eliminating the design element of having a separate family respondent to answer
questions about all members of the family, relying instead on collection of a more limited scope of information
as part of the remaining sample adult and sample child modules. Commenters were also asked to suggest
validated questions that could be asked in the areas under development.
This call for feedback generated 319 emails. Most of these comments (264) were form letters or variations of
form letters based on templates circulated by groups opposed to the elimination of the family questionnaire.
Commenters emphasized the importance of having data on the health insurance status and type of all family
members, requested that additional demographic information be collected about each family member, and
called for improved measurement of same-sex and multiracial couples. Other comments emphasized the need
to improve measurement of mental health, with some commenters advocating for a particular measure of
mental health distress, depression, and/or anxiety.
In reaction to the concerns expressed by commenters about the loss of content from discontinuation of the
family section, NCHS released a public statement to allay these concerns and clarify the content areas that

would remain on the survey, albeit integrated into the adult and child sections. Also, NCHS reconsidered and
ultimately retained questions on the education and employment of all adults in the household.
iii.

Third call for feedback

In June 2016, the third call for public feedback was announced on the NCHS website and through the NHIS
listserv to solicit reactions on content drafts of both the sample adult and child questionnaires. Prior
commenters were explicitly invited to review the new content drafts and provide additional comments.
This third call resulted in approximately 70 comments from the public, academics, state and federal
government, and nonprofit/advocacy groups, and many repeat commenters. Most comments addressed the
changes to the family section, disability, injury, and mental health.
iv.

Federal Register Notice

A formal call for public comments was undertaken via a non-Paperwork Reduction Act (PRA) 30-day Federal
Register Notice (FRN), which was published on October 7, 2016. Attached to this FRN were the draft
questionnaires for the sample adult and sample child annual and rotating core, which included the proposed
wording of each question, skip patterns, universe of people of whom the question would be asked, and
proposed answer categories.
There were 60 comments received in response to this FRN. Two thirds of these comments had to do with four
topics. These included the measurement of disability, and request for new or additional questions on oral
health and tooth pain, specific food and other types of allergies, and the developmental status and behaviors of
young children aged 0-4 years.
v.

Response to the calls for feedback

Each individual comment received as part of the three calls for feedback and the non-PRA 30-day FRN was
recorded in a database, noting the topic, affiliation of commenter, survey section to which the comment applied
(i.e. family, sample adult, sample child, etc.) and whether the comment was suggesting an addition, deletion, or
change.
Each comment received was read and considered; in some cases, changes were made to the questionnaire.
Commenters responding to the three calls for public feedback disseminated via the NCHS website and listserv
were sent tailored responses or, in cases where the comments came in the form of form letters, were sent a
form letter. Commenters responding to the 30-day FRN were sent the standard CDC response.

NHIS REDESIGN TECHNICAL EXPERT PANEL ROSTERS
CHILD
First Child Technical Expert Panel (met on 2/2/16, 2/25/16, 3/8/16)
Rachel Caskey, MD, MAPP
Division Chief
Academic Internal Medicine and Geriatrics
University of Illinois College of Medicine at Chicago
[email protected]
312.413.1595
Reem Ghandour, DrPH, MPA
Director
Division of Epidemiology
Office of Epidemiology and Research
Maternal and Child Health Bureau
[email protected]
301.443.3786
Amy Houtrow, MD, PhD, MPH
Chief
Division of Pediatric Rehabilitation Medicine
Children’s Hospital of Pittsburgh
[email protected]
412.692.5097
Alex Kemper, MD, MPH, MS
Professor of Pediatrics
Duke University School of Medicine
[email protected]
919.668.8038
Russell Kirby, PhD, MS
Chair and Professor
Department of Community and Family Health
USF College of Public Health
[email protected]
813.396.2347
Michael Kogan, PhD
Director
Office of Epidemiology and Research
Maternal and Child Health Bureau
[email protected]
301.443.3145

Milton Kotelchuck, PhD, MPH
Professor of Pediatrics
Harvard Medical School
[email protected]
617.724.3594
Cheri Pies, MSW, DrPH
Clinical Professor
School of Public Health
University of California, Berkeley
[email protected]
510.643.1250
Heather Ringeisen, PhD, MS
Director
Children and Families Research Program
RTI International
[email protected]
919.541.6000
Edward Schor, MD
Senior Vice President
The Lucile Packard Foundation for
Children’s Health
[email protected]
650.736.2663
Snehal Shah, MD, MPH
Director
Research and Evaluation Office
Boston Public Health Commission
[email protected]
617.414.5946
Deborah Klein Walker, EdD
Vice President
Public Health and Epidemiology Practice
Abt Associates
[email protected]
617.492.7100
Bharathi Zvara, PhD
Research Assistant Professor
Maternal and Child Health
UNC Gillings School of Global Public Health
[email protected]
919.966.1370

Second Child Technical Expert Panel (met on 4/21/16, 4/28/16)
Wendy Baldwin, PhD
Former President and CEO
Population Reference Bureau
[email protected]
202.939.5409
Christy Bethell, PhD
Director
Child & Adolescent Health Measurement Initiative
Johns Hopkins Bloomberg School of Public Health
[email protected]
443.287.5092
J. Nadine Gracia, MD, MSCE
Deputy Assistant Secretary
Office of Minority Health
[email protected]
202.960.7694
Lynn Olson, PhD
Director
Department of Research
American Academy of Pediatrics
[email protected]
847.434.7631
Minh Wendt, PhD
Public Health Advisor
Office of Minority Health
[email protected]
202.960.7694

CHRONIC PAIN
Charles Helmick, MD
Medical Officer, IV
National Center for Chronic Disease Prevention and Health Promotion
Centers for Disease Control and Prevention
[email protected]
770.488.5456
Sean Mackey, MD, PhD
Chief, Division of Pain Medicine
Director of the Systems Neuroscience and Pain Laboratory (SNAPL) and Redlich Professor of Anesthesiology,
Perioperative and Pain Medicine, Neurosciences and Neurology
Stanford University
(Past President of American Academy of Pain Medicine)
[email protected]
650.498.6477
Linda Porter, PhD
Director
Office of Pain Policy
National Institute of Neurological Disorders and Strokes
National Institutes of Health
[email protected]
301.451.4460
Michael Von Korff, ScD
Senior Investigator
Kaiser Permanente Washington Health Research Institute
[email protected]
206.287.2874

INCOME
Jason Fields, PhD
Survey Director
Survey of Income and Program participation & National Children’s Health Survey
U.S. Census Bureau
[email protected]
301.763.2465
Steven Hill, PhD
Senior Economist
Center for Financing, Access, and Cost Trends
Agency for Healthcare Research and Quality
[email protected]
301.427.1672
David Johnson, PhD
Research Professor & Deputy Director (Panel Study of Income Dynamics)
Survey Research Center
University of Michigan
[email protected]
734.647.4076
Trivellore Raghunathan, PhD
Director
Survey Research Center
University of Michigan
[email protected]
734.764.8365
Thomas Selden, PhD
Director
Center for Financing, Access, and Cost Trends
Agency for Healthcare Research and Quality
[email protected]
301.427.1677
Kathleen Short, PhD
Research Economist
Social, Economic & Housing Statistics Division
U.S. Census Bureau
[email protected]
Joan Turek, PhD
Senior Economist
Office of the Assistant Secretary for Planning and Evaluation
[email protected]
202.690.5965

INJURY
Abay Asfaw
Economist
National Institute for Occupational Safety and Health
[email protected]
202.245.0635
Anasua Bhattacharya
Economist
National Institute for Occupational Safety and Health
[email protected]
513.533.8331
Matt Breiding
Research Scientist
National Center for Injury Prevention and Control
[email protected]
770.488.1396
Tim Bushnell
Economist
National Institute for Occupational Safety and Health
[email protected]
513.458.7108
Susan Carlson
Health Scientist
National Center for Chronic Disease Prevention and Health Promotion
[email protected]
770.488.6091
Lily Chen
Health Scientist
National Center for Health Statistics
[email protected]
202.205.4908
Ann Dellinger
Branch Chief
National Center for Injury Prevention and Control
[email protected]
770.488.4811
Carolyn DiGuiseppi
Associate Dean for Faculty, Professor, and Director of the Preventive Medicine Residency Program
Colorado School of Public Health
[email protected]
303.724.4415

Andrea Gielen
Professor
Johns Hopkins Center for Injury Research and Policy
[email protected]
410.955.2397
Christina Greene
University of Maryland School of Medicine
[email protected]
Siobhan Gruschow
Clinical Research Coordinator
The Children's Hospital of Philadelphia
[email protected]
Holly Hedegaard
Medical Officer
National Center for Health Statistics
[email protected]
301.458.4460
Ken Kolosh
Manager of Statistics
National Safety Council
[email protected]
Marcie-Jo Kresnow-Sedacca
Mathematical Statistician
National Center for Injury Prevention and Control
[email protected]
770.488.4753
Marie-Claude Lavoie
Epidemiologist
University of Maryland School of Medicine
[email protected]
David Lombardi
Principal Research Scientist
Liberty Mutual Research Institute for Safety
[email protected]
Karen Mack
Science Officer
National Center for Injury Prevention and Control
[email protected]
770.488.4389

Suzanne Marsh
Health Statistician
National Institute for Occupational Safety and Health
[email protected]
304.285.6009
Helen Marucci-Wellman
Senior Research Scientist/Epidemiologist
Center for Injury Epidemiology, Liberty Mutual Research Institute for Safety
[email protected]
Ted Miller
Senior Research Scientist
Pacific Institute for Research and Evaluation
[email protected]
301.593.7471
Ryan Mutter
Lead Social Science Analyst
Agency for Healthcare Research and Quality
[email protected]
240.276.2754
Corinne Peek-Asa
Professor, Associate Dean for Research
University of Iowa College of Public Health
[email protected]
319.335.4895
Keisha Pollack
Professor
Johns Hopkins Center for Injury Research and Policy
[email protected]
410.502.6272
Rose Rudd
Health Scientist
National Center for Injury Prevention and Control
[email protected]
770.488.3712
Yahtyng Sheu
Statistician
National Center for Health Statistics
[email protected]
301.443.1426

Gordon Smith
Adjunct Professor of Epidemiology & Public Health
University of Maryland School of Medicine
[email protected]
410.328.3847
Lorann Stallones
Professor, Director of the Graduate Program in Public Health
Colorado State University
[email protected]
970.491.6156
Ray Tapas
Senior Service Fellow
National Institute for Occupational Safety and Health
[email protected]
Margaret Warner
Statistician
National Center for Health Statistics
[email protected]
301.458.4556
Huiyun Xiang
Director and Principal Investigator
Center for Injury Research and Policy and Center for Pediatric Trauma Research, The Research Institute at
Nationwide Children's Hospital
[email protected]
614.355.5850