OMB Ss-a_0290_5 29 14

9

2014-2017 National Surveys on Drug Use and Health:

Methodological Field Tests

SUPPORTING STATEMENT

A. JUSTIFICATION

1. Circumstances of Information Collection

The Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Behavioral Health Statistics and Quality (CBHSQ), is requesting approval for an extension to the generic clearance for methodological field tests for the National Survey on Drug Use and Health (NSDUH) (OMB No. 0930-0290) which expires on May 31, 2014. The NSDUH, sponsored by the Substance Abuse and Mental Health Services Administration (SAMHSA), is a national survey of the U.S. civilian, non-institutionalized population aged 12 and older. The NSDUH has been conducted on a periodic basis from 1971-1988, and annually since 1990. The conduct of the NSDUH (OMB No. 0930-0110) is paramount in meeting a critical objective of SAMHSA’s mission to maintain current data on the prevalence of substance use in the United States.

The NSDUH is authorized by Section 505 of the Public Health Service Act (42 USC 290aa4 – Data Collection).  Section 505 specifically authorizes annual data collection for monitoring the incidence and prevalence of illicit substance use and mental health problems, as well as the abuse of licit substances in the United States population.

NSDUH data are used by SAMHSA, the National Institute on Drug Abuse (NIDA), the Centers for Disease Control and Prevention (CDC), the Office of National Drug Control Policy (ONDCP), other Federal agencies, Congress, and various State and local government agencies interested in the incidence and prevalence of substance use and mental health statistics. The NSDUH questionnaire asks the minimum information necessary to meet the needs of Federal policymakers and the substance abuse research, prevention, and treatment communities. In conjunction with other data sources, NSDUH data are used to:

• design prevention programs;

• respond to inquiries on the extent of substance use;

• estimate treatment need;

• study the social and economic impact of substance abuse;

• identify the correlates of substance use;

• evaluate the overall impact that Federal and State programs have on drug demand and reducing youth substance use;

• assess and improve outcomes of prevention and treatment services;

• measure program performance and improvement, including Quality Outcome Measures, Government Relations and Public Affairs (GRPA), and other requirements; and

• identify areas where serious substance abuse problems exist and provide assistance to States to help them develop and adopt targeted responses for those problems.

In order to continue producing current data, SAMHSA’s Center for Behavioral Health Statistics and Quality (CBHSQ) must periodically update the NSDUH to reflect changing substance use and mental health issues. CBHSQ plans to redesign the NSDUH for the 2015 survey year to achieve two main goals: 1) revise the questionnaire to address changing policy and research data needs, and 2) modify the survey methodology to improve the quality of estimates and the efficiency of data collection and processing. In addition, CBHSQ may implement other changes to NSDUH that could be tested during 2014 through 2017.

These methodological tests will continue to be designed to examine the feasibility, quality, and efficiency of new procedures or revisions to existing survey protocol. Specifically, the tests will measure the reliability and validity of certain questionnaire sections and items through multiple measurements on a set of respondents; assess new methods for gaining cooperation and participation of respondents with the goal of increasing response and decreasing potential bias in the survey estimates; assess customer satisfaction; and assess the impact of new sampling techniques and technologies on respondent behavior and reporting. Research will involve focus groups, cognitive laboratory testing, and field tests. (Details are included under “2. Purpose and Use of Information”.)

Methodological testing activities are expected to also focus on assessing questionnaire modifications through cognitive interviews and improving response rates among persons residing in controlled access communities (locked apartment buildings, gated communities, college dormitories, etc.) and other hard-to-reach populations. Other activities currently under consideration are targeted at assessing the characteristics of nonrespondents and determining the feasibility of alternative sample designs and modes of data collection. Some studies may be combined to introduce efficiencies. Particular attention will be given to minimizing the impact of design changes so that survey data continue to remain comparable over time. If these tests provide successful results, current procedures or data collection instruments may be revised.

This submission is for generic approval for these methodological testing activities. As these tests are developed, their materials will be submitted to OMB for an expedited review. This will enable CBHSQ to continue methodological testing activities in a timely manner, given the tight data collection schedule. CBHSQ requests that OMB review the individual submission on an expedited basis and provide comments or approve the request within two weeks of receipt.

2. Purpose and Use of Information

The methodological field tests will assess the potential effectiveness of proposed revisions to the NSDUH data collection instruments and procedures, and will determine their impact on trends before implementation. The intent of each study will be to assess, maintain, or improve the overall quality of the NSDUH data. The individual studies to be submitted under this clearance will not be designed to produce population estimates. The methods proposed for use under this generic clearance are briefly described below.

Focus Groups

Focus groups have been a useful tool in developing new questions and materials for the NSDUH. They are useful as an early step in exploring a new issue or gathering opinions about a topic that has social relevance. Under the current generic clearance, focus groups were conducted to assess potential changes in the NSDUH contact materials. As new topics and requests arise frequently, CBHSQ anticipates the need for additional focus groups as one tool used in the efforts to be responsive to changing times.

Cognitive laboratory testing

New questions and modules are introduced into the NSDUH questionnaire on a regular basis. Currently, they undergo cognitive laboratory testing to assess problems with question wording, meaning and flow within the questionnaire. Potential groups recruited for cognitive testing include prescription drug users, clients of drug treatment centers, adolescents and members of the general public. Since federal requirements dictate OMB approval of studies dealing with 10 or more human subjects, the amount of testing that can be completed in a given survey year is limited due the time constraints involved in obtaining clearance. NSDUH annual cognitive testing would benefit from more time for development. Usability testing may also be conducted in a laboratory setting to explore different data collection modes, new software and/or hardware.

Field Tests

For the purposes of this clearance, field tests are defined as small data collections of 500 cases or less, designed to assess modifications in the survey instrument and/or data collection procedures. Field tests will incorporate the findings of any pre-tests and assesses the impact of the changes on a larger scale. The tests will provide a more in-depth examination of context effects associated with questionnaire changes with a small subset of the NSDUH population. They also present the opportunity to study how data collection protocol and materials changes can potentially affect response rates. Field tests will involve administering the entire survey, including any new questions and procedures, to a random sample of respondents. Several potential design changes methodological investigations have been proposed that would require a field test.

• An examination of the impact of the American Psychiatric Association’s adoption of DSM-5 criteria for mental disorders on the NSDUH.  This work may include exploring how DSM-5 revisions affect Major Depressive Disorder (MDE), Substance Use Disorder (SUD) (and correspondingly treatment need for SUD), and Any Mental Illness (AMI) and Serious Mental Illness (SMI) estimates.

• Adaptive sampling techniques could be investigated to increase the yield of the drug-using population.

• A nonresponse follow-up study may be conducted in which a subset of selected respondents who initially did not complete the NSDUH are re-contacted. An incentive would be offered for these individuals to complete the interview and to provide information regarding the reasons for refusal, as well as information on their drug use and other characteristics to allow for the assessment of bias.

• A study could be conducted to assess the feasibility of obtaining interviews from three respondents within a given household where three or more potential respondents reside (currently up to two persons within a household can be selected).

• The feasibility of various longitudinal-type survey designs (e.g., following persons over time, rotating panel designs, etc.) could be assessed.

• “Responsive designs” could be investigated where an optimal call limit on screening and interviewing households is determined based on the examination of collected survey data

• New computer hardware and software may be tested prior to implementation.

• Alternative sampling frames may be explored.

• A pretest may be warranted to test modified procedures or to capture timing data for new questions in a given survey year. It would be conducted among a small subset of the NSDUH population, approximately 200 cases.

Field tests would be employed to determine the feasibility of these and possibly other design modifications with a small subset of the NSDUH population. They would be used to determine the potential benefits and costs of these changes if they were adopted in the main study.

3. Use of Information Technology

Information will be collected through the use of face-to-face interviews and/or focus groups, telephone interviews, self-administered questionnaires including customer satisfaction surveys, or clinical evaluations, depending upon the subject matter being addressed. The face-to-face interviews and self-administered questionnaires will be conducted using computer-assisted interviewing (CAI). The main NSDUH study has been administered via CAI since 1999.

4. Efforts to Identify Duplication

Before each new methodological test is developed, CBHSQ will review existing literature on the proposed topic, and consult with outside experts to evaluate available information in similar studies with comparable populations.

5. Involvement of Small Entities

The methodological tests will not include small businesses or other such entities as respondents.

6. Consequences If Information Collected Less Frequently

CBHSQ is responsible for providing quality, timely data to the public on an annual basis. Methodological tests are necessary to keep up with changes in substance use and mental health without affecting trend measurement. Therefore, timely methodological tests are key for NSDUH data quality. Prior to each methodological test, a separate clearance memo (under this generic clearance) will be presented to OMB for review of each test.

7. Consistency With the Guidelines in 5 CFR 1320.5(d)(2)

CBHSQ recognizes the need to collect information in a manner that places minimal burden on each respondent. Therefore, when CBHSQ recruits prospective participants for each methodological test, they will explain the purpose of the study, the approximate length of time that it will take, and the voluntary nature of participation. All efforts will be made to keep the data collection instrument for each test short and well focused. This data collection is fully consistent with 5 CFR 1320.5(d)(2).

8. Consultation Outside the Agency

A number of experts on survey methodology, substance abuse and mental health have provided consultation on key issues related to the 2014-2015 redesign of the NSDUH. Consultations with experts in these in these fields will continue as methods study development progresses. Consultants will be identified based on the topic of the individual submissions (i.e., each methodological study) and their names and contact information will be provided in the respective clearance packages.

A Federal Register notice was published on December 16, 2013 (Vol. 78, page 76151) and one comment was received. CBHSQ received comments from Altria Client Services during the 60 Day Federal Register Notice (see Attachment C).  Their comments centered around two issues:  testing the menthol cigarettes survey item and coordinating with other surveys regarding data relevant to the Family Smoking Prevention and Tobacco Control Act (FSPTCA).  

SAMHSA and the Office of National Drug Control Policy (ONDCP) within the Executive Office of the President maintain the position that preserving trends is imperative to the continued success of the study.  With that in mind, a firm decision has been made to leave the core drug modules essentially unchanged, with the exception of the prescription drug modules and one minor change to smokeless tobacco items (combining chewing tobacco and snuff).  Altering questions within a module has the potential to cause unintended context effects on subsequent questions.  This effect has been seen in NSDUH non-core modules where the simple deletion of one question caused significant changes in the responses to the questions that followed.  Making the types of modifications suggested could introduce these types of effects and disrupt trends within the tobacco or successive modules for other substances. Nevertheless, SAMHSA continues to explore these topics for future non-core modules and maintains communications with the FDA and the Office on Smoking and Health to remain well-informed of emerging tobacco issues.

The Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation (ASPE) reviews all national surveys. ASPE reviewed this OMB request in February 2014.

There are no unresolved issues resulting from any consultation at this time.

The following persons are the current consultants on the main NSDUH study.

Michael Arthur, Ph.D., Project Director (206) 685-3858

Social Development Research Group

University of Washington

Raul Caetano, M.D., Ph.D., Assistant Dean (214) 648-1080

Dallas Satellite MPH Program

University of Texas at Houston

John Carnevale, Ph.D., President (301) 963-2151

Carnevale Associates

Bill Kalsbeek, Ph.D., Associate Professor/Director (919) 962-3249

Survey Research Unit, Biostatistics

University of North Carolina at Chapel Hill

Graham Kalton, Ph.D. (301) 251-8253

Senior Vice President

Westat

Philip Leaf, Ph.D., Professor (410) 9553962

Department of Mental Hygiene, Mental Health and Psychiatry

School of Public Health

Johns Hopkins University

Patrick O’Malley, Ph.D., Senior Research Scientist (734) 763-5043

Survey Research Center, The Institute for Social Research

University of Michigan

University of Maryland, School of Public Affairs

Peter Reuter, Ph.D. (301) 405-6367

School of Public Policy

University of Maryland

9. Payment to Respondents

Survey research literature suggests that monetary incentives have a strong positive effect on response rates and no known adverse effect on reliability. It is standard practice in methodological research to offer recruited respondents an incentive to help assure their participation. The fee for each field test will be established during the development phase, and will be included in the materials for each expedited review.

10. Assurance of Confidentiality

Concern for the confidentiality and protection of respondents’ rights has always played a central part in the implementation of the NSDUH and will continue to be given the utmost emphasis. Information provided by respondents will only be used by authorized personnel for statistical purposes and cannot be used for any other purpose. Prior to any data collection, respondents will be advised of the following: the nature of the activity; the purpose and use of the data collected; SAMHSA sponsorship; and the fact that participation is voluntary at all times. Since responses are voluntary, respondents will be assured there will be no penalties if they decide not to respond, either to the information collection as a whole or to any particular questions.

On November 9, 2006, the OMB approved CBHSQ as a statistical unit. As a result, CBHSQ is now required to follow the Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA) implementation guidelines in their sponsored surveys, including the NSDUH. CIPSEA provides a uniform set of confidentiality protections to all individually identifiable data collected for statistical purposes under a pledge of confidentiality. Under CIPSEA, penalties are imposed for willfully disclosing information to a person or agency not entitled to receive it; unlawful disclosure could be considered a class E felony with up to 5 years imprisonment or fines not to exceed $250,000.

As CIPSEA agents, all Contractor staff complete an annual CIPSEA training and sign either a notarized Confidentiality Agreement (Attachment A) or a notarized Data Collection Agreement (Attachment B) – each specific to an individual’s role on NSDUH – certifying they will keep all respondent information confidential.

As a further assurance of confidentiality, all presentation of data in reports will be in aggregate form, with no links to individuals being preserved. Reports will only be used by the project staff for research purposes and for the development of specific data collection questions and procedures. There will be no publication of estimates from the methodological tests.

Although some personal information will be collected, data will not be retrieved by personal identifiers during data analysis and data file preparation, and thus the Privacy Act does not apply to these activities.

11. Questions of a Sensitive Nature

Some studies may require the inclusion of people who match the characteristics of the target population for specific questions. This sometimes requires asking a question about race/ethnicity, income, education, and/or drug abuse or mental health problems on the initial screening questionnaire used for recruiting. Potential participants are informed that the reason these questions are asked is to make sure that CBHSQ speaks with the appropriate people for each particular test. Again, respondents will be assured that the information is voluntary and will be handled in a confidential manner.

Since the NSDUH survey deals with issues on drug use and mental health, some methodological tests may involve asking questions about (or discussing) personal experiences with such problems. Questions of this nature require some sensitivity in how they are worded and approached. In face-to-face data collections, every attempt will be made to ensure that the interview is conducted in as private a setting as possible.

Raw data from data collections that include sensitive information (for example, screening questionnaires, paper cognitive interviewing questionnaires and audio tapes) will not be retained once the data has been extracted and aggregated; nor will the information become part of a system of records containing permanent identifiers that can be used for retrieval.

12. Estimates of Annualized Hour Burden

The number of respondents to be included in each field test will vary, depending on the nature of the subject being tested and the target population. However, the total estimated response burden is 8,225 hours. This estimate is based on our previous generic clearance submissions and activities anticipated for the next several years. The exact number of subjects and burden hours for each test are unknown at this time, but will be clearly outlined in each individual submission. The table below, however, describes the anticipated burden for each of the major testing activities for which generic approval is being tested.

Estimated Burden for NSDUH Methodological Field Tests

Activity	Number of Respondents	Responses per Respondent	Total Number of Responses	Average Burden per Response	Total Burden (Hrs.)	Hourly Wage Rate	Total Hour Cost
a. Focus Groups	270	1	270	2.0 hrs.	540	$14.61	$7,889.40
b. Respondent screening for a.	337	1	337	0.083 hr.	28	$14.61	$409.08
c. Cognitive laboratory testing	200	1	200	1.0 hr.	200	$14.61	$2,922.00
d. Respondent screening for c.	250	1	250	0.083 hr.	21	$14.61	$306.81
e. Field Tests	6,600	1	6,600	1.0 hr.	6,600	$14.61	$96,426.00
f. Household screening for e.	8,910	1	8,910	0.083 hr.	740	$14.61	$10,811.40
g. Screening Verification for e.	445	1	445	0.067 hr.	30	$14.61	$438.30
h. Interview Verification for e.	990	1	990	0.067 hr.	66	$14.61	$964.26
TOTAL	9,497	–	9,497	–	8,225	–	$120,167.25
Annual Average (Total divided by 3 years)	3,165	–	3,165	–	2,741	–	$40,046.01

Estimated Annualized Burden for NSDUH Methodological Field Tests

Activity	Number of Respondents	Responses per Respondent	Total Number of Responses	Average Burden per Response	Total Burden (Hrs.)
a. Focus Groups	90	1	90	2.0 hrs.	180
b. Respondent screening for a.	112	1	112	0.083 hr.	9
c. Cognitive laboratory testing	67	1	67	1.0 hr.	67
d. Respondent screening for c.	83	1	83	0.083 hr.	7
e. Field Tests	2,200	1	2,200	1.0 hr.	2,200
f. Household screening for e.	2,970	1	2,970	0.083 hr.	246
g. Screening Verification for e.	148	1	148	0.067 hr.	10
h. Interview Verification for e.	330	1	330	0.067 hr.	22
Total	3,165	–	3,165	–	2,741

13. Estimates of Annualized Cost Burden to Respondents

Respondents participate on a voluntary basis as private individuals and, therefore, are subject to no direct costs other than their time to participate; there are no start-up or maintenance costs.

14. Estimates of Annualized Cost to the Government

As stated earlier, the exact number of tests and subjects are unknown at this time. Therefore, the cost estimate is based on a number of assumptions and is likely to change. The total estimated cost to the Federal Government will be approximately $445,380. This includes CBHSQ’s cost to manage the study. Each package developed for this generic clearance will have the estimated costs summarized in the supporting statement. The annualized cost burden is $148,460.

15. Changes in Burden

There is a 9 hour program decrease from the previous approval due to the CBHSQ’s experience with cognitive interviewing recruitment.

16. Time Schedule, Publication and Analysis Plans

The data collection plan, schedule and analysis for each field test will be determined by the objectives of the methodological test, and will be included in the materials for expedited review.

The process for developing the analytical plan for the methodological tests will be similar to that used in any formal evaluation. CBHSQ staff will review the material to be pretested, discuss the objectives with the individuals responsible for developing the materials, determine the analytic questions to be addressed in the test, and then after resolution of any problems, approve the test procedures, instruments, and data analysis plan.

Techniques will primarily include qualitative analyses (for example, content analysis for results of cognitive studies), although some results may be summarized quantitatively using descriptive statistics. No complex analytic techniques will be used.

17. Display of Expiration Date

Approval is requested to not display the expiration date on Methodological Field Test materials. The exclusion of the expiration date provides the opportunity to use any applicable main study materials during the field tests and reduces the time and expense of printing all new materials.

18. Exceptions to Certification Statement

The certifications are included in this package and fully comply with 5 CFR 1320.9.