National Survey on Drug Use and Health:
Methodological Field Tests
SUPPORTING STATEMENT
A. JUSTIFICATION
1. Circumstances of Information Collection
Since the Substance Abuse and Mental Health Services Administration’s (SAMHSA) has the leadership responsibilities in the substance abuse and mental health communities, it has a responsibility to provide data of the utmost quality on a yearly basis. In order to accomplish this, SAMHSA’s Center for Behavioral Health and Quality (CBHSQ) 1 must update the National Survey on Drug Use and Health (NSDUH; OMB No. 0930-0110) regularly to reflect changing substance abuse and mental health issues, without impacting trend data. CBHSQ is planning to redesign the NSDUH for the 2015 survey year. The redesign will seek to achieve three main goals: 1) to bring the NSDUH costs in line with anticipated budget levels, 2) to revise the questionnaire to address changing policy and research data needs, and 3) to modify the survey methodology to improve the quality of estimates and the efficiency of data collection and processing.
In March 2008, CBHSQ received a three-year renewal of its generic clearance for methodological field tests (OMB No. 0930-0290; see Attachments A-C for reports and summaries generated from studies under this clearance). At this time, CBHSQ is requesting another renewal of the generic clearance to continue methodological field tests over the next three years, with conditions similar to the previous clearance. These methodological studies will be used to inform decisions regarding sample design, data collection methods, questionnaire format, data processing and estimation. Through these studies and other efforts, CBHSQ is hoping to realize a cost-efficient survey that collects high quality data.
These methodological tests will continue to be designed to examine the feasibility, quality, and efficiency of new procedures or revisions to existing survey protocol. Specifically, the tests will measure the reliability and validity of certain questionnaire sections and items through multiple measurements on a set of respondents; assess new methods for gaining cooperation and participation of respondents with the goal of increasing response and decreasing potential bias in the survey estimates; and assess the impact of new sampling techniques and technologies on respondent behavior and reporting. Research will involve focus groups, cognitive laboratory testing, and field tests.
The NSDUH has been conducted on a periodic basis from 1971-1988, and annually since 1990. The NSDUH is authorized by Section 505 of the Public Health Service Act (42 USC 290aa-4). Section 505 specifically authorizes annual data collection for monitoring the prevalence of illicit substances and the abuse of licit substances in the United States population.
NSDUH data are used by CBHSQ, the National Institute on Drug Abuse (NIDA), the Centers for Disease Control and Prevention (CDC), the Office of National Drug Control Policy (ONDCP), and other Federal agencies interested in the prevalence of substance use. This information collection is used to design prevention programs, respond to inquiries on the extent of substance use, estimate treatment need, study the social and economic impact of substance abuse, identify the correlates of substance use, and evaluate the overall impact that Federal and State programs have on drug demand. The NSDUH provides a useful indicator of individual states’ overall success at reducing youth substance abuse. In conjunction with other data sources, the NSDUH data provide a means for assessing and improving outcomes of prevention and treatment services.
The next wave of methodological tests will continue to examine ways to increase data quality, lower operating costs, and gain a better understanding of sources and effects of nonsampling error on the NSDUH estimates. Particular attention will be given to minimizing the impact of design changes so that survey data continue to remain comparable over time. If these tests provide successful results, current procedures or data collection instruments may be revised.
Methodological testing activities are expected to focus on assessing questionnaire modifications through cognitive interviews and improving response rates among persons residing in controlled access communities (locked apartment buildings, gated communities, college dormitories, etc.) and other hard-to-reach populations. Other activities currently under consideration are targeted at assessing the characteristics of nonrespondents and determining the feasibility of alternative sample designs and modes of data collection. Some studies may be combined to introduce efficiencies.
This submission is for generic approval for these methodological testing activities. As these tests are developed, their materials will be submitted to OMB for an expedited review. This will enable CBHSQ to continue methodological testing activities in a timely manner, given the tight data collection schedule. CBHSQ requests that OMB review the individual submission on an expedited basis and provide comments or approve the request within 2 weeks of receipt.
Focus groups have been a useful tool in developing new questions and materials for the NSDUH. They are useful as an early step in exploring a new issue or gathering opinions about a topic that has social relevance. Under the current generic clearance, focus groups were conducted to assess potential changes in the NSDUH contact materials. As new topics and requests arise frequently, CBHSQ anticipates the need for additional focus groups as one tool used in the efforts to be responsive to changing times.
Adaptive sampling techniques could be investigated to increase the yield of the drug-using population.
A nonresponse follow-up study may be conducted in which a subset of selected respondents who initially did not complete the NSDUH are recontacted. An incentive would be offered for these individuals to complete the interview and to provide information regarding the reasons for refusal, as well as information on their drug use and other characteristics to allow for the assessment of bias.
New computer hardware and software may be tested prior to implementation.
A pretest may be warranted to test modified procedures or to capture timing data for new questions in a given survey year. It would be conducted among a small subset of the NSDUH population, approximately 200 cases.
Field tests would be employed to determine the feasibility of these and possibly other design modifications with a small subset of the NSDUH population. They would be used to determine the potential benefits and costs of these changes if they were adopted in the main study.
Information will be collected through the use of face-to-face interviews, telephone interviews, self-administered questionnaires, or clinical evaluations, depending upon the subject matter being addressed. The face-to-face interviews and self-administered questionnaires will be conducted using computer-assisted interviewing (CAI). The main NSDUH study has been administered via CAI since 1999.
Before each new methodological test is developed, CBHSQ will review existing literature on the proposed topic, and consult with outside experts to evaluate available information in similar studies with comparable populations.
The methodological tests will not include small businesses or other such entities as respondents.
CBHSQ is responsible for providing quality, timely data to the public on an annual basis. Methodological tests are necessary to keep up with changes in substance use and mental health without affecting trend measurement. For the majority of planned field tests, respondents will be interviewed once and will not be re-contacted.
CBHSQ recognizes the need to collect information in a manner that places minimal burden on each respondent. Therefore, when CBHSQ recruits prospective participants for each methodological test, they will explain the purpose of the study, the approximate length of time that it will take, and the voluntary nature of participation. All efforts will be made to keep the data collection instrument for each test short and well focused. This data collection is fully consistent with 5 CFR 1320.5(d)(2).
A number of experts on survey methodology, substance abuse and mental health have provided consultation on key issues related to the redesign of the NSDUH. Consultations with experts in these in these fields will continue as methods study development progresses. Consultants will be identified based on the topic of the individual submissions and their names and contact information will be provided in the clearance packages.
A Federal Register notice published on October 8, 2010 (75 FR 62403) solicited one comment on the 2011 NSDUH. The comment was received from the New York State Office of Alcoholism and Substance Abuse Services NYS (OASAS).
The letter from OASAS, along with CBHSQ’s response, is in Attachment D. In summary, OASAS had comments on the frequency of the survey and asked that various questions on cocaine, crack, prescription drugs, dependence, and adult mental health utilization be revised. The letter also contained a request that questions be added for special populations, such as those in recovery, lesbian, gay, bisexual, and transgender respondents.
CBHSQ’s reply stated that the measurement of trends in the NSDUH is critical to understanding the progress made in the effort to reduce the use of alcohol, tobacco, and illegal drugs in the U.S and also to track mental health issues in the U.S. population. The changes requested would require a comprehensive redesign of the survey. Unless there is a significant error, CBHSQ attempts to avoid questionnaire modifications with the aim of preserving trend data. The next major redesign is currently planned for 2014, pending approval from management within the Department of Health and Human Services, ONDCP, and OMB.
CBHSQ’s response indicated that the new design will focus largely on the prescription drug module, including some of the revisions suggested. CBHSQ noted that several of OASAS’s other suggestions were already under study for the 2014 redesign, including a biennial design and the addition of recovery measures.
It is DHHS policy that all national surveys are reviewed by the Office of the Assistant Secretary for Planning and Evaluation (ASPE). The review was coordinated by Dale Hitchcock, Director, Division of Data Policy, Office of Science Policy, ASPE, (202) 690-7100. The DHHS Data Council has been kept informed about the status and plans for the 2011 NSDUH.
There are no unresolved issues resulting from any consultation at this time.
The following persons are the current consultants on the main NSDUH study.
Michael Arthur, Ph.D., Project Director (206) 685-3858
Social Development Research Group
University of Washington
Raul Caetano, M.D., Ph.D., Assistant Dean (214) 648-1080
Dallas Satellite MPH Program
University of Texas at Houston
John Carnevale, Ph.D., President (301) 963-2151
Carnevale Associates
Barbara Delaney (212) 973-3509
Director of Research
Partnership for a Drug-Free America
Bill Kalsbeek, Ph.D., Associate Professor/Director (919) 962-3249
Survey Research Unit, Biostatistics
University of North Carolina at Chapel Hill
Graham Kalton, Ph.D. (301) 251-8253
Senior Vice President
Westat
Philip Leaf, Ph.D., Professor (410) 955‑3962
Department of Mental Hygiene, Mental Health and Psychiatry
School of Public Health
Johns Hopkins University
Patrick O’Malley, Ph.D., Senior Research Scientist (734) 763-5043
Survey Research Center, The Institute for Social Research
University of Michigan
University of Maryland, School of Public Affairs
Peter Reuter, Ph.D. (301) 405-6367
School of Public Policy
University of Maryland
Survey research literature suggests that monetary incentives have a strong positive effect on response rates and no known adverse effect on reliability. It is standard practice in methodological research to offer recruited respondents an incentive to help assure their participation. The fee for each field test will be established during the development phase, and will be included in the materials for expedited review.
Concern for the confidentiality and protection of respondents’ rights has always played a central part in the implementation of the NSDUH and will continue to be given the utmost emphasis. Information provided by respondents will only be used by authorized personnel for statistical purposes and cannot be used for any other purpose. Prior to any data collection, respondents will be advised of the following: the nature of the activity; the purpose and use of the data collected; SAMHSA sponsorship; and the fact that participation is voluntary at all times. Since responses are voluntary, respondents will be assured there will be no penalties if they decide not to respond, either to the information collection as a whole or to any particular questions.
On November 9, 2006, the OMB approved CBHSQ as a statistical unit. As a result, CBHSQ is now required to follow the Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA) implementation guidelines in their sponsored surveys, including the NSDUH. CIPSEA provides a uniform set of confidentiality protections to all individually identifiable data collected for statistical purposes under a pledge of confidentiality. Under CIPSEA, penalties are imposed for willfully disclosing information to a person or agency not entitled to receive it; unlawful disclosure could be considered a class E felony with up to 5 years imprisonment or fines not to exceed $250,000.
As a further assurance of confidentiality, all presentation of data in reports will be in aggregate form, with no links to individuals being preserved. Reports will only be used by the project staff for research purposes and for the development of specific data collection questions and procedures.
Although some personal information will be collected, data will not be retrieved by personal identifiers during data analysis and data file preparation, and thus the Privacy Act does not apply to these activities.
Some studies may require the inclusion of people who match the characteristics of the target population for specific questions. This sometimes requires asking a question about race/ethnicity, income, education, and/or drug abuse or mental health problems on the initial screening questionnaire used for recruiting. Potential participants are informed that the reason these questions are asked is to make sure that CBHSQ speaks with the appropriate people for each particular test. Again, respondents will be assured that the information is voluntary and will be handled in a confidential manner.
Since the NSDUH survey deals with issues on drug use and mental health, some methodological tests may involve asking questions about (or discussing) personal experiences with such problems. Questions of this nature require some sensitivity in how they are worded and approached. In face-to-face data collections, every attempt will be made to ensure that the interview is conducted in as private a setting as possible.
Raw data from data collections that include sensitive information (for example, screening questionnaires, paper cognitive interviewing questionnaires and audio tapes) will not be retained once the data has been extracted and aggregated; nor will the information become part of a system of records containing permanent identifiers that can be used for retrieval.
The number of respondents to be included in each field test will vary, depending on the nature of the subject being tested and the target population. However, the total estimated response burden is 8,225 hours. This estimate is based on our previous generic clearance submissions and activities anticipated for the next several years. The exact number of subjects and burden hours for each test are unknown at this time, but will be clearly outlined in each individual submission. The table below, however, describes the anticipated burden for each of the major testing activities for which generic approval is being tested.
Estimated Burden for NSDUH Methodological Field Tests
Activity |
Number of Respondents |
Responses per Respondent |
Total Number of Responses |
Average Burden per Response |
Total Burden (Hrs.) |
Hourly Wage Rate |
Total Hour Cost |
a. Focus Groups |
270 |
1 |
270 |
2.0 hrs. |
540 |
$14.71 |
$7,943.40 |
b. Respondent screening for a. |
337 |
1 |
337 |
0.083 hr. |
28 |
$14.71 |
$411.88 |
c. Cognitive laboratory testing |
200 |
1 |
200 |
1.0 hr. |
200 |
$14.71 |
$2,942.00 |
d. Respondent screening for c. |
250 |
1 |
250 |
0.083 hr. |
21 |
$14.71 |
$308.91 |
e. Field Tests |
6,600 |
1 |
6,600 |
1.0 hr. |
6,600 |
$14.71 |
$97,086.00 |
f. Household screening for e. |
8,910 |
1 |
8,910 |
0.083 hr. |
740 |
$14.71 |
$10,885.40 |
g. Screening Verification for e. |
445 |
1 |
445 |
0.067 hr. |
30 |
$14.71 |
$441.30 |
h. Interview Verification for e. |
990 |
1 |
990 |
0.067 hr. |
66 |
$14.71 |
$970.86 |
TOTAL |
9,497 |
– |
9,497 |
– |
8,225 |
– |
$120,989.75 |
Annual Average (Total divided by 3 years) |
3,165 |
– |
3,165 |
– |
2,741 |
– |
$40,329.92 |
Estimated Annualized Burden for NSDUH Methodological Field Tests
Activity |
Number of Respondents |
Responses per Respondent |
Total Number of Responses |
Average Burden per Response |
Total Burden (Hrs.) |
a. Focus Groups |
90 |
1 |
90 |
2.0 hrs. |
180 |
b. Respondent screening for a. |
112 |
1 |
112 |
0.083 hr. |
9 |
c. Cognitive laboratory testing |
67 |
1 |
67 |
1.0 hr. |
67 |
d. Respondent screening for c. |
83 |
1 |
83 |
0.083 hr. |
7 |
e. Field Tests |
2,200 |
1 |
2,200 |
1.0 hr. |
2,200 |
f. Household screening for e. |
2,970 |
1 |
2,970 |
0.083 hr. |
246 |
g. Screening Verification for e. |
148 |
1 |
148 |
0.067 hr. |
10 |
h. Interview Verification for e. |
330 |
1 |
330 |
0.067 hr. |
22 |
Total |
3,165 |
– |
3,165 |
– |
2,741 |
Respondents participate on a voluntary basis as private individuals and, therefore, are subject to no direct costs other than their time to participate; there are no start-up or maintenance costs.
As stated earlier, the exact number of tests and subjects are unknown at this time. Therefore, the cost estimate is based on a number of assumptions and is likely to change. The total estimated cost to the Federal Government will be approximately $438,429. This includes $72,917 representing the estimated CBHSQ cost to manage the study. Each package developed for this generic clearance will have the estimated costs summarized in the supporting statement. The annualized cost burden is $146,143.
Currently 6,097 of the 6500 requested burden hours are remaining in the OMB inventory. CBHSQ is requesting 8,225 hours for this clearance. Experience from the last round of methods field tests, as well as an increase in expected testing activities in preparation for the redesign, prompted CBHSQ to make a program change, increasing the number of hours being requested by 1,725.
The data collection plan, schedule and analysis for each field test will be determined by the objectives of the methodological test, and will be included in the materials for expedited review.
The process for developing the analytical plan for the methodological tests will be similar to that used in any formal evaluation. CBHSQ staff will review the material to be pretested, discuss the objectives with the individuals responsible for developing the materials, determine the analytic questions to be addressed in the test, and then after resolution of any problems, approve the test procedures, instruments, and data analysis plan.
Techniques will primarily include qualitative analyses (for example, content analysis for results of cognitive studies), although some results may be summarized quantitatively using descriptive statistics. No complex analytic techniques will be used.
Approval is requested to not display the expiration date on Methodological Field Test materials. The exclusion of the expiration date provides the opportunity to use any applicable main study materials during the field tests and reduces the time and expense of printing all new materials.
The certifications are included in this package.
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
Conducting field tests includes a variety of methods and approaches. The methods chosen for use depend on the subject tested, as well as their intended target population. Recommended methodologies and sample sizes will be based on a review of the relevant literature and consultation with experts in the field. As the methodologies and sample sizes for each field test are determined, they will be detailed in submissions sent to OMB for expedited review.
Information collection procedures will be different for each methodological test, but will generally involve one of the four major methods outlined in section A2 and will be included in the materials for each expedited review.
Consistent with survey methodology, the design of each field test will include approaches to maximize response rates, while retaining the voluntary nature of the effort. CBHSQ will typically propose incentives at the level approved by OMB for cognitive laboratory testing ($40) and focus groups (up to $75). If a higher level incentive is proposed for approval, a meaningful justification will be provided. These details will be included in the materials for expedited review.
The activities to be conducted under this approval are in themselves tests of procedures. Interview guides and questionnaires to be used in the field tests will all be carefully developed and given careful scrutiny and limited, informal testing to assure completeness and smooth flow.
Michael Jones, Mathematical Statistician, Division of Population Surveys, CBHSQ, SAMHSA is the Government Project Officer, (301) 443-2674. Joseph Gfroerer, Director, Division of Population Surveys, CBHSQ, SAMHSA is the primary mathematical statistician responsible for overall project management, (301) 443-7977. RTI International statisticians contributing to the design are Dr. James Chromy, Chief Scientist and Director of Statistical Operations, Dr. Ralph Folsom, Chief Scientist and Director of Small Area Estimation, and Dr. Douglas Currivan, Operations Director for Methodological Analysis.
ATTACHMENTS
Attachment A ‑ OMB Executive Summary of NSDUH Youth Mental Health Service Questions
Attachment B - NSDUH State Data User Survey Final Report
Attachment C ‑ NSDUH Focus Groups for Redesigned Contact Materials Final Report
Attachment D - Response to Federal Register Notice
1 On August 17, 2010, the name of SAMHSA’s Office of Applied Studies was changed to the Center for Behavioral Health Statistics and Quality. http://edocket.access.gpo.gov/2010/pdf/2010-19856.pdf
| File Type | application/msword |
| File Title | 2003 National Survey on Drug Use and Health |
| Author | lky |
| Last Modified By | dicy.painter |
| File Modified | 2011-05-09 |
| File Created | 2011-05-09 |