National Survey on Drug Use and Health:
Methodological Field Tests
SUPPORTING STATEMENT
A. JUSTIFICATION
1. Circumstances of Information Collection
The Substance Abuse and Mental Health Services Administration (SAMHSA), sponsor of the National Survey on Drug Use and Health (NSDUH; OMB No. 0930-0110), formerly known as the National Household Survey on Drug Abuse (NHSDA), requests a generic clearance to conduct methodological field tests over the next three years. These field tests will be designed to examine the feasibility, quality, and efficiency of new procedures or revisions to existing survey protocol. Specifically, the tests will measure the reliability and validity of certain questionnaire sections and items through multiple measurements on a set of respondents; assess new methods for gaining cooperation and participation of respondents with the goal of increasing response and decreasing potential bias in the survey estimates; and assess the impact of new sampling techniques and technologies on respondent behavior and reporting. Research will involve household survey interviews and cognitive interviewing.
The NSDUH has been conducted on a periodic basis from 1971-1988, and annually since 1990. The NSDUH is authorized by Section 505 of the Public Health Service Act (42 USC 290aa4). Section 505 specifically authorizes annual data collection for monitoring the prevalence of illicit substances and the abuse of licit substances in the United States population.
NSDUH data are used by SAMHSA, the National Institute on Drug Abuse (NIDA), the Centers for Disease Control and Prevention (CDC), the Office of National Drug Control Policy (ONDCP), and other Federal agencies interested in the prevalence of substance use. This information collection is used to design prevention programs, respond to inquiries on the extent of substance use, estimate treatment need, study the social and economic impact of substance abuse, identify the correlates of substance use, and evaluate the overall impact that Federal and State programs have on drug demand. The NSDUH provides a useful indicator of individual states’ overall success at reducing youth substance abuse. In conjunction with other data sources, the NSDUH data provide a means for assessing and improving outcomes of prevention and treatment services.
Because mental health issues are correlates of substance abuse, SAMHSA made the decision that the NSDUH would be an appropriate survey for the inclusion of questions on mental health and utilization of mental health services. Of the different types of mental disorders classified in the DSM-IV, depression has become the greatest concern to federal policymakers, because it is related to suicide and it is viewed as a source of economic loss. To look specifically at depression, the 2004 NSDUH introduced two depression modules – one for adults and one for youths. The data collected focuses on lifetime and past year prevalence of major depressive episodes, past year treatment for it, and its severity and impact on functioning. These data are being used to obtain the prevalence and need for treatment of depression in the U.S., and allow for ongoing research into the interaction between depression and drug use.
In December, 2006 a meeting of expert consultants was convened by the Center for Mental Health Services (CMHS), SAMHSA, to solicit recommendations for mental health surveillance data collection strategies. The panel recommended conducting methodological studies to calibrate NSDUH mental health and impairment screening tools with a ‘gold standard’ clinical psychiatric interview to create a statistically sound measure that may be used to estimate the prevalence of serious mental illness (SMI) among adults (age 18+). As an initial step, a small Feasibility Study was conducted in June of 2007 to test procedures for collecting data on an expanded mental health screener module and for conducting follow-up telephone psychiatric interviews with selected respondents. Information from this Feasibility Study was used to inform the design and protocol development for a full calibration study to be conducted as an embedded split-sample follow-up study within the 2008 NSDUH.
Because of SAMHSA’s leadership responsibilities in the substance abuse and mental health communities, it has a responsibility to provide data of the utmost quality on a yearly basis. In order to accomplish this, SAMHSA must change the data collection instruments every year to reflect changing substance abuse and mental health issues, without impacting trend data.
The NSDUH methodological field tests will examine ways to increase data quality, lower operating costs, and gain a better understanding of sources and effects of nonsampling error on the NSDUH estimates. Particular attention will be given to minimizing the impact of design changes so that survey data continue to remain comparable over time. If these tests provide successful results, current procedures may be revised and incorporated into the main study (e.g., questionnaire changes).
Field test activities are expected to include improving response rates among persons residing in controlled access communities (locked apartment buildings, gated communities, college dormitories, etc.) and other hard-to-reach populations; and conducting a nonresponse follow-up study. Cognitive laboratory testing will be conducted prior to the implementation of significant questionnaire modifications. Other field tests currently under consideration include assessing questionnaire modifications through pre-tests, varying incentive amounts to understand the effectiveness of the current monetary incentive, examining the relationship between incentives and veracity of reporting, determining the feasibility of alternative sample designs and modes of data collection, and testing the feasibility of text-to-speech software. Lastly, a customer satisfaction survey of NSDUH data users will be conducted to improve the utility of the NSDUH data. Some studies may be combined to introduce survey efficiencies.
This submission is for generic approval to develop and conduct the field tests listed above. As these field tests are developed, their materials will be submitted to OMB for an expedited review. OMB will then approve or provide comments within 10 working days of receipt of the individual request.
The methodological field tests will assess the potential effectiveness of existing and proposed revisions to the NSDUH data collection instruments and procedures, and will determine their impact on trends before implementation. The individual field tests identified to date are briefly described below.
Several potential design changes have been proposed to improve participation among hard-to-reach populations. Adaptive sampling techniques could be used to increase the yield of the drug-using population. For controlled access and older populations, adjustment of interviewer training, carrying out local media campaigns, and revising lead letters and refusal conversion letters to emphasize salient concepts, have all been suggested. A field test could determine the potential benefits and costs of these changes if they were adopted in the main study.
This study would recontact a subset of potential respondents who initially did not complete the NSDUH. We anticipate offering a large incentive for these individuals to complete the interview and to collect in-depth information regarding the reasons for refusal and potential methods to avert other refusals of the type, as well as information on their drug use and other characteristics to allow us to assess bias.
This study would be conducted in concert with the Questionnaire Validity Study and would be designed to examine the effects of incentives on survey response. At the first request, the sample would receive no incentive. Those who initially refuse the interview will be offered a monetary incentive as an appeal to participate. In addition, all respondents will be offered an incentive to provide a hair and urine specimen after they complete the interview. This study would be especially informative regarding the relationship between incentives and truthfulness in reporting.
For the drug use validity component, biological specimens would be collected from select persons whose responses to the core drug modules showed varying degrees of drug usage. This study could provide information about the effects of any design modifications required for collecting biological specimens from respondents. It could thus serve as a predecessor for small, targeted studies to be embedded in the NSDUH over time.
New questions and modules are introduced into the NSDUH on an annual basis. Currently, they undergo cognitive laboratory testing to assess problems with question wording, meaning and flow within the questionnaire. Since federal requirements dictate OMB approval of studies dealing with ten or more human subjects, the time constraints involved in obtaining OMB approval limit the amount of testing that can be completed. NSDUH annual cognitive testing would benefit from more time for development, including Spanish translation.
A portion of the field test budget could be set aside for annual pre-testing of new questionnaire items and modules prior to their adoption in the main study. Several new or revised modules are foreseen at this point, including Dependence & Abuse and Drug Use & Life History as well as Drug Consumption. This project could also test relatively minor developments, such as increased font size, participant recruitment video clips, use of electronic calendars and pill cards, etc.
The field test incorporates the findings of any pre-tests and assesses the impact of the changes on a larger scale. The test will provide a more in-depth examination of context effects associated with questionnaire changes. It also presents the opportunity to study how data collection protocol and materials changes can potentially affect response rates. The field test will involve administering the entire survey, including any new questions and procedures, to a random sample of respondents.
The NSDUH uses a professional speaker, or “voice”, to provide audio recordings of the ACASI portion of the survey. The use of text-to-speech (TTS) software would eliminate the need to manually update the voice files, and would head off any problems should the “voice” become unavailable. The field test would determine the feasibility of using this technology, and compare human vs. computer-generated voices in terms of respondent acceptance, comprehension, and willingness to answer sensitive questions.
The testing of alternative sample designs could include, but are not limited to, the following: 1) investigating the feasibility of obtaining interviews from three respondents within a given household where three or more potential respondents reside (currently up to two persons within a household can be selected); 2) investigating the feasibility of various longitudinal-type survey designs (e.g., following persons over time, rotating panel designs, etc.); 3) use of adaptive sampling methods to increase the yield of drug users in the sample; 4) use of “responsive designs” where an optimal call limit on screening and interviewing households would be determined based on the examination of collected survey data; and 5) use of alternative sampling frames. Field tests would be employed to determine the feasibility of these and possibly other approaches.
The NSDUH employs ACASI and CAPI methods in a face-to-face interview. Several cost-saving mechanisms can be tested for NSDUH data collection. These include mixed mode designs in which part of the interview is conducted face-to-face and part is conducted using some other mode. Web-based data collection, video computer-assisted interviewing (VCAI), and telephone screening are among the methods for consideration.
In August of 2007, the Office of Applied Studies instituted a Customer Satisfaction Survey. There is a link to the survey embedded in each staff member’s email signature. This study will explore options for a similar survey specifically designed for NSDUH data users. Research will be conducted using NSDUH data users to determine the content and administration of the survey. The study will also assist in the development of a plan for reviewing and addressing user comments.
Information will be collected through the use of face-to-face interviews, self-administered questionnaires, or clinical evaluations, depending upon the subject matter being addressed. The face-to-face interviews and self-administered questionnaires will be conducted using computer-assisted interviewing (CAI). The main NSDUH study has been administered via CAI since 1999.
The text-to-speech field test would use a self-administered ACASI instrument to evaluate the differences between human and computer-generated voices.
Before each new field test is developed, SAMHSA will review existing literature on the proposed topic, and consult with outside experts to evaluate available information in similar studies with comparable populations.
The methodological field tests will not include small businesses or other such entities as respondents.
SAMHSA is responsible for providing quality, timely data to the public on an annual basis. Field tests are necessary to keep up with changes in substance use and mental health without affecting trend measurement. For the majority of planned field tests, respondents will be interviewed once and will not be re-contacted.
SAMHSA recognizes the need to collect information in a manner that places minimal burden on each respondent. Therefore, when SAMHSA recruits prospective participants for each field test, they will explain the purpose of the study, the approximate length of time that it will take, and the voluntary nature of participation. All efforts will be made to keep the data collection instruments for such field tests short and well focused. This data collection is fully consistent with 5 CFR 1320.5(d)(2).
A Federal Register notice published on September 4, 2007 (Volume 72, Number 170, page 50685) solicited comments on the proposed NSDUH methodological field tests. No comments were received. The following persons were consulted concerning plans for the NSDUH methodological field tests.
Ron Kessler, PhD. (617) 423-3587 Professor Department of Health Care Policy Harvard Medical School Boston, MA
|
Michael First, M.D. (212) 543-5531 Professor of Clinical Psychiatry Department of Psychiatry Columbia University Medical Center New York, NY
|
Phillip Wang, MD, PhD (301) 443- 6233 Director, Division of Services and Intervention Research National Institute of Mental Health Bethesda, MD
|
Michael Schoenbaum, PhD (301) 435-8760 Senior Advisor for Mental Health Services, Epidemiology and Economics National Institute of Mental Health Bethesda, MD |
Peter Reuter, PhD. (301) 405-6367 Professor School of Public Policy University of Maryland College Park, MD
|
Philip Leaf, PhD. (410) 955-3962 Professor of Mental Hygiene, Mental Health and Psychiatry School of Public Health Johns Hopkins University Baltimore, MD
|
Jonathan Caulkins, PhD. (412) 268-6076 Professor of Operations Research and Public Policy Carnegie-Mellon University Pittsburgh, PA
|
Rosalie Pacula, PhD. (310) 393-0411 Senior Economist RAND Corporation Santa Monica, CA |
Deborah Dawson, PhD. (301) 435-2255 Clinician National Institute on Alcohol Abuse and Alcoholism Rockville, MD
|
Bill Rhodes, PhD. (617) 349-2731 Principal Scientist Abt Associates Cambridge, MA |
John Pepper, PhD. (434) 924- 3402 Associate Professor Department of Economics University of Virginia Charlottesville, VA
|
Carol Petrie (202) 334-3732 Associate Chair National Research Council Washington, DC |
Graham Kalton, PhD. (301) 251-8253 Senior Vice President Westat Rockville, MD |
Patrick O’Malley, PhD. (734) 763-5043 Senior Research Scientist Survey Research Center The Institute for Social Research University of Michigan Ann Arbor, MI |
There are no unresolved issues resulting from these consultations.
Survey research literature suggests that monetary incentives have a strong positive effect on response rates and no known adverse effect on reliability. It is standard practice in methodological research to offer recruited respondents an incentive to help assure their participation. The fee for each field test will be established during the development phase, and will be included in the materials for expedited review.
Concern for the confidentiality and protection of respondents’ rights has always played a central part in the implementation of the NSDUH and will continue to be given the utmost emphasis. Information provided by respondents will be treated in confidence and kept private, except as otherwise required by law. This will be communicated to respondents by means of introductory letters, scripts read prior to interviews, and/or consent forms. Prior to any data collection, respondents will be advised of the following: the nature of the activity; the purpose and use of the data collected; SAMHSA sponsorship; and the fact that participation is voluntary at all times. Since responses are voluntary, respondents will be assured there will be no penalties if they decide not to respond, either to the information collection as a whole or to any particular questions.
On November 9, 2006, the OMB approved SAMHSA’s Office of Applied Studies (OAS) as a statistical unit. As a result, OAS is now required to follow the Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA) implementation guidelines in their sponsored surveys, including the NSDUH. CIPSEA provides a uniform set of confidentiality protections to all individually identifiable data collected for statistical purposes under a pledge of confidentiality. Under CIPSEA, penalties are imposed for willfully disclosing information to a person or agency not entitled to receive it; unlawful disclosure could be considered a class E felony with up to 5 years imprisonment or fines not to exceed $250,000.
As a further guarantee of confidentiality, all presentation of data in reports will be in aggregate form, with no links to individuals being preserved. Reports will only be used by the project staff for research purposes and for the development of specific data collection questions and procedures.
Although some personal information will be collected, data will not be retrieved by personal identifiers during data analysis and data file preparation, and thus the Privacy Act does not apply to these activities.
As mentioned above, some studies require the inclusion of people who match selected characteristics of the target population which SAMHSA is trying to reach. This sometimes requires asking a question about race/ethnicity, income, education, and/or drug abuse or mental health problems on the initial screening questionnaire used for recruiting. Potential participants are informed that the reason these questions are asked is to make sure that SAMHSA speaks with the appropriate people for each particular field test. Again, respondents will be assured that the information is voluntary and will be handled in a confidential manner.
Since the NSDUH survey deals with issues on drug use and mental health, some field tests may involve asking questions about (or discussing) personal experiences with such problems. Questions of this nature require some sensitivity in how they are worded and approached. In face-to-face data collections, every attempt will be made to ensure that the interview is conducted in as private a setting as possible.
Raw data from data collections that include sensitive information (for example, screening questionnaires, paper cognitive interviewing questionnaires and audio tapes) will not be retained once the data has been extracted and aggregated; nor will the information become part of a system of records containing permanent identifiers that can be used for retrieval.
The number of respondents to be included in each field test will vary, depending on the nature of the subject being tested and the target population. However, the Estimated Response Burden tables below provide an estimate of respondents and hours by type of data collection and an overall annual average. All prospective field test respondents will be from various subgroups of the general population.
The hourly wage of $14.02 was calculated based on weighted data from the 2006 NSDUH respondents' personal annual income.
Estimated Burden for NSDUH Methodological Field Tests
Activity |
Number of Respondents |
Responses per Respondent |
Average Burden per Response |
Total Burden (Hrs.) |
Hourly Wage Rate |
Total Hour Cost |
a. Improving participation among controlled access and other hard-to-reach populations |
417 |
1 |
1.0 hr. |
417 |
$14.02 |
$5,846.34 |
b. Nonresponse follow-up |
2000 |
1 |
1.0 hr. |
2000 |
$14.02 |
$28,040.00 |
c. Incentive / validity study |
2000 |
1 |
1.0 hr. |
2000 |
$14.02 |
$28,040.00 |
d. NSDUH questionnaire validity studies |
2500 |
1 |
1.0 hr. |
2500 |
$14.02 |
$35,050.00 |
e. Cognitive laboratory testing |
90 |
1 |
1.0 hr. |
90 |
$14.02 |
$1,261.80 |
f. Annual questionnaire pre-test |
670 |
1 |
1.0 hr. |
670 |
$14.02 |
$9,393.40 |
g. Field testing alternative questions, data collection protocol, contact materials |
1000 |
1 |
1.0 hr. |
1000 |
$14.02 |
$14,020.00 |
h. Text-to-speech software for voices in computer-assisted interviewing |
100 |
1 |
1.0 hr. |
100 |
$14.02 |
$1,402.00 |
i. Testing alternative sample designs (including alternative sampling frames) |
5000 |
1 |
1.5 hr. |
7500 |
$14.02 |
$105,150.00 |
j. Alternative modes of data collection (e.g., T-ACASI for Nonresponse follow-up) |
100 |
1 |
1.0 hr. |
100 |
$14.02 |
$1,402.00 |
k. Customer satisfaction survey of NSDUH data users |
100 |
1 |
.25 hr. |
25 |
$14.02 |
$350.50 |
Household screening for a-d, f-g, i-j |
12,471 |
1 |
0.083 hr. |
1,769 |
$14.02 |
$24,801.38 |
Screening Verification for a-d, f-g, i-j |
997 |
1 |
0.067 hr. |
43 |
$14.02 |
$602.86 |
Interview Verification for a-d, f-g, i-j |
1,497 |
1 |
0.067 hr. |
78 |
$14.02 |
$1,093.56 |
TOTAL |
28,942 |
– |
– |
18,292 |
– |
$256,453.84 |
Annual Average (Total divided by 3 years) |
9,647 |
– |
– |
6,097 |
– |
$85,484.61 |
Respondents participate on a voluntary basis as private individuals and, therefore, are subject to no direct costs other than their time to participate; there are no start-up or maintenance costs.
The total annual cost to the Federal Government will be approximately $835,000. This estimate is based on annual performance of up to: 1 in-person survey at $700,000; and 2 cognitive studies at $35,000 each. These figures include the costs of study design, data collection, analysis, and report/publication writing.
In addition, there will be annual costs of approximately $65,000 representing SAMHSA costs to manage this study. This estimate assumes that applicable SAMHSA staff will spend an average of 5% of their time per year on methodological field tests.
This is a new project.
The data collection plan, schedule and analysis for each field test will be determined by the objectives of the field test, and will be included in the materials for expedited review.
The process for developing the analytical plan for the field tests will be similar to that used in any formal evaluation. SAMHSA staff will review the material to be pretested, discuss the objectives with the individuals responsible for developing the materials, determine the analytic questions to be addressed in the field test, and then after resolution of any problems, approve the field test procedures, instruments, and data analysis plan.
Techniques will primarily include qualitative analyses (for example, content analysis for results of cognitive studies), although some results may be summarized quantitatively using descriptive statistics. No complex analytic techniques will be used.
The expiration date will be displayed.
The certifications are included in this package.
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
Conducting field tests includes a variety of methods and approaches. The methods chosen for use depend on the subject tested, as well as their intended target population. Recommended methodologies and sample sizes will be based on a review of the relevant literature and consultation with experts in the field. As the methodologies and sample sizes for each field test are determined, they will be detailed in submissions sent to OMB for expedited review.
The methodologies planned for use in this submission represent the standard state-of-the art approaches adapted from survey research. The following methodologies will be used:
Cognitive Interviews. These one-on-one interviews will be conducted using verbal probing and concurrent “think-aloud” techniques, in order to gain insight into the cognitive processes a respondent uses to answer survey questions. Annual cognitive interviews will be conducted in two rounds. Each round will include approximately 80 participants, evenly split between English speaking adults, Spanish speaking adults, English speaking adolescents and Spanish speaking adolescents.
|
R1 Adults |
R1 Adolescents |
R2 Adults |
R2 Adolescents |
Total |
English-Speaking |
20 |
20 |
20 |
20 |
80 |
Spanish-Speaking |
20 |
20 |
20 |
20 |
80 |
Total |
40 |
40 |
40 |
40 |
160 |
In-Person Interviews. For face-to-face interviews, the same methodology that is currently used in the main NSDUH study will be replicated. Households will be selected from the "retired" main study segments (e.g., segments used 2 to 3 quarters earlier that will no longer be used for the survey), and a probability based sample of persons will be selected from these households. Person probabilities of selection will be pre-loaded on the field interviewer's handheld computer screening device such that the selection(s) from the household roster is automatic.
Sample sizes and stratification will be specific to each field test and will be detailed in a submission to OMB prior to selecting the sample.
Information collection procedures will be different for each field test, and will be included in the materials for each expedited review.
Consistent with survey methodology, the design of each field test will include approaches to maximize response rates, while retaining the voluntary nature of the effort. These approaches will be included in the materials for expedited review.
The activities to be conducted under this approval are in themselves tests of procedures. Interview guides and questionnaires to be used in the field tests will all be carefully developed and given careful scrutiny and limited, informal testing to assure completeness and smooth flow.
Arthur Hughes, Mathematical Statistician, Division of Population Surveys, OAS, SAMHSA is the Government Project Officer, (301) 443-2674. Joseph Gfroerer, Director, Division of Population Surveys, OAS, SAMHSA is the primary mathematical statistician responsible for overall project management, (301) 443-7977. RTI International statisticians contributing to the design are Dr. James Chromy, Chief Scientist and Director of Statistical Operations, Dr. Ralph Folsom, Chief Scientist and Director of Small Area Estimation, and Mr. Joseph Murphy, Operations Director for Methodological Analysis.
| File Type | application/msword |
| File Title | 2003 National Survey on Drug Use and Health |
| Author | lky |
| Last Modified By | SKING |
| File Modified | 2007-10-25 |
| File Created | 2007-10-11 |