National Survey on Drug Use and Health:
2015 Text-to-Speech Pretest
SUPPORTING STATEMENT
A. JUSTIFICATION
The Substance Abuse and Mental Health Services Administration (SAMHSA) is requesting OMB approval to conduct 66 interviews as part of a Text-to-Speech (TTS) Pretest prior to the implementation of TTS software within the 2015 National Survey on Drug Use and Health (NSDUH) questionnaire. This Pretest will include two phases: a cognitive interview phase with 36 interviews and a pilot test phase with 30 interviews. TTS uses a computer-generated voice to read text displayed on-screen to respondents, rather than relying on pre-recorded audio files from a human voice. Using TTS offers an opportunity for work process efficiencies and cost savings in NSDUH’s software development. This TTS Pretest package is submitted under the NSDUH Methodological Field Tests generic OMB clearance (OMB No. 0930-0290).
NSDUH is sponsored by SAMHSA’s Center for Behavioral Health Statistics and Quality (CBHSQ) and approved under OMB No. 0930-0110. The data collection is a national survey of the U.S. civilian, non-institutionalized population aged 12 or older. This survey is paramount in meeting a critical objective of SAMHSA’s mission—to maintain current data on the incidence and prevalence of substance use and mental health problems in the United States. NSDUH has been conducted on a periodic basis from 1971 to 1988, and annually since 1990. The 2015 NSDUH will represent the thirty-fifth in the series.
NSDUH is authorized by Section 505 of the Public Health Service Act (42 USC 290aa4 – Data Collection). Section 505 specifically authorizes annual data collection for monitoring the incidence and prevalence of illicit substance use and mental health problems, as well as the abuse of licit substances in the U.S. population.
In order to continue producing current data, CBHSQ must periodically update the NSDUH to reflect changing substance use and mental health issues. CBHSQ plans to redesign the NSDUH for the 2015 survey year to achieve two main goals: 1) revise the questionnaire to address changing policy and research data needs, and 2) modify the survey methodology to improve the quality of estimates and the efficiency of data collection and processing. Planned revisions for the 2015 NSDUH to the questionnaire content, methodology and materials have already been thoroughly tested and refined, first in 2012 as part of the NSDUH Questionnaire Field Test (QFT) (OMB No. 0930-0334), then again in 2013 during the NSDUH Dress Rehearsal (DR) (OMB No. 0930-0334).
One element of the planned revisions not included in the QFT and DR was the use of TTS software within the audio computer-assisted self-interviewing (ACASI) portion of the NSDUH interview. Prior to implementing TTS for the 2015 NSDUH, SAMHSA is requesting approval of this TTS Pretest to conduct cognitive interviews in August 2014 and a pilot test in October-November 2014. The purpose of the TTS Pretest is to:
evaluate whether the use of TTS has any effect on the comprehension of specific survey items or administration of those items compared to the human voice;
determine the best TTS presentation speed for administering the survey questions;
evaluate participants’ preferences and opinions regarding the voices used for ACASI;
identify any major changes in administration time; and
identify unanticipated issues with the use of TTS.
The findings from the cognitive interviews will be prepared by early September 2014 to assist CBHSQ in a determination of whether TTS should be used on the 2015 NSDUH, and if so, the preferred speaking rate. If it is decided that TTS will be implemented on the 2015 NSDUH, the cognitive interview findings will be used to make refinements to the TTS audio to improve comprehensibility (e.g., altering the speed or modifying pronunciations), and then these refinements will be tested in the pilot test phase to identify any major issues with administration time or any unanticipated issues with the use of TTS. If determined that TTS will not be used for the 2015 NSDUH, then NSDUH will continue using audio files of a pre-recorded human voice.
The main purpose of the TTS Pretest is to evaluate whether the use of TTS on NSDUH has any effect on the comprehension of specific survey items or administration of those items compared to using the pre-recorded human voice, and then make a decision whether or not to implement TTS in the 2015 NSDUH based on the findings.
In 2013, CBHSQ completed an informal, in-house investigation into the features and quality of various TTS software packages in order to determine which, if any, software would be best suited for use in the ACASI portion of the NSDUH. This investigation’s objective was to determine whether NSDUH could transition from using a human voice for the English and Spanish ACASI portions of the survey to using an automated voice created through TTS software. This investigation concluded that advances in TTS technology have enabled realistic, accurate, and clear-sounding voices with a reduced level of effort to program and customize. The purpose of this proposed study request is to cognitively test the TTS-produced voice with potential NSDUH respondents, especially focusing on those who are likely to have some cognitive difficulty with the survey and rely more heavily on the audio component. These individuals are likely to be the youngest and oldest respondents (i.e. those aged 12-17 and 65 or older), as well as respondents with low levels of literacy and non-native English speakers who would complete the cognitive interview in English and Spanish. Additionally, the TTS software must be evaluated in the pilot test to determine any impact on administration time and any unanticipated issues in the field.
The cognitive interviews and pilot test planned as part of this TTS Pretest will run in a phased approach, with the pilot test phase following the cognitive interview phase, if TTS is deemed successful. If SAMHSA decides not to use TTS in the 2015 NSDUH based on the cognitive interview findings, then the pilot test will not be conducted. The cognitive interview phase will include approximately 36 participants across a diverse population to evaluate the comprehensibility of the TTS software on the NSDUH. The pilot test will include approximately 74 completed screenings and 30 completed interviews, 20 of which will be in English and 10 will be in Spanish, with selected participants using the 2015 NSDUH questionnaire to evaluate the timing and any unanticipated field issues associated with the use of TTS. The pilot test sample will be selected in Los Angeles, CA and Miami, FL to provide exposure to a variety of conditions and respondent demographics.
As explained in Section A.1, the findings from the cognitive interviews will assist CBHSQ in a determination of whether TTS should be used on the 2015 NSDUH, and if so, the preferred speaking rate. If it is decided that TTS will be implemented, the cognitive interview findings will be used to make refinements to the TTS audio to improve comprehensibility (e.g., altering the speed or modifying pronunciations), and then these refinements will be tested in the pilot to identify any major issues with administration time or any unanticipated issues with the use of TTS in the field. If determined that TTS will not be used for the 2015 NSDUH, then NSDUH will continue using audio files of a pre-recorded human voice.
Cognitive Interviews
Data for these cognitive interviews will be collected by Contractor staff in a face-to-face interview setting at the Contractor’s cognitive laboratory facility. Using select portions of the 2015 NSDUH CAI instrument, participants will listen to select survey questions read using three voices: human voice, TTS at moderate speed, and TTS at slow speed. Participants will only hear the questions and will not be able to see the text shown on screen. While this differs from the main study protocol, the purpose is to test the clarity of the audio without helping the participants by showing them the question text. Participants will not be asked to answer the recorded survey questions, rather they will be asked follow-up probes about the questions to assess how well they understood the words presented in the question. The complete list of survey questions and cognitive probes to be asked during these cognitive interviews is included as Attachment A, Protocols for Cognitive Interviews.
Each cognitive interview will be assigned a unique case number so that a respondent’s name will not be stored with their responses. The audio of the entire cognitive interview sessions will be recorded, including the CAI questions, interviewer probes, and participants’ responses. The audio will be recorded directly onto interviewer laptops using digital recording software (such as Audacity). The digital files of the recordings will be labeled with the respondent case number on laptops secured with Checkpoint Endpoint disk encryption software. The links between the numeric file names and respondent identities will be kept separately from the audio recordings at all times. The audio files will not be transcribed, and they will not be transferred to any removable media, such as a CD.
Cognitive interviewers will take notes electronically on their laptops, which are also secured with Checkpoint Endpoint disk encryption software. Thus, the data on the laptops will be encrypted. Other electronic files containing information such as telephone numbers will be password protected, with the password set to expire within four weeks after the final memo with the cognitive interview findings is completed and approved by CBHSQ. Both the electronic files and hardcopies will be destroyed at that time.
Pilot Test
NSDUH pilot test data will be collected in a face-to-face interview setting in respondents’ homes using the same materials, equipment and protocols as planned for the 2015 NSDUH, and were previously tested in the QFT and DR. The NSDUH field interviewers (FIs) selected to contact respondents and conduct interviews for the pilot test were previously trained on and took part in the QFT and/or the DR. Interviews will be administered using TTS within the ACASI modules for sensitive questions, which represent most of the interview.
The pilot test will utilize seven-inch, touch-screen Android tablets similar to those tested during the QFT and DR to conduct household screening interviews for these cases, as well as interview respondent selection, answering FI observation questions, and case management. The primary advantage of this computer-assisted methodology is accuracy in selecting the correct household member or members for an interview. The computer automatically selects the correct household member or members based on the demographic variables entered, thus substantially reducing the probability for human error. The hand-held computers also provide the benefits of complex case management tools and quick, secure electronic transfer of data.
Light-weight, ultra-book laptops, similar to those evaluated in the DR in 2013 and planned for use on the 2015 NSDUH, will be used for interviews completed as part of this pilot test. These have the advantage of being easy for FIs to transport in the field while providing ample processing power for the necessary computer programs. Because these laptops have solid state drives, they are more durable and reliable than previous generations of NSDUH data collection laptops.
The instrument fielded for the pilot test will be the 2015 NSDUH CAI Instrument. When completing the pilot interviews, respondents will wear headphones to listen to the ACASI questions read aloud by TTS software offered by Microsoft Speech Platform, which features a dynamic implementation mode that uses the TTS engine to read question text in real time and eliminates the use of pre-recorded audio files. The remainder of the interview will also be administered by the FIs using computer-assisted personal interviewing (CAPI). This mode has been used on NSDUH since 1999.
CBHSQ is in contact with major Federal health survey managers and is aware of no other pretest being conducted to assess the implementation of TTS software in the NSDUH questionnaire and how that might affect respondent comprehension and questionnaire administration time. To date, no duplication of effort has been identified.
This survey does not involve small businesses or other such entities.
The redesigned NSDUH survey will enter the field in January 2015. In order to meet this deadline, collection and reporting of the TTS Pretest results needs to take place in late 2014. This project is a one-time collection and will not be repeated.
This data collection is integral to implementing a redesigned NSDUH with TTS in 2015. Without testing prior to implementation, SAMHSA will not know the potential impacts the TTS software might have on respondent comprehension and questionnaire administration time.
This information collection fully complies with 5 CFR 1320.5(d)(2).
Appendix A of this Supporting Statement contains a listing of current consultants on the main NSDUH questionnaire.
There are no unresolved issues resulting from these consultations.
Cognitive Interviews
Both adult (aged 18 or older) and youth respondents (aged 12 to 17) in the TTS cognitive interviews will be given $40 cash for completion of the cognitive interview. The cognitive interviews will last, on average, 60 minutes. This incentive amount will be sufficient to compensate for the participants’ time and any travel expenses incurred (Willis, 2005). Given the relationship of the schedule for cognitive interviewing to the schedule for fielding the revised NSDUH instrument by January 2015, significant delays in recruiting participants with a lower incentive amount could adversely affect the timely implementation of these future activities.
The incentive for the cognitive interview is mentioned in the following materials: Protocols for Cognitive Interviews, Attachment A; Recruitment Flyers, Attachment B; Recruitment Screening Scripts, Attachment C; Parental Permission Forms, Attachment D; Participant Informed Consent Forms, Attachment E; and Receipt for Participation, Attachment F.
Pilot Test
Both adult (aged 18 or older) and youth respondents (aged 12 to 17) selected to complete one of the 30 interviews in the TTS Pretest pilot test will be given $30 cash upon completion of the full interview – as is also planned for the 2015 NSDUH. This incentive is consistent with the current NSDUH incentive, which was approved by OMB on October 18, 2001, for use in the 2002 NSDUH survey. Prior OMB approval was provided for the continued use of the $30 incentive for the 2003-2014 NSDUH surveys.
The incentive is mentioned in the following materials: Pilot Test NSDUH CAI Questionnaire Content, Attachment G; Lead Letter, Attachment H; Contact Cards – Sorry I Missed You & Appointment Cards, Attachment I; Study Description, Attachment J; Introduction and Informed Consent Scripts, Attachment K; Pilot Test Screening Questions, Attachment L; Question & Answer Brochure, Attachment M; and Interview Incentive Receipt, Attachment N.
Concern for the confidentiality and protection of respondents’ rights has always played a central part in the implementation of NSDUH and will continue to be given the utmost emphasis.
The Contractor’s Institutional Review Board (IRB) was granted a Federalwide Assurance (Attachment O) by the Office for Human Research Protections (OHRP) and HHS in compliance with the requirements for the protection of human subjects (45 CFR 46). The Contractor’s IRB will approve the protocols and consent forms for the 2015 NSDUH, which will be used for the pilot test. The Contractor’s IRB will also approve the protocols and consent forms for this TTS pretest prior to any respondent contact. The IRB’s primary concern is protecting respondents’ rights, one of which is maintaining the confidentiality of respondent information. By obtaining IRB approval for NSDUH procedures and materials, CBHSQ is assured that respondent confidentiality will be maintained.
Under CIPSEA, data may not be released to unauthorized persons. CIPSEA safeguards the confidentiality of individually identifiable information acquired under a pledge of confidentiality by controlling access to, and uses made of, such information. CIPSEA includes fines and penalties for any knowing and willful disclosure of individually identifiable information by an officer, employee, or agent of SAMHSA. Willful and knowing disclosure of protected data to unauthorized persons is a felony punishable by up to five years imprisonment and up to a $250,000 fine.
As CIPSEA agents, all Contractor staff complete an annual CIPSEA training and sign a notarized Confidentiality Agreement (Attachment R). FIs and Field Supervisors (FSs), who work for a subcontractor to the Contractor, will also complete CIPSEA and project training on ensuring respondent confidentiality and will have signed a notarized Data Collection Agreement (Attachment S) certifying they will keep all respondent information confidential.
Cognitive Interviews
The cognitive interviews for the NSDUH TTS Pretest will incorporate several procedures to ensure that respondents’ rights will be protected. The recruitment flyers (Attachment B) will advertise to the participants that, “All responses will be kept confidential under Federal law.” Also, the recruitment screening scripts (Attachment C), parental permission form (Attachment D), and the participant informed consent forms (Attachment E) all indicate to the participants that the interview will be conducted in private to ensure the following:
no one else will overhear their answers;
all of their answers will be kept private and confidential;
information given by the participants will not be shared with any persons outside the project staff;
their name will never be connected with the answers they provide;
and that federal law (CIPSEA) requires that their answers be kept confidential and used only for statistical purposes.
In these same study materials, participants are informed that their responses are voluntary and are assured there will be no penalties if they decide not to respond, either to the information collection as a whole or to any particular question.
Adolescents (i.e., aged 12-17) and older respondents (i.e., aged 65 and older), as well as respondents with low levels of literacy and non-native English speakers who would complete the interview in English, will be targeted for recruitment. All recruited participants will be screened over the telephone using a number provided on the advertisement. Once a caller is identified as being aged 12 to 17, the recruitment interviewer will ask the adolescent for permission to speak with a parent or legal guardian who can provide permission for the adolescent to participate. The interviewer will then follow a screening script (Attachment C, Recruitment Screening Scripts) to provide an initial explanation of the study for the parent before attempting to screen the adolescent.
During the recruitment process, information will be collected on the age, phone number, city of residence, any significant physical limitations that would preclude participation, and information about participation in other research studies. The telephone numbers collected will be used to either remind the participants about their upcoming appointments or to recruit additional participants who were placed on a “will call” list in the event that spaces opened up for them to be interviewed.
If selected for the cognitive interview, recruiters will schedule a time to conduct one-on-one interview appointments at the Contractor’s cognitive laboratory facilities. Parents will accompany adolescents to the interview. Upon arrival at the interview, the interviewer will review the consent form and assent form (Attachment D, Parental Permission Form) with both the adolescent and the guardian and will receive verbal consent. In the event that the participant is an adult, interviewers will review the consent form (Attachment E, Participant Informed Consent Forms) with the participant and collect verbal consent. To protect respondent anonymity, the informed consent/assent form will be signed only by the interviewer after receiving verbal consent/assent from the participant. Participants will receive a copy of the consent, assent, and parental consent forms.
Only those respondents who give verbal consent/assent to participate will be interviewed. Participants will also be asked to provide consent to have the interview audio recorded. In the event that observers are present, participants will provide consent for observation to take place. If participants decline to have the interview recorded or observed, the interview will still be conducted without any recording or observations.
All internal communication regarding a participant will include only the first name of the participant and time of interview. An example of internal communication would be when the recruiter notifies the interviewer of a scheduled appointment with a participant. Any other materials (including the recruitment screeners and the “will call” list of potential back-up participants who were screened but not interviewed) connecting the first name of the participant with his/her last name, telephone number, etc. will be locked in a cabinet (if in hardcopy form) or password protected (if in electronic form). Both the electronic files and hardcopies containing identifying information will be destroyed within four weeks after CBHSQ approves the final memo with the cognitive interview findings.
During the cognitive interviews, information will be collected on the age and gender of all participants but only to inform the questions and wording in the CAI program. Personal identifying information will not be included in the data or final memo delivered to CBHSQ.
Pilot Test
All procedures planned, and listed below, for the TTS Pretest pilot test are identical to those being implemented for NSDUH main study data collection in 2015. All NSDUH FIs are thoroughly trained in methods for maximizing a respondent’s understanding of the government’s commitment to confidentiality. Furthermore, FIs make every attempt to secure an interview setting in the respondent’s home that is as private as possible, particularly when the respondent is a youth.
Several procedures ensure that respondents’ rights are protected. First, the FI introduces himself or herself and the study using the Introduction and Informed Consent Scripts (Attachment K), reading the scripted text aloud to each interview respondent. This statement will appear in the Showcard Booklet (Attachment P) and is read aloud to each interview respondent. As part of the process for obtaining informed consent, respondents are given a Study Description (Attachment J), which includes information on the Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA, included as Title V in the E-Government Act of 2002, P.L. 107-347) and the protection that it affords. This statute prohibits disclosure or release, for non-statistical purposes, of information collected under a pledge of confidentiality. Specifically, the Study Description states that respondents’ answers will be used only by authorized personnel for statistical purposes and cannot be used for any other purpose. If a respondent is aged 12 to 17, except in rare instances where a 17-year-old lives independently from his or her parent or guardian (in which case the 17-year-old provides his or her own consent), when the youth is selected for the interview, the FI will read the parental introductory script (Attachment Q) to the parent or guardian requesting permission to speak with the youth about NSDUH. After that introduction, parental consent for the interview is obtained from the selected respondent’s parent or guardian, youth assent is requested and at least one parent, guardian or another adult must remain present in the home throughout the interview. After obtaining informed consent, FIs make every attempt to secure an interview setting in the respondent’s home that is as private as possible. In addition, the pilot test interview process, by design, includes techniques to afford privacy for the respondent. The ACASI portion of the questionnaire maximizes privacy and confidentiality by giving control of the sensitive questionnaire sections directly to the respondent. The ACASI methodology allows the respondent to listen to questions through a headset and/or to read the questions on the computer screen, and then key his or her own responses into the computer via the keyboard. At the end of the ACASI portion, the respondent’s answers are locked so that no one can see the responses until after the data are transmitted, processed, and aggregated by the Contractor.
To further ensure confidentiality, the respondent’s name, address, or other identifying information are never noted. The one exception is the Quality Control Form (Attachment T), which the respondent is asked to voluntarily complete at the end of the pilot test interview. The FI explains the procedures in advance, asking the respondent to record his or her phone number and current address on the Quality Control Form and then place the form in an envelope and seal the envelope. The Quality Control Forms are mailed directly to the Contractor’s office in North Carolina and used only for verification purposes.
Each day they work, FIs electronically transmit all completed screening and interview data to the Contractor’s servers via secure encrypted data transmission. On the data files, respondents are distinguished only by a unique number assigned to screenings and interviews. The data are processed immediately upon receipt at the Contractor’s facilities, and all associations between the respondent’s data and address are destroyed after all data processing activities are completed.
After delivery and acceptance of the final data files, all Quality Control Forms are destroyed, thus eliminating records of sample dwelling unit (SDU) addresses. The permanent sampling records show only the general location in which pilot test interviews were conducted; there is no record of specific dwelling units contacted.
This data collection is subject to the Privacy Act of 1974.1 Furthermore, the most recent Privacy Impact Assessment (PIA), updated by SAMHSA on December 26, 2013, would cover the 2015 NSDUH and thus this TTS Pretest (since this is processed annually).
Cognitive Interviews
Using the CAI instrument, participants in the TTS Pretest cognitive interviews will listen to select 2015 survey questions, many of which concern topics that are likely to be of a sensitive nature to many respondents. Participants will not be asked to answer the recorded survey questions, rather they will be asked follow-up probes about the questions to assess how well they understood the words presented in the question. For example, the probes will ask, “Can you tell me in your own words what this question was asking?” or “Do you recall the time frame that was asked about in this question?”
Measures will be taken to reduce risks to the respondents. Raw data from the screening questionnaires, paper cognitive interviewing protocols, and audio recordings that include sensitive information will be stored in locked cabinets (if in hardcopy form) or password protected (if in electronic form) during the recruiting and interview process. None of this information will be retained once the data has been extracted and aggregated; nor will the information become part of a system of records containing permanent identifiers that can be used for retrieval.
Pilot Test
Many of the NSDUH interview questions concern topics that are likely to be of a sensitive nature to many respondents. The content of those questions included in this TTS Pretest pilot test have already been successfully tested as part of the QFT and DR and are identical to those planned for implementation in the 2015 NDUH. As with those instruments, many safeguards, including the ACASI mode of questionnaire administration, improve the privacy of data collected on sensitive issues in the TTS pilot test. As a part of the interview introduction, the FI informs the respondent why the information is necessary, indicates who sponsors the study, requests consent to conduct an interview, and explains the procedures that ensure confidentiality. As noted in section A.10, for respondents between the ages of 12 and 17—except in rare instances where a 17-year-old lives independently without a parent or guardian and provides his or her own consent—verbal consent is obtained from both the parent or guardian and then the youth. (See Attachment K, Introduction and Informed Consent Scripts, for verbal consent text.) Once parental consent is obtained, every attempt is made to ensure that the actual interview is conducted without parental observation or intervention, though at least one parent, guardian or another adult must remain present elsewhere in the home throughout the interview.
Answers to sensitive questions, including all substance use, mental health, and sexual orientation and attraction questions, are obtained by closed interview design. In the ACASI portion of the interview, the respondent enters his or her answers directly into the computer. The FI does not see these answers. Several items previously in the CAPI portion of the NSDUH interview, including questions about school and work attendance, were tested in the DR in ACASI to offer increased privacy and will remain in ACASI for 2015 and the pilot test.
As explained in section A.10, all NSDUH data collected using Computer Assisted Interviewing (CAI) are transmitted regularly to the Contractor via secure encrypted data transmission and distinguished only with a unique number, which is a code associated with the SDU. The questionnaire data are processed immediately upon receipt at the Contractor’s facilities, and all associations between a questionnaire and the respondent’s address are destroyed after all data processing activities are completed. The listings of SDU addresses are kept under secured conditions and destroyed after all data processing activities are completed.
No signed consent forms are used; however, verbal consent is obtained as explained above.
Cognitive Interviews
Thirty-six participants will be interviewed in this portion of the study. It is expected that approximately 50 screenings with potential participants will be completed to obtain the 36 study participants.
Administration of the screening questionnaire during the cognitive interview recruitment process will take an average of five minutes per participant. It is estimated that the average amount of time required to conduct each cognitive interview will be approximately 60 minutes.
Both the recruitment and cognitive interviewing phases will span approximately three weeks in August 2014.
The respondent burden for this study is shown in Table 1 below. The hourly wage of $14.61 was calculated based on weighted data from the 2012 NSDUH and respondents' reported personal annual income.
Table 1. Estimated Burden for Cognitive Interviews
Activity |
Number of Respondents |
Responses per Respondent |
Average Burden per Response (Hours) |
Total Burden (Hours) |
Hourly Wage Rate |
Total Hour Cost |
Screening |
50 |
1 |
0.083 |
4.15 |
$14.61 |
$60.63 |
Full Cognitive Interviews |
36 |
1 |
1.000 |
36.00 |
$14.61 |
$525.96 |
TOTAL |
50 |
– |
– |
40.15 |
– |
$586.59 |
Pilot Test
For the pilot test, the sample has been designed to yield approximately 30 completed interviews. It will be necessary to sample approximately 113 households and complete approximately 74 screenings to obtain the requisite number of interviews. This sample size is required to ensure sufficient testing of the instrument across age groups and to include a sufficient number of interviews completed in both English (20) and Spanish (10).
Based on experience with the 2014 screening process and experience gained in 2012 and 2013 with the QFT and DR, administration of the screening questions is expected to take an average of five minutes per SDU.
Initial timing data indicate the NSDUH questionnaire tested in the DR took about 60 minutes to administer, on average. Since there are only a few changes to the 2015 questionnaire from the DR questionnaire, it is estimated that the average amount of time required to administer the 2015 NSDUH CAI Questionnaire (Attachment G) in the TTS pilot test will also be approximately 60 minutes, including two minutes for the Quality Control Form (Attachment T). The pilot test questionnaire will mirror that for the 2015 NSDUH.
The data collection field period for the pilot test is a little more than two weeks, spanning the period from October 26 – November 9, 2014. The annualized estimated respondent burden for the pilot test is shown in Table 3. The hourly wage of $14.61 was calculated based on weighted data from the 2012 NSDUH and respondents' reported personal annual income.
Table 2. Annualized Estimated Respondent Burden for Pilot Test
Instrument |
No.
of |
Responses per respondent |
Total number of responses |
Hours per response |
Total burden hours |
Hourly |
Total hour cost |
Household Screening |
74 |
1 |
74 |
0.083 |
6.14 |
$14.61 |
$89.71 |
Interview |
30 |
1 |
30 |
1.000 |
30.00 |
$14.61 |
$438.30 |
Total |
74 |
|
74 |
|
36.14 |
|
$528.01 |
There are no capital, startup, operational, or maintenance costs to respondents.
Total costs associated with the cognitive interviews and pilot test are estimated to be $136,727 over a 5-month period. Of the total costs, $123,690 are for contract costs (e.g., sampling, data collection, processing, reports), and approximately $13,037 represents CBHSQ costs to manage/administrate the surveys.
Currently there are 2,741 total burden hours in the OMB inventory. For the TTS Pretest cognitive interviews and pilot test, SAMHSA is requesting 40.15 and 36.14 burden hours respectively, or 76.29 total burden hours, leaving 2,665 burden hours.
The purpose of the TTS Pretest is to evaluate whether the use of TTS on NSDUH has any effect on the comprehension of specific survey items or administration of those items compared to using the pre-recorded human voice to help CBHSQ make a decision whether or not to implement TTS in the 2015 NSDUH based on the findings. Due to the limited number of participants, the results of this study will not be generalizable. The sample size and design do not allow for statistical inference to be conducted, and therefore, the analyses will be observational and anecdotal.
Cognitive Interviews
Debriefings with the cognitive interviewers will be conducted to learn about participants’ preference and comprehension of the survey questions and interviewer probes (included as Attachment U, Cognitive Interview Debriefing Questions). The findings from the cognitive interviews will be prepared in a brief memo by early September 2014 to assist CBHSQ in a determination of whether TTS should be used on the 2015 NSDUH, and if so, the preferred speaking rate.
The time schedule for the cognitive interviews is included in Table 3 below.
Table 3. Schedule for Cognitive Interviews
Subtask |
Due Date |
|
Recruiting for cognitive interviews begins |
August 7, 2014 |
|
Cognitive interviews begin |
August 8, 2014 |
|
Recruiting for cognitive interviews ends |
August 26, 2014 |
|
All cognitive interviews completed |
August 28, 2014 |
|
Submit draft memo with TTS cognitive interview results and recommendations to CBHSQ |
September 5, 2014 |
|
Submit final memo with TTS cognitive interview results and recommendations to CBHSQ |
September 18, 2014 |
|
Pilot Test
If it appears the TTS was successful during the cognitive interviewing phase, the cognitive interview findings will be used to make refinements to the TTS audio to improve comprehensibility (e.g., altering the speed or modifying pronunciations), and then these refinements will be tested in the pilot to identify any major issues with administration time or any unanticipated issues with the use of TTS. Timing data from the pilot test will be benchmarked against the timing data from the Dress Rehearsal and those from the 2013 main study. Results will be evaluated to assess whether the implementation of TTS affects overall instrument timing. The results will be summarized in a brief memo by late November to help CBHSQ evaluate the potential impact of using TTS on 2015 NSDUH data quality.
The time schedule for the pilot test is included in Table 4.
Table 4. Schedule for Pilot Test
Subtask |
Due Date |
Select FIs to conduct data collection |
June 16, 2014 |
Design area sample and select sample segments from retired listed segments |
June 30, 2014 |
Select dwelling units and prepare lists and maps |
August 28, 2014 |
Finalize data collection materials |
September 26, 2014 |
Finalize screening and interview questionnaire on TTS pilot test equipment |
October 8, 2014 |
Finalize FI training materials |
October 17, 2014 |
Conduct FI training sessions |
October 23, 2014 |
Data collection begins |
October 24, 2014 |
Data collection ends |
November 9, 2014 |
Submit draft memo with TTS pilot test results and recommendations to CBHSQ |
November 26, 2014 |
Submit final memo with TTS pilot test results and recommendations to CBHSQ |
December 12, 2014 |
The OMB expiration date will be displayed on data collection instruments.
The certifications are included in this submission and fully comply with 5 CFR 1320.9.
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
Cognitive Interviews
The Contractor will recruit a total of 36 participants and conduct cognitive interviews with English-speaking and Spanish-speaking participants. The Contractor will recruit a convenience sample of 24 English-speaking participants who are most likely to rely on the audio component of the ACASI portion of the interview, with at least 6 interviews in each of the following groups:
Participants aged 12-17
Participants aged 65 or older
Participants with low levels of education and/or literacy
Non-native English speakers who would conduct the interview in English
Across all groups, the Contractor will try to recruit a diverse population with respect to gender, ethnicity, and race. Participants will be recruited from the Research Triangle Park, NC area, Washington DC metro area, and Chicago, IL.
The Contractor will recruit participants aged 12-17, participants aged 65 or older, and non-native English speakers who would conduct the interview in English to participate using online advertisements, flyers posted at various locations (Laundromats, shelters, private and public charitable organizations, retirement communities), and word of mouth. All participants will be screened over the telephone using a number provided on the advertisements.
To recruit participants with low literacy levels, the Contractor will work with adult literacy programs. On a previous project, the Contractor worked with the Literacy Volunteers and Advocates organization in Washington, DC to recruit participants with less than a high school education or low-literacy levels for cognitive interviews. When working with literacy agencies, the Contractor will identify a contact at the organization and explain the nature of the study. The Contractor will provide the contact person with the name and number of the recruiter. The literacy advocate will provide the name and number of the recruiter to interested individuals.
The Contractor will also conduct interviews with 12 Spanish-speakers with limited or no English-speaking ability to complete the cognitive interview in Spanish. These participants will all be recruited from the Research Triangle Park, NC through a local Hispanic community center. The Contractor has had success with this method in the past to recruit Spanish-speaking participants.
Pilot Test
The respondent universe for the TTS Pretest pilot test is the civilian, non-institutionalized population aged 12 or older residing in the sampled areas. Eligibility for the pilot test will be determined based on where the occupants of the sampled dwelling units (DUs) reside for most of October, November and December 2014. Data collection will take place in the fourth quarter of 2014.
The pilot test will include approximately 20 interviews completed in English and 10 interviews completed in Spanish. The sample will be selected from areas within Los Angeles, CA and Miami, FL to meet staffing needs and ensure a sufficient number of Spanish interviews.
Retired Quarter 1 2014 segments will be used for selection. Based on past experience with these segments, three will be selected to yield the desired number of interviews, and each of these three segments will have a historically-high proportion of Spanish interviews. After accounting for eligibility, nonresponse, and the person-level sample selection procedures, it is estimated that approximately 113 DUs will need to be selected to yield the 30 completed interviews in order to identify respondents from specific age groups, as listed below. Also, as discussed in the data collection section, FIs will not return to convert refusals for the pilot test. This will be taken into account when selecting the sample.
In order to sufficiently evaluate the impact of TTS across various age populations, the respondent sample will be allocated across age groups so that 10 interviews are completed by youth respondents aged 12-17 and 20 interviews are completed by respondents age 18 and over.
Cognitive Interviews
Cognitive interviews will be conducted by the Contractor’s staff. After completing the informed consent process described in Section A.10, participants will be played audio recordings of survey questions using three voices: Human voice, TTS at moderate speed, TTS at slow speed. Participants will only hear the questions and will not be able to see the questions. While this differs from the main study protocol, it will test the clarity of the audio without helping the participants by showing them the question text. Participants will not be asked to answer the recorded survey questions, rather they will be asked follow-up probes about the questions to assess how well they understood the words presented in the question. These probes will include things like, “Can you tell me in your own words what this question was asking?” or “Do you recall the time frame that was asked about in this question?”
All participants will receive the same questions and the same probes, but the interviewer will vary which questions are presented in which voice. Participants will be played audio recordings of survey questions using three voices: human voice, text-to-speech moderate speed, and text-to-speech slow speed.
After each set of audio recordings, interviewers will ask participants a set of debriefing questions to assess participants’ thoughts on speed, inflection, comprehensibility, sound quality, and overall affective response (how well participants liked the voice), as shown in Attachment U. All cognitive interviewing participants will be paid $40 cash.
A copy of the protocol including the survey questions and follow up probes is included in Attachment A. In some cases, interviews will be observed by SAMHSA staff. In that case, participants will informed during the consent process that observers are interested in observing from another room and will be asked to provide consent for observers to be present.
Pilot Test
Unless otherwise specified, the pilot test procedures described in this section follow the same processes as planned for use on the 2015 NSDUH; however materials referenced in the attachments are specific for the pilot test. The pilot test will use either the TTS at moderate speed or TTS at slow speed, depending on what participants in the cognitive testing prefer.
Prior to the FI’s arrival at the SDU, a Lead Letter (see Attachment H) will be mailed to the resident(s) briefly explaining the survey and requesting their cooperation. Upon arrival at the SDU, the FI will refer the resident to this letter and answer any questions. If the resident has no knowledge of the Lead Letter, the FI will provide another copy, explain that one was previously sent, and then answer any questions. If no one is home during the initial visit to the SDU, the FI may leave a Sorry I Missed You Card (Attachment I) informing the resident(s) that the FI plans to make another callback at a later date/time. Callbacks will be made as soon as feasible following the initial visit.
When in‑person contact is made with an adult member of the SDU and introductory procedures are completed, the FI will present a Study Description (Attachment J) and answer any questions that person might have concerning the study. A Question & Answer Brochure (Attachment M) that provides answers to commonly asked questions may also be given. In addition, FIs are supplied with copies of the NSDUH Highlights & Newspaper Articles (Attachment V) for use in eliciting participation, which can be left with the respondent.
Similar to 2015, for the pilot test, FIs will be able to utilize the multimedia capability of the touch screen tablet to play a short video for respondents (approx. 50 seconds run time) which provides a brief explanation of the study and why participation is important. The script for this video is included as Attachment W. Interviewers have the option to play this video for potential respondents as a tool for gaining cooperation.
If a potential respondent refuses to be screened, the FI has been trained to accept the refusal in a positive manner, thereby minimizing the possibility of creating an adversarial relationship that might preclude future opportunities for contact. However, for the pilot test, any screening and interview refusals will be finalized at the initial refusal without any refusal conversion attempts because of the short data collection period.
With respondent cooperation, the FI will begin screening the SDU by asking either the Housing Unit Screening questions, or the Group Quarters Unit Screening questions, as appropriate. The screening questions are administered using a seven-inch touch screen Android tablet computer. A paper representation of the housing unit and group quarters unit screening process is shown in Attachment L.
Once all household members aged 12 or older have been rostered, the hand-held computer performs the within-dwelling-unit sampling process, selecting zero, one, or two members to complete the interview. For cases with no one selected, the FI asks for a name and phone number for use in verifying the quality of the FI’s work, thanks the respondent, and concludes the household contact.
For each person selected to complete the full interview, the FI follows these steps:
If the selected individual is aged 18 or older, or aged 17 and living independently from his or her parent or guardian, and is currently available, the FI immediately seeks to obtain informed consent. Once consent is obtained, the FI begins to administer the questionnaire in a private setting within the dwelling unit. As necessary and appropriate, the FI may make use of the Appointment Card (in Attachment I) for scheduled return visits with the respondent.
If the selected individual is 12 to 17 years of age, except in rare instances where a 17-year-old is living independently from his or her parent or guardian, in which case the 17-year-old provides his or her own consent, the FI will read the parental introductory script (Attachment Q) to the parent or guardian before speaking with the youth about NSDUH. Subsequently, parental consent is sought from the selected individual’s parent or legal guardian using the Parent section of the youth version of the Introduction and Informed Consent Scripts (Attachment K). Once parental consent is granted, the minor is then asked to participate using the Youth section of the same document. If assent is received, the FI begins to administer the questionnaire in a private setting within the dwelling unit with at least one parent, guardian or another adult remaining present in the home throughout the interview.
As mentioned in section A.11, the FI administers the interview in a prescribed and uniform manner with sensitive portions of the interview completed via ACASI.
Race/ethnicity questions are FI-administered and meet all of the guidelines for the OMB minimum categories. The addition of the finer delineation of Guamanian or Chamorro and Samoan, which collapse into the OMB standard Native Hawaiian/Other Pacific Islander category, were a requirement of the new HHS Data Collection Standards and were added to the 2013 and 2014 NSDUH interviews. They will continue to be included in the 2015 questionnaire.2
In order to facilitate the respondent’s recollection of prescription-type drugs and their proper names, as tested during the QFT and DR, pill images will appear on the laptop screen during the ACASI portions of interviews as appropriate. Also, respondents will use an electronic reference date calendar, which displays automatically on the computer screens when needed throughout the ACASI parts of the interview. Finally, in the FI-administered portion of the questionnaire, showcards are included in the Showcard Booklet (Attachment P) that allow the respondent to refer to information necessary for accurate responses.
For interviews conducted as part of this pilot test, respondents will wear headphones to listen to the ACASI questions read aloud by TTS software offered by Microsoft Speech Platform, which features a dynamic implementation mode that uses the TTS engine to read question text in real time and eliminates the use of pre-recorded audio files.
When the entire interview is completed and before the verification procedures begin, each respondent is given a $30 cash incentive and an Interview Incentive Receipt (Attachment N) signed by the FI.
For verification purposes, interview respondents are asked to complete a Quality Control Form (Attachment T) that requests his/her current address and phone number for possible follow‑up to ensure that the FI did his or her job appropriately. Respondents are informed that completing the Quality Control Form is voluntary. If he or she agrees, the respondent completes this form, places it in an envelope and seals it. The form is then mailed to the Contractor’s office for processing. In previous NSDUHs, less than one percent of the verification sample refused to fill out Quality Control Forms. However, while verification information will be collected for the pilot and FIs will mail the Quality Control Form to the Contractor’s office, verification will not be conducted given the small sample and short period of data collection.
As noted above, all interview data are transmitted on a regular basis via secure encrypted data transmission to the Contractor’s offices, where the data are subsequently processed and prepared for reporting.
Cognitive Interviews
To assure the participation of the recruited cognitive interviewing participants, each selected person will receive a reminder telephone call the day before the interview with directions to the interview location and an opportunity to ask any questions about the purpose or logistics of the study. The $40 incentive will also help ensure participation.
Pilot Test
Procedures to maximize response rates for the pilot test will be based on past and current experience with NSDUH’s main study data collection. In this section, any differences from the main study are indicated below:
Lead letters will be sent to selected dwelling units with valid mailing addresses, but no follow-up contact will occur through mail. Refusal and unable-to-contact letters will not be sent for the pilot test.
Efforts will be made to successfully complete each case within the data collection period. However, given that data collection will only last approximately two weeks, any screening or interview refusals will be finalized at the initial refusal without any refusal conversion attempts. In addition, unless an interview appointment is pending, all cases will be finalized once 30 interviews are completed with approximately 20 completed English interviews and 10 completed Spanish interviews.
The $30 incentive will also help ensure participation.
The activities to be conducted under this approval are in themselves tests of the TTS technology to be used in the 2015 NSDUH.
The basic NSDUH design was reviewed by statistical experts, both within and outside SAMHSA. Statistical experts reviewing portions of prior NSDUHs designs include William Kalsbeek, PhD, University of North Carolina; Robert Groves, PhD, Georgetown University; and Michael Hidiroglou, PhD, Statistics Canada. Monroe Sirken, PhD, National Center for Health Statistics (NCHS); James Massey, PhD, (deceased) also of NCHS; Douglas Wright, CBHSQ, SAMHSA (retired); Joseph Gfroerer, CBHSQ, SAMHSA (retired); and Arthur Hughes, CBHSQ, SAMHSA were consulted on the 1992 and subsequent survey designs. Peter Tice, CBHSQ, SAMHSA is the Government Project Officer, (240) 276-1254. Arthur Hughes, CBHSQ, SAMHSA is the primary mathematical statistician responsible for overall project management, (240) 276-1262. RTI senior statisticians contributing to the design are James Chromy, PhD, and Ralph Folsom, PhD.
Current NSDUH Consultants
Michael Arthur, Ph.D., Project Director (206) 685-3858
Social Development Research Group
University of Washington
Raul Caetano, M.D., Ph.D., Assistant Dean (214) 648-1080
Dallas Satellite MPH Program
University of Texas at Houston
John Carnevale, Ph.D., President (301) 963-2151
Carnevale Associates
Bill Kalsbeek, Ph.D., Associate Professor/Director (919) 962-3249
Survey Research Unit, Biostatistics
University of North Carolina at Chapel Hill
Graham Kalton, Ph.D. (301) 251-8253
Senior Vice President
Westat
Philip Leaf, Ph.D., Professor (410) 955-3962
Department of Mental Hygiene, Mental Health and Psychiatry
School of Public Health
Johns Hopkins University
Patrick O’Malley, Ph.D., Senior Research Scientist (734) 763-5043
Survey Research Center, The Institute for Social Research
University of Michigan
University of Maryland, School of Public Affairs
Peter Reuter, Ph.D. (301) 405-6367
School of Public Policy
University of Maryland
Attachment A. Protocols for Cognitive Interviews
Attachment B. Cognitive Interview Recruitment Flyers
Attachment C. Cognitive Interview Recruitment Screening Scripts
Attachment D. Cognitive Interview Parental Permission Forms
Attachment E. Cognitive Interview Participant Informed Consent Forms
Attachment F. Cognitive Interview Receipt for Participation
Attachment G. Pilot Test NSDUH CAI Questionnaire
Attachment H. Lead Letter
Attachment I. Contact Cards – Sorry I Missed You Card and Appointment Cards
Attachment J. Study Description
Attachment K. Introduction and Informed Consent Scripts
Attachment L. Pilot Test Screening Questions
Attachment M. Question & Answer Brochure
Attachment N. Interview Incentive Receipt
Attachment O. Federalwide Assurance
Attachment P. Showcard Booklet
Attachment Q. Parental Introductory Scripts
Attachment R. Confidentiality Agreement
Attachment S. Data Collection Agreement
Attachment T. Quality Control Form
Attachment U. Cognitive Interview Debriefing Questions
Attachment V. NSDUH Highlights and Newspaper Articles
Attachment W. Tablet Screening Video Script
References
Willis, G.B. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage Publications.
1 The SAMHSA System of Record Notice covering NSDUH is 09-30-0036 and 09-30-0049. See http://beta.samhsa.gov/privacy/pia for more information.
2 http://aspe.hhs.gov/datacncl/standards/ACA/4302/index.shtml
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | 2003 National Survey on Drug Use and Health |
| Author | lky |
| File Modified | 0000-00-00 |
| File Created | 2021-01-27 |