B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
The Contractor will screen a convenience sample of approximately 500 individuals to recruit up to 198 respondents across three rounds of pretesting interviews. The targeted final sample size for completed pretesting interviews in Round 1 is 60 (26 English-speaking adolescents aged 12-17 and 34 English-speaking adults). The targeted final sample size for completed pretesting interviews in Rounds 2 and 3 is 69 per round (26 English-speaking adolescents, 34 English-speaking adults, and 9 Spanish-speaking adults). All participants will consist of volunteers who are screened and determined to meet recruitment criteria.
English-Speaking Participants
Across the three rounds, a total of up to 180 pretesting interviews will be conducted with English-speaking participants in the following geographic areas: Research Triangle Park (RTP), NC; Washington, DC; Baltimore, MD; Portland, OR; Chicago, IL; Greenville, SC; Minneapolis, MN; Atlanta, GA; Charleston, WV; Charleston, SC; and Lincoln, NE.
The objectives mentioned previously require the pretest to include people who match selected characteristics of the target population NSDUH is trying to reach based on experiences with substance use, treatment or other factors (e.g., chronic pain or sleep issues). Table 4 shows the sample targets for English-speaking adolescent and adult participants in each round of interviewing.
Table 4. Targeted Final Sample Sizes per Pretesting Interview Round, by Age Group and Recruitment Type
Recruitment Criteria |
Total |
12-17 |
18+ |
Vaping nicotine Vaping nicotine and smoking cigarettes, past 30 days Vaping nicotine without smoking cigarettes, past 30 days Vaping just flavors or other substances, past year |
12 4 4
4 |
9 |
3 |
Marijuana Edibles, oil, wax, tinctures Leaves/stems Legal State Illegal State Medicinal Recreational |
8 2-6 2-6 2-6 2-6 2-6 2-6 |
2 |
6 |
CBD without marijuana |
2 |
1 |
1 |
Past year opioid users Prescription fentanyl Heroin, prescription pain relievers, or IMF |
4 2 2 |
1 |
3 |
Past year alcohol treatment MAT or MAT only Detox or detox only Other |
4 1 1 2 |
2 |
2 |
Past year drug treatment MAT or MAT only Detox or detox only Other |
8 2 2 4 |
4 |
4 |
Past year mental health treatment |
12 |
6 |
6 |
Past year sleep issues |
4 |
2 |
2 |
Past year pain issues |
4 |
0 |
4 |
Previously Incarcerated |
4 |
0 |
4 |
Total Criteria |
62 |
27 |
35 |
Unique Participants |
45-60 |
20-26 |
25-34 |
Youth: English-speaking participants aged 12 to 17 years old will be recruited in several ways. Adolescents with a recent history of vaping or past year difficulty sleeping will be recruited directly through flyers posted within vape shops and community organizations such as the YMCA. Youth may also be recruited via school suspension programs and the local juvenile justice system. No other adolescent criteria will be targeted using these ads. These vaping-only adolescents will be directed to complete a brief web recruitment screener. The Contractor’s IRB has granted a waiver of consent/permission from parents of adolescents recruited via community ads for vaping or sleep issues so these adolescents may complete this web recruitment screener that asks only about vaping nicotine, past year sleep issues and demographics. These vaping-only adolescents will be required to have a parent or guardian accompany them to the in-person interview.
For all criteria other than vaping or sleep issues, adolescents will be recruited in one of four ways. First, from inpatient and outpatient substance use or mental health treatment centers and support groups in the relevant geographic areas surrounding Research Triangle Park, NC; Washington, DC; Baltimore, MD; Portland, OR; Chicago, IL; Greenville, SC; Minneapolis, MN; Atlanta, GA; Charleston, WV; Charleston, SC; and Lincoln, NE. Contractor staff will contact the treatment centers via email, telephone or in person. Staff will explain the purpose of the study, provide a letter of support from CBHSQ explaining the importance of the study (Attachment K), and provide any additional details about NSDUH, SAMHSA or the pretesting interviews as requested. Contractor staff will ask treatment center or support staff to identify adolescents who meet the eligibility criteria (based on experiences with substance use, treatment or chronic pain/sleep issues), and provide these individuals or their parents with a copy of the advertisement (Attachment A), which references the $40 incentive.
Second, English-speaking adolescents will be recruited via their parents through online ads (e.g., craigslist) and printed flyers distributed to community organizations such as the YMCA, as well as school suspension programs and the local juvenile justice system. Parents will be informed via these advertisements (Attachment A, which references the $40 incentive) that adolescents with past 12-month substance use are needed for a research study.
Third, Contractor staff will work with focus-group facilities such as First in Focus, L&E Research or similar facilities to post or otherwise disseminate the recruitment ad targeting parents of adolescents to their membership database.
Fourth, adolescents will be recruited via word of mouth by sharing the recruitment ad or recruitment screener phone number with others.
Interested adolescents recruited through these last four methods (excluding the vaping-and sleep issues only population) will be required to complete a brief telephone recruitment screener to verify eligibility (Attachment C). Prior to completing the telephone recruitment screener with the adolescents, recruiters will obtain consent to complete the screener from a parent or legal guardian. Because adolescents recruited for criteria other than vaping and sleep issues are required to have parental consent to participate and must have a recent history of substance use or mental health concerns, all adolescents will be recruited through these four avenues to allow access to a population where parents are aware of the youth’s illicit substance use.
Adults: English-speaking adult participants will be recruited in the relevant geographic areas using four primary methods: 1) advertising in the classified sections of internet sites; 2) posting or distributing flyers at to inpatient/outpatient substance use and mental health treatment centers; 3) posting flyers at vape shops, cannabis dispensaries, or other organizations or businesses in the community; and 4) word of mouth.
In addition, recently-incarcerated adults will be recruited through probation officers and staff at half-way houses. Contractor staff will explain the purpose of the study and any additional details about NSDUH or the pretesting interviews as requested. Probation officers/staff will not be informed who participated in the study.
All recruitment advertisements (Attachment A) reference the $40 incentive. Interested adults recruited through these routes will be required to complete a brief web recruitment screener to verify eligibility (Attachment B).
Individuals who responded to the recruitment in Round 1 or Round 2 but were placed on a waiting list could still be eligible to be interviewed in a subsequent round. They will be rescreened to verify they meet the past 12-month substance use criteria.
Spanish-speaking participants
Approximately 30 Spanish-speaking individuals will be screened with the goal of enrolling up to 18 total respondents in Spanish-language pretesting interviews across Rounds 2 and 3. All Spanish-language participants will be 18 or older.
Spanish-speaking adult participants will be recruited primarily via Hispanic community center organizations and via word of mouth. If these methods fail to produce adequate numbers as described in Table 4, the Contractor will place newspaper ads or distribute flyers in other locations (e.g., treatment centers, medical clinics, student unions) after having received appropriate permissions. All advertisements (printed in Spanish) will reference the $40 incentive for participation.
The Contractor will conduct Spanish language interviews in the Research Triangle Park, NC area; Miami, FL; and Houston, TX.
Spanish-speaking participants do not need to meet any specific recruitment criteria. However, priority will be given to screening respondents who have used alcohol or another substance in the past 12 months.
Recruitment and Screening: Potential English-speaking adult participants will complete a web recruitment screener (Attachment B) to determine eligibility. Adults without access to the internet will be provided with the option to complete the recruitment screener via telephone. The recruiter will enter the screening respondent’s responses into the web screener. Potential participants will be screened for demographic information (age, sex, race/ethnicity, and education), the presence of any physical limitations that would preclude effective participation, as well as the specific recruitment criteria listed in Table 4. A recruiter will contact eligible adults to schedule the interview at the treatment center (for those recruited there), at one of the Contractor’s private offices, or another private setting such as a private room in a public library or community center. Callers will be notified they will receive $40 upon completing the interview.
Spanish-speaking adults may either complete the recruitment screening over the telephone or in person. A recruiter will schedule eligible participants for the interview at the Contractor’s private offices, or another private setting agreed upon by the pretesting interviewer and participant (such as the private room in a public library). Screening respondents will be notified they will receive $40 upon completing the interview.
Adolescents who vape or have sleep issues will complete an online recruitment screener to determine eligibility. The online recruitment screener for adolescents asks only about vaping nicotine, past year sleep issues, the presence of any physical limitations that would preclude effective participation and demographics. The Contractor’s IRB granted a waiver for parental permission for those adolescents who vape or have sleep issues to complete the web recruitment screener. The recruitment screener will let adolescents know whether they are eligible. Eligible adolescents will be informed that parental consent is required to schedule the interview. The recruitment screener will also alert adolescents to what their parents will be told about the interview when called: “The purpose of the study is to get feedback on how easy or difficult different questions about tobacco and nicotine are to answer. Respondents do not need to have used tobacco or nicotine to participate.” If adolescents choose to participate, they will then be asked to enter the name and phone number for their parent. Eligible participants will also be informed that a parent must accompany them to the interview. Contractor staff will then contact the parent to schedule for the interview to be held at one of the Contractor’s private offices. Callers will be notified the adolescent participant will receive $40 upon completing the interview.
For all criteria other than vaping or sleep issues, potential adolescent participants or adults calling for their adolescent child will be directed to call into the study line to verify eligibility rather than complete a web screener. For adolescent callers, the consent of a parent or guardian will be required before the adolescent can be asked any screening questions. Potential participants will be screened for demographic information (age, sex, race/ethnicity, and education), and the presence of any physical limitations that would preclude effective participation, as well as the recruitment criteria listed in Table 4. The recruiter will inform potential participants a parent or guardian must accompany the adolescent to the interview. Eligible adolescent participants will then be scheduled for the interview to be held at the treatment center or at one of the Contractor’s private offices. Callers will be notified they will receive $40 upon completing the interview.
After the requisite number of participants have been recruited, potential participants who call the Contractor’s designated telephone number or complete the recruitment screener online will be placed on a wait list, with their permission, but only for the duration of the three rounds of interviewing in this pretest. Individuals on this wait list may be called to ask if they are available, such as if one or more originally-scheduled participants did not arrive for a scheduled interview. Recruitment and contact information will be kept in locked cabinets and via password-protected electronic files and not shared except with those who are assigned to complete the interviews.
Interview Process: Pretesting interviews will be conducted by Contractor staff who are survey methodologists trained in conducting cognitive and usability interviews. Prior to Round 1, an interviewer training will be held to discuss the goals of the project and train interviewers on the procedures specific to this pretest.
When each participant arrives for an interview, he or she will be greeted and asked to listen to instructions and informed consent/assent information from the interviewer, and parental permission when applicable (Attachments D and E). All pretesting interviews will be audio/screen recorded upon consent of each participant and a subset may be observed by authorized project staff (in person or remotely), again upon consent of each participant. Participants will have the right to decline to be recorded and/or observed without being excluded from participation.
Before the interview begins, the participant will read the Participant Informed Consent Form (Attachment E). The interview will commence after the participant has given his or her verbal consent. The interviewer will begin by asking the participant for demographic information (to determine how the interview program should route the participant through questions in the interview) and then will show each respondent how to navigate through the interview program. Participants then will complete a tutorial to learn how to complete the ACASI portion of the survey.
Adolescents recruited via online ads or vape shops will only receive the core demographics, ACASI tutorial, nicotine and sleep modules due to the potential risk of asking sensitive questions to those not already seeking treatment for substance use or mental health issues.
All other adolescents and adult pretesting participants will complete a key set of modules including core demographics, ACASI tutorial, and the main substance use modules (except for prescription drugs) even if they were not screened as a past year substance user. This will ensure pretesting participants have an experience like that of NSDUH respondents. To reduce burden, these pretesting interviews will use abridged versions of the substance use modules, asking about lifetime use, recency, frequency and needle use (if applicable). Age of first use will only be asked for new questions on vaping and marijuana. Although all interview participants will receive these questions, only certain participants will be probed on these modules by the interviewer based on predetermined recruitment criteria. The prescription pain relievers and stimulants modules will only be asked of participants who reported use in the screening questionnaire, due to their administration time and because only the needle use questions are being revised. These participants will be asked the corresponding misuse questions from the main modules and needle use questions (if applicable).
The other modules will only be offered to participants eligible as a result of recruitment screening, as well as a small number of participants who did not screen in. Youth suicide and NSSI questions will only be tested with participants who have received treatment for mental health in the past year. All other modules will be administered to a combination of participants who met specific recruitment criteria as well as some who did not. It is important to test the questions with participants who do not meet the criteria to ensure that the questions do not generate false positives.
Tables 5 and 6 list the questionnaire modules that will be evaluated during pretesting and the general recruitment criteria proposed for testing with English-speaking adolescents and adults. Modules will be administered to a mixture of participants who screened in for those particular criteria as well as some who did not. It is important to test the questions with participants who do not meet the criteria to ensure that the questions do not generate false positives.
Table 5. Pretesting Modules Administered to All Participants (Except adolescents recruited for vaping or sleep)
Module |
Participants that will be probed |
|
Adolescents |
Adults |
|
Core Demographic |
|
|
ACASI Tutorial |
|
|
Nicotine
|
|
|
Alcohol |
|
|
Marijuana
|
|
|
Other Substances Use (e.g., inhalants, hallucinogens, cocaine, heroin, methamphetamine)
|
|
|
Needle
Use |
|
|
Table 6. Pretesting Modules Administered to Targeted Participants
Module |
Participants that will receive and be probed on |
|
Adolescents |
Adults |
|
Prescription Drugs |
|
|
Synthetic
Fentanyl |
|
|
Synthetic Marijuana & Bath Salts (Special
Drugs) |
|
|
Vaping Other (Special Drugs) |
|
|
Alcohol Treatment |
|
|
Drug Treatment |
|
|
Mental Health Treatment |
|
|
Youth Suicide |
|
|
NSSI |
|
|
Sleep items (Health) |
|
|
Pain Items (Health) |
|
|
Perceived Risk
|
|
|
Criminal Justice |
|
|
Pretesting interviews will be iterative with changes made to the questionnaire after each of the first two rounds based on the findings from that previous round. For all rounds, participants will be given the option of having the questions played over the computer’s speakers or turning off the sound and reading the questions aloud. These procedures are required so the interviewers can ask probes to a participant after certain questions. Providing participants the option of hearing the questions or reading them allows the pretesting interview process to more closely mimic an actual interview where participants can turn down the volume and read the questions if desired.
Respondents will be instructed to “think-aloud” as they complete the survey. To encourage respondents to think-aloud, interviewers may prompt as needed (e.g., “keep talking,” “what are you thinking,” “tell me what you’re doing,” “tell me what you’re looking at,” “tell me more about that.”). Interviewers will also be provided with a series of probes that will further explore the quality of responses and whether these responses are meeting the researchers’ goals (Attachment I). Interviewers are instructed to use the probes as a guideline, but are not required to use all of them, be limited by them, or to read them exactly as written (Beatty 2004). Interviewers may probe based upon the content of the interview and participant responses. Interviewers may ask spontaneous probes based upon the content of the interview and participant responses. Spontaneous probes such as, “Can you tell me more about that?” will be used as needed to further assess whether potential issues with question comprehension, recall, interpretation, and mapping their answer to the response categories.
Interviewers will also make observations regarding usability issues, documenting any errors observed (e.g., the respondent’s think-aloud responses were different than the respondent’s selection, the respondent accidentally hit the back button when he or she meant to hit the next button). Interviewers will also observe how well participants are able to understand and resolve any errors. As needed, interviewers may ask probes to better understand potential usability issues, “When you clicked ‘X,’ tell me what you expected to happen?” or “Tell me what happened there.”
After the interview, participants will be thanked and given $40 cash and a participation receipt form (Attachment G).
At any point, if respondent exhibits signs of distress, agitation, or indication of imminent danger of harm to oneself or another based on indirect and direct statements, the interviewer will follow the Distressed Respondent Protocol (Attachment J).
To assure the participation of recruited participants, each selected person will receive a reminder telephone call the day before the interview by Contractor staff with directions to the interview location and an opportunity to ask any questions about the purpose or logistics of the pretest. The $40 incentive will also help ensure participation.
The activities to be conducted under this approval are in themselves tests of procedures.
The basic NSDUH design was reviewed by statistical experts, both within and outside SAMHSA. Statistical experts reviewing portions of prior NSDUH designs include William Kalsbeek, PhD, University of North Carolina; Robert Groves, PhD, Georgetown University; and Michael Hidiroglou, PhD, Statistics Canada. Monroe Sirken, PhD, National Center for Health Statistics (NCHS) (deceased); James Massey, PhD, (deceased) also of NCHS; Douglas Wright, CBHSQ, SAMHSA (retired); Joseph Gfroerer, CBHSQ, SAMHSA (retired); and Arthur Hughes, CBHSQ, SAMHSA (retired) were consulted on the 1992 and subsequent survey designs. Peter Tice, CBHSQ, SAMHSA is the Government Project Officer, (240) 276-1254. Jonaki Bose, Chief, Population Surveys Branch, CBHSQ, SAMHSA is the primary mathematical statistician responsible for overall project management, (240) 276-1257. RTI senior statisticians contributing to the design are Paul Biemer, PhD, James Chromy, PhD (retired), Ralph Folsom, PhD (retired), and Rachel Harter, PhD.
Attachment A Recruitment Flyers
Attachment B Web Recruitment Screeners
Attachment C Telephone Recruitment Screener
Attachment D Parental Permission Forms
Attachment E Participant Informed Consent/Assent Forms
Attachment F Pretesting Introductory Script
Attachment G Pretesting Interview Receipt for Participation
Attachment H Federalwide Assurance
Attachment I Pretesting Interview Survey Questions
Attachment J Distressed Respondent Protocol
Attachment K SAMHSA Letter of Support
References
Beatty, P. (2004). The dynamics of cognitive interviewing. In S. Presser, J. Rothgeb, M. Couper, J. Lessler, E. Martin, J. Martin, and E. Singer (Eds.), Methods for Testing and Evaluating Survey Questionnaires (pp. 45-66). New York: John Wiley & Sons.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Payne, Scott |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |