OMB Submission
2009 National Survey on Drug Use and Health
Cognitive Interviews for Youth Mental Health Services Utilization Questions
2009 National Survey on Drug Use and Health:
Cognitive Interviews for Youth Mental Health Services Utilization Questions
SUPPORTING STATEMENT
1. Purpose and Use of Information
The National Survey on Drug Use and Health (NSDUH), sponsored by the Substance Abuse and Mental Health Services Administration (SAMHSA), is a national survey of the U.S. civilian non-institutionalized population aged 12 and older. The conduct of the NSDUH is paramount in meeting a critical objective of SAMHSA’s mission, i.e., to maintain current data on the prevalence of substance use in the United States.
This package is submitted under the NSDUH Methodological Field Tests generic OMB clearance (OMB No. 0930-0290).
On February 16, 2008, SAMHSA requested that RTI International assist in the development of alternatives for presenting NSDUH adolescent mental health treatment data and in the potential modification of the NSDUH questionnaire to better capture the receipt of mental health services by adolescents. The NSDUH youth mental health services module currently includes 10 service categories from the Services Assessment for Children and Adolescents (SACA). Currently, NSDUH defines adolescent mental health treatment/counseling for emotional or behavioral problems as receipt of services from any of the 10 service categories. Questions are asked of all youth aged 12 to 17. SAMHSA is currently updating this approach by organizing the 10 service categories under the sectors of care often reported in child mental health services literature (i.e., specialty mental health; education; general medical; juvenile justice) to more specifically depict the types of providers and service delivery sites for youth mental health services.
To this end, earlier this year, SAMHSA requested that 3 specific tasks be completed:
Task 1. Develop and recommend new ways to categorize youth mental health service use data as they are currently collected in the NSDUH;
Task 2. Produce a short report based on 2005-2006 NSDUH data; and
Task 3. Investigate options for improving the 2009 NSDUH Youth Mental Health Services module.
All three tasks were completed in March and April of 2008, and a memo outlining recommendations for changes to the NSDUH Youth Mental Health module was submitted to SAMHSA in April of 2008 (See Attachment I). Part of this memo outlines changes that are recommended to be made for the 2009 questionnaire, including new questions on where youth receive mental health treatment.
Prior to implementation in the 2009 NSDUH instrument, qualitative cognitive interview data from 32 participants will be examined and reported on to evaluate how well potential NSDUH adolescent respondents understand the concept and language of the proposed new and revised questions on mental health services and whether they find them difficult or sensitive to answer.
Data will be collected in a face-to-face interview setting. Information will be collected electronically via audio computer-assisted self-interview (ACASI) for the majority of the interview. The sessions will be audio recorded so the interviewer can reference the recordings when refining his/her notes. The first part of the session will involve the respondents answering the current Youth Mental Health Services Utilization module questions on their own via ACASI. The latter part of the session will involve showing participants questions that are being added to the 2009 NSDUH Instrument. Respondents will be asked to read and answer these questions aloud and discuss them with the interviewer.
The OAS, SAMHSA, is in contact with all major Federal health survey managers and is aware of no other efforts to assess how potential adolescent respondents may react to changes made to the NSDUH instrument. To date, no duplication of effort has been identified.
4. Consequences if Information Collected Less Frequently
The timetable for finalizing the study participant materials for the 2009 NSDUH dictates the project be completed by August 31, 2008. This project will not be repeated.
A number of consultants conducted a review of these procedures. Heather Ringeisen, Senior Research Psychologist, RTI International, (919) 541-6931 coordinated the review. The review also included Kimberly Eaton Hoagwood, Ph.D.,Professor of Clinical Psychology in Psychiatry, Columbia University and New York State Office of Mental Health, (212) 543-5311 and Sarah M. Horwitz, Ph.D., Professor of Pediatrics, Stanford University,
(650) 724-5924.
There are no unresolved issues resulting from these consultations.
Participants will be paid $40 in the form of a VISA gift card made out to “RTI Respondent” for a forty-five minute session. This amount should be enough to compensate for the respondent’s time and any travel expenses incurred (Willis, 2005).
The payment for the interview is mentioned in the following materials: Recruitment Flyer [Attachment A], Recruitment Script [Attachment B], Parental Informed Consent Form [Attachment C], Participant Informed Assent Form [Attachment D], Protocol for Cognitive Interviews [Attachment E], and Receipt for Participation [Attachment F].
Concern for the confidentiality and protection of respondents’ rights has always played a central part in the implementation of the NSDUH and will continue to be given emphasis.
The Cognitive Interviews for Youth Mental Health Services Utilization Questions will incorporate several procedures to ensure that adolescents’ rights will be protected and to reduce the risk of distress due to disclosure about mental health or behavioral treatment/counseling. The recruitment flyer [Attachment A] will advertise to the participants that “Confidentiality is guaranteed.” Also, the recruitment script [Attachment B], parental informed consent form [Attachment C], and the participant informed assent form [Attachment D] all indicate that the interview will be conducted in private to ensure nobody else overhears their answers; all of their answers will be kept private and confidential; information given by the participants will not be shared with any persons outside the project staff; their name will never be connected with the answers they provide; and that federal law (CIPSEA) requires that their answers be kept confidential and used only for statistical purposes. In these same study materials, participants are informed that their responses are voluntary and are assured that there will be no penalties if they decide not to respond, either to the information collection as a whole or to any particular question. The only people who will have access to any of the responses given by the participants are researchers at RTI and SAMHSA who have signed NSDUH confidentiality pledges [Attachment G]. These pledges explicitly outline the CIPSEA federal law requirements mentioned in the consent/assent. The only exception to this promise of confidentiality is if the adolescent respondent discloses during the interview that they intend to harm themselves or someone else. For those instances, we have included a distressed respondent protocol for this study [Attachment H] that may involve releasing respondent contact information to either the national Lifeline hotline or a local 911 dispatcher.
Although parents may accompany their child to the interview, they will not be allowed in the interview setting itself, so they will not overhear the adolescent participant’s answers.
During the recruitment process, potential participants are told that we would like to audio record the interactions between them and the interviewer during the cognitive interview so we can make sure we have all of the information from them on how our new questions work. They are told that the recordings will be heard only by members of the research team and that having their interview recorded is voluntary so they can decline (see the recruitment script, [Attachment B]). Additional information on the recordings is provided in the Parental Informed Consent Form [Attachment C] and the Participant Informed Assent Form [Attachment D]. These documents inform the participants that the recording will remain on the project laptop at all times, which will be protected by a password, to protect their privacy. This information will not be transferred off of the laptop, and the password will expire within 4 weeks after the last cognitive interview is completed. At that point, all recordings will be destroyed.
All internal communication regarding an adolescent participant will only include the first name of the participant and time of interview. An example of internal communication would be when the recruiter notifies the interviewer of a scheduled appointment with a participant. Any other materials (including the recruitment screeners, “will call” list, and informed consent forms) connecting the first name of the participant with his/her last name, telephone number, etc. will be locked in a cabinet (if in hardcopy form) or password protected (if in electronic form). This password will be set to expire within 4 weeks after the last cognitive interview is completed and both the electronic files and hardcopies will be destroyed at that time.
Although some personal information will be collected during the recruitment process and the cognitive interviews, data will not be retrieved by personal identifiers, and thus the Privacy Act does not apply to these activities. More specifically, during the recruitment process, we will collect the age, phone number, city of residence, and school attended of potential participants who are aged 12 to 17. The telephone numbers collected will be used to either remind the participants about their upcoming appointments or to recruit additional participants who were placed on a “will call” list in the event that spaces opened up for them to be interviewed. For potential participants aged 12 to 17, the telephone number will also be used to verify with Directory Assistance that their parents are who they say they are. During the cognitive interviews, we will collect the age and gender of all participants only to inform the CAI program which questions to display and to tailor wording. No links to individuals will be preserved in the cognitive interview report.
The questions tested for the 2009 NSDUH deal with youth mental health treatment options. During the cognitive interview, the participants are instructed on how to employ ‘think-aloud’ techniques when answering the questions that have been added to the instrument. The interviewer will use a standard set of interviewing protocols to best understand how the participants understand each question and how features of the question impact responses.
As noted in Section 7 potential participants and the actual participants will be assured at all stages of the recruiting and interviewing process that the information they provide is voluntary and will be handled in a confidential manner. These efforts will be made to help participants feel more comfortable with the interview situation and more at ease with the interviewer. Additionally, since we are dealing with mental health issues, we have included a distressed respondent protocol that will be enacted if the respondent becomes very agitated or very upset during the interview, or inadvertently discloses that he/she has had thoughts or plans of harming himself/herself or someone else in the past two weeks. All distressed respondent protocol procedures are detailed in Attachment H. For respondents who ask for additional information on mental health or other treatment options, we will provide them with the national Lifeline hotline that we provide respondents in the main NSDUH survey. The Lifeline is a 24-hour hotline that a person can call to discuss mental health issues with a counselor and may provide referrals for affordable clinical services in their area. We will also provide instructions for what to do if he/she becomes distressed in the future (i.e. call 911 or go to a local emergency room).
Raw data from the screening questionnaires, ACASI questionnaire, paper cognitive interviewing protocols, and audio recordings that include sensitive information will be stored in locked cabinets (if in hardcopy form) or password protected (if in electronic form) during the recruiting and interview process. None of this information will be retained once data collection is over; nor will the information be extracted, aggregated, or become part of a system of records containing permanent identifiers that can be used for retrieval.
Thirty-two participants will be interviewed in this study. It is expected that approximately 50 screenings with potential participants will be completed to obtain the 32 study participants.
Administration of the screening questionnaire during the recruitment process will take an average of 5 minutes per participant. It is estimated that the average amount of time required to conduct each cognitive interview will be approximately 45 minutes.
Both the recruitment and cognitive interviewing phases of this study will span for approximately three weeks over the months of June and July 2008. The recruitment phase will begin as soon as OMB clearance is received, and interviews will begin directly after that.
The respondent burden for this study is shown in the following table. The hourly wage of $14.61 was calculated based on weighted data from the 2007 NSDUH respondents’ personal annual income.
Estimated Burden for 2009 Cognitive Interviews for YMHSU Questions
Activity |
Number of Respondents |
Responses per Respondent |
Average Burden per Response |
Total Burden (Hrs.) |
Hourly Wage Rate |
Total Hour Cost |
Screening |
50 |
1 |
0.083 hr. |
4.61 |
$14.61 |
$ 67.35 |
Cognitive Interviews |
32 |
1 |
.75 hr. |
24 |
$14.61 |
$350.64 |
TOTAL |
82 |
– |
– |
28.61 |
– |
$417.99 |
Total costs associated with the 2009 Cognitive Interviews for Youth Mental Health Services Utilization Questions are estimated to be $50,600 over a 4-month performance period. Of the total costs, $47,000 is for study design, preparation of materials for laboratory testing, recruiting, conducting the cognitive interviews, analysis and report/publication writing, and approximately $3,600 represents SAMHSA costs to manage/administrate the survey.
The 2009 Cognitive Interviews for Youth Mental Health Services Utilization Questions results will be used to gauge the impact of wording changes and question additions required for the 2009 NSDUH survey. The sample size and design do not allow for statistical inference to be conducted, and therefore, the analyses will be observational and anecdotal. Debriefings with the cognitive interviewers will be conducted to learn from their experiences about respondents’ reactions and responses to the additional questions in the 2009 NSDUH instrument.
When the study has been completed, the SAMHSA and RTI liaisons will discuss opportunities for presenting results of the study to the external research community at a professional conference and/or via external publication.
The time schedule for the 2009 Cognitive Interviews for Youth Mental Health Services Utilization Questions is included below.
2009 Cognitive Interviews for Youth Mental Health
Services Utilization Questions Schedule
Subtask |
Due Date |
|
RTI sends cognitive interview materials (including recruitment ads and scripts and the interviewing protocol) to SAMHSA for review |
May 19, 2008 |
|
SAMHSA provides comments to RTI on cognitive interview materials |
May 20, 2008 |
|
RTI submits IRB materials for approval |
May 21, 2008 |
|
RTI delivers final package (3 sets) for request of OMB blanket clearance to SAMHSA |
June 9, 2008 |
|
OMB package sent from OAS to OMB |
June 11, 2008 |
|
RTI IRB meeting |
June 2, 2008 |
|
RTI IRB approval obtained |
June 9, 2008 |
|
OMB notifies SAMHSA concerning outcome of review |
July 2, 2008 |
|
RTI begins recruiting for cognitive interviews |
July 3, 2008 |
|
RTI conducts cognitive interviews |
July 7-August 25, 2008 |
|
RTI sends final cognitive interview report to SAMHSA |
August 29, 2008 |
|
The sample of 32 participants to be recruited is non-probability based. It will consist of adolescent volunteers who are screened and determined to meet recruitment criteria. Adolescent volunteers will primarily be recruited from advertisements posted at local schools, summer camps, and boys and girls clubs in the three interview areas, and in the classified sections of internet sites. If these methods fail to produce adequate numbers, we will place newspaper ads.
The above mentioned objectives require the study to include people who match selected characteristics of the target population SAMHSA is trying to reach for the NSDUH. Details regarding the composition of potential participants for the cognitive interviews are as follows:
Half of the adolescent respondents in each round will be youth who have attended an alternative school in the past 12 months. This will ensure that we have youth in our sample who may be familiar with the terms in the new questions regarding treatment received at a juvenile justice facility, and can help us explore more appropriate terms that we may need to use.
Interviews will be conducted in each of the RTI offices in Research Triangle Park (RTP), NC, Washington, DC, and Chicago, IL, but we will not stipulate a certain number at each site.
The majority of the NSDUH interviews are conducted in English. Thus we will focus our efforts on the English speaking population for all of our cognitive interviews.
Exhibit 1 below details the estimated number of participants to be recruited according to the characteristics detailed above for the cognitive interviews. If after 50 screenings (see Section 9), some of the cells specified in the table are unfilled, potential participants who give permission to be put on a “will call” list (because the requisite number of participants for their criteria had already been recruited) will be re-contacted to fill the vacant interview slots to maximize the attempt to obtain 32 study participants.
Exhibit 1 – Recruiting Matrix
School Type |
Age |
Round |
Participants |
Alternative |
12-14 |
1 |
4 |
Alternative |
15-17 |
1 |
|
Traditional |
15-17 |
1 |
4 |
Traditional |
12-14 |
1 |
4 |
|
|
|
|
Alternative |
12-14 |
2 |
4 |
Alternative |
15-17 |
2 |
4 |
Traditional |
15-17 |
2 |
4 |
Traditional |
12-14 |
2 |
4 |
Total |
|
|
32 |
Potential participants who call into the study line will be screened for eligibility [Attachment B]. Adolescents who call in will be screened for their age and type of school(s) they attended within the past 12 months, and will require permission from their parent or legal guardian to participate in the study. Eligible respondents will be scheduled for a forty five minute cognitive interview.
Potential participants who call the RTI study line after the requisite number of participants has been recruited will be placed on a “will call” list with their permission, but only for the duration of the interviewing phase. During this phase, individuals on the “will call” list may be called to ask if they are available, should one or more originally scheduled participants not arrive for a scheduled interview. As specified in Section 7, recruitment and contact information will be kept in locked cabinets and via password protected electronic files and not shared except with those who are assigned to complete the interviews.
Parents/guardians will be asked to accompany their children to the interview, and to sign a Parental Consent Form [Attachment C] once they arrive. If the parent or guardian declines to accompany their child to the interview, a Parental Consent Form will be mailed to them with a pre-addressed return envelope and they will be asked to sign the form and return it to RTI. Adolescent respondents will not be interviewed until we have a signed Parental Consent Form from a parent or guardian. When each participant arrives for their interview, they will be greeted and asked to listen to instructions and information from the interviewer, including the acknowledgement of a Participant Assent Form [Attachment D]. All cognitive interviews will be audio recorded upon consent of each participant. Participants will have the right to decline to be audio recorded without being excluded from participation.
To begin the interview, the interviewer will set up a 12-month calendar with the respondent (the same one used in the main NSDUH survey), and will show each respondent how to navigate through the NSDUH CAI program. The respondent will then complete a tutorial, along with the beginning of the Youth Mental Health Services Utilization module from the main NSDUH survey [See Attachment E]. These modules will be completed via ACASI. At the end of the YMHSU module, the respondent will be instructed to read the new questions aloud and discuss them with the interviewer. Participants will deliver concurrent verbal reports of their thought process, or will provide thoughts on how they are answering the question before the question is answered (Forsyth and Lessler 1991). Interviewers will be provided with a series of probes and questions that will further explore the quality of responses and whether these responses are meeting the researchers’ goals. Interviewers are instructed to use the probes as a guideline, but are not required to use all of them, be limited by them, or to read them exactly as written (Beatty 2004). Interviewers may probe based upon the content of the interview and participant responses. All pre-scripted probes are found in the cognitive interviewing protocol in Attachment E. However, the interviewer requires the latitude to deviate from and/or make changes to these protocols “on the fly” during the interviewing process in order to follow up on new information that is gained from respondents during the course of the interviews. After the interview, participants will be thanked, paid for their time with a $40 Visa card, and will be asked to sign a participation receipt form [Attachment F].
To assure the participation of the recruited cognitive interviewing participants, each selected person will receive a reminder telephone call the day before the interview with directions to the facility and an opportunity to ask any questions about the purpose or logistics of the study. The $40 VISA gift card incentive will also help ensure participation.
The activities to be conducted under this approval are in themselves tests of procedures.
Arthur Hughes (240) 276-1261 Project Officer/Mathematical Statistician Division of Population Surveys OAS, SAMHSA
|
Dicy Painter (240) 276-1264 Alternate Project Officer Division of Population Surveys OAS, SAMHSA
|
Joseph Gfroerer (240) 276-1262 Division Director/Supervisory Mathematical Statistician Division of Population Surveys OAS, SAMHSA
Peggy Barker (240) 276-1258 Statistician Division of Population Surveys OAS, SAMHSA
|
Joel Kennet (240) 276-1265 Statistician Division of Population Surveys OAS, SAMHSA
Lisa Colpe (240) 276-1245 Statistician Division of Population Surveys OAS, SAMHSA
|
ATTACHMENTS
Attachment A - Recruitment Flyer
Attachment B - Recruitment Script
Attachment C - Parental Informed Consent Form
Attachment D - Participant Informed Assent Form
Attachment E ‑ YMHSU Module and Protocol for YMHSU Cognitive Testing
Attachment F ‑ Receipt for Participation
Attachment G - NSDUH Confidentiality Pledge
Attachment H - Distressed Respondent Protocol
Attachment I - April 10, 2008 YMHSU Recommendations Memo to SAMHSA
REFERENCES
Beatty, P. (2004). The dynamics of cognitive interviewing. In S. Presser, J. Rothgeb, M. Couper, J. Lessler, E. Martin, J. Martin, and E. Singer (Eds.), Methods for Testing and Evaluating Survey Questionnaires (pp. 45-66). New York: John Wiley & Sons.
Forsyth, B.H. and Lessler, J.T. (1991) Cognitive Laboratory Methods: A Taxonomy. In Measurement Error in Surveys. P.P Biemer, R.M. Groves, L.E. Lyberg, N.A. Mathiowitz and S. Sudman (eds). New York, Wiley, 393-418.
Willis, G. B. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design (p. 144). Sage Publications, Inc.: Thousand Oaks, CA.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |