Attachment AA - REVISED-CIPSEA Cognitive Interviewing Report

On November 9, 2006, the Office of Management and Budget (OMB) approved the Substance Abuse and Mental Health Services Administration's (SAMHSA) Office of Applied Studies (OAS) as a statistical unit. As a result, OAS is now required to follow the Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA) implementation guidance in their sponsored surveys, including the National Survey on Drug Use and Health (NSDUH). With OMB guidance, SAMHSA revised various contact materials used in the NSDUH to incorporate CIPSEA language. These materials included the Lead Letter to Selected Dwelling Units, Study Description, and Question and Answer Brochure. Prior to fully administering the revised materials in the 2008 NSDUH, 24 cognitive interviews were conducted to compare the current contact materials administered in the 2007 NSDUH with the revised contact materials to be administered in the 2008 NSDUH. The main purpose of these interviews was to evaluate whether potential NSDUH respondents noticed a difference between the two versions of the materials. If the potential NSDUH respondents did notice a difference between the two versions of the materials, then the cognitive interviews evaluated whether the revisions would (1) change their decision to respond to the survey and (2) change their responses to the questions within the survey. Highlights on the findings and recommendations resulting from this aspect of the cognitive interviews are presented in the bullets below, and detailed results can be found in Section 3.1 of this report.

Overall, the participants did not notice differences between the two versions of the NSDUH contact materials. There were only a couple of participants who noticed the actual changes in language due to CIPSEA. These comments were related to the language about confidentiality and the CIPSEA act (e.g., penalty for violation).
The findings from these cognitive interviews suggest that the revisions will not adversely affect sample members' decisions to participate in the survey, nor will the revisions affect how honest respondents will be in reporting their answers in the survey. If any minor effect occurs, it might be in a positive way in that a couple of participants mentioned that they liked the increased emphasis on confidentiality (and the CIPSEA act) in the revised materials.
There were no distinct patterns when looking at the differences in interview location (Location 1, versus Location 2), adults versus children, and drug use (users versus nonusers).
There did not appear to be any noteworthy order effects where participant responses depended on the order of presentation of the current and revised contact materials.
Overall, the findings mentioned above indicate that the CIPSEA language does not appear to have an impact on sample members' decisions to participate in the survey and how honest they would be in reporting answers within the survey. Thus, it is recommended that no additional research is needed at this time on the CIPSEA language included in the NSDUH contact materials.

A second purpose of the cognitive interviews was to evaluate how well potential NSDUH respondents understood the contact materials provided. During the cognitive interviews, participants were asked questions by the interviewers to gauge their understanding of various phrases used within different sections of the contact materials that were read earlier by the participants. The findings and recommendations for this aspect of the cognitive interviews can be found in Section 3.2 of this report.

Background

The Overview of the Office of Management and Budget's (OMB) proposed implementation guidance on the Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA) states, "Issues of privacy and confidentiality are of increasing concern to respondents of Federal government surveys. Agencies often seek to assuage these concerns by pledging to respondents that the agency will protect the information that respondents provide, and by using whatever statutory authority that the agency has to substantiate this pledge. However, many agencies do not have strong confidentiality provisions in their authorizing statues." It also states, "The CIPSEA is a new government-wide law that can provide strong confidentiality protections to many Federal agencies conducting statistical information collections such as surveys and censuses as well as other statistical activities including data analysis, modeling, sample design, etc." At large, CIPSEA protects information that is acquired for statistical purposes only, under a pledge of confidentiality. The protection of the information collected under this law is supported by a penalty of a Class E Felony for a knowing and willful disclosure of confidential information. This includes imprisonment for up to 5 years and fines up to $250,000 (OMB, 2006).

In October 2006, the OMB issued the above-mentioned implementation guidance on CIPSEA (OMB, 2006). The purpose of this guidance was to inform agencies about the requirements for using CIPSEA and clarify the circumstances under which CIPSEA may be used. The guidance included suggested CIPSEA language to incorporate into the lead letter, study description, informed consent materials, survey instrument, or other materials that are read to or by the respondent.

One of the key distinctions made in the CIPSEA guidance is that the only agencies that are authorized to designate agents (e.g., RTI International^¹) to perform exclusively statistical activities (including data collection) subject to CIPSEA limitations and penalties are agencies that have been recognized by OMB as "statistical units." Thus, for purposes of CIPSEA, on November 9, 2006, the OMB approved the Substance Abuse and Mental Health Services Administration's (SAMHSA's) Office of Applied Studies (OAS) as a statistical unit. As a result, OAS is now required to follow the CIPSEA implementation guidance in their sponsored surveys, including the National Survey on Drug Use and Health (NSDUH). Since the 2007 NSDUH was prepared to enter the field in January 2007, and OMB was aware that any revisions made to the NSDUH contact materials to incorporate CIPSEA language would need to be tested (e.g., via cognitive interviews), OAS was provided full clearance to conduct the 2007 NSDUH with the contact materials used in prior years. However, OMB did request that changes to NSDUH contact materials for purposes of CIPSEA be fully implemented in the 2008 NSDUH.

With OMB and Office of General Counsel (OGC) guidance, SAMHSA revised various contact materials used in the NSDUH to incorporate CIPSEA language according to the above-mentioned guidance. The revised contact materials included the Lead Letter to Selected Dwelling Units, Study Description, and Question and Answer Brochure. One key initial task prior to fully administering these materials in the 2008 NSDUH was to conduct cognitive tests comparing the current contact materials administered in the 2007 NSDUH with the revised contact materials to be administered in the 2008 NSDUH. The purpose of the cognitive testing was to first evaluate whether potential NSDUH respondents noticed a difference between the current and revised materials and, if so, whether the revisions would (1) change their decision to respond to the survey and (2) change their responses to the questions within the survey. The cognitive testing also evaluated how well potential NSDUH respondents understood the materials provided.^²

This report documents the qualitative research conducted by RTI to evaluate the current and revised NSDUH contact materials. Chapter 2 provides a detailed description of the methodology used to conduct the research, and Chapter 3 provides the findings obtained from the cognitive interviews and recommendations based on these findings.

Methodology

Cognitive interviews were conducted with a total of 24 participants: 12 in each of two locations. This section of the report summarizes the main objectives of the research; describes the protocol developed for the project; and details the methodology for identifying, recruiting, and interviewing participants.

Objectives of the Cognitive Interviews

The main purpose of the cognitive interviews was to evaluate whether potential NSDUH respondents noticed a difference between the contact materials currently being administered in the 2007 survey and those revised to incorporate CIPSEA language that are to be administered in the 2008 NSDUH. If the potential NSDUH respondents did notice a difference between the two versions of the materials, then the cognitive interviews evaluated whether the revisions would (1) change their decision to respond to the survey and (2) change their responses to the questions within the survey. A second purpose of the cognitive interviews was to evaluate how well potential NSDUH respondents understood the materials provided. Cognitive interview participants were asked to respond to specific probes by an in-person cognitive interviewer.

Cognitive Interview Protocol

Three interview protocols were developed to be used in the 24 cognitive interviews: one for participants aged 12 to 17, one for adult participants recruited as parents or legal guardians of adolescents aged 12 to 17, and one for adult participants recruited as adults aged 18 or older. All three protocols were developed to administer the cognitive interview as follows:

Participants were asked to read through one version of the NSDUH contact materials.
Depending on their age, participants were either read the "Introduction and Informed Consent for Interview Respondents Aged 18+" or the "Introduction and Informed Consent for Interview Respondents Aged 12-17."
Participants were shown a short 2-minute video of a typical interaction between an interviewer and a potential respondent.
Based on the materials they just read and heard and the video they saw, the participants were asked to respond to pre-scripted questions by the cognitive interviewer.
Participants were asked to read through a second version of the NSDUH contact materials.
The appropriate NSDUH "Introduction and Informed Consent for Interview Respondents Aged XXX" was re-read to the participants.
Based only on the second set of materials they read and heard, participants were asked pre-scripted questions by the cognitive interviewer.
The cognitive interviewer asked the participants pre-scripted questions comparing the two different versions of the NSDUH contact materials.
The cognitive interviewer asked the participants pre-scripted questions about their general understanding of various terms and phrases in the NSDUH contact materials read by the participants earlier in the interview.

The version of the contact materials received first and second by the participants during the cognitive interviews was randomly assigned to reduce the potential for order effects. Twelve participants were asked to review the current version being administered in the 2007 survey first and then review the revised version, which incorporates the CIPSEA language and is to be administered in the 2008 NSDUH. The other 12 participants were asked to review the contact materials in the opposite order.

The NSDUH contact materials administered by the protocols developed for participants recruited as parents or legal guardians of adolescents aged 12 to 17 and for participants recruited as adults aged 18 or older included the Lead Letter to Selected Dwelling Units, Study Description, and Question and Answer Brochure. Since it is assumed that adolescents themselves are unlikely to see the lead letter in the main NSDUH, the Lead Letter to Selected Dwelling Units was not included with the NSDUH contact materials administered by the protocol developed for participants aged 12 to 17.

The protocol developed for participants recruited as parents or legal guardians of adolescents aged 12 to 17 included a few more pre-scripted questions than the protocols developed for participants aged 12 to 17 and participants recruited as adults aged 18 or older. These additional questions were used to evaluate whether the parent would allow their child to participate in the survey after reading through each version of the NSDUH contact materials.

Four cognitive interview participants were administered the protocol for participants aged 12 to 17, 4 were administered the protocol for participants recruited as parents or legal guardians of adolescents aged 12 to 17, and 16 were administered the protocol for participants recruited as adults aged 18 or older. See Appendix G for the completed cognitive interview protocols for this study.

Participant Recruitment

To ensure that the cognitive interview participants represented more than one geographical location in the United States, 12 participants were recruited from two different metropolitan areas. Within each location, the recruitment criteria were established to ensure that the cognitive interview participants represented various characteristics of the NSDUH main study sample (e.g., varying age groups and education levels, drug users and non-drug users). Thus, within each location, it was recommended that participants be recruited according to the matrix presented in Table 1 below.

Table 1. Recruiting Matrix (for each location)

Characteristics	Requested Number of Participants
Aged 12 to 17	2
Parents or legal guardians of adolescents aged 12 to 17	2
Aged 18 to 49, illegal drug user in past 12 months, less than high school education	1
Aged 18 to 49, illegal drug user in past 12 months, at least a high school education	1
Aged 18 to 49, never used illegal drugs in lifetime, less than high school education	1
Aged 18 to 49, never used illegal drugs in lifetime, at least a high school education	1
Aged 50 or older, illegal drug user in past 12 months, less than high school education	1
Aged 50 or older, illegal drug user in past 12 months, at least a high school education	1
Aged 50 or older, never used illegal drugs in lifetime, less than high school education	1
Aged 50 or older, never used illegal drugs in lifetime, at least a high school education	1

To summarize, participants were either recruited as adolescents aged 12 to 17, as parents or legal guardians of adolescents aged 12 to 17, or as adults aged 18 or older. During the recruitment process, only potential participants being screened as adults aged 18 or older were asked (1) if they were a high school graduate (including GED) and (2) whether they used illegal drugs in their lifetime, the past 12 months, and/or in the past 30 days. For purposes of this study, use of illegal drugs included the use of marijuana, cocaine, crack, heroin, hallucinogens (i.e., PCP, LSD, and ecstasy), and/or the illegal use of methamphetamine. Use of alcohol, tobacco, and/or prescription drugs was not considered illegal drug use.

Recruitment was conducted by placing ads on craigslist (craigslist.org), a free online community-based bulletin board. Different ads were posted for adolescents aged 12 to 17, parents or legal guardians of adolescents aged 12 to 17, and adults aged 18 or older. All ads invited interested persons to call a toll-free number to determine whether they were eligible to participate in the study. The text for the ads is included in Appendix H.

When interested participants called the toll-free number, they were screened for eligibility (see Appendix I for the recruitment screeners). Persons aged 12 to 17 were considered eligible to participate only after permission was obtained from a parent during the screening process (see Appendix J for the script for parental informed consent). If, for one of the two locations, interested participants calling in met the eligibility requirements for one of the cells in the matrix detailed in Table 1 above that had already been filled (e.g., a parent of an adolescent was calling in from Location 1, and we had already recruited two parents of adolescents in Location 1), they were placed on a "will call back" list in the event that we needed to fill vacant cells at the end of the recruiting process to conduct a total of 24 interviews.

After screening 83 interested participants between the two locations, two thirds of the cognitive interview participants were recruited according to the various eligibility requirements detailed in the matrix in Table 1 above. No participants of the following types were identified:

Location 1

Person aged 18 to 49, illegal drug user in past 12 months, less than high school education
Person aged 50 or older, illegal drug user in past 12 months, less than high school education
Person aged 50 or older, illegal drug user in past 12 months, at least a high school education
Person aged 50 or older, never used illegal drugs in lifetime, less than high school education

Location 2

Person aged 18 to 49, illegal drug user in past 12 months, less than high school education
Person aged 18 to 49, never used illegal drugs in lifetime, less than high school education
Person aged 50 or older, illegal drug user in past 12 months, less than high school education

Based on SAMHSA feedback, the above-mentioned difficult-to-recruit participants were respectively replaced with participants on the "will call back" list who had met the following eligibility criteria:

Location 1

4 persons aged 18 to 49, illegal drug user in past 12 months, at least a high school education

Location 2

Person aged 18 to 49, illegal drug user in past 12 months, at least a high school education
Person aged 18 to 49, never used illegal drugs in lifetime, at least a high school education
Person aged 50 or older, illegal drug user in past 12 months, at least a high school education

Once a caller was screened in for participating, the recruiter provided additional information about the study, including

the purpose of the study,
the amount of time required to participate (approximately 1 hour),
the location of the cognitive interviews,
the amount of incentive ($50 gift card for first 9 participants, $40 gift card for remaining 15 participants^³),
the voluntary nature of participation, and
the fact that sessions would be audio recorded.

Callers who were interested in participating and who were determined to be eligible were scheduled for an appointment and provided directions to the cognitive interview location. A total of 24 participants were selected for the cognitive interviews. Tables 2 and 3 include various characteristics (collected during the recruitment process and the cognitive interview) for each of the participants who took part in this research, by geographic location. Table 4 includes an overall summary of the demographic and drug use information for the complete set of participants who took part in this research, combining geographic locations.

Table 2. Characteristics of Cognitive Interview Participants—Location 1

Participant	Recruited as	Age	Gender	Education	Race/Hispanicity*
L1-1	Adult 18+	40	Female	Some College	NH White
L1-2	Adult 18+	55	Female	Some College	NH White
L1-3	Adult 18+	35	Female	4-Year College Degree	NH White
L1-4	Adolescent	14	Male	Less than High School	NH Asian
L1-5	Parent	50-55	Female	Some College	NH Asian
L1-6	Adult 18+	20	Female	Less than High School	NH White/ NH American Indian
L1-7	Adolescent	15	Female	Less than High School	NH White/ NH American Indian
L1-8	Parent	45	Female	Some College	NH White/ NH American Indian
L1-9	Adult 18+	43	Male	4-Year College Degree	NH Black
L1-10	Adult 18+	28	Female	Some College	NH Black
L1-11	Adult 18+	41	Male	4-Year College Degree	NH White
L1-12	Adult 18+	42	Female	Some College	NH White

*NH = Non-Hispanic.

Table 3. Characteristics of Cognitive Interview Participants—Location 2

Participant	Recruited as	Age	Gender	Education	Race/Hispanicity*
L2-1	Adult 18+	63	Female	Less than High School	NH Black/ NH American Indian
L2-2	Adult 18+	61	Female	4-Year College Degree	NH Black
L2-3	Adult 18+	30	Male	4-Year College Degree	NH White
L2-4	Adult 18+	19	Male	Some College	NH Black
L2-5	Adult 18+	58	Male	Some College	NH White
L2-6	Parent	44	Female	High School Graduate	NH Black
L2-7	Adolescent	12	Male	Less than High School	NH Black
L2-8	Adult 18+	25	Female	4-Year College Degree	NH Other ("bunch of stuff")
L2-9	Adult 18+	51	Male	Some College	NH White
L2-10	Adult 18+	26	Female	Some College	Hispanic
L2-11	Parent	47	Female	4-Year College Degree	NH White
L2-12	Adolescent	12	Female	Less than High School	NH White

*NH = Non-Hispanic.

Table 4. Summary of Demographic and Drug Use Information

Demographic Characteristic	Criteria	Total
Age	12-17	4
	18-25	3
	26-34	3
	35-49	8
	50 or Older	6
Gender	Male	8
	Female	16
Education	Less than High School	6
	High School Graduate	1
	Some College	10
	4-Year College Degree	7
Race/Hispanicity	Non-Hispanic White	10
	Non-Hispanic Black	6
	Non-Hispanic Asian	2
	Non-Hispanic Two or More Races	5
	Hispanic	1
Drug Use Status*	Never Used	7
	Used in PY	4
	Used in PM	5
	N/A	8

*During the screening process, only respondents recruited as adults aged 18 or older were asked whether they ever used drugs, used drugs in the past year (PY), and/or used drugs in the past 30 days (PM). Parents and adolescents were not asked about drug use during the screening process. Thus, this question was not applicable (N/A) to them.

Conducting the Cognitive Interviews

Every cognitive interview was conducted by only one interviewer. All cognitive interview sessions were audio recorded. The interviewer began each cognitive interview by providing in written format and reading aloud the respective informed consent form (Appendix K). Once this was read to the participant and signed by the interviewer, the interviewer began the audio recording (if participant consented to it) and read the cognitive interviewing introduction using the respective protocol.

Upon completion of the cognitive interview, the interviewer provided the promised incentive payment and signed the receipt for participation form, indicating that the participant had received the gift card in appreciation for his or her participation in the project (Appendix L). All cognitive interviews were completed between June 14, 2007, and July 10, 2007.

Findings and Recommendations

Recruiting efforts on this study were made to maximize the diversity of the participants in this research. However, the results presented in this report are qualitative in nature and do not reflect a statistically representative sample of any kind. This chapter summarizes the results of the cognitive interviews by the two main objectives of conducting the testing.

Current versus Revised Contact Materials

As mentioned in Section 2.1 (Objectives of the Cognitive Interviews), the main purpose of the cognitive interviews was to evaluate whether potential NSDUH respondents noticed a difference between the contact materials currently being administered in the 2007 survey and those revised to incorporate CIPSEA language that are to be administered in the 2008 NSDUH. If the potential NSDUH respondents did notice a difference between the two versions of the materials, then the cognitive interviews evaluated whether the revisions would (1) change their decision to respond to the survey and (2) change their responses to the questions within the survey. Thus, the cognitive interview protocols were set up so that during the first half of the interviews, the participants (1) read through both versions of the contact materials, answering pre-scripted questions after each one to determine their level of participation and honesty in answering the questions within the NSDUH, and (2) were asked pre-scripted questions comparing the two versions of the contact materials. Both the overall and question-specific findings for the first half of the cognitive interviews are detailed below.

3.1.1 Overall Findings and Recommendations

Overall, the participants did not notice differences between the two versions of the NSDUH contact materials. There were only a couple of participants who noticed the actual changes in language due to CIPSEA. These comments were related to the language about confidentiality and the CIPSEA act (e.g., penalty for violation).

The findings from these cognitive interviews suggest that the revisions will not adversely affect sample members' decisions to participate in the survey, nor will the revisions affect how honest respondents will be in reporting their answers in the survey. If any minor effect occurs, it might be in a positive way in that a couple of participants mentioned that they liked the increased emphasis on confidentiality (and the CIPSEA act) in the revised materials. In addition, there were no distinct patterns when looking at the differences in interview location, adults versus children, and drug use (users versus nonusers). Any discernable differences are identified for the specific question in the question-specific summaries below. Finally, there did not appear to be any noteworthy order effects where participant responses depended on the order of presentation of the current and revised contact materials. In other words, there were no overall differences in the responses to questions Q1 through Q3 versus questions Q4 through Q6, by version of the contact materials received. Thus, the fact that the participants did not perceive the differences in the two versions of the contact materials is reinforced. The only order effects that were noticed were related to participants thinking that the second set of materials they received were easier to read (e.g., more concise) or contained different information. From the debriefing questions, most of these perceived differences were not real. However, this implies that if the order in which participants received the current and revised contact materials had not been randomized, we might have concluded that participants preferred the second set of materials over the first.

Overall, the findings mentioned above indicate that the CIPSEA language does not appear to have an impact on sample members' decisions to participate in the survey and how honest they would be in reporting answers within the survey. Thus, it is recommended that no additional research is needed at this time on the CIPSEA language included in the NSDUH contact materials.

3.1.2 Question-Specific Findings

Q1. (If you were the contacted participant in the video/If your parent or legal guardian were the person contacted in the video and he or she gave the interviewer permission to interview you), how likely would you be to agree to participate in the interview? Indicate a number from 1 to 5, where 1 means you would be not at all likely to participate, and 5 means you would be very likely to participate.

Rating	Adults 18+	Children 12-17	Total
1 (Not at all likely)	1	0	1
2	1	0	1
3	4	0	4
4	7	2	9
5 (Very likely)	7	2	9
Total	20	4	24

Was there anything in the materials you just read and heard that would influence how likely you would be to agree to participate in the interview? If yes, what?

Only a few of the participants said the materials would influence their decision to participate in the NSDUH interview. These comments were positive and not related to any of the differences between the two versions of the materials. A few of the comments about the materials included (1) the materials explained the process and purpose of the survey; (2) responses would count for other persons, which made the study seem important; (3) the materials mentioned the incentive payment; and (4) the materials mentioned the notion of confidentiality. While the materials in general were not a major factor in whether these participants would participate in the NSDUH, there were other factors cited including the willingness (and sometimes perceived importance) to participate in studies. There were only a couple of negative comments provided. One participant did not want to be contacted by an outside researcher, and another participant thought the interviewer in the video was pushy and invasive.

Q1 (As Parent of Child). If you were the person contacted in the video, how likely would you be to agree to allow your 12-17 year old child to participate in the interview? Indicate a number from 1 to 5, where 1 means you would be not at all likely to allow him/her to participate, and 5 means you would be very likely to allow him/her to participate.

Rating	Parents
1 (Not at all likely)	0
2	1
3	0
4	0
5 (Very likely)	3
Total	4

Was there anything in the materials you just read and heard that would influence how likely you would be to agree to allow your 12-17 year old child to participate in the interview? If yes, what?

Two of the participants answered "Yes" to the follow-up question. One cited the confidential nature of the information and the other cited the incentive payment and the importance of doing the study. The one parent (who answered "2") said he or she does not like to force his or her kids to do anything they don't like to do.

Q2. As a reminder, the survey being presented in the materials you have read asks about drug use or non-use. If you agreed to participate in the survey, how likely would you be to answer these questions honestly? Indicate a number from 1 to 5, where 1 means you would be not at all likely to answer honestly, and 5 means you would be very likely to answer honestly.

Rating	Adults 18+	Children 12-17	Total
1 (Not at all likely)	0	0	0
2	0	0	0
3	0	0	0
4	4	0	4
5 (Very likely)	16	4	20
Total	20	4	24

Was there anything in the materials you just read and heard that would influence how likely you would be to answer questions about drug use or non-use honestly? If yes, what?

About half of the participants said the materials would not influence how honest they would answer the questions about drug use or nonuse because they would answer honestly regardless. The other participants cited confidentiality and names not associated with answers as the main points in the materials that would influence them to answer the questions honestly.

Q3. The survey being presented in the materials you have read also asks about mental health. If you agreed to participate in the survey, how likely would you be to answer these questions honestly? Indicate a number from 1 to 5, where 1 means you would be not at all likely to answer honestly, and 5 means you would be very likely to answer honestly.

Rating	Adults 18+	Children 12-17	Total
1 (Not at all likely)	0	0	0
2	0	0	0
3	1	0	1
4	4	0	4
5 (Very likely)	15	4	19
Total	20	4	24

Was there anything in the materials you just read and heard that would influence how likely you would be to answer questions about mental health honestly? If yes, what?

Almost all of the participants cited the same reasons for answering honestly as provided in Q2, and their responses about the materials did not change much. There was one participant who said the term "mental health" has a negative stigma in society and so using this term in the materials would decrease honest answers because persons would not want to reveal a mental health problem.

Q4. (If you were the contacted participant/If your parent or legal guardian were the person contacted and he or she gave the interviewer permission to interview you), how likely would you be to agree to participate in the interview? Indicate a number from 1 to 5, where 1 means you would be not at all likely to participate, and 5 means you would be very likely to participate.

Rating	Adults 18+	Children 12-17	Total
1 (Not at all likely)	1	0	1
2	0	0	0
3	5	0	5
4	7	2	9
5 (Very likely)	7	2	9
Total	20	4	24

Was there anything in the materials you just read and heard that would influence how likely you would be to agree to participate in the interview? If yes, what?

Nothing from the responses to this question would suggest that the materials would have any negative impact on whether these participants would agree to participate in the NSDUH. Similar to Q1, the comments about the materials were mostly positive and not related to any of the differences between the two versions of the materials. The participants provided similar answers and comments as in Q1.

Some of the participants thought both versions of the materials were the same and so their answers would not change from Q1. Others thought there were differences and even mentioned that they liked the second version of materials better, but they only cited differences that did not truly exist.

There was one participant (same as in Q1) who answered negatively and was wary about being selected and contacted by an outside researcher.

Q4 (As Parent of Child). If you were the person contacted, how likely would you be to agree to allow your 12-17 year old child to participate in the interview? Indicate a number from 1 to 5, where 1 means you would be not at all likely to allow him/her to participate, and 5 means you would be very likely to allow him/her to participate.

Rating	Parents
1 (Not at all likely)	1
2	0
3	0
4	0
5 (Very likely)	3
Total	4

The same answers were provided by these parents as in Q1 (As Parent of Child), so there were no changes in the answers. The one participant who answered "1" said that they would have to talk about it as a family before deciding.

Q5. As a reminder, the survey being presented in the materials you have read asks about drug use or non-use. If you agreed to participate in the survey, how likely would you be to answer these questions honestly? Indicate a number from 1 to 5, where 1 means you would be not at all likely to answer honestly, and 5 means you would be very likely to answer honestly.

Rating	Adults 18+	Children 12-17	Total
1 (Not at all likely)	0	0	0
2	0	0	0
3	1	0	1
4	4	0	4
5 (Very likely)	15	4	19
Total	20	4	24

Was there anything in the materials you just read and heard that would influence how likely you would be to answer questions about drug use or non-use honestly? If yes, what?

Almost all of the participants provided the same (or very similar) answers and comments as in Q2. Confidentiality was again cited as an important factor in answering the questions about drug use or nonuse honestly.

There was one participant who changed his or her answer from 4 (in Q2) to 3. The participant commented that the second version (current) mentioned confidentiality but the first version (revised) repeated the notion of confidentiality more. Another participant changed his or her answer from 5 (in Q2) to 4. In both Q2 and Q5, the participant mentioned that there was nothing in the materials that would influence how honest he or she would answer the questions about drug use or nonuse. In Q2, he or she added that he or she liked to help others.

Q6. The survey being presented in the materials you have read also asks about mental health. If you agreed to participate in the survey, how likely would you be to answer these questions honestly? Indicate a number from 1 to 5, where 1 means you would be not at all likely to answer honestly, and 5 means you would be very likely to answer honestly.

Rating	Adults 18+	Children 12-17	Total
1 (Not at all likely)	0	0	0
2	0	0	0
3	2	0	2
4	4	0	4
5 (Very likely)	14	4	18
Total	20	4	24

Was there anything in the materials you just read and heard that would influence how likely you would be to answer questions about mental health honestly? If yes, what?

The participants provided similar answers and comments about answering honestly for the questions about mental health as in Q3. The same participant commented about the negative stigma of the term "mental health," and the other participant (mentioned in Q5) again provided the same comment about confidentiality being emphasized more in the first version (revised).

F1. (Not administered to children 12 to 17) Did you notice any differences between the LEAD LETTER you read at the beginning of this interview and the one you just read? If yes, what differences did you notice?

None of the participants noticed the actual differences between the current and revised lead letters. Some participants indicated that they did notice differences, but the differences that were cited were not real. For example, a couple of participants thought the two versions of the lead letter provided different numbers of persons who were selected for the study. Also, a couple of participants thought the second version was easier to read, was more concise, and had a different feel (note the order of the materials was not the same for these participants).

F2. Did you notice any differences between the STUDY DESCRIPTION you read at the beginning of this interview and the one you just read? If yes, what differences did you notice?

Most of the participants said there was no difference between the two versions of the Study Description. A few participants cited differences that did not exist. One of the adult participants said the first version (revised) was more detailed about confidentiality but could not specifically remember the details. Another adult participant stated that the revised version of the Study Description mentioned the penalty for violating CIPSEA but the current version did not. This participant indicated this difference in the Study Description after mentioning the same difference in F3 below for the Question and Answer Brochure.

F3. Did you notice any differences between the QUESTION AND ANSWER BROCHURE you read at the beginning of this interview and the one you just read? If yes, what differences did you notice?

Some of the differences cited were not real differences. However, three participants (including one child participant) stated that the revised version of the brochure mentioned the penalty for violating CIPSEA but the current version did not.

F4a. (IF F1 or F2 or F3=YES) Is one version of the materials more convincing in getting someone to respond to the survey being presented? If yes, which version and why? If no, why?

The four participants who mentioned the additional wording regarding the penalty for violating CIPSEA in the revised versions of the Study Description and Question and Answer Brochure in F2 and F3 above thought the revised materials were more convincing in getting someone to participate because of the emphasis on confidentiality and the mention of the CIPSEA act and fines.

Did you include the lead letter, study description, AND the question and answer brochure when telling me which version of the contact materials is more convincing? (If not all three): How did you decide which material(s) to tell me about?

Of the four participants who thought the revised materials were more convincing in getting someone to participate, two were including all three materials (or both materials for the children) in answering the question, one was including the Study Description and the brochure, and one was including just the brochure.

F4b. (IF F1 and F2 and F3=NO) Were the materials convincing in getting someone to respond to the survey being presented? Why?

All of the participants who were asked this question answered "Yes," the materials were convincing in getting someone to participate in NSDUH. The participants' comments varied somewhat but included remarks about confidentiality, providing sufficient information about the study, and the importance of the research.

Did you include the lead letter, study description, AND the question and answer brochure when telling me whether the materials were convincing? (If not all three): How did you decide which material(s) to tell me about?

Most of the participants said they were thinking about all three materials (or both materials for the children). However, one participant thought the Study Description went into more detail about what the survey would be asking. Another participant thought the brochure was the most effective because he or she thought the lead letter was ineffective (a turnoff).

F4a (As Parent of Child). (IF F1or F2 or F3=YES) Is one version of the materials more convincing in getting a parent to allow his/her 12-17 year old child to respond to the survey being presented? If yes, which version and why? If no, why?

None of the parents noticed the actual differences between the current and revised versions of the materials.

F4b (As Parent of Child). (IF F1 and F2 and F3=NO) Were the materials convincing in getting a parent to allow his/her 12-17 year old child to respond to the survey being presented? Why?

One participant said "No," not for him or her personally, but for others the materials might be convincing. The other participants said the materials were convincing for reasons such as their confidentiality was assured and the survey was government based.

All of the parents were thinking of all three materials.

F5a. (IF F1or F2 or F3=YES) If you agreed to participate in the survey, would either version of the materials make you more or less likely to give honest answers to the questions in the survey about drug use or non-use? If yes, which version(s) and why? If no, why?

The four participants who mentioned the additional wording regarding the penalty for violating CIPSEA in the revised versions of the Study Description and Question and Answer Brochure in F2 and F3 above said they would give honest answers to the drug use or nonuse questions regardless of which version of the materials they received. They thought the materials were similar and would not affect their honesty in responding.

Did you include the lead letter, study description, AND the question and answer brochure when telling me which version would make you more or less likely to give honest answers? (If not all three): How did you decide which material(s) to tell me about?

All four participants mentioned above who said they would give honest answers to the drug use or nonuse questions regardless of which version of the materials they received were including all three materials (or both materials for the children) in answering the question.

F5b. (IF F1 and F2 and F3=NO) If you agreed to participate in the survey, would the materials make you more or less likely to give honest answers to the questions in the survey about drug use or non-use? Why?

Most of the participants mentioned confidentiality as a factor for being honest in answering the drug use or nonuse questions. Some participants said the materials would not make a difference because he or she would answer honestly regardless. Only one participant answered "Less likely" because it is a government study (and this participant was wary of the government collecting personal information).

Did you include the lead letter, study description, AND the question and answer brochure when telling me whether the materials would make you more or less likely to give honest answers? (If not all three): How did you decide which material(s) to tell me about?

Most of the participants were thinking about all three materials (or both materials for the children), though a few mentioned the brochure in particular.

F6a. (IF F1or F2 or F3=YES) If you agreed to participate in the survey, would either version of the materials make you more or less likely to give honest answers to the questions in the survey about mental health? If yes, which version(s) and why? If no, why?

The four participants who mentioned the additional wording regarding the penalty for violating CIPSEA in the revised versions of the Study Description and Question and Answer Brochure in F2 and F3 above said they would give honest answers to the mental health questions regardless of which version of the materials they received. They thought the materials were similar and would not affect this aspect.

Did you include the lead letter, study description, AND the question and answer brochure when telling me which version would make you more or less likely to give honest answers? (If not all three): How did you decide which material(s) to tell me about?

All four participants above who said they would give honest answers to the mental health questions regardless of which version of the materials they received were including all three materials (or both materials for the children) in answering the question.

F6b. (IF F1 and F2 and F3=NO) If you agreed to participate in the survey, would the materials make you more or less likely to give honest answers to the questions in the survey about mental health? Why?

Most of the participants said they would be more likely to answer honestly about the questions on mental health because of the emphasis on confidentiality and also because they would answer honestly if they were to participate. Two participants said the materials would not make a difference because they would answer honestly regardless of them.

Most of the participants were thinking about all three materials (or both materials for the children), though a couple of them mentioned the brochure in particular.

Comprehension of Contact Materials

As mentioned in Section 2.1 (Objectives of the Cognitive Interviews), a second purpose of the cognitive interviews was to evaluate how well potential NSDUH respondents understood the contact materials provided. Thus, the cognitive interview protocols were set up so that during the second half of the interviews, the interviewer read out loud different sections of the contact materials read earlier by the participants. After reading through a given section, the interviewer asked the participants pre-scripted questions about their understanding of various phrases within that section of the contact materials. The findings for each of the pre-scripted questions are detailed below. Unless specifically noted, the responses provided by the participants did not differ among age or geographical region. This section of the report concludes with some recommendations to consider based on the findings presented.

3.2.1 Findings

U1: The first paragraph of the Lead Letter to Selected Dwelling Units was read to all participants except those aged 12 to 17. This paragraph is the same in both the current and revised versions of the letter. After reading this paragraph, participants were asked, "What did the phrase "health-related issues" mean to you as used in this letter?" Most of the participants understood this phrase to mean overall health issues and included medical conditions or diseases such as cancer, diabetes, and heart problems in addition to drug or mental health issues. A couple of participants indicated that the phrase "health-related issues" did not make them think of alcohol or drug issues.

U2: The first paragraph of the Study Description was read to all participants. This paragraph is the same in both the current and revised versions of the description. After reading this paragraph, participants were asked, "What did the phrase "randomly chosen" mean to you as used in this Study Description?" About two thirds of the participants understood this phrase in the correct manner. Phrases used by these participants to explain "randomly chosen" included "I was just picked out of a hat" or "Pick a number out of a bag, a lotto." A few participants even mentioned that the selection was not based on their specific race/ethnicity, gender, income, etc. Three participants did not know what the phrase "randomly chosen" meant. A couple of other participants incorrectly described the phrase to mean they were chosen based on specific demographic information (e.g., white, male, between the ages of 18 and 25) or the type of neighborhood they lived in (e.g., lower income).

U3: The second paragraph of the current Study Description was read to all participants. Then participants were asked, "What did the phrase "research purposes" mean to you as used in this Study Description?" Most of the participants interpreted this phrase to mean data would be collected or information would be gathered to find out more about persons in general. A couple of participants went deeper and indicated that this information would then be used to help allocate funds for various programs such as substance abuse prevention. Some participants, particularly a handful from Location 2, associated the phrase "research purposes" with collecting statistics or gathering information for statistical data use. A couple of other participants understood "research purposes" to be related to statistics without actually mentioning the term. For example, one participant said, "Stuff you might see in the newspaper such as drug use is down among teens." None of the participants incorrectly described the phrase "research purposes" to mean information gathered to find out more about persons at an individual level.

U4: The second paragraph of the revised Study Description was read to all participants. Then participants were asked, "What did the phrase "statistical purposes" mean to you as used in this Study Description?" A majority of the participants associated the phrase "statistical purposes" with numbers. For example, participants mentioned "information gleaned for number crunching," "To put numbers on just to compare," "It would be gathered on a ratio or percentage," and "How many people do illegal drugs, how many people don't." Some participants understood the phrase "statistical purposes" to mean the same as "research purposes." One participant preferred the use of the term "statistical purposes" versus "research purposes" because it was clearer. Two participants did not know and could not explain the meaning of "statistical purposes." Three participants who did know the meaning of "statistical purposes" expressed concerns about the phrase being too difficult and complicated for their peers to understand.

U5 through U10: The remainder of the revised Study Description was read to all participants, including the CIPSEA reference below the signature. Then all participants were asked the following six questions.

U5: "In your own words, can you tell me what type of information is collected by the National Survey on Drug Use and Health? Do you think this type of information is important to research? Why?" Most participants specifically mentioned that the information collected by the NSDUH included the use and nonuse of drugs, alcohol, and tobacco as well as information about mental health. Some participants also indicated that the type of information collected by the NSDUH included health issues in general, attitudes toward health behaviors, and demographic and socioeconomic information. All of the participants thought this type of information was important to research for various reasons, which can be summarized into three main categories:

Substance use and mental health have a big influence on the Nation's health, which everyone should be aware of. For example, one participant mentioned that many persons today are stressed out and tend to turn toward a substance of some type (e.g., alcohol) to cope. Thus, there has to be some sort of study done to help the general population develop coping skills that do not utilize substances. Another participant mentioned that substance abuse was an incidental part of why the health insurance and pharmaceutical costs were spiraling out of control.
The information collected from the research on current drug use and mental health would be helpful to allocate funding to assist persons who suffer from substance abuse or mental health problems.
The information collected on substance use and mental health could be used to design more effective programs for treatment and prevention.

U6: "Would you be convinced to do the interview for $30? Why or why not?" All but five of the participants said "Yes" to being convinced to do the interview for $30. These participants thought either $30 was reasonable for a 1-hour interview or the topic was very important to research and wanted to help. Several of these participants said they would participate in the interview regardless of the money since it was the right thing to do. All participants aged 12 to 17 were convinced to do the interview. Only 2 of the 24 participants said they would not be convinced to do the interview for $30. They thought the money would not make it worth their time to provide personal and sensitive information even though they had nothing to hide. Three other participants said they might be convinced to do the interview, depending on their financial situation or what they were doing when the interviewer showed up at their door.

U7: "What did the phrase "any project staff or authorized data user" mean to you as used in this Study Description?" Most of the participants understood this phrase to mean employees of RTI or government staff who were involved with any aspect of this project. Some participants narrowed the meaning by limiting it to interviewers and data entry staff. A few participants broadened the meaning by including all persons who worked for RTI or the government. Although all adult participants gave an answer for this question, two participants aged 12 to 17 were confused by this phrase and did not provide an answer.

U8: "What did the phrase "your confidentiality" mean to you as used in this Study Description?" All 24 participants seemed to understand the concept of confidentiality. Some participants used the words "anonymous" or "privacy" to describe the concept. Other participants explained that their name or any other personal identifying information would not be associated with the answers they provided. One participant indicated that he or she represented a demographic figure head (e.g., white, male, between the ages of 18 and 25).

U9: "In your own words, can you tell me what the penalty is for "any project staff or authorized data user who violates CIPSEA"?" None of the participants specifically said the penalty was "a jail term of up to 5 years, a fine of up to $250,000, or both." Four participants gave the most accurate description of the penalty as "5 years in jail, a $250,000 fine, or both." Five participants gave a similar description but left out the "or both" part. One participant did not remember. All of the other participants mentioned either just the jail term, just the fine, or both. However, some of these answers did not provide specific terms or amounts. Others provided a term or amount that was incorrect.

U10: "What does it mean for someone to "violate CIPSEA"? (PROBE FOR EXAMPLES OF WHAT PARTICIPANT CONSIDERS TO BE VIOLATIONS)" Most of the participants understood this phrase to mean giving out individual information (e.g., name, address, answers to the survey) to someone for purposes not related to the particular research study. Examples of "violating CIPSEA" included "if the interviewer went and told my family, friends, or parents about my information" and "To leak out your name with the answer you provide." A couple of participants indicated more extreme cases of CIPSEA violations such as "someone getting hold of the data and selling it or using it to harass or blackmail" and "Posting name/address on a website with information about what you said."

U11: The first bullet under the "Why Should I Participate?" section of the Question and Answer Brochure was read to all participants. This bullet is the same in both the current and revised versions of the brochure. After reading this bullet, participants were asked, "What did the phrase "accurate policy decisions" mean to you as used in this Question and Answer Brochure?" The interpretation of this phrase varied among the participants and some had difficulty providing an explanation that did not include the term "policy." However, the majority of the answers provided can be summarized into two main themes, using the supporting survey data to (1) make better decisions on what programs need to be developed for mental health and substance abuse and (2) allocate funds accurately (e.g., for treatment programs and laws). Some participant responses included "Policies can be effective in addressing and identifying problems," "Determine if more programs need to be in the mental health field or drug addictions or other addictions," "Developing programs for substance abuse and mental health. Ensuring data will be correct to help develop the programs," "How much money, when the physical budget comes up, is each program going to get versus other programs (e.g., alcohol versus methamphetamine use)," and "How to distribute the money for treatment and law enforcement." Five participants had difficulty providing an answer to this question. Four of these participants answered "don't know" and mentioned that the phrase "accurate policy decisions" could possibly be referring to laws ("anywhere, in workplace and society" or "regarding drug use"). One of these participants answered that it was "not very clear" and then followed up with "assumed the research was conducted in order to help people and know more about the population."

U12: The second half of the cognitive interviews concluded with the interviewer asking each of the participants the following question, "Thinking about both versions of the Lead Letter, Study Description, and Question and Answer Brochures that you read earlier in this interview, are there any other terms or phrases in these materials that are unfamiliar or confusing to you? If yes, what are they and why?" More than half of the participants said there were no other unfamiliar or confusing terms or phrases used in the materials they had read earlier in the interview. While most of the comments provided by participants who answered "Yes" to this question were unique to that individual, there were a couple of issues that were mentioned by more than one participant. The first issue pointed to participants' confusion as to how persons are selected to participate in the NSDUH. One participant questioned why they received the lead letter if it says "it is possible that no one from your household will be chosen to be interviewed." Another participant mentioned that "the idea of being random and possibly having one or two people chosen and then possibly not being chosen, and then if you don't participate someone else not being chosen" was all a little confusing. The second issue pointed to the phrase "similar to you" as used in the "Introduction and Informed Consent for Interview Respondents Aged XX." Two participants questioned what this meant (e.g., "like me as in drug addicts, mental health problems, or same demographic characteristics").

3.2.2 Recommendations

The following recommendations focus on the terms and phrases that proved to be the most difficult for participants to understand based on the findings presented above.

A quarter of the participants either did not know or incorrectly described the meaning of the phrase "randomly chosen" as it was used in the Study Description. While the selection process is described in more detail in the Question and Answer Brochure, not all participants receive this information during the main study. Thus it is recommended that additional information be provided in the Study Description to describe the phrase "randomly chosen" in more common terms.
More participants seemed to better understand the phrase "research purposes" as it was used in the current Study Description versus the phrase "statistical purposes" as it was used in the revised Study Description. Participants appeared to understand the concept of "research purposes" without difficulty and to some extent related it to statistics. On the contrary, two participants could not explain what the phrase "statistical purposes" meant, and three participants who could provide an explanation were concerned that their peers could not. Thus, it may be the case that the phrase "statistical purposes" is less understood among the general population than "research purposes."
While the adult participants could provide an explanation of what the phrase "any project staff or authorized data user" meant as it was used in the Study Description, two participants aged 12 to 17 could not. Thus, it is recommended that a more detailed explanation be provided for this phrase where it is used in the NSDUH contact materials for adolescent participants who are not familiar with the current verbiage.
The phrase "accurate policy decisions" as it was used in the Question and Answer Brochure seemed to be the most difficult for the participants to explain among all of the phrases provided. Thus, it is recommended that a more detailed explanation, with examples of how the results of the NSDUH assist with making accurate policy decisions, be added to the text in the Question and Answer Brochure to better illustrate this concept.

For each of the items mentioned above, it is recommended that alternative language be developed and tested in multiple rounds of controlled, experimental settings such as focus groups and/or cognitive interviews. These types of settings are extremely important to ensure that there is no introduction of response bias in the NSDUH data resulting from the introduction of any revised language.

References

Office of Management and Budget. (2006). Implementation Guidance for Title V of the E-Government Act, Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA) (Web page). Retrieved July 27, 2007, from http://www.whitehouse.gov/omb/inforeg/proposed_cispea_guidance.pdf

Appendix A

Lead Letter (Current Version)

Appendix B

Study Description (Current Version)

Appendix C

Q&A Brochure (Current Version)

Appendix D

Lead Letter (Revised Version)

Appendix E

Study Description (Revised Version)

Appendix F

Q&A Brochure (Revised Version)

Appendix G

Protocols for Cognitive Interviews

Participants Aged 12 to 17

Protocol for Cognitive Testing

(Participants Aged 12 to 17)

National Survey on Drug Use and Health (NSDUH)

Date of Interview__________________ Time Interview Started:__________

ENTER AFTER INTERVIEW COMPLETION:

Interviewer Initials__________ Time Interview Ended: __________

NOTE TO INTERVIEWER:

READ (OR PARAPHRASE) THE FOLLOWING TO THE SUBJECT AFTER THE INFORMED CONSENT IS SIGNED:

(Thank you for participating in our study. The National Survey on Drug Use and Health is a large survey given to about 70,000 people across the country every year. It collects information on many health-related issues. The aim is to better serve all segments of the American population. Right now we're interested in changing some of the printed materials like letters and brochures that we use to tell people about the study. Before we do this, we want to see how well people understand these materials and how these changes may affect whether or not people will participate in the study.)

During this process, I will give you some materials to read through, show you a short video of an interaction between an interviewer and a potential respondent, and ask you some questions about the materials. My job is to take a lot of notes and to figure out how potential respondents think about these materials. There are no right or wrong answers to the questions I ask. Our main goal is to make sure that the materials make sense and tell people what they need to know about the study. You can help us by pointing out anything in the materials you find confusing or unclear. If something doesn't make sense, tell me that. Or, if you're not sure about your response, tell me that, too. When we're done you'll receive a $40 gift card for participation.

Do you have any questions? Let's begin.

GIVE PARTICIPANT ONE VERSION OF THE FOLLOWING CONTACT MATERIALS (BASED ON RANDOM ASSIGNMENT SPREADSHEET):

- STUDY DESCRIPTION

- QUESTION AND ANSWER BROCHURE

The following materials are presented to people who are contacted to participate in the National Survey on Drug Use and Health. When the interviewer arrives at their door, the contacted participants are asked to read the "Study Description" describing the survey. If the contacted participants have further questions regarding the survey, they can be provided with a "Question and Answer Brochure".

Please read through these materials and let me know when you are finished.

WHEN THE PARTICIPANT INDICATES HE/SHE IS FINISHED READING THROUGH THE CONTACT MATERIALS SAY:

The following information is read to all participants of the National Survey on Drug Use and Health before starting the interview:

"This year, we are interviewing about 70,000 people across the nation. You have been randomly chosen to take part. You will represent over 1,000 young people in this country who are similar to you. You may choose not to take part in this study, but no one else can take your place. We will pay you $30 when you finish the interview.

This study asks about tobacco, alcohol, and drug use or non-use, knowledge and attitudes about drugs, mental health, and other health issues. It takes about one hour. You will answer most of the questions on the computer, so I will not see your answers. Your answers will never be seen by either your parents or your school. We are only interested in the combined responses from all 70,000 people, not just one person's answers. This is why we do not ask for your name and we keep your answers separate from your address. RTI may contact you by phone or mail to ask a few questions about the quality of my work. This is why we ask for your phone number and mailing address at the end of the interview.

While the interview has some personal questions, federal law keeps your answers private. We hope that protecting your privacy will help you to give accurate answers. You can quit the interview at any time and you can refuse to answer any questions.

If it is all right with you, let's get started."

AFTER THE INFORMED CONSENT STATEMENT HAS BEEN READ TO THE PARTICIPANT SAY:

Now I will show you a short 2-minute video of a typical interaction between an interviewer and a contacted participant.

PLAY VIDEO.

Based on the information you have read and heard, and the video you have seen, please answer the following questions.

Q1 If your parent or legal guardian were the person contacted in the video and he or she gave the interviewer permission to interview you, how likely would you be to agree to participate in the interview? Indicate a number from 1 to 5, where 1 means you would be not at all likely to participate, and 5 means you would be very likely to participate:

INTERVIEWER – CIRCLE THE RESPONDENT'S ANSWER

1 2 3 4 5 DK/REF

Was there anything in the materials you just read and heard that would influence how likely you would be to agree to participate in the interview? If yes, what?

Q2 As a reminder, the survey being presented in the materials you have read asks about drug use or non-use. If you agreed to participate in the survey, how likely would you be to answer these questions honestly? Indicate a number from 1 to 5, where 1 means you would be not at all likely to answer honestly, and 5 means you would be very likely to answer honestly.

INTERVIEWER – CIRCLE THE RESPONDENT'S ANSWER

1 2 3 4 5 DK/REF

Was there anything in the materials you just read and heard that would influence how likely you would be to answer questions about drug use or non-use honestly? If yes, what?

Q3 The survey being presented in the materials you have read also asks about mental health. If you agreed to participate in the survey, how likely would you be to answer these questions honestly? Indicate a number from 1 to 5, where 1 means you would be not at all likely to answer honestly, and 5 means you would be very likely to answer honestly.

INTERVIEWER – CIRCLE THE RESPONDENT'S ANSWER

1 2 3 4 5 DK/REF

Was there anything in the materials you just read and heard that would influence how likely you would be to answer questions about mental health honestly? If yes, what?

GIVE PARTICIPANT OTHER VERSION OF THE FOLLOWING CONTACT MATERIALS (BASED ON RANDOM ASSIGNMENT SPREADSHEET):

- STUDY DESCRIPTION

- QUESTION AND ANSWER BROCHURE

Here is another version of the materials. Again, these materials are presented to people who are contacted to participate in the National Survey on Drug Use and Health. Please read through these materials and let me know when you are finished.

WHEN THE PARTICIPANT INDICATES HE/SHE IS FINISHED READING THROUGH THE CONTACT MATERIALS SAY:

Again, the following information is read to all participants of the National Survey on Drug Use and Health before starting the interview:

If it is all right with you, let's get started."

THEN SAY:

Based only on the version of the materials you have just read and heard, please answer the following questions.

Q4 If your parent or legal guardian were the person contacted and he or she gave the interviewer permission to interview you, how likely would you be to agree to participate in the interview? Indicate a number from 1 to 5, where 1 means you would be not at all likely to participate, and 5 means you would be very likely to participate:

INTERVIEWER – CIRCLE THE RESPONDENT'S ANSWER

1 2 3 4 5 DK/REF

Was there anything in the materials you just read and heard that would influence how likely you would be to agree to participate in the interview? If yes, what?

Q5 As a reminder, the survey being presented in the materials you have just read asks about drug use or non-use. If you agreed to participate in the survey, how likely would you be to answer these questions honestly? Indicate a number from 1 to 5, where 1 means you would be not at all likely to answer honestly, and 5 means you would be very likely to answer honestly.

INTERVIEWER – CIRCLE THE RESPONDENT'S ANSWER

1 2 3 4 5 DK/REF

Was there anything in the materials you just read and heard that would influence how likely you would be to answer questions about drug use or non-use honestly? If yes, what?

Q6 The survey being presented in the materials you have just read also asks about mental health. If you agreed to participate in the survey, how likely would you be to answer these questions honestly? Indicate a number from 1 to 5, where 1 means you would be not at all likely to answer honestly, and 5 means you would be very likely to answer honestly.

INTERVIEWER – CIRCLE THE RESPONDENT'S ANSWER

1 2 3 4 5 DK/REF

Was there anything in the materials you just read and heard that would influence how likely you would be to answer questions about mental health honestly? If yes, what?

KEEP OUT COPIES OF BOTH SETS OF MATERIALS WHEN ASKING THE PARTICIPANT THE FOLLOWING QUESTIONS.

Based on the two versions of the materials you have read, please answer the following questions.

F1 IS NOT ASKED FOR PARTICIPANTS 12 to 17.

F2 Did you notice any differences between the STUDY DESCRIPTION you read at the beginning of this interview and the one you just read?

If yes, what differences did you notice?

F3 Did you notice any differences between the QUESTION AND ANSWER BROCHURE you read at the beginning of this interview and the one you just read?

If yes, what differences did you notice?

F4a (IF F2 or F3=YES)

Is one version of the materials more convincing in getting someone to respond to the survey being presented?

If yes, which version and why? If no, why?

Did you include the study description AND the question and answer brochure when telling me which version of the contact materials is more convincing?

(If not both): How did you decide which material(s) to tell me about?

F4b (IF F2 and F3=NO) Were the materials convincing in getting someone to respond to the survey being presented?

Why?

Did you include the study description AND the question and answer brochure when telling me whether the materials were convincing?

(If not both): How did you decide which material(s) to tell me about?

F5a (IF F2 or F3=YES) If you agreed to participate in the survey, would either version of the materials make you more or less likely to give honest answers to the questions in the survey about drug use or non-use?

If yes, which version(s) and why? If no, why?

Did you include the study description AND the question and answer brochure when telling me which version would make you more or less likely to give honest answers?

(If not both): How did you decide which material(s) to tell me about?

F5b (IF F2 and F3=NO) If you agreed to participate in the survey, would the materials make you more or less likely to give honest answers to the questions in the survey about drug use or non-use?

Why?

Did you include the study description AND the question and answer brochure when telling me whether the materials would make you more or less likely to give honest answers?

(If not both): How did you decide which material(s) to tell me about?

F6a (IF F2 or F3=YES) If you agreed to participate in the survey, would either version of the materials make you more or less likely to give honest answers to the questions in the survey about mental health?

If yes, which version(s) and why? If no, why?

Did you include the study description AND the question and answer brochure when telling me which version would make you more or less likely to give honest answers?

(If not both): How did you decide which material(s) to tell me about?

F6b (IF F2 and F3=NO) If you agreed to participate in the survey, would the materials make you more or less likely to give honest answers to the questions in the survey about mental health?

Why?

Did you include the study description AND the question and answer brochure when telling me whether the materials would make you more or less likely to give honest answers?

(If not both): How did you decide which material(s) to tell me about?

Now I'm going to read out loud different sections of the materials you read earlier. After each section I'm going to ask you some questions about your understanding of various phrases within these materials.

U1IS NOT ASKED FOR PARTICIPANTS 12 to 17.

READ 1^st PARAGRAPH OF STUDY DESCRIPTION (THIS IS THE SAME IN BOTH VERSIONS)

U2 What did the phrase "randomly chosen" mean to you as used in this Study Description?

READ 2nd PARAGRAPH OF ORIGINAL STUDY DESCRIPTION

U3 What did the phrase "research purposes" mean to you as used in this Study Description?

READ 2nd PARAGRAPH OF REVISED STUDY DESCRIPTION

U4 What did the phrase "statistical purposes" mean to you as used in this Study Description?

READ REMAINING SECTIONS OF THE REVISED STUDY DESCRIPTION (INCLUDING THE CIPSEA REFERENCE AT THE BOTTOM)

U5 In your own words, can you tell me what type of information is collected by the National Survey on Drug Use and Health? Do you think this type of information is important to research? Why?

U6 Would you be convinced to do the interview for $30? Why or why not?

U7 What did the phrase "any project staff or authorized data user" mean to you as used in this Study Description?

U8 What did the phrase "your confidentiality" mean to you as used in this Study Description?

U9 In your own words, can you tell me what the penalty is for "any project staff or authorized data user who violates CIPSEA"?

U10 What does it mean for someone to "violate CIPSEA"? (PROBE FOR EXAMPLES OF WHAT PARTICIPANT CONSIDERS TO BE VIOLATIONS)

READ THE "WHY SHOULD I PARTICIPATE" SECTION OF THE QUESTION AND ANSWER BROCHURE (THIS IS THE SAME IN BOTH VERSIONS)

U11 What did the phrase "accurate policy decisions" mean to you as used in this Question and Answer Brochure?

U12 Thinking about both versions of the Study Description and Question and Answer Brochures that you read earlier in this interview, are there any other terms or phrases in these materials that are unfamiliar or confusing to you? If yes, what are they and why?

D1 Let me confirm that you are _______ years old.

D2 And let me confirm that you are a [MALE/FEMALE]?

1 MALE

2 FEMALE

D3 What is the highest level of education you have completed?

1 Less than high school (includes no schooling through 11^th grade)

2 High school graduate (includes GED)

3 Some college

4 College graduate (at least a 4-year degree)

D4 What is your race? You may identify more than one category. Would you say you are . . .

[Multiple responses allowed]

1 White or Caucasian

2 Black or African American

3 Asian

4 Native Hawaiian or other Pacific Islander, or

5 American Indian or Alaskan Native?

6 Other (specify ____________________)

D5 Are you of Hispanic, Latino, or Spanish origin or descent?

1 YES

2 NO

GIVE THE RESPONDENT A $40 GIFT CARD AND A COPY OF THE SIGNED 'RECEIPT FOR PARTICIPATION' AND THANK HIM/HER FOR THEIR HELP WITH THE INTERVIEW.

Participants – Parents or Legal Guardians of 12-17 Year Olds

Protocol for Cognitive Testing

(Participants - Parents or Legal Guardians of 12-17 Year Olds)

National Survey on Drug Use and Health (NSDUH)

Date of Interview__________________ Time Interview Started:__________

ENTER AFTER INTERVIEW COMPLETION:

Interviewer Initials__________ Time Interview Ended: __________

NOTE TO INTERVIEWER:

READ (OR PARAPHRASE) THE FOLLOWING TO THE SUBJECT AFTER THE INFORMED CONSENT IS SIGNED:

Do you have any questions? Let's begin.

GIVE PARTICIPANT ONE VERSION OF THE FOLLOWING CONTACT MATERIALS (BASED ON RANDOM ASSIGNMENT SPREADSHEET):

- LEAD LETTER TO SELECTED DWELLING UNITS

- STUDY DESCRIPTION

- QUESTION AND ANSWER BROCHURE

The following materials are presented to people who are contacted to participate in the National Survey on Drug Use and Health. People who are contacted by this survey will first receive a "Lead Letter" in the mail to introduce them to the survey and notify them that an RTI interviewer will be in their neighborhood to provide them with additional information. When the interviewer arrives at their door, the contacted participants are asked to read the "Study Description" describing the survey. If the contacted participants have further questions regarding the survey, they can be provided with a "Question and Answer Brochure".

Please read through these materials and let me know when you are finished.

WHEN THE PARTICIPANT INDICATES HE/SHE IS FINISHED READING THROUGH THE CONTACT MATERIALS SAY:

The following information is read to all participants aged 18 or older of the National Survey on Drug Use and Health before starting the interview:

"This year, we are interviewing about 70,000 people across the nation. You have been randomly chosen to take part. You will represent over 3,000 other people who are similar to you. You may choose not to take part in this study, but no one else can take your place. We will pay you $30 when you finish the interview.

This study asks about tobacco, alcohol, and drug use or non-use, knowledge and attitudes about drugs, mental health, and other health issues. It takes about an hour. You will answer most of the questions on the computer, so I will not see your answers. We are only interested in the combined responses from all 70,000 people, not just one person's answers. This is why we do not ask for your name and we keep your answers separate from your address. RTI may contact you by phone or mail to ask a few questions about the quality of my work. This is why we ask for your phone number and mailing address at the end of the interview.

If it is all right with you, let's get started."

THEN SAY:

The following information is read to all parents or legal guardians of 12-17 year old participants of the National Survey on Drug Use and Health before starting the interview.

"Your (AGE) year-old child has been selected to be in this study. Your child's participation is voluntary. This interview asks about tobacco, alcohol, and drug use or non-use, knowledge and attitudes about drugs, mental health, and other health related issues. All of your child's answers will be confidential and used only for [research (IF ORIGINAL VERSION)/statistical (IF REVISED VERSION)] purposes. Since your child will answer most of the questions on the computer, I will never see the answers, and you are not allowed to see them either. If it is all right with you, we'll get started."

AFTER THE INFORMED CONSENT STATEMENTS HAVE BEEN READ TO THE PARTICIPANT SAY:

Now I will show you a short 2-minute video of a typical interaction between an interviewer and a contacted participant.

PLAY VIDEO.

Based on the information you have read and heard and the video you have seen, please answer the following questions.

Q1(Parent)

If you were the contacted participant in the video, how likely would you be to agree to participate in the interview? Indicate a number from 1 to 5, where 1 means you would be not at all likely to participate, and 5 means you would be very likely to participate:

INTERVIEWER – CIRCLE THE RESPONDENT'S ANSWER

1 2 3 4 5 DK/REF

Was there anything in the materials you just read and heard that would influence how likely you would be to agree to participate in the interview? If yes, what?

Q1(As Parent of Child)

If you were the person contacted in the video, how likely would you be to agree to allow your 12-17 year old child to participate in the interview? Indicate a number from 1 to 5, where 1 means you would be not at all likely to allow him/her to participate, and 5 means you would be very likely to allow him/her to participate:

INTERVIEWER – CIRCLE THE RESPONDENT'S ANSWER

1 2 3 4 5 DK/REF

Was there anything in the materials you just read and heard that would influence how likely you would be to agree to allow your 12-17 year old child to participate in the interview? If yes, what?

INTERVIEWER – CIRCLE THE RESPONDENT'S ANSWER

1 2 3 4 5 DK/REF

Was there anything in the materials you just read and heard that would influence how likely you would be to answer questions about drug use or non-use honestly? If yes, what?

INTERVIEWER – CIRCLE THE RESPONDENT'S ANSWER

1 2 3 4 5 DK/REF

Was there anything in the materials you just read and heard that would influence how likely you would be to answer questions about mental health honestly? If yes, what?

GIVE PARTICIPANT OTHER VERSION OF THE FOLLOWING CONTACT MATERIALS (BASED ON RANDOM ASSIGNMENT SPREADSHEET):

- LEAD LETTER TO SELECTED DWELLING UNITS

- STUDY DESCRIPTION

- QUESTION AND ANSWER BROCHURE

WHEN THE PARTICIPANT INDICATES HE/SHE IS FINISHED READING THROUGH THE CONTACT MATERIALS SAY:

Again, the following information is read to all participants aged 18 or older of the National Survey on Drug Use and Health before starting the interview:

If it is all right with you, let's get started."

THEN SAY:

The following information is read to all parents or legal guardians of 12-17 year old participants of the National Survey on Drug Use and Health before starting the interview with the child.

THEN SAY:

Based only on the version of the materials you have just read and heard, please answer the following questions.

Q4(Parent)

If you were the contacted participant, how likely would you be to agree to participate in the interview? Indicate a number from 1 to 5, where 1 means you would be not at all likely to participate, and 5 means you would be very likely to participate:

INTERVIEWER – CIRCLE THE RESPONDENT'S ANSWER

1 2 3 4 5 DK/REF

Was there anything in the materials you just read and heard that would influence how likely you would be to agree to participate in the interview? If yes, what?

Q4(As Parent of Child)

If you were the person contacted, how likely would you be to agree to allow your 12-17 year old child to participate in the interview? Indicate a number from 1 to 5, where 1 means you would be not at all likely to allow him/her to participate, and 5 means you would be very likely to allow him/her to participate:

INTERVIEWER – CIRCLE THE RESPONDENT'S ANSWER

1 2 3 4 5 DK/REF

Was there anything in the materials you just read and heard that would influence how likely you would be to agree to allow your 12-17 year old child to participate in the interview? If yes, what?

INTERVIEWER – CIRCLE THE RESPONDENT'S ANSWER

1 2 3 4 5 DK/REF

Was there anything in the materials you just read and heard that would influence how likely you would be to answer questions about drug use or non-use honestly? If yes, what?

INTERVIEWER – CIRCLE THE RESPONDENT'S ANSWER

1 2 3 4 5 DK/REF

Was there anything in the materials you just read and heard that would influence how likely you would be to answer questions about mental health honestly? If yes, what?

KEEP OUT COPIES OF BOTH SETS OF MATERIALS WHEN ASKING THE PARTICIPANT THE FOLLOWING QUESTIONS.

Based on the two versions of the materials you have read, please answer the following questions.

F1 Did you notice any differences between the LEAD LETTER you read at the beginning of this interview and the one you just read?

If yes, what differences did you notice?

F2 Did you notice any differences between the STUDY DESCRIPTION you read at the beginning of this interview and the one you just read?

If yes, what differences did you notice?

F3 Did you notice any differences between the QUESTION AND ANSWER BROCHURE you read at the beginning of this interview and the one you just read?

If yes, what differences did you notice?

F4a (Parent)

(IF F1or F2 or F3=YES)

Is one version of the materials more convincing in getting someone to respond to the survey being presented?

If yes, which version and why? If no, why?

Did you include the lead letter, study description, AND the question and answer brochure when telling me which version of the contact materials is more convincing?

(If not all three): How did you decide which material(s) to tell me about?

F4b (Parent)

(IF F1 and F2 and F3=NO) Were the materials convincing in getting someone to respond to the survey being presented?

Why?

Did you include the lead letter, study description, AND the question and answer brochure when telling me whether the materials were convincing?

(If not all three): How did you decide which material(s) to tell me about?

F4a (As Parent of Child)

(IF F1or F2 or F3=YES) Is one version of the materials more convincing in getting a parent to allow his/her 12-17 year old child to respond to the survey being presented?

If yes, which version and why? If no, why?

Did you include the lead letter, study description, AND the question and answer brochure when telling me which version of the contact materials is more convincing?

(If not all three): How did you decide which material(s) to tell me about?

F4b (As Parent of Child)

(IF F1 and F2 and F3=NO) Were the materials convincing in getting a parent to allow his/her 12-17 year old child to respond to the survey being presented?

Why?

Did you include the lead letter, study description, AND the question and answer brochure when telling me whether the materials were convincing?

(If not all three): How did you decide which material(s) to tell me about?

F5a (IF F1or F2 or F3=YES) If you agreed to participate in the survey, would either version of the materials make you more or less likely to give honest answers to the questions in the survey about drug use or non-use?

If yes, which version(s) and why? If no, why?

Did you include the lead letter, study description, AND the question and answer brochure when telling me which version would make you more or less likely to give honest answers?

(If not all three): How did you decide which material(s) to tell me about?

F5b (IF F1 and F2 and F3=NO) If you agreed to participate in the survey, would the materials make you more or less likely to give honest answers to the questions in the survey about drug use or non-use?

Why?

Did you include the lead letter, study description, AND the question and answer brochure when telling me whether the materials would make you more or less likely to give honest answers?

(If not all three): How did you decide which material(s) to tell me about?

F6a (IF F1or F2 or F3=YES) If you agreed to participate in the survey, would either version of the materials make you more or less likely to give honest answers to the questions in the survey about mental health?

If yes, which version(s) and why? If no, why?

Did you include the lead letter, study description, AND the question and answer brochure when telling me which version would make you more or less likely to give honest answers?

(If not all three): How did you decide which material(s) to tell me about?

F6b (IF F1 and F2 and F3=NO) If you agreed to participate in the survey, would the materials make you more or less likely to give honest answers to the questions in the survey about mental health?

Why?

Did you include the lead letter, study description, AND the question and answer brochure when telling me whether the materials would make you more or less likely to give honest answers?

(If not all three): How did you decide which material(s) to tell me about?

READ 1^st PARAGRAPH OF LEAD LETTER (THIS IS THE SAME IN BOTH VERSIONS)

U1 What did the phrase "health-related issues" mean to you as used in this letter?