SS Part B

OMB No. xxxx-xxxx October 12, 2010 2011-CMH-PartB.doc

Supporting Statement B for Request for Clearance:

Calibration of the Short Strengths and Difficulties Questionnaire (SDQ) in the National Health Interview Survey (NHIS)

OMB 0920-NEW

Contact Information:

Catherine M. Simile, Ph.D., Project Officer

National Center for Health Statistics/CDC

3311 Toledo Road, Room 2115

Hyattsville, MD. 20782

301-458-4499

301-458-4035 (fax)

[email protected]

November 18, 2010

Supporting Statement B for Request for Clearance:

Calibration of the Short Strengths and Difficulties Questionnaire (SDQ) in the National Health Interview Survey (NHIS)

B. Statistical Methods

1. Respondent Universe and Sampling Methods

The sample design will be driven by the primary research objective which is to determine the optimal cutoff point in the short SDQ for discriminating between “gold standard” positive and negative cases of serious emotional disturbance (SED) as assessed by clinical interviews.

The calibration study sample will be designed to yield approximately 800 follow-up clinical interviews for parents of children aged 4 to 17 years, and approximately 600 follow-up clinical interviews for children aged 11 to 17 years. The sample will be designed to increase the yield of respondents expected to have SED. Mark and Buck (2006) used the 90th percentile of the total difficulties score from the SDQ as the cutoff point for determining SED cases, and they also state that the SED prevalence rate in the US is approximately 9% to13%. Therefore, a stratified sampling design is proposed so that respondents belonging to strata with higher total difficulties scores from the SDQ will have a higher probability of being selected. The stratified sampling design displayed in Table 1 is designed to substantially increase the yield of SED cases in the sample. In this design, four strata will be defined by percentiles of the total difficulties score from the SDQ: (1) ≥ 90th percentile, (2) 75th to <90th percentile, (3) 50th to <75th percentile, (4) < 50^th percentile. Each stratum is designed to yield an equal number of selected cases (333), so the sampling rates will vary from 0.78431 for Stratum 1 to 0.15686 for Stratum 4 (i.e., respondents with higher total difficulties scores will have a higher probability of being selected in an attempt to increase the yield of SED cases). Assuming a respondent agreement rate of 75% and a response rate of those agreeing to participate of 80%, an overall participation rate of 60% should result. This provides a yield of 200 completed cases per stratum. The sample will also be designed to yield approximately 150 completed cases of children and adolescents aged 11 to 17 per stratum.

Table 1. Stratified Sampling Design

Strata	Sample Size	Sampling Rate	Selected Cases	Agreement Rate	Response Rate	Completed Cases
Total	4,250		1,333	0.75	0.80	800
≥ 90th percentile	425	0.78431	333	0. 75	0.80	200
75th to <90th percentile	638	0.52288	333	0.75	0.80	200
50th to <75th percentile	1,062	0.31372	333	0.75	0.80	200
< 50th percentile	2,125	0.15686	333	0.75	0.80	200

The sample size of 800 completed cases is based on experience derived from similar analyses used to determine serious mental illness (SMI) among adult respondents in the National Survey of Drug Use and Health (NSDUH OMB No. 0930-0110). In the SMI calibration analyses RTI conducted three analyses: two interim analyses and a final analysis. The first interim analysis was conducted on a total sample size of 250 cases. Estimates of sensitivity in this analysis were poor, and it was difficult to identify an optimal cutoff point. The second interim analysis was conducted on a total sample size of 400 cases. Estimates of sensitivity were much improved, but it was only with the final analysis with a total sample size of 750 that the estimates had stabilized to the point where there was confidence about the receiver operating characteristic (or ROC) analyses, and the optimal cutoff point that was identified.

Analysis weights for this study will need to be appropriately modified; that is, the weights will likely include the following components: main NHIS study analysis weight; inverse of probability of selection for clinical follow-up; nonresponse adjustment for clinical interview; and poststratification adjustments. The NHIS design variables also will need to be modified (e.g., collapsing of empty strata). The assumptions made above (e.g., SED prevalence rates, respondent agreement and response rates) will be monitored on an ongoing basis to ensure that the desired sample is achieved. If any of those assumptions are demonstrated to be somewhat incorrect, then adjustments to the sampling algorithm may need to be made during the course of the study. Therefore, it is important to ensure that the sampling algorithm is designed to be dynamic, so that changes can be made midstream, if required.

2. Procedures for the Collection of Information

The NHIS Computer Assisted Personal Interview (CAPI) instrument will be programmed to select the sampled respondents. The Census Bureau will deliver the names and telephone numbers for households identified for follow-up to NCHS and NCHS will provide this information to the contractor (Research Triangle Institute (RTI) International) through secure means. Because potential respondents for the follow-up are drawn from respondents to a personal visit survey, most of the addresses are expected to be accurate.

For the households selected to participate in the calibration study, an advance letter will be sent to the parents who answered questions about their children in the National Health Interview Survey. Included in the advance letter will be a $5 incentive for the parents. In addition, the parent and child will each be promised $25 at the completion of the interview. Parents will receive $25.00 in cash and children will receive a $25 gift card.

A team will be assembled at the RTI call center in Raleigh, NC, including interviewers, on-site clinical supervisors, and program management staff. Interview fidelity, supervision, and quality control measures will be designed. Inter-rater reliability exercises will be conducted. Interviewers will be selected, to the extent possible, from the cadre of trained telephone interviewers used for the calibration of the mental health assessment conducted by RTI for the National Survey of Drug Use and Health (NSDUH) for adults ages 18 and older.

Interviewers will call the household and ask to speak to the parent or guardian of the sample child. Interviewers will explain the purposes of the study, risks, and the need for privacy during the follow-up interview, and the amount of time the interview is expected to take. Respondents will be told all elements of informed consent. All respondents are informed about the purpose of the survey, the survey content and expected duration, the confidentiality of their responses, the authorizing legislation, and the voluntary nature of the survey. In general, the following procedures will be used. Parents or guardians are asked for consent for an interview with themselves about their child’s mental health and for children aged 4-17 years; they are also asked for permission for an interviewer to interview their children, 12-17. Children aged 12-17 years are asked to give their assent for participation in the survey. Permission will be requested to record the interviews so that the coding of responses may be verified.

In addition, interviewers may provide respondents with information that generally discusses mental health issues among children/youth and information about how respondents can obtain information about services available in their particular local area. They will also be required to follow specified protocols for handling children or parents of children who have serious mental health problems.

All parents will be administered either the Preschool Age Psychiatric Assessment (PAPA) (see Attachment C) or the Parent Child and Adolescent Psychiatric Assessment (CAPA) (see Attachment D) , depending on the child’s age. They will also be administered the short Strengths and Difficulties Questionnaire (SDQ) (see Attachment F). Children aged 12- 17 years will be administered the Child and Adolescent Psychiatric Assessment (CAPA) (see Attachment E).

Based on the sample size and interview time required for this study, RTI will hire and train a total of 12 Clinical Interviewers (CIs) to complete the pilot and main calibration according to schedule. Two RTI staff psychologists will supervise the work conducted by the Clinical Interviewers. They will be responsible for conducting quality checks on a subset of all the clinical interviews.

Key skills recommended for CIs include strong conceptual skills; good attention to detail; the ability to accurately administer a complex interview protocol; the ability to develop and maintain strong rapport with respondents, including the ability to adjust interview style to competencies and the personality of the respondent; and the flexibility and capacity to work as part of a team and accept constructive feedback.

Based on RTI’s experience with NSDUH as well as Duke University’s experience in conducting their epidemiological studies, to recruit the Clinical Interviewers, contact information will be drawn from APA-accredited graduate programs in clinical and counseling psychology, professional psychology internships, and postdoctoral training (APA web directory; American Psychology Postdoctoral and Internship Centers [APPIC] directory. As interviewers will be required to conduct interviews in the Raleigh located Call Center, we will recruit heavily on the clinical training programs of area universities including University of North Carolina Chapel Hill, Duke University, University of North Carolina Greensboro and North Carolina State University. Each university has psychology, counseling, social work or marriage and family therapy graduate training programs.

Supervision. The Clinical Interviewers (CIs) will be supervised by an RTI Clinical Supervisor (CS). The Clinical Supervisor is well-versed in the complexities of managing a data collection effort and can confidently analyze problems and identify solutions in a timely manner. The supervisors will provide the coaching, feedback, and retraining essential to a high quality, efficient data collection effort.

The Clinical Interview Team Leader, Dr. Rhonda Karg, is a licensed clinical psychologist and certified health care provider. She also served in this role on the NSDUH MHSS. She was integrally involved in supervising the CIs, and was on-call any time that a distressed respondent could be encountered so that level of risk could be verified, and consultation and debriefing could be provided.

Duke’s Center for Developmental Epidemiology (CDE) will lead the 5-day in-person training program and certification of the CAPA and PAPA with RTI’s assistance. RTI and Duke will work together to develop clinical interviewer manuals and training materials. The trainer contracted to do the CAPA and PAPA training is an employee of Duke University who has conducted all similar trainings for Duke’s previous studies using these interview protocols. Duke researchers have conducted multiple epidemiological studies using the CAPA and PAPA protocols which total several thousand interviews conducted over the past decade. These studies have employed interviewers with parallel expertise to that required in this study and who were recruited in a manner consistent with current study plans. Training on telephone interviewing techniques and procedures will be lead by RTI drawing from their corporate experience conducting telephone interviews with children, adolescents and their families.

The multi-disciplinary training program developed for this study will be built on principles of adult education, including self-study manuals, class instruction and demonstrations, and hands-on practice such as conducting interview mocks. In order to successfully graduate from training, Clinical Interviewers will also be required to pass a standardized certification process, which includes successfully conducting 2 practice interviews with Duke clinicians according to protocol. The in-person training program and certification process will ensure that the clinical interviewers are well-prepared, knowledgeable and confident to begin work immediately after training.

Field Manuals and Home Study Materials. The clinical interviewers will receive field manuals and home study exercises 2 weeks before arriving at training. The clinical interviewer manual will contain detailed descriptions of all procedures that CIs will follow during data collection, including distressed respondent protocols. The manual also will contain information about the hardware and software for the tablet computer (responses are written long hand) and the audio recording device (all interviews are taped so inter interviewer reliability can be assured), including troubleshooting solutions for various computer issues that CIs may encounter, such as transmission problems. This manual will serve as a reference tool during training and data collection. The home study materials will be designed to ensure that all CIs review the manual and are familiar with key points before attending training.

Clinical Interviewer Training Program. The clinical interviewer training program will be conducted in the RTP, NC area. The 5-day CI training session will focus on key protocols and procedures specific to recruiting participants, scheduling and administering the CAPA and PAPA instruments, handling distressed respondents and other adverse events, telephone etiquette with adults and children, typing and data capture accuracy, and transmitting data. The training will also include instruction on basic interviewing techniques, including gaining informed consent and cooperation, probing and avoiding bias, maintaining confidentiality, sensitivity, and data security. General administrative tasks will be discussed as well, including instructions on the case management system, transmission and completing timesheets. Clinical interviewers will receive one-on-one feedback and coaching from a trainer on any items missed during the certification process. The training and certification process will enable trainers to assess the CIs’ ability to follow project protocols and allow them to provide direct feedback and retraining prior to beginning work, thus maximizing data quality.

The contractor will continually monitor the data collection process, and report to NCHS on a weekly basis in phone conversations or written reports.

The interviews are coded by the interviewers directly following the interview. A Duke University programmer will apply to the interviewer data a Duke-developed diagnostic algorithm designed to generate DSM-IV diagnosis representing Serious Emotional Disturbance (SED) to produce a full SAS dataset to be delivered to RTI. The symptom information gathered using the CAPA and PAPA is being used to determine whether or not each child meets the criteria for one or more of five DSM-IV diagnoses: depression, anxiety, attention deficit disorder, oppositional defiance disorder, and conduct disorder. An RTI statistician will conduct the calibration analysis and determine the cutoff score to be used in the NHIS. A recode will be placed on the 2010 NHIS microdata file signaling children likely to have SED as determined by their SDQ score.

NCHS staff will monitor all aspects of the calibration.

3. Methods to Maximize Response Rates and Deal with Non-response

There are three methods used to maximize response and cooperation. First, the calibration is being done as a follow-up to the in-person NHIS. As a result of that positive experience respondents are expected to be willing to participate in a follow-up study about their children’s mental health. Furthermore, the follow-up interviews will be conducted within a few months of the successful NHIS interview, so locating respondents is not expected to take a lot of time. Second the protocol calls for replacing cases that are either difficult to locate or who are uncooperative. Finally, the protocol proposes the use of incentives. The sponsor of this project, SAMHSA, conducted an incentive experiment for the NSDUH, a survey that contains substantial mental health sections. A randomized, split-sample, experimental design was included with the main study data collection to compare the impact of $20 and $40 incentive treatments to a $0 control group on measures of respondent cooperation, data quality survey costs, and substance abuse prevalence estimates. Overall, the study found that the use of monetary incentives significantly increases response rates. Specifically it showed substantial gains in cooperation with when a $20 incentive versus no incentive was offered (78.8% vs. 69. 2%), with more modest gains when a $40 vs $20 incentive was used (78.8 vs. 83.3).

The findings were especially striking among 12-17 years olds where a $20 incentive shifted the cooperation rates from 78.8 to 91.1%, with the $40 incentive bumping the cooperation rate to 95.1%. Teenagers are an integral part of the proposed survey because only they, not their parents can give accurate information about symptoms that help to determine whether or not the teenager is experiencing such internalizing disorders as depression and anxiety.

Other research indicates that prepaid and promised tokens of appreciation may have an impact on increasing response rates to telephone surveys (Cantor, O’Hare, and O’Connor, 2008).

For this study, the protocol calls for the inclusion of a $5 dollar prepaid incentive with the advance letter that promises an additional $25 once the interview is implemented in whole or in part. The combination of these two incentives should maximize cooperation rates without incurring significant survey costs.

4. Tests of Procedures or Methods

There are no proposed tests of the procedures.

5. Statistical Consultants

NCHS Statistical consultant:

Catherine M. Simile, Ph.D.

National Center for Health Statistics

Hyattsville, MD 20782

[email protected]

RTI International

Heather Ringeisen, Ph.D.

Senior Research Psychologist

Survey Research Division

RTI International

P.O. Box 12194

3040 Cornwallis Road

Research Triangle Park, NC  27709

Phone:  919-541-6931

Fax:  919-541-1261

Email:  [email protected]

References:

Cantor, D, O’Hare, BC and KS O’Connor. 2008. The Use of Monetary Incentive to Reduce Nonresponse in Random Digit Dial Telephone Surveys. Pp. 471-498 Lepkowski, JM, C Tucker, JM Brick, E deLeeuw, L Japec, PJ Lavrakas, MW Link and RL Sangster (eds.) Advances in Telephone Survey Methodology.

Mark, T.L., & Buck, J.A. (2006). Characteristics of U.S. youths with Serious Emotional Disturbance: Data from the National Health Interview Survey. Psychiatric Services, 57(11), 1573-1578.

List of Attachments

1. Authorizing legislation

A1. NCHS Authorizing Legislation

A2. CMHS authorizing legislation

2. Federal Register notice - 60 day notice

3. Preschool Age Psychiatric Assessment (PAPA) (for parents of children aged 4-8 years)

C0-C40 (pages)

C41-C90

C91-C149

C150-C199

C200-C250

C251-C300

C301-C316

4. Child and Adolescent Psychiatric Assessment (CAPA)( for parents of children aged 9-17 years)

D1-D51

D52-D101

D102-D150

D151-D200

D201-D232

5. Child and Adolescent Psychiatric Assessment (CAPA)( for children aged 12-17 years)

E1-E50

E51-E100

E101-E150

E151-E203

6. Short Strength and Difficulties Questionnaire

7. NCHS ERB Approval

8. Consent Scripts and Letters

7