Supporting Statement Part B_FINAL

National Survey of Child and

Adolescent Well-Being Second Cohort

(NSCAW II)

OMB Supporting Statement Part B

Submitted by:

Mary Bruce Webb, Ph.D.

Administration for Children and Families

370 L’Enfant Promenade SW

Washington, DC 20447

PART B: COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1 Respondent Universe and Sampling Methods

Overview

The following section describes the sample design for NSCAW II as it is currently being implemented, including the target population, methods used to select the sample, sample allocations across domains, and response rates expected based on our experience to date and final response rate projections. Since the NSCAW II is essentially a new cohort of the NSCAW I, we begin by briefly reviewing the NSCAW I sample design and indicating how the NSCAW II design differs.

The NSCAW I sample was designed to achieve several important goals related to the research objectives. First, a national probability sample was required in order that the sample represent all components of the child welfare system nationwide. Second, as mandated by the legislation authorizing the study, the NSCAW I sample design provided for state estimates to the extent possible. Third, during the development of the survey objectives, it was determined that the sample size should be controlled for certain analytic domains in order to ensure adequate precision for certain key subgroups of the target population. These groups were defined by age, type of abuse/neglect, placement outside the home, and receipt of services, if any. Fourth, the NSCAW I design included a supplemental sample of children in foster care for at least one year; this was the so-called Long-Term Foster Care sample supplement. Finally, the sample sizes for all domains and the supplement were calculated to provide adequate precision and power for key statistical analyses while satisfying the budgetary constraints of the project.

The NSCAW II sample design mirrored that used in NSCAW I with several notable exceptions. After assessing uses of the data by the research community, it was decided to eliminate the Longer Term Foster Care sample component while retaining the Child Protective Service (CPS) sample component. Similarly, the ability to calculate state estimates has not been an important NSCAW I feature, and was therefore not retained in the NSCAW II sample design. The allocation of the child sample to domains of analytic interest was updated to focus more on infants, children receiving services and children in out-of-home placement, thus eliminating the abuse type domains. This more streamlined array of domains covers many research objectives and maintains an overall design effect comparable to that from NSCAW I. In addition, child age eligibility was 0 to 14 years for the NSCAW I. The upper range was extended 17 ½ years in NSCAW II due to increasing interest in adolescents in the child welfare system. An important feature of the NSCAW II sample is the re-use of the NSCAW I PSUs. This was done to boost the precision for estimating and comparing trends between NSCAW I and II system-level data.

The fundamental features of the NSCAW II sample design are as follows:

 The NSCAW I and II primary sampling units (PSUs) were defined as geographic areas that encompass the population served by a child welfare agency (CWA). In most cases, these areas are counties or groups of counties. However, in large metropolitan areas, smaller geographic areas were defined.

 At the second stage of sampling, a total of approximately 5,341 children will be interviewed¹. In most agencies, approximately 40–100 children will be randomly selected over the course of 15 months. In some counties, the number of children selected will be larger in order to reflect growth in the number of child abuse and neglect cases since the original first-stage sample was taken in 1999–2000 and to represent the 2008 child welfare population accurately. In addition, in some counties, the data collection period was extended by 1-3 months to achieve the desired sample sizes in some key analytic domains.

 We have finished data collection activities for sample months March through December, and we are in the process of closing the January sample month. As illustrated in Exhibit B.1-1, overall, the response rate for these first five sample months is currently 60%. Based on current production levels and the pending sample in months February through April, we are anticipating an overall response rate of 63-64% - a response rate that mirrors that obtained in NSCAW I. We will analyze the potential for nonresponse bias using sample frame file data, and communicate any necessary cautions to analysts and data users. See appendices H and J for more detailed analysis regarding nonresponse.

Exhibit B.1-1 NSCAW II Baseline Response Rates by Sample Month

Sample Month	Response Rate
March	71.6%
April	65.9%
May	67.8%
June	67.9%
July	68.8%
August	63.2%
September	62%
October	55.9%
November	57.7%
December	51.1%
Cumulative through October Sample Month	60%

• Longitudinal follow-up interviews will be conducted for the 5,341 children in the cohort whose baseline interview is obtained. This 18-month follow-up of Wave 1 respondents (referred to as Wave 2 and the focus of this request for extension) is scheduled to begin in October 2009.

• As of May 31, 2009, we had completed approximately 86% of the planned number of baseline interviews. Completed interview by respondent type are provided in Exhibit B-1.2.

Exhibit B.1-2 NSCAW II Baseline Completed Interviews (as of 5/31/09)

Respondent Type	Completed Interviews
Child	4,009
Caregiver	4,519
Caseworker	3,820
Key	4,525

Target Population

The target population for the NSCAW II sample is all children who are subjects of child abuse and neglect investigations (or assessments) conducted by CPS between February 2008 and April 2009. Thus, the target group is restricted to children who are reported to CPS and who are subjects of either an investigation or family assessment for child abuse or neglect. Although a number of these will go on to receive services, the target population also includes cases that are not substantiated and cases that are substantiated but do not subsequently receive services. We also anticipate that the sample eligibility criteria will result in the inclusion of some children in the system for other reasons such as dependency cases, status offenders, children on probation or children in need of supervision (ChINS), and children of families who voluntarily seek child welfare services. Cases that receive only such services such as well-baby visits from a public health nurse, mental health services, or preventive services to teen mothers are not included, except when these services are provided because of a child abuse or neglect report, the child is in child welfare custody, or the services are provided under the child welfare budget.

According to 2006 data from the National Child Abuse and Neglect Data System (NCANDS), over 3 million referrals of abuse or neglect, concerning approximately 6 million children, were received by child protective services (CPS) agencies. Almost 62 percent of those referrals were accepted for investigation or assessment.

Selection of PSUs

As noted previously, the NSCAW II sample uses the same PSUs that were used in NSCAW I. Therefore, we briefly review the method for selecting the NSCAW I PSUs and then outline some of the analyses conducted to show the optimality of this PSU sample for NSCAW II.

Review of the NSCAW I Sample Design

The NSCAW I PSUs were selected in 1998 with probability proportional to size measures derived from the 1996 NCANDS data. First, the entire U.S. child welfare (CW) population was partitioned into PSUs which were essentially counties or, in a few cases, groups of counties. Extremely large counties (i.e. MSAs) were subdivided into smaller units, such as areas delineated by CPS branch office jurisdictions, to create more manageable PSUs. For simplicity, one can associate PSUs with counties since that is the case in the vast majority of PSUs. PSU size measures were calculated as composites of domain sizes using the method in Folsom, Potter and Williams (1987). This method provides an approach for controlling domain sample sizes while maximizing the efficiency of the design. The composite size measure reflects the size of the sample that would fall into the PSU if a national random sample of children were selected with the desired sampling rates for all domains but without PSU clustering.

As is standard practice for national samples surveys, counties with fewer than some minimum number of eligible children were deleted from the first-stage sampling frame for reasons of sampling efficiency and field costs reduction. Altogether, 710 counties were deleted due to their small size (defined as 67 eligible children or less), accounting for approximately 22,362 children or 0.9 percent of the target population. The PSU frame was then stratified into nine major strata. The eight key states where we made state-level estimates constituted the first eight strata. The PSUs in the remaining 42 states and D.C. were grouped into the ninth stratum. Within the nine strata, the PSUs were implicitly stratified by urbanicity and Census regions. Finally, Chromy’s minimum replacement probability proportional to size sampling method (Chromy, 1979) was implemented to obtain the sample of 100 PSUs.

Within PSUs, eight sampling domains were defined based upon child age, abuse type, whether in foster care and whether receiving CPS services. Sample sizes for these domains were determined based upon extensive power analyses and the requirements of analysts in relevant research fields. Within each PSU, the number of children selected in each PSU was dictated by the composite size measure theory in Folsom, et al (1987). As the CPS agencies/sample PSUs were recruited, the projected domain sizes were refined and the sampling rates by domain by PSU were modified accordingly. Customized software was developed to apply these sampling rates to the domains during the 15-month second-stage sampling period which began in October, 1999 and continued until December, 2000.

During the recruitment process, eight agencies declined participation due to policies within these states which precluded their participation in the study. These policies essentially require CPS agencies to obtain written consent from caregivers before releasing any information on children for purposes of NSCAW sampling. These so-called agency first contact states were deemed out of scope for the study and excluded from the target population. This reduced the coverage of the sample frame to about 94.8%, which was considered a very small loss of coverage for the purposes of the study. An analysis of the potential bias associated with the exclusion of these agencies was performed to determine if inference could be extended to include these out-of-scope states. That analysis revealed little or no bias in the estimates for the key domains of analysis.

NSCAW II Sample Design

As previously mentioned, the NSCAW II sample design makes maximum use of the 92 primary sampling units (PSUs) from the NSCAW I. This was done primarily to improve the precision of comparisons between NSCAW I and NSCAW II. In addition, NSCAW analysts are quite interested in evaluating the effects of changes at the agency level on child outcomes. However, there are also practical benefits in the re-use of NSCAW I PSUs. Inarguably this approach is also the most expedient approach to fielding the survey since the project has worked in these PSUs for many years, has an established and very experienced workforce in each, and has built a solid cooperative relationship with the child welfare agencies in the sites. Drawing a completely new sample of agencies would have lengthened the NSCAW II schedule for getting into the field and would have required the recruitment and training of a large number of new interviewers. In addition, agency-level nonresponse as well as the incidence of other data-quality-related problems might be expected to be higher with a new sample of PSUs.

A number of technical issues were investigated before the decision to use the NSCAW I PSUs was finalized. For example, the precision of the NSCAW estimates could suffer from the use of PSUs selected using outdated size measures. The reduction in precision might be substantial if the distribution of the child welfare population has changed dramatically since 1998. Therefore, we examined the change in county-level child welfare populations from 1998 to the present and the effect these changes could have on the precision of NSCAW II results.

We also conducted a cost-error optimization analysis aimed at determining the optimal number of PSUs to sample when costs and errors are simultaneously considered. It is possible that fewer PSUs and a larger sample size within PSUs could yield greater precision for essentially the same cost. If so, then it would be possible to reduce the number of PSUs while increasing the overall sample size for the study. Further analysis showed the current number of PSUs to be near optimal. Although adding a few PSUs and reducing the average sample size per PSU would increase precision slightly, the cost of recruiting the new agencies and other logistical considerations offset the modest precision gains. Changes in frame coverage were also examined since the NSCAW II PSU sample design is based upon the NSCAW I PSU sampling frame. Recall that 710 extremely small counties were deleted from NSCAW I and this had a very slight effect on frame coverage. We determined that only a small number of these counties now meet the criterion of 67 or more cases per year to be included in the sampling frame. Representing these counties in the NSCAW II sample would only increase CW population coverage from about 98% to 99.3%, a trivial increase. Therefore, based upon cost-error optimization, it was decided that the coverage of the NSCAW I frame is quite adequate for the purposes of NSCAW II.

With regard to the within PSU sampling domains, it is important that sufficient sample be achieved in the various domains of interest in order to provide adequate power for analysis. However, disproportionate sampling of domains can reduce the precision of the general population estimates due to the unequal weighting effects. It is important to achieve a balance between the objectives of subgroup analysis and general population analysis.

Consultation with ACF and a team of external experts in the field identified five analytic domains for the study. These were:

1. children who receive no services,

2. infants (less than 1 year old) in out of home (OOH) care,

3. non-OOH infants who are receiving services,

4. older (age 1-17 ½ ) children in OOH care and

5. older non-OOH children who are receiving services.

To maximize sampling efficiency and control precision, it is essential to have very good estimates of the sizes of these domains for each PSU in the sample since they, together with the PSU by domain sample size, determine the within PSU sampling weights. Initially, data from the 2006 NCANDS data file were used to project the sizes of these five domains. As sampling has progressed, these domain size estimates have been updated monthly using a type of empirical Bayes estimator of the annual domain population size. Likewise, the domain allocations have been adjusted each month in order to minimize the variation in the final base weights.

One advantage of the composite size measure used in NSCAW I is the ability to control design effects while keeping interviewer workloads approximately equal across PSUs. In NSCAW I, the achieved sample size in each PSU was about 60 interviews. However, the NSCAW I composite size measures are ineffective for NSCAW II since they are based on 1996 NCANDS data and, thus, no longer reflect actual population sizes. To compensate for this, NSCAW II PSUs sample sizes vary in the range of 40 to 100 for the most part with a few large PSUs having workloads in excess of 100. Although this has resulted in somewhat uneven staffing of the fieldwork across PSUs, it has provided a much reduced design effect compared to an equal allocation across PSUs.

Additional Agency First Contact States

In NSCAW I, four states (involving 8 PSUs) were excluded from the target population due to agency first contact requirements. In NSCAW II, four additional states (involving 9 additional PSUs) were removed from the target population for this reason. We estimated the proportion of the CW population lost from the study by this exclusionary target population definition. After removing the four states, the NSCAW I sample represented approximately 94.8% of the total CW population in the US. Eliminating the four additional states in NSCAW II drops the coverage to around 87.0%. Given the information available from NSCAW I on the NSCAW II new exclusions, it may be possible to retain representation of these states to some extent in NSCAW II using a weighting coverage adjustment. This will be discussed in more detail in Section B2.2.

The two states identified as agency-first-contact after sampling and data collection had started had the most impact on sampling operations because their sample allocations (a total of 359 cases) were still being included in monthly updates to the sample size through November 2008. We considered the question as to what to do with the cases allocated to them. One option was to retain the 359 cases and spread them optimally (according to the minimum design effect criterion) to other PSUs. Alternatively, we could simply reduce the sample by 359 cases, thus eliminating the costs associated with, and burden on other agencies of, interviewing these cases in other PSUs. Our cost-error analysis showed unequivocally that reducing the sample size by 359 cases was optimal since retaining the cases had little impact on the effective sample sizes for any domain due to the large design effect associated with their reallocation. Thus, with the elimination of these 359 interviews, the target achieved sample size for the study was revised downward to 5,341 completed interviews.

Comparison of NSCAW I and NSCAW II PSUs

Starting in late July, 2007, agency recruiters from RTI and ICF Caliber began contacting all counties that participated in NSCAW I and requesting their continued participation in NSCAW II. Recruitment was finalized on September 30, 2008. Fewer than 15 total NSCAW I counties refused to continue participation in the study. For each refusing county, a replacement county similar to the original county was selected. Agency recruiters contacted each new county to introduce NSCAW and gain their cooperation.

A total of 81 primary sampling units (PSUs) are participating in NSCAW II. Exhibit B1-3 shows the distribution of the 100 PSUs originally selected for NSCAW I by NSCAW II participation status.

Exhibit B1-3. NSCAW I and II PSU Comparison

NSCAW I Status	NSCAW II Status				Total
	Cooperating	Refusal, Replaced	Refusal, Not Replaced	Out-of-scope
Cooperating	67	8	2	9	86
Refusal, Replaced	2	4	0	-	6
Refusal, Not Replaced	0	0	0	-	0
Out-of-scope	-	-	-	8	8
Total	69	12	2	17	100

As the table indicates, of the 92 PSUs that cooperated in NSCAW I (the 86 in the Cooperated Total cell + the 6 replacement counties in the Refusal, Replaced Total cell), 67 (73%) are also participating in NSCAW II. Two NSCAW I replacements refused to continue in NSCAW II and were replaced with the original NSCAW I refusing county (the 2 in the intersection of the Cooperating column and Refusal, Replaced row). Of the 14 NSCAW I counties that refused to continue in NSCAW II, 12 were replaced, 1 could not be replaced (we tried three other counties), and 1 made the decision to refuse well after the start of data collection and so was not replaced. The total number of PSUs – 81 participating – does not include 5 new PSUs selected for the state supplementary study.

We have completed child level sampling in all 81 participating PSUs with the sample of investigations closed in April 2009. As in NSCAW I, each of the 81 participating PSUs sends a monthly sampling frame file containing records of investigations/assessments that closed during the previous month. A sample of children is then selected from each file. The anticipated number of completed interviews is estimated from the sampled cases using observed eligibility and completion rates. Based on these projections, it is expected that only a few PSUs would need samples of the April 2009 cases.

The actual number of cases completed is being closely monitored in order to estimate the number that will be completed from those to be finalized. If projections suggest that we will fall well below our overall or domain sample allocation targets, we will release more April 2009 cases. Note that the index investigations in the sample are more highly clustered than was true in NSCAW I; that is to say that the majority of the investigations sampled closed in the eight-month period between July 2008 and March 2009, rather than an even distribution across the sampling period. This was necessitated by the very slow starts we experienced because of review and approval processes (one took 13 months), unavailability of programming or other staff to begin sample frame file submission, and problems with frame file contents.

Statistical Power

It is impossible to specify the exact statistical power for every type of analysis planned for these data because they include many different research questions, variables, subpopulations and analysis methods. However, the expected cohort size of 5,341 children is only slightly lower than the NSCAW I survey, which had 5,501 completed key respondent interviews. Our experience with the NSCAW I analysis suggests the sample was quite adequate for the many types of analysis conducted to date, both for cross-sectional and longitudinal analyses. As an example, for the descriptive estimates contained in Wave 1 Analysis Report (DHHS, 2005), the coefficients of variation (cv’s) of the estimates never exceeded 20%. Further, for comparing estimates between independent subgroups or for comparing estimates of the same subgroup at two points in time, the precision of the comparisons was quite adequate for the eight planned domains of analysis as well as for a number of ad hoc domains.

To illustrate the statistical power of the NSCAW II sample, Exhibit B.1-4 shows sample sizes for the five sampling domains, their corresponding effective sample sizes, and the poer for detecting small effect sizes (defined as a difference in the estimates as small as 20% of the standard error of the difference). These allocations are approximations, and may change based on the final achieved sample. The effective sample sizes in the exhibit were computed as the actual sample size divided by the appropriate design effect. The design effect for the total sample is predicted to be about 6.9. It varies by domain from 2.8 to 3.7. These design effects, which account for both clustering and the unequal weighting but not post-survey adjustments, are comparable with those observed for NSCAW I prior to the final weighting adjustments. Following the post-survey weighting adjustments, the NSCAW I design effects were reduced by as much as 20% or more. To account for these potential gains in precision, we list a range of effective sample sizes and their corresponding power ranges. Power is based on a two-sided test of size α = 0.05 comparing two independent means or proportions; for example, the mean total score on the CBCL for children in two different domains. As previously noted, these calculations assume that a “small effect size” (i.e., an effect size of 0.2 on Cohen’s effect size scale; Cohen, 1988) is to be detected.

Exhibit B.1-4 Sample Size, Effective Sample Size and Power by Domain

NSCAW II Sampling Domain				Actual N	Range of Effective N	Range of Power for Small Effect Sizes
Age	Open or Closed		In Foster Care?
Less than 1 year	Open		Yes	1,031	295-353	0.69-0.75
1-17.5 years	Open		Yes	1,031	368-442	0.78-0.84
All Ages	Closed		No	1,217	393-471	0.80-0.87
Less than 1 Year	Open		No	1,031	279-334	0.68-0.73
1-17.5 years	Open		No	1,031	368-442	0.78-0.85
All Domains				5341	774-029	0.97-0.99

The power shown in Exhibit B.1-4 represents a worst case scenario since we have assumed independent means and small effect sizes. For example, increasing the Cohen effect size from 0.2 to 0.35 (which is still in the small range according to Cohen’s rule) increases the smallest power in the table from 0.69 to 0.99. For dependent means (e.g., change estimates) the power may be close to 1. For example, assuming a moderate correlation between the means will increase smallest power in the table to from 0.69 to 0.95. For medium and larger effect sizes (i.e., effect sizes of 0.50 and 0.80, respectively) the power is always close to 1 for testing for differences between independent means. This analysis suggests that even for small subgroups in the population, the design provides adequate power for a wide range of analyses. In addition, these calculations ignore the special sample supplement in one state that also boosts precision for the national estimates.

Wave 2 Sample Design

The Wave 1 cohort includes every sampled child for whom a Wave 1 interview has been completed with the key respondent. At Wave 2, we will conduct interviews with each of the children in the Wave 1 cohort, their current caregivers and services caseworkers (if services have been received since the Wave 1 caseworker interview), and their teacher if the child is school-age and not home-schooled. Currently, the number of children we will be interviewing at Wave 2 is 5,341, and we will follow them in the 81 PSUs they were sampled from, and to many other locations (for those who have moved). Our target response rates are 80-82% for children, 83-85% for caregivers, and 93% for caseworkers.

Statistical Power

For Wave 2 the statistical power is estimated in much the same way as in Wave 1. Assuming approximately 80% of the 5,341 children interviewed in Wave 1 will respond in Wave 2 our overall Wave 2 sample size is projected to be 4,273. Exhibit B.1-5 shows the projected Wave 2 sample sizes by domain and the effective sample size ranges adjusting for the predicted design effects. Since we will be comparing the survey results of the same children in Wave 1 and Wave 2, we calculated the power using a dependent two sample t-test for means and small effects (again assuming effect size of 0.2 on Cohen’s effect size scale).

Exhibit B.1-5 Sample Size, Effective Sample Size and Power by Domain for Wave 2

NSCAW II Sampling Domain			N Assuming 80% retention in Wave 2	Range of Effective N	Range of Power for Small Effect Sizes
Age	Open or Closed	In Foster Care?
Less than 1 year	Open	Yes	825	235-282	0.86-0.92
1-17.5 years	Open	Yes	825	294-353	0.93-0.96
All Ages	Closed	No	973	304-365	0.93-0.96
Less than 1 Year	Open	No	825	223-267	0,85-0.89
1-17.5 years	Open	No	825	294-353	0.93-0.96
All Domains			4273	619-743	0.99-~1.0

For both Wave 1 and Wave 2, the effective sample sizes and power calculations are based upon our best estimates of the final design effects after all post survey weighting adjustments have been applied.  Also, similar to the Wave 1 results, the Wave 2 power is quite adequate for all domains and for a range of analyses.

B.2 Information Collection Procedures

This section describes the procedures for data collection. Also addressed are weighting procedures and analytical techniques to be employed.

B.2.1 Data Collection Procedures

Procedures for data collection mirror closely those developed for and used successfully on NSCAW I, and to date on NSCAW II Wave 1.

Advance letters and project brochures have been prepared for the different types of respondents and will be provided prior to interviews. (See draft letters and project FAQ included in Appendix F.)

Data Collection from Caseworkers

18-Month (Wave 2) Follow-Up Interviews with Services Caseworkers. Eighteen months after the close of the index investigation or assessment, the field representative will interview a service caseworker. This caseworker interview will focus on the services recommended for and received by the sampled child and her/his family since the baseline interview, the case history after the index investigation including any subsequent reports or investigations, the living environment in the household, caseworker involvement with the family, progress made by the family, and information about the caseworker. To facilitate the caseworker’s preparations for these interviews, the field representative will send a letter with a list of topics to be addressed. Because of the detailed nature of many of the questions, we will request that the caseworker have the child’s case record available for the interview session. The caseworker interview will average 60 minutes.

Collection of Data from the Adult Caregiver

Selection of the Adult Caregiver. In many instances there is little question regarding who should be interviewed about the child. In other situations we have to carefully sift through information to identify the most appropriate adult respondent. Regardless of the family situation, the guiding principle is to interview the adult in the current household who knows the sampled child best and who can accurately answer as many of the questions as possible.

Informed Consent Procedures. The adult respondent is asked to consent to participate for both her/himself and the selected minor child. Written consent procedures will be repeated in each personal interview (baseline, 18 months). Draft Wave 2 consent forms, and authorization forms for teachers to release information and for data linkage are included with other project materials in Appendix E.

Field representatives are carefully trained to confirm with the caseworker that the adult respondent chosen has legal guardianship and the resulting legal right to consent to the child’s participation. If there are two legal guardians in the household, the field representative attempts to secure signed consent from both guardians. If the chosen adult respondent does not have guardianship rights, the field representative identifies and contacts the person who does have the authority to consent for the child. In some sites the agency has guardianship for out-of-home placement children; in other sites the family court or juvenile court may hold guardianship. The field representative contacts the guardian, explains the study and the child’s selection, and seeks permission to interview the child and authorization for others (i.e., a teacher) to release information about the child.

In cases where a non-custodial parent retains guardianship and must be contacted to obtain informed consent for the child’s participation, the field representative talks with the caseworker to get her/his suggestions on the best approach. Depending on the specific case situation, the field representative may send a letter notifying the parent of his/her child’s selection into the study sample and the need to discuss our request further, telephone the parent to deliver introductory information, or arrange to meet with the parent in person.

Interview Content. This CAPI questionnaire is focused on the child’s health, mental health, services received by the child and the family, the family environment, and experiences with the child welfare system. The 18-month follow-up will be fielded two weeks before the anniversary of the investigation closing. Using timing data from the current baseline interview, the 18-month interview is expected to average an administration time of 115 minutes for each in-person caregiver interview.

Collection of Data from Sampled Children

Informed Consent for Selected Children. As noted above, we obtain signed informed consent from the legal guardian of each sampled child before approaching that child for participation in the study. Additionally, we have assent forms for children 7 through 17 ½ years old and read the appropriate assent form to the child before beginning the interview, as a means to introduce the child to the study, to assure the child that what they tell us will be kept confidential (with the exceptions surrounding expressed suicidal intent and suspected ongoing serious abuse), and to provide the child with an understanding of the voluntary nature of participation and their right to refuse to answer any question we ask of them. Given the vulnerability of this particular population, we believe it is especially important to provide these children with this information. From our NSCAW I pretest findings, we do not believe that it is feasible to go through informed assent procedures with children younger than 7, as they do not comprehend some of the fundamental concepts necessary to meaningfully process the information.

Collecting Data from Children. Sampled children under 18 years of age will be interviewed in the same visit to the household as the adult caregiver. Once a signed consent form has been obtained from the legal guardian and the study has been explained to the adult caregiver (who may also be the same person), the field representative seeks to conduct a CAPI interview with sampled children and collect physical measurement and observation data for infants and toddlers. The timing of the adult caregiver and child interviews varies by circumstances and the convenience of respondents; field representatives schedule both interviews in the same visit to the household when possible (about 70% of NSCAW I interviews).

Sampled children who, at the 18-month follow-up, are now 18 years or older will be interviewed in their place of residence, but a caregiver interview will not be required. Children ages 18 and older (Young Adults) will sign consent (rather than assent) forms.

The interview protocol varies considerably depending on the age of the child. Only physical measures (length, weight, and head circumference) and physical development measures are taken from the very youngest infants; older babies are assessed through the Battelle Development Inventory. Toddlers and young children will complete several cartoon-based and other simple measures in addition to the physical measures of height and weight. The interview protocol for older children includes questions on physical health, mental health, assessments of cognitive development and academic achievement, and for 11 and older, questions in Audio Computer-Assisted Self Interview (A-CASI) mode about events that led to their involvement with the child welfare system and their relationship with caregivers. The A-CASI sections include questions on substance abuse, sexual activity, delinquency, injuries, and maltreatment. Using timing data from the current baseline interviews, the 18-month interviews with sampled children will average 76 minutes, depending on age.

Special Sensitivities Necessary for Children in this Study. We provide field representatives with special training on balancing the needs of data collection (e.g., keeping the child focused and on task and remaining emotionally detached and unbiased) with the needs of respondents (e.g., processing emotions evoked by the interview questions and feeling respected and supported) and their own needs (e.g., being confident in their ability to deal with the survey topics and specific sensitive questions and displaying that confidence without threatening or coercing respondents.) In our NSCAW I and II experience to date, a respondent becoming distressed is a rare event, but we provide field representatives with training on what to do if they encounter such a situation. Lists of local mental health resources are provided to all adult respondents at the close of their interview.

We also provide training on interviewing children and especially children who bring various challenges to the interview setting. The training helps field representatives become more aware of the issues of trust and confidence and of building rapport with children of all ages, and provides them with skills for handling the situation if a child becomes upset during the interview.

The variety and different administration procedures for children of various ages, in addition to the content of the interview, makes this study a challenge for respondents and the field representatives alike. To the extent possible, we will maintain our cadre of field representatives currently working the baseline wave and provide them with Wave 2 training. As needed, we will recruit, hire, and train persons with demonstrated experience with children, and preferably with experience administering assessments. We will recruit from the local pool of school counselors, social workers, pediatric nurses, teachers, and experienced interviewers.

Collection of Data from Teachers

The purpose of the teacher survey is to obtain an independent measure of the child’s academic performance, cognitive abilities, social skills, and relationships with other children. In order to protect the child and parent respondents against their involvement with the child welfare system becoming known, we have created a different name to describe the NSCAW in the teacher survey—the National Teacher’s Survey of Children and Adolescents. The survey of teachers will be implemented through a mailed self-administered instrument (that can also be completed via the web), with promptings of nonrespondents by mail, telephone, and email. The teacher is identified in the adult caregiver interview. Note, however, that teachers are only contacted if the signed authorization form is obtained by the field representative from the legal guardian. This insures that no teacher is contacted for participation without the guardian’s express approval.

B.2.2 Weighting Plan

Overview

These procedures to be used are essentially identical to those used for the NSCAW I CPS sample. Maintaining comparable sample weighting strategies for the two surveys will help ensure that the estimates are also comparable. Like NSCAW I, the NSCAW II sample design presents many significant departures from simple random sampling. First, the NSCAW II sampling design is clustered by county thus introducing correlations in the observations that violate the independence assumptions of simple random sampling. Second, the sample is highly stratified according to child characteristics, placement outside the home, and services received. Third, nonresponse and sample attrition may be related to outcome measures and case characteristics, and therefore cannot be ignored.

Inference to the CW target population derived from the NSCAW II data must consider the complex sample design and incorporate the appropriate sample weights that compensate for departures from simple random sampling. This requirement is routine for large federal demographic surveys.

Details about the weighting procedures and adjustments for non-response can be found in Appendix H.

B.2.3 Analytic Techniques to Be Employed

In Section A.2, the research questions that will be addressed in the NSCAW analysis were described. The analysis of data from a stratified and clustered national sample is necessarily more complex and problematic than data from a sample selected using a simple random sample. Unfortunately, many statistical software packages that are readily accessible by researchers employ analysis techniques that assume simple random sampling. In spite of this, the benefits of using analysis methods that are appropriate for the sample design employed include improved statistical inference and less reliance on untenable assumptions which increases the robustness of the estimates. To support data licensees’ use of the NSCAW I data, the project team has developed a manual that includes guidance on the application of the appropriate weight for the specific analysis, methods for imputing missing data, cross-sectional analysis, longitudinal analysis, and multilevel modeling.

B.3 Methods to Maximize Response Rates

NSCAW’s ability to gain the cooperation of potential respondents and maintain their participation through the subsequent waves of the study is key to the success of this endeavor. In preparation for NSCAW I, we carefully reviewed procedures for obtaining respondent cooperation on a wide array of studies, incorporated the best practices of those studies into our data collection procedures, and adapted procedures for continuous improvement through the follow-up waves. The response rates achieved in NSCAW I are presented in Exhibit B.3-1.

Exhibit B.3-1 NSCAW I CPS Sample Component Response Rates, by Wave and Respondent Type

	Wave 1	Wave 2	Wave 3	Wave 4	Wave 5*
Key Respondent (weighted) (Child if >11 years old, Current Caregiver if < 11 at Wave 1; caregiver or child Waves 2-5)	64.2%	82.4%	86.6%	85.3%	80.2%
Caseworker (unweighted)	92.6%	85.2%	94.0%	97.0%	94.9%
Teacher (unweighted)	69.0%		67.8%	66.4%	83.2%

A caregiver’s and child’s willingness to participate, both initially and long-term, is affected by the combination of circumstances surrounding the nature of each selected case. These situations define whether our approaches for gaining cooperation are the more conventional and standard practices implemented on household-type surveys, or whether more resourceful and persistent actions are required. In cases in which the child remains in the environment of the reported neglect or abuse and with all non-custodial biological parents who retain legal guardianship we face more difficulty in obtaining cooperation from the caregiver and the child. Participant confidentiality is of course emphasized, as is the lack of association between the study and the child welfare agency and any other law enforcement agency. We also rely on obtaining information from the caseworker about the family situation in order to develop an effective approach to the potential respondents. Advance materials also emphasize that participation in the survey is an opportunity to provide information on how the system works based on their experiences. Additional NSCAW I procedures included:

 Field representatives thoroughly trained on NSCAW procedures and on the resources available and the processes to (1) overcome respondent objections, (2) resolve restricted access problems, (3) safely and successfully work in dangerous neighborhoods, and (4) reach difficult-to-contact respondents such as those seldom at home and teenagers

 Advance mailings of a letter on ACF letterhead and a project brochure with information customized for that type of respondent on the frequently asked questions about the study

 Use of tailored letters addressing specific reasons for nonparticipation

 Review and approval of all noninterview cases by the field supervisor

 Sufficient numbers of bilingual interviewers so cases are rarely lost due to a Spanish-language barrier

In addition, extensive analyses for non-response bias were conducted in NSCAW I, which will inform our approaches in NSCAW II. Please see Appendix E for details of those analyses.

B.4 Tests of Procedures

The NSCAW I study and the NSCAW II Wave 1 experience to date are the best test of procedures for preparation for NSCAW II Wave 2. For both NSCAW I sample components combined through the fifth wave of follow-up, we completed 25,292 interviews with caregivers, 20,149 interviews with sampled children, 15,027 interviews with investigative and services caseworkers, and received 7,129 completed mail surveys from teachers. This experience, coupled with commitment to continuous improvement, is the best test.

B.5 Statistical Consultants

Statistical consulting was provided by

J.N.K. Rao, Ph.D.

Mathematics Department

Carleton University

1125 Coloney By Drive

Ottawa, Ontario K1S 5B6

(613) 788-214

References

Asparouhov, T. (2004). Weighting for Unequal Probability of Selection in Multilevel Models. MPlus Web Notes, No. 8 (www.statmodel.com/).

Campbell, D.T. and Kenny, D.A. (1999). A Primer on Regression Artifacts. New York: Guilford.

Chromy, J.R. (1981). “Variance Estimators for a Sequential Sample Selection Procedure,” in Krewski, Platek, and Rao (eds) Current Topics in Survey Sampling, New York, Acedemic Press.

Folsom, R.E. (1991). Exponential and Logistic Weight Adjustments for Sampling and

Nonresponse Error Reduction. Proceedings of the American Statistical Association, Social Statistics Section, pp.197-202.

Folsom, R., Potter, F, Williams, S. (1987). “Notes on a Composite Size Measure for Self-Weighting Samples in Multiple Domains,” Proceedings of the American Statistical Association Survey Methods Research Section, pp. 792-796.

Lanza, S.T., Flaherty, B.P., and Collins, L.M. (2003). Latent Class and Latent Transition Analysis. In Schinka, J.A. and Velicer, W.F. (Eds). Handbook of Psychology: Research Methods in Psychology, Vol. 2. (pp. 663-685). New York: John Wiley & Sons, Inc.

Lessler, J.T. & O’Reilly, J.M. (1997). “Mode of Interview and Reporting on Sensitive Issues: Design and Implementation of Audio-Computer Assited Self-Interviewing.” In The Validity of Self-Reported Drug Use: Improving the Accuracy of Survey Estimates, a monograph to be published by the National Institute on Drug Abuse, 366-382.

McLaws, Mary-Louise; Oldenburg, Brian; Ross, Michael W., and Cooper, David A. Sexual Behaviour in AIDS-Related Research: Reliability and Validity of Recall and Diary Measures. The Journal of Sex Research. 1990 May; 27(2):265-281.

Mosher, William et al. CAPI, Event Histories, and incentives in the NSFG Cycle 5 Pretest. In: American Statistical Association (editor), 1994 Proceedings of the Section on Survey Research methods. Alexandria, VA. Pages 59‑63. 1994.

Pfeffermann, D., Skinner, C., Holmes, D. , Goldstein, H. and Rasbash, J. (1998). “Weighting for Unequal Selection Probabilities in Multilevel Models.” Journal of the Royal Statistical Society, B 60, 23-56.

Turner, C.F., Ku L., Rogers, S.M., Lindberg, L.D., Pleck, J.H., & Sonenstein, F.L. Adolescent Sexual Behavior, Drug Ause, and Violence: Incresed Reporting with Computer Survey Technology. Science, 1998 May 8; Vol. 280:867-873.

Von Thurn, D.R., Moore, J.C., & Martin, E.A. (1993). National Health Interview Survey Redesign: An Anthropological Investigation of Mental Health Concepts. Proceedings of the American Statistical Association, Section on Survey Research Methods, 571-576.

Wald, M. (1975). Child development and public policy: Juvenile justice. Transcription of speech presented to the Society for Research in Child Development Panel Symposium: Denver, CO.

.

1 This sample size does not include 600 additional cases that were added to one state as part of a special supplemental sample funded by that state. Although these supplemental cases will improve the precision of the national estimates to some extent, they were designed specifically to boost the precision of the estimates for this state. Therefore, these cases are not included in the Core sample size discussions or power calculations. For this reason, our estimates of precision and power are somewhat understated.