OMB-ACF Q&A August 2011

OMB-ACF Questions & Answers – August 2011

NSCAW-II

1. First, with regards to the bias analysis, please confirm whether the Agency intends to repeat the bias analysis in the next round.  If so, please update (and upload) the supporting statement to reflect this intention.

ACF Response: Yes, the analysis is conducted following each wave. We have updated the supporting statement to reflect this.

2. Second, with regards to the confidentiality statement, while we certainly understand that this language has been in the survey for a long time, OMB policy towards confidentiality statements has evolved over the last decade.  As a result, we now require language in line with our suggestions, unless there is compelling evidence of a specific need to deviate.

ACF Response: While we understand OMB’s concerns, this language has been carefully negotiated not only with OMB, but also with the IRB for the contractor as well as by multiple state and local IRBs. Although the changes seem relatively minor, there are a number of considerations that make us extremely hesitant to reopen these discussions. First, we believe that it is important to be able to have uniform language in all of the materials that will be read by respondents; getting agreement across multiple parties has been an extremely laborious process, and the refusal by a single agency to incorporate any or all of the changes could result in variations that would introduce error into our data in unknown ways. Second, the timing of IRB decisions is beyond our control; many of the local IRBs meet infrequently, and the process of obtaining approvals can be quite slow. Significant delays to our schedule would adversely affect the quality of the data, which is meant to represent a specific time period after entry into the child welfare system. Finally, our relationships with state and local child welfare agencies are crucial to the success of the project, and reopening any issue with the agencies, including IRB reviews, results in additional burden to agencies that can jeopardize those relationships. In short, although the changes in language suggested seem small, we believe that the risks incurred in pursuing those changes would outweigh any benefits.

-----------------------------------------------------------------------------------------

Responses to 7/14/2011 OMB Questions

National Survey of Child and Adolescent Well-Being

Questions raised as “comments” in the supporting statement:

B1 (p.6). Besides changes in kinship care, are there other significant changes to the NSCAW model that have been made?

The reference to kinship care is given as an example of a national policy issue that has taken on more prominence since the first cohort of NSCAW was fielded, rather than a change to the conceptual model that guides NSCAW data collection. The ecological framework (Exhibit A1-1) is designed to allow for shifting policy emphases.

B2 (p. 9). Was there a substantive non-response bias study conducted at the end of the baseline survey, and if so, what were the results?

The nonresponse analysis included in the NSCAW Wave 1 (Baseline) Data File User’s Manual is included at the end of this document. The findings indicate that the adjustments made during weighting reduce potential bias due to incomplete coverage and nonresponse.

B3 (p. 19). If there were some changes in the overall model (e.g. adding kinship care), why did the Agency decide not to make changes to model constructs here?

Please see the response to B1.

B4 (p. 20). Has ACF conducted any studies to evaluate the data quality and accuracy of the AFCARS and other tracking systems?

AFCARS and NCANDS includes data elements that are used for the Child and Family Service Reviews, as well as for Title IV-E eligibility reviews, allocations to states for the Chafee program, and other policy decisions; the Children’s Bureau monitors data quality for AFCARS on an ongoing basis. NCANDS data also contribute to CFSR; they are collected by the states on a voluntary basis, but the Children’s Bureau provides extensive technical assistance through a contract to promote data quality.

B5 (p. 25). Clarify to state that “some” teachers’ unions require token payment for survey participation, and perhaps add addition information about why compensation is appropriate (e.g. requires teachers to stay past working hours, etc.).

This has been changed in the supporting statement.

B6 (p. 26). As Certificate has already been obtained, change to “has obtained”

This has been changed in the supporting statement

B7 (p. 29). Please correct data in ROCIS to ensure that it equals 15,872 (or the number of respondents).

This has been corrected in the supporting statement.

B8 (p. 31). Recommend adding (1) a paragraph about the public use plan or the licensing arrangement processing (2) a commitment to making the results publically available.

This has been added to the supporting statement.

Additional questions in the email:

1. In the enclosed participant letter, in the second to last paragraph, please either change the phrase “as required by law” to “as permitted/authorized by law” or drop the sentence.  We have some concerns that this phrase implies researchers will be unable to release information, rather than stating that they have the authority to not release information. 

Similarly, in the FAQ in the “confidentiality” section (pp. 12 of appendix D), please change “All staff members have signed a confidentially agreement guaranteeing…” to “assuring.”

The language in this sentence was very carefully negotiated between OMB and the various relevant IRBs in 1999, just prior to the first study’s clearance and baseline data collection. We are reluctant to change any wording in this sentence or in any of the letters or materials because of the delicate negotiations and the high level of scrutiny each has received over the dozen years of the study, both by OMB, by the contractor’s institutional IRB, and by a dozen IRB committees across the country.

2. Teacher Survey Instrument:

On page 15, for the question that currently asks “How is the student classified? What is the PRIMARY special education handicapping code?,” we would recommend changing the language to read “what is the child’s PRIMARY disability code.”  The recommended language corresponds with Department of Education recommended language. 

In addition, the survey states that IEP stands for “individual education plan,” when it should be stated as “individualized education program.” 

These changes will be incorporated into the teacher questionnaire (attached).

More generally, we were curious as to whether ACF has consulted with the Department of Education on this survey, and we would be happy to facilitate such a conversation if the Agency has not done so (either now or for future updates). 

The Department of Education National Center for Education Statistics Elementary and Secondary Education branch was represented in initial study review and design discussions. The ED’s Office of Special Education Programs has been represented in review of materials and stakeholder meetings and discussions by various staff over the years of the project. We have invited Department of Education staff to meetings of our technical workgroup, and will continue to do so in the future.

Cont. from question B2:

Analysis of Unit Nonresponse and Undercoverage Bias in the NSCAW II Wave 1

An investigation has been conducted in order to provide information on the extent of the bias arising from unit nonresponsethe failure to obtain an interview from a NSCAW II sample member. An estimate of the nonresponse bias is the difference between the sample estimate (based only on respondents) and a version of the sample estimate based upon respondents and nonrespondents. In the NSCAW II, a limited amount of frame information is available for sample children who did not respond to the survey. Thus, it is possible to compare nonrespondents and respondents for some characteristics in order to investigate the potential nonresponse bias in the NSCAW II results. There is also bias in NSCAW II due to frame undercoverage; in particular, unsubstantiated cases were not included on the sampling frame in a few large states, and an adjustment was made to account for this.

In the remainder of this section, we briefly summarize the results of an investigation of the bias in the NSCAW II results due to nonresponse using the data on nonrespondents available from the frame. Unlike NSCAW I, caseworker data was not collected for nonresponding sample members at NSCAW II and therefore caseworker data was not used in this analysis. We also estimated the bias in the results by comparing respondents to the population represented by NCANDS data, to determine if the adjustments were successful in reducing bias due to frame undercoverage.

Three measures were used to examine the impact and magnitude of the nonresponse bias – the bias, the relative bias, and Cohen’s effect size. These measure were examined for a variety of characteristics to see if bias was reduced as a result of the nonresponse adjustments that were made to the weights. Cohen’s effect sizes were also used to examine the magnitude of the nonresponse bias for these same characteristics.

This investigation was conducted in two steps. First, estimates for characteristics available on the sample frame (sampling strata, sampling domain, region, urbanicity, receipt of services, substantiated or unsubstantiated outcome of the case, foster care, child’s age, child’s gender, child’s race, and child’s hispanic origin) were compared for respondents and nonrespondents, using the base weight and the nonresponse adjusted weight to see if the nonresponse adjustment was successful in reducing the bias due to nonresponse for the sample.

Next, estimates for characteristics available for the inferential population of NSCAW II that were obtained from the NCANDS data were compared to weighted NSCAW II estimates using the final coverage adjusted weight to see if the coverage adjustment was successful in reducing the bias due to frame undercoverage. Characteristics used were sampling strata size, urbanicity, and substantiated or unsubstantiated outcome of the case; fewer characteristics were available than for the sample because not all states report all variables to NCANDS.

7.3.2.1 Estimated Bias and Relative Bias

Using the data available from the frame for sample members at Wave 1, we estimated the bias due to using only the data for those with a key respondent interview. Let denote the true average of the characteristic C based upon the entire target population; i.e., is the average value of C that we would estimate if we conducted a complete census of the target population. Thus, is the target parameter that we intend to estimate with . Then bias in as an estimate of is simply the difference between the two, viz.,

(1)

The bias can be estimated as follows. Let denote the estimate of the average value of C for the unit nonrespondents in the sample; i.e., is a computed as but over the nonrespondents in the sample rather than the respondents. For example, we may have information on the characteristic C that is measured in the child interview from the sampling frame. If that is true, then can be computed. From this, we can form an estimate of using the following formula:

(2)

where is the unit nonresponse rate for the interview corresponding to the characteristic C. Thus, an estimator of the bias in is obtained by substituting in (2) for in (1). This results in the following estimator

(3)

or, equivalently,

(4)

That is, the estimator of the nonresponse bias for C is equal to the nonresponse rate for the interview that collects C times the difference in the average of C for respondents and nonrespondents.

We estimated these means and their standard errors using the weights and accounting for the survey design, as described in Section 7.1. We estimated using the unadjusted base weight. We estimated the mean for respondents, , in two ways: (1) using the unadjusted base weight, and (2) using the nonresponse adjusted weight or the final analysis weight. This allowed us to see if the bias was reduced by applying the nonresponse and post-stratification adjustments to the weights.

We first tested the null hypothesis that the bias is 0 with α=0.05, i.e., HO: Bias=0. We used a t-statistic for the test, and Taylor series linearization to estimate the standard errors. Variables with fewer than 20 cases in the denominators of the proportions or means were excluded from the analyses. We examined the variables with significant bias. The biases, while statistically significant due to the large NSCAW sample size, were generally small and not practically significant. For this reason, we also tested a hypothesis of practical significance. We tested that the relative bias is small, specifically, we tested the null hypothesis H_O: |Relative Bias|<5 percent, where the relative bias is calculated as 100*Bias/ .

Exhibit 7-1 shows whether the null hypothesis was rejected at α =0.05, using the base weight. Exhibit 7-2 shows whether the null hypothesis was rejected at α =0.05, using the nonresponse adjusted weight. Variables showing practically significant bias due to nonresponse (compared to the sampling frame) in the NSCAW II sample were sampling domain (foster care less than one year old and no foster care receiving services and less than one year old), children in foster care, child’s age (3 months to 1 year old, and 12 years old to 17.5 years old), and Hispanicity (missing information). After the nonresponse adjustment, none of the variables show practically significant bias compared to the sampling frame.

7.3.2.2 Cohen’s Effect Size

Cohen’s effect size (Cohen, 1988, Section 7.2) was also used as a measure of the magnitude of the bias. For a variable with K categories, Cohen’s effect size (CES) for the variable is computed

where is the estimated proportion of respondents that are in category i, and is the estimated proportion of all sample members that are in category i.

The effect sizes were computed and examined for the characteristics using the distributions computed with the base weight and the nonresponse adjusted analysis weight. In this analysis, the analysis weights are considered to do an adequate job of reducing the bias due to nonresponse if they reduce the number of significant or large biases in the data.

Following Cohen’s recommendation, the magnitude of the effect for a variable was classified as:

• Small, if CES < 0.2

• Medium, if 0.2 <= CES <= 0.8, and

• Large, if CES > 0.8.

This “rule of thumb” was used to identify survey items for which the biasing effects of nonresponse would be considered medium or large using a well-known standard for such judgments like the CES. However, a shortcoming of this approach is that a bias may be classified as medium or large while the practical implications of the bias may still be small or even trivial. For example, a large bias by the CES scheme may be considered to be practically insignificant if its effect analysis and decision making is unimportant. Likewise, a bias may be classified as small by the CES scheme while its practical implications are quite important for analysis and decision making. In such cases, it may be more relevant to consider the relative bias. We have somewhat arbitrarily used 5 percent or more as the level at which the relative bias may be considered as practically significant.

The results of this analysis are also shown in Exhibit 7-1 and Exhibit 7-2. Using both the base weight and the nonresponse-adjusted weight, all of the characteristics have “small” values of the CES when compared to the sampling frame.

Exhibit 7-3 compares the values for the target population of NSCAW II as obtained from the 2006 and 2007 NCANDS data with the estimates obtained using the NSCAW II base weights, nonresponse adjusted weights, and the final coverage adjusted analysis weights. As described earlier in this chapter, the NSCAW II weights were adjusted to NCANDS totals (adjusted for the observed NSCAW II eligibility) in order to compensate for cases missing from the sampling frame, especially unsubstantiated cases in a few large states. Fewer variables are used in Exhibit 7-3 compared to Exhibit 7-1 and Exhibit 7-2 because many of the states do not report to NCANDS by all of the variables. Exhibit 7-3 shows that the relative bias (compared to the NCANDS distributions) is reduced by the coverage adjustment. Using the base weight and the nonresponse adjusted weight, the bias is statistically significant for the substantiated and unsubstantiated categories, and the absolute relative bias is statistically greater than 5% for the substantiated category when using the base weight. None of the categories have statistically significant bias or absolute relative bias greater than 5% when using the final coverage-adjusted weight. All of the values of Cohen’s effect size are considered small.

The unit nonresponse analysis suggests that the nonresponse and coverage adjustments applied to the analysis weights reduce potential nonresponse bias.

Exhibit 7-1. Distributions of Demographic Characteristics Before Nonresponse Weight Adjustments, Wave I of NSCAW II

	Using Base Weight (Prior to Nonresponse Adjustment)
	Total eligible sample		Respondents		Nonrespondents
Characteristics	Number	Percentage	Number	Percentage	Number	Percentage	Difference between total and respondents		Absolute relative difference		Cohen's effect size
Size of state/sampling stratum											0.0449	Small
Largest states	6,050	54.3	3,607	52.0	2,443	57.1	-2.3		4.1
Remainder	3,559	45.7	2,266	48.0	1,293	42.9	2.3		4.9
Sampling domain											0.0516	Small
No foster care, no services, all ages	2,552	67.6	1,422	66.3	1,130	69.2	-1.3		1.9
Foster care, < 1yr, services & no services	1,639	1.3	1,158	1.7	481	0.8	0.4	*	29.9	*
No foster care, services, <1 yr	1,741	2.4	1,105	2.8	636	1.9	0.4	*	14.8	*
Foster care, 1-18, services & no services	1,935	5.5	1,144	6.0	791	4.9	0.5	*	8.7
No foster care, services, 1-18	1,577	21.5	949	21.4	628	21.8	-0.2		0.9
Missing	165	1.6	95	1.8	70	1.4	0.2		13.1
Region											0.0345	Small
Northeast	888	7.9	582	8.9	306	6.8	1.0	*	11.7
Central	2,752	19.4	1,610	19.3	1,142	19.7	-0.1		0.9
South	3,650	48.1	2,297	47.7	1,353	48.5	-0.4		0.7
West	2,319	24.6	1,384	24.2	935	25.1	-0.4		1.7
Urbanicity of PSU											0.0365	Small
Urban	8,577	82.3	5,203	80.9	3,374	84.1	-1.4		1.7
NonUrban	1,032	17.7	670	19.1	362	15.9	1.4		7.9
Services (Frame)											0.0278	Small
Receiving services	6,520	29.9	4,112	30.9	2,408	28.7	1.0		3.2
Not receiving services	2,895	68.4	1,644	67.3	1,251	69.9	-1.1		1.7
Missing	194	1.6	117	1.9	77	1.4	0.3		13.0

Exhibit 7-1. Distributions of Demographic Characteristics Before Nonresponse Weight Adjustments, Wave I of NSCAW II (Continued)

	Using Base Weight (Prior to Nonresponse Adjustment)
	Total eligible sample		Respondents		Nonrespondents
Substantiated or Unsubstantiated (frame)											0.0456	Small
Substantiated	5,702	30.9	3,642	33.0	2,060	28.2	2.1	*	6.8
Not substantiated	3,907	69.1	2,231	67.0	1,676	71.8	-2.1	*	3.1
Foster care (frame)											0.0357	Small
Foster care	3,574	6.9	2,302	7.7	1,272	5.8	0.8	*	12.8	*
Not in foster care	5,946	92.5	3,518	91.5	2,428	93.6	-1.0	*	1.0
Missing	89	0.7	53	0.7	36	0.6	0.0		8.9
Child's age											0.0794	Small
Infants < 3 months	2,089	4.9	1,409	5.2	680	4.4	0.3		7.4
3 months - < 1yr	1,556	5.0	1,023	6.0	533	3.6	1.0	*	21.9	*
1 - 5 years	2,315	35.5	1,439	35.4	876	35.7	-0.1		0.3
6 - 11 years	2,019	32.1	1,185	33.4	834	30.4	1.3		4.1
12 - 17.5 years	1,630	22.5	817	19.9	813	25.8	-2.6	*	11.8	*
Child's gender											0.0246	Small
Male	4,506	46.8	2,762	47.1	1,744	46.4	0.3		0.7
Female	4,722	49.6	2,898	49.7	1,824	49.4	0.1		0.2
Missing	381	3.6	213	3.2	168	4.2	-0.4		12.6
Child's race											0.0277	Small
Black	2,869	21.7	1,821	21.9	1,048	21.3	0.2		1.2
White	4,758	58.6	2,852	57.9	1,906	59.6	-0.7		1.2
Other	718	6.9	420	6.6	298	7.3	-0.3		4.8
Missing	1,264	12.8	780	13.6	484	11.8	0.8		6.1
Child's Hispanic origin											0.0366	Small
Hispanic	1,987	23.3	1,199	23.0	788	23.7	-0.3		1.4
Non-Hispanic	6,577	67.3	4,015	66.6	2,562	68.3	-0.7		1.1
Missing Hispanicity	1,045	9.3	659	10.4	386	8.0	1.1	*	11.4

Note: * indicates statistical significance at the 0.05 level.

Exhibit 7-2. Distributions of Demographic Characteristics After Nonresponse Weight Adjustments, Wave I of NSCAW II

	Total eligible sample (using base weight)		Using Nonresponse Adjusted Weight
			Respondents		Nonrespondents
Characteristics	Number	Percentage	Number	Percentage	Number	Percentage	Difference between total and respondents		Absolute relative difference				Cohen's effect size
Size of state/sampling stratum													0.0000	Small
Largest states	6,050	54.3	3,607	54.3	2,443	57.1	0.0		0.0
Remainder	3,559	45.7	2,266	45.7	1,293	42.9	0.0		0.0
Sampling domain													0.0000	Small
No foster care, no services, all ages	2,552	67.6	1,422	67.6	1,130	69.2	0.0		0.0
Foster care, < 1yr, services & no services	1,639	1.3	1,158	1.3	481	0.8	0.0		0.0
No foster care, services, <1 yr	1,741	2.4	1,105	2.4	636	1.9	0.0		0.0
Foster care, 1-18, services & no services	1,935	5.5	1,144	5.5	791	4.9	0.0		0.0
No foster care, services, 1-18	1,577	21.5	949	21.5	628	21.8	0.0		0.0
Missing	165	1.6	95	1.6	70	1.4	0.0		0.0
Region													0.0000	Small
Northeast	888	7.9	582	7.9	306	6.8	0.0		0.0
Central	2,752	19.4	1,610	19.4	1,142	19.7	0.0		0.0
South	3,650	48.1	2,297	48.1	1,353	48.5	0.0		0.0
West	2,319	24.6	1,384	24.6	935	25.1	0.0		0.0
Urbanicity of PSU													0.0000	Small
Urban	8,577	82.3	5,203	82.3	3,374	84.1	0.0		0.0
NonUrban	1,032	17.7	670	17.7	362	15.9	0.0		0.0
Services (Frame)													0.0009	Small
Receiving services	6,520	29.9	4,112	30.0	2,408	28.7	0.1		0.1
Not receiving services	2,895	68.4	1,644	68.4	1,251	69.9	0.0		0.1
Missing	194	1.6	117	1.6	77	1.4	0.0		0.1

Exhibit 7-2. Distributions of Demographic Characteristics After Nonresponse Weight Adjustments, Wave I of NSCAW II (Continued)

	Total eligible sample (using base weight)		Using Nonresponse Adjusted Weight
			Respondents		Nonrespondents
Substantiated or Unsubstantiated (frame)													0.0000	Small
Substantiated	5,702	30.9	3,642	30.9	2,060	28.2	0.0		0.0
Not substantiated	3,907	69.1	2,231	69.1	1,676	71.8	0.0		0.0
Foster care (frame)													0.0075	Small
Foster care	3,574	6.9	2,302	6.9	1,272	5.8	0.0		0.0
Not in foster care	5,946	92.5	3,518	92.5	2,428	93.6	0.1		0.1
Missing	89	0.7	53	0.6	36	0.6	-0.1		9.0
Child's age													0.0000	Small
Infants < 3 months	2,089	4.9	1,409	4.9	680	4.4	0.0		0.0
3 months - < 1yr	1,556	5.0	1,023	5.0	533	3.6	0.0		0.0
1 - 5 years	2,315	35.5	1,439	35.5	876	35.7	0.0		0.0
6 - 11 years	2,019	32.1	1,185	32.1	834	30.4	0.0		0.0
12 - 17.5 years	1,630	22.5	817	22.5	813	25.8	0.0		0.0
Child's gender													0.0000	Small
Male	4,506	46.8	2,762	46.8	1,744	46.4	0.0		0.0
Female	4,722	49.6	2,898	49.6	1,824	49.4	0.0		0.0
Missing	381	3.6	213	3.6	168	4.2	0.0		0.0
Child's race													0.0000	Small
Black	2,869	21.7	1,821	21.7	1,048	21.3	0.0		0.0
White	4,758	58.6	2,852	58.6	1,906	59.6	0.0		0.0
Other	718	6.9	420	6.9	298	7.3	0.0		0.0
Missing	1,264	12.8	780	12.8	484	11.8	0.0		0.0
Child's Hispanic origin													0.0000	Small
Hispanic	1,987	23.3	1,199	23.3	788	23.7	0.0		0.0
Non-Hispanic	6,577	67.3	4,015	67.3	2,562	68.3	0.0		0.0
Missing Hispanicity	1,045	9.3	659	9.3	386	8.0	0.0		0.0

Note: * indicates statistical significance at the 0.05 level.

Exhibit 7-3. Comparison of the NSCAW II Eligible Population from NCANDS with the Estimates using the NSCAW II Wave 1 Weights

Characteristic	Population percentage (2006 and 2007 NCANDS)	Base weight							Nonresponse adjusted weight
		Percentage	Estimated Bias		Absolute relative bias		Cohen's Effect Size		Percentage	Estimated Bias		Absolute relative bias		Cohen's Effect Size
Size of state/sampling stratum							0.0700	Small						0.0245	Small
Largest states	55.5	52.0	-3.5		6.3				54.3	-1.2		2.2
Remainder	44.5	48.0	3.5		7.8				45.7	1.2		2.8
Urbanicity of PSU							0.1584	Small						0.1900	Small
Urban	74.0	80.9	6.9		9.4				82.3	8.3		11.3
NonUrban	26.0	19.1	-6.9		26.7				17.7	-8.3		32.1
Substantiated or Unsubstantiated (frame)							0.1975	Small						0.1483	Small
Substantiated	24.5	33.0	8.5	*	34.7	*			30.9	6.4	*	26.1	*
Not substantiated	75.5	67.0	-8.5	*	11.3	*			69.1	-6.4	*	8.5

Exhibit 7-3 (continued). Comparison of the NSCAW II Eligible Population from NCANDS with the Estimates using the NSCAW II Wave 1 Weights

Characteristic	Population percentage (2006 and 2007 NCANDS)	Final NSCAW II Wave 1 analysis weight
		Percentage	Estimated Bias		Absolute relative bias		Cohen's effect size
Size of state/sampling stratum							0.0000	Small
Largest states	55.5	55.5	0.0		0.0
Remainder	44.5	44.5	0.0		0.0
Urbanicity of PSU							0.0762	Small
Urban	74.0	77.3	3.3		4.5
NonUrban	26.0	22.7	-3.3		12.8
Substantiated or Unsubstantiated (frame)							0.0000	Small
Substantiated	24.5	24.5	0.0		0.0
Not substantiated	75.5	75.5	0.0		0.0

Note: * indicates statistical significance at the 0.05 level.

National Survey of Child Wave 1 Data File User’s Manual

and Adolescent Well-Being II 7-8 June 2010