Non-Substantive Change to the National Survey of Children’s Health Information Collection Request

Non-Substantive Change to the National Survey of Children’s Health Information Collection Request

The Census Bureau requests approval of a non-substantive change to the current approved National Survey of Children’s Health (NSCH) Information Collection Request. Due to lower than expected response rates for the 2016 survey, the NSCH sponsor has requested the Census Bureau expand the current incentive plans to include an additional incentive in a planned mailing of topical questionnaires. Census recommended to the sponsor that we also implement the additional incentive with an experimental design, as was done for the incentivized initial mailings in June 2016.

The initial sample size and completion estimates are included in Table 1. A response rate for participation by web was estimated to be between 42% and 48% based on incentive group, 2015 pretest response rates, and other survey response rates. The mailing design was a ‘web-push’ strategy, which attempts to generate as much response as possible by web before switching to a more costly paper screener and paper topical strategy. The paper screener response rates were estimated to be 25% of the remaining web non-response, for an expected total screener response of 58.8%.

Table 1: Expected Sample Sizes of 2016 NSCH Incentive Treatment Groups

Initial Sample	Stratum Stratum I: 61.2% Stratum II: 38.8%		Incentive Group $0: 33% $2: 33% $5: 33%		Valid 89%	Screeners Web ($0): 42% Web ($2): 45% Web ($5): 48% Paper: (1-Web)*.25		Households With Kids Stratum I: 77.1% Stratum II: 10.2%		Completed Topicals Web: 95% Paper: 25%
						Web:	Paper:	Web:	Paper:	Web:	Paper:
364,153	I:	222,751	$0	74,251	66,083	27,755	9,582	21,394	7,386	20,324	1,846
			$2	74,250	66,083	29,737	9,086	22,921	7,004	21,775	1,751
			$5	74,250	66,083	31,720	8,591	24,449	6,622	23,227	1,655
	II:	141,402	$0	47,134	41,949	17,619	6,083	1,795	620	1,705	155
			$2	47,134	41,949	18,877	5,768	1,923	588	1,827	147
			$5	47,134	41,949	20,136	5,453	2,051	556	1,949	139
Totals					324,096		190,406		97,307	76,500
per State					6,355		3,733		1,908	1,500

The incentives used in June 2016 were included with the first web-invite introductory letter mailed to sampled households and were split into thirds with $0, $2, and $5 cash monetary incentives offered.  There have been no further incentives offered.

The initial invitation letter began as a mail out on June 10, 2016 to all mailable addresses in the sample. Table 2 below details the number of households in the final sample by initial incentive treatment.

Addresses that received an incentive were more likely to respond in the first month (see “As of July 16”), with the size of the effect proportional to the size of the incentive. Once the incentive’s initial effect was observed, the remaining non-responding households did not show a continued impact from the incentive treatment (see “Since July 16”).

Table 2. NSCH Response Rates – June 2016 Incentive

		Incentive
	Total	$0	$2	$5
Sample	364150	121346	121420	121384
Total Screener Returns	129755	40149	43599	46007
% of sample	35.63%	33.09%	35.91%	37.90%
As of July 16	38970	8679	13286	17005
% of sample	10.70%	7.15%	10.94%	14.01%
Since July 16	90785	31470	30313	29002
% of remaining	27.92%	27.93%	28.03%	27.79%
% Web Topical Returns	10.56%	8.90%	10.55%	12.22%
% Total Topical Returns	11.54%	9.95%	11.51%	13.16%

With current daily web screener return rates indicating that we should expect limited further progress by web, and with the current percent returned between 33% and 38%, significantly below the expected 57% to 61%, the sponsor of the survey has asked for options to incentivize respondents to return more paper topical interviews. Together, we are focusing on procedures that could both help reduce the effect of high non-response, and provide critical information for planning the 2017 NSCH.

In the 2015 pretest, a $10 incentive was used to improve response and attempt to reduce bias in the completion of the paper topical interview. We proposed following that procedure, and testing incentives at the topical stage to see if $10 (and lesser amounts) could be effective in increasing the response to the topical interview. The third topical paper mailing was the next available questionnaire mail-out. We are proposing to identify the response benefit associated with including $0, $2, $5, and $10 cash monetary incentives and a Thank You note in the material being sent to respondents in the third mailing. That would require the use of the remaining available incentive funds of about $91,795, but is expected to increase response (estimates in Table 4), and provide valuable information for us to use in planning the 2017 NSCH to develop and refine a cost-effective data collection strategy. There is no increase in the respondent burden, as this was a planned mailing. The only change to this mailing is adding the monetary incentive. Below is a table of the expected mail-out distribution.  This distribution produces power estimates (see Table 3 below) above .8 for all comparisons.

Table 3. Expected NSCH Mail-Out – Incentivized Topical Questionnaire

	Total	First time topical recipients	Second and Third time topical recipients
Total estimated mailing	19,369	9,417	9,952
Control Group (0$)	2,274	1,105	1,169
$2 Group	6,620	3,219	3,401
$5 Group	5,239	2,548	2,691
$10 Group	5,236	2,545	2,691

Initial estimates below in Table 4 are based on the mail-out in Table 3, with 19,369 topical questionnaires.  The distributions above have been increased for the $5 and $10 groups to improve their power.  The estimated response rates below in Table 4 are our most conservative estimates and trying to detect a very small difference (3% point) between groups.

Table 4. Estimated NSCH Response Rates – Incentivized Topical Questionnaire

 

	Expected Response Rate to third topical mailing, no incentive (%)	Expected Response Rate Difference due to Incentive (%)	Expected Response Rate (%)	Estimated Number of Mailings	Estimated Additional Topicals Received Above a 25.2% Response
Total			--	19,369
Control Group (0$)	25.2	0	25.2	2,274	0
$2 Group	25.2	3.0	28.2	6,620	199 (+11.9%)
$5 Group	25.2	6.0	31.2	5,239	314 (+23.8%)
$10 Group	25.2	9.0	34.2	5,236	471 (+35.7%)

Table 5. Power Calculations

	p=0.05	p=0.10
Control ($0) vs $2	0.791*	0.870
Control ($0) vs $5	≈1.000	≈1.000**
Control ($0) vs $10	≈1.000	≈1.000
$2 vs $5	0.943	0.971
$2 vs $10	≈1.000	≈1.000
$5 vs $10	0.902	0.946

 

 

 

* With 2,274 households receiving $0 incentive and 6,620 households receiving $2 incentive, there will be a 79% chance of detecting a significant difference.

Stated differently, a power of 0.791 means that 79% of the time a significant difference will be detected, with a 5% probability of a false positive result.

** With 2,274 households receiving $0 incentive and 5,239 households receiving $5 incentive, there will be a 100% chance of detecting a significant difference.