Part B NHES Supporting Statement 2011-2012 Field Test

NHES:2011/2012 Field Test

Request for OMB Review

OMB# 1850-0768 v.7

Part B

June 30, 2010

TABLE OF CONTENTS

DESCRIPTION OF STATISTICAL METHODOLOGY 1

B.1 Statistical Design and Estimation 1

B.1.1 Sampling Households 1

B.1.2 Assigning Addresses to Experimental Treatment Groups 3

B.1.3 Within-Household Sampling 9

B.1.4 Expected Yield 9

B.1.5 Estimation Procedures 11

B.2 Survey Procedures 12

B.3 Methods for Maximizing Response Rates 20

B.4 Individuals Responsible for Study Design and Performance 21

References 22

List of Tables

Table Page

3 Number and percent of children ages 5-17 in linguistically isolated households 3

4 Expected numbers of sampled cases: NHES:2011 field test 4

5 Expected numbers of cases by screener mailing group: NHES:2011 field test 5

6 Expected detectable differences for screener experiments: NHES:2011 field test 6

7 Expected detectable differences for topical experiments: NHES:2011 field test 8

8 Percentage of households with eligible children, by sampling domain: ACS 2007 10

9 Expected numbers sampled and expected numbers of completed screeners and topical surveys in the NHES:2011 and NHES:2012 10

List of Exhibits

Exhibit

5 Experimental groups showing screener version assignment during nonresponse follow‑up. 14

6 NHES:2011 field test contact strategies and experimental treatments: screener and topical surveys 16

7 NHES:2011 field test contact strategies and experimental Spanish language treatments: screener and topical surveys 17

PART B. DESCRIPTION OF STATISTICAL METHODOLOGY

B.1 Statistical Design and Estimation

Historically, an important purpose of the National Household Education Surveys Program (NHES) has been to conduct repeated measurements of the same phenomena at different points in time. However, the primary goal of both the 2009 pilot study and the planned 2011 field test is to develop a methodology that will carry the NHES program into the future. Like many other telephone surveys and ongoing periodic survey programs, the NHES has been experiencing declining response rates. Screener response rates for the NHES have declined from above 80 percent in the early 1990s to 53 percent in 2007. Meanwhile, with the increasing prevalence of households having only cellular telephone service, landline telephone coverage rates have declined from about 93 percent of households in early 2004 to 74 percent of households in the second half of 2009 (Blumberg and Luke 2009). Additionally, the standard list-assisted method used for random digit dial (RDD) sampling of landline telephone numbers now fails to cover about 20 percent of landline numbers (Fahimi, Brick, and Kulp 2009).

The NHES:2011 field test and NHES:2012 full-scale collections are address-based samples covering the 50 states and the District of Columbia. The NHES:2011 will be conducted from January through May 2011 and the NHES:2012 will be conducted from January through May 2012. Households will be randomly sampled as described in section B.1.1, and a screening questionnaire will be sent to each sampled household. Demographic information about household members provided on the screener will be used to determine whether anyone is eligible for the Early Childhood Program Participation (ECPP) or Parent and Family Involvement in Education (PFI) surveys. In order to limit respondent burden, regardless of the number of eligible children, no more than one child per household will be sampled for the topical surveys and no more than one topical survey will be administered in a household.

The target population for the ECPP Survey consists of children age 6 or younger (as of December 31, 2010 for the field test and December 31, 2011 for the NHES:2012) who are not yet in kindergarten. The target population for the PFI survey includes children/youth ages 20 or younger who are enrolled in kindergarten through twelfth grade (homeschool version will be administered in 2012 but not in 2011).

B.1.1 Sampling Households

For the NHES:2011 field test, a nationally representative sample of 60,000 addresses will be used. This nationally representative sample of addresses will be drawn in a single stage from a file of residential addresses maintained by a vendor, based on the United States Postal Service (USPS) Computerized Delivery Sequence File (CDSF). To accommodate the use of telephone follow-up as described in section B.2, the samples of addresses will be reverse-matched to landline telephone directories; it is expected based on the NHES:2009 pilot study and other experience with this process that a telephone number will be obtained for about 60 percent of addresses through the reverse matching.

In order to examine particular aspects of the redesign associated with English literacy and the bilingual (English and Spanish) and Spanish instruments¹, the NHES:2011 sampling frame will be stratified by linguistic isolation and presence of a Spanish surname. Addresses in areas with high linguistic isolation and those with Spanish surnames will be oversampled. The most recent ACS tract-level data available at the time of sample selection will be used to identify those areas

Table 3 gives estimates from the 2007ACS of the percent of children ages 5-17 years who live in linguistically isolated households. (A linguistically isolated household is “one in which no member 14 years old and over (1) speaks only English or (2) speaks a non-English language and speaks English ‘very well.’ In other words, all members 14 years old and over have at least some difficulty with English.”) The “ages 5-17” subgroup was used because that was the subgroup for which the ACS estimates have been tabulated and that corresponds most closely to the NHES:2011 target population.

Table 3 shows that 5.7 percent of children ages 5-17 live in linguistically isolated households. Of those, 74.3 percent (4.3 percent of all children ages 5-17) live in linguistically isolated households in which the child speaks Spanish (and it seems reasonable to assume that in virtually all of these, the adult(s) would also speak Spanish). Thus by including a Spanish option for NHES we are potentially improving our coverage by 4.3 percent.

Table 3.  Number and percent of children ages 5-17 in linguistically isolated households

Subgroup	Total	Percent of children ages 5‑17 in linguistically isolated households	Percent of children ages 5‑17
Total children ages 5-17	53,237,254
Children ages 5-17 in linguistically isolated households	3,045,784	100.0%	5.7%
Child speaks only English	156,428	5.1%	0.3%
Child speaks Spanish	2,262,774	74.3%	4.3%
Child speaks other Indo-European languages	230,151	7.6%	0.4%
Child speaks Asian and Pacific Island languages	334,713	11.0%	0.6%
Child speaks other languages	61,718	2.0%	0.1%

SOURCE: 2007 American Community Survey (ACS) 1-Year Estimates (Accessed April 17, 2009.)

The NHES:2012 will utilize similar sampling techniques to the NHES:2011 but will select 198,000 addresses. As in past NHES surveys, the NHES:2012 will oversample black and Hispanic households using Census and frame data. This oversampling is necessary to produce more reliable estimates for subdomains defined by race and ethnicity. As an option, NCES may select a sample that is representative at the state level in order to provide data for tracking the effectiveness of the American Recovery and Reinvestment Act and the Elementary and Secondary Education Act. If this option is exercised, it is possible a larger sample will be required.

B.1.2 Assigning Addresses to Experimental Treatment Groups

As discussed in section B.2, the NHES:2011 field test design will include a number of embedded experiments. These will permit the evaluation of the costs and effectiveness of various aspects of the protocol. The experimental design will involve the random pre-designation of addresses to particular treatment groups, with the allocation to experimental groups done in such a way as to facilitate comparisons among treatment conditions. The twelve experiments are:

1. Use of a prenotification letter before the initial screener questionnaire mailing in a random subsample of 1,500 addresses.

2. Use of varying levels of monetary incentive included with the initial screener mailing.

3. The effect of including a token magnet with the initial screener questionnaire mailing.

4. Testing different versions of the screener.

5. Testing the effect on response of switching the version of the screener mailed to the sampled address for nonresponse follow-up.

6. Comparing the effect on response of two rush delivery methods (USPS priority mail and FedEx²) for the third screener questionnaire mailing (second nonresponse follow-up).

7. Comparing the effect of different first-class carrier envelopes to the use of USPS Priority Mail delivery for the initial topical mailing for the topical survey follow-up.

8. Examining the effect of different levels of a prepaid monetary incentive level among households assigned to receive their initial topical survey by first-class mail.

9. Examining the effect of different levels of monetary incentive at the final nonresponse mail stage (second topical follow-up).

10. Including a split-ballot test of topical instruments to test the efficacy of alternate wordings for selected questions.

11. Examining the effect of randomly assigning addresses from the Spanish language sample group to receive one of two versions of a screener bilingual mail package.

12. Using the separate English and Spanish questionnaire packages to test procedures aimed at evaluating whether there is any effect on both response rates and respondent characteristics associated with including Spanish questionnaires in mailings (in general; i.e., to non-Spanish surname households outside the high density linguistically isolated stratum).

The primary objectives of the field test are methodological in nature: to compare alternative treatments in order to determine the approach to be taken in future NHES studies. The field test will permit assessment of the various methods with respect to response and cooperation rates. The NHES:2012 design will be shaped by taking the best performing groups from each of these experiments. Table 4 shows the expected sample sizes for the NHES:2011 screener and topical surveys, by overall survey type. Table 5 shows the mailing packages for the screener experiments and their proposed sample sizes at the time of the initial mailing. Table 6 shows the expected levels of detectable differences in response rates for the comparisons of interest using a two tailed test with a .05 significance level for the screener experiments. Table 7 shows the expected levels of detectable differences in response rates for the comparisons of interest as related to the topical experiments.

Table 4.  Expected numbers of sampled cases: NHES:2011 field test

	Expected sample size
Screener cases
TOTAL	60,000
Non-Linguistically Isolated/Spanish Surname	40,000
Pilot treatment	5,000
Other treatment	35,000
Linguistically Isolated/Spanish Surname	20,000
Expected topical cases
TOTAL	11,340
ECPP	3,251
Non-Linguistically Isolated	2,167
Linguistically Isolated	1,084
PFI	8,089
Non-Linguistically Isolated	5,392
Linguistically Isolated	2,696

Table 5.  Expected numbers of cases by screener mailing group: NHES:2011 field test

Mailing package	Expected number of cases receiving the particular mailing package
	60,000
First class, $2 cash (initial)	30,000
First class, $5 cash (initial)	30,000
Pilot screenout First class, $2 cash (initial)	5,000 2,500
First class, $5 cash (initial)	2,500
Screenout with name	15,101
First class, $2 cash (initial)	7,551
First class, $5 cash (initial)	7,550
Screenout without name	8,434
First class, $2 cash (initial)	4,217
First class, $5 cash (initial)	4,217
Bilingual screenout	6,667
First class, $2 cash (initial)	3,334
First class, $5 cash (initial)	3,333
Spanish screenout with name	7,932
First class, $2 cash (initial)	3,966
First class, $5 cash (initial)	3,966
Engaging screener with name	8,434
First class, $2 cash (initial)	4,217
First class, $5 cash (initial)	4,217
Engaging screener without name	8,434
First class, $2 cash (initial)	4,217
First class, $5 cash (initial)	4,217

* For the screener, 1,500 households will be getting an English only prenotification letter. Additionally, 1,750 households will be getting a magnet.

NOTE: Mailing packages and sample sizes shown pertain to the initial mailing.

Table 6.  Expected detectable differences for screener experiments: NHES:2011 field test

Screener	Sample size*	Expected Screener response rate for baseline group (%)	Expected detectable difference between groups in Screener response rates	Expected detectable difference between groups in Topical response rates
Screener incentive
$2 cash	27,000	57	0.7%	1.7%
$5 cash	27,000
Screener advance letter
Yes	1,350	57	2.3%	5.4%
No	30,150
Screener magnet
Yes	1,710	57	2.0%	5.1%
No	47,790
Screener questionnaire version
Engaging for initial mailing/ Engaging for follow-up mailings	7,591	57	1.4%	3.2%
Engaging for initial mailing/ Screen-Out for follow-up mailings	7,591
Screen-Out for initial mailing/ Engaging for follow-up mailings	7,591
Screen-Out for initial mailing/ Screen-Out for follow-up mailings	7,591
Screener name
With name	15,181	57	1.0%	2.3%
Without name	15,181
Screener mailing service for 2nd follow-up**
FedEx	15,181	30	1.3%	3.9%
Priority Mail	15,181

See notes at end of table.

Table 6.  Expected detectable differences for screener experiments: NHES:2011 field test—Continued

Screener	Sample size*	Expected Screener response rate for baseline group (%)	Expected detectable difference between groups in Screener response rates	Expected detectable difference between groups in Topical response rates
Screener language test for non-Spanish surname, non-linguistically isolated households
Dual (English and Spanish) for all mailings, bilingual letter	569	52	4.9%	11.8%
English for first mailing, dual (English and Spanish) for followup only, English letter	569
Screener baseline
Screener pilot treatment	4,500	†	†	†
Screener linguistically isolated/Spanish surname
Initial mailing English only	6,000	46	1.8%	4.5%
Initial mailing Bilingual	6,000
Initial mailing Dual	6,000

† Not applicable.

* Sample sizes have been adjusted to account for an estimated 10% ineligibility rate.

** Rates given are 2nd followup completion rates.

NOTE: Unless otherwise indicated, the detectable difference shown is the difference between the two experimental groups. For each experimental treatment, the first group listed is considered the “baseline” group.

Table 7.  Expected detectable differences for topical experiments: NHES:2011 field test

Topical	Sample sizea	Expected Topical response rate for baseline group (%)	Expected detectable difference in Topical response rates
Initial Topical mailing: Standard envelope	3,088	70
Initial Topical mailing: Distinctive envelope	2,928		2.4%b
Initial Topical mailing: Priority mail	1,544		2.8%c
Initial Topical incentive
$0	1,203	70
$5	1,203		4.1%d
$10	1,203		3.9%e
$15	1,203		3.7%f
$20	1,203		3.4%g
Topical incentive in 2nd follow-up mailing*
$5	232	30	8.6%
$15	232
Questionnaire content**
Mainline version	3,843	50	2.9%
Alternate version	1,647
Linguistically Isolated or Spanish Surname
Initially standard envelope, $0	504	60
Initially standard envelope, $5	504		6.8%h
Initially standard envelope, $10	504		6.4%i
Initially standard envelope, $15	504		6.0%j
Initially standard envelope, $20	504		5.4%k
Initially priority mail, $5 2nd followup*	189	30	8.0%
Initially priority mail, $15 2nd followup	189
Initially standard envelope, all incentive groups	2,520	60	2.8%
Initially priority mail	1,260

† Not applicable.

^* Rates given are 2nd followup completion rates.

^** Baseline “rate” and “detectable difference” for this treatment are in terms of item statistics, rather than response rates. Sample sizes and detectable differences shown here are based on ECPP and PFI combined.

^a Sample sizes have been adjusted to account for an estimated 10% ineligibility rate.

^b The detectable difference shown is between the standard and distinctive envelope groups.

^c The detectable difference shown is between the distinctive and Priority mail envelope groups.

^d The detectable difference shown is between the initial topical $0 and $5 incentive groups.

^e The detectable difference shown is between the initial topical $5 and $10 incentive groups.

^f The detectable difference shown is between the initial topical $10 and $15 incentive groups.

^g The detectable difference shown is between the initial topical $15 and $20 incentive groups.

^h The detectable difference shown is between the initial linguistically isolated $0 and $5 incentive groups.

ⁱ The detectable difference shown is between the initial linguistically isolated $5 and $10 incentive groups.

^j The detectable difference shown is between the initial linguistically isolated $10 and $15 incentive groups.

^k The detectable difference shown is between the initial linguistically isolated $15 and $20 incentive groups.

NOTE: Unless otherwise indicated, the detectable difference shown is the difference between the two experimental groups. For each experimental treatment, the first group listed is considered the “baseline” group.

B.1.3 Within-Household Sampling

As noted in section B.1.1, eligible persons within households that have a completed screener will be sampled for the ECPP or PFI topical surveys. One key criterion in the development of the sampling scheme for NHES is minimizing respondent burden. With a mail survey, this is more of a concern than with a CATI instrument, since the customization that is possible with CATI is impossible or impractical with a hard-copy mail instrument. First, the inclusion of multiple topical survey instruments would result in a bulky mailing that would likely depress response rates. Second, the redundancy of some items (e.g., the household characteristics section, and the mother and father sections for children having the same parents) would result in increased respondent burden or the need for potentially complicated instructions to the respondent. As a result, the decision for the NHES:2011 and 2012 is to restrict the number of topical survey interviews to no more than one per household.

Each household will be randomly pre-designated as either an “ECPP household” or a “PFI household.” This pre-designation will come into play only when a household has children in both domains. In any household with a child/children in the eligible population for only one survey (either ECPP or PFI, but not both), one child will be selected in that domain. When a household has children eligible for both surveys, then only children eligible for the pre-designated survey will be sampled, and one child will be selected in that pre-designated domain.

To carry out this sampling scheme, one flag and one random number will be set prior to screening (i.e., at the time the sample of addresses is drawn). As described above, the flag will indicate whether the household is pre-designated as an “ECPP household” or a “PFI household,” should the household have eligible children in both sampling domains. Once the sampling domain for a particular household has been determined, the random number will be used to sample from amongst the eligible children, if the household has more than one child in the sampling domain.

B.1.4 Expected Yield

As described above, the 60,000 addresses in the nationally representative sample will be assigned to various experimental treatment groups, and a key objective of the 2011 field test is to estimate and compare response rates among treatment groups in this multi-mode context. An expected screener response rate of 60 percent is assumed. (The overall screener response rate in the NHES:2009 pilot study was 59 percent). This response rate, in addition to accounting for an estimated 10 percent screener ineligibility rate³ would yield 32,400 completed screeners.

The ECPP and PFI topical surveys will be administered to households with completed screeners that have eligible children. Tabulations of the 2007 American Community Survey (ACS) Public Use Microdata Sample (PUMS) data file showed that about 35 percent of households are expected to have at least one eligible child. Estimates of the percentage of households with eligible children in each sampling domain are given in table 8, as well as the expected number of screened households in the nationally representative sample, based on the distribution of household composition and assuming a total of 32,400 completed screeners.

Table 8.  Percentage of households with eligible children, by sampling domain: ACS 2007

Household composition	Percent of households	Expected number of screened households
Households with no eligible children	65.1	21,092
Households with eligible children	34.9	11,308
Households with at least one child ages 0 through 6 and not yet in kindergarten, and no child enrolled in grades kindergarten through 12	7.1	2,300
Households with at least one child enrolled in grades kindergarten through 12, and no child ages 0 through 6 and not yet in kindergarten	20.8	6,739
Households with at least one child ages 0 through 6 and not yet in kindergarten, and at least one child enrolled in grades kindergarten through 12	7.0	2,268

NOTE: The distribution in this table assumes 32,400 screened households for the NHES:2011 field test. Detail may not sum to totals because of rounding.

SOURCE: U.S. Department of Commerce, U.S. Census Bureau, American Community Survey (ACS), 2007; Public Use Microdata Sample (PUMS) file, accessed December 2, 2008 (independent tabulations).

Table 9 summarizes the expected numbers of completed interviews for the NHES:2011 and NHES:2012.

Table 9.  Expected numbers sampled and expected numbers of completed screeners and topical surveys in the NHES:2011 and NHES:2012

Survey	Expected number sampled NHES:2011	Expected number of completed interviews NHES:2011	Expected number sampled NHES:2012	Expected number of completed interviews NHES:2012
Household screeners	60,000	32,400	198,000	106,920
ECPP	3,251	2,438	10,728	8,046
PFI	8,089	6,067	26,694	20,021

B.1.5 Estimation Procedures

There are no plans to release survey estimates from the NHES:2011 field test, the aim of which is to provide a large-scale methodological evaluation of alternative approaches. To allow for unbiased estimation of response rates, weights that account for differential selection probabilities will be computed. Fully weighted datasets from the NHES:2012 will be available. The estimation weights for the NHES:2012 survey will be formed in stages. The first stage is the creation of a base weight for the household, which is the inverse of the probability of selection of the address. The second stage is a screener non-response adjustment. The third stage is the adjustment of the base weights for households with multiple addresses. The fourth stage is the poststratification adjustment of the weights to Census Bureau estimates of household totals by household demographic characteristics. Variables that may be used include region and presence of children in the household. These household-level weights include nonresponse and undercoverage adjustments. National household-level estimates may be produced using these final, poststratified household weights.

The poststratified household-level weights are the base weights for the person-level weights. For each extended interview, the person-level weights also undergo a series of adjustments. The first stage is the adjustment of these weights for the probability of selecting the person within the household. The second stage is the adjustment of the weights for topical survey nonresponse. The third stage is the raking adjustment of the weights to Census Bureau estimates of the target population. The variables that may be used for raking at the person level include race and ethnicity of the sampled person, household income, home tenure (own/rent/other), region, age, grade of enrollment, gender, family structure (one parent or two parent), and highest education attainment in household. These variables (e.g., family structure) have been shown to be associated with response rates. The final, raked person-level weights include undercoverage adjustments as well as adjustments for nonresponse.

Standard errors of the estimates will be computed using a jackknife replication method. The replication process repeats each stage of estimation separately for each replicate. The replication method is especially useful for obtaining standard errors for complex statistics such as quantiles. The standard errors may be computed using the complex survey data analysis package WesVar Complex Samples Software or other software packages that use replication methods such as Stata, SAS, SUDAAN or the AM software package. Also, PSU and STRATUM variables will be available on the data files for users who wish to use Taylor series linearization to compute standard errors.

B.2 Survey Procedures

This section describes the data collection procedures to be used in the NHES:2011 field test. As noted previously, many experiments have been embedded into each step of the field test. The NHES:2012 procedures will follow the best performing treatment groups wherever budget and time constraints allow. The basic approach is illustrated in exhibit 6, and special procedures to be tested in high-density linguistically isolated (Spanish) areas and for households with Spanish surnames are illustrated in exhibit 7. As discussed in Part A of this clearance submission, the survey instruments include:

• A screener to engage respondents, determine whether eligible persons live in the household, and sample persons for topical surveys;

• The ECPP Survey to be administered to the parents/guardians of children from birth through age 6 who have not yet started kindergarten; and

• The PFI Survey to be administered to the parents/guardians of children enrolled in kindergarten through grade 12 (homeschool version will be administered in 2012 but not in 2011).

Screener Procedures

A small sample of respondents will be sent a prenotification letter approximately one week before the initial screener is mailed. For the remaining respondents, the initial screener contact will be a questionnaire mailing with a monetary incentive. All sampled addresses will receive the initial screener followed by a thank you/reminder postcard. Nonresponding households will be mailed a second screener questionnaire. If households that have been mailed a second screener questionnaire do not respond, a third and final screener will be mailed by a special rush delivery method described below.

Remaining cases that have not responded to any questionnaire mailings and for which there is a telephone number match available (i.e., provided by a vendor) will receive follow-up by telephone contact. Telephone contact will include up to 10 contact attempts (including up to one refusal conversion attempt if necessary) before the case becomes final nonresponse. If response rates do not meet targets in NHES:2012, it is possible that a fourth questionnaire mailing will be sent after telephone nonresponse follow-up. This additional mailing may also be sent to telephone non-match cases. As part of this final mail attempt in NHES:2012, we may experiment with additional incentives. If this approach is necessary, it would be submitted to OMB with further detail during the survey field period.

The NHES:2011 field test includes several experiments to test approaches aimed toward decreasing unit and item nonresponse. These experiments are illustrated in exhibit 5. Experiments 1 through 4 will be included in the initial contact stage of screener data collection procedures, and experiments 5 and 6 will be implemented during screener nonresponse follow-up.

Experiment 1 will test the use of a prenotification letter before the initial screener questionnaire mailing in a random subsample of 1,500 addresses. Due to the potential for numerous mail contacts using a two-phase mail approach, the NHES Technical Review Panel recommended against including a separate prenotification letter in the NHES:2009 pilot study; as a result, a prenotification letter was not included in the contact procedures in the pilot study. The benefit from a prenotification letter is well-established (see, for example, Dillman, Smyth, and Christian 2008), but there is a lack of empirical research on its effect in the context of a two-phase mail methodology with multiple nonresponse mailings. This experiment will explore whether screener response can be positively affected with the use of a prenotification letter in a two-phase mail survey.

Experiment 2 tests the use of varying levels of monetary incentive included with the initial screener mailing. Addresses will be randomly assigned to receive either $2 or $5 with the initial screener mailing. The NHES:2009 pilot study included a $2 monetary incentive in all initial screener mailings and did not include any experimental manipulations of this amount. The test will examine whether any gain in initial or overall screener response can be realized with a $5 initial monetary incentive.

Experiment 3 will examine the effect of including a token magnet with the initial screener questionnaire mailing. A subsample of 750 addresses will be randomly assigned to receive a magnet with the initial screener mailing. The magnet will feature the NHES logo, the URL for the study website, and toll-free number respondents may call for more information. The purpose of this test is to see if the inclusion of a token magnet with the initial screener mailing can increase response by adding weight to the package, thereby encouraging respondents to open the initial mail package. This could increase the salience of the monetary incentive over respondents who do not open the initial screener package.

Experiment 4 will test two versions of the screener. These include a “screen-out” version which seeks to identify households with children and then roster the children and an “engaging” version with substantive (civic involvement and adult education) questions to engage respondents in addition to the child roster. Due to form constraints, the child roster is presented differently across the versions. Within the two different versions of the screeners are the “name” and “no name” versions of enumerating children within the household. The “name” versions ask for the child’s name, nickname, or initials. The “no name” versions omit the request for the child’s name, nickname, or initials. Households will be randomly assigned to a screener version and to the “name” or “no name” enumeration. In addition to the aforementioned screener versions, a subsample of respondents will be randomly assigned to a “pilot” group, which will receive the screen-out version of the screener that was used in the NHES:2009. The pilot version will also include the same letter and procedures that were used in the NHES:2009 and will be excluded from all NHES:2011 screener-level experiments (Experiments 1 through 6). Including a pilot version that follows the same procedures and includes the same materials will provide a control group for comparisons. This will result in five screener versions:

• Engaging with request for child’s name

• Engaging without request for child’s name

• Screen-out with request for child’s name

• Screen-out without request for child’s name

• Pilot screen-out as used in NHES:2009 pilot study

Experiment 5 will be implemented at the first follow-up screener contact. This experiment will test the effect on response of switching the version of the screener mailed to the sampled address for nonresponse follow-up. Based on the screener versions outlined in experiment 4, respondents will be randomly assigned to two conditions. The first group will receive the same screener questionnaire for all nonresponse follow-up mailings as their initial screener mailing. The second group will receive a different version from their initial screener mailing for all nonresponse follow-up mailings. A component of experiment 4 is the manipulation of the request for the names of eligible children in the household. Addresses in experiment 5 assigned to either the name or no name condition will remain in the same assigned condition if they are also assigned to receive a different screener version for nonresponse follow-up. The groups are illustrated in exhibit 5 below:

Exhibit 5.  Experimental groups showing screener version assignment during nonresponse follow‑up

	Initial Mailing	1st Nonresponse follow-up	2nd Nonresponse follow-up
Group 1 (Non-switchers)	Screen-out with name	Screen-out with name	Screen-out with name
	Screen-out without name	Screen-out without name	Screen-out without name
	Engaging with name	Engaging with name	Engaging with name
	Engaging without name	Engaging without name	Engaging without name
Group 2 (Switchers)	Screen-out with name	Engaging with name	Engaging with name
	Screen-out without name	Engaging without name	Engaging without name
	Engaging with name	Screen-out with name	Screen-out with name
	Engaging without name	Screen-out without name	Screen-out without name

Experiment 5 will serve to test two hypotheses. The first hypothesis is that switching the screener questionnaire version will increase response. If the initial version did not motivate response from the sampled household then it is possible that offering a different stimulus (questionnaire) will increase response. The second hypothesis is that the order of the different screener versions is more important in increasing response. There is evidence from the pilot that the shorter screener version (screen-out) elicited higher overall response. Using the screen-out version for nonresponse follow-up may motivate response from households that felt the engaging version was too burdensome. Alternatively, since the screen-out version only requests basic identifying information on children in the household, it may be perceived as less relevant or too sensitive without the additional context of other survey questions. Using the engaging version for nonresponse follow-up may motivate response from households who did not feel the screen-out version was relevant.

Experiment 6 will compare the effect on response of two rush delivery methods for the third screener questionnaire mailing (second nonresponse follow-up). Nonresponse cases at this stage will be randomly assigned to either USPS priority mail delivery or FedEx delivery.

Topical Survey Procedures

Topical survey mailings will follow procedures similar to the screener procedures. Only households identified through a completed returned screener to have a child eligible for a topical survey will be mailed a topical survey. For the NHES:2011, the initial topical mailing will include either a monetary incentive or the use of USPS Priority Mail delivery without an incentive. Following the initial topical mailing, all households will receive a thank you/reminder postcard. Nonresponding households will be mailed a second topical questionnaire. If households that have been mailed a second topical questionnaire do not respond, a third and final topical will be mailed either by first-class mail with a monetary incentive or by USPS Priority Mail (with no incentive), as described below.

Remaining cases that have not responded to any topical questionnaire mailing and for which there is a telephone number available (either from the vendor match or from the completed screener) will receive follow-up by telephone. The telephone effort will include up to 5 contact attempts (with no refusal conversion) before the case becomes final nonresponse.

In addition to the screener experiments (Experiments 1 through 6) described above, the NHES:2011 field test includes several experiments to test approaches aimed at decreasing unit nonresponse to the topical survey. These experiments are illustrated in exhibit 6.

Exhibit 6.  NHES:2011 field test contact strategies and experimental treatments: screener and topical surveys

Exhibit 7.  NHES:2011 field test contact strategies and experimental Spanish language treatments: screener and topical surveys

The NHES:2012 will follow similar procedures utilizing the design elements that prove most successful at boosting response during the NHES:2011.

Experiment 7 will compare the effect of different first-class carrier envelopes to the use of USPS Priority Mail delivery for the initial topical mailing. Households identified from the screener to be eligible for a topical survey will be randomly assigned to either one of two first-class envelope conditions or USPS Priority Mail delivery. This assignment will take into consideration the delivery method of their last screener mailing. Households that were mailed a third screener questionnaire (second follow-up) will not be eligible for USPS priority mail delivery for their initial topical mailing. (The reason for this exclusion is to distinguish the initial topical mailing from the last screener mailing; the third screener questionnaire mailing is conducted by either USPS Priority Mail or FedEx delivery.)

With first-class delivery, there is a question of whether using an envelope that is also used for the screener mailing affects response. One hypothesis is that using the same envelope elicits a positive effect due to respondent recognition of the survey, while another hypothesis suggests that respondents may feel they are receiving another screener mailing and discard the topical mailing without opening. The high topical response rates attained in the NHES:2009 pilot study suggest little support for the latter hypothesis. Testing a different first-class carrier envelope will provide empirical evidence on whether the envelope is a factor in topical response.

Experiment 8 will randomly assign households receiving their initial topical mailing by first-class mail to a prepaid monetary incentive level of: $0, $5, $10, $15, or $20. Based on the substantial increase observed during the pilot study when using $15 (a topical unit response rate of 81 percent, compared to 74 percent with a $5 topical incentive and 70 percent with no topical incentive), it is possible the ceiling effect of increasing incentive amount has not yet been realized and substantial gains can be realized with $20. The NHES:2011 field test has added a $10 condition to examine if the effect observed for $15 can be achieved with $10. Conversely, a $20 incentive condition is included to test whether further increases can be achieved in topical response. Households assigned to USPS Priority Mail for the initial topical mailing (see Experiment 7) will not include a monetary incentive in that mailing.

Experiment 9 will examine the effect of different levels of monetary incentive at the final nonresponse mail stage (second topical follow-up). Households in this group will include only those that did not receive a monetary incentive in their initial topical mailing and instead were assigned to the group to receive their initial mailing by USPS Priority mail. Remaining nonresponding households will be randomly assigned to receive either $5 or $15 in the second follow-up topical mailing.

Experiment 10 will include a split-ballot test of topical instruments to test the efficacy of alternate wordings for selected questions. Households will be randomly assigned to each version. Response patterns will be compared to previous NHES results as well as external surveys to identify the item wording that is yielding the best response. These items will be included in the NHES:2012.

Spanish Screener Experiments

As described in section B.1.1, the NHES:2011 field test will include an oversample of households in a high density linguistically isolated (Spanish) stratum and Spanish surname households. The general procedures for this group will follow the same screener and topical procedures described above. The approach for this group is illustrated in exhibit 7.

Some of the experiments described earlier (Screener Experiments 4 and 5 and topical Experiment 10) will not be included for this sample group. Addresses from this sample group will be randomly assigned to receive one of two versions of a screener bilingual mail package (Experiment 11). Two additional screener versions will be available in this experiment (1) a Spanish translation of the “screen-out” screener with the request for names of eligible children; and (2) a bilingual version that includes both English and Spanish versions of the “screen-out” screener with the request for names of eligible children in one questionnaire. Addresses assigned to receive the Spanish instrument will also receive an English version of the same instrument in their screener mail package; i.e., such households will receive separate English and Spanish questionnaires in the same screener mailing.

A comparison group is included that will be randomly assigned to receive English only versions of the screen-out questionnaire for the initial and first nonresponse follow-up mailing. A random subsample from this group of nonrespondents to the second mailing will be assigned to receive either English-only questionnaire or both English and Spanish questionnaires in their third screener mailing.

While content is essentially the same between the two experimental conditions, the purpose of this experiment is to test different approaches to eliciting response from Spanish-speaking households. Additionally, it is important for Spanish speaking households to be identified as Spanish-speaking so that a Spanish topical survey may be mailed. One hypothesis under consideration is that the use of separate English and Spanish questionnaires will make it easier for Spanish-speaking households to be identified.

Experiment 12 uses the separate English and Spanish questionnaire package, but will test procedures aimed at evaluating whether there is any effect on both response rates and respondent characteristics associated with including Spanish questionnaires in mailings (in general; i.e., to non-Spanish surname households outside the high density linguistically isolated stratum). Results from the pilot study were not conclusive and suggest that response amongst non-Spanish-speaking households may have been depressed with the use of a bilingual screener questionnaire. This experiment will randomly assign a subsample of non-Spanish surname addresses outside the high-density linguistically isolated areas to receive either an English-only initial screener package or both English and Spanish screener questionnaires. Nonrespondents to the initial screener for both groups will receive both English and Spanish screener questionnaires for all nonresponse follow-up

Survey Monitoring

Mail survey returns will be processed upon receipt, and reports from the survey management system will be prepared at least weekly. The reports will be used to continually assess the progress of data collection. Weekly reports of telephone follow-up status will also be produced from the CATI management system to monitor survey progress and inform the case management process.

B.3 Methods for Maximizing Response Rates

The NHES:2011 field test and NHES:2012 designs incorporate a number of features to maximize response rates. This section discusses those features.

Total Design Method/Respondent-Friendly Design. The approach combines the attributes of the least expensive and best methods available, beginning with the least labor intensive mode to a mode requiring increasingly greater amounts of labor. While this places an emphasis on utilization of resources, these procedures create a respondent-friendly approach that uses design attributes, a scheduled sequence of contacts, and survey mode to motivate and encourage survey participation. Surveys that take advantage of respondent-friendly design have demonstrated increases in survey response (Dillman, Smyth, and Christian 2008; Dillman, Sinclair, and Clark, 1993).

Engaging Respondent Interest and Cooperation. The content of respondent letters and frequently asked questions (FAQs) will be focused on communicating the legitimacy and importance of the study. Past experience has shown that the NHES survey topics are salient to most parents. However, questions to engage the interest of all respondents will be tested in the screener to ascertain whether they help to engage the interest and cooperation of those without children. Interviewer training for nonresponse follow-up and interviewing will focus on strategies for communicating the importance and legitimacy of the survey and gaining cooperation.

Nonresponse Follow-up. The data collection protocol includes several stages of nonresponse follow-up. In addition to the numbers of contacts, changes in method (mail, FedEx, and telephone) are designed to capture the attention of potential respondents. In telephone nonresponse follow-up, up to 10 call attempts will be made to complete screeners and up to 5 call attempts will be made for topical interviews.

Flexibility in Scheduling Interviews. In situations where a telephone respondent is unavailable, a call appointment will be entered into the CATI management system with notations on the best time to reach the respondent.

B.4 Individuals Responsible for Study Design and Performance

The persons listed below participated in the study design and are responsible for the collection and analysis of the data.

• Andrew Zukerberg, NCES 202/219-7056

• Christopher Chapman, NCES 202/502-7414

• Jill Montaquila, Westat 301/517-4046

• Michael Brick, Westat 301/294-2004

• Kwang Kim, Westat 301/517-4078

• Douglas Williams, Westat 240/453-2934

References

Blumberg, S.J. and Luke, J.V. (2009). Wireless substitution: Early release of estimates from the National Health Interview Survey, July-December 2009. National Center for Health Statistics. Available from: http://www.cdc.gov/nchs/nhis.htm. May 18, 2010.

Dillman, D.A., Sinclair, M.D., and Clark, J.R. (1993). Effects of questionnaire length, respondent-friendly design, and difficult questions on response rates for occupant-addressed Census mail surveys. Public Opinion Quarterly, 57, 289-304.

Dillman, D.A., Smyth, J.D., and Christian, L.M. (2008). Internet, mail, and mixed mode surveys: The Tailored Design Method. New York: Wiley.

Fahimi M., Kulp D., Brick J.M. (2009). A Reassessment of List-Assisted RDD Methodology. Public Opinion Quarterly, 73(4):751–60.

1 The translations to create the Spanish topical instruments will be done after approval has been obtained for the English instruments.

2 Westat is currently investigating possible rate increases for FedEx for residential delivery and will compare these rates with those of other vendors, such as USPS. At this point we do not have a recommendation to change the vendor.

3 Once a screener mailing for an address is returned as a postmaster return, the address will be coded ineligible.