Part B NHES2009 Pilot 2009-05-22 Supporting Statement

NHES:2009 Pilot Test

Request for

IMT/OMB Review

Part B

February 20, 2009

TABLE OF CONTENTS

Section Page

PART B. DESCRIPTION OF STATISTICAL METHODOLOGY

B.1. Statistical Design and Estimation 1

B.1.1. Sampling Households 2

B.1.2. Assigning Addresses to Experimental Treatment Groups 3

B.1.3. Within-Household Sampling 5

B.1.4. Expected Yield 5

B.1.5. Estimation Procedures 8

B.2. Survey Procedures 9

B.3. Methods for Maximizing Response Rates 12

B.4. Tests of Procedures and Methods 13

B.5. Individuals Responsible for Study Design and Performance 14

LIST OF TABLES

Table Page

3 Expected numbers of cases by treatment group: NHES:2009 Pilot Test 4

4 Percentage of households with eligible children, by sampling domain: ACS 2007 6

5 Expected number of screened households with eligible children, by sampling domain: NHES:2009 Pilot Test 7

6 Expected numbers sampled and expected numbers of completed Screeners and topical surveys in the NHES:2009 Pilot Test 8

LIST OF EXHIBITS

Exhibit Page

5 NHES:2009 Pilot Test contact strategies and experimental treatments: screener and topical surveys 10

PART B. DESCRIPTION OF STATISTICAL METHODOLOGY

B.1. Statistical Design and Estimation

Historically, an important purpose of the National Household Education Surveys Program (NHES) has been to conduct repeated measurements of the same phenomena at different points in time. However, the 2009 Pilot Test and the planned large-scale Field Test in 2011 both constitute a break from this goal of estimating change over time in key measures. Rather, the primary goal of both the 2009 Pilot Test and the 2011 Field Test is to develop a methodology that will carry the NHES program into the future. Like many other telephone surveys and ongoing periodic survey programs, NHES has been experiencing declining response rates NHES Screener response rates have declined from above 80 percent in the early 1990s to 53 percent in 2007. Meanwhile, with the increasing prevalence of households having only cellular telephone service, landline telephone coverage rates have declined from about 93 percent of households in early 2004 to 80 percent of households in the first half of 2008 (Blumberg and Luke 2008).

The NHES:2009 Pilot Test is an address-based sample covering the 50 states and the District of Columbia. For the NHES 2011 Field Test, an independent random digit dial (RDD) sample, covering both landline and cell phone numbers, is also being considered; however, this is not being tested in the 2009 Pilot Test because there is considerable experience with RDD samples and telephone data collection, both within the NHES program (landline only) and at Westat (with both landline only and landline plus cell).

The NHES:2009 Pilot Test will be conducted from September through December 2009. Households will be randomly sampled as described in section B.1.1, and a screening interview will be administered to an adult household respondent. Demographic information about household members will be used to determine whether anyone is eligible for the Early Childhood Program Participation (ECPP) or Parent and Family Involvement in Education (PFI) Surveys. In order to limit respondent burden, regardless of the number of eligible children, no more than one child per household will be sampled for the topical surveys and no more than one topical survey will be administered in a household.

The ECPP Survey will be administered to a parent or guardian in the household who is knowledgeable about the care and education of the sampled child age 6 or younger (as of December 31, 2009) who is not yet in kindergarten.¹ A parent/guardian who is knowledgeable about the care and education of the sampled child age 20 or younger who is enrolled in kindergarten through twelfth grade will be administered the PFI Survey.

B.1.1. Sampling Households

The NHES:2009 Pilot Test will include only an address-based sample; for the 2011 Field Test, an independent RDD sample, which would allow for estimation of within-study mode effects and will also provide insight as to the current levels of response rates for an RDD NHES administration, is under consideration. For the NHES:2009 Pilot Test, both a nationally representative sample of 11,000 addresses and a supplemental targeted sample of 800 additional addresses will be used.²

The nationally representative sample of addresses will be drawn in a single stage from a file of residential addresses maintained by a vendor, based on the United States Postal Service (USPS) Delivery Sequence File (DSF). In order to examine particular aspects of the redesign associated with English literacy and the bilingual (English and Spanish) instruments³, the sampling frame will be stratified by linguistic isolation and, for addresses in areas with low linguistic isolation, by educational attainment. Addresses in areas with high linguistic isolation and those in areas with low educational attainment will be oversampled.

The targeted sample will be drawn from a list of addresses provided by a vendor that maintains a rich address-level database containing information on characteristics of the addresses. One of the characteristics that will be targeted is the presence of children in the household. Targeting households with children will boost the numbers of completed topical surveys, to allow for examination of particular issues related to the instruments and procedures used for the topical surveys. Another characteristic that may be used to target households is Internet usage, to allow for assessment of whether response propensities and mode preferences are related to Internet usage.

To accommodate the use of telephone follow-up as described in section B.2, the samples of addresses will be reverse-matched to landline telephone directories; it is expected based on other Westat experience with this process that a telephone number will be obtained for about 60 percent of addresses through this reverse matching.

B.1.2. Assigning Addresses to Experimental Treatment Groups

As discussed in section B.2, the design will include a number of embedded experiments. These will permit the evaluation of the costs and effectiveness of various aspects of the protocol, including up to five versions of the Screener, mail versus telephone follow-up, varying incentive levels for the topical survey, and USPS (Priority Mail) versus FedEx⁴ for the topical survey follow-up. The experimental design will involve the random pre-designation of addresses to particular treatment groups, with the allocation to experimental groups done in such a way as to facilitate comparisons among treatment conditions. Although the 800 addresses in the targeted sample will also be randomized to the experimental treatment groups, it is expected that only the nationally representative sample of addresses will be used in analyses of response rates.

The primary objectives of the Pilot Test are to evaluate the operational success of the proposed methods and to get a preliminary assessment of the various methods with respect to response and cooperation rates.

Table 3.  Expected numbers of cases by treatment group: NHES:2009 Pilot Test

Experiment number	Treatment	Expected number of cases receiving the treatment
Overall		11,000
1	Screener-Core version with just the basic items needed to determine eligibility and sample, and for nonresponse adjustment	3,934
	Screener-Engagement version with additional substantive questions to engage the respondent	3,133
	Screener- Screen out version designed to quickly determine household eligibility	3,934
2	Screener-1st follow-up by mail	6,700
	Screener-1st follow-up by phone	1,000
3	Screener-2nd follow-up by FedEx	3,690
	Screener-2nd follow-up by phone	1,000
4	Topical survey $0 prepaid incentive	684
	Topical survey $5 prepaid incentive	684
	Topical survey $15 prepaid incentive	684
5	Topical survey follow-up by USPS	359
	Topical survey follow-up by FedEx	359
6	Topical survey promised incentive for phone group: $0	1,000
	Topical survey promised incentive for phone group: $5	1,000

NOTE: Excludes targeted sample of 800 addresses which will receive a bilingual version of the Core Screener. See section B.1.2 for discussion of each of the experiments and the treatment groups.

B.1.3. Within-Household Sampling

Persons within households that have a completed Screener will be sampled for the ECPP or PFI topical surveys. One key criterion in the development of the sampling scheme for NHES is minimizing respondent burden. With a mail survey, this is more of a concern than with a CATI instrument, since the customization that is possible with CATI is impossible or impractical with a hard-copy mail instrument. First, the inclusion of multiple topical survey instruments would result in a bulky mailing that would likely depress response rates. Second, the redundancy of some items (e.g., the household characteristics section, and the mother and father sections for children having the same parents) would result in increased respondent burden or the need for potentially complicated instructions to the respondent. As a result, the decision for the NHES:2011 Field Test and the 2009 Pilot Test is to restrict the number of topical survey interviews to no more than one per household.

Each household will be pre-designated as either an “ECPP household” or a “PFI household,” with half of the households pre-designated for each survey. This pre-designation will come into play only when a household has children in both domains. In any household with a child/children in the eligible population for only one survey (either ECPP or PFI, but not both), one child will be selected in that domain. When a household has children eligible for both surveys, then only children eligible for the pre-designated survey will be eligible to be sampled, and one child will be selected in that pre-designated domain.

To carry out this sampling scheme, one flag and one random number will be set prior to screening (i.e., at the time the sample of addresses is drawn). As described above, the flag will indicate whether the household is pre-designated as an “ECPP household” or a “PFI household,” should the household have eligible children in both sampling domains. Once the sampling domain for a particular household has been determined, the random number will be used to sample from amongst the eligible children, if the household has more than one child in the sampling domain.

B.1.4. Expected Yield

As described above, the 11,000 addresses in the nationally representative sample will be assigned to various experimental treatment groups, and a key objective of both the 2009 Pilot Test and the 2011 Field Test is to estimate and compare response rates among treatment groups, in this multi-mode context. Although there is considerable uncertainty as to the effectiveness of various treatments, an expected Screener response rate of 60.6 percent is assumed. This response rate would yield 6,664 completed Screeners in the nationally representative sample.

The ECPP and PFI topical surveys will be administered to knowledgeable parents/guardians of a sample of children. Tabulations of the 2007 American Community Survey (ACS) Public Use Microdata Sample (PUMS) data file showed that about 35 percent of households are expected to have at least one eligible child. Estimates of the percentage of households with eligible children in each sampling domain are given in table 4.

Table 4.  Percentage of households with eligible children, by sampling domain: ACS 2007

Household composition	Percent of households
Households with no eligible children	65.1
Households with eligible children	34.9
Households with at least one child ages 0 through 6 and not yet in kindergarten, and no child enrolled in grades kindergarten through 12	7.1
Households with at least one child enrolled in grades kindergarten through 12, and no child ages 0 through 6 and not yet in kindergarten	20.8
Households with at least one child ages 0 through 6 and not yet in kindergarten, and at least child enrolled in grades kindergarten through 12	7.0

SOURCE: U.S. Department of Commerce, U.S. Census Bureau, American Community Survey (ACS), 2007; Public Use Microdata Sample (PUMS) file, accessed December 2, 2008 (independent tabulations).

Table 5 shows the expected number of screened households in the nationally representative sample, based on the distribution of household composition shown in table 4 and assuming a total of 6,664 completed Screeners.

Table 5.  Expected number of screened households with eligible children, by sampling domain: NHES:2009 Pilot Test

Household composition	Expected number of Screened households
Households with no eligible children	4,337
Households with eligible children	2,327
Households with at least one child ages 0 through 6 and not yet in kindergarten, and no child enrolled in grades kindergarten through 12	474
Households with at least one child enrolled in grades kindergarten through 12, and no child ages 0 through 6 and not yet in kindergarten	1,384
Households with at least one child ages 0 through 6 and not yet in kindergarten, and at least child enrolled in grades kindergarten through 12	468

NOTE: The distribution in this table assumes 6,664 screened households for the NHES:2009 Pilot Test. These figures do not include the 800 addresses in the targeted sample. Detail may not sum to totals because of rounding.

SOURCE: U.S. Department of Commerce, U.S. Census Bureau, American Community Survey (ACS), 2007; Public Use Microdata Sample (PUMS) file, accessed December 2, 2008 (independent tabulations).

The experimental design described in section B.2 and the within-household sampling approach described in section B.1.3 will also be applied to the targeted sample of 800 addresses, but these addresses are not expected to be combined with the nationally representative sample for analysis of the experiments. However, some analyses may compare the effectiveness of treatments in households with children (from the targeted list) to households in general (from the nationally representative list). Assuming a Screener response rate of 60.6 percent, it is expected that 485 completed Screeners will be obtained from the targeted sample. Since part of the targeted sample will be targeted to households with children, the proportion of households in the targeted sample with children selected for topical surveys will be higher than that of households in the nationally representative sample of addresses. Of the 485 screened households in the targeted sample, it is expected that 338 will have eligible children, and that a total of 272 topical surveys will be completed for these eligible children.

Table 6 summarizes the expected numbers of completed interviews for the NHES:2009 Pilot Test.

Table 6.  Expected numbers sampled and expected numbers of completed Screeners and topical surveys in the NHES:2009 Pilot Test

Survey	Expected number sampled	Expected number of completed interviews
Household Screeners	11,800	7,149
Nationally representative address sample	11,000	6,664
Targeted sample of addresses	800	485
ECPP	812	653
Nationally representative address sample	709	570
Targeted sample of addresses	103	83
PFI	1,853	1,490
Nationally representative address sample	1,618	1,301
Targeted sample of addresses	235	189

B.1.5. Estimation Procedures

There are no plans to release survey estimates from the NHES:2009 Pilot Test; the aim of the Pilot Test is to obtain a preliminary assessment of the effectiveness of procedures that will be tested in the 2011 Field Test. To allow for unbiased estimation of response rates, weights that account for differential selection probabilities will be computed. Standard errors of the response rate estimates will be computed using a jackknife replication method. The replication process repeats each stage of estimation separately for each replicate. The standard errors may be computed using the complex survey data analysis package WesVar Complex Samples Software or other software packages that use replication methods such as Stata, SAS, SUDAAN or the AM software package.

B.2. Survey Procedures

This section describes the data collection procedures to be used in NHES:2009 Pilot Test. The approach is illustrated in Exhibit 5. As discussed in Part A of this clearance submission, the interviews include:

• A Screener, to engage respondents, determine whether eligible persons live in the household, and sample persons for extended interviews;

• The ECPP Survey, to be administered to the parents/guardians of children from birth through age 6 who have not yet started kindergarten; and

• The PFI Survey, to be administered to the parents/guardians of children enrolled in kindergarten through grade 12 or homeschooled for those grades.

Screener Procedures

The initial Screener contact will be a questionnaire mailing with a $2 prepaid incentive. Multiple versions of the screener will be used (Experiment 1), including an “engagement” version with substantive questions to engage respondents versus a “core-only” version with just the items needed for eligibility, sampling, and nonresponse adjustment; and a screen-out version that quickly determines household eligibility (presence of eligible children). The Screener version will be randomized. Following a thank-you/reminder postcard (sent to all addresses), a subsample of the nonresponding cases with matched telephone numbers will be assigned to receive future Screener contacts by telephone and the remaining nonresponse cases will receive a second screener mailing (Experiment 2).

If cases that receive their second Screener contact by mail do not respond, the third Screener contact will be done either by FedEx or by telephone. Cases with telephone matches will be randomly assigned to either telephone or a FedEx mailing (Experiment 3). For nonmatched cases, the third Screener contact will be by FedEx. For cases completing the Screener by telephone, only the core version of the screener will be administered by phone and the administration of the topical survey will also be by telephone. If a knowledgeable parent/guardian respondent is available at the time the Screener is completed, the topical survey will be administered at that time. Otherwise, the name of the parent/guardian respondent will be collected for future telephone contacts.

Exhibit 5.  NHES:2009 Pilot Test contact strategies and experimental treatments: screener and topical surveys

Cases that do not respond to all other designated Screener contacts (mail or telephone) will be sent a reduced (nonresponse) Screener that collects a few key items that might be useful for weighting and/or nonresponse bias analysis. This reduced instrument will not include any rostering of children, and households that complete the reduced Screener will not be administered a topical survey. Thus, this instrument has a very different function than the regular screening instruments.

Topical Survey Procedures

The first topical survey mailing will include a prepaid incentive, with the amounts ($0, $5, or $15) randomized (Experiment 4). Although $15 might be considered rather high, it is worth testing in the Pilot Test to gauge how much of an effect it might have (and we believe we can interpolate what to expect from other amounts, such as $10).

As noted above, the topical survey will be administered by telephone if the Screener is administered by telephone. If possible, the topical interview will be done in the same telephone call as the Screener. For these cases, which will not receive a topical survey mailing, up to 10 callback attempts will be made to complete the topical survey if the parent/guardian respondent is not available at the time the Screener is completed.

All households designated to receive topical surveys by mail will be sent a thank-you/reminder postcard for the topical survey. For this group (the “topical by mail” path), a second topical survey will be mailed to all nonrespondents. For all households that have not responded after the second topical mailing a third topical survey contact will again be by mail, randomized to either USPS (Priority Mail) or FedEx (Experiment 5). For the “topical by mail” path, the fourth and final topical survey contact will be by telephone, provided a telephone number is available (either from the Screener or through matching). Up to five call attempts will be made.

For topical interviews being administered by telephone, a promised incentive of $5 will be offered verbally to a random 50 percent subsample and will be sent upon completion of the topical survey. After telephone contact has been exhausted and the case remains a nonresponse to the topical questionnaire, a final topical questionnaire can be mailed to these households. Whether this option is exercised may depend upon phone response and the timing of the mailing (optional Experiment 6).

Westat is currently investigating possible rate increases for FedEx for residential delivery and will compare these rates with those of other vendors, such as UPS. It is not clear that another vendor will provide better rates than FedEx and at this point we do not have a recommendation to change the vendor.

Survey Monitoring

Mail survey returns will be processed upon receipt, and reports from the survey management system will be prepared at least weekly. The reports will be used to continually assess the progress of data collection and assign cases to mail or telephone followup according to the sample design and data collection protocol. Weekly reports of telephone followup status will also be produced from the CATI management system to monitor survey progress and inform the case management process.

B.3. Methods for Maximizing Response Rates

The NHES:2009 Pilot Test design incorporates a number of features designed to maximize response rates. This section discusses those features.

Total Design Method/Respondent Friendly Design. The approach combines the attributes of the least expensive and best methods available beginning with the least labor intensive mode to a mode requiring increasingly greater amounts of labor. While this places an emphasis on utilization of resources, these procedures create a respondent friendly approach that uses design attributes, a scheduled sequence of contacts, and survey mode to motivate and encourage survey participation. Surveys that take advantage of respondent friendly design have demonstrated increases in survey response (Dillman, Smyth, and Christian 2008; Dillman, Sinclair, and Clark, 1993).

Engaging Respondent Interest and Cooperation. The content of respondent letters and frequently asked questions (FAQs) will be focused on communicating the legitimacy and importance of the study. Past experience has shown that the NHES survey topics are salient to most parents. However, questions to engage the interest of all respondents will be tested in the Screener to ascertain whether they help to engage the interest and cooperation of those without children. Interviewer training for nonresponse followup and interviewing will focus on strategies for communicating the importance and legitimacy of the survey and gaining cooperation.

Nonresponse Followup. The data collection protocol includes several stages of nonresponse followup. In addition to the numbers of contacts, changes in method (mail, FedEx, and telephone) are designed to capture the attention of potential respondents. In telephone followup, up to 5 call attempts will be made to complete Screeners and up 5 additional call attempts will be made for topical interviews for those cases that receive multiple mailings. For cases assigned to the telephone mode after one Screener mailing (and continuing through the topical survey), up to 10 call attempts will be made.

Flexibility in Scheduling Interviews. Whenever possible, telephone interviewers will attempt to complete the Screener and topical interview (if any household member is selected) during one call. In situations where a respondent is unavailable, a call appointment will be entered into the CATI management system with notations on the best time to reach the respondent.

B.4. Tests of Procedures and Methods

As noted in earlier sections of this document, this request is for clearance of a test of survey procedures and methods. Prior to the NHES:2009 Pilot Test, Westat will conduct cognitive laboratory research, for which a prior request for clearance was submitted.

B.4.1. Cognitive Laboratory Research for NHES:2009

A separate request for OMB clearance of cognitive laboratory research has been submitted. The cognitive laboratory research being conducted for the NHES:2009 Pilot Test will use concurrent think-aloud and retrospective probing techniques in the context of individual cognitive interviews. This research will examine issues including respondent reactions to the survey materials, comprehension of the instrument content and instructions, and navigation of the hard copy instrument. A special focus will be placed on the different versions of the Screener and respondent comprehension and navigation of the various methods for listing household members. Respondent perceptions of the Screener content questions, designed to engage respondents, will also be examined.

The cognitive laboratory research is expected to lead to some refinements in the survey instruments. The revised instruments will be submitted to OMB prior to the Pilot Test.

B.5. Individuals Responsible for Study Design and Performance

Those persons listed below participated in the study design and are responsible for the collection and analysis of the data.

• Christopher Chapman, NCES 202/502-7414

• Andrew Zukerberg, NCES 202/219-7056

• Mary Hagedorn, Westat 301/251-4273

• Jill Montaquila, Westat 301/517-4046

• Michael Brick, Westat 301/294-2004

• Douglas Williams, Westat 240/453-2934

1 Because the proportion of 7-year-olds who are not enrolled in school is very small (about 1.5 percent), an upper age limit of 6 will be used for the ECPP Survey.

2 See section B.1.2 for a discussion of considerations for the analysis involving these two samples of addresses.

3 The translations to create the bilingual instruments will be done after approval has been obtained for the English instruments.

4Westat is currently investigating possible rate increases for FedEx for residential delivery and will compare these rates with those of other vendors, such as UPS. It is not clear that another vendor will provide better rates than FedEx and at this point we do not have a recommendation to change the vendor.