NHES 2015 Part B

NHES:2015

Full-scale Data Collection

Request for OMB Review

OMB# 1850-0768 v.10

Part B

July 8, 2014

TABLE OF CONTENTS

Page

DESCRIPTION OF STATISTICAL METHODOLOGY 1

B.1 Respondent Universe and Statistical Design and Estimation 1

B.1.1 Sampling Households 1

B.1.2 Within-Household Sampling 2

B.1.3 Expected Yield 3

B.1.4 Reliability 5

B.1.5 Estimation Procedures 6

B.1.6 Nonresponse Bias Analysis 6

B.2 Statistical Procedures for Collection of Information 8

B.3 Methods for Maximizing Response Rates 13

B.4 Tests of Procedures and Methods 14

B.5 Individuals Responsible for Study Design and Performance 14

List of Tables

Table Page

1 Expected percentage of households with eligible children or adults, by sampling domain 4

2 Expected numbers sampled and expected numbers of completed screeners and topical surveys for households in Screener Group A in the NHES:2015 4

3 Expected numbers sampled and expected numbers of completed screeners and topical surveys for households in Screener Group B in the NHES:2015 4

4 Numbers of completed topical interviews in previous NHES surveys administrations 5

5 Margins of error for the screeners 5

6 Margins of error for topical surveys in Screener Group A 5

List of Figures

Figure

1 Screener Data Collection 10

2 Screener Internet Data Collection Mailing Plan 11

3 Topical Mailing Plan – ECPP/PFI 12

4 Topical Mailing Plan – CWS/TWS 13

PART B. DESCRIPTION OF STATISTICAL METHODOLOGY

B.1 Respondent Universe and Statistical Design and Estimation

Historically, an important purpose of the National Household Education Surveys Program (NHES) has been to collect repeated measurements of the same phenomena at different points in time. Recently, decreasing response rates required NCES to redesign the NHES. This redesign involved changing the sampling frame from a list-assisted Random Digit Dial (RDD) to an Address-Based Sample (ABS) frame. The mode of data collection has also changed from telephone, interviewer-administered to mail, self-administered.

The NHES:2015 will be an address-based sample covering the 50 states and the District of Columbia and will be conducted from January through August 2015. The household target population is all residential addresses except PO Boxes that are not flagged as the only way to get mail and is estimated at 126,115,692 addresses.

Households will be randomly sampled as described in section B.1.1, and a screening questionnaire will be sent to each sampled household. Demographic information about household members provided on the screener will be used to determine whether anyone is eligible for the Early Childhood Program Participation (ECPP) survey, the Parent and Family Involvement in Education (PFI) survey, the Credentials for Work Survey (CWS), or the Training for Work Survey (TWS). In order to limit respondent burden, regardless of the number of eligible people in a household, no more than one child or one adult per household will be sampled for the topical surveys.

The target population for the ECPP survey consists of children age 6 or younger who are not yet in kindergarten. The target population for the PFI survey includes children/youth ages 20 or younger who are enrolled in kindergarten through twelfth grade or are homeschooled for equivalent grades. The target population for the CWS and TWS surveys includes adults ages 16-65 who are not enrolled in grade 12 or below or homeschooled for equivalent grades. Ages will be calculated based on information provided in the screener.

B.1.1 Sampling Households

A nationally representative sample of 200,000 addresses will be used. This nationally representative sample of addresses will be drawn in a single stage from a file of residential addresses maintained by a vendor, Marketing Systems Group (MSG), based on the United States Postal Service (USPS) Computerized Delivery Sequence File (CDSF). MSG provided the sample for the previous NHES administration, NHES:2012, and for the NHES Feasibility Study (NHES-FS) in 2014. As in past NHES surveys, the NHES:2015 will oversample black and Hispanic households using Census and vendor-appended frame data. This oversampling is necessary to produce more reliable estimates for subdomains defined by race and ethnicity. NHES:2015 will use the stratification methodology that was used for the NHES:2012 administration. The sampling design concentrates on achieving a significant sample of black and Hispanic households. The universe consists of Census tracts likely to contain relatively high proportions of these subgroups. The three strata for the NHES:2015 are defined as households in:

1) Tracts with 25 percent or more Black persons

2) Tracts with 40 percent or more Hispanic persons (and not 25 percent or more Black persons)

3) All other tracts

The Hispanic stratum contains a high concentration of households in which at least one adult age 15 or older speaks Spanish and does not speak English very well and will be used for the assignment of Spanish screener mailing materials. The sample allocation to the three strata will be 20 percent each to the black and Hispanic strata and 60 percent to the third stratum. This allocation will provide improvement in the precision of estimates by race/ethnicity compared to use of a uniform sampling rate across each stratum, and will protect against unknown factors that may affect the estimates for key subgroups, especially differential response rates.

In addition to stratifying by the race/ethnicity groups mentioned above, these three strata will be sorted by a poverty variable and the sample will be selected systematically from the sorted list in order to maintain the true poverty level proportions in the sample. The Census tracts will be used to define the two categories of poverty level as households in:

1. Tracts with 20% or more below the poverty line

2. Tracts with less than 20% below the poverty line

The NHES:2015 will include a self-response Internet screener experiment. After stratifying the sample of 200,000 addresses by the same characteristics defined above, a systematic sample of 15,000 households will be selected to participate in the experiment. These 15,000 households will be referred to as Screener Group B. The 15,000 households will be evenly split into two panels with two different contact strategies. The description of these two strategies is described in Section B.2. This experiment is being conducted to determine if a self-response Internet screener results in an acceptable response rate. If an acceptable response rate is achieved, the self-response Internet screener could be administered in place of the paper screener in the future. The remaining 185,000 households will receive a paper screener, and they will be referred to as Screener Group A.

B.1.2 Within-Household Sampling

One key criterion in the development of the sampling scheme for NHES is minimizing respondent burden. With a mail survey, this is more of a concern than with a CATI instrument, since the customization that is possible with CATI is impossible or impractical with a hard-copy mail instrument. First, the inclusion of multiple topical survey instruments would result in a bulky mailing that would likely depress response rates. Second, the redundancy of some items (e.g., the household characteristics section, and the parent characteristics sections for children having the same parents) would result in increased respondent burden or the need for potentially complicated instructions to the respondent. As a result, the decision for 2015 is to restrict the number of topical survey interviews to no more than one per household, either a child or an adult. In order to administer only one topical survey per eligible household, multiple stages of sampling will be used.

First-Stage Sampling: Age Group (Child or Adult)

The first stage of sampling is at the age group level (child or adult). Each household will be randomly pre-designated as either a “child household,” or an “adult household.” This pre-designation will be referred to as the age group pre-designation. A household will be pre-designated as a “child household” with a .75 probability and an “adult household” with a .25 probability. Because eligible children comprise a smaller portion of the population compared to eligible adults, differential sampling is applied to ensure a sufficient sample size of the children. The pre-designation will only be used when a household has both an eligible child/children and an eligible adult/adults. If a household only has an eligible child/children and no eligible adults, that household will receive a child topical survey, regardless of pre-designation. Likewise, if a household only has an eligible adult/adults and no eligible children, that household will receive an adult topical survey, regardless of pre-designation. (It is possible that households with only eligible adults will be subsampled.) For example, suppose the Doe household has four eligible children and two eligible adults. If this household was pre-designated as a “child household,” a child will be selected. As another example, suppose the Smith household has two eligible adults and no eligible children. Even if this household was pre-designated as a “child household,” an adult will be selected instead since there are no eligible children.

Second-Stage Sampling for Households That Are Selected to Receive a Child Topical Survey: Domain (ECPP or PFI)

The second stage of sampling is at the topical survey level, or the domain level. There are no sampling differences between Screener Group A and Screener Group B for households that are selected to receive a child topical survey. There are two topical surveys that a child could possibly be eligible for: the Early Childhood Program Participation (ECPP) Survey or the Parent and Family Involvement (PFI) Survey. A child’s age, among other factors, will determine if the child is eligible for the ECPP or the PFI topical surveys. The eligibility criterion only allows a child to be eligible for one of the topical surveys, not both.

In addition to the age group pre-designation, each household will be pre-designated as an “ECPP household” or a “PFI household.” This pre-designation will be referred to as the child topical survey pre-designation. A household will be pre-designated as an “ECPP household” with a .70 probability and a “PFI household” with a .30 probability. Because children eligible for the ECPP survey comprise a smaller portion of the population compared to children eligible for the PFI survey, differential sampling is done to ensure a sufficient sample size of the ECPP group. Similar to the age group pre-designation, this designation will only be used when a household has a child/children eligible for the ECPP topical survey and a child/children eligible for the PFI topical survey. If a household only has a child/children eligible for the ECPP topical survey and not the PFI topical survey, that household will receive an ECPP topical survey, regardless of pre-designation. Likewise, if a household only has a child/children eligible for the PFI topical survey and not the ECPP topical survey, that household will receive a PFI topical survey, regardless of pre-designation. Referring to the same family above, suppose the Doe household has two children eligible for the ECPP topical survey and two children eligible for the PFI topical survey. If the household was pre-designated as an “ECPP household,” a child will be selected to receive the ECPP topical survey. Alternatively, if the Johnson household has two children eligible for the PFI and no children eligible for the ECPP, they will receive a PFI topical survey even if they are designated as an “ECPP household” because there are no eligible ECPP children in the household.

Second-Stage Sampling for Households That Are Selected to Receive an Adult Topical Survey: Domain (CWS or TWS)

For Screener Group A, adults that are eligible to receive a topical survey are eligible to receive either the Credentials for Work Survey (CWS) or the Training for Work Survey (TWS), because there are no differing eligibility criteria for these two topical surveys. However, an eligible adult will only receive one of the topical surveys, not both.

In addition to the age group and child topical survey pre-designations, each household will be pre-designated as a “CWS household” or a “TWS household” for Screener Group A. This pre-designation will be referred to as the adult topical survey pre-designation. A household will be pre-designated as a “CWS household” with a .78 probability and a “TWS household” with a .22 probability in order to achieve a TWS topical survey sample size of approximately 16,000. This designation will be used every time when sampling the adults since eligible adults are eligible for both topical surveys.

The second stage of sampling for households that are selected to receive an adult topical survey is the only stage where Screener Group B differs from Screener Group A. For Screener Group B households, the TWS topical survey will not be administered. Therefore, if a Screener Group B household is selected to receive an adult topical survey and the household has at least one eligible adult, the household will receive a CWS.

Third-Stage Sampling: Person Level

The third stage of sampling is at the person level. If any household has only one person that is eligible for the domain that was selected in the second stage of sampling, that person is selected to receive that topical survey. If any household has two or more people eligible for the domain that was selected in the second stage of sampling, one of those persons will be randomly selected to receive that topical survey.

Therefore, at the end of the three stages of sampling, one eligible child or one eligible adult within a household that has a completed screener will be sampled for the ECPP/PFI topical survey or the CWS/TWS, respectively, in Screener Group A. For Screener Group B, one eligible child or one eligible adult within a household that has a completed screener will be sampled for the ECPP/PFI topical survey or the CWS, respectively.

B.1.3 Expected Yield

As described above, the initial sample will consist of approximately 200,000 addresses, which splits into a 185,000 household sample that will be sent paper screeners and a 15,000 household sample that will be selected to participate in the self-response Internet screener experiment. An expected screener response rate of 70 percent and an address ineligibility¹ rate of 11 percent are assumed for Screener Group A, based on results from the NHES:2012. Since there is not a comparable study that offers an Internet screener response, the same assumptions are applied to Screener Group B. Under these assumptions, the total number of expected screeners is 124,600.

For households in Screener Group A, the ECPP/PFI or the CWS/TWS topical surveys will be administered to households with completed screeners that have eligible children or adults, respectively. For households in Screener Group B, the ECPP/PFI or the CWS topical surveys will be administered to households with completed screeners that have eligible children or adults, respectively. For the NHES:2015, we expect to achieve a total percentage of households with eligible children of approximately 28 percent, and for eligible adults a total percentage of approximately 83 percent.² Expected estimates of the percentage of households with eligible children or adults overall and in each sampling domain are given in Table 1, as well as the expected number of screened households in the nationally representative sample of 200,000, based on the distribution of household composition and assuming 124,600 total completed screeners.

Table 1.  Expected percentage of households with eligible children or adults, by sampling domain

Household composition	Percent of households	Expected number of screened households
Total households with eligible adults	83	103,418
Total households with eligible children	28	34,888
Households with at least one ECPP eligible child and no PFI eligible children	6	7,476
Households with at least one PFI eligible child and no ECPP eligible children	16	19,936
Households with at least one ECPP eligible child and at least one PFI eligible child	6	7,476

NOTE: The distribution in this table assumes 124,600 screened households. Detail may not sum to totals because of rounding.

Estimates are based on calculations from NHES:2012.

Tables 2 and 3 summarize the expected numbers of completed interviews for the NHES:2015. These numbers take into account within-household sampling. A topical response rate of 75 percent is assumed for ECPP and PFI based on results from the NHES:2012, and 70 percent is assumed for CWS and TWS based on the 2010 ATES Pilot Study response rate. Based on an initial sample of 185,000 addresses for Screener Group A and an expected eligibility rate of 89 percent, the expected number of completed screener questionnaires is 115,255. Of these, we expect to have approximately 8,968 children selected for the ECPP questionnaire, 15,229 children selected for the PFI, 56,041 adults selected for the CWS questionnaire, and 16,000 adults selected for the TWS questionnaire. Based on an initial sample of 15,000 addresses for Screener Group B, the expected number of completed screener questionnaires is 9,345. Of these, we expect to have approximately 727 children selected for the ECPP questionnaire, 1,235 children selected for the PFI, and 5,841 adults selected for the CWS questionnaire.

Table 2.  Expected numbers sampled and expected numbers of completed screeners and topical surveys for households in Screener Group A in the NHES:2015

Survey			Expected number sampled NHES:2015			Expected number of completed interviews NHES:2015
Household screeners			185,000			115,255
	ECPP			8,968	6,726
	PFI			15,229	11,422
	CWS			56,041	42,031
TWS		16,000				11,200

Table 3.  Expected numbers sampled and expected numbers of completed screeners and topical surveys for households in Screener Group B in the NHES:2015

Survey	Expected number sampled NHES:2015			Expected number of completed interviews NHES:2015
Household screeners	15,000			9,345
ECPP		727	545
PFI		1,235	926
CWS		5,841	4,089

Table 4 shows the number of ECPP and PFI completed interviews from previous NHES surveys administrations.

Table 4. Numbers of completed topical interviews in previous NHES surveys administrations
Number of completed topical interviews	NHES surveys administration
	1993	1995	1996	1999	2001	2003	2005	2007	2012
ECPP	†	7,564	†	6,939	6,749	†	7,209	†	7,893
PFI	19,144	†	17,774	17,652	†	12,422	†	10,681	17,563
† Not applicable; children in this category were not eligible for these topical interviews.

B.1.4 Reliability

The reliability of the sample sizes is measured by the margins of error. The reliability of the sample sizes of Screener Groups A and B is shown in Table 5. The value of p represents a survey estimate proportion, so if a survey estimate proportion is 70 percent the margin of error is below .03 for an overall, 50 percent, 20 percent, and 10 percent characteristics for Screener Group A. The same is true with a survey estimate proportion of 60 or 50 percent. For Screener Group B, the margin of error is below .03 for an overall and a 50% characteristic for all three survey estimate proportions.

Table 5. Margins of error for the screeners††

Screener	value of p	1.96 times standard error - overall	1.96 times standard error - 50% characteristic	1.96 times standard error - 20% characteristic	1.96 times standard error - 10% characteristic
Screener Group A	0.7	0.0034	0.0049	0.0077	0.0109
	0.6	0.0037	0.0052	0.0082	0.0116
	0.5	0.0038	0.0053	0.0084	0.0119
Screener Group B	0.7	0.0121	0.0171	0.0270	0.0382
	0.6	0.0129	0.0183	0.0289	0.0408
	0.5	0.0132	0.0186	0.0295	0.0417

†† A design effect of 1.3 was used in the calculations for this table.

Once the screener is completed, the topical mailing procedures are identical. Therefore, at the topical level, the margins of error for Screener Group A and Screener Group B combined are shown in Table 6.

Table 6. Margins of error for topical surveys in Screener Group A and Screener Group B††

Topical	value of p	1.96 times standard error - overall	1.96 times standard error - 50% characteristic	1.96 times standard error - 20% characteristic	1.96 times standard error - 10% characteristic
ECPP	0.7	0.0137	0.0194	0.0306	0.0433
	0.6	0.0146	0.0207	0.0327	0.0463
	0.5	0.0149	0.0211	0.0334	0.0473
PFI	0.7	0.0105	0.0149	0.0235	0.0332
	0.6	0.0112	0.0159	0.0251	0.0355
	0.5	0.0115	0.0162	0.0256	0.0363
CWS	0.7	0.0056	0.0079	0.0125	0.0177
	0.6	0.0060	0.0085	0.0134	0.0190
	0.5	0.0061	0.0087	0.0137	0.0194
TWS	0.7	0.0110	0.0156	0.0247	0.0349
	0.6	0.0118	0.0167	0.0264	0.0373
	0.5	0.0120	0.0170	0.0269	0.0381

†† A design effect of 1.3 was used in the calculations for this table.

B.1.5. Estimation Procedures

The data sets from the NHES:2015 will have weights assigned to facilitate estimation of nationally representative statistics. All households responding to the screener will be assigned weights based on their probability of selection and a non-response adjustment, making them representative of the household population. All individuals responding to the topical questionnaires will have a record with a person weight designed such that the complete data set represents the target population.

The estimation weights for the NHES:2015 surveys will be formed in stages. The first stage is the creation of a base weight for the household, which is the inverse of the probability of selection of the address. The second stage is a screener nonresponse adjustment to be performed based on characteristics available on the frame and screener.

The household-level weights are the base weights for the person-level weights. For each completed topical questionnaire, the person-level weights also undergo a series of adjustments. The first stage is the adjustment of these weights for the probability of selecting the person within the household. The second stage is the adjustment of the weights for topical survey nonresponse to be performed based on characteristics available on the frame and screener. The third stage is the raking adjustment of the weights to Census Bureau estimates of the target population. The variables that may be used for raking at the person level include race and ethnicity of the sampled person, household income, home tenure (own/rent/other), region, age, grade of enrollment, gender, family structure (one parent or two parent), and highest educational attainment in household. These variables (e.g., family structure) have been shown to be associated with response rates. The final raked person-level weights include undercoverage adjustments as well as adjustments for nonresponse.

Standard errors of the estimates will be computed using a jackknife replication method. The replication process repeats each stage of estimation separately for each replicate. The replication method is especially useful for obtaining standard errors for the statistics that the NHES:2015 produces.

B.1.6 Nonresponse Bias Analysis

To the extent that those who respond to surveys and those who do not differ in important ways, there is a potential for nonresponse biases in estimates from survey data. The estimates from NHES:2015 are subject to bias because of unit nonresponse to both the screener and the extended topical surveys, as well as nonresponse to specific items. Per NCES statistical standards, a unit-level nonresponse bias analysis will be conducted if the NHES:2015 overall unit response rate (the screener response rate multiplied by the topical response rate) falls below 85 percent. Additionally, any item with an item-level response rate below 85 percent will be subject to an examination of bias due to item nonresponse.

Unit nonresponse

To identify characteristics associated with unit nonresponse, a multivariate analysis will be conducted using a categorical search algorithm called Chi-Square Automatic Interaction Detection (CHAID). CHAID begins by identifying the characteristic of the data that is the best predictor of response. Then, within the levels of that characteristic, CHAID identifies the next best predictor(s) of response, and so forth, until a tree is formed with all of the response predictors that were identified at each step. The final result is a division of the entire data set into cells by attempting to determine sequentially the cells that have the greatest discrimination with respect to the unit response rates. In other words, it divides the data set into groups so that the unit response rate within cells is as constant as possible, and the unit response rate between cells is as different as possible. Since the variables considered for use as predictors of response must be available for both respondents and nonrespondents, demographic variables from the sampling frame provided by the vendor (including household education level, household race/ethnicity, household income, age of head of household, whether the household owns or rents the dwelling, and whether there is a surname and/or phone number present on the sampling frame) and data from the screener (including number of children in the household and the number of adults in the household) will be included in the CHAID analysis.

In addition to the above, the magnitude of unit nonresponse bias and the likely effectiveness of statistical adjustments in reducing that bias will be examined by comparing estimates computed using adjusted weights to those computed using unadjusted weights. The unadjusted weight is the reciprocal of the probability of selection, reflecting all stages of selection. The adjusted weight is the extended interview weight adjusted for unit nonresponse (without the raking adjustment). In this analysis, the statistical significance of differences in estimates will be investigated only for key survey estimates including, but not limited to, the following:

All surveys

• Age/grade of child or age of adult

• Census region

• Race/ethnicity

• Parent 1 or adult’s employment status

• Parent 1 or adult’s home language

• Educational attainment of parent 1/adult

• Family type

• Household income

• Home ownership

Early Childhood Program Participation (ECPP)

• Child receiving relative care

• Child receiving non-relative care

• Child receiving center-based care

• Number of times child was read to in past week

• Someone in family taught child letters, words, or numbers

• Child recognizes letters of alphabet

• Child can write own name

• Child is developmentally delayed

• Child has health impairment

• Child has good choices for child care/early childhood programs

Parent and Family Involvement in Education (PFI)

• School type: public, private, homeschool

• Whether school assigned or chosen

• Contact from school about child’s behavior

• Contact from school about child’s school work

• Child’s overall grades

• Parents participate in 5 or more activities in the child’s school

• Parents report school provides information very well

About how child is doing in school

About how to help child with his/her homework

About why child is placed in particular groups or classes

About how to help child plan for college or vocational school

About the family's expected role at child’s school

• Parents attended a general school meeting (open house), back-to-school night, meeting of parent-teacher organization

• Parents went to a regularly scheduled parent-teacher conference with child’s teacher

• Parents attended a school or class event (e.g., play, sports event, science fair) because of child

• Parents acted as a volunteer at the school or served on a committee

• Parents check to see that child's homework gets done

Credentials for Work Survey/Training for Work Survey (CWS/TWS)

• Certification or license

• Certificate

• Completed work-related training last 12 months

The final component of the bias analysis will include comparisons between respondent characteristics and known population characteristics from extant sources including the Current Population Survey (CPS) and the American Community Survey (ACS). Additionally, for substantive variables, weighted estimates will be compared to prior NHES administrations if available. While differences between NHES:2015 estimates and those from external sources as well as prior NHES administrations could be attributable to factors other than bias, differences will be examined in order to confirm the reasonableness of the 2015 estimates.

Item nonresponse

In order to examine item nonresponse, all items with response rates below 85 percent will be listed. Alternative sets of imputed values will be generated by imposing extreme assumptions on the item nonrespondents. For most items, two new sets of imputed values—one based on a “low” assumption and one based on a “high” assumption—will be created. For most continuous variables, a “low” imputed value variable will be created by resetting imputed values to the value at the fifth percentile of the original distribution; a “high” imputed value variable will be created by resetting imputed values to the value at the 95^th percentile of the original distribution. For dichotomous and most polytomous variables, a “low” imputed value variable will be created by resetting imputed values to the lowest value in the original distribution, and a “high” imputed value variable will be created by resetting imputed values to the highest value in the original distribution. Both the “low” imputed value variable distributions and the “high” imputed value variable distributions will be compared to the unimputed distributions. This analysis helps to place bounds on the potential for item nonresponse bias through the use of “worst case” scenarios.

B.2 Statistical Procedures for Collection of Information

This section describes the data collection procedures to be used in the NHES:2015. These procedures represent a combination of best practices to maximize response rates based on findings from the ATES 2010 Pilot Study, the NHES:2011 Field Test, NHES:2012, the 2013 the National Adult Training and Education Survey (NATES), and the 2014 NHES-Feasibility Study (NHES-FS) within the Agency’s budget constraints. The NHES is a two-phase self-administered survey. In the first phase, households are screened to determine if they have eligible adults or children. In the second phase, a detailed topical questionnaire is sent to households with the eligible adults or children. The NHES employs multiple contacts with households to maximize response. These include an advance letter and up to four questionnaire mailings for both the screener and the topical surveys. In addition, households will receive one reminder postcard after the initial mailing of a screener or topical questionnaire and an automated phone call reminder for nonrespondents in conjunction with the third mailing. Respondent contact materials are provided in appendix A and questionnaires are provided in appendices B (screener) and C (topical questionnaires).

Screener Procedures

Figure 1 presents a flow chart for the NHES:2015 data collection which will begin with the mailing of an advance notification letter in early January 2015. The NHES:2011 Field Test showed that using an advance letter was effective at raising response rates to the first questionnaire mailing which reduces the amount of follow-up required. Based on this finding NHES:2012, NATES, and NHES-FS have also used an advance letter. A questionnaire package will be mailed to households one week after the advance letter. The packages will contain a cover letter, household screener, business reply envelope, and a $5 incentive. Many years of testing in the NHES have shown the effectiveness of a small cash incentive on increasing response and the NHES:2011 Field Test showed that a $5 incentive was more effective at increasing screener response than a $2 incentive. A Spanish version will be offered with the second, third, and fourth mailings in addition to the English version. Cases that are sampled as part of the Hispanic oversample, or where the frame vendor indicates a Spanish surname is associated with the address, will receive both the English and Spanish versions with the first mailing. The same version of the questionnaire sent in the first mailing will be sent in all subsequent nonresponse follow-up mailings. A thank you/reminder postcard will be sent to all sampled addresses approximately one week after the first mailing. All nonresponse follow-up questionnaire mailings will contain a cover letter, replacement screener questionnaire, and business reply envelope. A second questionnaire mailing will be sent to nonresponding households approximately two weeks after the postcard. Approximately three weeks after the second mailing, a third questionnaire mailing will be sent to non-responding households using rush delivery (FedEx or USPS Priority Mail³). For addresses for which the frame includes a telephone number, an automated reminder phone call will be made on the same day as the third questionnaire mailing to encourage households to complete the study as soon as possible. Results from the 2009 and 2011 tests indicated that households with eligible children tend to respond to later mailings, for this reason we added an additional mailing to our process and extended the data collection period for the NHES:2012 and NHES-FS data collection. We will continue that protocol in the NHES:2015. The fourth questionnaire package will be sent to nonresponding households approximately three weeks after the third questionnaire mailing.

For the first time, the NHES:2015 will implement a test of self-response using an Internet mode. This test will follow the traditional mailout strategy of having an advance letter, four contacts, and a reminder postcard but with a different contact strategy specifically designed for the Internet (see Figure 2). The self-response Internet screener questionnaire will only be available in English. The test, which is only being conducted for the screener, will be a response rate test that compares self-response on an Internet questionnaire to the paper questionnaire to determine if a self-response Internet screener results in an acceptable response rate. This test includes two treatment groups and examines different contact strategies.

The contact strategy for the two experimental treatment groups will be the same until the fifth contact. Households will receive an advance letter as the first contact. The second contact will be a mailing that includes a $5 incentive and a letter inviting the household to participate via the Internet. The letter will contain a unique username and password required to complete the survey. The third contact will be a thank you/reminder post card. Nonresponding households will be sent another letter inviting the household to participate in the survey via the Internet by using a unique username and password provided in the letter as the fourth contact.

The contact strategy for the two treatment groups differs for the fifth household contact. Nonresponding households in Group 1 will be sent a letter via FedEx that invites the household to participate in the survey via the Internet using a unique username and password. Nonresponding households in Group 2 will be sent both a letter that invites the household to participate in the survey via the Internet using a unique username and password, as well as a paper questionnaire. The letter and paper questionnaire will be mailed using FedEx. The sixth and final contact for nonresponding households in both groups will be a letter containing the unique username and password to participate via Internet, as well as a paper questionnaire. This letter and paper questionnaire will be sent using USPS First Class mail.

Testing the two alternative contact strategies for the self-response Internet survey will allow us to test the most efficient way to encourage web response while also providing households without Internet access the opportunity to participate in the survey.

Topical Procedures

Topical survey mailings will follow procedures similar to the screener procedures. Households will receive a topical package in English or Spanish depending on the language they used to complete the screener. As described earlier, households with at least one eligible person will be assigned to one of four topical survey mailing groups:

1. Households that have been selected to receive an ECPP questionnaire only;

2. Households that have been selected to receive a PFI questionnaire only;

3. Households that have been selected to receive a CWS questionnaire only;

4. Households that have been selected to receive a TWS questionnaire only.

Within each group, questionnaires will be mailed in waves to minimize the time between the receipt of the screener and the mailing of the topical. Regardless of treatment group or wave, households with a selected sample member will have the same topical contact strategy. The initial topical mailing will include a cover letter, questionnaire, business reply envelope, and a cash incentive. All nonresponse follow-up mailings will contain a cover letter, replacement questionnaire, and business reply envelope. The NHES:2011 Field Test, the NHES:2012, and the NHES-FS showed that a $5 incentive was effective with most respondents at the topical level. However, late screener respondents (those that responded after the third or fourth mailing) were more likely to respond to the $15 incentive amount. As a result, we will offer these late screener responders a $15 prepaid incentive with their first topical questionnaire mailing. All other respondents will receive $5 with their initial mailing. One week after the initial topical mailing, all households will receive a thank you/reminder postcard.

Nonresponding households will be mailed a second topical questionnaire approximately two weeks after the reminder postcard. If households that have been mailed a second topical questionnaire do not respond, a third package will be mailed by rush delivery (FedEx or USPS Priority Mail) approximately three weeks after the second mailing. Most households will be sent packages via FedEx delivery; only those households with a PO Box-only address will be sent USPS Priority Mail. An automated reminder phone call will also be made to households where the frame vendor has a phone number. If the household does not respond to the third mailing, a fourth mailing will be sent via USPS First Class mail. Figures 3and 4 show the topical mailing plans.

Figure 1: Screener Data Collection

Figure 2: Screener Internet Data Collection Mailing Plan

Figure 3: Topical Mailing Plan – ECPP/PFI

Figure 4: Topical Mailing Plan – CWS/TWS

Survey Monitoring

Mail survey returns will be processed upon receipt, and reports from the survey management system will be prepared at least weekly. The reports will be used to continually assess the progress of data collection.

B.3 Methods for Maximizing Response Rates

The NHES:2015 design incorporates a number of features to maximize response rates. This section discusses those features.

Total Design Method/Respondent-Friendly Design. Surveys that take advantage of respondent-friendly design have demonstrated increases in survey response (Dillman, Smyth, and Christian 2008; Dillman, Sinclair, and Clark 1993). We have honed the design of the NHES forms through multiple iterations of cognitive interviewing and field testing. These efforts have focused on the design and content of all respondent contact materials. As noted previously, we will include a respondent incentive in the initial screener mailing. Respondent incentives will also be used in the initial topical mailing. Many years of testing in the NHES have shown the effectiveness of a small cash incentive on increasing response. The Census Bureau will maintain an email address and a toll-free questionnaire assistance (TQA) line to answer respondent questions or concerns. If a respondent chooses to provide their information to the TQA staff, staff will be able to collect the respondent’s information on an Internet-based screener questionnaire. Additionally, as a test, a subsample of respondents may choose to provide their information using a self-response Internet-based screener questionnaire. If successful in the NHES:2015, this online questionnaire could be used for future NHES self-response. Additionally, the questionnaires contain frequently asked questions (FAQs) and contact information for the Project Officer.

Engaging Respondent Interest and Cooperation. The content of respondent letters and FAQs is focused on communicating the legitimacy and importance of the study. Past experience has shown that the NHES child survey topics are salient to most parents. In the NHES:2012, Census Bureau “branding” was experimentally tested against Department of Education branding. Response rates to Census Bureau branded questionnaires were higher compared to Department of Education branded questionnaires. Based on this finding, we will highlight the Census Bureau’s participation in the data collection.

Nonresponse Follow-up. The data collection protocol includes several stages of nonresponse follow-up at each phase. In addition to the number of contacts, changes in method (USPS First Class mail, FedEx, and automated reminder phone calls) are designed to capture the attention of potential respondents.

B.4 Tests of Procedures and Methods

Internet Screener Experiment: The NHES:2015 will include a self-response Internet screener experiment to determine if a self-response Internet screener results in an acceptable response rate. If an acceptable response rate is achieved, the self-response Internet screener could be administered in place of the paper screener in the future. After stratifying the initial screener sample of 200,000, a systematic sample of 15,000 households will be selected to participate in the experiment. The 15,000 households will be evenly split into two panels with two different contact strategies. The description of these two strategies is provided in Section B.2.

TWS Pilot: NCES is also requesting clearance for a small pilot study of a complementary Training for Work Survey (TWS) that focuses on participation in work-related education and training. NCES expects to field the TWS as part of NHES:2017. Adults that are eligible to receive a topical survey will receive either the Credentials for Work Survey (CWS) or the Training for Work Survey (TWS) but not both. Each pre-designated adult household will then be pre-designated as either a “CWS household” or a “TWS household”. A household will be pre-designated as a “TWS household” with a .22 probability in order to achieve a TWS topical survey sample size of approximately 16,000.

B.5 Individuals Responsible for Study Design and Performance

The persons listed below participated in the study design and are responsible for the collection and analysis of the data:

Sharon Boivin, NCES

Lisa Hudson, NCES

Andrew Zukerberg, NCES

Carolyn Pickering, Census

References

Dillman, D.A., Sinclair, M.D., and Clark, J.R. (1993). Effects of questionnaire length, respondent-friendly design, and difficult questions on response rates for occupant-addressed Census mail surveys. Public Opinion Quarterly, 57, 289-304.

Dillman, D.A., Smyth, J.D., and Christian, L.M. (2008). Internet, mail, and mixed mode surveys: The Tailored Design Method. New York: Wiley.

1 Ineligible addresses are those that are undeliverable. Once a screener mailing for an address is returned as a postmaster return (PMR), the address will be coded ineligible.

2 Percentages based on population proportions from the Annual Social and Economic Supplement (ASEC) of the Current Population Survey (CPS)

3 Because FedEx cannot deliver to PO Boxes, PO Box-only addresses will be delivered using USPS Priority Mail.