Att_NHES2007 OMB part B

Att_NHES2007 OMB part B.doc

2007 National Household Education Surveys Program (NHES: 2007)

OMB: 1850-0768

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 1850-0768 can be found here:

Document [doc]

Download: doc | pdf

NHES:2007

Request for

IMT/OMB Review

August 30, 2006

TABLE OF CONTENTS

Section Page

PART A: JUSTIFICATION 1

Request for Clearance 1

Introduction 1

A.1. Circumstances Necessitating Collection of Information 10

A.2. Purposes and Uses of the Data 14

A.3. Use of Improved Information Technology 15

A.4. Efforts to Identify Duplication 16

A.5. Collection of Data from Small Businesses 25

A.6. Consequences of Less Frequent Data Collection 25

A.7. Special Circumstances 26

A.8. Public Comment and Consultations Outside the Agency 26

A.9. Payments to Respondents 34

A.10. Assurance of Confidentiality 35

A.11. Sensitive Questions 38

A.12. Estimated Response Burden 39

A.13. Annualized Cost to Respondents 41

A.14. Annualized Cost to the Federal Government 41

A.15. Reasons for Program Changes 41

A.16. Publication Plans and Project Schedule 41

A.17. Approval for Not Displaying the Expiration Date for OMB Approval 51

A.18. Exceptions to the Certification Statement 51

PART B. DESCRIPTION OF STATISTICAL METHODOLOGY 53

B.1. Statistical Design and Estimation 53

B.2. Survey Procedures 88

B.3. Methods for Maximizing Response Rates 95

B.4. Tests of Procedures and Methods 97

B.5. Individuals Responsible for Study Design and Performance 109

PART C: JUSTIFICATION OF NHES:2007 QUESTIONNAIRES 111

C.1. Screener 111

C.2. School Readiness and Parent and Family Involvement in Education Interview 118

C.3. Adult Education for Work-Related Reasons Interview 138

C.4. PFI Reinterview 149

References 151

Appendix A: Screener, School Readiness and Parent and Family Involvement in Education Interview, Adult Education for Work-Related Reasons Interview, and PFI
Reinterview A-1

Appendix B: Respondent Materials B-1

Appendix C: Details About Sample Size Requirement C-1

LIST OF TABLES

Table Page

Table 1. Estimated response burden for NHES:2007 40

Table 2. Race/ethnicity distribution of completed interviews in NHES:2003: Actual counts and percentages compared to counts and percentages expected without oversampling in the high minority stratum: 2003 62

Table 3. Screener response rates and ever refusal rates, by incentive group: 2003 65

Table 4. Expected numbers of sampled telephone numbers and completed Screeners, and expected residency and Screener unit response rates, by stratum: 2007 73

Table 5. Percentage of telephone households with eligible children, by age/grade group: CPS 2003 80

Table 6. Expected number of screened households in NHES:2007, by household composition: CPS 2003 81

Table 7. Expected number of adults sampled for AEWR interviews, by number of adults and presence of eligible children in household: 2007 82

Table 8. Expected numbers sampled and expected numbers of completed interviews in the telephone sample for NHES:2007 83

Table 9. Comparison of expected number of persons sampled for extended interview in NHES:2007 to the numbers sampled in previous survey administrations 85

Table 10. Comparison of expected numbers of completed interviews in NHES:2007 to the numbers completed in previous survey administrations 86

LIST OF EXHIBITS

Exhibit Page

Exhibit 1. Topics addressed in surveys conducted under the National Household Education Surveys Program and years administered 2

Exhibit 2. NCES Affidavit of Nondisclosure 36

Exhibit 3. Westat Confidentiality Pledge 37

Exhibit 4. NHES 2007 schedule of major activities 51

Exhibit 5. NXXType codes 58

Exhibit 6. Key design features of the NHES:2007 nonresponse bias evaluation 68

Exhibit 7. Overview of the sampling scheme for selecting adults based on household composition 79

LIST OF EXHIBITS-Continued

Exhibit Page

Exhibit C-1. Item-by-item description of the NHES:2007 Screener 116

Exhibit C-2. Item-by-item description of NHES:2007 School Readiness Interview and Parent

and Family Involvement in Education Interview 127

Exhibit C-3. Item-by-item description of NHES-2007 Adult Education for Work-Related Reasons

Interview 143

PART B. DESCRIPTION OF STATISTICAL METHODOLOGY

B.1. Statistical Design and Estimation

An important purpose of the National Household Education Surveys Program (NHES) is to conduct repeated measurements of the same phenomena at different points in time, and this goal is reflected in the sample design of NHES:2007. NHES:2007 is a random digit dialing (RDD) telephone survey covering the 50 states and the District of Columbia. It will be conducted from January through mid-April 2007. Households will be randomly sampled, and a screening interview will be administered to a household respondent age 18 or older.^¹ Demographic information about household members will be used to determine whether anyone is eligible for the School Readiness (SR), Parent and Family Involvement in Education (PFI), or Adult Education for Work-Related Reasons (AEWR) Surveys.

The SR Survey will be administered to the parent or guardian^² in the household who is most knowledgeable about the care and education of the sampled child age 3 through age 6, as of December 31, 2006, who is not yet in kindergarten.^³ For the PFI Survey, the parent/guardian most knowledgeable about the care and education of the sampled child age 20 or younger who is enrolled in kindergarten through twelfth grade will be interviewed.^⁴ The SR and PFI Surveys will be administered in a single instrument; however, the sample design considerations discussed in this report are unaffected by this. The AEWR Survey will be administered to sampled persons 16 years or older who are not currently enrolled in twelfth grade or below and are not institutionalized or on active duty in the U.S. Armed Forces.

B.1.1. Sampling Telephone Numbers

The sampling method used for NHES:2007 will be a list-assisted method described by Casady and Lepkowski (1993) and by Tucker, Lepkowski, and Piekarski (2002). This method was used previously in NHES:1995, NHES:1996, NHES:1999, NHES:2001, NHES:2003, and NHES:2005.^⁵ The list-assisted method is a single-stage, unclustered method that produces a self-weighting sample of telephone numbers. In a list-assisted sample, a simple random sample of telephone numbers is selected from all telephone numbers that are in 100-banks (the set of numbers with the same first eight digits) in which there is at least one residential telephone number listed in the white pages directory. This is called the listed stratum.^⁶ Telephone numbers in 100-banks with no listed telephone numbers, the zero-listed stratum, are not sampled. The telephone numbers in the listed stratum include both listed and unlisted numbers and both residential and nonresidential numbers.

Differences in telephone coverage rates, especially differential rates among population subgroups, such as those defined by region, age, race/ethnicity, and household composition, are of concern to telephone survey methodologists because they can introduce bias in the estimates. The largest component of coverage bias in a telephone survey such as the NHES is probably due to the prevalence of nontelephone households^⁷ and the differences between such households and those with telephones. Based on recent findings (Tucker et al. 2004 and Blumberg et al. 2004), it is expected that by 2007, the percentage of households with no telephone service will be about 2 percent, and the percentage of households with cell phone service alone will be about 5 to 10 percent. Tucker et al. (2004) and Blumberg et al. (2004) examined differences in characteristics among persons and households having no telephone service, cellular service only, and landline service (including both landline only, and landline and cellular). Although there are differences (e.g., young adults, adults in 1-person households, renters, and Blacks and Hispanics) in landline coverage, raking to population totals for these subgroups is used in NHES to statistically adjust for and reduce undercoverage bias.

Additionally, coverage bias may arise with this sampling scheme because not all telephone households are included in the listed stratum; households in the zero-listed stratum have no chance of being included in the sample. Empirical findings were presented in Brick, et al. (1995) to address the question of coverage bias associated with excluding the zero-listed stratum. The results show that the percentage of telephone numbers in the zero-listed stratum that are residential is small (about 1.4 percent) and that about 3 to 4 percent of telephone households are in the zero-listed stratum. The results also indicate that households in the zero-listed stratum are not very different from households in the listed stratum. Because the proportion of telephone households that are in the zero-listed stratum is small and the persons living in these households are not very different from those living in households in the listed stratum, the bias resulting from excluding the zero-listed stratum is generally very small. Giesbrecht, Kulp, and Starer (1996) examined coverage bias due to exclusion of the zero-listed stratum using data from the Current Population Survey (CPS) and also found the bias to be small. Telephone exchanges are classified by NXXType,^⁸ a code that indicates the types of telephone numbers assigned within the exchange (e.g., mobile only, cellular only, etc.). A complete list of NXXType codes is given in exhibit 5. Due to newly enacted legislation, by the time NHES:2007 is fielded, exchanges previously limited to land lines may include cellular numbers. For NHES:2007, as in previous NHES studies, telephone numbers will be sampled from exchanges having NXXTypes 00 or 52 only, which cover about 99 percent of listed households.^⁹ However, for future NHES studies, this restriction should be re-examined; in particular, NXXTypes 50, 51, and 54 should be considered.^¹⁰ Telephone numbers assigned for Voice Over Internet Protocol (VDIP) are classified as “plain old telephone service” and are thus in the NXXTypes included in the frame for NHES.

In NHES:2007, as in previous NHES administrations, procedures will be used prior to data collection to reduce the number of unproductive calls. Prior to NHES:2001, Marketing Systems Group’s (MSG’s) Genesys ID process was used.^¹¹ The Genesys ID process included tritone^¹² checks for nonworking numbers and purging of listed business numbers (i.e., numbers listed in the yellow pages but not in the white pages). In NHES:2001 and NHES:2003, a more extensive procedure, the Genesys ID-PLUS process, was used prior to the field period. With the ID-PLUS utility, a telephone number is dialed by Genesys and allowed to ring up to two times (compared with one ring in the Genesys ID tritone test). If the telephone call is answered, a representative is available to speak to the respondent. In such cases, the representative attempts to ascertain whether the telephone number is a business number.^¹³ For NHES:2005, a more comprehensive prescreening procedure, the Genesys Comprehensive Sample Screening (Genesys-CSS) procedure, was used. Like the Genesys ID and ID-PLUS utilities, the Genesys-CSS utility also includes the white and yellow pages matches. The primary differences between Genesys-CSS and the ID-PLUS procedure are enhanced identification of all types of wireless numbers and the pre-dialing of numbers listed in the white pages.^¹⁴ With the Genesys-CSS utility, each telephone number is classified into one of the following categories:

LB (Listed Business)

UR (Unlisted Residence)

UB (Unlisted Business)

FM (Fax/Modem)

LA (Language Barrier)

NR (No Ring Back)

NW (Nonworking)

BX (Blocked Exchanges)

PM (Privacy Manager^¹⁵)

WR (Wireless)

CP (Cell Phone)

DK (Undetermined: Residential/No Answer/Busy)

Because the Genesys-CSS method is more comprehensive than the ID-PLUS process, Genesys-CSS will be used in NHES:2007. As in NHES:2005, telephone numbers identified by Genesys-CSS as LB, NW, WR, or CP, as well as UB telephone numbers for which no mailing address could be obtained, will be excluded from dialing in NHES:2007. In NHES:2005, these exclusions amounted to 35 percent of the sample of telephone numbers; it is expected that the percent of telephone numbers excluded from dialing in NHES:2007 will be similar. All telephone numbers that are not excluded from dialing as a result of the Genesys-CSS results will be sent to up to two address vendors to obtain mailing addresses. (The second vendor will attempt to obtain mailing addresses for only those telephone numbers for which the first vendor is unable to provide a match.) In NHES:2003, address matches were obtained for 43 percent of the full phase 1 sample^¹⁶ and 66 percent of the phase 1 sample excluding the cases that were not dialed (the NW, LB, and nonmailable UB cases).

Exhibit 5. NXXType codes

Code	Description

00	Regular
01	Mobile radio
02	Paging
03	Packet switching
04	Cellular
05	Test code
06	Maritime
07	Air to ground
09	900 service
10	Called party pays
11	Information provider
13	Directory assistant
14	Special calling cards
15	Official exchange carrier service
16	Originating only
17	Billing only
30	Broadband
50	Shared among three or more services
51	Shared between plain old telephone service (POTS) and mobile
52	Shared between POTS & paging
54	Shared between POTS & cellular
55	Special billing options – Cellular
56	Special billing options – Paging
57	Special billing options – Mobile
58	Shared among two or more
60	Intra-Local Access and Transport Area (IntraLATA) billing option – Cellular
61	IntraLATA billing option – Paging
62	IntraLATA billing option - Mobile
63	Combination of 60, 61, and 62
64	Personal communication service (PCS)
65	Miscellaneous
66	Shared between POTS and miscellaneous
67	PCS/Miscellaneous service
68	Selective local exchange, IntraLATA special billing option - PCS/Misc.

SOURCE: TPM^TM Data Source (Telcordia^TM TPM^TM Data Source), Data for the telecommunications industry that describes and supports the local network environment. CD produced by Telcordia^TM Routing Administration(TRA), Telcordia^TM Technologies, Inc. , October 15, 2003.

B.1.1.1 Oversampling Blacks and Hispanics

The general precision requirement for each survey in NHES:2007 is the ability to detect a 10 to 15 percent relative change for an estimate of between 30 and 60 percent (see appendix C for details). As in previous NHES administrations, one goal of NHES:2007 will be to produce reliable estimates for race/ethnicity subdomains (in particular, Blacks, Hispanics, and Asians). The method used in NHES surveys to date has been to stratify by the concentration of Blacks and Hispanics in the exchange and oversample telephone numbers in the high minority stratum. To avoid the possible introduction of biases as a result of this oversampling, the weights used for analysis account for selection probabilities at all stages of selection. The sample design for NHES:2007 is based on the approach used in NHES:2003 and NHES:2005, which was a slight modification of the NHES:2001 approach. Prior to NHES:2001, a re-evaluation of the approach used in previous NHES studies to oversample blacks and Hispanics was undertaken. In NHES surveys prior to NHES:2001, the high minority stratum was defined as the set of exchanges in which at least 20 percent of the population was black or at least 20 percent of the population was Hispanic. The re-evaluation was warranted for several reasons:

Since the original evaluation of the oversampling method (based on the NHES:1989 field test), the method of sampling telephone numbers had changed from the modified Mitofsky‑Waksberg method to the list-assisted method.^¹⁷
Demographic changes, especially the distribution and concentration of race/ethnicity subgroups, could affect the effectiveness of oversampling.
Changes in residency rates could affect the effectiveness of oversampling, particularly if there are disproportionate changes across strata.
An alternative under consideration was differential sampling of telephone numbers based on whether or not they are listed in the white pages directory (i.e., “listed” vs. “unlisted” numbers).
The sampling frame used to select the sample of telephone numbers had been enhanced to include information about the percent Asian in the exchange. In light of the interest in the ability to produce reliable estimates of characteristics of Asian Americans, an evaluation of the effect of the alternatives on the expected yield for Asians was warranted.

For the evaluation, several alternative stratification schemes were considered. The alternative definitions of a “high-minority” stratum considered were as follows:

At least 10 percent Black or at least 10 percent Hispanic;
At least 20 percent Black or Hispanic;
At least 20 percent Black or at least 20 percent Hispanic;
At least 30 percent Black or Hispanic; and
At least 30 percent Black or at least 30 percent Hispanic.

In addition to information about the race/ethnicity distribution of the exchange, it is possible to obtain telephone number-level information that is associated with cost and operational efficiency. Specifically, for each telephone number in the sample, the listed status of the telephone number (i.e., whether the number is listed in the White Pages Directory) and the mailable status (i.e., whether a mailable address may be obtained for the telephone number) are available. In the design of the 2001 surveys, listed status was considered in addition to minority stratum. The evaluation compared the expected precision of estimates across alternatives, holding the total cost fixed. It was determined that among the alternatives considered, stratification involving both minority strata and the listed status of the telephone number was optimal, and that the alternative in which “high minority” is defined as “at least 20 percent Black or at least 20 percent Hispanic” was optimal.^¹⁸ For the 2001 surveys, minority stratification was used for selecting an initial, larger sample of telephone numbers. The definition of a “high minority” stratum (at least 20 percent Black or at least 20 percent Hispanic) was the same as that used prior to 2001. Once this sample had been selected, the listed status was obtained for each sampled telephone number, and within each minority stratum, telephone numbers were sampled differentially based on listed status.

Research conducted at Westat subsequent to the selection of the NHES:2001 sample suggested that using mailable status, rather than listed status, could further improve the efficiency of the sample. Within each minority stratum, mailable status is a better discriminator of residency and response rates than listed status. As a result of this research, three alternatives were considered for NHES:2003:

Minority stratification only,
Minority by listed stratification, and
Minority by mailable stratification.

For each of these alternatives, the minority strata were defined using the same definition that had been used in previous NHES studies.

The Black and Hispanic populations are heavily concentrated in the high minority stratum, and a substantial proportion of the population in the high minority stratum (approximately 50 percent of the population) is Black or Hispanic. These two aspects are what make minority stratification an effective means of increasing the sample yield for Blacks and Hispanics. When listed status or mailable status are considered, within each minority stratum, there are considerable differences in both residency and response rates.^¹⁹ These differences are more pronounced when mailable status is considered. These differences in residency and response rates allow for a more efficient design when either mailable status or listed status is used in stratification and mailable or listed numbers are oversampled.

In light of the findings of this study, differential sampling of mailable and nonmailable telephone numbers was used in conjunction with minority stratification for NHES:2003 and NHES:2005. Table 2 gives the actual race/ethnicity distribution of completed interviews in NHES:2003, compared to the expected distribution if oversampling of telephone numbers in the high minority stratum had not been used.

Because the minority stratification has been effective in improving the sample yield for Blacks, Hispanics, and Asians and stratification on mailable status was effective in improving the operational efficiency of the sample, these characteristics will be used to stratify the NHES:2007 sample of telephone numbers. Race/ethnicity distributions are available on the sampling frame. However, the mailable status of telephone numbers is not available on the frame. The standard procedure is to match the sample of telephone numbers to address listings by telephone number to obtain the mailable status of each sampled telephone number. Therefore, in order to stratify on both minority concentration and mailable status, it will be necessary to select the sample of telephone numbers in two phases. The first phase will involve minority stratification only. The mailable status will be obtained for each first-phase telephone number, and the second phase will involve subsampling from the first-phase sample using strata defined by the combination of minority stratum and mailable status.

Race/ethnicity	NHES:2003 actual		NHES:2003 expected without oversampling of telephone numbers in high minority stratum
Race/ethnicity	Number of completed interviews	Percent of completed interviews	Number of completed interviews	Percent of completed interviews

PFI
Total	12,426	100.0	12,426	100.0
Black, non-Hispanic	1,628	13.1	1,230	9.9
Hispanic	2,576	20.7	2,087	16.8
Asian/Pacific Islander	363	2.9	345	2.8
Other	7,859	63.3	8,763	70.5

AEWR
Total	12,725	100.0	12,725	100.0
Black, non-Hispanic	1,343	10.6	1,018	8.0
Hispanic	1,318	10.4	1,062	8.3
Asian/Pacific Islander	371	2.9	344	2.7
Other	9,693	76.2	10,301	81.0

NOTE: Due to rounding, subdomain counts may not add to totals.

SOURCE: U.S. Department of Education, National Center for Education Statistics, Parent and Family Involvement in Education Survey of the National Household Education Surveys Program, 2003; and Adult Education for Work-Related Reasons Survey of NHES, 2003.

B.1.1.2 Subsampling Cases for Followup

In NHES, substantial effort is undertaken to make contact with households and secure their cooperation in the interviews. In previous NHES administrations, for each case in which a potential respondent refused to respond to the interview, with the exception of hostile (i.e., profane or abusive) refusals, a refusal conversion was attempted by specially trained interviewers. In recent NHES studies, Screener cases that finalized as “no answer” or “no answer, answering machine” due to failure to make contact were refielded for additional call attempts. Cases that finalized as “maximum call” due to failure to complete an interview after making contact with the household were re-released for additional call attempts.

Although these followup methods have proven to be effective in previous NHES studies, they may be more effective if these efforts can be concentrated on a portion of the sample that is “front-loaded” (i.e., this portion of the sample would be designated to be worked at the beginning of the study). The portion of the sample not designated for these followup efforts would be released a few weeks into the study, to allow sufficient resources for working the cases that will receive the followup efforts. Since all cases designated for followup efforts would be released at the beginning of the study and all efforts initially would focus on these cases, this approach is believed to allow for more efficient and effective use of followup procedures in a study with a short data collection period such as NHES.

This “followup subsampling” approach has been used in other surveys conducted by Westat^²⁰ including NHES:2005, and is a cost-effective procedure that may have a positive effect on response rates.^²¹ Although it is not clear why this procedure has a positive effect on response rates, it is possible that it is because it enables resources to be concentrated early on the cases receiving followup and, by holding other cases until a later release date, reduces the amount of unproductive work at the end of the field period. The subsampling procedure is especially effective with an incentive scheme such as that used for NHES:2005, in which a refusal conversion incentive was mailed to the subsample of cases designated for refusal conversion (provided an address was available), but no advance incentive was mailed. In NHES:2007, as discussed in the next section, all cases for which a mailable address is obtained will be sent a small precontact incentive in the advance letter. Although refusal conversion incentives alone are not proposed for NHES:2007, the front-loading approach is still worthwhile from an operational standpoint.

For NHES:2007, as in NHES:2005, 60 percent of the basic sample of telephone numbers will be randomly designated to receive followup efforts if attempts to complete the Screener result in a refusal, finalization of the case with “no answer” or “no answer, answering machine” status, or finalization of the case as a “maximum call” case with at least 14 call attempts. For the remaining 40 percent of the sample, no followup efforts will be attempted for the Screener if any of these conditions occur. The cases receiving followup attempts will be appropriately weighted to account for the cases that are subsampled out.^²² These procedures apply only to Screener cases; all extended interview cases will be fielded using procedures similar to those used in the past including refusal conversion and refielding of maximum call cases.

B.1.1.3 Methods for Improving Response Rates

Declining Screener unit response rates in NHES over the years led to the design and execution of an experiment in NHES:2003 to examine the effects of respondent incentives as a means to improve response (see Brick et al., forthcoming). The Screener incentive experiment implemented in the 2003 NHES administration included 10 conditions with varied combinations of mailing procedures (first class and priority mail) and respondent incentive amounts ($0, $2, and $5). Experimental conditions were set to study advance mailing treatments and refusal conversion treatments.

The results of this experiment indicated that small cash incentives are effective in improving NHES Screener unit response. As shown in table 3, in NHES:2003, refusals occurred less often among those who received advance incentives of $2 or $5 than those who received no incentive; the benefit of the incentive in the initial refusal rate was 5 to 7 percentage points, depending on the incentive amount. While the rate of refusal was lower for those who received an advance incentive of $5 compared with those who received $2, a diminishing effect per dollar of incentive was observed.

Table 3. Screener response rates and ever refusal rates, by incentive group: 2003

Incentive group	Sample size	Response rate		Ever refused
Incentive group	Sample size	Percent	s.e.	Percent	s.e.
1 - ($0 brochure/1st $0)	4,574	64.1	0.67	48.8	0.76
2 - ($0 /1st $2)	4,524	67.3	0.71	47.1	0.84
3 - ($0 /1st $5)	4,518	69.5	0.64	45.4	0.74
4 - ($0 /Priority $0)	5,422	63.7	0.69	46.6	0.69
5 - ($0 /Priority $2)	4,543	66.7	0.71	45.2	0.83
6 - ($2 /1st $0)	5,424	67.9	0.67	40.9	0.67
7 - ($2 /Priority $0)	4,558	68.9	0.72	42.1	0.69
8 - ($2 /Priority $2)	4,498	69.1	0.76	42.4	0.81
9 - ($5 /1st $0)	4,505	69.7	0.70	38.9	0.76
10 - ($2 /1st $2)	4,540	69.9	0.56	42.1	0.70

NOTE: The parenthetical descriptions of the experimental conditions give the advance mailing condition (before the / mark) and the initial refusal condition (after the / mark). All advance letters were sent by first class mail in a U.S. Department of Education business envelope, as were initial refusal letters in conditions not specifying priority mail. The letters sent by priority mail were sent in the U.S. Postal Service priority mail envelopes. Refusal letters included a colored NHES project brochure. Nonresidential telephone numbers are excluded from the table.

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys Program, 2003.

The NHES:2003 experiment also showed respondent incentives were effective in obtaining cooperation at the refusal conversion stage for the Screener. (See Brick et al., forthcoming.) Experimental treatment groups that included payments were more likely to respond at the first refusal conversion stage compared with those who had not received payments. In addition, the advance incentive treatments and refusal incentive treatments combined to yield higher response rates after the first refusal conversion stage for those who had received payments of $4 (two payments of $2) and those who had received $5 (in one payment).

Given the positive effects of incentives on Screener unit response, the costs of effective strategies were examined during the design of NHES:2005. Treatments that resulted in low response rates and more costly treatments that yielded results similar to less costly treatments were eliminated from future consideration. The strategy used for NHES:2005 was an advance letter with no incentive, mailed first class, and a refusal conversion letter with a $5 incentive, mailed first class, with subsampling of 60 percent of cases for nonresponse followup efforts (refusal conversion, refielding of maximum calls and no answer cases). Additionally, an experiment was conducted in NHES:2005 to compare conversion rates for Federal Express and Priority Mail for second refusal letters, and the results of that experiment will inform the final NHES:2007 procedures.

Due to concerns about sending incentives to refusal cases only, the incentive treatment proposed for NHES:2007 is an advance letter with a $2 incentive, mailed first class, and a refusal conversion letter with a $2 incentive, also mailed first class. Second refusal letters will be sent via either FedEx or Priority Mail.^²³ As discussed in section 1.2, for both alternatives, the subsampling scheme used for NHES:2005, in which 60 percent of cases are subsampled for nonresponse followup efforts, is proposed.

B.1.1.4 Field Study to Evaluate Bias in NHES:2007

Over the past several NHES survey administrations, an increasing array of methods has been used in an attempt to contact potential respondents and gain cooperation. These methods include refined call scheduling protocols, advance mailings and refusal conversion letters, refielding of cases for additional call attempts, and prepaid incentives. In an attempt to evaluate an additional method for improving survey response, the NHES:2005 Field Test included a test of the feasibility of conducting a national RDD survey that contains a subsample of nonrespondents to be contacted by in-person interviews (see Westat 2004 for details.) Further testing of the in-person followup approach was conducted during the NHES:2005 main study data collection and NHES:2007 field test.

The goals of these feasibility tests were to determine whether an in-person component would improve response rates enough to make it a cost-effective strategy for future NHES surveys and to assess the feasibility of using in-person followup with nonrespondents to study nonresponse bias. The results of this initial feasibility study were promising in some ways (field interviewers completed 41 percent of telephone maximum calls cases and 42 percent of telephone refusal cases that were determined to be residential). However, some major operational issues were identified, including the limited utility of such an approach when the study protocol greatly limits which cases can be fielded for in-person collection. As a result, an additional study was conducted in conjunction with NHES:2005 to examine operational issues and identify any key factors needing further study, to provide information on the refusal conversion rate for the cases that are not subject to the restrictions imposed in the NHES:2005 Field Test, and to provide estimates free of the compressed schedule effects of the NHES:2005 Field Test.

For the NHES:2005 study of in-person followup, a subset of cases in two predetermined sites were sent to the field after having been finalized on the initial refusal (sample release wave 2 only), after having been finalized on the third refusal (sample release wave 1 only), and after having been finalized as maximum call or language problems (both sample release waves). Of the 134 first refusal cases sent to the field, 36 cases (27 percent) were completed in the field. Of the 206 third refusal cases sent to the field, 48 cases (23 percent) were completed in the field. Among maximum call and language problem cases, 8 of the 60 cases (13 percent) that were sent to the field were completed.

For NHES:2007, a more comprehensive study to evaluate nonresponse bias will be conducted. This study will permit an assessment of bias in addition to an assessment of operational issues and cost-effectiveness of in-field followup; however, the primary purpose of the study is to evaluate bias. For the nonresponse bias study, a nationally representative three-stage sample will be selected. At the first stage, 30 primary sampling units (PSUs), which will be defined as single counties or groups of a few contiguous counties, will be randomly selected from the 50 states and DC with probabilities proportional to size. At the second stage, ten area segments (Census blocks or groups of blocks) will be selected with probability proportional to size in each sampled PSU. At the third stage, a two-phase sample will be used to select addresses, to obtain a final sample size of 250 addresses in each PSU. Lists of residential addresses within each sampled PSU will be obtained from a vendor who maintains address lists based on the U.S. Postal Service delivery files. In the first phase, a larger sample of addresses will be selected. These sampled addresses will be sent to commercial vendors to be matched to white pages telephone directory listings to obtain telephone numbers, where possible. In the second phase, within each area segment, addresses with telephone number matches will be sampled at twice the rate of addresses without telephone number matches. Exhibit 6 summarizes key design features of the nonresponse bias study.

The sample sizes for the bias study were set to allow for detection of a 5 percentage point bias in key statistics from each of the surveys.^²⁴ For example, if the work-related adult education participation rate from the standard NHES:2007 RDD survey is 40 percent, and the participation rate from the bias study is 45 percent, such a difference would be expected to be statistically significant, and the difference would be attributable to bias. Detection of a bias of 5 percentage points was set as the criterion because smaller differences are generally not substantively important in NHES.

Exhibit 6. Key design features of the NHES:2007 nonresponse bias evaluation

Design characteristic

Approach

Type of sample

Address sample

Mode of data collection

CATI (originating in TRC) for cases finalized in the TRC;

CATI (via cell phone or household land line) for cases attempted in the field

Instrument

Expanded screener and standard NHES:2007 extended interviews

Respondent

Standard NHES respondents

Cases attempted in the field

Followup with telephone nonrespondents and nonmatches

Number of PSUs (sites)

Number of screener cases attempted (total across sites)

7,500

Expected number of completed screeners/extended interviews (total across sites)

Screeners completed by phone: 2,970

Screeners completed in field: 2,265

Extended interviews completed: 1,144

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys Program (NHES), 2007.

Cases with matched telephone numbers will first be attempted by telephone, using the standard procedures used in NHES:2007. These cases will receive the full complement of telephone nonresponse followup (e.g., refusal conversion attempts, refielding of maximum call and no answer cases). Following the full protocol of mailings, incentives, and telephone attempts, in-person data collection will be attempted with four types of Screener nonresponse cases sampled for the study. These include:

Final refusal cases: Cases that have received three non-hostile refusals,^²⁵ after having experienced the mailings and hold periods designated in the NHES:2007 data collection protocol. Cases that have received one or two non-hostile refusals, but have not been completed or have received a final (third) refusal prior to reaching the NHES Screener maximum call limit of 20 call attempts.^²⁶
Maximum call cases: Cases that had at least one human contact but never refused and reached the maximum allowed number of attempts under the protocol.
Language cases: Cases that were not completed for which at least one call attempt was coded as a language problem, restricted to Spanish language only.
Cases with no correct matched telephone number: Cases for which no telephone number match could be obtained, or an incorrect or nonworking telephone number match was obtained.

The study sample will thus facilitate the analysis of nonresponse from both refusals and noncontact cases as well as undercoverage due to households that only have cell phones and households without telephones. The sample size may not be large enough to detect a 5 percentage point bias due to a particular source of error (e.g., the undercovered population may be too small by itself), but reliable estimates of overall error should be obtained. When the analysis indicates an overall bias of 5 percentage points or more, separate estimates that examine bias due to undercoverage and bias due to nonresponse will be computed.

It is expected that about 60 percent of cases sampled for the bias study (or 4,530 of the 7,500 sampled addresses) will be attempted in the field. Among cases attempted in the field, it is expected that Screeners will be completed with about 50 percent, based on the experiences in the NHES:2005 Field Test and the study of in-field followup conducted in conjunction with NHES:2005, with additional consideration of the length of the field period and the incentive (discussed below). Trained field interviewers will be provided with the address, telephone number (if available), and Telephone Center contact history for the selected cases. Field interviewers will attempt to make in-person contact with these cases and secure the household’s cooperation in the study. The field interviewers will offer a cash incentive of $20 for participation in the study. (See section A.9 for justification of the proposed $20 incentive.) The interviews will be conducted via cell phone using the NHES CATI instruments.^²⁷ An expanded Screener, containing additional questions about the household and its members, will be administered,^²⁸ and household members will be sampled for extended interviews using an algorithm similar to that used for the NHES:2007 surveys completed in the Telephone Research Center. The sampling rates applied in the within-household sampling algorithm will be designed to yield 1,144 completed extended interviews (344 SR interviews, 400 PFI interviews, and 400 AEWR interviews), assuming unit response rates of 90 percent, 83 percent, and 80 percent for the SR, PFI, and AEWR surveys, respectively.^²⁹

If household members are sampled for extended interviews, an attempt will be made to conduct the interview(s) immediately following the completion of the Screener, while the field interviewer is present in the household. When all extended interviews in the household cannot be completed immediately after the Screener, the field interviewer will make additional callbacks to the household. If the household agrees to a callback from the TRC, this will be attempted. However, if the interview is not completed as scheduled, the field interviewer will return to the household. The procedures for the nonresponse bias study were tested in the Phase 1 Field Test (see section B.4.2), and full implementation will occur in conjunction with the NHES:2007 main study.

Data from the study will be used to assess bias in NHES survey estimates due to nonresponse and noncoverage of households without landline telephones. While, from a statistical standpoint, these data could be combined with NHES:2007 data for analytic purposes, this will not be feasible under the current data release schedule (since the data collection period for the bias study will extend beyond the data collection period for NHES:2007, and the additional processing to combine the two would take additional time). Additionally, the relatively small sample sizes and the design effects due to clustering and due to differential weighting would mitigate the gains from combining the bias study sample with the NHES:2007 sample.

Since responses will not be obtained from all cases sent to the field, the bias study is itself subject to nonresponse bias. In order to mitigate the effects of potential nonresponse bias to the bias study, weighting adjustments will applied. A Chi-Square Automatic Interaction Detection (CHAID) analysis will be used to identify characteristics most associated with Screener nonresponse. These characteristics will be used to form the cells for nonresponse adjustment of the household weights. The base weights of the bias study nonrespondent cases will be distributed to the base weights of the respondent cases within the nonresponse adjustment cell. A similar process will be used to adjust for nonresponse at the extended interview level.

To further facilitate the assessment of bias, additional measures will be used to obtain information about cases that are nonrespondents to the field effort. These additional measures include a brief postpaid form asking for a few household characteristics, and an interviewer form used to report select characteristics of the neighborhood and the residence (to be completed by the interviewer, for both responding and nonresponding cases). After the final in-person contact attempt, the interviewer will return to the household one final time to leave the form on the doorstep with a $5 incentive. The form will contain a brief questionnaire, aimed at capturing a few household characteristics that may be associated with nonresponse, including the number of persons in the household, the number of children in the household, home tenure, and the highest level of education of any household member. The form will be addressed to Westat and will have pre-paid postage, and will include instructions for a household member to complete it and send it back. Although the preferred outcome is a completed interview, these fallback measures will allow these cases to be included in a more limited analysis of differences among telephone respondents, field respondents, and field nonrespondents.

B.1.1.5 Sampling Frame

The sampling frame for NHES:2007 RDD sample will be MSG’s Genesys frame of all telephone numbers in 100-banks with one or more telephone numbers listed in the white pages in the fourth quarter of 2006.^³⁰ MSG is a commercial firm that has produced samples of telephone numbers for previous NHES studies. The sampling frame contains estimates from the 2000 census of the race/ethnicity distributions of persons in the telephone exchange.

B.1.1.6 Number of Sampled Telephone Numbers

The primary function of the screening interview in NHES:2007 will be to assess the eligibility of members of the household for the extended interviews. As a result, the number of households that must be sampled for each type of extended interview is largely a function of the precision requirements for the extended interviews, which are discussed in the next section. The total number of completed Screeners needed in NHES:2007 is driven by the sample size requirement to produce reliable estimates for preschoolers. A target of 62,000 completed Screeners was set for NHES:2007. This number of screeners is expected to be sufficient to meet the precision requirements of the NHES:2007 surveys and accounts for expected design effects incurred as a result of differential sampling of telephone numbers, subsampling Screener cases for nonresponse followup, unequal within household selection probabilities, and the effects of weighting adjustments. Further details on the precision requirements for NHES and the sample size implications of those requirements are given here and in appendix C. The number of telephone numbers to be sampled was determined by inflating the target of 62,000 completed Screeners to account for the expected residency rates and unit response rates; in doing so, the expected effects of the incentive treatment and the subsampling of cases for followup were also taken into account.

Based on the optimal allocation used for NHES:2005, the sampling rate in the high minority concentration stratum will be approximately twice that of the low minority stratum and that in the first phase of selection, a total of about 336,371 telephone numbers will need to be selected—167,512 telephone numbers from the high minority stratum and 168,859 telephone numbers from the low minority stratum.

In the second phase, within each minority stratum, the sampled telephone numbers will be stratified as mailable or nonmailable according to whether a mailing address was able to be matched to the telephone number. Within each of the four strata defined by the combinations of minority concentration and mailable status, telephone numbers will be subsampled at different rates. To attain the sample sizes and optimal allocation under the stratification based on minority concentration and mailable status, it is estimated that a total of about 262,885 telephone numbers will need to be sampled for NHES:2007. The expected numbers of sampled telephone numbers and completed Screeners, as well as residency and Screener unit response rates, are shown by stratum in table 4. All differential sampling, including differential sampling of telephone numbers based on minority concentration and mailable status, will be properly accounted for in the calculation of base weights.

Table 4. Expected numbers of sampled telephone numbers and completed Screeners, and expected residency and Screener unit response rates, by stratum: 2007

Stratum	Expected number of sampled telephone numbers*	Expected percent of sampled telephone numbers that are residential	Expected Screener unit response rate (percent)	Expected initial cooperation rate (percent)	Expected number of completed screeners

Total	262,885	43	64	40	62,000

Mailable, High minority	65,388	73	63	39	25,628
Mailable, Low minority	69,949	76	69	42	31,076
Not mailable, High minority	68,899	11	44	36	3,089
Not mailable, Low minority	58,649	9	46	36	2,207

* Phase 2 sample only. Figures do not include reserve sample or cases in the Phase 1 sample that are not selected into the Phase 2 sample.

NOTE: These figures assume 62,000 screened households for NHES:2007. The figures given here are also based on the proposed subsampling of 60 percent of cases for nonresponse followup efforts. Figures given in this table are not final; they will be revised prior to selection of the NHES:2007 sample to reflect updated figures from the MSG database.

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys Program (NHES), 2007.

To facilitate the release of additional telephone numbers, in the event that sample yield falls considerably below expectations, a reserve sample of telephone numbers will be selected for NHES:2007. The reserve sample will be 20 percent of the size of the basic sample, and will be selected using the same sampling rates used to select the original sample. Like the basic sample, the reserve sample of telephone numbers will be matched to address listings and subsampled based on mailable status. Because this reserve sample will be selected using the same sampling rates as the original sample, if the same within-household sampling algorithm and data collection protocols are applied to the reserve sample, it would be expected to yield numbers of completed interviews that are approximately 20 percent of those yielded by the original sample if the full data collection protocol was used.^³¹

B.1.2. Within-Household Sampling

Persons within households that have a completed Screener will be sampled for the SR, PFI, and AEWR surveys. One key criterion in the development of the sampling scheme for NHES:2007 is minimizing respondent burden. Considerations of the numbers of persons within a household sampled for extended interviews and the combinations of extended interviews also weigh heavily in the development of the sampling scheme.

B.1.2.1 Precision Requirements

The general precision requirement for all three surveys is the ability to detect a 10 to 15 percent relative change for an estimate of between 30 and 60 percent. The following paragraphs provide further detail on more specific requirements for each survey. In NHES:2007, the overall screening sample will be largely determined by the need to produce precise estimates of indicators for children, particularly preschoolers (age 3–6, not yet in kindergarten)^³². The focus on preschoolers stems from the fact that they have the lowest prevalence in households among the subdomains of children (as discussed in section 3). It is useful to assess how the NHES:2007 sample can be combined with estimates from earlier NHES surveys to examine change over time. In a simple comparison, a t-test statistic is

where p is the estimated percentage, d is the design effect, n is the sample size, and the subscripts 1 and 2 denote the two time periods. The current survey’s sample size requirements for detecting change are highly dependent on the sample sizes and precision achieved in previous surveys. Thus, increasing the sample size in NHES:2007 drastically above the levels of previous surveys will not substantially improve the precision of estimates of change over time. However, one important consideration is that if larger sample sizes are anticipated for future surveys, then having larger sample sizes in NHES:2007 will facilitate the detection of change over time in the future.

Of course, the t-statistic is only one of the many methods that can be used to detect and characterize change over time with data from NHES. Regression analysis or simple trend analyses of the various surveys over time are other ways of analyzing these data. For nearly all the methods, increasing sample sizes drastically over those in previous survey administrations does not result in large increases in the power or the precision of the estimates.

The sample requirements for estimates of change are more stringent than those for cross-sectional estimates. Bearing in mind the effects of sample sizes from previous administrations on the capacity to detect change over time, the sample size requirements for key estimates were derived. For the SR survey, key sample size determinants were the requirements to detect changes in estimates of type of care arrangement, literacy, and numeracy, by single year of age and by race/ethnicity (White, non-Hispanic; Black, non-Hispanic; and Hispanic). The key estimates were selected to represent statistics that have been published from the SR-NHES:1993, Parent-NHES:1999, and ECPP-NHES:2001 surveys, and to reflect topics of interest to experts in the field. The subgroups were chosen because they are key subgroups used in analyses of NHES data for preschoolers.

For the PFI survey, the key estimates considered in designing the sample were the percentage of children whose parents participate in three or more activities in the child’s school, the percentage of children whose parents report that school practices^³³ are done very well, and the percentage of children whose parents participated in six or more home learning activities;^³⁴ the key analytic subgroups were race/ethnicity (White, non-Hispanic; Black, non-Hispanic; and Hispanic), 2-year grade groups with kindergarten as a separate group, parents’ educational attainment (high school diploma or below, beyond high school diploma), school type (public, private), and school size (under 300; 300-599; 600-999; 1,000 or more). The key estimates were selected to represent statistics that have been published from the PFI/CI-NHES:1996 and PFI-NHES:2003 surveys, to reflect topics of interest to experts in the field, and to include measures of both in-school and out-of-school involvement. The subgroups were chosen because they are key subgroups used in analyses of NHES data for school-age children.

As a result of the analysis of the precision requirements, target sample sizes (in terms of numbers of completed interviews) of about 3,790 for the SR Survey and 14,150 for the PFI Survey were established. Details of the derivation of these sample sizes are provided in appendix C.

For the AEWR survey, the key sample size determinants are the requirements to detect changes in estimates of the percentage of adults who participate in work-related adult education activities and the percentage of adults who participate in employer-supported AEWR. The key analytic subgroups were race/ethnicity (White, non-Hispanic; Black, non-Hispanic; and Hispanic), employment status (employed, unemployed but looking for work), and educational attainment (less than high school; high school and higher). A sample size of about 32,700 completed AEWR interviews is required to meet the precision requirement for all of these characteristics. The estimates with the most stringent sample size requirements are AEWR participation estimates for Hispanics and for adults who are unemployed but looking for work. The latter subgroup requires a relatively high overall number of completed interviews because adults who are unemployed but looking for work comprise such a small proportion of the adult population (about 6 percent). With a sample size of 15,000 completed AEWR interviews, the precision requirement can be met for all estimates considered with the exception of these two.

It should be noted that many of the key characteristics from the AEWR and PFI Surveys fall outside the 30 to 60 percent range specified in the precision requirement. Larger sample sizes than those required to meet the minimum precision requirement are needed in order to measure change in many key statistics that fall outside the 30 to 60 percent range. As noted above for unemployed adults who are looking for work, extraordinarily large sample sizes would be needed in order to measure these key statistics for some groups, and for some small groups (e.g., those defined by race and ethnicity) no sample size would be adequate to assess the relative change specified in the precision requirement. Response burden considerations and cost considerations were also considered in establishing the final sample size targets.

Thus, in order to improve the precision of estimates of characteristics that fall outside the 30 to 60 percent range, the target sample size for the AEWR Survey was set at 15,000 completed interviews. Adult education participants will be sampled at a higher rate than nonparticipants in order to improve the precision of estimates of characteristics of participants. Details of the derivation of sample sizes for adults are given in appendix C.

The sample requirements for the extended interviews were determined based on a set of assumptions about extended interview unit response rates.^³⁵ Specifically, the assumed unit response rates are 86 percent for the SR survey and 83 percent for the PFI survey. For the AEWR survey, the assumed unit response rate is 80 percent for adults sampled as adult education participants, and 74 percent for adults sampled as nonparticipants.

B.1.2.2 Sampling Scheme for Within-Household Sampling

The sampling scheme for within-household sampling is designed to satisfy the sample requirements discussed earlier while keeping the respondent burden to a minimum. The following are the primary goals and features of the sampling scheme for within-household sampling in NHES:2007:

No more than three persons will be sampled in a given household.
Exactly one preschooler will be sampled in every household that has at least one, and exactly one child enrolled in kindergarten through twelfth grade will be sampled in every household that has at least one.
Because adult education participants are of particular interest, they will be sampled at a higher rate than other adults.
In households with eligible children, adults will be sampled at lower rates than in households without eligible children. Additionally, adults in households with children sampled for both SR and PFI interviews will be sampled at about half the rates of adults in households with only one child sampled.

To carry out this sampling scheme, several flags and/or random numbers will be set prior to screening (i.e., at the time the sample of telephone numbers is drawn). The first will specify whether the adult sampling algorithm (to determine whether an adult is selected) is to be run for the particular household. Each telephone number will receive one of three possible designations:

1) Household is designated for the adult sampling algorithm to run;

2) Household is designated for the adult sampling algorithm to run only if there are no eligible children in the household; or

3) Household is not designated for the adult sampling algorithm to run.

This flag will be set such that households with eligible children are designated for adult sampling at one-half the rate of households without eligible children (about 27 percent vs. 55 percent).

In the NHES:2001, NHES:2003, and NHES:2005 survey administrations, the Screener contained a “screen-out” question to determine whether there are any eligible children in the household. The response to that question and the values of the aforementioned sampling flags determined the extent of the household enumeration. Because a child will be sampled in every household containing an SR- or PFI-eligible child, NHES:2007 will feature full enumeration in all households with children, and in households without children that are designated for the adult sampling algorithm to run. That is, the only households that will be screened out in NHES:2007 are households without children that are predesignated for no adult sampling. (These households are expected to comprise about 31 percent of screened households.)

Following the enumeration of children, if the household has at least one preschooler, then exactly one will be randomly sampled for the SR survey. If the household has at least one child ages 3 through 20 enrolled in kindergarten through twelfth grade, then exactly one will be randomly sampled for the PFI survey. For each survey, pre-assigned random numbers will be used to sample from amongst all eligible children in the household.

In households in which an adult is to be sampled, adult education participants will have twice the probability of selection of nonparticipants. Exhibit 7 shows all possible combinations of household compositions for sampling adults based on the presence of children in the household and adult education participation status, with the respective domain probabilities of selection for adults. The maximum rate at which adults in households without children will be sampled is 55 percent. That is, in 45 percent of households without children, no enumeration will be required. Further details about the differential sampling of adults are given in section 2 of appendix C.

Exhibit 7. Overview of the sampling scheme for selecting adults based on household composition

Child in household	Household composition		Domain probability of selection
Child in household	Adult education participant	Adult education nonparticipant	Adult education participant	Adult education nonparticipant

No			0	0.2728
No			0.5456	0
No			0.3637	0.1819
Yes			0	0.1364
Yes			0.2728	0
Yes			0.1819	0.0909

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys Program (NHES), 2007.

B.1.3. Expected Yield

This section presents the expected yield for each extended interview survey.

B.1.3.1 SR and PFI Surveys

The SR and PFI interviews will be conducted with the parents of a sample of preschoolers and children ages 3 through 20 enrolled in kindergarten through twelfth grade, respectively. Estimates from the October 2003 CPS were used to determine the sampling rates for sampling children for the SR and PFI surveys and to develop the sampling scheme.

Tabulations of the October 2003 CPS data showed that about 32 percent of households are expected to have at least one eligible child. Estimates of the percentage of households with eligible children or youth by age/grade group are given in table 5. To balance screening requirements against household burden and minimize the effect of intra-household clustering, the sampling scheme for NHES:2007 will involve sampling one preschooler and one child enrolled in grades kindergarten through 12 in every household in which a child in either domain is present.

Table 5. Percentage of telephone households with eligible children, by age/grade group: CPS 2003

Household composition	Percent of households

Households with no eligible children	68.3

Households with eligible children	31.7
Households with at least one child ages 3 through 6 and not yet in kindergarten, and no child enrolled in grades kindergarten through 12	3.2
Households with at least one child enrolled in grades kindergarten through 12, and no child ages 3 through 6 and not yet in kindergarten	24.3
Households with at least one child ages 3 through 6 and not yet in kindergarten, and at least child enrolled in grades kindergarten through 12	4.1

SOURCE: U.S. Department of Commerce, U.S. Census Bureau, Current Population Survey (CPS), October 2003 School Enrollment Supplement data file (independent tabulations).

Table 6 shows the expected number of screened households based on the distribution of household composition shown in table 5. The majority of screened households (about 42,000 households) are expected to have no eligible children. Thus, the sampling scheme for within-household sampling was developed such that the screened households with children (about 20,000 households) will provide the sample sizes needed to meet the precision requirements while minimizing respondent burden.

B.1.3.2 AEWR Survey

Persons 16 years or older who are not enrolled in grade 12 or below, not institutionalized, and not on active duty in the U.S. Armed Forces will be eligible for the AEWR-NHES:2007 survey. Because sampling adults for AEWR interviews is required in only about one-third of screened households, a subsample of households without eligible children will be designated for adult enumeration.

Table 6. Expected number of screened households in NHES:2007, by household composition: CPS 2003

Household composition	Expected number of households

Households with no eligible children	42,398

Households with eligible children	19,602
Households with at least one child ages 3 through 6 and not yet in kindergarten, and no child enrolled in grades kindergarten through 12	1,985
Households with at least one child enrolled in grades kindergarten through 12, and no child ages 3 through 6 and not yet in kindergarten	15,084
Households with at least one child ages 3 through 6 and not yet in kindergarten, and at least child enrolled in grades kindergarten through 12	2,533

NOTE: The distribution in this table assumes 62,000 screened households for NHES:2007.

SOURCE: U.S. Department of Commerce, U.S. Census Bureau, Current Population Survey (CPS), October 2003 School Enrollment Supplement data file (independent tabulations).

Table 7 shows the expected number of adults sampled for an AEWR interview, by number of adults in the household and presence of eligible children. Based on the sampling scheme described earlier, it is expected that 10,441 adults will be sampled as adult education participants and 10,543 adults will be sampled as nonparticipants. In NHES:2003, about 18 percent of those sampled as adult education nonparticipants who completed extended interviews were found to be AEWR participants, and about 23 percent of persons sampled as participants who completed extended interviews were identified as AEWR nonparticipants. Taking into account the NHES:2003 “switching” rates and assuming unit response rates (based on the AEWR-NHES:2003 unit response rates and experience from NHES:2005) of 74 percent for adults sampled as participants and 69 percent for adults sampled as nonparticipants, it is expected that about 7,295 AEWR interviews will be completed with AEWR participants and about 7,707 AEWR interviews will be completed with nonparticipants. Unit response rates in recent NHES surveys of adults have remained relatively constant over time, and the NHES:2007 screening approach and AEWR Survey are similar to those in NHES:2003; thus, it is reasonable to assume that the unit response and switching rates for the AEWR Survey will be similar to those in the AEWR-NHES:2003 Survey.

Table 7. Expected number of adults sampled for AEWR interviews, by number of adults and presence of eligible children in household: 2007

Number of adults in household	Children in household?	Expected number of sampled adults
Number of adults in household	Children in household?	Sampled as adult education participants	Sampled as nonparticipants	Total

1	Yes	412	279	691
1	No	2,215	3,339	5,553
2	Yes	1,639	1,156	2,795
2	No	4,115	4,473	8,588
3	Yes	394	248	642
3	No	1,002	723	1,725
4	Yes	120	63	183
4	No	413	198	611
5 or more	Yes	40	20	61
5 or more	No	91	44	135
Overall		10,441	10,543	20,984

NOTE: The distributions in this table assume 62,000 screened households for NHES:2007 Due to rounding, details may not add to totals.

SOURCE: U.S. Department of Commerce, U.S. Census Bureau, Current Population Survey (CPS), October 2003 School Enrollment Supplement data file (independent tabulations).

B.1.4. Summary of the Sample Design

Table 8 summarizes the expected numbers of completed interviews for the RDD sample to be selected for NHES:2007. As shown in table 8, the expected numbers of completed interviews are 3,790 for SR, 14,150 for PFI, and 15,000 for AEWR.

To facilitate comparison with previous NHES administrations, expected numbers of persons sampled for extended interviews in NHES:2007 are given in table 9, along with numbers of persons sampled for extended interviews in NHES:1991, NHES:1993, NHES:1995, NHES:1996, NHES:1999, NHES:2001, and NHES:2003, and the expected number for NHES:2005. Table 10 gives the expected numbers of completed interviews in NHES:2007, along with the actual numbers of completed interviews in NHES:1991, NHES:1993, NHES:1995, NHES:1996, NHES:1999, NHES:2001, and NHES:2003, and the expected number of completed interviews in NHES:2005. Appendix C contains details about the expected precision of estimates from the three NHES:2007 surveys.

Table 8. Expected numbers sampled and expected numbers of completed interviews in the telephone sample for NHES:2007

Sample population	Expected number sampled	Expected number of completed interviews

Household Screeners	100,846	62,000

SR	4,518	3,790

PFI	17,617	14,150

AEWR
Total adults	20,984	15,000
Total AEWR participants	†	7,295
AEWR participants sampled as participants	†	5,954
AEWR participants sampled as nonparticipants	†	1,341
Total AEWR nonparticipants	†	7,707
AEWR nonparticipants sampled as participants	†	1,773
AEWR nonparticipants sampled as nonparticipants	†	5,934

† Not applicable.

NOTE: Due to rounding, details may not add to totals.

SOURCE: U.S. Department of Commerce, U.S. Census Bureau, Current Population Survey (CPS), October 2003 School Enrollment Supplement data file (independent tabulations).

B.1.5 Supplemental Homeschooler Sample

NHES is the only source of data on homeschooling collected from a nationally representative sample of households. A question has been raised as to whether homeschooling families are less likely to respond to telephone surveys about education, since they have chosen to educate their children outside of the formal public and private school system. The NHES design permits the examination of differences in response rates to the PFI extended interview once a child has been sampled. However, it is not possible, from the RDD sample alone, to ascertain whether homeschooling families are less likely to complete the NHES Screener stage.

In order to determine whether screening-level unit response rates differ between homeschooling families and the population as a whole, a seeded sample will be included in the NHES:2007 data collection. The homeschool seeded sample is a methodological supplement and those cases will not be part of the public-use data analysis file. Under a separate contract, NCES has worked with a contractor to identify available lists of homeschooling families. These include extensive lists of those who belong to homeschooling organizations, have attended homeschooling conferences, and have purchased homeschooling materials. The selected lists are from Response Unlimited and include Home Schooling Families as the primary list and Christian Home School Connection Buyers and Home Schooling Today Magazine as supplementary lists. A sample will be selected from the lists, matched to telephone numbers, and included in the NHES:2007 data collection.

The goal of this investigation is to detect a difference of 3 percentage points in the Screener unit response rate between the homeschooling population and the overall population. In order to arrive at the total seeded sample size, some assumptions were required. We have assumed that about 70 percent of the selected sample will have correct telephone matches (“true matches”), that about 75 percent of households in that “true match” group will contain current homeschoolers at the time of the NHES survey, and that the overall RDD Screener response rate will be about 60 percent. The total sample size required to detect a 3 percent difference is 2,420 cases, yielding 1,694 “true matches,” and resulting in about 762 completed screeners. One child will be sampled in each household; assuming a PFI unit response rate of 83 percent, 633 completed PFI interviews are expected from the supplemental homeschooler sample.

B.1.6 Reinterview Sample

As noted previously, NHES:2007 will include a brief reinterview for the Parent and Family Involvement in Education Survey. The nature of the questions to be addressed in the reinterview requires that completed PFI interviews be sampled at different rates based on responses given in the original interview, so that sufficient numbers of cases of particular types are included in the reinterview pool. The total sample selected for the reinterview will be approximately 1,250, in order to yield 1,000 completed reinterviews. In order to assess the response variability of items concerning school choice, the reinterview sample will be chosen so that approximately one-half of the sample is composed of students who attend their assigned public schools, about one-fourth is composed of those attending public schools chosen by their families, and about one-fourth is composed of private school students. In order to support the analysis of the tutoring items, one-third of the sample will be selected from among those students who have received some tutoring services in the current school year.

Table 9. Comparison of expected numbers of persons sampled for extended interview in NHES:2007 to the numbers sampled in previous survey administrations

Sample	Survey administration
Sample	NHES: 1991 (actual)	NHES: 1993 (actual)	NHES: 1995 (actual)	NHES: 1996 (actual)	NHES: 1999 (actual)	NHES: 2001 (actual)	NHES: 2003 (actual)	NHES: 2005 (actual)	NHES: 2007 (expected)


Number of completed Screeners	60,314	63,884	45,465	55,838	55,929	48,385	32,049	58,140	62,000
Number of persons sampled for an extended interview
Total	34,118	27,437	40,319	26,435	36,125	32,966	30,946	33,901	43,119
Infants (0–2 yrs.)	(†)	(†)	4,341	(†)	3,435	5,750	(†)	4,253	(†)
Preschoolers (3–not yet in K)	9,925¹	5,635	4,372	3,594⁴	4,316	2,223⁷	(†)	4,228¹⁰	4,518
Grades K–2	9,967¹	7,270²	5,227	4,460	4,841	2,745⁸	3,470⁹	3,741¹¹	3,902
Grades 3–5	(†)	2,882	1,841³	4,847	4,788	2,967	3,395	3,918¹²	4,206
Grades 6–12	(†)	11,650	(†)	10,934	10,631⁵	5,423⁶	8,077	5,951^6, 13	9,509
Adults	14,226	(†)	24,538	2,600	8,114	13,858	16,004	11,810	20,984
Adult education participants	12,464	(†)	14,355	--	4,542	6,615	8,264	5,265	10,441
Adult education nonparticipants	1,730	(†)	10,183	--	3,572	7,243	7,740	6,545	10,543

† Not applicable; persons in this category were not eligible for extended interviews.

--These categories are not applicable because the NHES:1996 survey was not an adult education survey.

¹The sample size for “preschoolers” is actually strictly 3–5 years old, regardless of enrollment status; this sample size includes 2,959 ineligible children. The sample size for “grades K–2” is actually strictly 6–9 years old, regardless of enrollment status or grade; this sample size includes 1,798 ineligible children and 22 of unknown age.

²The sample size for grades K–2 includes 158 children who were enrolled in transitional kindergarten, prefirst, special education, or ungraded.

³The sample size for grades 3–5 includes only 3rd grade; this sample size includes 36 children enrolled in special education or ungraded.

⁴The sample size for preschoolers includes children up to age 7 who are not enrolled.

⁵The sample size for grades 6–12 includes 5 children whose grade was unknown and 9 children who were enrolled in special education or ungraded.

⁶This sample size reflects only middle schoolers (grades 6–8).

⁷The sample size for preschoolers includes 3 children with unknown enrollment status.

⁸The sample size for grades K–2 includes 38 children with unknown grade and 5 children who were ungraded or in special education.

⁹The sample size for preschoolers includes 82 children with unknown enrollment status, in special education or ungraded.

¹⁰ The sample size for preschoolers includes 7 children with unknown enrollment status, in special education, or ungraded.

¹¹ The sample size for grades K-2 includes 8 children with unknown enrollment status.

¹² The sample size for grades 3-5 includes 12 children in unknown enrollment status, in special education, or ungraded.

¹³ The sample size for grades 6-8 includes 36 children with unknown enrollment status, in special education, or ungraded.

NOTE: The distributions in this table for NHES:2005 assume 59,380 screened households. The distributions in this table for NHES:2007 assume 62,000 screened households. Due to rounding, subdomain counts may not add to totals.

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys Program (NHES), 1991-2003.

Table 10. Comparison of expected numbers of completed interviews in NHES:2007 to the numbers completed in previous survey administrations

Sample	Survey administration
Sample	NHES: 1991 (actual)	NHES: 1993 (actual)	NHES: 1995 (actual)	NHES: 1996 (actual)	NHES: 1999 (actual)	NHES: 2001 (actual)	NHES: 2003 (actual)	NHES: 2005 (actual)	NHES: 2007 (expected)


Number of completed Screeners	60,314	63,884	45,465	55,838	55,929	48,385	32,049	58,140	62,000
Number of completed extended interviews
Total	26,460	23,568¹	33,786	23,042¹	31,297¹	27,205	25,151	27,797	32,940
Infants (0–2 yrs.)	(†)	(†)	4,135	(†)	3,378	3,599	(†)	3,855	(†)
Preschoolers (3–not yet in K)	5,085²	4,424	3,429	3,012⁵	3,561	3,150	(†)	3,354⁶	3,790
Grades K–2	7,322²	6,447³	4,830	4,037	4,330	2,350⁶	2,834⁶	3,212	3,212
Grades 3–5	1,464	2,580	1,666⁴	4,348	4,182	2,559	2,837	3,363⁶	3,354
Grades 6–12	(†)	10,117	(†)	9,389⁶	9,140⁶	4,670⁷	6,751	5,109⁷	7,584
Other/unknown	21	0	4	6	9	4	4	0	0
Adults	12,568	(†)	19,722	2,250	6,697	10,873	12,725	8,904	15,000
Adult education participants	11,149	(†)	11,890	--	3,953	5,348	6,738	4,732	7,295
Adult education nonparticipants	1,419	(†)	7,832	--	2,744	5,525	5,987	4,172	7,707

† Not applicable; persons in this category were not eligible for extended interviews.

--These categories are not applicable because the NHES:1996 survey was not an adult education survey.

¹ Excludes extended interviews completed with sampled older children (Youth).

²The sample size for “preschoolers” is actually strictly 3–5 years old, regardless of enrollment status. The sample size for “grades K–2” is actually strictly 6–9 years old, regardless of enrollment status or grade.

³The sample size for grades K–2 includes children who were enrolled in transitional kindergarten, prefirst, special education, or ungraded.

⁴The sample size for grades 3–5 includes only 3rd grade; this sample size includes children enrolled in special education or ungraded.

⁵The sample size for preschoolers includes children up to age 7 who are not enrolled.

⁶The sample size includes children who were enrolled in special education or ungraded, distributed to the modal grade for their age.

⁷This sample size reflects only middle schoolers (grades 6–8).

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys Program (NHES), 1991-2003.

B.1.7 Estimation Procedures

The estimation weights for the NHES:2007 survey will be formed in stages. The first stage is the creation of a base weight for the household, which is the inverse of the probability of selection of the telephone number. The second stage is the adjustment of the base weights for households with multiple telephone numbers. The third stage is a non-response adjustment. The fourth stage is the poststratification adjustment of the weights to Census Bureau estimates of household totals by household demographic characteristics. Research will be done regarding the choice of household-level variables to use for this adjustment in light of any new findings on characteristics associated with household-level landline telephone coverage; variables that may be used include region and presence of children in the household. These household-level weights implicitly include nonresponse and undercoverage adjustments. National household-level estimates may be produced using these final, poststratified household weights.

The poststratified household-level weights are the base weights for the person-level weights. For each extended interview, the person-level weights also undergo a series of adjustments. The first stage is the adjustment of these weights for the probability of selecting the person within the household. The second stage is the adjustment of the weights for nonresponse. The third stage is the raking adjustment of the weights to Census Bureau estimates of the target population. The variables that may be used for raking at the person level include race and ethnicity of the sampled person, household income, home tenure (own/rent/other), region, age, grade of enrollment, gender, family structure (one parent or two parent), and education level. These include important analysis variables (e.g., family structure) and characteristics that have been shown to be associated with telephone coverage (e.g., race/ethnicity). The final, raked person-level weights implicitly include undercoverage adjustments as well as additional implicit adjustments for nonresponse.

Standard errors of the estimates will be computed using a jackknife replication method. The replication process repeats each stage of estimation separately for each replicate. The replication method is especially useful for obtaining standard errors for complex statistics such as quantiles. The standard errors may be computed using the complex survey data analysis package WesVar Complex Samples Software or other software packages that use replication methods such as Stata, SAS, SUDAAN or the AM software package. Also, PSU and STRATUM variables will be available on the data files for users who wish to use Taylor series linearization to compute standard errors.

B.2. Survey Procedures

This section describes the data collection procedures to be used in NHES:2007. The NHES data collection approach has evolved over time to include mailings, extended calling protocols, multiple stages of refusal conversion, and modest cash incentives. The NHES:2007 approach builds upon the experience gained in the prior NHES collections, and the procedures are designed to maximize the survey response rate. Some new approaches have been incorporated in order to meet the challenges of conducting the NHES collection in a relatively brief time frame, expand the pool of potential interviewers, and increase operational efficiency.

As discussed in Part A of this clearance submission, the interviews include:

A Screener, to determine whether eligible persons live in the household and to sample persons for extended interviews;
The SR/PFI interview, to be conducted with the parents/guardians of children from age 3 through 12th grade; and
The AEWR interview, to be conducted with civilian, noninstitutionalized adults who are age 16 and older and not enrolled in grade 12 or below.

The interviews will be conducted in the following sequence. First, the brief household screening interview (Screener) will be administered to an adult household member. Upon completion of the Screener, the contact with the household will be terminated if no household members have been sampled for extended interviews. If one or more household members have been sampled for interviews, the interviewer will continue with any interviews that are to be conducted with the Screener respondent. Following completion of any interviews with the Screener respondent, or if no interviews are to be conducted with him or her, the interviewer will ask to speak with any other selected respondents in the household. Callback appointments will be made as necessary to reach respondents for extended interviews if they are not available at the time the Screener is completed.

Conducting Telephone Interviews

An advance letter will be sent to all cases for which a telephone-address match has been made (see Appendix B). The letter, on U.S. Department of Education stationery, will explain the purpose and sponsorship of the study, and a cash incentive of $2 will be enclosed. The letter will be sent via first class mail, and mailings will be staggered so that receipt of the letter is fairly close in time to the initial call into the household.

It is not possible to sent an advance letter to all cases in the RDD sample, because all telephone numbers cannot be matched to addresses. We are considering using an Interactive Voice Response (IVR) system to deliver an advance announcement to a sample of telephone numbers to ascertain its usefulness in notifying potential respondents of the survey and its effect on initial cooperation rates. This brief advance message would introduce the study and its sponsorship and inform respondents that they will be receiving a call from an interviewer. Those receiving the message would be able to press zero to speak with a staff member if they wish. We are considering using this approach for a 50 percent sample of telephone numbers without an address match and a small sample of about 5,000 telephone numbers with address matches.

As in prior NHES collections, interviews will be conducted at Westat’s Telephone Research Centers (TRC). In addition, in NHES:2007 we will take advantage of new technology that allows for a distributed interviewing approach, in which some of the interviewing staff will conduct interviews from their homes. The home interviewing approach involves the recruitment of interviewing staff who have a suitable secure and quiet space that is dedicated to this activity, appropriate computer hardware, and high-speed internet access in their homes, in addition to meeting the usual requirements for interviewer qualifications. More of the technical details of the at-home environment are discussed below.

The NHES training approach will be somewhat different than in the past. The new approach will be more efficient and productive and ensure equivalent training for both the center-based and the distributed interviewers. A web-based home study program using interactive tutorials and exercises will be completed prior to the beginning of the trainer-led sessions. Interviewers at the TRCs will complete the same training either in their homes or at the TRCs, but then will be administered the trainer-led sessions in person at the centers as in the past. Those interviewing from their homes will be trained on-line and their trainer-led sessions will be done via computer and voice lines in small groups, with the capability to ask questions or report problems using instant messaging. All trainees will complete role plays that will be monitored and evaluated by TRC supervisors and all will complete required study tests.

Interviewers will connect to the training website and the data collection site using Citrix. When using a Citrix ICA (Independent Computing Architecture) Client, all application logic executes on the server and only screen updates, mouse movements, and keystrokes are transmitted via the Citrix ICA session. Thus, while the server-based CATI application would appear to run locally on the client machine, in reality the application and data reside on the server. Citrix is highly secure and fully encrypted with built-in 128-bit AES end-to-end encryption.

All interviewers, whether in the TRC or in their homes, will be using the same secure application and all data will reside on Westat servers. Distributed interviewers will not have a way to download respondent information to their home computers, nor will any respondent information be sent to them in hard copy. The interviewing staff will work from a single database and call scheduler, which maximizes the efficiency of the case management process.

Because the sample will span four time zones, Westat intends to maximize the number of hours that the interviewing staff is scheduled to cover and will make full use of evening and weekend hours, when respondents to household surveys are most likely to be available. In household telephone surveys, evening and weekend calls are the most productive because adult household members are more likely to be home at these times than during weekdays. The exceptions are Friday and Saturday evenings. As a result, more staff will be assigned to weekday evenings and weekend days.

Expected hours of TRC operation will be Monday through Friday 9:00 a.m. to 12:00 midnight, Saturday 10:00 a.m. to 9:00 p.m., and Sunday 2:00 p.m. to 10:00 p.m. (in local time zones of each TRC). Unless an appointment is specifically requested at another time, respondents will be called only between 9:00 a.m. and 9:00 p.m. on weekdays, 10:00 a.m. to 6:00 p.m. on Saturday, and 2:00 p.m. to 9:00 p.m. on Sundays in their own time zones. Home interviewers will also be scheduled during these periods. Hours of operation are periodically extended in order to reach respondents in Hawaii and Alaska.

Telephone interviewers will initially make at least four attempts to screen households in order to determine the presence of eligible household members. These calls will be staggered on different days of the week and at different times of the day. An approach that is being considered for NHES:2007 is the use of predictive dialing for cases not contacted after four attempts. Specifically, if no contact has been made with a person at a sampled telephone number after four attempts, the case may be set aside, and such cases may be sent in batches to Westat’s subcontractor for additional calls to be made by predictive dialing. Predictive dialing involves the dialing of telephone numbers by a computer and the transfer of a case to a staff member only if a person answers the telephone. The cut-off of four call attempts was chosen because our experience in NHES is that more than 90 percent of households at which a Screener is completed have had contact with a household member by the fourth call. If the predictive dialing effort results in a contact, the respondent will be asked if they are willing to complete the interview at that time and the call will be forwarded to a trained NHES interviewer. This approach maximizes the effective use of trained interviewer time by focusing their efforts on cases they are more likely to contact and complete, rather than on noncontact cases they are unlikely to reach. It also improves interviewer morale by allowing them to spend more time working cases in which they are actually speaking to respondents.

Once contact is made with a household member, a total of 20 call attempts will be made to complete the Screener in the 60 percent of the sample selected for nonresponse followup, and up to 14 attempts will be made to complete the Screener in the remaining 40 percent of the sample. Screener cases that reach this number of attempts without an interview being completed will be designated as “maximum call” cases. Once household members have been sampled for extended interviews, up to 24 total attempts will be made to complete SR, PFI, and AEWR interviews.

Westat’s CATI system will schedule cases automatically based on an algorithm that will be customized for NHES:2007. CATI will assign cases to interviewing time periods in the following order of priority:

Cases that have specific appointments;
Cases that resulted in a busy signal earlier in the same time period;
Cases that have unspecified appointment/general callback;
Cases that were previously attempted with no contact; and
Cases that are new and have never been worked.

When potential respondents are encountered who speak Spanish but not English, an interview will be conducted in Spanish. Non-Spanish speaking interviewers will code a case as a “language problem” when they encounter a non-English-speaking respondent and cannot identify an English-speaking household member. (Spanish-speaking interviewers will code a case a language problem if no household member speaks either English or Spanish.) All cases designated as a language problem will be called by a bilingual (English/Spanish) interviewer. An interview will be conducted in Spanish if that is the respondent’s language; all other non-English languages will receive a final disposition of “language problem.” The Spanish versions of the interviews will be programmed in CATI following the programming of the English interviews. Bilingual interviewers will be able to switch to the Spanish version during the interview process so that they can conduct an interview in Spanish without the respondent having to be called back.

When a refusal is received from a case in the 60 percent of the RDD sample that is subsampled for refusal conversion or in the bias study address sample, a refusal conversion letter and $2 incentive will be sent by first class mail, and an interviewer will call after a hold period of about two weeks and attempt to obtain the completed Screener. If a second refusal is received, a letter will be sent via FedEx and another refusal conversion attempt will be made after another hold period of about two weeks.

All cases in the bias study address sample and in the supplemental homeschooler sample will be assigned to the same telephone interviewing protocol as the 60 percent of RDD sample cases designated for nonresponse followup efforts. That is, they will receive refusal conversion mailings and calls and will have a maximum Screener call limit of 20 call attempts.

Conducting Interviews During In-Person Collection

Four groups of address sample cases will be assigned to in-person data collection: those with no matched telephone number, those with incorrect or nonworking matches, those that refuse at the Screener level during telephone collection, and those that are not completed at the Screener level during telephone collection despite many contact attempts (that is, maximum call cases and Spanish language cases not completed by telephone). Only non-hostile refusals and those who have not directly contacted the government to refuse will be contacted in the field. Field interviewers will contact these addresses in person, determine whether the sampled address is an occupied dwelling unit, and attempt to secure the household’s cooperation in the study.

Each field interviewer will be equipped with a cellular telephone, and will connect the household to the Telephone Research Center using the cell phone or the household’s landline telephone by using Westat’s toll-free TRC number. The interviews will be completed with the same CATI system used to collect data from the RDD sample. Thus, the Screener will be completed with a household member age 18 years or older^³⁶ and SR/PFI or AEWR interviews will be conducted with sampled household members (if any), beginning with the Screener respondent if he/she is identified as a selected respondent.

Field interviewers will make up to 10 attempts to contact a household member at each sampled address. Visits to households will be planned to maximize the efficiency of interviewer time. Interviewers will visit addresses located near one another during the same shift. Also, for refusal and maximum call cases, information on telephone attempts will be used to select times that are likely to be productive; for example, if a household’s telephone number was never answered during a weekday, the interviewer would begin with an evening or weekend visit. Field staff will make visits to the sampled addresses at different times of the day and on different days of the week in order to establish contact with a household member. Field staff members will be provided with “Sorry I Missed You” cards to leave at households where no one is home. These cards will identify the study and its purpose, the sponsor, and field interviewer; they will be used for the first noncontact (if applicable) and first missed appointment (if applicable). Once contact has been made, additional visits, if needed, will be scheduled at the convenience of household members.

When field staff contact household members, they will identify themselves, showing a Westat/NHES photo identification badge; explain the reason for the visit; offer the study letter on U.S. Department of Education letterhead and study brochure; and attempt to enlist the household’s cooperation in the study. Upon identifying an adult household member (i.e., a person eligible to complete the Screener) the field staff will offer to connect the respondent to Westat’s TRC using a toll-free number and using either the study cellular telephone or using the household landline telephone. The field staff member will offer the person the cash incentive of $20 for cooperation in the study. The incentive will be paid when the Screener is completed, regardless of whether household members are sampled for extended interviews. A signed receipt for the incentive will be obtained.

While field staff members will attempt to have all extended interviews completed during the same visit as the Screener, this may not be possible due to respondents being unavailable at that time. If the household has a telephone and will accept a callback appointment for the interview, the TRC will schedule this callback. If the appointment is missed, however, the field interviewer will visit the household again to attempt to get the interview completed. If the household does not have a telephone, the field interviewer will return to the household for completion of any extended interviews that remain to be done.

If all efforts to contact the household member and complete the Screener are unsuccessful, or if the household refuses to complete the Screener, the field staff member will leave a brief response form requesting basic information about household characteristics and an incentive of $5, asking that the household complete and return the postage-paid form. To protect the household’s privacy, the form will be created to be folded and sealed (rather than an open postcard).

B.3. Methods for Maximizing Response Rates

The factors that influence the overall interview completion rate can be divided into the following three broad categories: the ability to gain cooperation from the respondent, flexibility in scheduling interviews, and effective refusal conversion procedures.

Obtaining Respondent Cooperation. Westat’s TRC interviewer training emphasizes obtaining cooperation as well as administering the questionnaire items. Multiple training segments on gaining respondent cooperation will be conducted during interviewer training. The sessions will be led by highly experienced supervisors who will guide the interviewers on ways to quickly gain respondent cooperation and avoid respondent breakoffs. Interviewers will be taught specific techniques and provided with easily accessible answers to many typical respondent questions. They will also be provided with Westat’s toll-free number to give to respondents who are concerned about the legitimacy of the survey and with a contact at the Department of Education should respondents require that information. Early in the data collection period, intensive interviewer monitoring and individualized coaching sessions will help interviewers build on skills learned in training. Throughout data collection, continual monitoring will be conducted, and feedback will be provided to interviewers on a regular basis.

Field staff training will also emphasize gaining cooperation, as this is the primary task of the field staff members. Emphasis will be placed upon being able to discuss the study knowledgably and confidently, providing printed materials to respondents, addressing specific respondent concerns (such as confidentiality and the use of the information), and using incentives to facilitate obtaining cooperation. Following lectures on these topics, field staff will practice a number of scenarios in the training session, being monitored by study staff, followed by debriefing and discussion. Throughout field collection, supervisors will assist the field interviewers by advising them on how to respond to questions and strategies to deal with difficult cases.

Previous NHES experience demonstrates that notifying sampled households in advance of calling them also increases cooperation. An advance letter will be mailed to all cases for which an address can be matched to a sampled telephone number (appendix B). This letter will describe the study and its sponsorship, will state the voluntary nature of the study and the confidentiality of responses, and will include frequently asked questions and answers on the reverse; the OMB approval number will be included. The advance letter will be printed on U.S. Department of Education letterhead and will be mailed in a U.S. Department of Education business envelope. As noted in section A.9, a modest incentive of $2 will be sent in the advance letter.

All field interviewers will carry a letter on U.S. Department of Education letterhead explaining the study and a study brochure, and will wear a study-specific Westat photo identification badge while working on the study. As discussed above, field interviewers will offer the households assigned to in-person collection a cash incentive of $20 for participation in the study.

Flexibility in Scheduling Interviews. Whenever possible, the interviewer will attempt to complete all interviews for the household at the time of screening. In situations where one or more of the respondents are unavailable, a call appointment record will be entered into the CATI management system with notations on the best time to reach the respondent(s). As noted above, up to 20 call attempts will be made to complete Screeners and up 24 call attempts will be made for extended interviews. Experience with previous NHES surveys has indicated that this is an effective method of increasing response.

For address sample cases referred to in-person collection, flexibility will also be important to case completion. As noted above, up to 10 visits will be made at different times of the day and different days of the week to establish contact and complete the interviews. The interviewer will make an appointment to return to the household to complete extended interviews with sampled household members who are not available if the household does not have a telephone or declines a telephone appointment. If the household accepts a telephone callback appointment, the TRC will call the household at the designated time to complete the extended interview. If the respondent does not keep the appointment, one additional telephone attempt will be made. If the interview is still not completed, the field interviewer will return to the household in person. This TRC calling approach will not be used for screening the household, since any adult household member may complete the Screener, but will only be used to complete extended interviews with sampled persons (SR, PFI, or AEWR).

Refusal Conversion Procedures. Another technique that Westat will use to bolster the NHES response rate will be to train TRC interviewers in established and successful refusal conversion procedures. During data collection, TRC interviewers skilled in gaining respondent cooperation will be identified and given advanced training in converting cases for which one refusal has been received. Refusal conversion efforts for NHES:2007 will incorporate an approach that was tested in the NHES:2003 respondent incentive experiment and found to be effective: a refusal conversion letter will be sent to cases for which an address is available (appendix B). This letter will be on Department letterhead and sent via first class mail in a Department envelope. Also, a cash incentive of $2 will be enclosed. Initial refusal cases will be refielded for refusal conversion attempts following a holding period of about 2 weeks. Hostile refusal cases (i.e., those that are profane or abusive) and cases that directly contact the Department of Education to refuse will not be refielded.

If a second refusal is received when refusal conversion has been attempted, a second letter will be sent to the households for which addresses are available. This letter will be sent via Federal Express, to capture the household’s attention and convey the importance of the study. Interviewers will attempt to complete interviews with second refusal cases after a holding period.

In order to limit survey costs and accomplish the data collection in the target field period, refusal subsampling will be employed with the RDD sample as described in section B.1.1.2.

Survey Monitoring

Throughout the data collection period, interviewers will be monitored by TRC supervisors and project staff, and will be given individual feedback based on their performance. At least once each week, the CATI management system will produce computer-generated reports that display response rates, refusal rates, and refusal conversion rates for the NHES:2007 interviewers. These reports will assist the interviewing supervisors in identifying interviewer performance problems that may not be detected through monitoring. In addition, these reports will be used to continually assess the progress of data collection and adjust data collection staffing levels as needed to complete data collection.

B.4. Tests of Procedures and Methods

The NHES is an established survey system. Surveys have been administered in eight times from 1991 to 2005. These previous survey cycles have provided thorough tests of the methodology employed in the NHES and have led to refinements in the system. Because NHES:2007 includes measures contained in previous NHES cycles, the prior surveys have also served as a test of many of the questions in the NHES:2007 instruments as well. The methodology reports for each previous NHES survey contain information on the methodological approach of NHES and the enhancements that have been made since the inception of the survey system. This section discusses the cognitive research and field test conducted specifically for NHES:2007.

In addition to the developmental testing described in this section, NHES includes a reinterview program that has been used to evaluate selected survey items and inform both analysis of NHES data and subsequent survey development. NHES:2007 will continue this program with a PFI reinterview, about 5 minutes in length. The selection of items typically focuses on questions that are new to NHES and/or have not been tested in previous NHES administrations. In addition, emphasis is given to items that are key statistics for the survey topic, required for critical skip patterns, and are amenable to a reinterview procedure in that they are not time dependent. The topics selected for the NHES:2007 PFI reinterview are school choice, the identification of the child’s school using a school look-up, tutoring, television viewing, and factors affecting parent participation.

B.4.1. Cognitive Laboratory Research for NHES:2007

The NHES:2007 surveys contain many items that have been fielded in previous NHES administrations. As a result, cognitive research focused on the relatively few new research issues identified by experts and on items about which experts expressed concern regarding respondent knowledge or recall. There were two stages of cognitive research conducted for NHES:2007: focus groups and individual cognitive interviews.

Participants for the cognitive laboratory research were recruited using flyers, personal networks of Westat employees, and the website craigslist.com. Westat employees and their immediate families were not eligible to participate, nor were those who had participated in an activity of the same type in the previous year. Recruiting goals and results are described briefly under each activity below. Participants were each paid an honorarium of $40 for attending a focus group or completing an individual interview.

Phase I: Focus Groups

At the first stage of cognitive interviewing, prior to the preparation of the full interviews, focus groups were conducted for the SR and PFI interviews. These surveys were chosen because their development included addressing new issues and measures, whereas few new issues were identified by experts for the AEWR survey. Two focus groups were held for the SR survey: one with parents of preschoolers (those age 3 to 5 and not yet in kindergarten) and one with parents of children in kindergarten through second grade. Three focus groups were held for the PFI survey: one each with parents of elementary school children (kindergarten through grade 6), parents of secondary school children (grades 6 through 12), and homeschooling parents (with children at levels equivalent to any grade from kindergarten through 12th grade).

Recruitment of the participants focused on including parents with a variety of characteristics. The first criterion for each group was defined by the population of interest for the particular group (e.g., preschoolers) as described above. Additional goals for the focus groups were:

At least three participants who were nonwhite;
At least three participants with a high school diploma or less education;
At least three participants from households in which the highest occupation is nonprofessional;
No more than two participants from private schools,
Three immigrant parents for each PFI group;
Parents from both two-parent and single-parent homes; and
A parent whose first language is not English.

Due to a miscommunication, not all participants were screened in advance for occupation. In part because of this, but also because recruitment is dependent upon those who volunteer, some recruiting goals were not met. Specifically, the distribution of participants includes fewer families in which parents do not have any postsecondary education or have nonprofessional jobs, although some such parents were included. In addition, few immigrant parents volunteered.

The SR focus groups were helpful in eliciting parent perspectives on school readiness (cognitive and behavioral), family activities that contribute to readiness (e.g., reading, arts and crafts), the parents' awareness of information of interest (e.g., specific types of child language development), and parent practices and experiences related to areas of expert interest such as television viewing and computer/Internet use. Focus groups results confirmed that much of the material contained in the draft SR instrument developed for a previous NHES administration, along with items recommended by experts, were meaningful to parents and the parents felt equipped to provide answers about those topics.

The PFI confirmed that many of the PFI-NHES:2003 questions were appropriate and relevant, including those about types of involvement at school, school-family communication, school practices to involve parents, involvement in homework, and family activities. Experts had recommended expanding NHES coverage of some topics (barriers to involvement and computer/Internet use) and the addition of community networks. Parents cited work and family obligations as major barriers to participation, and indicated that language and cultural barriers are experienced by some parents. The majority of parents of elementary school children indicated that they do not know many parents of other children and do not discuss school-related issues with other parents. Some parents of older children reported knowing parents of their children’s friends and classmates, whereas others did not. In general, schools were not regarded as fostering inter-parent communication, with the exception of PTA meetings.

Regarding computer/Internet use, most elementary school parents indicated that little computer work is required of their children, but parents and children may use the computer or Internet for help. Both computer-based or CD-based programs and Internet sites were mentioned as sources of help. Some parents of these younger children (kindergarten through fifth grade) reported trying to limit their children’s time playing computer games and tried to provide educational programs. In contrast, among secondary students, everyday use for school work and homework was common, and all parents reported children using the Internet at school. Parents of 6th through 12th graders reported both game-playing and educational use at home, and reported efforts to limit type of use or amount of use.

Parents of children who are homeschooled provided extensive information on their reasons for homeschooling, the structure of their own homeschooling, participation in homeschooling groups, and sources of material for homeschooling. There was a large amount of variability in parents’ homeschooling practices, but all participating parents belonged to a homeschool group and reported group activities. Most reported obtaining curriculum material from multiple sources including websites, and several reported that their children took special classes or lessons outside the home. Sources of instructional materials included homeschooling organizations, publishers, bookstores, libraries, and websites representing schools and universities, National Geographic, and the Public Broadcasting System (PBS).

The information obtained in the focus group, together with previous NHES instruments, feedback from experts, and reviews of the extant research literature contributed to the development of the draft instruments. Those instruments (following Technical Review Panel comment for SR and PFI) were tested in the second phase of cognitive research.

Phase II: Cognitive Interviews

Following development of the instruments, individual cognitive interviews were conducted, using concurrent probing techniques to explore knowledge, recall, and comprehension for the targeted items. A particular strength of individual interviews is that the interviewer can focus on one respondent at a time and tailor the cognitive approach to each case. Probes were used to assess the participants’ understanding of terms used in the questions and to ensure that items are salient and unambiguous. Specific probes were employed to assess respondent comprehension of new items included in the questionnaires or items about which knowledge or recall were of concern. An example of a probe that was used concerned adults’ comprehension of the definition given for distance education. With the concurrent methodology, probes are presented immediately following the response to an item. The advantage to this strategy is immediacy, and the drawback is interruption of the interview flow. In some interviews, additional probing was conducted when the interview was over in order to explore specific issues further. Although specific items had been targeted for examination, respondents were encouraged prior to the start of the interview to call the staff member’s attention to any items that they found unclear or difficult to answer. Also, cognitive interviews were used to evaluate the flow and order of the questions.

As in the first phase, the recruiting goals sought parent participants with a wide variety of characteristics. For SR and PFI interviews with parents, the goals were similar:

Parents of children with an approximately equal distribution across age/grade subgroups (preschoolers, kindergarteners, and children in the first and second grades for SR; kindergarten through grade 5 and grades 6 through 12 for PFI) ;
At least three parents of children in private school and at least three parents of children in public school for PFI;
At least one parent of a homeschooler for PFI;
Parents of at least two children from single-parent families for each survey;
Three parents whose level of education is not above high school diploma for each survey;
Four parents in a professional/managerial occupation for each survey;
Four parents who are not White, non-Hispanic for each survey;
At least one interview with a parent of a child with a disability for each survey; and
At least one interview with a parent who reported family receipt of public assistance (Food Stamps, WIC, TANF) in the last 12 months for each survey.

Nine SR parent interviews were conducted. The number of parent of kindergartners was one fewer than the goal, as was the number of parents whose education was not above a high school diploma; the number of nonwhite parents was two below the goal. Nine PFI parent interviews were conducted. Most PFI recruiting goals were met, but there were a few exceptions. No homeschooling parents volunteered and the private school goal fell short by one.

Specific recruiting goals for the AEWR intensive interview participants were as follows:

At least five adults who worked in the past 12 months;
Four or five participants with education beyond a high school diploma or GED, and at least three participants with no more education than a high school diploma or equivalent;
At least five participants who report having participated in formal adult educational activities for work-related reasons in the past 12 months;
Three participants who are employed and were not participants in any formal work-related educational activities in the past 12 months;
Three participants who are not of White, non-Hispanic origin; preferably one who is an immigrant and/or non-native English speaker;
A combination of participants engaged in professional/managerial, sales/service, and trade or labor occupations.

Only one of the six preceding recruiting goals was not met. The recruitment of participants who are employed and are not participants in any formal work-related educational activities for work-related reasons fell short by one. All other recruitment goals were met. Nine AEWR interviews were conducted.

The phase two cognitive interviews indicated that nearly all items in the questionnaires were comprehended appropriately by the respondents and they were able to provide responses to the questions. A few respondents to the SR/PFI interviews indicated that they were unsure about the accuracy of their responses concerning school size. Several parents of children in primary grades (kindergarten to grade 3) could not answer questions about their children’s school grades, since traditional grades (A, B, C, etc.) are not given in their children’s schools until fourth grade. Parents reported that some open-ended items (e.g., number of books their child has) would be easier to answer if ranges were provided. Some parents suggested separating questions about school meetings and PTA meetings because of their different auspices and functions. Some adjustment to wording was suggested because middle school and high school students typically have multiple teachers. Some items were also identified for further monitoring in the field test.

AEWR cognitive interviews indicated little need for instrument changes. The need to allow for the recording of two fields of study at the highest degree level was identified. Difficulty in recalling activities over 12 months was reported by some respondents. Respondent confusion between provider and location of activities (i.e., the organization that provided the instruction versus the place a class or program was taken) was identified and was further monitored in the field test.

B.4.2. NHES:2007 Field Test

The NHES:2007 field test was designed to be conducted in two phases. Phase one of the NHES:2007 field test had three purposes. The first goal was to qualitatively assess the NHES:2007 survey questionnaires by monitoring telephone interviews and debriefing the telephone interviewers. This assessment of the instrument focused on interview flow, how the interviews sounded in “live” administration with respondents, respondent comprehension, and the operation of the CATI system. A second goal was to obtain interview administration timings from the CATI system for the SR, PFI, and AEWR interviews. The SR and PFI interviews were of particular concern based on preliminary timings conducted by and with in-house Westat staff that showed the instruments took too long to administer. Expected administration times were around 20 minutes. Preliminary SR and PFI interviews, however, took over 30 minutes each. In order to meet these two goals, 50 completed interviews of each type (SR, PFI, and AEWR) were considered sufficient for phase one of the field test.

A third goal of phase one of the field test was the implementation and evaluation of the planned field procedures for the NHES:2007 bias study. This portion of the field test was conducted in one county in the mid-Atlantic region, and provided an opportunity to identify areas in which the field procedures should be adjusted prior to the full implementation of the bias study in 2007.

Phase two of the field test shared the same two goals as the RDD portion of the phase one field test: evaluation of the survey instruments and assessment of interview administration times. By administering larger numbers of interviews (200 each for SR, PFI, and AEWR), further qualitative assessment of interview flow, respondent comprehension, and the operation of the CATI system would be possible. In addition, quantitative review of the survey data was an additional goal of the phase two field test. By examining item distributions and “other, specify” responses, survey managers would be able to identify items lacking in variation, having high item nonresponse rates, or having large numbers of “other” responses that might suggest the need for additional response categories.

B.4.2.1 Field Test Samples

Two samples were selected to meet the field test goals. First, a random-digit-dial sample was selected that was sufficient to meet the target number of interviews for SR, PFI, and AEWR. Because the goals of the field test involved assessment of the instruments and survey administration times, and not estimation to the population, some deviations from normal random-digit-dial sample selection were implemented. The telephone numbers were selected in the Eastern and Central time zones, only telephone numbers identified by the vendor as residential numbers were selected, and the sample was selected so that approximately two-thirds of the sampled telephone numbers were those flagged as likely containing at least one household member under the age of 15. These changes were implemented to improve the operational efficiency of the field test. Households selected in this manner are not likely to be different from the population in ways that would affect the results of the field test in terms of evaluating the working of the instruments or the survey timings. A total of 7,000 telephone numbers was selected, with 2,000 of these being allocated to the phase one field test; the remaining 5,000 telephone numbers were reserved for phase two.

The second sample, selected for the phase one field test only, was an address sample in the county selected for the test of bias study procedures. Within the county, 10 local segments were selected to improve the operational efficiency of field efforts; this will also be a feature of the main study. A sample of 400 addresses was selected from residential postal delivery data files; the selected addresses were then matched to telephone numbers using a commercial vendor. In a second phase of sampling, the cases with telephone matches and those without telephone matches were subsampled at differential rates to arrive at a final sample composed 75 percent of matched cases and 25 percent of nonmatched cases. The final field test sample contained 120 addresses with telephone number matches and 30 addresses without telephone matches.

B.4.2.2. Field Test Data Collection Procedures

Telephone interviewers attempted to contact sampled telephone numbers, secure their cooperation, and administer the NHES:2007 interviews. Different data collection procedures were used for the RDD samples in phase one and two and the bias study address sample in phase one.

For the RDD samples in phase one and phase two, the goal was to dial the sampled telephone numbers and complete the target numbers of SR, PFI, and AEWR interviews. Callbacks were made to telephone numbers at which no contact was made or the household agreed to an appointment. No advance mailings were sent to these households. No refusal conversion was attempted with the RDD sample in phase one. In phase two, because the cooperation rate was low, refusal conversion was attempted with initial screener refusals that were coded as mild.

The bias study portion of the phase one field test included the procedures developed for the full-scale implementation of the study discussed in section B.2. The protocol was based on experience from past NHES collections and other recent survey experience regarding the efficacy of survey mailings and cash incentives and the benefits of relatively high call limits.

Following the telephone collection period for the address sample, three types of address sample cases were assigned to in-person data collection. These were cases for which a telephone match was not found (30 cases) or the number that was matched to the address was incorrect or nonworking (33 cases), cases that had received 20 call attempts without completion but had never refused (22 cases, including maximum call, no answer, and answering machine results),^³⁷ and non-hostile refusals that were not converted by telephone interviewers (25 cases). About half of the 25 refusal cases (13) had given three refusals, having completed the protocol described above. However, 12 refusal cases had not given three refusals by the beginning of the field collection period on April 29: 5 cases that had reached the maximum number of calls (20) had one refusal; 5 cases that had reached the maximum call limit had two refusals; and 2 cases with two refusals had not yet reached the maximum call limit of 20.

In addition to the nonmatch and nonresponse cases noted above, an additional eight cases that had been completed in the TRC were assigned to in-person interviewers. These were cases in which the address given by the respondent did not exactly match the sampled address, but it was not clear that they were mismatches. Field interviewers were assigned these cases in order to ascertain whether the interview had been conducted at the sampled address.

All telephone calls to the address sample cases ended on April 24, at which time cases were prepared for the April 29 field staff training. In-person field work began on April 30 and continued through June 14. Thus, the in-person field work for the bias study portion of the phase one field test partially overlapped the telephone interviewing for phase two of the field test, which began on May 26 and continued through July 2.

B.4.2.3 Completed Field Test Interviews

The target number of completed phase one field test interviews was 50 interviews each for the SR, PFI, and AEWR surveys. During the phase one field test, 55 SR interviews, 64 PFI interviews, and 74 AEWR interviews were completed. These figures include interviews completed with RDD sample telephone numbers and those completed with address sample cases.

Ninety seven (97) phase one address cases that were not completed in the TRC were attempted in the field, including 67 cases attempted in the TRC and 30 address cases that were not attempted in the TRC because they were not matched to a telephone number. Forty nine cases were completed as a result of in-person efforts. Of those that were completed, 17 were cases that were not matched to a telephone number; 4 were cases finalized as maximum call cases in the TRC; 17 were cases that were finalized as no answer, answering machine, not working or nonresidential in the TRC; 4 were cases that refused 3 times in the TRC, 1 was a case that refused twice in the TRC, and 6 were cases that refused up to 2 times but finalized with maximum number of calls in the TRC.

In phase two, the goal was to complete 200 interviews each for SR, PFI, and AEWR. The Screener initial cooperation rates (complete/complete + refusal) was lower than expected (32 percent). While it is common for field tests to experience lower initial cooperation rates than main studies, this low cooperation rate negatively affected the completion of extended interviews. The final numbers of completed phase two interviews were as follows: SR, 154; PFI, 253; and AEWR, 167.

B.4.2.4 Field test results

Interview Administration Times

Screener Administration Time. The average field test administration time for the RDD Screener was 4.9 minutes. This is longer than expected during the full-scale data collection; the full-scale data collection will include households in which no person is enumerated, whereas the field test involved enumeration in all households. Based on prior NHES administrations, we expect that the actual screener administration time for the main study will be about 3.5 minutes.

SR and PFI Interview Administration Times. Based on a small number of timings conducted prior to the field test, the length of the SR and PFI surveys was of concern; the phase one field test bore out that concern. The average administration time for the SR survey was 30 minutes, and the average for the PFI survey was 36 minutes. Following the phase one field test, substantial reductions in the SR/PFI interview were made in consultation with the Technical Review Panel in order to reduce the administration time. The changes to the instrument were effective in reducing the administration times. The average time for the SR interviews was reduced to 20 minutes and the average for PFI interview was reduced to 28 minutes. Additional deletions were made to the PFI survey following phase two of the field test to reduce the timing further.

Adult Education for Work-Related Reasons Interview Administration Time. In the first phase, the average field test administration time for the AEWR interview in phase one was 21 minutes. The average administration time was 29 minutes for AE participants and 10 minutes for nonparticipants. In the second phase, the average administration time was slightly shorter and when adjusted based on the number of participants and nonparticipants expected in the NHES:2005 sample design plan, the estimated administration time is 17 minutes.

Field Test Instrument Evaluation Results

The field test indicated that, in general, the items in all interviews were working very well, and only minor changes were required to clarify items or to improve item wording. To a large extent, this result reflects the fact that most of the survey items had been tested and administered in prior NHES collections. In addition, the NHES:2007 questionnaires had undergone expert review and cognitive testing. However, because administration times were of concern, the SR/PFI surveys were reduced in length in consultation with the Technical Review Panel.

Screener. The core of the NHES Screener has been thoroughly tested in prior administrations, but the NHES:2007 Screener includes additional items for the bias study sample. The field test evaluation focused particularly on these items, while still evaluating the Screener as a whole. Additional wording was added to explain why we are asking about telephone numbers in the household, and the wording of a question about participation in work-related courses (administered to bias study cases assigned to in-person collection) was modified so that a lengthy explanation will not be read repeatedly when there are multiple adults in the household.

AEWR Survey. Very few changes were made to the AEWR survey. The survey items had nearly all been administered previously and had been well tested in prior administrations. However, some adjustments were made based upon interview monitoring, interviewer feedback, and a review of the field test data.

Minor wording changes were made for consistency, as some interviewers were observed to misread or stumble over wording slightly in a few items.
An item concerning the first language the adult learned to speak was added for consistency with the parent background section of SR/PFI.
The definition of distance education was observed to be very long. It was shortened following field test phase one, and was observed to work well in phase two.
Following phase two, the “other, specify” responses to item AIL1120 through AIL1340 concerning the types of skills taught in less formal educational activities, suggested that two additional response options and minor wording changes to two others would be useful in classifying responses.

SR and PFI Surveys. Most items in the SR and PFI surveys had been administered previous, but some important items, such as the school lookup and items on tutoring services, were new and required assessment in the field test. Relatively few changes were needed as a result of respondent confusion or problems with specific questions.

The question concerning participation in a daycare center or preschool was unclear to some respondents, who included home-based group child care (generally known as family daycare). The introduction to the section was revised to clarify the intent of the questions.
The survey included questions about parent contact with parents of children in their child’s school and with parents in their community. Interviewers noted that often these were the same parents, so the items were combined.
The “other, specify” responses concerning the main reason for choosing the child’s school were reviewed following the field test. Two additional response categories were added to the question as a result of this review: “ religious and other spiritual reasons,” and “cost and financial reasons.” In addition the item concerning school size was expanded to include both school and class size.

As noted earlier, the length of the SR and PFI surveys was of concern and the results of the phase one field test indicated a need to reduce their lengths. Substantial reductions were made for both SR and PFI to reduce the interview administration times. Members of the Technical Review Panel were asked to review the questionnaire and assign priorities to survey items and content areas, and these were taken into account in revising the instrument.

Skip patterns were revised so that the SR interviews will focus primarily on school readiness issues, and many items about parent involvement will no longer be asked for the preschool population. This approach substantially reduced the SR survey administration time.
Developmental items concerning sounding out words, buttoning clothes, writing versus scribbling, paying attention well, and telling a story to an adult were deleted to reduce administration time. Some TRP members recommended their deletion because they felt that parents might not be good sources for this information.
Followup questions about which household members attended school meetings or functions or helped with homework were eliminated.
Followup questions about whether specific types of meetings or functions had been held at the child’s school were deleted.
Several items concerning how well the parent believes the school does various things were deleted. Those that were retained focused on the child’s current education, and those that were deleted focused on school transitions, community services, and planning for future education.
The section concerning parent beliefs about school readiness was deleted from the SR survey.
The nonresidential parent section was deleted from the PFI survey.
Throughout the instrument, items were combined or shortened to reduce the administration time while still capturing some information on topics of interest.

The above are illustrative examples. Details concerning the changes to the survey instruments are provided in the NHES:2007 field test report, which accompanies this clearance request. The final questionnaires appear in Appendix A.

B.4.2.5 Bias Study Findings

There were two main goals of the bias study field test. The first goal was to test the protocol for training interviewers, obtaining cooperation, making contact with the TRC with a willing participant, and assessing the approach used to monitor progress in the field. No problems with the bias study approach were identified. While none of the households at which a nonrespondent postcard was left actually returned the postcard to Westat, this result does not suggest that the postcard approach should be abandoned for the full study, due to the small number of cases in the field test.

The second goal of the bias study field test was to evaluate respondent reaction to the in-person approach. This was in response to concerns that sampled households might perceive in-person efforts as harassment following telephone refusals resulting in complaints to ED or Congressional Representatives. Over the course of the field test of the bias study the negative responses were minimal. While interviewers reported that a small number of final refusals were very strong and one was hostile, the experience was not different from in-person refusal conversion efforts on other studies.

B.5. Individuals Responsible for Study Design and Performance

Those persons listed below participated in the study design and are responsible for the collection and analysis of the data.

Christopher Chapman, NCES 202/502-7414

Gail Mulligan, NCES 202/502-7491

Mary Hagedorn, Westat 301/251-4273

Jill Montaquila, Westat 301/517-4046

J. Michael Brick, Westat 301/294-2004

Kevin O’Donnell, Westat 301/738-3594

Soumya Alva, Westat 240/314-2450

This page is intentionally blank.

1 Any household member age 18 or older will be eligible to respond to the screening interview. However, if there are no household members age 18 or older, the male or female head of the household will complete the Screener. Household members are defined as persons who considered that household as their residence, kept their possessions there, and have no other place to live.

2 The respondent for the SR and PFI Surveys will be identified by the Screener respondent as the household member most knowledgeable about the care and education of the sampled child. For ease of discussion, the respondent is referred to as the parent/guardian.

3 Because the proportion of 7-year-olds who are not enrolled in school is very small (about 1.5 percent), an upper age limit of 6 will be used for the SR Survey.

4 Some SR Survey items will be administered about children enrolled in kindergarten through second grade.

5For the NHES:1991 and NHES:1993 surveys, a modified Mitofsky-Waksberg method was used to select the sample of telephone numbers. The advantages and disadvantages of this method are discussed in Collins et al. (1997).

6 Here, the term listed stratum is used to refer to the set of telephone numbers in 100-banks having at least one listed number; that is, at least one number listed in a white pages directory. Later in this report there is a discussion of differential sampling of telephone numbers based on listed status. Note that, unlike the reference to listed status later in this report, the listed stratum referred to here does not refer to the listed status of the particular telephone number.

7 Nontelephone households include cellular phone-only households, in addition to households with no telephone service.

8These were previously referred to as “Bellcore types.”

9Independent tabulation of the Marketing Systems Group’s 1^st quarter 2005 Genesys database.

10The NXXType restriction was reconsidered for NHES:2007. In the past, NXXTypes 50, 51, and 54 have been excluded from the frame because of ethical concerns about cellular telephone customers having to pay for incoming calls. Although changes in pricing plans have ameliorated those concerns, sampling cell phone numbers is not recommended for NHES due to concerns for a household survey about the definition of the sampling unit and selection probabilities, and because of concerns about low response rates.

11MSG is the vendor that provides the sampling frame for the selection of telephone numbers. Genesys is the name of the system that generates the sampling frame.

12A tritone is the three-note sound heard when dialing a nonworking telephone number.

13With the ID-PLUS utility, the telephone numbers in the NHES:2001 and NHES:2003 samples were dialed by Genesys representatives prior to the beginning of the field period in order to help ascertain whether they were nonworking or business numbers.

14 With Genesys-CSS, all telephone numbers not identified as business numbers (including listed residential) numbers are dialed and allowed to ring up to two times, in order to identify business, cellular, and nonworking numbers. The dialing is done during the hours of 9 a.m. to 5 p.m. local time by specially trained agents.

15 Privacy Manager is a device that works with caller ID to screen and manage incoming calls.

16 Results for the phase 1 sample are given here because the subsampling in phase 2 was based on mailable status and thus results from the phase 2 sample would be skewed.

17 In the modified Mitofsky-Waksberg procedure, telephone numbers are grouped in 100-banks that are treated as primary sampling units (PSUs). One telephone number in each PSU is randomly selected (the prime number) and is dialed. If the prime number is residential, then the PSU is retained in the sample, otherwise the PSU is eliminated. The screening of PSUs continues until the required number of residential PSUs is identified. See Brick and Waksberg (1991) for further information. The change to the list-assisted method eliminated the need to screen prime numbers and gives an unclustered sample, resulting in a reduction in sample variance.

18In addition to having a higher concentration of Blacks and Hispanics, the high minority stratum was found to have a higher concentration of Asians than the low minority stratum. Therefore, oversampling in the high minority stratum was expected to raise the sample yield for Asians (as compared to an equal probability design), even though Asians are not explicitly considered in the definition of “high minority.”

19 In NHES:2001, the Screener response rates for telephone numbers with mailable addresses were 75 percent for those sent an advance letter and 70 percent for those not sent an advance letter; for telephone numbers with no mailable address, the Screener response rate was 55 percent (Nolin et al. 2004).

20 Westat used this subsampling in the 2002 National Survey of America’s Families (NSAF) and in the 2003 California Health Interview Survey (CHIS). For details on the 2002 NSAF methodology, refer to Report 2 on the website http://www.urban.org/content/Research/ NewFederalism/NSAF/Methodology/2002MethodologySeries/2002.htm. For details on the 2003 CHIS, refer to a forthcoming report to be published on the website http://www.chis.ucla.edu/methods.html. Subsampling for nonresponse followup has also been used in non-RDD surveys such as the American Community Survey (ACS). (See Tersine and Starsinic 2003 for details.)

21 In NHES:2005, initial cooperation rates for mailable cases in the release group not designated for followup were 2 percentage points higher than for mailable cases in the group designated for followup; initial cooperation rates for nonmailable cases did not differ between the two release groups.

22 The expected design effect due to the unequal weighting to account for the proposed subsampling of cases for followup is 1.06. The expected increase in variance was accounted for in the determination of the sample size requirements.

23 FedEx will be used for all eligible addresses. Priority Mail will be used for Post Office boxes and rural route addresses, because they cannot be sent by FedEx.

24 Key statistics for the SR survey include participation in care arrangements, recognition of all colors, ability to count higher than 10, knowing all letters, and ability to write own name. Key statistics for the PFI survey include parent participation in 3 or more activities in the child’s school, parent participation in home learning activities, and parent assessment of school practices. Key statistics for the AEWR survey include participation in adult education for work-related reasons and participation in employer supported adult education.

25 Hostile refusals are profane or abusive refusals.

26 In addition to coding refusal call results as mild/firm/hostile, the NHES interviewers also code a variety of other characteristics of the refusal (whether the respondent hung up during the introduction; whether the respondent expressed confidentiality concerns; whether the respondent said he/she was too busy or not interested; whether the respondent indicates a negative reaction to the survey; whether the respondent indicates a negative reaction to the government/all surveys; whether the respondent reports that the interview sounds too long or is taking too long; sex of the person refusing; who refused the extended interview; whether there is a hearing or language problem (and, if applicable, what language); and whether the refuser sounds elderly).

27 If the respondent prefers, the household’s telephone may be used to dial in to the Westat Telephone Research Center using a toll-free number.

28 The additional questions in the expanded Screener will be repeated in the extended interview for sampled Screener respondents. Data from the additional questions in the expanded Screener will be used only if the extended interview is not completed. The question wording will be as similar as possible to the corresponding question in the extended interview. While it is possible that there will be context and proxy effects, these are expected to be small, and will be examined by comparing responses to the expanded Screener and extended interview items for cases with completed extendeds.

29 These expected extended interview unit response rates are slightly higher than the rates attained in recent NHES surveys of the same or similar subpopulations. However, in general, in-person interviews attain higher unit response rates than telephone interviews.

30If the fourth quarter, 2006, Genesys frame is not available at the time of telephone number sampling for NHES:2007, then the third quarter frame will be used.

31 It is likely that the reserve sample, if released, will be treated as part of the cases subsampled to not receive followup efforts.

32 Throughout this report, the subgroup of children age 3 through 6 not yet enrolled in kindergarten is referred to simply as “preschoolers.”

33 The school practices considered were the following: School tells family how child is doing in school; school helps family understand child’s development; school tells about chances to volunteer; school advises about home learning; and school gives information about community services.

34 The home learning activities considered were the following: Telling the child a story; working on arts or crafts with the child; involving the child in household chores; taking the child to the library; taking the child to a play, concert, or other live show; taking the child to an art gallery, museum, or historical site; taking the child to a zoo or aquarium; working on a project with the child such as building, making, or fixing something; talking with the child about the family history or ethnic heritage; playing board games or working puzzles with the child; and discussing with the child how to manage time.

35 The unit response rate expectations were derived from the unit response rates from NHES:2005.

36 If no household member is age 18 or older, the Screener will be completed with the head of household.

37 In a few cases, the maximum call limit was not reached because the household had one or more appoints scheduled during telephone collection, which resulted in the case being “on hold” for the appointment for a period of time.

File Type	application/msword
Author	WAITS_T
Last Modified By	DoED
File Modified	2006-09-18
File Created	2006-09-18