NHES:2015
Full-scale Data Collection
Request for OMB Review
OMB# 1850-0768 v.10
Part A
July 8, 2014
TABLE OF CONTENTS
Page
JUSTIFICATION 1
A.1 Circumstances Necessitating Collection of Information 7
A.2 Purposes and Uses of the Data 8
A.3 Use of Improved Information Technology 8
A.4 Efforts to Identify Duplication 8
A.5 Collection of Data from Small Businesses 9
A.6 Consequences of Less Frequent Data Collection 9
A.7 Special Circumstances of Data Collection 10
A.8 Consultations Outside the Agency 10
A.9 Payments to Respondents 12
A.10 Assurance of Confidentiality 13
A.11 Sensitive Questions 13
A.12 Estimated Response Burden 14
A.13 Cost to Respondents 14
A.14 Cost to the Federal Government 15
A.15 Reasons for Program Changes 15
A.16 Publication Plans and Project Schedule 15
A.17 Approval to Not Display the Expiration Date for OMB Approval 15
A.18 Exceptions to the Certification Statement 15
List of Tables
Table
1 Estimated response burden for NHES:2015 14
List of Exhibits
Exhibit
1 Surveys conducted under the National Household Education Surveys Program, by years administered: 1991 through 2012 6
2 NHES:2015 schedule of major activities 15
JUSTIFICATION
In 2008, the National Household Education Surveys program (NHES) began a redesign effort to convert from a system of landline random digit dial (RDD) surveys to a self-administered mail survey using an address-based sample (ABS). This redesign was prompted by declines in response rates to the telephone survey and concerns about population coverage using the landline telephone frame (due to increasing conversion to cellular-only coverage). The goals of the redesign effort were to develop and assess approaches to collecting important information on education topics from households while improving response rates and coverage from the previous design. A feasibility test of the new design was conducted in 2009 followed by a field test in 2011. The field test results helped to inform the final design of a full-scale NHES collection in 2012 (OMB# 1850-0768 v.9).
During the same period of time, NCES began supporting developmental work on new questionnaire items for federal household surveys on work-related education, training, and credentials for adults and out-of-school youth. The Interagency Working Group on Expanded Measures of Enrollment and Attainment (GEMEnA) is a collaboration among federal statistical agencies established by the OMB Office of Statistical and Science Policy, the Council of Economic Advisors, and the Under Secretary of Education to improve federal household statistics on the attainment of non-degree credentials such as industry-recognized certifications, occupational licenses, and educational certificates. In 2012, GEMEnA’s commission expanded to include the development of new and revised measures of enrollment or participation in education and training for work. One of GEMEnA’s roles is to guide NCES’s development of a new household survey on these topics to support research and policy analysis. NCES conducted two pilot studies (OMB# 1850-0803), first a two-stage telephone survey (42% response rate) and then a single-stage self-administered mail survey (69% response rate).
NHES Feasibility Study (NHES-FS) Summary
One of NCES’s goals from the beginning of the GEMEnA project was to see whether it would be feasible to eventually incorporate a revised survey on adult credentials back into the NHES. In 2014 a Feasibility Study testing the integration of an adult topical survey into NHES mail operations and processing was conducted. The Feasibility Study (NHES-FS) included several experiments to inform the final design of the 2015 full-scale NHES which will include the Credentials for Work Survey (CWS) and the two child-focused surveys fielded in the NHES:2012, the Parent and Family Involvement in Education (PFI), and Early Childhood Program Participation (ECPP) surveys. The 2015 NHES will also pilot the adult Training for Work Survey (TWS), planned for a future NHES administration. Using one household survey platform for both child and adult surveys provides greater efficiency in the data collection and reduces overall national burden by maximizing the use of a single household sample draw. Before adding an adult education survey back into the NHES it was important to test the feasibility of using a mail survey to screen households for both adults and children, and to test different approaches to collecting topical data from households (e.g., sampling either an adult or a child from the same household for topical follow up compared to sampling both and adult and a child for follow up).
In addition to testing different approaches to collecting topical data from households, the NHES-FS included several other experiments to test approaches aimed toward decreasing unit and item nonresponse. These experiments are described below.
Experiment 1 tested unit and item response rate differences between a screener that asked for age measured in years versus a screener that asked for age measured as year and month of birth. Both screeners are included in Appendix B. Based on the NHES:FS results, NCES plans to collect age measured as year and month of birth.
Experiment 2 tests different response option orders for the question about household members’ school enrollment status. In one version of the screener, the response option “homeschool instead of school for some classes or subjects” is the first response option, after “public or private school, or preschool”. In the other version, “homeschool instead of school for some classes or subjects” is the first option and “public or private school, or preschool” is the second response option. Analysis of the NHES:2012 data suggested there may have been under-reporting of homeschooling on the screener. This experiment is designed to test whether response distributions are sensitive to response option order. Based on the NHES:FS results, NCES plans to position the homeschool response option first.
Experiment 3 tested response rate differences between households that did not receive a cash incentive with their initial screener questionnaire mailing and households that received the NHES standard $5 incentive. Preliminary results from the NHES-FS indicate that the $5 incentive is associated with higher response rates than no incentive. Based on this NCES plans to include a $5 incentive in the initial screener mailing in the NHES:2015.
Experiment 4 tested the effectiveness of mailing a large credit-card sized magnet with the screener to serve as a reminder to complete the survey. This experiment was designed to test whether the magnet was associated with higher response rates in the second stage of the survey. Preliminary results indicate that the magnet did not improve topical response rates so NCES does not plan to include a magnet incentive in NHES:2015.
Experiment 5 tested unit response rates when a household was mailed both an adult topical questionnaire and a child topical questionnaire (the After-School Programs and Activities, or ASPA, survey) versus a single CWS or ASPA questionnaire. Due to the operational complexity of fielding two topical surveys per household and the relatively small increase in efficiency for dual households, households will only receive one questionnaire (adult or child) in the NHES:2015.
Experiment 6 included a split-ballot test of the ASPA topical instrument to test the efficiency of different question wordings and formats (grid versus non-grid) for selected items. The analysis is ongoing to determine the results of the ASPA split ballot test. NCES is not requesting clearance for the ASPA for the NHES:2015.
Experiment 7 included a split-ballot test of a full Adult Training and Education Survey (ATES) instrument including sections on work-related training and non-credit college classes versus a shorter CWS instrument including only key items from those sections. In NHES:2015, NCES plans to field the CWS and pilot test the TWS for NHES:2017.
Overview of NHES:2015 Target population
Adults ages 16 to 65 who are not enrolled in grade 12 or below will be eligible for the CWS and TWS topical surveys and children from birth through 12th grade who are ages 20 or younger will be eligible for the child-focused topical surveys. The NHES:2015 will screen 200,000 households, which splits into a 185,000 household sample that will be sent paper screeners and a 15,000 household sample that will be selected to participate in a self-response Internet screener experiment. The Internet screener experiment will allow NCES to examine response rates and coverage when initiating the NHES survey using a web mode. We will also experimentally test nonresponse follow-up by mail at either the 3rd mailing and 4th mailing or the 4th mailing only. The Internet screener will be developed and hosted by the Census Bureau. For both groups, an expected screener response rate of 70 percent and an address ineligibility1 rate of 11 percent are assumed, bringing the total number of expected screeners to 124,600. From these completed screeners, it is expected that approximately 56 percent or 69,341 households will contain an eligible adult but no eligible children; approximately 27 percent or 34,167 households will contain an eligible adult and an eligible child; and approximately 0.4 percent or 534 households will contain an eligible child but no eligible adults (for example, children who live with grandparents above age 65)2.
NHES is NCES’s principal mechanism for addressing education topics appropriate for households rather than establishments. Such topics cover a wide range of issues, including early childhood care and education, children’s readiness for school, parent perceptions of school safety and discipline, before- and after-school activities of school-age children, participation in adult education and training, parent involvement in education, school choice, homeschooling, and civic involvement. The NHES consists of a series of rotating surveys using a two-stage design in which a household screener collects household membership and key characteristics for sampling and then appropriate topical survey(s) are mailed to sample members.
Data from the NHES are used to provide national cross-sectional estimates on populations of special interest to education researchers and policymakers. For surveys about children, the population of interest is defined by age or grade in school, or both, depending on the particular survey topic and research questions. For surveys of adults, the population of interest is those aged 16 to 65 who are not enrolled in grade 12 or below, excluding those on active duty military service and those who are institutionalized. The NHES targets these populations using specific screening and sampling procedures.
The NHES design also yields estimates for subgroups of interest for each child and adult survey, as defined by age (or grade for children), education level for adults, Hispanic origin, and racial background for all populations of interest3. In addition to providing cross-sectional estimates, the NHES is designed to produce estimates from repeated cross sections to measure changes in key statistics.
The last NHES data collection prior to the redesign was conducted in 2007 and included the Parent and Family Involvement in Education Survey (PFI) and the School Readiness Survey (SR)4. The NHES:2012 was the first full-scale data collection using an addressed-based sample and a self-administered questionnaire--it included the PFI survey and ECPP survey. The overall screener plus topical response rate was approximately 58 percent for both the PFI and the ECPP in 2012, compared to the 2007 overall response rate of 39-41% (depending on the survey). The results suggest that the new methodology has the ability to address the response rate and coverage issues identified in the 2007 data collection.
The 2015 NHES will reintroduce an adult survey component. The Credentials for Work Survey (CWS) will provide a means to investigate issues related to the prevalence and key characteristics of credentials for work that cannot be adequately studied through the Center’s institution-based data collection efforts. NCES is also requesting clearance for a small pilot study of a complementary Training for Work Survey (TWS) that focuses on participation in work-related education and training. NCES expects to field the TWS as part of NHES:2017.
Survey data from the NHES have been used for a large number of descriptive and analytic reports and articles, including NCES publications, publications of other Federal agencies, policy analyses, theses and dissertations, conference papers, and journal articles. A list of NHES publications issued by NCES can be found on the NHES website, http://nces.ed.gov/nhes.
As shown in exhibit 1, each administration of the NHES has included more than one topical survey. However, thus far under the new mail data collection methodology, households have only been screened for and topical surveys have only focused on children. The NHES:2015 will include three topical surveys: the CWS, the PFI, and ECPP surveys; and a pilot test of the TWS.5 To develop the CWS survey, NCES has been redeveloping and testing new survey content since 2009, including pilot testing, expert review, cognitive testing, and focus groups.
As a result of analysis of the NHES-FS results, some of the proposed content in the 2015 PFI, related to homeschooling, has been revised. NHES has traditionally experienced under-reporting of homeschoolers, and the switch to the mail mode likely exacerbated under-report of homeschoolers because of the use of self-administered screener, which was a change from prior NHES administrations when a trained interviewer helped respondents who homeschooled their children (full-time or part-time) navigate to the homeschool path rather than the enrolled path. However, at the second stage of the mail survey, we found evidence that the NHES is yielding over-reports of homeschooling. Specifically, results from the NHES-FS indicated that a smaller proportion of the respondents to the ASPA who indicated that their child was homeschooled would actually be considered a homeschooler under NCES reporting conventions. NCES considers any child who attends school for more than 25 hours per week to be enrolled rather than homeschooled. Therefore, although we still believe that the mail screener operation results in under-reporting of homeschooling, we know from the NHES-FS that when we ask respondents to a questionnaire designed for enrolled students whether or not the child is homeschooled, we get over-reporting.
For the reasons discussed above, we have revised the 2015 PFI-Enrolled questionnaire so that it:
Moves the homeschooling questions to the end of the first section (we believe that having the homeschooling question toward the beginning of the first section of the NHES-FS ASPA instrument led respondents to believe that we were trying to ask them something about their child’s enrollment rather than asking if the child may actually be homeschooled for some or all classes).
Adds an additional confirmation question which attempts to further filter out students who are enrolled rather than homeschooled from the homeschooling sequence (the NHES CATI instruments used up until 2007 utilized multiple confirmation questions to try to clarify homeschooling status; adding multiple confirmation questions to NHES 2015 will parallel the approach used in the CATI instruments).
Asks parents of students of homeschoolers who receive the PFI-Enrolled questionnaire their main reason for homeschooling (the item comes from the PFI-Homeschooling questionnaire) so that we can get more accurate estimates of this key data point.
Because the homeschooling questionnaire design poses these challenges and also attempts to measure constructs that are observed in a small proportion of the school-aged population, we plan to cognitively test these items in early October 2014 with 5-9 respondents. NCES staff will conduct the cognitive interviews, and will recruit participants through personal networks and contacts with homeschooling policy organizations. If the proposed content is found to introduce new sources of measurement error, we will change the content to address the error. Should the cognitive interviews result in any wording changes, we will provide the updated questionnaire through a change request.
Exhibit 1. Surveys conducted under the National Household Education Surveys Program, by years administered: 1991 through 2012
Topical survey |
NHES survey administration |
|
||||||||
1991 |
1993 |
1995 |
1996 |
19991 |
2001 |
2003 |
2005 |
2007 |
2012 |
|
Early childhood education/ program participation |
√ |
|
√ |
|
√ |
√ |
|
√ |
|
√ |
Adult education |
√ |
|
√ |
|
√ |
√ |
√ |
√ |
|
|
School readiness |
|
√ |
|
|
√ |
|
|
|
√ |
|
School safety and discipline |
|
√ |
|
|
|
|
|
|
|
|
Parent and family involvement in education |
|
|
|
√ |
√ |
|
√ |
|
√ |
√ |
Civic involvement |
|
|
|
√ |
√ |
|
|
|
|
|
After-school programs and
|
|
|
√2 |
|
√ |
√3 |
|
√ |
|
|
Household library use |
|
|
|
√ |
|
|
|
|
|
|
Homeschooling |
|
|
|
|
√ |
|
√ |
|
√ |
√ |
1 The NHES:1999 was a special end-of-decade administration that measured key indicators from the surveys fielded during the 1990s.
2 The After-School Programs and Activities Survey of the NHES:1995 only asked about children in first through third grades.
3 The After-School Programs and Activities Survey of the NHES:2001 also included items on before-school programs.
SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys Program (NHES), 1991–2012.
The PFI and ECPP surveys are repeat administrations of the topics administered in the mail survey mode in the NHES:2012. NCES decided to repeat these surveys in 2015 to establish the first trend data for the newly designed NHES. Tracking changes in the population over time is a key research goal of the NHES program.
The Early Childhood Program Participation Survey (ECPP), previously conducted in 1991, 1995, 2001, 2005, and 2012, surveys families of children ages 6 or younger who are not yet enrolled in kindergarten and provides estimates of children’s participation in care by relatives and non-relatives in private homes and in center-based daycare or preschool programs (including Head Start and Early Head Start). Additional topics addressed in ECPP interviews have included family learning activities; out-of-pocket expenses for nonparental care; continuity of care; factors related to parental selection of care; parents’ perceptions of care quality; child health and disability; and child, parent, and household characteristics.
The Parent and Family Involvement in Education Survey (PFI), previously conducted in 1996, 2003, 2007, and 2012, surveys families of children and youth enrolled in kindergarten through 12th grade or homeschooled for these grades, with an age limit of 20 years, and addresses specific ways that families are involved in their children’s school; school practices to involve and support families; involvement with children’s homework; and involvement in education activities outside of school. Parents of homeschoolers are asked about their reasons for choosing homeschooling and resources they used in homeschooling. Information about child, parent, and household characteristics is also collected. To minimize response burden and potential respondent confusion, both enrolled and homeschool versions of the PFI questionnaire were created for self-administration.
The Credentials for Work (CWS) provides a means to investigate issues related to the prevalence and key characteristics of non-degree credentials that cannot be adequately studied through the Center’s institution-based data collection efforts. It targets non-institutionalized adults in the United States ages 16 to 65, who are not enrolled at grade 12 or below. The CWS will collect information on educational attainment, prevalence and characteristics of certifications and licenses and their holders, prevalence and characteristics of educational certificates and certificate holders, and key measures of work-related training collected on the TWS. It will also collect detailed employment and background information.
The Training for Work Survey (TWS) will collect information on initial on-the-job training programs (such as apprenticeships and internships), college coursetaking, work-related training and key measures of non-degree credentials collected on the CWS. It also includes detailed employment and background items.
The instruments are provided in Appendices B (screener) and C (topical questionnaires).
The Education Sciences Reform Act of 2002 (ESRA 2002: 20 U.S. Code § 9543) defines the legislative mission of NCES to collect, report, analyze, and disseminate statistical data related to education in the United States and in other nations.
The NHES is specifically designed to support this mission by providing a means to investigate education issues that cannot be adequately studied through the Center’s institution-based data collection efforts. For example, school-age children participate in many types of informal or formal care settings. There is no available sample frame that includes all of these types of care arrangements. Likewise, although attaining a postsecondary credential has become increasingly important for securing opportunities to get high-return jobs in the United States in the 21st century, NCES has traditionally only collected data on postsecondary certificates and degrees awarded through credit-bearing instruction in institutions of higher education that participate in Title IV federal student aid programs. These comprise only a portion of subbaccalaureate education and training American adults seek and complete to learn the skills they need to find and keep good-paying jobs.
It is efficient and economical to interview parents about their children’s participation in child care programs and family participation in school and other education activities through a household-based approach rather than incurring the cost and nonresponse involved in enlisting schools, obtaining lists of parents, and sampling parents from those lists. Similarly, it is also most efficient to interview adults through a household-based approach rather than trying to obtain lists from a myriad of private credential awarding bodies. Also, the household approach allows for inclusion of adults who do not participate in training or have a credential, providing a point of comparison.
Repeating the NHES:2012 child surveys will provide the first trend data available under the new NHES design. Tracking trends in education topics over time on a regular, repeating basis is a key research goal of the NHES program. Adding the adult education component will provide the first publicly available adult education data from the NHES in over a decade, and provide detailed data about previously unmeasured non-degree credentials.
The NHES:2015 data collection will provide policymakers and researchers with data on early childhood education, parent and family involvement in education, homeschooling, and adult credentials for work that are not available elsewhere. The data contribute directly to NCES education indicators, reports, and statistical abstracts. Researchers nationwide rely on NHES data for important policy analyses.
The data for the NHES:2015 will be collected for NCES by the United States Bureau of the Census using four complementary survey systems - (1) Amgraf One Form Plus, (2) Docuprint, and (3) integrated Computer Assisted Data Entry (iCADE), chosen for their efficiency and accuracy in the data collection process and (4) Enterprise Internet Solutions (EIS).
Forms Design. Questionnaires will be created using Amgraf One Form Plus. Completed hardcopy forms can be processed by iCADE to capture responses through optical mark recognition (OMR) and keying from image (KFI). Questionnaires will be printed, trimmed, and stitched through an in-house print on-demand process using a Docuprint system which allows personalization, and the ability to tailor items to each specific respondent. The data from the questionnaires will be captured by the iCADE technology/software, which automatically extracts all check box entries (OMR) and captures and displays an image of all other entries to an operator for KFI.
Image Preprocessing. iCADE applies image preprocessing to the forms in their image format in order to correct any skewing at the time of scanning, and the iCADE software performs registration to align the individual questionnaire page template with the appropriate scanned image. The scanner despeckles the image to remove unwanted pixels.
Data Capture. iCADE reads the form image files, checks the presence of data, processes all check box fields through OMR, and presents an image of the handwritten fields to an operator for KFI.
Verification. Extracted KFI data are subject to 100% field validation according to project specifications. If a data value violates validation rules, the value is flagged for review by verifiers who interactively review the images and the corresponding extracted data, and resolve validation errors.
Archiving. Images will be scanned and archived to magnetic storage located on a secured server in case they are needed later. This eliminates the need to save paper copies of the completed questionnaires.
Data from a web instrument. The Enterprise Internet Solutions (EIS) area of the Application Services Divisions (ASD) will host a web based Screener instrument. Data will be transmitted and processed daily. The web application will be hosted on the fully certified and accredited Centurion system (infrastructure, security, and framework).
CWS/TWS
Senior policy officials in the Departments of Education, Commerce, and Labor, foundations including the Gates Foundation and Lumina, and research organizations such as the Georgetown Center for Education and the Workforce have recognized that there is a lack of valid statistical information on prevalence of industry-recognized certifications and education certificates, and called for the development of new data sources. A series of meetings during the fall of 2009 launched a broad effort to begin to define and enumerate these credentials. NCES conducted a review of research literature and data collections since the work of a previous Interagency Committee in 2000, from which NCES developed a bank of existing survey items on certifications (completed 11/2009) and education certificates (completed 1/2010). This research found no surveys that adequately capture comprehensive data on adult training, education, and non-degree credentials.
PFI and ECPP
Population: Most other surveys do not address the topics covered in NHES for the populations of interest. For example, the Head Start Family and Child Experiences Survey (FACES) focuses on children in Head Start, whereas all children who have not yet started kindergarten are of interest in the ECPP Survey. Likewise, the National Survey of Early Care and Education (NSECE) focuses primarily on low income children and their program participation. The National Survey of Parents of Public School Students and Survey of Family and School Partnerships in Public Schools focus on parents of children in public schools; those whose children attend private schools or are homeschooled are not represented. Some studies, such as the Early Childhood Longitudinal Study, Birth Cohort (ECLS-B); the Early Childhood Longitudinal Study, Kindergarten Class of 1998-1999 (ECLS-K); and the Early Childhood Longitudinal Study-Kindergarten Class of 2010-11 (ECLS-K:2011) focus on single-year cohorts that are followed over time and therefore do not provide nationally representative data on different age groups. The NHES surveys are designed to complement these longitudinal collections with more frequent and more inclusive cross-sectional data.
Survey Content: Extant studies are limited in the content that they include relative to the goals of the NHES surveys. Studies such as the National Survey of America’s Families and the National Study of the Changing Workforce collect some information on child care or program participation, but their primary emphasis is on other topics, and the depth of information on early care and education experiences is limited. The Head Start FACES project collects information on Head Start program participation and some family measures, but does not account for all nonparental care and programs. The Current Population Survey October Education Supplement is limited to a relatively small number of items on education participation and does not address the roles that parents play in their children’s school, schoolwork, and home activities. Also, no nationally representative study other than the NHES collects detailed data on homeschooling.
Current Estimates and Measuring Change Over Time. Many of the extant surveys follow one cohort or periodic cohorts (e.g., the Early Childhood Longitudinal Studies, Head Start FACES, NSECE) or are no longer conducted (e.g., the National Survey of America’s Families, Family Involvement in Education: A National Portrait). As a result, they cannot meet the NHES goal of providing up-to-date cross-sectional estimates and measures of change over time for all children who have not started kindergarten or for children in kindergarten through 12th grade, as is provided by the NHES.
Due to these limitations in extant studies and the importance of data collected by the NHES, NCES plans to continue to conduct the ECPP and PFI surveys under the NHES program. Appendix D contains a review prepared for NCES of other surveys that cover topics similar to those in the NHES child surveys. The review shows that there is little overlap between the NHES and these other surveys. There are no other federal surveys that collect information on the attainment of non-degree credentials; filling this gap is GEMEnA’s charge.
Not applicable.
This request is for clearance of the NHES:2015. Topics covered in child questionnaires for this NHES collection have been addressed in previous NHES administrations; repeating the surveys biennially allows for analysis of trends over time. Less frequent collection would result in incomplete tracking of these trends. The last full NHES study was conducted in 2012. Because of budget constraints, NCES did not conduct the NHES in 2014 but plans to resume a biennial data collection schedule beginning in 2015.
None of the special circumstances listed in the instructions for completing the supporting statement apply to the NHES:2015.
A Technical Review Panel (TRP) comprising leading experts in survey methodology was established to provide input to the redesign of the NHES system. Most members of the panel met in February 2010 to discuss the proposed design for the field test, and their comments and suggestions led to changes reflected in this submission.
Technical Review Panel Participants and Their Affiliation at the Time of TRP Recruitment
Nancy Bates
U.S. Census Bureau
649 A. St. N.E.
Washington, DC 20002
E-mail: [email protected]
Paul Beatty
National Center for Health Statistics
Division of Health Care Statistics
3311 Toledo Road,
Hyattsville, MD 20782
E-mail: [email protected]
Johnny Blair
Survey Sampling and Methodology
Abt Associates Inc.
4550 Montgomery Avenue
Bethesda, MD 20814-3343
E-mail: [email protected]
Stephen Blumberg
National Center for Health Statistics
3311 Toledo Road
Hyattsville, MD 20782
E-mail: [email protected]
Mick Couper
Survey Research Center
University of Michigan
ISR, 426 Thompson Street
Ann Arbor, MI 48104
E-mail: [email protected]
Don Dillman
Social and Economic Sciences Research Center, Professor
Washington State University
133 Wilson Hall
Pullman, WA 99164-4014
E-mail: [email protected]
Robert Groves
Survey Research Center, Institute for Social Research
University of Michigan
426 Thompson Street
Ann Arbor, MI 48106-1248
E-mail: [email protected]
Scott Keeter
Pew Research Center
1615 L. St. NW. Suite 700
Washington, DC 20036
E-mail: [email protected]
Kristen Olsen
Survey Research and Methodology
University of Nebraska-Lincoln
201 N. 13th St.
Lincoln, NE 68588-0241
E-mail: [email protected]
Roger Tourangeau
Joint Program in Survey Methodology
University of Maryland
1218 LeFrak Hall, University of Maryland
College Park, MD 20742
E-mail: [email protected]
Gordon Willis
Division of Cancer Control / Population Sciences
National Cancer Institute
6130 Executive Blvd, MSC 7344, EPN 4005
Bethesda, MD 20892-7344
E-mail: [email protected]
The content of the NHES:2015 topical surveys repeats the content developed for the NHES:2012 administration and prior NHES administrations. As a result, the PFI and ECPP surveys reflect the cumulative input of many experts in the field and past NHES Technical Review Panels. In order to ensure that the ECPP and PFI Surveys address important issues in the topical areas of interest and incorporate important emerging issues, the design phase of the 2012 study included consultations with experts in the substantive areas addressed in the surveys. These experts included persons in government agencies, academe, and research organizations.
Substantive Experts: ECPP and Their Affiliation at the Time of TRP Recruitment
Jerry West - Mathematica
Mathematica Policy Research, Inc.
600 Maryland Ave., SW, Suite 550
Washington, DC 20024-2512
E-mail: [email protected]
Ann Collins – Abt Assoc. Cambridge, MA
Abt Associates Inc.
55 Wheeler Street
Cambridge, MA 02138-1168
E-mail: [email protected]
Ron Haskins – Brookings Institution and Casey Foundation
The Brookings Institution
1775 Massachusetts Ave., NW
Washington, DC 20036
E-mail: [email protected]
Ivelisse Martinez-Beck – HHS Division of Child and Family Development
Administration for Children and Families
370 L’Enfant Promenade, S.W.
7th Floor West, Room 7A011
Washington, D.C. 20447
E-mail: [email protected]
Lynda Laughlin – Census
U.S. Census Bureau
4600 Silver Hill Road
Suitland, MD 20746
E-mail: [email protected]
Substantive Experts: PFI and Their Affiliation at the Time of TRP Recruitment
Richard Brandon – Univ. of Washington
Human Services Policy Center, Evans School of Public Affairs
University of Washington
1107 NE 45th St.
Seattle, WA 98105
E-mail: [email protected]
Annette Lareau – Univ. of Pennsylvania
Department of Sociology
University of Pennsylvania
McNeil Hall
Philadelphia, PA 19104
E-mail: [email protected]
Joyce Epstein – The Johns Hopkins University
Center for Social Organization of Schools
3003 N. Charles St., Suite 200
Baltimore, MD 21218
E-mail: [email protected]
Lawrence Aber - NYU
Steinhardt School of Culture, Education, and Human Development
New York University
82 Washington Square East
New York, NY 10003
E-mail: [email protected]
As noted above, the CWS and TWS are products of ongoing work guided by GEMEnA, which has met monthly since October 2009 and consists of senior staff from the Bureau of the Census, the Bureau of Labor Statistics, the Council of Economic Advisors, the National Center for Education Statistics, the National Center for Science and Engineering Statistics, the Office of Statistical and Science Policy (OMB), and the Office of the Under Secretary of Education. In addition, GEMEnA established an Expert Panel of substantive experts in the fields of workforce education, economic development, and non-degree credentials that met in November of 2012 and March of 2014 to provide input to content for CWS and TWS.
Survey and Methodology Experts: GEMEnA Member Agency Representatives
Census Bureau
Bob Kominski
Stephanie Ewert
Bureau of Labor Statistics
Dori Allard
Harley Frazis
National Center for Science and Engineering Statistics
Dan Foley
John Finamore
Council of Economic Advisors
Jordan Matsudaira
OMB Office of Statistical and Science Policy
Shelly Martinez
Department of Education – Office of the Under Secretary
Jon O’Bergh
National Center for Education Statistics
Sharon Boivin
Lisa Hudson
Kashka Kubzdela
Sarah Crissey
Sarah Carroll
Andy Zukerberg
Substantive Experts: GEMEnA Expert Panel Members
Jim
Van Erden
Senior
Policy Advisor
National Association of State Workforce
Agencies/
Information Technology Support Center
Washington,
DC
Evelyn
Ganzglass
Director
of Workforce Development
CLASP
Washington, DC
Parminder
Jassal
Executive
Director
ACT Foundation
Austin, TX
Morris
Kleiner
Professor
and Director of Graduate Studies
Humphrey School of Public
Affairs
University of Minnesota
Minneapolis, MN
James
Parker
Senior
Research and Policy Associate
Council for Advancement of Adult
Literacy
New York City, NY
Kent
Phillippe
Associate
Vice President,
Research and Student Success
American
Association of Community Colleges
Washington, DC
Kenneth
Poole
CEO/President
Center
for Regional Economic Competitiveness
Arlington, VA
Andrew
Reamer
Research
Professor
George Washington Institute of Public Policy
George
Washington University
Washington, DC
Jesse
Rothstein
Associate
Professor of Public Policy and Economics
Richard & Rhoda
Goldman School of Public Policy
University of California,
Berkeley
Berkeley, CA
Jeff
Strohl
Director
of Research
Center on Education and the Workforce
Georgetown
Public Policy Institute
Georgetown University
Washington,
DC
Michelle
Van Noy
Researcher
Heldrich
Center for Workforce Development
Rutgers, The State University
of New Jersey
New Brunswick, NJ
Holly
Zanville
Senior
Research Officer
Lumina Foundation
Indianapolis, IN
The NHES:2003 included an extensive experiment in the use of small cash incentives to improve unit response. The experiment demonstrated that gains in respondent cooperation could be realized with relatively modest cash incentives (Brick et al. 2006). Such incentives were used in NHES:2005 and NHES:2007. The NHES:2011 Field Test included an incentive experiment at the screener level testing the effect of including a $2 cash incentive on response rates compared to a $5 cash incentive in the initial screener mailing. The $5 screener was associated with higher response rates than the $2 incentive, so the $5 incentive was used in the NHES:2012. Preliminary results from the NHES-FS indicate that the $5 incentive is associated with higher response rates than no incentive. We will continue with this approach in the NHES:2015 and use a $5 cash incentive in the first screener questionnaire mailing.
The NHES:2012 included an incentive experiment at the topical level to further refine an optimal strategy for the use of incentives in the NHES. For those households in which a child was selected as the subject of an ECPP or PFI questionnaire, cases that responded to the first or second mailing of the screener received a $5 cash incentive with the initial topical survey mailing. Evidence from the 2011 Field Test indicated that topical response rates could benefit significantly by providing later screener respondents with a larger topical incentive. To confirm this finding, we subsampled late screener respondents (those responding to the 3rd or 4th questionnaire mailing) to receive either a $5 or $15 cash incentive with their first topical survey mailing. The results from the NHES:2012 indicate that among later screener responders, the $15 incentive was associated with higher response rates compared to the $5 incentive. Based on these findings, we will use the same strategy in the NHES:2015 and send a $5 cash incentive in the initial topical mailing to cases that responded to the first or second screener mailing and a $15 cash incentive in the initial topical mailing to any cases that responded later than three days after the third screener mailing.
Respondents will be informed of the voluntary nature of the survey and of the confidentiality provision in the initial cover letter and on the questionnaires, stating that their responses may be used for statistical purposes only and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002), 20 U.S. Code § 9573].
Additionally, all staff members and subcontractors working on the NHES and having access to the data are required to sign the NCES Affidavit of Nondisclosure. Notarized affidavits are kept on file by the contractor and submitted to NCES quarterly. In addition, all contractor staff members who have access to confidential data and work on the project more than 30 days are required to have a federal background check.
The NHES is a voluntary survey, and no persons are required to respond to it. In addition, respondents may decline to answer any question in the survey. Respondents are informed of the voluntary nature of the survey in the cover letter that accompanies the questionnaire, as well as on the actual questionnaire. At the same time, some items in the surveys may be considered sensitive by some respondents:
CWS and TWS: The CWS and TWS currently ask a question about income that may be considered sensitive:
Personal earnings in the past year (categorical).
A measure of income is important because educational attainment is statistically associated with income, and the empirical properties of the survey measures may differ for people with different income levels. The American Community Survey (ACS) was the source for most of the CWS and TWS employment and background items. Item response rates for income questions were reasonably high in the 2013 pilot test. The item response rate for personal earnings was 96.4.
PFI and ECPP: Child development and education experts consider economic disadvantage and children’s disabilities to be important factors in children’s school experiences and their activities outside of school. As a result, the child surveys contain measures of these characteristics, including:
Household income;
Receipt of public assistance in the form of Temporary Assistance to Needy Families (TANF), food stamps, and the Women, Infants, and Children program (WIC); and
Children’s disability conditions.
Measures of household income and government assistance are important because access to early childhood programs by children at-risk and the education involvement of families of children from different socioeconomic backgrounds is of interest to policymakers, child development specialists, and educators. These items are important in identifying children at-risk and have been administered successfully in previous NHES studies. Respondents are also asked the age at which they first became a parent to a child. This may be sensitive for parents in some situations.
The 2012 response rates for these items were very high. For total household income, the 2012 PFI survey had an item response rate of 95.4 percent. Item response rates for receipt of public assistance were also high: for Temporary Assistance to Needy Families, 97.9 percent; for the Women, Infants, and Children Program, 97.7 percent; and for Food Stamps, 98.4 percent. In the 2012 mail survey, it is not possible to examine item missing data for child disability because of the multiple response, list format of the question. Missing data may indicate either unreported data or that the child does not have a disability. However, in prior NHES collections, response to this item was high; in the 2007 PFI, the item response rates were over 99 percent. In the 2012 PFI, the item response rate for age at which the child’s parent first became a parent to any child was 96.2 for the first parent reported and 96.0 for the second parent reported.
ECPP Survey: In addition to the items above, the ECPP survey also includes questions about assistance to pay for child care. This measure is important to understand families’ and children’s access to early childhood programs.
PFI Survey: The PFI survey includes items concerning children’s school performance and difficulties in school. Among these are:
Children’s school performance and difficulties, including school grades, suspensions, and expulsions; and
Identification of children’s schools.
Items concerning school performance and difficulty are important to the PFI survey as correlates of parent and family involvement in children’s education. These items were asked in the NHES:2012 PFI and item response rates for these items were high: 99.0 percent for children’s grades, 97.6 percent for out-of-school suspension, and 97.5 percent for expulsion.
Another element of the surveys that may be sensitive to some parents is the identification of children’s schools. This feature allows analysts to link the NHES data to other NCES datasets containing additional information about schools, greatly enhancing the ability to examine the relationships between students’ and families’ experiences and the characteristics of schools. The item response rate for the identification of the child’s school was 97.0 percent in 2012.
The response burden per instrument and the total response burden are shown in table 1. The administration times for the screener, adult questionnaires, and child questionnaires are based on practice administrations and past experience. The expected number of respondents and number of responses are based on the expected numbers of completed surveys of each type, discussed in section B.1.3. The hourly rate of $21.50 is based on the average for all civilian workers from the March 2013 National Compensation Survey (http://www.bls.gov/news.release/ecec.t02.htm). There are no other costs to respondents. There are also no recordkeeping requirements associated with NHES. For the NHES:2015, a total of 32,240 burden hours are anticipated, resulting in a burden cost to respondents of approximately $693,160.
Table 1. Estimated response burden for NHES:2015
Interview forms |
Estimated time (minutes) |
Number sampled |
Anticipated Response Rate |
Estimated Number of respondents |
Estimated Number of responses |
Total time (hours) |
Screener |
8 |
200,000* |
70% |
124,600 |
124,600 |
16,613 |
CWS questionnaire |
10 |
61,882 |
70% |
43,317 |
43,317 |
7,220 |
TWS questionnaire |
10 |
16,000 |
70% |
11,200 |
11,200 |
1,867 |
ECPP questionnaire |
20 |
9,695 |
75% |
7,271 |
7,271 |
2,424 |
PFI questionnaire |
20 |
16,464 |
75% |
12,348 |
12,348 |
4,116 |
Study Total |
|
|
|
198,736 |
198,736 |
32,240 |
* Approximately 11% of addresses will be returned by USPS as invalid, reducing the final sample size to 178,000 addresses.
There are no costs beyond those presented in section A.11.
The total cost of NHES:2015 to the government is approximately $7.2 million over a period of 20 months. This includes all direct and indirect costs of the design, data collection, analysis, and reporting phases of the study, as well as the delivery of data sets to NCES.
The increase in the respondent burden is due to the fact that the last OMB clearance was for NHES 2012 full-scale data collection, which included only two topical surveys (ECPP and PFI), while the NHES 2015 full-scale data collection in this submittal includes four topical surveys (CWS, TWS, ECPP, and PFI).
Exhibit 2 presents the schedule of project activities for NHES:2015. The NHES:2015 will produce datasets, statistics, and reports. The following are the planned outcomes of the NHES:2015:
A fully documented public-use data set that will be available for download from the NCES website;
A codebook with weighted and unweighted frequencies of all variables; and
First Look Reports that highlight key findings from the study.
Exhibit 2. NHES:2015 schedule of major activities
Task |
Date of Scheduled Conduct/Completion |
Survey Instruments Formatting and Printing |
November-December, 2014 |
Data Collection Begins (advance letter mailing) |
January 2, 2015 |
Data Collection Ends |
August 31, 2015 |
Public-use data files released |
August 31, 2016 |
Restricted-use data files released |
September 30, 2016 |
The OMB authorization number and expiration date will be displayed on the questionnaires and web screener.
There are no exceptions to the certification statement.
1 Ineligible addresses are those that are undeliverable. Once a screener mailing for an address is returned as a postmaster return (PMR), the address will be coded ineligible.
2 Percentages based on population proportions from the Annual Social and Economic Supplement (ASEC) of the Current Population Survey (CPS).
3 NCES collects information on the following racial/ethnic groups in the NHES topical surveys: American Indian or Alaska Native, Asian, Black or African American, Hispanic, Native Hawaiian or other Pacific Islander, and White. Reported estimates by race typically use the following categories: White, non-Hispanic; Black, non-Hispanic; Hispanic; Asian or other Pacific Islander, non-Hispanic; and other, non-Hispanic. Areas with higher concentrations of Blacks and Hispanics are typically oversampled in order to ensure sufficient samples sizes to generate reliable estimates for these subgroups. In the Feasibility Study, these groups were not oversampled because no estimates will be published, instead comparisons will primarily be made across item distributions for the two split panel forms.
4 NHES:2007 also included an adult education module, but data collection was stopped due to low response rates.
5 For survey administration purposes only, NCES has titled both the CWS and the TWS as the Adult Training and Education Survey. NCES believes that this title has the most relevance to respondents and will promote higher response rates.
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
Author | WAITS_T |
File Modified | 0000-00-00 |
File Created | 2021-01-27 |