Justification

Volume I:

Request for Clearance for Cognitive Interviews for the 2010 National Household Education Survey (NHES) Study Draft Questionnaires

1850-0803

February 25, 2010

Justification

The Random Digit Dial (RDD) National Household Education Survey (NHES) like most RDD surveys, has experienced a rapid decline in response rates. As a result of these declining response rates, the National Center for Education Statistics (NCES) has implemented a multi-stage redesign of the study. The primary goal for a revised NHES is to increase response rates without increasing measurement or coverage error in the study. To achieve this objective, a number of modes including face-to-face interviews and Internet data collection were evaluated for the redesign. Based on this evaluation, it was determined that a mail out- mail back survey with telephone non response follow up design had the greatest potential to achieve the response rate goal within the budget and precision constraints of the study. The NHES is a two stage study. In the first stage households are screened to determine if they contain eligible members. If eligible members are in the household, within household sampling is performed. Finally, topical module(s) are administered to the selected household members. Initially, there were concerns about the feasibility of conducting this screening operation by mail. A small pilot test of the methodology was undertaken during Fall 2009. The results of this pilot indicate that it is possible to conduct the NHES by mail and obtain a reasonable response rate while maintaining coverage of the target population. A large scale field test is planned to further refine the data collection protocols in 2011. The purpose of this submission is to conduct cognitive interviews to refine instruments and procedures for the 2011 field test. A request for clearance of the 2011 field test will be submitted separately in July 2010.

Cognitive testing has been a feature of the NHES program in past surveys. The objective of the interviewing in 2010 is to identify and correct problems of ambiguity or misunderstanding in question wording and respondent materials. The result should be a set of questionnaires that are easier to understand and therefore less burdensome for respondents while also yielding more accurate information. The interviews will focus on three key areas:

Layout and Navigation. The 2009 pilot results indicate that respondents had difficulty following certain skip patterns in the questionnaire. Additionally, item non-response was elevated on ‘mark all that apply’ items. The cognitive interviews will look at alternative skip patterns and question formats.

Screener. Four screeners were tested in the 2009 pilot: a screenout screener that only asked for sampling information, a core screener that asked for sampling information and some household characteristics, an engaging screener that asked a series of education and policy questions in addition to the core screener items, and an English/Spanish version of the core screener. The screeners appeared to be successful at obtaining response during the 2009 pilot, however some areas for improvement were uncovered. For example, respondents did not provide children’s names at the same rate in each of the screener forms. Response and coverage rates varied across the different screener versions. In 2011, we plan to test a modified screenout, engaging, and bilingual questionnaires. Proposed changes to the questionnaires include: not asking for children’s names, collecting phone number on the screenout instrument, new engaging items, and revisions to the mailing package (cover letter, cover page, and envelopes ). These cognitive interviews will be used to inform the design process and test changes to the instrument for respondent comprehension and appeal.

Screener to topical transition. Because of the NHES two-stage mailing administration, the interviews will evaluate the transition from the screener to the topical survey. It is important that we evaluate the respondent’s understanding of the transition from the screener to the topical to ensure the screener survey encourages response at the second stage. The screener is also critical to correctly sampling a child for participation in the second stage.

Content. As a result of the change to self administered mode, many of the NHES items were rewritten. An expert review has suggested that some items be further revised to improve data quality. It is important to test all of the items with respondents for comprehension and consistency of response. This round of cognitive interviewing will focus heavily on item wording.

Design

Cognitive interviews are intensive, one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions, or to answer a series of questions about the items they just answered. Techniques include asking probing questions, as necessary, to clarify points that are not evident from the think-aloud comments and responding to scenarios. Probes that will be used include,

• probes to verify respondents’ interpretation of the question (e.g. asking for specific examples of activities in which the respondent reports participating),

• probes about the meaning of specific terms or phrases used in the questions, or

• probes for experiences or ideas that the respondent did not think were covered by the question but we would have considered relevant.

Interviews will be conducted by trained cognitive interviewers. Volume II of this submission includes draft questionnaires and mailing materials that will be tested during these interviews, as well as preliminary protocols for conducting the interviews. Multiple variations of the screener and topical instruments will be tested. It is expected that the instruments, mailing materials, and interview protocols will evolve during testing.

Screener Variations for testing:

• Screenout with phone number

• Screeners with and without items for children’s name

• New respondent engagement questions

Topical Module Variations for testing:

• Parent-Family Involvement (PFI)

• Early Childhood Program Participation (ECPP)

Contact Strategies:

• Letters

• Cover page

• FAQ’s

• Envelopes

• Bilingual forms

Approximately 30 participants will only be asked to complete the screener and provide feedback on mailing materials. These interviews will allow us to focus more intensely on the initial contact package. These interviews are expected to last one hour or less. Another 40 respondents will complete the screener and topical survey. These interviews are expected to last 1.5 hours. To simulate the two stage design of the NHES, some topical survey participants will be asked to complete and return the screener by mail prior to arriving for their in person interview. The research will be iterative, in that question wording and forms design may change during the testing period in response to early findings.

To adequately test the NHES instruments, it is necessary to distribute the cognitive interviews across respondents who can represent the major variations of experience in the target population and, correspondingly, to raise the total number of participants so as to obtain sufficient numbers of similarly-situated respondents.

We propose to conduct a total of 70 interviews across the four surveys. The interviews will be conducted in two rounds. Table 1 shows the key demographic groups for the screener interviews. Tables 2 and 3 show the key groups for the topical survey interviews.

Table 1. Approximate key respondent characteristics for recruitment – screener and topical questionnaires.

Approximately 30 screener and mailing package only interviews:

5 parents or guardians of students aged K-12 in public or private school

8 parents of children under 2

5 parents of children 2-6 but not yet in kindergarten

7 respondents from households without children

5 households with children not in school

Approximately 40 screener and topical interviews:

15 parents or guardians of students aged K-12 in public school.

5 parents or guardians of students aged K-12 in private school.

About 10 parents or guardians of children under 2.

About 10 parents or guardians of children between 2 and 6 but not yet in kindergarten.

Among the 20 people who have children ages 0-6:

3-5 parents or guardians of a child who is now receiving care, regularly scheduled at least once a week, from a relative other than a parent (e.g. from grandparents, brothers or sisters, or any other relatives).

3-5 parents or guardians of a child who is now receiving care, regularly scheduled at least once a week, in a home from someone who is not related to the child.

10 parents or guardians of a child who is now attending a daycare center, preschool, or prekindergarten, not in a private home, regularly at least once a week.

Among all 70 people:

5-10 people whose highest level of education is high school or less.

10-15 people who are non-White.

2 parents or guardians of children with a condition such as a specific learning disability, an orthopedic impairment, a speech or language delay, a serious emotional disturbance, deafness or another hearing impairment, blindness or another visual impairment not corrected with glasses, mental retardation, autism, Attention Deficit Disorder, ADD or ADHD, Pervasive Developmental Disorder or PDD, or another health impairment lasting 6 months or more.

3-5 people who speak only Spanish

The primary deliverable from this study will be revised questionnaires and contact materials. A report highlighting key findings will be prepared and submitted to the NCHS Qbank.

Consultations Outside the Agency

Past versions of the questionnaires, from the 1989 field test through the 2007 national study, have been reviewed during development by technical review panels composed of individuals with expertise on issues included in those studies. The cognitive laboratory work proposed here will focus on items developed for the 2007 Parent and Family Involvement in Education Survey (PFI) and the 2005 Early Childhood Program Participation Survey (ECPP). In addition, items on the Screener will be drawn from the 2009 NHES Pilot Screener, some items from the 2005 Adult Education Survey (AE), and longstanding Gallup, Pew, and other public opinion surveys about education.

Recruiting and Paying Respondents

Participants completing the one-hour screener cognitive interview will receive $40 and those completing the 1.5 hour topical interview will receive $60. These are the same amounts that were offered in the 2009 NHES cognitive interviews. These amounts are necessary to attract a wide spectrum of income and demographic groups. The participants receiving $60, in addition to filling out and evaluating the content of topical questionnaires, will also evaluate the transition from the screener to the topical, requiring them to fill out the screener in advance and to comment during the interview about the ease of understanding of the purpose of the topical as related to the initial screener. It is important that we evaluate the respondent’s perceptions about the ease and understanding of the transition from the screener to the topical to ensure the topical survey encourages response to the second survey and accuracy of reporting about the sampled child selected at the screener level.

Participants will be recruited using multiple sources, including a database of prescreened research volunteers, advertisements in print and online publications, community organizations, and personal contacts. People who have participated in cognitive studies or focus groups in the past 12 months and employees of the firms conducting the research will be excluded from participating. The items used to screen respondents for participation are in Volume II. We anticipate that it will take 3 minutes per screening interview.

Assurance of Confidentiality

This study is being conducted under the Education Sciences Reform Act (Public Law 107-279). All responses that relate to or describe identifiable characteristics of individuals will be used only for statistical purposes and will not be disclosed, or used, in identifiable form for any other purpose, unless otherwise compelled by law. Participation is voluntary and a written consent will be obtained from respondents. No personally identifiable information will be maintained after the cognitive laboratory analyses are completed. A copy of the respondent consent form is attached at the end of this document.

Estimate of Hour Burden

Table 2. Burden time by response type.

Respondents	Number of Respondents	Number of Responses	Burden Hours per Respondent	Total Burden Hours
Screener Recruitment Interview	840	840	0.05	42
Screener Review	30	30	1	30
Screener and Topical Review	40	40	1.5	60
Total	840	910	-	132

We expect the cognitive interviews for the screener questionnaire to be approximately one hour in length. This will result in 30 burden hours. The topical instrument cognitive interviews are expected to last approximately one and one-half hours. This will result in 60 burden hours. Screening potential participants will require 3 minutes per screening and we anticipate approximately 12 screening interviews per eligible participant. This will result in 42 hours of burden. Thus, the estimated total respondent burden for this research will be: 132 hours.

Estimate of Cost Burden

There is no direct cost to respondents.

Cost to the Federal Government

The total cost for conducting the cognitive interviews is expected to be $120,000.

Project Schedule

The project schedule calls for the first phase of cognitive research focusing on the topical interviews to be started the week of March 15. Interviews focused on the screener instruments will be conducted during the week of April 12. Interviewing is expected to last until June 14th. After interviewing, instruments will be finalized and submitted as part of a request for clearance of the 2011 field test in July 2010.

Assurance of Confidentiality

The American Institutes for Research and XXX are conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. This study is authorized by law under the Education Sciences Reform Act (Public Law 107-279). Your participation is voluntary. Your responses are protected from disclosure by federal statute (P.L. 107-279, Title 1, Part E, Sec. 183). All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose, unless otherwise compelled by law.

Please sign below to indicate that you have read the assurance of confidentiality.

Print Name ___________________________________

Signature _____________________________________

Date_________________________________________

2