Justification

Volume I:

Request for Clearance for Cognitive Interviews for the 2009 National Household Education Survey (NHES) Study Draft Questionnaires

1850-0803

February 11, 2009

Justification

The Random Digit Dial (RDD) National Household Education Survey (NHES) like most RDD surveys, has experienced a rapid decline in response rates. As a result of these declining response rates, the National Center for Education Statistics (NCES) has implemented a multi-stage redesign of the study. The primary goal for a revised NHES is to increase response rates without increasing measurement or coverage error in the study. To achieve this objective, a number of modes including face-to-face interviews and Internet data collection were evaluated for the redesign. Based on this evaluation, it was determined that a mail out- mail back survey with telephone non response follow up design had the greatest potential to achieve the response rate goal within the budget and precision constraints of the study.

NHES screens households to identify eligibility for one of three possible topical interviews: Early Childhood Program Participation (ECPP), and two Parent-Family Involvement (PFI) in Education (School and Home schooled) versions. In order to minimize the mode effects that can arise from the switch to self-administered questionnaires, and maximize response, NCES plans to conduct extensive pretesting of the revised instruments and methods. The first step in this testing is to conduct cognitive interviews on the revised questionnaires and respondent contact materials.

Cognitive testing has been a feature of the NHES program in past surveys. The objective of the interviewing in 2009 is to identify and correct problems of ambiguity or misunderstanding, while maintaining the integrity of the items in the shift from interviewer administered to self administered. The result should be a set of questionnaires that are easier to understand and therefore less burdensome for respondents while also yielding more accurate information. The interviews will focus on three key areas:

Layout and Navigation. In past NHES administrations, the respondents did not see the questionnaires, and the instrument navigation and flow was controlled by a Computer Assisted Telephone Interviewing (CATI) program. In the self-administered form, both the appearance of the instrument and ease of correcting navigation are critical elements. The design, layout, and formatting of the questionnaires will be tested to learn whether participants find the instrument presentation motivating, whether the skip patterns are easily navigated, and to learn about the perceived burden associated with the instruments.

Screener. While some prior NHES cognitive research focused on the screener, the research was limited and conducted in the context of a telephone administration. The proposed redesign includes a self-administered screener. In addition to issues of layout and navigation, the cognitive research must examine engagement and salience, sensitivity, and motivation to complete the screener. Because of the proposed two-stage design, motivation to possibly continue with a later stage of the survey is an important aspect of respondent reactions to the screener.

Content. In the past, the cognitive research focused on respondent comprehension, knowledge, and motivation in order to assess the content, meaning, and sensitivity of the survey questions. These features of the instrument are still of great importance and must be examined in the self-administered format. It will be particularly important to examine interpretation and comprehension of questions that have been modified for self-administration.

Design

Cognitive interviews are intensive, one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions, or to answer a series of questions about the items they just answered. Techniques include asking probing questions, as necessary, to clarify points that are not evident from the think-aloud comments and responding to scenarios. Probes that will be used include,

• probes to verify respondents’ interpretation of the question (e.g. asking for specific examples of activities in which the respondent reports participating),

• probes about the meaning of specific terms or phrases used in the questions, or

• probes for experiences or ideas that the respondent did not think were covered by the question but we would have considered relevant.

Volume II of this submission includes draft questionnaires and mailing materials that will be tested during these interviews. Multiple variations of the screener and topical instruments will be tested. In order to boost response to the Screener, we will be testing the impact of adding respondent engagement items during a forthcoming pilot test. During this round of cognitive interviewing, we plan to test a variety of engagement items to identify those that generate the most interest. Some of the proposed items are included in the Volume II screeners. Additional items that will be qualitatively tested are listed in Attachment B.

Screener Variations:

• Grid and non-grid child rosters

• With and without respondent engagement questions

Topical Module Variations:

• Parent-Family Involvement (PFI) – homeschool and enrolled versions

• Early Childhood Program Participation (ECPP)

Contact Strategies:

• Letters (cover, FAQs,)

Respondents that do not meet the criteria for topical interviews (generally, those without children younger than 18) will only be asked to complete the screener and provide feedback on mailing materials. These interviews are expected to last one hour or less. Other respondents will complete the screener and topical survey. These interviews are expected to last 1.5 hours. To simulate the two stage design of the NHES, some topical survey participants will be asked to complete and return the screener by mail prior to arriving for their in person interview. The research will be iterative, in that question wording and forms design may change during the testing period in response to early findings.

To adequately test the NHES instruments, it is necessary to distribute the cognitive interviews across respondents who can represent the major variations of experience in the target population and, correspondingly, to raise the total number of participants so as to obtain sufficient numbers of similarly-situated respondents.

We propose to conduct a total of 81 interviews across the four surveys. The interviews will be conducted in two rounds. Table 1 shows the key demographic groups for the screener interviews. Tables 2 and 3 show the key groups for the topical survey interviews.

Table 1. Proposed key respondent characteristics for recruitment – screener questionnaire.

Key respondent characteristics			Round 1	Round 2
Parental Education	Number of Children in HH	Child Age/ Survey	N	N	Burden Hours (Total)
Less than HS	None		1	2	3
	1	0-5 (ECPP)	1	2	3
		5-18 (PFI)	1	2	3
	2-6	0-5 (ECPP)	1	2	3
		5-18 (PFI)	1	2	3
	7+	0-18 (ECPP & PFI)	1	2	3
HS Diploma	None		1	2	3
	1	0-5 (ECPP)	1	2	3
		5-18 (PFI)	1	2	3
	2-6	0-5 (ECPP)	1	2	3
		5-18 (PFI)	1	2	3
	7+	0-18(ECPP & PFI)	1	2	3
4 – yr Degree	None		1	2	3
	1	0-5 (ECPP)	1	2	3
		5-18 (PFI)	1	2	3
	2-6	0-5 (ECPP)	1	2	3
		5-18 (PFI)	1	2	3
	7+	0-18 (ECPP & PFI)	1	2	3
Total			18	36	54

NOTE: Racial/ethnic diversity will be sought across the above categories, but no fixed recruiting requirement for race/ethnicity pertains to any one group. Among the 27 households with children in the PFI population, four homeschooling families will be sought. In Round 2, at least five participants will be native Spanish speakers.

When the household has eligible children and time remains in the cognitive interview period, respondents will be presented with a topical survey package to ascertain their response to the package and their understanding of which child is shown as having been selected for the topical survey. If time permits, they will be asked to complete some or all of the topical survey instrument.

Table 2. Proposed key respondent characteristics for recruitment – PFI questionnaire.

Key respondent characteristics
Parental Education	Enrollment Type	Child Grade	N	Burden Hours (Total)
Less than HS	Public/Private	K-5	2	3
		6-12	2	3
	Homeschool	K-5	1	1.5
		6-12	1	1.5
HS Diploma	Public/Private	K-5	2	3
		6-12	2	3
	Homeschool	K-5	1	1.5
		6-12	1	1.5
4 – yr Degree	Public/Private	K-5	2	3
		6-12	2	3
	Homeschool	K-5	1	1.5
		6-12	1	1.5
Total			18	27

NOTE: Racial/ethnic diversity will be sought across the above categories, but no fixed recruiting requirement for race/ethnicity pertains to any one group. Two native Spanish speakers will be included.

Table 3. Proposed key respondent characteristics for recruitment – ECPP questionnaire.

Key respondent characteristics
Parental Education	Care Type	Child Age	N	Burden Hours (Total)
Less than HS	Non-Parental	0-2	1	1.5
		3-5	1	1.5
	None / Parental Care	0-5	1	1.5
HS Diploma	Non-Parental	0-2	1	1.5
		3-5	1	1.5
	None / Parental Care	0-5	1	1.5
4 – yr Degree	Non-Parental	0-2	1	1.5
		3-5	1	1.5
	None / Parental Care	0-5	1	1.5
Total			9	13.5

NOTE: Racial/ethnic diversity will be sought across the above categories, but no fixed recruiting requirement for race/ethnicity pertains to any one group. Two native Spanish speakers will be included.

Consultations Outside the Agency

Past versions of the questionnaires, from the 1989 field test through the 2007 national study, have been reviewed during development by technical review panels composed of individuals with expertise on issues included in those studies. The cognitive laboratory work proposed here will focus on items developed for the 2007 Parent and Family Involvement in Education Survey (PFI) and the 2005 Early Childhood Program Participation Survey (ECPP). In addition, items on the Screener will be drawn from the 2007 NHES Screener, some items from the 2005 Adult Education Survey (AE) and longstanding Gallup and Pew surveys about education.

Recruiting and Paying Respondents

Participants completing the one-hour screener cognitive interview will receive $40 and those completing the 1.5 hour topical interview will receive $60. The amount offered is consistent with honorariums offered by Westat for cognitive interviews on other Federally sponsored projects and is consistent with NCES practice. Additionally, this amount is in line with incentives offered by focus group facilities and research firms for general population interviews of this length in the Washington, DC area. The amount is necessary to attract a wide spectrum of income and demographic groups.

Westat and Child Trends will recruit participants using multiple sources, including a database of prescreened research volunteers, advertisements in print and online publications, community organizations, and personal contacts. People who have participated in cognitive studies or focus groups in the past 12 months and Westat employees will be excluded from participating.

Assurance of Confidentiality

Participation is voluntary and written consent will be obtained from respondents or from legal guardians of minor respondents before interviews are conducted. No personally identifiable information will be maintained after the cognitive laboratory analyses are completed. A copy of the respondent consent form can be found in Attachment A.

Estimate of Hour Burden

We expect the cognitive interviews for the screener questionnaire to be approximately one hour in length. For the round 1 interviews this will result in 18 burden hours, for round 2 the interviews will result in 36 burden hours. The topical instrument cognitive interviews are expected to last approximately one and one-half hours. For the PFI questionnaire interviews this will result in 27 burden hours. For the ECPP questionnaire interviews this will result in 13.5 burden hours. Thus, the estimated total respondent burden will be: 94.5 hours.

Estimate of Cost Burden

There is no direct cost to respondents.

Cost to Federal Government

The estimated cost to the Federal government to conduct this cognitive research is $40,000.

Project Schedule

Two stages of screener testing and one stage of topical interview testing are planned. The project schedule calls for the first phase of cognitive research, focusing on the screener, to be conducted during the week of March 9. Interviews focused on the ECPP and PFI instruments will be conducted during the week of April 13. The second round of screener interviews is planned for the week of May 18. Please note that this cognitive research will not be completed before pilot test materials need to be submitted for OMB review.

Supporting Documents – Volume II Table of Contents

File Name	Description
SCNR Letter V2 09-0206	Cover letter that will accompany the screener.
NHES Screener FAQ’s 09-206	Frequently asked questions that may be included with the cover letter or on the questionnaire
SCNR – Core V1 09-126	Core (no engagement items) screener questionnaire with non grid response format
SCNR – Core V2 09-128	Core (no engagement items) screener questionnaire with grid response format
SCNR – Grid option 2 for cog test 09-0205	An alternate grid format for testing
SCNR – Survey V1 09-126	Extended screener questionnaire (contains engagement items) with non grid response format
SCNR – Survey V2 09-128	Extended screener questionnaire (contains engagement items) with grid response format
ECPP Mail Draft Feb 10	Current draft of the Early Childhood Program Participation (ECPP) questionnaire.
PFI Enrolled Mail Draft Feb 10	Current draft of the Parent Family Involvement (PFI) questionnaire for students enrolled in a school
PFI Homeschooled Mail Draft Feb 10	Current draft of the Parent Family Involvement questionnaire for students that are homeschooled
Topical Letter Feb 11	Cover letter that will accompany a topical survey (PFI or ECPP)

CONSENT FORM

Westat is conducting research for the U.S. Department of Education. The purpose of this interview is to discuss your reactions to survey questions about education issues. Your opinions are important and will assist in the development of the questionnaire.

• The interview will be audio-taped and notes will be taken. In addition, project researchers may be observing the interview.

• The interview should take about an hour.

• Your participation is completely voluntary. You may stop at any time, and you do not have to answer any questions you do not wish to answer.

• Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002). Your responses will not be published in an identifiable way. Your information will be used to help design future surveys.

• You will receive $40 cash token of our appreciation for completing the session.

If you agree to participate in this interview, please sign the following statement:

I have read this consent form and understand the proposed project.

I consent to participate in this study.

Signature Date

Printed Name

Additional Items To Be Tested Qualitatively

Do you think all children in America's are receiving the education they need? (modified from CER/ICR 1997 poll)

Do you know if your local teachers are paid on their performance or on some other criteria? (CER/ICR 1997 poll)

Do you believe the federal government should play a major role, a minor role, or no role at all in making policy decisions for schools? (CER/ICR 1997 poll)

Here is a list of problems schools might face. For each one, please tell me how big a problem you think it is — a major problem, a minor problem, or not a problem for the public schools in your community. How about… ? (NPR/Kaiser study)

a. Students who are undisciplined and disruptive

b. Lack of adequate academic standards

c. Overcrowded classrooms

d. Discrimination against children because of race or gender

e. Public school facilities that are unsafe or unhealthy

f. Lack of computers and technology

g. Violence and lack of school safety

h. Lack of parental involvement

i. Poor quality teachers

j. Student use of alcohol or illegal drugs

k. Poor school administration

l. Inequality in funding among school districts

Which is a more important role for schools — to give students academic skills and other knowledge to prepare them for a job, a career, or college OR to develop students' character so they can make responsible decisions as adults in society about such things as drugs, sex, family, and money? (NPR/Kaiser study)

Which of the following two statements comes closer to your view: Schools need to return to their main function of giving knowledge to students, OR schools need to focus on teaching children how to think critically rather than worrying about how much detailed knowledge they have? (NPR/Kaiser study)

Which of the following two statements comes closer to your view: Schools have gotten too far away from the basics, like reading, writing, and math, OR schools need to teach about a broader range of subjects than they used to, because the world is more complex today? (NPR/Kaiser study)

As you look back on your elementary and high school education, is it your impression that children today get a better or worse education than you did? (NPR/Kaiser study)

In your opinion, how much does the amount of money spent on a public school student's education affect the quality of his or her education—a great deal, quite a lot, not too much, or not at all?

(NPR/Kaiser study)

Do you think that the Federal government is spending too much money on public schools, too little money, or about the right amount of money on public schools? (SRBI/Time poll 2006)

1. Too much

2. Too little

3. About the right amount

9. (VOL) No answer/Don’t know

Do you think that the public schools overall are teaching students the skills they will need for our economy in the 21st century, or not? (SRBI/Time poll 2006)

1. Teaching skills they will need

2. Not teaching them

9. (VOL) No answer/Don’t know

What percent of American students do you think graduate from high school?

Your best estimate is fine. (SRBI/Time poll 2006)

____ ____% Range (0-99)

100%

101 (VOL) No answer/Don’t know

Who is most to blame when some public schools fail to improve test scores? Is it the students themselves, the parents, the teachers, the school districts, or is it because the schools don’t have enough money to improve scores? (SRBI/Time poll 2006)

1. Students

2. Parents

3. Teachers

4. School districts

5. Not enough money

9. (VOL) No answer/Don’t know

Please indicate how much confidence you have in Public Schools a great deal, quite a bit, some, very little, or none at all?... (modified from NBC News/Wall Street Journal Poll January, 2009)

2