Justification

Volume I:

Request for Clearance for Cognitive Interviews for the Year 2008 National Indian Education Study Draft Questionnaires

1850-0803

February 29, 2008

Justification

The National Indian Education Study (NIES) is designed to describe the condition of education of American Indian/Alaska Native (AI/AN) students in the United States. The study is being conducted by the National Center for Education Statistics (NCES) for the U.S. Department of Education, Office of Indian Education (OIE). A component of this study involves background questionnaires for students, teachers, and school administrators to collect information on the characteristics and educational circumstances of AI/AN students. An initial data collection occurred in May 2005, a second in 2007, and a third is planned for 2009.

Although the NIES background questionnaires were administered successfully in 2005 and 2007, anecdotal evidence from the field staff, along with comments from a technical review panel (TRP) and members of the Native American/Alaska Native community indicated that there could be problems with the interpretation of some items for some respondents. In other cases respondents might question the relevance of items to their circumstances. Both of these types of problems reflect the challenges of designing questionnaires that work well across a broad range of respondents who have divergent experiences relevant to the constructs being surveyed. For the constructs of interest to the NIES, respondents’ experiences will vary along several important dimensions depending upon whether they attend or serve schools with high or low concentrations of American Indian and Alaska Native students. In addition, the experience that particular respondents bring to the survey task will be significantly influenced by the variations in American Indian and Alaska Native cultural practices, and the vocabulary used to describe these practices, in different regions of the country.

For 2008 we have the opportunity to re-evaluate and correct the more problematic items from previous surveys, as well as to address some new constructs of interest to the American Indian and Alaska Native education community. In-depth interviewing using cognitive interview techniques offers the best tool for evaluating all of these issues of item interpretation.

The objective is to identify and correct problems of ambiguity or misunderstanding, or other difficulties respondents have answering questions. The result should be a set of questionnaires that are easier to understand and therefore less burdensome for respondents while also yielding more accurate information.

Design

Cognitive interviews are intensive, one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions. Techniques include recording the think-aloud comments and asking probing questions, as necessary, to clarify points that are not evident from the think-aloud comments. These probes might include, for example

• probes to verify respondents’ interpretation of the question (e.g. asking for specific examples of activities in which the respondent reports participating),

• probes about the meaning of specific terms or phrases used in the questions, or

• probes for experiences or ideas that the respondent did not think were covered by the question but we would have considered relevant.

Volume II of this submission includes draft questionnaires and corresponding cognitive interview protocols for each respondent type.

The numbers of respondents who must be interviewed in order to identify item problems can be estimated from the usability testing literature. Nielsen and Molich (1990)¹ have shown that usability testing can detect almost half of all major usability problems in a product with three participants. In addition, Virzi (1992)² found that four to five participants were adequate to detect 80 percent of the usability problems with a product. With ten participants, all of the global problems and 90 percent of the problems were detected. To adequately test the NIES instruments, however, it is necessary to distribute the cognitive interviews across respondents who can represent the major variations of experience in the target population and, correspondingly, to raise the total number of participants so as to obtain sufficient numbers of similarly-situated respondents.

We propose to sample 30 schools, in five geographic regions, to reach the major American Indian and Alaska Native cultural groups. Among the 30 schools, we will also include different types of schools (e.g., public schools, charter schools, Bureau of Indian Education operated schools, tribal operated schools, and other private schools; day schools and boarding schools; urban schools and rural schools; high density schools and low density schools). At each school, we plan to interview two adults and two to four students using the revised survey instruments. The larger student samples allow us to try different variants of some items, including re-evaluating question wording used in previous surveys.

The four to six respondents per school should provide adequate coverage to identify problems with survey questions related to the particular circumstances represented by that school. At the same time, we will cumulate responses to each type of survey (i.e., school administrator, grade 8 teacher, grade 4 teacher, grade 8 student, and grade 4 student) across schools, which will provide more than adequate opportunity to identify any more general problems with the survey instruments.

Specifically, we plan to distribute our school sample as shown in the following table. We have examined the distribution of high density schools and believe that schools in these areas are sufficiently clustered to allow us to visit all of the schools on a single itinerary. Furthermore, there are sufficient high density schools in these areas to allow some latitude in recruiting. With regard to low density schools, there are many more of these across the country, so it should not be a problem to recruit low density schools in each site visit area. Because close to 50% of American Indian and Alaska Native students attend low density schools, the sample for cognitive interviewing reflects this distribution.

Proposed distribution of schools for cognitive interviewing
Region	Number of high density schools	Number of low density schools
Atlantic	3	3
North Central	3	3
South Central	3	3
Mountain	3	3
Pacific	3	3

Respondents will be distributed across survey instruments as follows:

Numbers of respondents per survey instrument
Administrators	20
Grade 8 teachers	20
Grade 4 teachers	20
Grade 8 students	40
Grade 4 students	40
TOTAL	140

The cognitive interview period will be spread across four to five weeks, which will also allow us time to review responses on a flow basis and to develop and try out new variants of particularly problematic questions as the interviews progress.

Consultations Outside the Agency

The questionnaires have been reviewed during development by a technical review panel composed of individuals with expertise on American Indian and Alaska Native education as well as personnel from the Office of Indian Education. These individuals, who are listed in Attachment A, will also review the revisions prompted by the cognitive interviews, along with supporting evidence.

Recruiting and Paying Respondents

Student interview respondents will receive a stipend of $35. Adult respondents will receive a stipend of $45. The amount offered is consistent with NCES guidelines.

To recruit for this study, we are following a two-stage process used successfully in previous rounds of questionnaire development. We first identify a school willing to help recruit appropriate respondents. The school point of contact then distributes informational fliers to teachers and administrators and puts us in touch with willing volunteers. For students, the school sends informational fliers home to parents and secures parent permission forms in advance of our visit.

Schools decide whether they make adult respondents available to us to carry out the interviews during the respondents’ regular work day, or whether respondents meet with us on their own time. Similarly, schools decide whether they want students to meet with us at some point during the school day or after school.

For these reasons, we plan to let schools decide whether subject payments be combined into a single school contribution or paid to the individual respondents. In the prior round of questionnaire development, some schools chose one option and some chose the other.

In either case, the $35 incentive for students is important as a motivator for the school to pursue the appropriate parent permissions and/or for the parent to return the consent forms. It is, of course, critically important that the parent permissions are collected in a timely manner so that our time at the school is not wasted.

Assurance of Confidentiality

Participation is voluntary and written consent will be obtained from respondents, and from legal guardians of minor respondents, before interviews are conducted. No personally identifiable information, either by schools or respondents, will be gathered.

Justification for Sensitive Questions

Throughout the item development process, effort has been made to avoid asking for information that might be considered sensitive or offensive. Consultants and internal reviewers have been asked to identify and eliminate potential bias in questions.

Estimate of Hour Burden

Each cognitive interview is expected to take one hour. Therefore, the estimated respondent burden will be:

Respondent	Hours per respondent	Number of respondents	Total
Administrators	1	20	20 hours
Grade 8 teachers	1	20	20 hours
Grade 4 teachers	1	20	20 hours
Grade 8 students	1	40	40 hours
Grade 4 students	1	40	40 hours
TOTAL	1	140	140 hours

Estimate of Cost Burden

There is no direct cost to respondents.

Cost to Federal Government

Cost to the Federal government to conduct the cognitive interviews, analyze the results, and revise the draft questionnaires, is estimated to be $140,000.

Project Schedule

February 15, 2008	ETS finalizes surveys for review
February 29, 2008	NCES/ETS completes technical panel review and expanded committee reviews
February 29, 2008	ETS submits fast-track OMB approval package for cognitive laboratory study
March 1—March 21, 2008	ETS trains Windwalker and AIR staff in Washington, DC.
March 21, 2008	OMB provides fast-track approval for cognitive laboratory study
March 21, 2008 – May 1, 2008	ETS recruits respondents and conducts cognitive interviews
May 1, 2008 – May 17, 2008	ETS completes revisions based on cognitive interviews; submits report and revised surveys
May 17, 2008	ETS submits NIES surveys to NCES
May 17 – May 28	NCES Review
May 28 – June 3	ETS incorporates NCES revisions
June 3 – June 9	NCES submits to NAGB
June 9 – 23, 2008	NAGB reviews NIES surveys
June 30, 2008	ETS incorporates NAGB revisions
July 7, 2008	ETS Submits OMB Clearance Package to NCES
July 15, 2008	NCES Submits Clearance Package to OMB
September 15, 2008	OMB Approval

Name/Home Address	Business Address	E-Mail and Telephone
henry braun 514 Bergen Street Lawrenceville, NJ 08648	Lynch School of Education Boston College 140 Commonwealth Avenue Chestnut Hill, MA 02467	Home: 609-577-4355 Business: E-Mail: [email protected]; [email protected]
BRYAN BRAYBOY 726A North Chandalar Drive Fairbanks, AK 99775	Visiting President’s Professor of Education University of Alaska-Fairbanks (2007-08) and Borderlands Associate Professor, Division of Educational Leadership and Policy Studies Arizona State University	Home: Business: E-Mail: [email protected]
ROBERT COOK 6690 Leisure Lane Black Hawk, SD 57718	Crazy Horse Memorial Cultural Specialist/Education Outreach Avenue of the Chiefs Crazy Horse, SD 57730	Home: Business: E-Mail: [email protected]
STEVE CULPEPPER 2100 Snowmass Circle Broomfield, CO 80020	Assistant Professor Department of Human Services Campus Box 12, P. O. Box 173362 Metropolitan State College of Denver Denver, Colorado 80217-3362	Home: Business: 303-556-3585 E-Mail: [email protected]
SUSAN FAIRCLOTH 1124 Old Boalsburg Road State College, PA 16801	The Pennsylvania State University Educational Leadership Suite 300, Rackley Building University Park, PA 16802	Home: 814-777-3290 Business: E-Mail: [email protected]
VALERIA LITTLECREEK Route 4, Box 159 Okemeh, OK 74854	Oklahoma State Department of Education Director of Tribal Affairs Indian Education Section, Suite 215 2500 North Lincoln Boulevard Oklahoma City, Oklahoma 73105-4599	Home Business 405-522-1591 Fax: 405-522-0611 Email: [email protected]
CHRISTOPHER LOHSE 1047 Breckenridge Helena, MT 56901	Director of Policy Research and Federal Liaison Office of Public Instruction PO Box 202501 Helena, MT 59620-2501	Home: 918-290-0586 Business: E-Mail : [email protected] [email protected]
LARRY LUDLOW 23 Duggan Drive Framingham, MA 01702	The Lynch School of Education Chair The Department of Educational Research, Measurement and Evaluation Boston College Campion Hall 336C 140 Commonwealth Ave Chestnut Hill, MA 02467-3813	Home Business: 617- 552-4221 E-Mail [email protected]
DEBRA NORRIS 4165 N. Western Winds Drive Tucson, AZ 85705	Indian Education Arizona Department of Education Bin #32 1535 West Jefferson Street Phoenix, AZ 85007	Business: 602-542-2784 Home: E-Mail: [email protected]
TED WRIGHT 2600 2nd Avenue #414 Seattle, WA 98121	Director, School-Community Leadership Consortium. Core faculty, Center for Programs in Education Antioch University Seattle 2326 Sixth Avenue Seattle, WA 98121	Home Business: E-Mail: [email protected]; [email protected]
TARAJEAN YAZZI-MINTZ P.O. Box 7674 Bloomington, IN 47407	Indiana University School of Education, Curriculum & Instruction 201 N. Rose Avenue Bloomington, IN 47405-1006	Business: Home: 812-331-0935 E-Mail: [email protected]

Extended Committee Members

NAME	AFFLIATION	E-MAIL	ADDRESS	PHONE
Marilyn Balluta	ASD Education Center	[email protected]	5530 E. Northern Lights Blvd.	742-4445
Leslie Caye	Ronan School District #30	[email protected]
Keith Moore	South Dakota Department of Education	[email protected]		605-773-6118
Ann Saenz		[email protected]		907-330-8013
LaRayne Leija		[email protected]	704 S. River Street Chamberlain, SD 57325	605-234-5233
Mike Tulee		[email protected]		206-367-1123

1 Nielsen, J. & Molich, R. (1990). Heuristic evaluation of user interfaces. Proceedings of the ACM CHI'90, pp. 249-256.

2 Virzi, R. (1992). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 34, 457-468.

2