Response to OMB Comments

NAEP Wave 1 2010 Responses to OMB's Questions.docx

National Assessment of Educational Progress 2008-2010 System Clearance

Response to OMB Comments

OMB: 1850-0790

Document [docx]
Download: docx | pdf

Questions for NAEP: Response to public comment:




1.       Explain what statistical method was used to estimate the interview time.    


Two methods are used to estimate the time it takes to complete survey questions. One is a review of the survey literature in various fields. The results of that review are attached. The review suggests that most Likert question types, which are predominant on the Teacher and School surveys, require approximately 7 to 13 seconds per item. This puts both surveys, which ask up to 45 and 58 questions, respectively, well within the burden guidelines of 20 and 30 minutes, respectively.  The second method is a cognitive lab timing study. Cognitive labs are conducted whenever a new framework is implemented for a survey. The most recent new framework implemented was for the 2011 Writing Assessment. A cognitive lab was conducted for that study in December, 2008. However, timing results were only determined for new items; although all items were new for the student questionnaire, only a few items were new for the Teacher and School questionnaires.



2.       What procedures will be used in FY2009 and future years to validate the accuracy of these estimates?


In the future we will conduct additional cognitive lab studies for a subset of complete Teacher and School Questionnaires administered each year, even when such questionnaires do not reflect new frameworks. The general method will involve having participants first complete the survey for timing purposes, and then have participants go back through the survey question by question, to help clarify ambiguous wording, identify misunderstandings, and serve the primary purpose of the cognitive labs, which is to improve the quality of the survey and the data obtained from it.


Completion Time in Adult Surveys/Questionnaires


  • Literature crosses psychology, sociology, politics (particularly polling), and public policy

  • Table below shows representative set of studies, where completion time is indicated in the study

  • Because target is teacher surveys, the topic search was restricted to adult samples

  • Format, administration type, question length, and item readability appear to account for the range of reported values (see also, Fricker, Galesic, Tourangeau, & Yan, 2005, Public Opinion Quarterly, 69, 370-392)

  • Studies where response time alone has been recorded, suggest the following: “Questions about facts that are less salient or that require a simple frequency estimate take between 1 and 1.4 sec. Simple attitude questions take between 1.4 and 2 sec., whereas more complex attitude questions take between 2 and 2.6 sec” (Bassili & Fletcher, 1991, Public Opinion Quarterly, 55, 339; see also Johnson, 2004, Political Psychology, 25, 679-702).



Authors

Sample

Sample Item

Response Scale

Format

Number of Items

Mean Time (minutes)

SD Time (minutes)

Average Time Per Item (sec)

Source

Naemi, Kyllonen, & Roberts (2009)

122 ETS Customer Service Representatives

I work hard

6-point: Very true of me to Very Untrue of Me

Web, proctored

127

12.86

3.41

6.10

Unpublished Study

Jackson, Paunonen, & Tremblay

1067 community dwelling volunteers

Personality measure

Likert scale (Not specified)

Paper and pencil, proctored

108

20 (max. time)

--

11.11

Technical Manual for Six Factor Personality Questionnaire

Montag & Reuter (2008)

47 university students

Affect measures (sample items not given)

4-point: Strongly Agree to Strongly Disagree

Paper and pencil, proctored

110

12.94

2.76

7.05

Cyber-psychology and Behavior, 11, 719-721.

Couper, Traugott, & Lamias (2001)

327 university students

For each of these categories, do you approve or disapprove of this aspect of the admissions policy? Applicants from economically disadvantaged backgrounds receive additional points:

6-point: Strongly Approve to Strongly Disapprove

Web, un-proctored

11

2.14

-

11.65

Public Opinion Quarterly, 65, 230-253.

Montag & Reuter (2008)

575 university students

Affect measures (sample items not given)

4-point: Strongly Agree to Strongly Disagree

Web, un-proctored

110

22.48

42.69

12.27

Cyber-psychology and Behavior, 11, 719-721.

Tourangeau, Couper, & Conrad (2004)

2,568 community dwelling respondents

I have to be very ill before I go to the doctor

5-point: Strongly Agree to Strongly Disagree

Web, un-proctored

1



10.8

Public Opinion Quarterly, 68, 368-393.


Questions for NAEP: General pass-back questions

1.       For student questionnaires, prepare a chart showing burden for each of the questionnaire components as in the example table below.


Example:

Questionnaire              Grade  Wave   Subject               Background (burden hours)                 Assessment (burden hours)

                                                                                    Core     Subject Specific    Total                                                           

Civics 4,8,12 1 5 10 15 65 mins. (inc bq and cognitive)

Geography 4,8,12 1 5 10 15 65 mins. (inc bq and cognitive)

History 4,8,12 1 5 10 15 65 mins. (inc bq and cognitive)

Writing 4 1 paper/pencil 5 10 15 65 mins. (inc bq and cognitive)

Writing 8,12 1 computer-deliv. 5 10 15 75 mins. (inc bq and cognitive)

Math 4, 8 1 5 10 15 65 mins. (inc bq and cognitive)


* As part of the Wave 1 supporting statement there is a burden chart with time per respondent for each survey (page 6) and another chart (page 7) that shows the total burden hours for all respondents by grade and by subject.


2.       Can a student be selected to complete more than one questionnaire per year? 

No, a student will only be selected for one subject.


  1. What is the maximum annual burden possible per student? 

15 minutes for background questions

50-60 minutes for cognitive sections

---------------------------------------------

65-75 minutes total

File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File TitleQuestions for NAEP:
Authorjoconnell
File Modified0000-00-00
File Created2021-02-04

© 2024 OMB.report | Privacy Policy