NAEP 2017 Change Memo

MEMORANDUM OMB # 1850-0928 v.2

DATE: August 2016

TO: Robert Sivinski, Office of Information and Regulatory Affairs, Office of Management and Budget

FROM: Patricia Etienne, National Center for Education Statistics

THROUGH: Kashka Kubzdela, National Center for Education Statistics

SUBJECT: National Assessment of Educational Progress (NAEP) 2017 Final Materials Change Request

The National Assessment of Educational Progress (NAEP) 2017-2019 data collections were recently approved in August 2016.

This request is to update the 2017-2019 NAEP clearance with the updated materials for 2017. Specifically, the final survey questions have been selected for 2017. The questions have been selected from those that appear in the “library” presented in Appendix F. In addition, some minor wording changes have been made to many of the items based on pilot and cognitive interview results and final reviews. All of the changes are non-substantive in nature. The table presented on the following pages indicates the items that have changed wording from the original version presented in Appendix F, as well as the rationale for the change. The final questionnaires to be administered are presented as a new package document, Appendix I. Note that some of the questionnaires will utilize spiraling in order to minimize respondent burden while maximizing the amount of items that can be piloted. The spiral maps for these questionnaires are also included in Appendix I.

In addition, the final recruitment materials have been updated for 2017 to reflect the specific assessments to be administered. These materials (similar to those presented in Appendix D) are provided as a new appendix, Appendix J. Notes that two of the materials (Appendices J-15 and J-16) are not yet finalized. The final versions for 2017 will be submitted as a subsequent change request.

This amendment does not change the estimated respondent burden or the cost to the federal government.

#	Resp*	Approved in OMB #1850-0928 v. 1	Amended August 2016	Rationale for Change
CORE OPERATIONAL AND PILOT
1	St			Given that approximately 25% of write-in responses for other piloted technology items stated “Chromebooks”, “(including Chromebooks)” was added to clarify the original sub-item f (currently sub-item e).
2	St			Given that approximately 25% of write-in responses for other piloted technology items stated “Chromebooks”, “(including Chromebooks)” was added to clarify this item.
3	T			Given that approximately 25% of write-in responses for other piloted technology items stated “Chromebooks”, “(including Chromebooks)” was added to clarify this item. Additionally, the response options were reordered based on comments from an editorial review.
4	T			The response options were reordered based on comments from an editorial review.
5	T			Given that approximately 25% of write-in responses for this item stated “Chromebooks”, “(including Chromebooks)” was added to clarify this item.
6	T			Because very few pilot participants used option e (i.e., Most of the computers…), this option was collapsed into option d.
7	T			Given that approximately 25% of write-in responses for this item stated “Chromebooks”, “(including Chromebooks)” was added to clarify this item. Further, because very few pilot participants used option e (i.e., Most of the computers…), this option was collapsed into option d.
8	T			Because very few pilot participants used option e (i.e., Most of the computers…), this option was collapsed into option d.
9	S			Given that approximately 25% of write-in responses for this item stated “Chromebooks”, “(including Chromebooks)” was added to clarify this item.
10	S			Because very few pilot participants used option a (i.e., Less than 1 year), this option was collapsed into option b. Additionally, the remaining response options were revised for clarity based on expert panel review.
11	S			Given that approximately 25% of write-in responses for other piloted technology items stated “Chromebooks”, “(including Chromebooks)” was added to clarify this item.
12	S			Given that approximately 25% of write-in responses for other piloted technology items stated “Chromebooks”, “(including Chromebooks)” was added to clarify this item. Additionally, because very few pilot participants used option a (i.e., Less than 1 year), this option was collapsed into option b. The remaining response options were revised for clarity based on expert panel review.
13	S			Given that approximately 25% of write-in responses for other piloted technology items stated “Chromebooks”, “(including Chromebooks)” was added to clarify this item. Additionally, the response option “On mobile carts” (new sub-item f) was added due to participants regularly referring to them in write-in responses.
14	S			Because very few pilot participants used option a (i.e., Less than 1 year), this option was collapsed into option b. Additionally, the remaining response options were revised for clarity based on expert panel review.
15	S			The response option “On mobile carts” (new sub-item f) was added due to participants regularly referring to them in write-in responses.
16	S			The word “Please” was added to response option f to improve consistency across survey questionnaire items.
WRITING OPERATIONAL
17	St			To ensure consistency across questionnaires, the phrase "desktop or laptop computer" in original sub-item e was revised to "computer or other digital device".
18	St			The word "spell-check" was revised to "spell-checker", per Webster's dictionary: "Use a spell-checker in word processing software".
19	St			To further clarify original sub-item b the phrase "including Chromebooks" was added. Further, given that approximately 75% of the write-in responses for sub-item d stated "phone" or some form of mobile phone, we updated this sub-item to: "Smartphone (for example, iPhone, Samsung Galaxy, HTC One)".
20	St			To further clarify original sub-item b the phrase "including Chromebooks" was added. Further, given that approximately 85% of the write-in responses for sub-item d stated "phone" or some form of mobile phone, we updated this sub-item to: "Smartphone (for example, iPhone, Samsung Galaxy, HTC One)".
21	St			To further clarify original the sub-item d "Write journal entries" was revised to "Write journal entries or blog posts".
22	St			To further clarify original sub-item b "someone about" was revised to "someone of".
23	St			To ensure references to technology are up-to-date, "using the Internet" was revised to "computer or other digital device". Further, the examples were placed in parentheses and "e-mails, blogs" were revised to "writing e-mails, blog posts".
24	T			For consistency across all teacher and school survey questionnaires, "for example" was revised to "e.g.," in original sub-item c.
25	T			Disaggregated original sub-item d into two sub-items: “Digitally based writing tests” and “Extended constructed-response questions”. Revised wording "digitally based" instead of "digital-based" per the updated NAEP style guide.
26	T			The word "spell-check" was revised to "spell-checker", per Webster's dictionary: "Using a spell-checker in word processing software".
27	T			To further clarify original sub-item a the phrase "including Chromebooks" was added. For consistency across all teacher and school survey questionnaires, "for example" was revised to "e.g.," in original sub-item b.
28	T			The phrase "8th grade" was revised to "eighth-grade" for consistency with trend wording and NAEP style guidelines.
29	T			The phrase "8th grade" was revised to "eighth-grade" for consistency with trend wording and NAEP style guidelines. Further, the word "spell-check" was revised to "spell-checker", per Webster's dictionary: "Spell-checker".
30	T			The phrase "8th grade" was revised to "eighth-grade" for consistency with trend wording and NAEP style guidelines.
31	S			As a result of dropping sub-items b and c in the original item VH312559 (leaving only sub-item a), we replaced this matrix item with a discrete multiple selection multiple choice item type.
32	S			As a result of dropping sub-items b and c in the original item VH312570 (leaving only sub-item a), we replaced this matrix item with a discrete multiple selection multiple choice item type.
33	S			For consistency across all teacher and school survey questionnaires, "for example" was revised to "e.g.,".
34	S			The phrase "8th grade" was revised to "eighth-grade" for consistency with trend wording and NAEP style guidelines.
35	S			For consistency with other Writing items, the word "writing" was bolded. Further, to account for the transition to DBA the sentence "Fill in one oval on each line" was revised to "Select one circle in each row."
36	S			The phrase "8th grade" was revised to "eighth-grade" for consistency with trend wording and NAEP style guidelines. Further, for consistency across all teacher and school survey questionnaires, "for example" was revised to "e.g.,".
37	S			The phrase "8th grade" was revised to "eighth-grade" for consistency with trend wording and NAEP style guidelines.
READING OPERATIONAL
38	St			For consistency with other options, option d was revised from "Four to five times" to "Four or five times".
39	St			To reduce reading burden response options b through f were revised from "30 minutes up to 1 hour, 1 up to 2 hours, etc." to "About 30 minutes, About 1 hour, etc.".
40	St			For clarity original sub-item l was revised from "in a text" to "from a text".
41	St			To reduce reading burden response options b through f were revised from "30 minutes up to 1 hour a day, 1 up to 2 hours a day, etc." to "About 30 minutes a day, About 1 hour a day, etc.".
42	St			For clarity and to be more inclusive, "Autobiographies" was revised to "Biographies".
43	T			For consistency with sub-item a, sub-items b and d were revised from "Increase" to "Build".
44	T			For consistency with sub-item a, sub-items b and d were revised from "Increase" to "Build".
45	S			To ensure consistency with the grade 8 questionnaire and the language used throughout the grade 4 questionnaire "reading content or the teaching of reading" was revised to "English/language arts content or the teaching of English/language arts".
46	S			Disaggregated original sub-item c into two sub-items: “Results from district assessments” and “Results from state assessments”. For consistency with other Reading items "reading/language" was revised to "English/language". For clarity original sub-item i was revised to "Resources found on the Internet". Sub-items c and h were adopted from the Social Studies school questionnaire.
47	S			For clarity and consistency with the Social Studies school questionnaire the example "quizzes or tests created by teachers" was added to original sub-item c.
48	S			For clarity and consistency with the Social Studies school questionnaire the example "quizzes or tests created by teachers" was added to original sub-item c.
READING PILOT
49	St			To reduce reading burden response options b through f were revised from "30 minutes up to 1 hour, 1 up to 2 hours, etc." to "About 30 minutes, About 1 hour, etc.".
50	T			For clarity "individualized instruction" was revised to "differentiated instruction" and a brief definition was added (sub-item f); "assigned plan or goal" was revised to "assignment or task" (sub-item g); and "goal" was revised to "task" (sub-item h). For consistency with other teacher and school questionnaires "Other (specify)" was revised to "Other strategies (Please specify)".
51	T			For consistency with sub-item a, sub-items b and d were revised from "Increase" to "Build". Further, for clarity original sub-item e was revised to "Conduct research for projects".
52	S			For clarity sub-item a was revised from "on various reading topics" to "with various reading strategies" and examples were added.
53	S			"ELL" was revised to be plural (i.e., ELLs) to be consistent with the phrase "English-language learners".
MATHEMATICS OPERATIONAL
54	St			"Hand-held" was revised to "handheld" as per Webster dictionary.
55	St			For clarity the parenthetical "excluding handheld calculators" was added.
56	St			To ensure consistency across questionnaires, the phrase "desktop or laptop computers" in original sub-item c was revised to "computers or other digital devices".
57	St			As per NAEP style, a period was added to "lbs" and "67 dollar" was revised " 67-dollar". For clarity "by" was revised to "among" in original sub-item b.
58	St			For tense consistency "things I learned" was revised to "things I learn".
59	St			"Hand-held" was revised to "handheld" as per Webster dictionary.
60	St			"Hand-held" was revised to "handheld" as per Webster dictionary.
61	St			"Hand-held" was revised to "handheld" as per Webster dictionary. Further, for consistency with other options, option d was revised from "Four to five times" to "Four or five times".
62	T			"Hand-held" was revised to "handheld" as per Webster dictionary.
63	T			Based on pilot data for the write-in response, "Materials you have created" was added.
64	T			For clarity original sub-item d was disaggregated into two sub-items: “Make assumptions” and “Make approximations”.
65	T			For clarity original sub-item d was disaggregated into two sub-items: “Make assumptions” and “Make approximations”. Further, in original sub-item c a comma was added between "ways" and "including".
66	S			To ensure consistency across all teacher and school questionnaires "teacher led" was revised to "teacher-led".
67	S			Disaggregated original sub-item c into two sub-items: “Results from district assessments” and “Results from state assessments”. For clarity original sub-item i was revised to "Resources found on the Internet". Sub-items c and h were adopted from the Social Studies school questionnaire.
68	S			For clarity and consistency with the Social Studies school questionnaire the example "quizzes or tests created by teachers" was added to original sub-item c.
69				For clarity and consistency with the Social Studies school questionnaire the example "quizzes or tests created by teachers" was added to original sub-item c.
70	S			For clarity "enrolls" was revised to "enroll" and "including summer school two-block classes" was revised to "including summer school or two-block classes".
71	S			For clarity the phrase "fourth-grade" was added.
72	S			For clarity "has gone" was revised to "have gone".
MATHEMATICS PILOT
73	St			Per style a comma was added after "smartphone".
74	St			Per style a comma was added after "smartphone".
75	St			For clarity "draw" was revised to "create" in original sub-item.
76	St			For clarity "or incorrect" was removed from original sub-item a.
77	St			For clarity "notation" was revised to "mathematical words and symbols" in original sub-items a and b.
78	T			For consistency a period was added to the response option "No, I do not prefer to use this resource". For clarity original sub-item i was revised to "Printed workbooks" and original sub-item j was revised to "Other materials not listed above (Please specify)".
79	T			For clarity original sub-item a was revised to "Use clear and precise language to discuss problem solving and reasoning" and sub-item c was revised to "Represent a problem in multiple ways including using numbers, words, pictures, and charts. Per style, "real life" was revised to "real-life".
80	T			For consistency across mathematics questionnaires the sub-items were revised so that the gerund ends in "-ing" (e.g., posing, using). For clarity "for student learning" was removed from original sub-item b.
81	T			For clarity "tools" was revised to "tables" in original sub-item d.
82	S			For clarity original sub-item i was revised to "Resources found on the Internet".
SOCIAL STUDIES PILOT
83	St			The stem wording was revised to reference "civics and/or United States government" in the item stem and the subject reference was removed from the sub-items.
84	St			The stem wording was revised to specifically reference "civics and/or United States government" the item stem and the subject reference was removed from the sub-items.
85	St			The sub-items were revised for consistency across questionnaires and all instances of "civics or United States government" were revised to "civics and/or United States government". Sub-items were also revised to improve the alignment between the wording and response options.
86	St			The stem and scale were revised to follow the wording that had been tested in cog labs with other civic engagement items. Sub-item f was revised to "Three years from now..." to avoid any potential problems with interpretation.
87	St			Based on cog labs where some students were confused by the phrase, "world affairs", the wording was changed to "events in other countries" in sub-item c. Sub-item d was revised for clarity. Sub-item e was revised to replace the term "public affairs". For sub-item f, "and/or" was revised to "and" to simplify the wording.
88	St			The term “and” was revised to “or” in the parenthetical examples in current sub-item c. For original sub-item d (current sub-item b) the phrase "social issues" was revised.
89	St			All instances of "civics and United States government" were revised to "civics and/or United States government." As per cog lab feedback, "I don't know" was revised to "I don't remember" for more accuracy.
90	St			The sub-items listing the 3 branches of government in isolation were combined into sub-item b. Sub-item e wording was revised to improve clarity. For previous sub-item i (current g), additional examples of international organizations were added. Previous sub-items j and k were revised and combined into current sub-item h for improved clarity.
91	St			Revisions were made to consistently list plural items in parentheses.
92	St			For sub-item c, the wording was revised to replace the phrase “public officials”. For sub-item e, the wording was revised to replace the phrase "public affairs". Based on feedback from cog labs, sub-item f was revised to add examples for what "others" refers to.
93	St			All instances of "civics or United States government" were revised to "civics and/or United States government". Sub-item b was revised to include “or” in the parenthetical examples. Sub-item e was revised to match the wording in the teacher sub-item.
94	St			All instances of "civics and United States government" were revised to "civics and/or United States government". The verb tense was aligned with the item stem for current sub-items b, d, and e. The word "position" was revised to "opinion" for improved clarity in sub-item c.
95	St			In the item stem, the wording of "civics or United States government" was revised to "...and/or..." and the stem was revised overall to improve alignment with the answer choices. Based on probing in cog labs for alternative response options, the options were revised. Sub-item c wording was revised to replace the phrase "world affairs" for improved clarity. Sub-item d was revised for improved clarity and grammar. For sub-item e, wording was revised to replace the phrase "public affairs". For sub-item f "and/or" was replaced with "and".
96	St			All instances of "civics or United States government" were revised to "civics and/or United States government". Sub-item b wording was revised to replace "multimedia" with "different forms of media". Sub-item d was revised to be consistent with the conventions used when listing parenthetical examples.
97	St			The item stem was revised to specifically reference geography. Two different versions of this item will be compared in pilot (one with the subject in the stem and one with the subject in the sub-items).
98	St			The stem wording was revised to specifically reference geography, while the subject reference was removed from sub-item c. Previous sub-item e (current d) was revised to replace the term “class” with “schoolwork.”
99	St			Sub-items e and f were revised to improve the alignment between sub-item wording and response options.
100	St			Sub-item b was revised to be consistent with the conventions used when listing parenthetical examples. For sub-item c, the wording was revised. Sub-item g was added to separate "affect and adjust to" into two sub-items, the other being sub-item f. Sub-item i was revised to provide simpler wording for "census data" and to align with a revision to the parallel classroom instruction sub-item.
101	St			Wording of the response option changed from "I don't know" to "I don’t remember" based on cog lab data.
102	St			The item stem and response options were revised based on cog lab data. Sub-items were revised to be consistent in format for parenthetical examples.
103	St			Sub-item c was revised to consistently list plural items in parentheses.
104	St			The stem was revised to be consistent with the stem for outside-of-school student engagement items found in civics and United States history. Sub-item a was revised to include another common map, and current sub-item f was revised for clarity of wording.
105	St			Sub-item b was revised to consistently list plural items in parentheses. Sub-item e was revised to match the teacher item to provide simpler and clearer wording.
106	St			The verb tense was aligned between sub-items b and d and the item stem. The word "position" was revised to "opinion" for improved clarity in sub-item c.
107	St			Sub-item b was revised to consistently list plural items in parentheses. For sub-item c, the wording was revised for improved clarity. In order to disaggregate the parallel teacher item, sub-items f and g were formed and the wording revised for improved clarity. Sub-item i was revised to provide simpler wording for "census data".
108	St			Sub-items were revised to consistently list plural items in parentheses. The response options were revised to those used in parallel 2016 Mathematics pilot items. Sub-items a and c were revised to provide simpler wording in place of "anthropologist" and "multimedia".
109	St			The stem wording was revised to specifically reference "United States history" in order to clarify the item and the subject reference was removed from the sub-items.
110	St			The stem wording was revised to specifically reference "United States history" in order to clarify the item and the subject reference was removed from the sub-items.
111	St			The wording on sub-item a was revised to be consistent with the wording of other sub-items. Sub-items e and f were revised to improve the alignment between sub-item wording and response options.
112	St			Based on cog lab responses, the wording of current sub-items b, c, and e was revised for improved clarity. Current sub-item d was revised for improved clarity. Current sub-item f was revised to match the teacher item for clearer language.
113	St			As per cog lab feedback, "I don't know" was revised to "I don't remember" for more accuracy.
114	St			The stem and response options were revised based on item-specific probing of respondents. All sub-items were revised to consistently list plural items in parentheses.
115	St			Sub-item c was revised for consistency in listing parenthetical examples. Sub-item e was revised to match the parallel teacher sub-item.
116	St			The item stem wording was revised to "hours a day" to match the response options.
117	St			Sub-item b was revised for consistency in listing parenthetical examples. Sub-items c and f were revised for consistency within the item and with the parallel teacher item, respectively.
118	St			Sub-items b, d, and e had revisions to the verb tense. Sub-item c was revised for replacing the term "position" with "opinion" for improved clarity.
119	St			Based on cog lab responses, response options were revised, and the wording for sub-items b, c, and d was revised for improved clarity. Sub-item f was revised for consistency with a parallel teacher sub-item.
120	St			The response options were revised to those used in 2016 Mathematics pilot items. Based on cog lab responses, wording in sub-items a and c was revised to improve clarity. Sub-item e was revised for consistency in listing parenthetical examples.
121	T			The item wording was revised to clarify instructions for what social studies class teachers should think about when providing a response.
122	T			The phrase "civics or United States government” was revised to "civics and/or United States government".
123	T			The phrase "civics or United States government” was revised to "civics and/or United States government". "Please" was capitalized in sub-item d for consistency with other subjects.
124	T			In order to maintain consistency with the abbreviation conventions used across all teacher and school questionnaires, "for example" was replaced with "e.g." in all instances.
125	T			To align with a parallel student sub-item, minor revisions were made to sub-item d. For sub-item e, the word "position" was revised to "opinion" for improved clarity and consistency with the revised wording of the parallel student item.
126	T			The phrase "civics or United States government” was revised to "civics and/or United States government".
127	T			For consistency with the abbreviation conventions used across all teacher and school questionnaires, "for example" was replaced with "e.g."
128	T			Based on cog lab responses, the response options were revised. For consistency with the abbreviation conventions used across all teacher and school questionnaires, "for example" was replaced with "e.g." in all instances. For current sub-items b and c, the words "anthropologist" and "multimedia" were revised, respectively, to improve clarity and to achieve consistency with the revised wording of the parallel student item.
129	T			The item stem wording was revised to "hours of homework a day" to match the response options.
130	T			Wording revisions were made to sub-item b to refer more generally to "sources" rather than "documents" and the word "or" was added to the terms in parentheses.
131	T			The phrase "civics or United States government” was revised to "civics and/or United States government". Based on cog lab responses, the response options were revised. Based on cog lab responses for a parallel student item, wording revisions were made to sub-items e, f, g, and h. The sub-items listing the 3 branches of government in isolation were combined into sub-item b.
132	T			Based on cog lab responses, the response options were revised. For current sub-items c, d, e, f, and h, the wording was revised to improve clarity and to achieve consistency with the revised wording of the parallel student items.
133	T			Based on cog lab responses, the item stem wording was revised to clearly refer to "civics and/or United States government." The response options were expanded to a multiple response format to account for non-mandated professional development.
134	T			The item stem and response options were revised to match a similar Civics curriculum content item. Sub-items b and c were revised to be consistent with the wording used in a parallel Geography student item. The wording for sub-items d and e was revised for improved clarity.
135	T			The stem and response options were revised to maintain consistency with a similar Civics instructional strategy item in teacher. "For example" was replaced with "e.g." for consistency with the abbreviation conventions used across all teacher and school questionnaires. Previous sub-item d (current c) and current sub-item i had revisions to wording for improved clarity. To achieve consistency with the revised wording of the parallel student item, current sub-items f and g were the result of disaggregation of the previous sub-item version.
136	T			Based on cog lab responses, the item stem wording was revised to clearly refer to "geographic skills." The response options were expanded to a multiple response format to account for non-mandated professional development.
137	T			The stem and response options were revised to match the parallel Civics item for teacher. All sub-items had minor revisions for punctuation, capitalization, and grammar.
138	T			The stem and response option wording was revised to match a similar Civics curriculum content item for teacher.
139	T			The stem and response option wording was revised to match a similar Civics curriculum content item for teacher. The wording for sub-items b, c, and d was revised for improved clarity. "For example" was replaced with "e.g." for consistency with the abbreviation conventions used across all teacher and school questionnaires.
140	T			Based on cog lab responses, the item stem wording was revised to clearly refer to "history". The response options were expanded to a multiple response format to account for non-mandated professional development.
141	S			The response option "5th grade" was added to achieve consistency with similar items found in the Civics, Geography, and U.S. History student questionnaires. For sub-item a and d, the phrase "civics or United States government” was revised to "civics and/or United States government". The word "or" was added before "United States history" in sub-item f.
142	S			The parenthetical examples in the item stem were reordered alphabetically and revised for more clarity. Sub-items e and j were revised for clarity and to eliminate overlap between these two sub-items. The wording of sub-items f and k was revised for improved clarity.
143	S			The parenthetical examples in the item stem were reordered and revised for more clarity. Parenthetical examples for sub-items d and e were added for more clarity.
144	S			All instances of "civics or United States government" were revised to "civics and/or United States government". The parenthetical examples in the item stem were reordered alphabetically and revised for more clarity. Minor grammatical revisions were made to sub-items g and h.
145	S			The inline directions were corrected to "Select one or more squares in each row" for consistency across all subjects. The "i.e." in the item stem was replaced with "e.g." for consistency in abbreviations across questionnaires. "Professional development" was added to sub-item c for more clarity.
146	S			The wording for sub-items b, c, and e was revised for improved clarity and grammar.