Carried over - MGLS 2017 IV Field Test 2016 Questionnaire Edits Change Memo

This document is being carried over from November 2015 clearance

MEMORANDUM OMB # 1850-0911 v.8

DATE: November 20, 2015

TO: Robert Sivinski, Office of Information and Regulatory Affairs, Office of Management and Budget

FROM: Carolyn Fidelman, National Center for Education Statistics

THROUGH: Kashka Kubzdela, National Center for Education Statistics

SUBJECT: Middle Grades Longitudinal Study of 2017–18 (MGLS:2017) 2016 Item Validation Field Test (IVFT) Questionnaire Edits Change Request

The Middle Grades Longitudinal Study of 2017-18 (MGLS:2017) is the first study conducted by the National Center for Education Statistics (NCES), within the Institute of Education Sciences (IES) of the U.S. Department of Education, to follow a nationally representative sample of students as they enter and move through the middle grades (grades 6-8). An Item Validation Field Test (IVFT) will be conducted beginning in January 2016 to determine the psychometric properties of items and the predictive potential of assessment and survey items so that valid, reliable, and useful assessment and survey instruments can be composed for the main study. The IVFT was approved in November 2015 (OMB# 1850-0911 v.4) and OFT recruitment in December 2015 (OMB# 1850-0911 v.4).

This submission requests approval (1) to modify language in the student questionnaire and the facilities checklist to improve item clarity; and (2) to include a flyer with parent permission materials. A separate change request will be submitted in January 2016 requesting approval to similarly modify language in the parent, administrator, math teacher, and special education questionnaires. Two separate submissions for edits to the questionnaires are necessary because the student questionnaire and facilities checklist must be finalized before the start of the student data collection in January, while data collection for the other questionnaires will commence in February 2016.

Modify Student Questionnaire Language

The purpose of the IVFT is to test the instruments on at least 1,200 students in each of grades 6 through 8, at least 350 students in grade 5, and at least 200 students in each of three disability groups: specific learning disability, autism, and emotional disturbance. Students will complete an assessment and questionnaire on Chromebook computers. The approved questionnaires were programmed for the electronic instrument, at which time it was determined that minor wording changes were needed to improve clarity of the items. At this time, changes have been finalized and are included here for the student questionnaire and the facilities checklist. These changes do not affect the approved MGLS:2017 IVFT respondent burden, nor the cost to the federal government.

In the student questionnaire, the demographics items were moved to the beginning of the questionnaire for all students, regardless of booklet assignment in the spiral design, to ensure that these important items are collected for all. Some wording changes were required to parse out this component from the rest of the student questionnaire. Other changes were made to add clarity to the item.

In the facilities checklist, three types of changes were made: (1) revised introductory text, (2) added an item erroneously left out of the original OMB submission, and (3) added a response option.

Exhibit 1 provides a table documenting and justifying each change to the two instruments. Part C and Appendices U and V have been updated to reflect these changes.

Flyer

A flyer, designed to explain why MGLS:2017 is different from other state or national assessments that students are required to take, has been included in the materials sent to parents. This flyer was developed in response to growing concerns about assessment in schools and concerns expressed by schools that large numbers of parents have been opting out of mandatory assessments. It is important to the success of the study for parents to understand how the study is different from other assessments that students are asked to take in the hope that they will permit their child to participate in MGLS:2017. The flyer has been added as the Appendix A.I.a.

Exhibit 1. Edits to the Student Questionnaire and Facilities Checklist

Instrument	Item #		Original Item	New/Updated Item	Justification
Student (all)	(All “Other specify” items)		“Please specify”: For “other” questions, respondents are asked to “please specify” what they were referring to when selecting a response option	“Please type in your answer”	There is concern that with “please specify,” respondents of this age group may not know the meaning of “specify”. Changing it to “Please type in your answer” would be more appropriate.
Student (all)	Skipping Questions Information		We hope you will respond to each item, but if you do not wish to answer then you can skip it. In order to skip an item, simply press the "Next" button on that screen. If you do press the "Next" button without answering the question, a message will pop-up to make sure you meant to skip that question. If so, just click the "Continue" button next to that message and move on, but if not, please provide your response to the question you missed and then click the“Next” button.	We hope you will respond to each question, but if you do not wish to answer a question then you can skip it. In order to skip a question, simply press the "Next" button on that screen. If you press the "Next" button without answering the question, a message will pop up to make sure you meant to skip that question. If you did, just click the "Next" button again to move on. But if you did not mean to skip it, please provide your response and then click the“Next” button.	We would like to update the skipping questions instructions to eliminate any references to a “Continue” button since this is no longer being included in the programmed instrument.
Student Introduction (all)	How to complete the survey		The "Save and Come Back Later" button can be used to save your responses and finish later (button is at the top right of each page).	The "Save and Come Back Later" button can be used to save your responses and finish later (button is at the bottom of each page)	The text was updated to reflect the new placement of the "Save and Come Back Later" button.
Student Demographics	Survey Information		We want to learn about some of the things you may think or do. This survey asks about what your school, friends, and home life are like, activities you do, and how you think about yourself and school.	We want to learn about you and some of the things you may think or do. This survey is voluntary and you can skip questions you do not want to answer.	The demographic questions are moving to the beginning of the student instrument. We modified the Survey Information page text prior to the demographics questions to eliminate references to topics that appear only in the student questionnaire, which comes later in the instrument.
Student Demographics	Introductory text		Now we have a few questions about your background.	N/A	We would like to remove the “Now we have a few questions about your background.” and used it as section introduction text on the first page of demographic questions.
Student Demographics	AA01		N/A	We have a few questions about your background.	Add this section introduction text (which previously appeared on the prior page) to help orient the respondents to what the ensuing questions will cover.
Student Demographics	AA01 Hard Check	HARD CHECK: IF MONTH = 1, 3, 5, 7, 8, 10, OR 12, DAY CAN BE 1-31. IF MONTH = 2, DAY CAN BE 1-28 [EXCEPT FOR IF YEAR= 2000, 2004 OR 2008, DAY CAN BE 1-29]. IF MONTH = 4, 6, 9, 11, DAY CAN BE 1-30. IF AA01= OUT OF RANGE: Please review the birth date you selected to make sure that it is correct.		PROGRAMMER BOX: DAY RANGE DISPLAY IF MONTH= 1, 3, 5, 7, 8, 10, OR 12, THE DAY DROP DOWN MENU CAN DISPLAY 1-31 IF MONTH= 2, THE DAY DROP DOWN MENU CAN DISPLAY 1-29 IF MONTH= 4, 6, 9, 11, THE DAY DROP DOWN MENU CAN DISPLAY 1-30 HARD CHECK: IF MONTH = 1, 3, 5, 7, 8, 10, OR 12, DAY CAN BE 1-31. IF MONTH = 2, DAY CAN BE 1-29. IF MONTH = 4, 6, 9, 11, DAY CAN BE 1-30. IF MONTH = 2 & DAY = 29 & YEAR ≠ 2000, 2004 OR 2008, OR IF MONTH = 2 & DAY = 30 OR 31, AA01= OUT OF RANGE. IF MONTH = 4, 6, 9, 11 & DAY =31, AA01= OUT OF RANGE. IF AA01= OUT OF RANGE: Please review the birth date you selected to make sure that it is correct.	Add a programmer box to indicate which day ranges should appear in the DAY drop down menu after the respondent selects their birth MONTH. Indicate that the value of “29” can appear to respondents automatically instead of appearing only after the respondent selects a YEAR option that coincides with a leap year. This allows the respondent to work sequentially across the screen. Modify the hard check programmer box to further clarify which scenarios would be out of range and would therefore trigger a hard check message.
Student Demographics	AA01	YEAR range (2000-2009)		YEAR range (1998-2009)	Widen the YEAR range to help the question be more inclusive and reflective of the possible age ranges of the study sample.
Student Demographics	AA04	Mexican, Mexican-American, or Chicano		Mexican, Mexican-American, or Chicano/Chicana	Add “/Chicana” to the response option to be consistent with question AA03 where respondents are asked whether they are “Latino/Latina”.
Student Demographics	AA06	Other (Please specify)		Other Asian (Please type in your answer)	Modify the response option to include “Asian” to be consistent with AA04 where the response option is “Other Hispanic or Latino/Latina”. Change “please specify” to “please type in your answer” as we think this will be clearer to respondents in this age group)
Student Booklet1/ Booklet 3	AA07_B1 AA07_B3	This class is ungraded or uses alternative grading		This class is ungraded or does not use letters for grades	Revise this response category label to use language more appropriate for this age group.
Student Booklet1/ Booklet 3	AA07_B1 AA07_B3	N/A		New response category: “Did not take this class in the last grading period”.	Given the possibility that some students may not have taken a class asked about in this item series, these students may leave the question blank. Add this new response category to help reduce the number of missing responses.
Student Booklet1/ Booklet 2	E06_B1c E06_B2c	No matter who you are, you can significantly change your intelligence.		No matter who you are, you can change your intelligence.	We removed “significantly” since this may not be appropriate for respondents in this age group and it does not change the meaning of the question.
Student Booklet1/ Booklet 2	E06_B1e/ E06_B2e	You can always substantially change your intelligence.		You can always change your intelligence.	We removed “substantially” since this may not be appropriate for respondents in this age group and it does not change the meaning of the question.
Student Booklet 2	A01a	I was tardy for school.		I was late for school.	We revised “tardy” to “late” to include vocabulary that is more appropriate for respondents in this age group.
Student Booklet 2	C01d	My classmates like me as much as they like others.		My classmates like me as much as they like other classmates.	We revised “others” to “other classmates” to clarify the question text.
Student Booklet 2	C02	Of the people your age who you hang out with, how important is it to them that they...		How important is it to the people your age who you hang out with that they...	We revised the question stem to reduce unnecessary wording and improve readability.
Student Booklet 2	D02	SOFT CHECK: IF 5:00AM-4:00PM; Please check that this is the time when you usually go to bed on school nights to make sure that it is correct.		SOFT CHECK: IF 2:00AM-7:00PM; Please check that this is the time when you usually go to bed on school nights to make sure that it is correct.	We widened the time range that would trigger the soft check message to further help identify potential response errors on behalf of the respondent.
Student Booklet 2	D02/D03	Question ordering: time go to sleep, time wake up.		Question ordering: time wake up, time go to sleep	We have reversed the order of these two questions so that respondents will now first indicate what time they wake up and then (in the next question) what time they go to sleep.
Student Booklet 2	E07	When I become confused about something I’m learning at school, I go back and try to figure it out.		When I become confused about something I’m learning at school, I try to figure it out.	We removed “go back and” to reduce unnecessary wording while not changing the meaning of the question.
Booklet 2	E08	Answer choice: Very Often		Answer choice: Very often	We revised the display of response category label to be consistent with the appearance of other labels.
Student Booklet 2 Student Booklet 3	E09/G03 E03/E04/ F06	Answer choice: Very True		Answer choice: Very true	We revised the display of response category label to be consistent with the appearance of other labels.
Student Booklet 2	E10a/ E10b	N/A		Adding the wording “American Society” to E10a and “School” to E10b	Because the items are similar in their appearance, include this wording to help respondents distinguish that after completing E10a, they are advancing to a new question (E10b), instead of potentially thinking they were remaining on the same survey page.
Student Booklet 2	E12	How would you describe your gender?		Deleted E12	Request approval to remove this item due to its sensitivity for respondents of this age group.
Student Booklet 2	G04	SOFT CHECK: IF G04=6 AND ANOTHER RESPONSE; You selected that you have not talked to anyone about what math courses to take next year and another response option. Please select either “I have not talked to anyone about this” or the other response, but not both.		PROGRAMMER BOX B04 PROGRAMMERS: IF G04=6 THEN GREY OUT ALL OTHER RESPONSE OPTIONS IF RESPONSE G06≠6 THEN GREY OUT OPTION 6	The need for this soft check message would be eliminated by programming the instrument in such a way that would prevent this scenario that would require such a message.
Student Booklet 2	G05	SOFT CHECK: IF G05=6 AND ANOTHER RESPONSE; You selected that you have not talked to anyone about going to college and another response option. Please select either “I have not talked to anyone about this” or the other response, but not both.		PROGRAMMER BOX B04 PROGRAMMERS: IF G05=6 THEN GREY OUT ALL OTHER RESPOSNE OPTIONS IF RESPONSE G05≠6 THEN GREY OUT OPTION 6	The need for this soft check message would be eliminated by programming the instrument in such a way that would prevent this scenario that would require such a message.
Student Booklet 2	G06	SOFT CHECK: IF G06=6 AND ANOTHER RESPONSE; You selected that you have not talked to anyone about possible jobs or careers when you are an adult and another response option. Please select either “I have not talked to anyone about this” or the other response, but not both.		PROGRAMMER BOX B04 PROGRAMMERS: IF G06=6 THEN GREY OUT ALL OTHER RESPONSE OPTIONS IF RESPONSE G06≠6 THEN GREY OUT OPTION 6	The need for this soft check message would be eliminated by programming the instrument in such a way that would prevent this scenario that would require such a message.
Student Booklet 3	A04	Think about all the teachers at your school, even if you don’t have a class with that teacher		Think about all the teachers at your school, even if you don’t have classes with all of them	Revision would help clarify the intent of the questions asked.
Student Booklet 3	B01	a. School sports b. Math or science clubs c. Other school clubs d. Performing arts e. Student government f. School yearbook, newspaper, or literary magazine		a. School sports b. Math or science clubs c. Performing arts d. Student government e. Other school clubs f. School yearbook, newspaper, or literary magazine	We moved the broader category of “other school clubs” down the list (e.g. above “other school sponsored activities) to follow the more specific categories. This helps address the potential concern is that a respondent, for example, in the performing arts club could say yes to “other school clubs” simply because they haven’t yet made it down to the performing arts question. We removed “chess” as an example of a “math or science club” since this does not fit within the types of clubs the questions seeks to understand. Revised help text to correspond to how the instrument is currently programmed. The respondents will no longer be taken to a new screen to record their response. This soft check was redundant with another soft check that was already in place.
Student Booklet 3	B01	Help text: Math or science clubs include, for example, math, robotics, or science bowl, or chess.		Help text: Math or science clubs include, for example, math, robotics, or science bowl	Remove “chess” as an example of a “math or science club” since this does not fit within the types of clubs the questions seeks to understand.
Student Booklet 3	B01	Help text: On the next screen, you will be asked to type in what those activities are.		Help text: Please then type in what those activities are	Revise help text to correspond to how the instrument is currently programmed. The respondents would no longer be taken to a new screen to record their response.
Student Booklet 3	B05	on demand		On Demand	We revised to reflect that this is a named source for watching television.
Student Booklet 3	B06	How many times per day do you post photos or updates on social media sites (such as Twitter, Instagram, Facebook, or Ask.fm)?		How many times per day do you post photos or updates on social media sites (such as Twitter, Instagram, Facebook, Snapchat, or Ask.fm)?	Add “Snapchat” as an example that can further help respondents understand what this question is asking.
Student Booklet 3	B06	SOFT CHECK: B06_wk > 99 NUMBER OF POSTS/UPDATES MIGHT BE TOO HIGH You entered [B06_wk] as the number of times per day on a typical weekday that you post photos or updates on social media sites. Is that correct?		Deleted soft check	The need for this soft check message would be eliminated by programming the instrument in such a way that would prevent this scenario that would require such a message.
Student Booklet 3	B06	SOFT CHECK: B06_wknd > 99 NUMBER OF POSTS/UPDATES MIGHT BE TOO HIGH You entered [B06_wknd] as the number of times per day on a typical weekend day that you post photos or updates on social media sites. Is that correct?		Deleted soft check	The need for this soft check message would be eliminated by programming the instrument in such a way that would prevent this scenario that would require such a message.
Student Booklet 3	B07	SOFT CHECK: B07_wk > 99 NUMBER OF EMAILS MIGHT BE TOO HIGH You entered [B07_wk] as the number of emails per day on a typical weekday that you send to friends, family members, teachers or others. Is that correct?		Deleted soft check	The need for this soft check message would be eliminated by programming the instrument in such a way that would prevent this scenario that would require such a message.
Student Booklet 3	B07	SOFT CHECK: B07_wknd > 99 NUMBER OF EMAILS MIGHT BE TOO HIGH You entered [B07_wknd] as the number of emails per day on a typical weekend day that you send to friends, family members, teachers or others. Is that correct?		Deleted soft check	Eliminate this soft check message to help respondents navigate through the instrument more readily.
Student Booklet 3	B07	PROGRAMMER DISPLAY BOX B07 ONLY ALLOW 0-99 NUMBER ENTRIES		PROGRAMMER DISPLAY BOX B07 ONLY ALLOW NUMBER ENTRIES	We have eliminated the soft check for this question to be more reflective of respondents’ experiences.
Student Booklet 3	B09	If you do not see yours listed, select the "Other talents, interests, or hobbies" option and type in that talent, interest, or hobby on the next screen. Physical activities (biking, running, martial arts, skateboarding, ice skating etc.)		If you do not see yours listed, select the "Other talents, interests, or hobbies" option and type in that talent, interest, or hobby. Physical activities (biking, running, martial arts, skateboarding, ice skating, etc.)	Remove “on the next screen” to reflect how the instrument is programmed. The respondent would not be taken to a different screen to record their answer. Added a “,” near the end of the response option to be more grammatically correct.
Student Booklet 3	B09	SOFT CHECK: IF B09=16 AND ANOTHER RESPONSE; You selected that you do not have any talents, interests, or hobbies that you are excited about and another response option. Please select either “I don’t have any talents, interests, or hobbies that I am excited about” or the other response, but not both.		PROGRAMMER BOX B04 PROGRAMMERS: IF B09=16 THEN GREY OUT ALL OTHER RESPONSE OPTIONS IF RESPONSE BESIDES B09=16 THEN GREY OUT OPTION 16	The need for this soft check message would be eliminated by programming the instrument in such a way that would prevent this scenario that would require such a message.
Student Booklet 3	B10	Booklet=3 & B09=1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,99		BOOKLET=3 & B09=1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,99 AND TWO OR MORE OPTIONS AT B09 ARE SELECTED	Update the skip logic so that this question will only appear for respondents that selected two or more response options in the preceding question.
Student Booklet 3	B14/B15	Thinking about your pet, or your favorite pet if you have several.		Think about your pet, or your favorite pet if you have several.	We revised the question instructions to improve readability.
Student Booklet 3	F02j	Do something else when I get good grades Help text: If there is something else your parents or guardians do when you get a good grade, please select how often they do it and click “Next”. On the next screen, you will be asked to type in what that other thing is that they do when you get a good grade. Do something else when I get bad grades Help text: If there is something else your parents or guardians do when you get a bad grade, please select how often they do it and click “Next”. On the next screen, you will be asked to type in what that other thing is that they do when you get a bad grade.		Do something else when I get good grades. Help text: If there is something else your parents or guardians do when you get a good grade, please select how often and then type in what that other thing is that they do when you get a good grade. Do something else when I get bad grades. Help text: If there is something else your parents or guardians do when you get a bad grade, please select how often they do it and then type in what that other thing is that they do when you get a bad grade.	Added a “.” at the end of question to be more grammatically correct. We revised the help text to reflect how the instrument is programmed. The respondent will not be taken to a different screen to record their answer. Added a “.” at the end of the question to be more grammatically correct. We revised the help text to reflect how the instrument is programmed. The respondent will not be taken to a different screen to record their answer.
Facilities Checklist	B1	The Section Intro Text "The following questions are about the condition of the immediate neighborhood/area around the school."		"The next questions are about the general upkeep of the entranceway and hallways."	There was an error in the OMB version. The change will provide the correct text for this item.
Facilities Checklist	B01 ordering	“Graffiti on the lockers” [at bottom of list]		“Graffiti on the lockers” [immediately following “Graffiti on the walls, doors, or ceilings”]	Re-ordering will facilitate the review process for Test Administrators who will be completing this checklist
Facilities Checklist	D1	"in which grade 6-8 students are taught"		"in which students in grades 6 through 8 are taught"	The wording has been revised to be less awkward.
Facilities Checklist	D1h	The next questions are about the general upkeep of classrooms. D.1 During a change in classes or other time when classes are not in session, enter one classroom in which grade 6-8 students are taught. For each item listed, indicate whether you observed it in the classroom. The item currently lists 10 rows.		Add one additional row: "Floors and walls appear clean"	The OMB version contained an error. It was missing a row from the grid. We would like to add the 1 item that was missing from the previously-approved OMB submission.
Facilities Checklist	F2	F.2The next questions are about signs that may be posted at or near the entrance of the school. Response categories of: “Observed Inside, “ “Observed Outside,” and “Not Observed.”		Keep this as a select one answer for each row but add new response category for “Observed both inside and outside, so that the four response categories would be: “Observed inside” “Observed outside” “Observed both inside and outside,” “Not observed”	The item currently contains four rows with response categories of “Observed inside,” “Observed outside,” and “Not observed.” Given the fact that these are “select one answer” items, they do not properly account for the possibility of signs posted both inside and outside. Thus, we would like to request approval to add one additional response category of “Observed both inside and outside.”
Facilities Checklist	F5e	How many parking spaces are designated for people with disabilities?		How many parking spaces are designated for people with disabilities? These parking spaces are typically marked by a sign or blue paint lines. Please count only the spaces in the parking lot nearest to the main school entrance.	Revised wording provides greater clarity for field staff completing the checklist.
Facilities Checklist	G1	is the resource room in an accessible location to all students?		is the resource room in a location accessible to all students?	This is a grammatical correction.

2