NTPS 2017-18 Change Request Memo

U NITED STATES DEPARTMENT OF EDUCATION

National Center for Education Statistics

MEMORANDUM

Date: June 29, 2017

To: Robert Sivinski, OMB

From: Amy Ho, NCES

Through: Kashka Kubzdela, NCES

Re: 2017–18 National Teacher and Principal Survey (NTPS 2017-18) Change Request (OMB# 1850-0598 v.19)

The 2017–18 National Teacher and Principal Survey (NTPS 2017-18) was approved in June 2017 (OMB# 1850-0598 v.18). This request updates the approved NTPS 2017-18 procedures and materials with: (a) details of the final sampling design, of the treatment analyses, and of the model to predict schools’ propensity to respond and assign them to treatment groups; (b) non-substantive revisions to questionnaire items to correct typos and errors found; (c) addition of the “letter of better understanding” to recruitment materials; and (d) updated screenshots of the NTPS portal to show their finalized versions. None of the changes described in this request affect the approved respondent burden or the cost to the federal government. This request updates the approved 1850-0598 v.18 documents as follows:

Part A

To reflect the revisions to the package materials submitted with this change request, the following revisions were made on page 3 of Part A:

The final sampling design, details of the responsive design model, and treatment analysis described in the Supporting Statement Part B will be completed in June 2017, at which time NCES will brief OMB and subsequently submit a change request with the final design and analysis details. This briefing and change request will alsoTo fulfill the last OMB terms of clearance that were provided to NCES as part of the NTPS 2015-16 approval in June 2015 (OMB# 1850-0598 v.11), and which stated that “OMB looks forward to having the results of NCES's adaptive survey design strategies included in its next submission.”, upon completion, a report of the 2015-16 NTPS adaptive design results will be submitted to OMB as a change request in late August 2017 in Appendix D – Adaptive Design Report.

Part B

1. Table 3 and Table 4 numbers in Part B were updated using the NTPS 2015-16 final response rates and CV of 25%.

2. We added information about the addition of the “letter of better understanding” to recruitment materials:

   1. At the end of the first paragraph on page 10 of Part B under section B.2.2.1 (Priority Schools):

If the Field Representative notes that the school has shown reluctance or initially refused to participate in the study, the Regional Office of the Field Representative will send out a “letter of better understanding” to help encourage participation.

2. As a new bullet point 9 on page 18 under section B.3.2 (Methods to Minimize Nonresponse):

9. Send a “letter of better understanding” to principals and teachers. After the 2015-16 NTPS collection, field representatives and the regional offices recommended that in NTPS 2017-18 “letters of better understanding” be sent to principals and teachers who may be hesitant to complete the survey to help them gain a better understanding of the survey by providing them information about how the data are used and referencing some of the published data from the SASS 2011-12 First Look Reports. These letters will be sent to principals and teachers in priority schools, which tend to exhibit high non-response.

3. We updated details of the model to predict schools’ propensity to respond and assign them to treatment groups (including the contributing variables and expected numbers of affected schools and teachers).

The schools will be assigned into one of eight experimental groups prior to the beginning of data collection. As such, the random assignment should result in similar TLF response rates across all groups by the beginning of Phase Two of the experiment. Approximately10,385 public schools and 4,000 private schools will be sampled for NTPS 2017-18. Because Phase One of the incentive experiment is independent of Phase Two of the data collection period, the random assignment of schools into the eight groups should result in similar TLF response rates across all groups at the start of Phase Two. Before the start of data collection, each experimental group will be assigned 1,298 public schools and 500 private schools. Therefore, approximately 5,192 public schools and 2,000 private schools will be assigned to the treatment groups that send teacher incentives during Phase One of the experiment.

Balancing treatment group assignments

To ensure a similar distribution of schools for each of the eight experimental groups, the sample will be sorted by an indicator for whether or not the school is covered on the vendor list, potentially an indicator for whether or not the teacher list was located during internet look-ups, and other selected school characteristics. The school characteristics will be selected for having significant influence on response based on past cycles of NTPS and on a model calculated to predict the likelihood of a school returning the TLF. For example, one of the school characteristics that may have a significant influence on response is the priority/non-priority school status (i.e., schools with higher potential to impact weighting and lower propensity to respond that are subject to a slightly different set of collection operations), which will be taken into account by using it as a sort variable in which the priority/non-priority status will be a stratification variable in the model.

A time-to-respond model will also be calculated to determine the likelihood of a case responding by any particular wave. This will ensure that schools that are more likely to return a TLF early in data collection do not randomly get selected into the same experimental group, which would make it difficult to separate the effect of incentives on response from the inherent likelihood of a school to return its TLF. The model to predict and assign schools to treatment groups will be completed in June 2017, at which time NCES will brief OMB on the details of the model (including the contributing variables and expected numbers of affected schools and teachers), and will subsequently submit a change request with the final details.To identify covariates that are predictive of whether and when a school will return the TLF for NTPS 2017-18, two response propensity models were analyzed, along with a time-to-event model, using NTPS 2015-16 data.

The NTPS 2015-16 frame data and response status were used to create the three models. The first response propensity model used logistic regression to predict the likelihood that a school would return a TLF by the end of data collection. This model was also used to calculate a TLF response propensity score for the NTPS 2017-18 schools, which will be used in the final sort order. The second model used logistic regression to predict the likelihood that a school would return the TLF early. In this model, the school’s return of the TLF was considered early if the school returned the TLF before the date that teachers were sampled from the vendor lists in place of the TLF. Lastly, the time-to-response model used Cox proportional hazard modeling to predict the number of days until the school would return the TLF.

Split-panel analysis was used to train and validate both the TLF response propensity model and the early vs. late TLF response propensity model. Using the sort order that was used to select the NTPS public schools into sample, data from the NTPS 2015-16 public schools were used for this analysis and were divided into two equal and representative partitions. One partition would be used to train the propensity models, while the other partition would be used for model validation.

Final Sort Order

After the three models were analyzed, the covariates from the models were compared for similarities. Covariates that appeared in two or three of the models were considered strong candidates for inclusion in the final sort order. However, given the sample size and number of experimental groups, not all of the model covariates could be included in the final sort order. Using too many covariates would result in implicit strata that were too small to be defined as effective. The number of effective implicit strata was determined by considering the number of schools per strata, and the distribution of school characteristics within the strata.

Based on the school characteristics that are significant predictors for TLF returns, and the optimal number of sorting strata, the final sort order for public schools was determined to be the following:

• vendor flag,

• priority flag,

• Census region,

• urban/rural locale code,

• response propensity for each school, and

• random number.

Private schools were not sampled in 2015-16, and so no predictions can be made about when private schools might return the TLF in NTPS 2017-18. The response propensity models created for public schools cannot be used to predict TLF returns for private schools with any confidence. Therefore, the final sort order for private schools will not include the response propensity score, and will simply be the vendor flag, the priority flag, the Census region, the urban/rural locale code, and a random number.

Part C - Survey Items Justification & Appendix B - Questionnaires

3. We updated questionnaire items to correct typos and errors found. A list of the updates to the questionnaire from the previous submitted version is provided below as Attachment 1.

Appendix A – Communication Materials

3. We added (s) the “letter of better understanding” on pages104-106 and (b) information about this letter on page 26:

Priority School Regional Office Mailout
Letter of Better Understanding		This letter provides information about how the data collected are used and why they are important. It also provides selected published statistics from the 2011-12 SASS First Look Reports. These letters will be mailed out during the field Phase 1 operations. There are three versions Public School Principal Letters Private School Principal Letters Teacher Letters

Appendix C – NTPS Portal

3. We updated screenshots of the NTPS portal to show their finalized versions. The list of these updates is provided below as Attachment 2. The following text was removed from the Appendix C title page:

A screenshot of the study homepage and several other pages in the website are provided here as examples.

An OMB package has been submitted for usability testing of the portal; a subsequent change request will be submitted upon completion of the testing, if any revisions to the portal are made.

Attachment 1

List of updates/corrections made to the approved in June 2018 (OMB# 1850-0598 v.18) NTPS 2017-18 Questionnaires in Part C and Appendix B:

Questionnaire	Page/Item	Update
NTPS-2A	Page 10, item 4-2a, Page 10, item 4-2c, Page 11, item 4-3a Page 11, item 4-3c	Added: '(apple icon) If none, please mark (X) the box.
	Page 12, item 4-4	[Old]: "teamwide, gradewide, or schoolwide?" [New]: "gradewide, teamwide, schoolwide, or districtwide?"’
	Page 13, item 4-6;	Changed text to: ‘Mark (X) one box on each line.’
	Page 17, item 6-1b	Changed text to: ‘During the LAST school year (2016-17), were you evaluated as a principal at THIS school?’ [Old]: "year," [New]: "year (2016-17),"
	Page 17, item 6-2a, 6-2f	[Old]: "principal or school head" [New]: "principal."
	Page 19, item 7-1	Capitalize ‘THIS’ [Old]: "this" [New]: "THIS"
	Page 19, item 7-4d	-Added period to ‘e.g.,’ [Old]: "(e.g," [New]: "(e.g.,"
NTPS-2B	Page 3, Instruction page	[Old]: "PRINCIPAL," [New]: "PRINCIPAL or SCHOOL HEAD,"
	Page 6, Item 2-2	[Old]: "concerningthe" [New]: "concerning the"
	Page 8, item 3-4	Deleted ‘or districtwide’ from question.
	Page 10, item 4-2a Page 10, item 4-2c Page 11, item 4-3a Page 11, item 4-3c	Added: '(apple icon)If none, please mark (X) the box.'
	Page 11, item 4-3a	Removed ‘district’ from text in parentheses. [Old]: "school, district," [New]: "school"
	Page 14, item 4-8d	Removed ‘district’ from question. [Old]: "school, district," [New]: "school"
	Page 17, item 6-1b	Changed text to: ‘During the LAST school year (2016-17), were you evaluated as a principal at THIS school?’ [Old]: "year," [New]: "year (2016-17),"
	Page 19, item 7-1	Capitalized ‘THIS’ [Old]: "this" [New]: "THIS"
NTPS-3A	Page 4, item 1-2; Page 6, item 1-10 Page 17, item 5-3a Page 18, item 5-6b Page 18, item 5-6c Page 19, item 5-7 Page 19, item 5-10	Changed to text: ‘October 2017’ [Old]: "October," [New]: "October 2017,"
	Page 10, item 2-2	Changed ‘MOST’ to ‘THIRD GRADE’ [Old]: "MOST" [New]: "THIRD GRADE"
	Page 13, item 3-1b; Page 13, item 3-1d; Page 14, item 3-2a	Removed comma after ‘i.e.’ [Old]: "(i.e.," [New]: "(i.e."
	Page 16, item 5-2a	1st bullet, changed ‘1-6’ to ‘1-7’ [Old]: "1-6," [New]: "1-7,"
	Page 18, item 5-6a	Changed text to: ‘GO TO item 5-7 on page 19.’ [Old]: "5-7a" [New]: "5-7"
	Page 18, item 5-6e	Changed text: ‘FRPL’ to ‘NSLP’ [Old]: "FRPL-eligible students.Under" [New]: "NSLP-eligible students. Under"
NTPS-3B	Page 4, item 1-1; Page 5, item 1-2; Page 6, item 1-3; Page 6, item 1-4b; Page 25, item 5-3a; Page 26, item 5-6b; Page 26, item 5-6c; Page 27, item 5-7	Changed to text: ‘October 2017’ [Old]: "October," [New]: "October 2017,"
	Page 8, item 1-9	Added school types: "MONTESSORI school" "EARLY CHILDHOOD PROGRAM OR DAY CARE CENTER – such as kindergarten only, prekindergarten and kindergarten and transitional first grade only, day care and transitional kindergarten only, etc."
	Page 10, item 1-10, first sentence	Added "or" before “program”
	Page 10, item 1-10, (third example text following )response box	[Old]: "10." [New]: "1-10."
	Page 13, item 1-15	[Old]: "U’Mesorah)" [New]: "Umesorah)"
	Page 14, item 1-15, (under OTHER SCHOOL ASSOCIATIONS OR ORGANIZATIONS)	Removed "Coalition of Alternative Community Schools (NCACS) National"
	Page 17, item 2-2	Changed ‘MOST’ to ‘THIRD GRADE’ [Old]: "MOST" [New]: "THIRD GRADE"
	Page 20, item 3-1b; Page 20, item 3-1d; Page 21, item 3-2a	Removed comma after ‘i.e.’ [Old]: "(i.e.," [New]: "(i.e."
	Page 22, item 3-4a (under ‘How is this funded?’)	Remove ‘district’ [Old]: "school, district" [New]: "school"
	Page 27, item 5-8b	Change answer box from 4 to 5 spaces.
NTPS-4A	Page 3, item 1-1 (on 3rd and 4th options)	Remove comma after ‘i.e.’ [Old]: "(i.e.," [New]: "(i.e."
	Page 19, Code 265, 266, 267	Philosophy was moved to code 266; Religious studies, theology, or divinity was moved to 267
NTPS-4B	Page 3, item 1-1 (on 3rd and 4th options)	Remove comma after ‘i.e.’ [Old]: "(i.e.," [New]: "(i.e."
	Page 19, Code 265, 266, 267	Philosophy was moved to code 266; Religious studies, theology, or divinity was moved to 267

Attachment 2

List of updates made to the approved in June 2018 (OMB# 1850-0598 v.18) NTPS 2017-18 Portal in Appendix C:

Item: Main Menu

• “Get Started” ”Go to Screener”; “Go to TLF”

• “Date Received”  “Date”

Item: Encourage Participation [we indicated in the approved (OMB# 1850-0598 v.18) that these changes would be made and now they are reflected in the screen shots]

• Revised wording for Sample 3:

You have a story to tell—about how you became a teacher, about the subjects you teach, about your kids, about how you feel about this school and teaching as a profession. The NTPS is a way for you to share your story with those who make decisions that affect us all. Remember, too, that your responses are confidential and your identity will never be disclosed (unless required by law). Remember, your responses are not visible to anyone at the school and your name will not be published in study reports. The results will only be published as summary statistics. Please take a few moments today to fill out the NTPS.

• Revised wording for Sample 4:

I’ve written before to ask you to lend your voice to those of other teachers who are filling out the NTPS to make sure that decision makers know what you think when they make policies about education. I wanted to make sure that you know that time is running out for you to take the survey and be heard in those discussions. If you will, please find some time within the next day or two to fill out the NTPS. I think it’s important for all of us.

• Revised wording for Sample 5:

It’s been really busy around here recently and I know that you’ve been busy too. I wanted to remind you, though-- your chance to fill out the NTPS is almost over. After the survey period ends, you won’t be able to add your voice to the thousands of other teachers who are participating in this important survey. It will only an hour or so take 40 minutes of your time and will provide really valuable information to decision makers about your views on education. They can only make good decisions with good information. Won’t Will you help by filling out the survey today?

Item: FAQ [we indicated in the approved (OMB# 1850-0598 v.18) that these changes would be made and now they are reflected in the screen shots]

• Revision to “Will my school’s and individual staff members’ data remain confidential?”:

Please be assured that both the U.S. Department of Education and the U.S. Census Bureau follow strict procedures to protect the confidentiality of study participants. All information you provide may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002), 20 U.S.C., §9573]. All of the provided information may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).Your participation is voluntary, but your responses are necessary to make the results of this study accurate and timely.

• Revision to “Who authorizes this study?”:

The U.S. Census Bureau will conduct this survey for NCES as authorized by the Education Sciences Reform Act of 2002, 20 U.S. Code §9541(b) and §9543(a). The National Center for Education Statistics (NCES), part of the U.S. Department of Education, is authorized to conduct NTPS by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). The data are being collected for NCES by the U.S. Census Bureau. The Office of Management and Budget (OMB) approved this survey. The OMB control number is 1850-0598 and the approval expiration date is 06/30/2020.

Item: Accessibility [we indicated in the approved (OMB# 1850-0598 v.18) that these changes would be made and now they are reflected in the screen shots]

• Removed old accessibility web links that brought users to: https://www.census.gov/about/policies/privacy/privacy-policy.html#par_textimage_1

• And replaced it with text only pertaining to Accessibility:

Item: Screener Interview Page 1 and Page 2

• Skip pattern correction: “(Survey Coordinator Contact Information)”  “(Verify School Type)”.

The following pages have been shifted

• Page 3  Page 11

• Page 4  Page 3

• Page 5  Page 4

• Page 6  Page 5

• Page 7  Page 6

• Page 8  Page 7

• Page 9  Page 8

• Page 10  Page 9

• Page 11  Page 10

Item: Page 3, Page 6, Page 7, Page 8, Page 9, Page 10

• Skip pattern: “Page 13 (End of Interview)”  “Page 11 (Survey Coordinator Contact Information)”

Item: Page 8

• If the respondent selects “Yes”, then a drop down box is presented to the respondent to select a grade.

Item: Page 11

• All respondents receive this screen. All in-scope respondents (respondents who have not been directed to Screener Interview Page 12 (Out of Scope) up to this point in the Screener Interview) receive this screen.

• Added text: Data collection for the National Teacher and Principal Survey (NTPS) will begin later this summer. At that time, your school will receive the necessary material to complete three related surveys – a Principal Questionnaire, a School Questionnaire, and a Teacher Listing Form.

• Revised text: The school coordinator is  A “school coordinator” is

• Skip pattern: “Page 4 (Verify School Type)”  “Page 13 (End of Interview)”

Item: TLF Teacher Compare

• Font color changed to black.

• Added screenshot for TLF Instructions page

Item: Provide a TLF

• Added link to TLF Instructions page

• Replaced box text with button text.

• “Get Stated”  “Continue to File Upload”

• “Get Stated”  “Continue to Data Entry”

Item: TLF Instructions

• Revised text: “Select each link to expand. Required information to complete your TLF:”

Item: Data Entry

• “Add Teacher” button moved to the right hand side; removed “clear entry” button

• “Teacher Name”  “First Middle Last Suffix” in example

• Revised text: “Please click the Instructions link for important information about the other staff that may teach at this school (e.g. itinerant teachers, substitute teachers, librarians, and principals.”

“Click here for important information about the other staff that may teach at this school (e.g. itinerant teachers, substitute teachers, librarians, principals). Enter the information for each teacher in your school into the table below.

Click "Add teacher" once you have completed each entry to add the teacher to the summary table at the bottom of this page. Once you have completed your teacher list, click "Submit" to submit your teacher list.”

Our mission is to ensure equal access to education and to promote educational excellence throughout the Nation.