OMB Qs and NCES As

MEMORANDUM OMB # 1850-0790 v.32

DATE: July 28, 2011

TO: Shelly Martinez

Office of Information and Regulatory Affairs, Office of Management and Budget

FROM: Patricia Etienne

National Center for Education Statistics

THROUGH: Kashka Kubzdela

National Center for Education Statistics

SUBJECT: Response to OMB passback on NAEP 2012 Wave 2 Submittal

1. Student background questionnaires:

1. Are we looking at what NCES considers the finally formatted paper questionnaires? 

2. There is switching in both the instructions and the actual items section to section from "square," to "circles" to "ovals" that seems like it needs to be cleaned up.

 Response: Yes, these are the final versions of the questionnaires, except that the grade 4 student writing questionnaire (core and writing-specific) will be delivered on computer. Thus, the text reflects the actual wording (‘squares’ for select all that apply items and ‘circles’ for single-selection items, rather than ‘ovals’) that will be on the computer screen. Writing is the only computer-based student questionnaire for 2012.

2. Pilot teacher questionnaire:

1. Question 8 uses language we are trying to avoid and have pointed out to NCES previously.  We would like you to use “alternative route to certification program”.  Although everyone understands what alternative certification program means, it creates a negative stigma about teachers that earn a certificate through one of these programs.

The item can be revised to the following:

Did you enter teaching through an alternative route to certification program? (An alternative route to certification program is a program that was designed to expedite the transition of non-teachers to a teaching career, for example, a state, district, or university alternative route to certification program.)

A. Yes ➔ Go to Question 9.

B. No ➔ Skip to Question 10.

2. Why are we asking questions about the National Board for Professional Teaching Standards?  It doesn’t seem as relevant as other questions we could ask.  Please provide a rationale for asking these set of questions (#10-13).

National Board for Professional Teaching Standards (NBPTS) certification is considered by many to be the premiere certification route and thus a topic of increasing interest regarding teacher development.  In a 2008 congressionally-mandated study, National Board Certification was recognized by the National Research Council as having a positive impact on student achievement, teacher retention, and professional development.

While the percentage of teachers who have National Board certification is small, we will review the post-pilot response data to determine if these items should be included in the 2013 operational questionnaires.

3. Question 14 is a little awkward.  Why are they asking about graduate coursework in such a way?  Why are we not asking if the teacher has a Masters degree in the subject that they are teaching in?

The question is designed to collect information about any post undergraduate coursework, not specifically completion of a Masters degree. If this item becomes operational in 2013, it will also serve as a predecessor to the following operational item (see below). If a respondent answers “No,” he or she will be directed to skip the next item, thereby reducing burden for those who have not taken any graduate-level courses.

(Please note that the set members for this item are revised depending on the subjects being administered)

Did you have a major, minor, or special emphasis in any of the following subjects as part of your graduate coursework? Fill in one oval on each line.
	Yes, a major	Yes, a minor or special emphasis	No
a. Reading, language arts, or literacy education	A	B	C
b. English	A	B	C
c. Other language arts-related subject	A	B	C
d. Elementary or secondary education	A	B	C
e. Special education (including students with disabilities)	A	B	C
f. English language learning	A	B	C

III. SD and ELL worksheets:

1. Please remove the word ‘student’ when modifying SD.  For example, the SD instructions state “What if an SD student is withdrawn…”  It sounds odd to state student twice here since the form notes that SD means students with disabilities.  In the same vein, it sounds odd to modify “student” with ELL since ELL means English Language Learners.

The following modifications have been made:

“SD students(s)” to “SD” and “ELL student(s)” to ELL(s)” on the worksheet instructions.

2. In general, SD should not have both an IEP and a 504 plan, but there might be schools that have both for some students.  If a student has both, how should a form completer respond to the question, “Record the student’s current SD classification using one of the codes below?”  The completer must choose an IEP, a Section 504 plan, or not SD.  There are no instructions for how a completer should respond if a student has both an IEP and 504 plan. 

The following instructions have been added:

“If a student has both an IEP and Section 504 Plan, record IEP as the student’s SD classification.”

3. How should a form completer respond to “What is the degree of this student’s disability(ies),” if there are no other SD in the student’s grade with the same type of disability?  The instructions ask the completer to pick a response, “based on the knowledge of the person most familiar with how the degree of this student’s disability(ies) compare(s) to other students with the same disability(ies) in the same grade.”  

The guidance has been revised to:

“The response should be based on the knowledge of the person most familiar with this student’s disability(ies) and based on this person’s professional judgment how it/they might compare to other student(s) with similar disability(ies) in the same grade.” 

4. On the ELL Worksheet, the completer is asked to record how long the student has been receiving academic instruction primarily in English.  What does academic instruction primarily in English mean or look like?  It might help to define this more concretely.

The following clarifying instructions have been added:

“Academic instruction primarily in English” means that the instructor presents academic content in English and the student receives only occasional translation of words or explanations of ideas/concepts in his/her native language.”

5. Finally, could we see a sample State-Specific SD Guidelines Summary and State-Specific ELL Guidelines Summary?

The 2011-2012 Guidelines are still being created. Samples from 2010-2011 are attached.

IV. SD/ELL Pilot:

1. Please clarify whether the 50 schools at each grade will be a random sample.

Yes, a random sample of schools is selected within both the Age 9 and Age 13 LTT school samples.  Note that only public schools with at least 25 students age 9 or 13, of which at least some must be 4th graders of age 9 or 8th graders of age 13, are eligible.

2. Please provide a power analysis and data analysis plan for measuring impact of the decision tree on inclusion rates.

The main purpose of the pilot is to determine how the revised SD/ELL Decision Tree and accompanying materials work in the field and, in particular, to identify any situations where the new decision tree doesn’t work properly. So, the data analysis plan revolves around conducting an evaluation based on the debriefing of the field staff and the corresponding schools on their experiences using the new decision tree and materials to determine if there are any failure points.

Power Analysis: Assuming a failure rate of 1 in 100 (i.e., a 1% chance the new decision tree does not work for a given SD/ELL student) an effective sample size of about 300 SD/ELL students would be required to be 95% confident that at least one such failure would be observed.   LTT has two subjects assessed: math and reading.  Exclusion rules are differential, so the effective sample size of 300 is needed for each subject for a total of 600 SD/ELL students required.  We estimate that about 800 SD/ELL students (math and reading) will be subject to the new decision tree in each age and therefore sufficient to meet these pilot goals.

3. Please explain what oversampling of students means in this context.

Within the schools selected for the SD/ELL Decision Tree Pilot a supplemental sample of up to 5 additional SD/ELL students were selected within the school.

How many students per school does NCES estimate?

For larger schools, this could potentially increase the total number of selected students from 64 to 69 students (SD/ELL and non-SD/ELL).

Will the remainder be included in "main" NAEP?

If by remainder OMB means the non-SD/ELL students, they will indeed be included in the “main” or operational NAEP sample.