ED Respone to OMB Comments

Mathematica Policy Research

OMB QUESTIONS ON NLTS 2012 BASELINE CLEARANCE REQUEST
AND RESPONSE

1. Do students with a 504 plan have a “Special Education” teacher?  If yes, should these teachers also be surveyed with the “Student’s Special Education Teacher” instrument?

In general students who have a section 504 plan (which specifies accommodations that will be made so the student can be served in a regular classroom) but do not have an Individualized Educational Plan (IEP) (which specifies services and accommodations necessary for a student’s free appropriate education) will not have a special education teacher. However, a key issue both for students with an IEP and for students with a section 504 plan is accommodations used in the classroom for the student. The math/language arts teacher survey requests information for members of both groups in section C of the math- language arts teacher survey.

2. Can the interviewer let the parent or student know that giving the student’s SSN and accessing their social security records will not affect their benefits?  And that in the confidentiality section please stress that the information that the interviewer gets will not affect any of the respondent’s public benefits.

In the baseline consent at A7, we are not requesting that the parent provide the student’s Social Security Number at this time. Rather, in order to fully disclose all elements of the study before the parent grants consent, we are telling the parent we may come back to the parent and student to request this information at a later date. If a decision is made to request the Social Security Number at a later date, we will stress in that subsequent request that whether the respondent provides or does not provide the Social Security Number will not affect receipt of any benefits.

3. Are there any questions that ask about whether the student receives vocational rehabilitation services?  We saw that question E6 of the Parent Survey asks whether a VR representative took part in the transition planning meeting, but couldn’t find others.  We’d like to know if and what kinds of VR services survey participants receive.  If needed, please work with RSA and Budget Service on developing additional questions.

The following questions are included in the Student’s School Program Survey.

• B5 Which of the following services has been provided to this student from or through the school system during this school year (2011-2012), including services contracted from other agencies? (Response “r” is Vocational Rehabilitation Services).

• C10 In Column A, indicate who has actively participated in the student’s transition planning. We define active participation as being involved in discussions regarding services and goals. (Response “k” is a counselor from the vocational rehabilitation agency.)

• C15 Have the following been contacted this school year about the students’ activities when he/she leaves high school? (Response “d” is Vocational Rehabilitation Agency.)

The response to question three asserts that not many students with disabilities receive VR services while in high school.  However, we have learned that RSA’s know a number of States such as Illinois, Arizona, Oregon, Tennessee, Iowa, and Florida, and others that are providing work experience and other services to students through a third party.  Unfortunately, RSA does not have national data to confirm this but this is something RSA has learned through their monitoring.  So, we still would like this survey to collect information from both the parent and student on the nature and frequency of VR services received while the student is in high school.

In consultation with RSA, we have included the following new questions in the School Program questionnaire to better understand how involved VR agencies are while students are still in school. The School Program questionnaire was selected because we and RSA believe that the most accurate response for this type of question will be provided by the special education teacher. Given the ‘third party’ nature of the services provided, it is unlikely that students and parents will know who is providing the services they are receiving.

• B5a. Did/does this student receive any services funded by Vocational Rehabilitation Services this school year (2011-2012)? Answer categories are: Yes, No, or Don’t Know.

• B5b. Which of the following services did/does this student receive from Vocational Rehabilitation Services? Answer categories are Yes, No, or Don’t Know for each.

Career counseling; Goal setting and career planning; Job assessment and appraisal; Health advice and promotion that supports working; Case management, referral, and service co-ordination; Interventions to remove environmental, employment and attitudinal barriers; and Job development, and placement services, including assistance with employment and job accommodations.

We anticipate that follow-up surveys (for which OMB clearance will be requested at a later date) will obtain information from youth and parents about the nature and extent of VR services received after the student has left school, and respondents’ perceptions about the services. Data on VR involvement from the baseline and first follow-up interview on behalf of students nearing completion of school, in conjunction with data obtained about post secondary VR services will be used to describe and explore the connections among early VR involvement in transition planning, post school student outcomes, and students’ perceptions of VR services.

4. Should the Math/LA teacher survey also ask how many students with 504 plans are in the class?  Right now question A5 only asks about students with IEPs.

Our best estimate of the proportion of all students ages 13-21 who have a Section 504 plan but no IEP appears to be relatively low – in the range of 1-2 percent. While the distribution of such students across classrooms is unknown, it is likely that most teachers will have zero students and others will have one. To limit burden on teachers, we decided not to include a question on the number of students with 504 plans because the incidence at the classroom level is likely to be so low. We are, however, asking the school principal for a count of the number of students with a Section 504 plan but no IEP in the School Characteristics Questionnaire (question B3). The School Characteristics Questionnaire will provide data with which to describe the schools attended by students who are between 13 and 21 and attending public schools with grades 7-12 in December 2011. Question B3 will allow tabulations of the mean and distribution of the percentage of students with Section 504 Plans but no IEP in the schools attended by this population nationally.  We will also be able to provide this tabulation separately for the main subgroups of our sample (IEP/non-IEP section 504/nonIEP, non- section 504 status, IDEA category, etc.).  

5. The parent and student surveys don’t ask about the academic components/challenges the student had in secondary school.  Why is that?  We thought that knowing this information could provide valuable context to a student’s post-school outcomes that cannot be gleaned from an academic transcript.

It is not clear what issues about academics the reviewer is suggesting should be addressed. We judged that relatively simple questions about whether and which academic components were challenging would provide very limited information with which to understand post school pathways. However, the parent interview questions F1-F4 ask about college courses the youth may have taken for credit in high school, either for advance placement credit or for exposure or preparation for a career. Additionally, the student interview at K2a and K2b asks about the extent to which the youth found school challenging.  Specifically, it asks the youth the extent to which she or he agrees with two statements: 1) “Class work was hard to learn.” 2) “I had trouble keeping up with the homework.” Considerable attention was given to obtaining information that could not be obtained from an academic transcript, but that is likely to condition successful transition. For example, Section D of the Math or Language Arts Teacher Survey asks about relative academic performance (D1), engagement (D2) and social skills (D3) of the student. Section K of the Student Interview asks a series of questions designed to gauge the youth’s engagement in school and sense of being connected to school. We would be happy to discuss this issue with OMB if there are specific items of interest.

RESPONSES TO OMB ON INSTRUMENTS

Teacher Survey Question A5. What does “usually” mean here? Either define or remove.

Response. The word “usually” was used to convey the concept of a typical day. To clarify the question, it will be modified to read: How many of the following are in this class on a typical day? (Include yourself in the count.)

Teacher Survey Question B2. Is the difference between “multimedia” and materials one can download (g and I, respectively) apparent?

Item g: Multimedia (the combined use of text, captioning, graphics, animation, pictures, video, and sound to present information)

Item i: Materials that students can download, including podcasts

Response. Item B2 looks at the ways the class as a whole and the specified student use a number of different types of instructional materials to engage with the course content. Item g (above) uses the term “multimedia” to capture learning tools embodying the principles of universal design instructional techniques, which call for using multiple approaches to conveying information that allow students to access the information in ways best suited to their needs and learning styles. Use of such tools may be especially useful and important for students with disabilities. These may, or may not, be downloaded by computer. Item “i” is intended to be distinct, in that it could involve just a single medium (such as a podcast) and is obtained via downloading.

While we appreciate that there is a conceptual difference between multimedia and downloaded materials, we believe that ED needs to make these categories clear somehow.  Perhaps a parenthetical definition next to the item?

Response. We have reordered the list of instructional materials to place “multimedia” closer to the first four types of materials (none of which refers to the computer), and we have combined prior items i. and h. Please see below for how we have revised and reordered the response categories. We believe that by moving the computer questions down the list and combining two of the response categories, we will reduce any confusion. Given that we already have parenthetical definitions next to these items, it was not clear how else we would elaborate.

(NLTS2 B3A/B modified)

B2. In column A, please indicate how often the class as a whole uses the following instructional materials.

In column B, indicate how often this student uses these materials.

	Column A The class as a whole			Column B This student
	Never or Rarely	Sometimes	Often	Never or Rarely	Sometimes	Often
a. Textbooks, worksheets, workbooks, curriculum-based materials	1 □	2 □	3 □	1 □	2 □	3 □
b. Fiction and non-fiction books, newspapers, and magazines	1 □	2 □	3 □	1 □	2 □	3 □
c. Manipulatives in class	1 □	2 □	3 □	1 □	2 □	3 □
d. Games and toys used for instructional purposes	1 □	2 □	3 □	1 □	2 □	3 □
e. Multimedia (the combined use of text, captioning, graphics, animation, pictures, video, and sound to present information)	1 □	2 □	3 □	1 □	2 □	3 □
f. Computers for word processing, spreadsheets, and other applications	1 □	2 □	3 □	1 □	2 □	3 □
g. Computers for academic drills and skills practice	1 □	2 □	3 □	1 □	2 □	3 □
h. Computers for accessing information or lessons on the internet or downloading materials, such as podcasts	1 □	2 □	3 □	1 □	2 □	3 □
i. Interactive white boards or smart boards	1 □	2 □	3 □	1 □	2 □	3 □
j. Tablet PCs such as iPads	1 □	2 □	3 □	1 □	2 □	3 □
k. eBooks, eBook readers such as Nook or Kindle, v‑books	1 □	2 □	3 □	1 □	2 □	3 □
l. Digital portfolios (a collection of the student’s work)	1 □	2 □	3 □	1 □	2 □	3 □
m. E-pals (electronic pen-pals)	1 □	2 □	3 □	1 □	2 □	3 □
n. Other – Specify: _________________________	1 □	2 □	3 □	1 □	2 □	3 □

Was this cog labed?

This instrument underwent pretesting with nine respondents, completed in both self-administered (mail) and interviewer administered (telephone) formats. Each participant provided feedback on the instrument after completion – identifying any items he / she felt were confusing, particularly sensitive, or overly burdensome to answer. This feedback was incorporated into the final revisions of the instruments, reflected in the final submission to OMB in October of 2011.

Teacher Survey Question B5. Are there proxies for these strategies? Do you actually expect teachers to respond “never” to any of these?

These items were drawn from PBIS (Positive Behavior Intervention Supports) and our goal is to determine the extent to which PBIS best-practices are implemented across different types of schools and settings. The language used for each item within B5 aligns closely with the PBIS measures intentionally, as it was deemed to be a straightforward way to approach each concept of interest. We elected to remove the word “positive” from the question stem to minimize the potential for social desirability to affect responses.

Findings from the 9-case pretest data indicate a fairly even split between the responses of “sometimes” and “always” – which shows respondents did not feel compelled to report strict adherence to these practices. In addition, one of the nine did use the “never” category for one item.

Details were as follows:

B5. To what extent do you use the following behavior strategies?

	N Never	N Sometimes	N Always	N Total
a. I convey my behavior expectations to students and reinforce those expectations through my curriculum	1	2	6	9
b. I provide a continuum of consequences for behavior infractions	0	3	6	9
c. I consistently reinforce positive behaviors	0	6	3	9
d. I share information on individual student behavior among teachers and use this information in my decision making	0	7	2	9
Total	1	18	17	36

It would be reasonable to assume teachers would be reticent to select the option of “never” for these behaviors, as it could be seen as an “absolute.” Therefore, we propose revising the text on the scale to be “Rarely” “Sometimes,” or “Often.” Since this is an addition, we don’t need to be concerned about alignment with NLTS 2.

Teacher Survey Question C2. Why no “other” category? Is this an exhaustive list, or do you just care about the supports listed?

Response. We will add the “other – specify” category item to C2 in alignment with the other items where the option is presented.

Teacher Survey Question E6. Why no neutral category in the scale? How does a forced answer benefit the data here?

The concepts of interest captured in item E6, such as whether a teacher feels he / she has adequate training and support to carry out his / her duties effectively, are not concepts that would evoke a neutral response. Should a teacher not feel especially strongly about the issue, he / she selects the middle range options within the scale. Likewise, the extent to which the school leadership is setting standards (item a) and promotes instructional improvement (item b) among staff are statements describing a process in the school that is either happening (agree) or not happening (disagree) – where a neutral category would not be applicable or logically consistent. Below are the responses to this item on the pretest.

E6: Please indicate the extent to which you agree or disagree with the following statements:

	Strongly Disagree	Disagree	Agree	Strongly Agree	TOTAL
The school leadership has high expectations and standards for all students and teachers:		3	4	2	9
The principal promotes instructional improvement among school staff:		1	5	3	9
This school is a safe place for students:		2	4	3	9
I have adequate training and support for teaching students with disabilities:		2	7		9
I have adequate training and support for teaching English Language Learners (ELL):	3	3	3		9
I have adequate training and support for teaching students who struggle academically:		3	6		9
TOTAL:	3	14	29	8	54

INCENTIVES QUESTION:

Unsure how a payment per student will actually increase the response rate, which is the purpose of an incentive.

As noted Part A section 9 of the OMB supporting statement dated October 11, 2011, we also propose to offer incentives to participation in the study to both parents ($20) and students ($10) as a “thank you” for completing the baseline interview. We believe this is important for promoting participation in the study during our first contact with parents and students. Also as noted, this first contact will occur in one of two ways: (1) when we call parents and students directly for consent and baseline interviewing, or (2) when they receive a consent form from the district asking to release contact information. Our discussion in the October 11, 2011 submission focused on evidence that such incentives can be effective for obtaining and sustaining over time the participation of adults in longitudinal studies. We believe offering similar incentives for youth in our study population is very important and will have several benefits for the study.

First, we note that the parent is the “gate keeper” because we must first have consent of the parents in most cases in order to request assent of the student. We believe the offer of a small “thank you” to the student will affect the willingness of the parent to have their student participate. In particular, it reinforces our statements to the parent that the experiences of their student are important for the study and valued, and may be especially persuasive for the parents of youth with disabilities.

Second, our experiences in studies of youth strongly suggest that the advantages outlined in the literature for adults participating in longitudinal studies also apply for youth participating in longitudinal studies. In the evaluation of the Youth Transition Demonstration Project, we offered a similar incentive for the youth at baseline, 12-month follow-up and 24-month follow-up. This study secured follow-up interviews from 90 percent of the sample at 12-month follow-up and 80 percent at the 24 month interview. The National Job Corps Study followed for a period of four years a sample of youth ages 16-21 who were enrolling in Job Corps. With incentives to this youth sample, that study was able to complete interviews with nearly 80 percent of the sample at a point four years after initial entry to the sample. Although we do not know what the response rates would have been for the youth in either of these studies without the incentives, our experience was consistent with the evidence reviewed by Laurie and Flynn (2008) as cited on page 17 of our Part A Supporting Statement Submission dated October 11, 2011.

We have an issue not with the amount given to the parents or students, but rather to the teachers.  Why should a teacher receive X amount per student, if what we’re dealing with are incentives and not payments?

We think that incentivizing teachers for their time to complete the survey is justified for several reasons. Although it has historically been OMB’s perspective that teachers should complete ED-sponsored surveys as part of their regular duties, we believe there are circumstances when that rationale is more or less compelling. For example, incentive for completion of a teacher survey may be less necessary in studies where there is otherwise a benefit to the school or classroom from participating; that benefit may include a “school” payment to offset study burden, a study-provided intervention (e.g., a promising curriculum or induction program), or a strong presence of the study team in the school (e.g., a significant set of participating students, classroom observation, conducting professional development, etc.). We might expect that these components of the study could affect the teacher’s motivation to complete the survey either extrinsically (the school/classroom is receiving benefits for participation) or intrinsically (teacher agreed to participate in larger study and is therefore more likely to fully participate).

However, in the case of NLTS 2012, the teacher is unlikely to have similar motivation because:

a) The teacher/class is not receiving any direct benefit for participation, thus not providing extrinsic motivation;

b) The prescriptive sample design required to obtain appropriate counts in each of the disability categories makes it possible that a math or language arts teacher may only have 1 or 2 sampled students in their class who have been selected to participate in the study; that level of exposure to the study is unlikely to provide sufficient extrinsic motivation to complete the survey; the special education teacher may have more exposure to the study (i.e. 3 or 4 students) but it is still unlikely to be enough to provide extrinsic motivation;

c) There will be no or very limited face-to-face contact with members of the study team which could serve to provide some social motivation for completing the survey. Most of the NLTS 2012 surveys will be completed via the web or telephone; and

d) There is no separate stage in which the teachers agree to participate in the study (the youth is the targeted sample) therefore we cannot count on teachers’ intrinsic motivation to complete the survey.

Finally, several considerations led us to recommend separate incentives for each student that a teacher reports on. 

• First, the information requested for individual students is unique to that student. The teacher is being asked to consider the unique experiences of each student (i.e. the second or third report takes as much thought, time, and effort as the first one).  Someone that only has to respond about one student is on the lower burden end, whereas multiple students moves a teacher to a higher burden level. 

• Second, the request for different students may reach a teacher at different points in time, given that the request cannot be made until after the parent/student have consented/assented. In this context it is important that the incentive be the same for each student, so the teacher does not become confused about whether s/he has already responded. 

• Third, we were concerned about the perception of equity across teachers if all teachers were to receive a one-time incentive without regard to the number of students reported on. Being able to assure district officials in the district recruitment process that teachers would receive a ‘thank you’ for each student they report on and that the “luck of the draw” would not create a perception of different treatment for different teachers has been important in the recruiting of districts and we believe it will enhance timely responses to our requests for data on the experiences of individual students.

12