State Director Interviews

NETI

The National Evaluation of Title III Implementation

____________________________________________________

Draft State Title III Director Interview Protocol

October 20, 2009

Prepared By: Prepared For:

American Institutes for Research U.S. Department of Education

Contract No. ED-04-CO-0025/0017

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB number. The valid OMB control number of this information collection is XXXX-XXXX. The time required to complete this information collection is estimated to average 60 minutes. If you have any comments concerning the accuracy of the time estimates(s) or suggestion for improving this form, please write to: U.S. Department of Education, Washington, 20202-4651. If you have comments or concerns regarding the status of your individual submission of this form, write directly to:

Policy and Program Studies Service, Office of the Deputy Secretary, US Department of Education, 400 Maryland Avenue, SW, Washington, DC 20202.

State Title III Director Interview Scheduling Email

Date: [October, 2009]

To: [State Title III Coordinator]

CC: [[email protected]; [email protected]]

Subject: National Evaluation of Title III Implementation

Attachments:

The National Evaluation

of Title III Implementation

Dear [State Title III Director],

Last October, you were copied on a letter to your state superintendent from the U.S. Department of Education introducing the National Evaluation of Title III Implementation,¹ being conducted by the American Institutes for Research (AIR).  This October, we are writing to schedule the State Title III Director telephone interview mentioned in that earlier letter.

This study, sponsored by the Policy and Program Studies Service of the U.S. Department of Education, is the first Federal study to take an in-depth look at the implementation of Title III at the state and local levels. The study will inform the U.S. Department of Education and the Congress about the implementation of the program and will inform the next reauthorization of the Elementary and Secondary Schools Act (ESEA). The study will provide important information about English Language Proficiency (ELP) standards, ELP assessments, annual measurable achievement objectives (AMAOs), and activities designed to improve outcomes for English language learners.

The telephone interview of State Title III Directors is one of the study’s most important data collections, SO we hope that you can find time to speak with us. The interview will take approximately one hour and we would be happy to schedule it at your convenience. We know that states are required to submit a good deal of documentation about their implementation of Title III, and we want to assure you that our research team has conducted a thorough review of state policy documents and reports to ensure that the survey only includes questions that we are not able to address well through extant sources. Prior to the interview, we will email you the data we have compiled about your state so that you can confirm its accuracy.

Please reply to this email to let me know if and when you are available for a one-hour interview during the next two weeks. Alternately, our scheduler, Lindsay Anderson, will follow up with you to schedule a time at your convenience.  Should you have any questions, please feel free to contact me.

Thanks very much in advance for your cooperation. We truly appreciate your willingness to provide the time and expertise needed for the success of this important study.

Sincerely,

[Name of Interviewer]

National Evaluation of Title III Implementation

[Interviewer’s Phone number and contact information]

CC:

Jennifer O’Day, Ph.D. James Taylor, Ph.D.

Principal Investigator Project Director

National Evaluation of Title III Implementation National Evaluation of Title III Implementation

The National Evaluation of Title III Implementation (NETI) is funded by the Policy and Program Studies Service (PPSS) of the US Department of Education and conducted by the American Institutes for Research (AIR) with its partners the Windwalker Corporation and edCount, LLC.  The valid OMB control number of this information collection is XXXX-XXXX.

Dear State Official,

Thank you for agreeing to participate in the Evaluation of State and Local Implementation of Title III Standards, Assessments, and Accountability Systems. To recap what our accompanying materials have already described:

• Purpose of Study: The purpose of this study is not to monitor Title III implementation, but to describe the implementation of Title III in states and school districts across the country.

• Sponsor: The study is being conducted by the American Institutes for Research, edCount, and Windwalker Corporation under a contract from the U.S. Department of Education.

• Response Burden: Preparation for this interview, including completion of the data confirmation document, should require less than 30 minutes of your time. The telephone interview itself should require approximately 60 minutes of your time.

• Benefits: Your participation will help inform policy makers, educators and researchers at the local, state, and national level of the implementation of No Child Left Behind.

• Participation: Your participation in the study is mandatory as a condition of receiving Title III funds.

• Confidentiality: Because these questions reflect your state’s policies or practices, they may be reported on a state-by-state basis, and we can’t ensure confidentiality.

• More Information: For questions or more information about this study, you may contact the Project Director, James Taylor, at AIR at 1-202-403-5000.

State Title III Director Data Confirmation Document-DRAFT

Instructions: Please read through the following items carefully to confirm the data we have collected for your state from various sources. If all the data are correct, no further action is needed. If any item is not correct, please cross out the pre-filled information and provide the correct response along with any explanation you feel is necessary. You may choose to mail, fax, or email any corrections you may have.

If we do not hear from you, we will assume the data to be accurate and it will be used in analyses for this study.

We appreciate the time you take to review the data for accuracy. If you have any questions or comments, please do not hesitate to contact us using the information at the end of this document. Thank you.

ELP Standards

1. ELP standards currently being used in your state (2009-10) were developed in the following manner: __pre-filled_________________________(1.1.3)

2. First year your current ELP standards (2009-10) were implemented: __pre-filled_________(1.1.1) (1.1.2)

ELP Assessments

3. ELP assessment(s) being used for NCLB accountability purposes in your state for 2009-10: __pre-filled_____________________________(2.1.4)

4. First year your current (2009-10) state English language proficiency assessment(s) was/were implemented? __pre-filled___________(2.1.1)

Annual Measureable Achievement Objectives (AMAOs)

4a. Does the following accurately reflect your current target(s) and definition of AMAO1 – making progress in learning English? __pre-filled___________

4b. Does the following accurately reflect your current target(s) and definition of AMAO2 – attaining English proficiency? __pre-filled___________

4c. Does the following accurately reflect your current target(s) and definition of AMAO3 – adequately yearly progress for the limited English proficient subgroup under Title I ? __pre-filled___________

Title III Subgrantees

5. Number of subgrantees in your state for the 2008-09 school year: ______pre-filled____________(5.1.1)

6. Number of subgrantees in your state that missed their AMAOs for 2008-09: _pre-filled_________(5.1.1)

7. Number of subgrantees in your state that have missed their AMAOs for 2 consecutive years (as of the 2008-09 school year): __pre-filled___(5.1.1)

8. Number of subgrantees in your state that have missed their AMAOs for 4 consecutive years (as of the 2008-09 school year): __pre-filled____(5.1.1)

State Data System

9. Your state data system has a unique student identifier: (yes/no) (4.2.2)

10. Your state data system has a unique teacher identifier: (yes/no) (4.2.3)

Highly Qualified Teacher Status and Professional Development

11. NCLB highly qualified teacher requirements currently apply to ESL and/or bilingual programs in your state (all, some, none): __pre-filled____(4.5.3)

12. Your state considers ESL/ESOL to be a core academic subject, to which NCLB’s highly qualified teacher requirements apply: __pre-filled__(4.5.3)

13. Incentives offered by your state to highly qualified teachers to teach in schools or classrooms serving large proportions or numbers of LEP students at any point since the passage of NCLB: __pre-filled_____(4.5.2)

14. Specific training requirements by your state for teachers of LEP students (mainstream or LIEP): __pre-filled__________________________(4.5.6)

Teacher Fluency

15. Teachers in your state demonstrate English language fluency through the following method(s): __pre-filled____________________________(4.5.1)

16. Teachers in your state demonstrate fluency in languages other than English through the following method(s): __pre-filled___________________(4.5.1)

17. Teacher language fluency requirements are determined/specified at the following level (district or state): __________________________(4.5.1)

Use of Student’s Native Language for Instruction

18. Your state’s policy on use of a student’s native language for instruction (e.g., bilingual education) is: __pre-filled______________________(3.2.3.5)

Title III State Interview Protocol - DRAFT

Pre-interview script

Evaluation of State and Local Implementation of Title III Standards, Assessments, and Accountability Systems

• Evaluation funded by the U.S. Department of Education

• New and different study than SSI-NCLB, but some questions will follow-up information collected from that study

• American Institutes for Research is an independent contractor conducting these interviews

• Collected as much information as possible through available documents

• Collaborated with the National Clearinghouse for English Language Acquisition (NCELA), which has already compiled much data on Title III implementation

• Today’s focus is on questions that we haven’t been able to address through these other sources.

• Confirm that Data Confirmation Form was completed and check for any outstanding questions

Recording

• In order to capture the large amount of data your responses will provide, your interview will be digitally recorded. This tape may be shared with others within our evaluation team, but only for purposes of analysis. Is it okay to begin recording now?

• [Press *2] There will be a brief pause and an automated voice will indicate that recording has begun.

Do you have any questions before we begin? Let’s get started.

Standards for English language proficiency

I’d like to ask you a few questions about your current English language proficiency standards.

1. Is your state planning any changes to your current ELP standards? (1.1.2)

• No

• Yes Please describe the anticipated changes. When will they be implemented? ________________________________________

2. Can you describe the primary way(s) in which the state (SEA or related state agencies) uses the ELP standards? [Initially, don’t probe, but check the boxes as they are mentioned] (1.1.6)

• Text selection/approval

• Curriculum development/selection/approval

• Support programs chosen (PD, etc…)

• Other _____________________________________________________

3. Does your state provide guidance or support to subgrantees on how to implement the standards? (1.1.6)

• No

• Yes Please describe support _____________________________

4. Has your state developed any linkages between your ELP standards and your state’s content standards? (1.1.4)

• No  Do you have any plans to make the linkages?_______________

• Yes  Can you briefly describe the linkage process, including who was involved, and which subjects were linked? ______________________

• Did you use an outside contractor to do the linkages or was it done in-house? _____________________________________________

Assessing LEP Students

Now I’m going to ask a few questions about how your state defines LEP and places students in LIEP programs.

5. How does your state define “LEP student”? _______________________ [If assessment results are included, obtain details of assessment--ask if assessment is the same as is used for accountability] (3.1.3.1) (2.1.3)

5a. Are there additional criteria used to identify LEP students not included in your state’s definition of “LEP student”? (3.1.3.1) (2.1.3)

• No

• Yes  Please describe the additional criteria ____________________

6. What criteria does your state use for the placement of LEP students into LIEP programs? ___________ [If assessment results are included, obtain details of assessment – ask if assessment is the same as is used for accountability] (3.1.3.1)

7. What criteria does your state use for determining the exit of LEP students from LIEP programs? ______________________________ [If assessment results are included, obtain details of assessment – ask if assessment is the same as is used for accountability] (3.1.3.1)

8. How does your state determine when an LEP student has been has exited the LEP subgroup for Title III purposes (been reclassified)?________________ (3.1.3.1)

8a. Is that the same definition used for Title I accountability? (3.1.3.1)

• No

• Yes

9. Are decisions about identification, placement, reclassification and exit of LEP students from LEP services made at the state level or district level? (2.1.3)

• State Level

• District Level

• Other [Please describe] _______________________________

10. Has your state aligned your ELP standards with your ELP assessment? (2.2.1)

• No  When does your state hope to undertake the alignment process? _________________________________________________________

• Yes  Can you briefly describe the alignment process, including who was involved? ________________________________________________

• Did you use an outside contractor to do the alignment?

  • No

  • YesDid you make any changes/adjustments to the alignment report completed by the contractor? [Y/N]

11. Can you describe for me how the results of the ELP assessment (used for accountability) are used at the state level? _________________________(2.1.4)

12. Please describe the development of your state’s AMAOs. [Capture open-ended data but probe for sub-items if not mentioned] _________________________________(3.1.1.1)

12a. What goals and contextual factors did you consider when setting your state’s current AMAOs? _________________________________(3.1.1.1)

12b. What kind of data did you use to develop them? [If assessment data was used, proceed with 12b1 and 12b2] _____________________________________________(3.1.1.1)

12b1. What is the name of the assessment? _____________(3.1.1.1)

12b2. Which years of assessment data were used to set the targets? ____

(3.1.1.1)

12c. Do your current AMAOs take into account the amount of time students have had access to LIEPs? (3.1.1.1)

• No

• Yes Describe how that element is included.___________________

  • Is this a change from past policy? ________________________

12d. Do you believe the targets that were set for your current AMAOs are goals that are realistically attainable/reachable? (3.1.1.1)

• No  Why not? ___________________

• Yes Why? ___________________

12e. Do you believe the targets are set in a way that challenges the subgrantees to improve their practices? (3.1.1.1)

• No  Why not? ___________________

• Yes Why? ___________________

13. For what school year were your current (2009-10) AMAOs first put into place? __________(3.1.1.4)

14. Can you briefly describe the changes (and the reasons for the changes) in your AMAOs from 2002-03 to now? _______________________________(3.1.1.4)

15. Does your state apply a minimum subgroup size (n-size) to AMAO determinations?

(3.1.3.1)

• No

• YesWhat is that number? ________ Is this the same n-size used for Title I accountability? _____________

15a. When did the state send notice of performance relative to annual measurable achievement objectives (AMAOs) under Title III based on the last school year (2008-2009)? ______

16. Does your state fund any consortia of subgrantees [for districts with too few LEP students to qualify individually]? (3.1.3.1)

• No  Is there a specific reason as to why not? _________________

• Yes

  • How many districts are in consortia (out of how many total)? ____

  • What criteria are used to determine if districts can form consortia? ___________________________________________

  • How are districts in consortia treated when it comes to AMAO determinations and accountability actions? (are they lumped together or separated?) _______________________________

  • What are rules on dissolving consortia and re-forming consortia with different districts in various years? ___________________

17. How is your state responding to the Notice of Interpretations released in fall 2008? __________________________________________________(3.1.1.1) (3.1.1.4)

Support

I’d like to talk about other consequences or state actions for districts that do not meet their AMAOs for 2 consecutive years or more under NCLB. Does your state have any districts/subgrantees in that situation?

18. If I were a superintendent in a district that did not meet the state’s AMAOs for 2 consecutive years or more, what actions or requirements should I expect from the state? [Capture open-ended data but listen for the following responses—probe on first 3 if necessary, especially TA] (3.2.3.1)

• Be required to write an improvement plan

• Receive technical assistance ask item 22

• Be required to develop professional development strategy in collaboration with SEA

• Be required to implement specific programmatic interventions (e.g., new curriculum, new language instruction educational program)

• Discontinuation of Title III funding

• Replacement of educational personnel

• Other (e.g., improvement grants) [Please describe] _________

PROBE: Are these actions ONLY for districts not meeting AMAOs or part of activities done with other districts across the state? ______________

19. To whom does your state apply these actions? [Don’t read list but code for the following variables and probe if necessary] (3.2.3.1)

• All districts not meeting their AMAOs for 2 consecutive years or more

• All Title III and Title I districts not meeting their AMAOs for 2 consecutive years or more

• All Title III districts not meeting their AMAOs for 2 consecutive years or more

• Target a subgroup of districts with a subgroup of actions [Proceed to 19a and 19b]

19a. Which districts are targeted? On what basis do you target these districts? ___________________________(3.2.3.1)

19b. Which sanctions are applied to which districts? _______________________________________(3.2.3.1)

20. How do you decide which actions to apply to these districts? [Capture open-ended data, but listen to whether or not a plan exists, if a needs assessment is done, or if it’s reactive] __________________________________________________(3.2.3.1)

21. [If state have subgrantees missing AMAOs for 4 consecutive years] Do the actions taken with subgrantees missing AMAOs for 4 consecutive years differ from what you just described for those missing for 2 years? (please describe differences) _____________________________________________________(3.2.3.1)

22. [Skip if technical assistance not mentioned in 18] Earlier you mentioned technical assistance as one of the actions the state provides districts not meeting their AMAOs. Can you describe the state-supported technical assistance provided to districts in this situation? We’re looking specifically at who received the assistance and what the focus was, and there’s a certain type of technical assistance we’re asking about. When I say technical assistance, I mean any assistance provided to district or school staff (both administrators and teachers) to help them implement Title III and any of the LEP student requirements outlined in NCLB. For this part of the interview, I don’t want you to include assistance that focus directly on improving teachers’ knowledge, skills, and instructional practices in the core content areas – in other words, interventions that we usually think of as professional development for teachers. That assistance will be addressed later in the interview. [Capture open-ended data. Don’t read lists, but code for the following variables and probe if necessary.] ____________________________________________(3.2.3.1)

22a. Recipients (3.2.3.1)

• District administrators

• School administrators

• Teachers (make sure it’s TA only and not PD!)

• Other (parapros, community, parents, etc…) [Please describe] _________________________________

22b. Focus (3.2.3.1)

• Assessment

• Data (use/collection)

• NCLB requirements

• Placement/reclassification of LEP students

• Curricular or instructional practices

• Other [Please describe]____________________________

23. Broadly speaking, does the state provide technical assistance to all subgrantees, regardless of AMAO status? If so, how might that be different from what is provided to those missing their AMAOs? __________________(3.2.3.1) (4.3.1)

24. Do you differentiate your support or actions based on which AMAOs were missed [1 and 2, vs. 3, for example]? (3.2.3.2)

• No

• Yes  How do you differentiate? _________________________

25. Do you have specific requirements for or provide specific kinds of supports for districts that have both missed their AMAOs and that have been identified for improvement under Title I? (3.2.3.2)

• No

• Yes  What are the plans? ______________________________

State Support Infrastructure

Now I’d like to talk a bit about how the SEA organizes its work respect to Title III and LEP issues.

26. How many FTEs do you have at the state level working with Title III? ______(4.1.3)

26a. How many people fill these FTEs? _______________________(4.1.3)

27. Could you tell me a bit about your position and responsibilities at the SEA as Title III director? ____________________________________________________(4.1.3)

27a. To whom do you report (what division)? ________________________

27b. What are your specific responsibilities with respect to Title III?

Are you responsible for calculating and tracking AMAOs or is that done elsewhere in the department? _______________________

27c. Do you work only on Title III or do you also work in other programs or arenas? ____________________________________________

28. Are there staff in other SEA units who have some responsibility for LEP issues? (4.1.3)

• No

• Yes Please describe ______________

  • Do you coordinate your work with any of these staff or do any of them work directly on title III? _________________________

29. Could you send us or direct us to an organizational chart for the state department of education so I can get a better idea of the structure? (4.1.3)

30. Can you please describe the expertise of those in the State Title III office and others in the SEA that work to implement Title III requirements (assessments, accountability, TA, PD, etc…)? ______________________________(4.1.4)

31. Please describe any coordination between the Title III and Title I offices _____________________________________________________(4.1.2) (4.1.3)

31b. If you coordinate, on what items do you work together? (4.1.2) (4.1.3)

Now I’d like to ask a few questions about your state data system.

32. Do you have a single data system at the state level holding all information on each student or is there a separate data system maintained for items specific to LEP students? (4.2.5)

• Single system

• Multiple systems Please describe the various systems and how they are used ________________________________________________

33. What are the primary purposes for which the state uses LEP-related data contained in the state data system? _________________________(2.1.3) (2.1.4) (2.1.5)

34. Who has access to the information in the state data system? (4.2.1)

• State officials only

• District officials

• Teachers

• Others  Please describe the sharing arrangement______________

35. How can [above mentioned individuals] access the data system? (4.2.1)

• Online access

• Request report from the state

• Other  Please describe the sharing arrangement______________

36. Does your state disaggregate Title III-funded LEP students from all LEP students in the state? (4.2.4)

• No

• Yes

37. Does your state data system have the ability to link different types of assessment results, for example, ELP assessment results of a student to math content test results? (4.2.4)

• No

• Yes

38. Can the state data system disaggregate achievement data by level of language proficiency? (4.2.4)

• No

• Yes

39. Can your state data system track former LEP students for more than 2 years after exiting programs?(4.2.5)

• No

• Yes

40. Can your state data system link data on teachers and their students? (4.2.2) (4.2.3)

• No

• Yes

41. Do you conduct any systematic analyses of teachers of LEP students? (4.2.5)

• No

• Yes [Please describe] _________________________________

42. To what extent do you believe you have appropriate resources to support Title III activities in the state? (financial, political, staffing) (4.1.1) (4.1.2)

• No  Why not? ________________________________________

• Yes Why? ________________________________________

43. How has your state handled implementation of the supplement not supplant provision of Title III? What guidance have you provided to districts on this issue? (4.1.1, 4.3.3)

43a. Has your state encountered any challenges in implementing this provision?

• No

• Yes [Please describe] _________________________________

43b. How has your state implemented the supplement not supplant provision of Title I with respect to meeting the needs (including linguistic needs) of ELL students in Title I schools?

• Have any challenges arisen?

Teacher Quality

44. Does your state have any state policies and/or initiatives to ensure mainstream classroom teachers receive professional development on LEP student issues? (4.5.6)

• Yes – state policies only  [Please describe] ____________________

• Yes – initiatives only  [Please describe] ____________________

• Yes – initiatives and state policies  [Please describe] ___________

Conclusion

45. Is there anything you’ve found to be particularly helpful in implementing Title III in your state? (1.1.5) (2.1.2)

46. Is there anything you’ve found to be particularly challenging in implementing Title III in your state? (1.1.5) (2.1.2)

47. Is there anything you’ve found to be especially helpful or challenging in addressing the needs of ELL issues in your state? (1.1.5) (2.1.2)

48. I know I’ve asked you a lot of questions. Is there anything else you’d like to add?

1 The study is formally known as the Evaluation of State and Local Implementation of Title III Standards, Assessments, and Accountability Systems.

C-16