Evaluation for Growth Model Piolet Program

Interview #: ________________

Transcript #: ________________

Audiofile #: ________________

Interviewer: ________________

Date of interview: __________

FINAL DRAFT 19June08

Revised 1July08

Evaluation of the Growth Model Pilot Program (GMPP)

under the No Child Left Behind Act of 2001

NORC Project Number 6489.08.62

In-Depth Interview Protocol

(Task 8)

Section Page

• Instructions to interviewer i

• Preliminary information 1

• About the interviewee 2

• Implementation 3

• Comparison to other state growth models 5

• Reporting 6

• Results 10

• Use & impacts 11

• Lessons learned 13

• Recommendations re additional contacts 14

Instructions to the interviewer

This interview is being conducted as part of an evaluation of the Growth Model Pilot Program (GMPP) under the No Child Left Behind Act of 2001 (NCLB) that is being conducted by NORC at the University of Chicago under contract to the U.S. Department of Education (ED).

About the pilot program

The Department initiated the GMPP in November 2005 with the goal of approving up to ten states to use growth models for determining adequate yearly progress (AYP) under NCLB; in December 2007, ED moved to expand participation in the growth model program and extended an invitation to all states to apply for inclusion.

About growth models

Growth models are defined as alternatives to the standard status model. The status model bases AYP on the proportion of a school’s students attaining proficiency in reading and mathematics. Growth models, in contrast, base AYP on some measure of how much students have gained from one year to the next. Growth models promise to provide better measures of school effectiveness and progress toward proficiency goals. The main objective of the GMPP is to develop and implement models for determining school-level AYP that incorporate measures of student growth.

About this evaluation

The experiences of nine states approved for participation in the pilot program are being assessed in this evaluation: Alaska, Arizona, Arkansas, Delaware, Florida, Iowa, North Carolina, Ohio, and Tennessee. The overall goal is to provide evidence on the strengths and weaknesses of growth models for tracking student achievement and school progress under NCLB. The study will also gather information on states’ experiences implementing approved growth models and on how model results are used and reported to stakeholders. These basic objectives are pursued in the context of each of the pilot states; cross-state comparisons will not be pursued.

About this interview

This is one of a small number of personal (telephone or face-to-face) interviews being conducted with state education agency (SEA) officials and particularly knowledgeable individuals SEA officials have identified who are well-placed to answer questions regarding each state’s experiences (a) implementing the growth model proposed and approved for use in the pilot program, and (b) using and reporting information on student proficiency and school AYP obtained from the application of the growth model. This in-depth interview protocol identifies key topics to be explored with interviewees under each of these headings. While we expect many interviewees will have less to say on some topics than others, and recognize that the semi-structured discussion you are about to have with the interviewee may involve deviations from the question order suggested below, it is important to raise all the topics included in the following in-depth interview protocol during the course of your discussion. Skip patterns are included to assist in focusing the interview on those topics with which the interviewee has the most experience and expertise. Guidelines are also included to assist you in probing for additional information on topics of primary importance to this evaluation. Throughout your discussion, it will be helpful to remember the information obtained from this interview will be used to answer the following study questions which guide this portion of the evaluation:

How have states in the pilot program implemented growth models?

Have growth models been implemented in accord with the plan proposed to the U.S. Department of Education?

What technical, administrative, and resource challenges did the State face in implementing its model? How and why did challenges differ by state?

How have pilot states used information from growth models, and what has been the local response?

How have states reported and used the results from growth models and how does this compare with information generated by status models? by State accountability models?

Have results been used for sanctions, incentives, and/or support for districts and schools to raise the achievement of the lowest performing schools and the lowest performing students? If so, what steps have districts and schools taken based on results?

Has the growth model resulted in new information for districts, principals, teachers and parents? Do these stakeholder view reporting from growth models as useful? Do their views differ depending on ratings of school performance?

Preparing for the interview

Before the interview, NORC’s Associate Project Director for this evaluation and a member of the analytic team will meet separately with the team of interviewers assigned to speak with interviewees from each pilot grantee state. At this meeting you will be provided with a briefing on key features of that state’s growth model that will assist you in refining specific probes regarding the implementation, use, and reporting of information from the growth model under NCLB in that state.

Interviewees are being asked to allow these interviews to be audio-recorded. NORC’s policies for the secure handling and storage of these audio files will be explained at the pre-interview briefing session, as will procedures for checking-out and returning audio equipment and interview materials. Your interview appointments will be scheduled for and confirmed with you and your interviewee in advance. If an emergency arises, and you are unable to conduct a scheduled interview, please notify the Associate Project Director and Task 8 Team Leader (Kevin Brown, (773) 256 6024) or the Project Director (Sarah-Kay McDonald, (773) 256 6199). Any additional questions you may have regarding the interview process or your role in this project should also be addressed to them.

The information you obtain from these interviews is extremely important to this evaluation. We appreciate your efforts to ensure the interviewees address each topic to the best of their ability, and understand how much we value their time and participation in this study.

Thank you for your contribution to this project.

I. Preliminary information

Thank you for speaking with me (us) today. My (Our) name(s) is (are) [INSERT]. I’m (We're) part of a research team working with NORC at the University of Chicago under contract to the U.S. Department of Education. We’re evaluating the Growth Model Pilot Program under the No Child Left Behind Act of 2001 (NCLB). As part of our evaluation, we are talking with a small number of people in each state who have been closely involved with the pilot program. We’d like to learn more about the implementation of [INSERT NAME OF STATE]’s growth model, and how the information from the growth model has been used to date. We are particularly interested to learn what is going well, and if any unanticipated challenges have arisen.

We expect this interview will take about an hour and a quarter of your time. The information we obtain from these interviews will be included in reports to the U.S. Department of Education. In those reports we will not attribute quotes to you without prior permission. However your answers will be identifiable as pertaining to your state and your organization, and because you will be one of a small number of individuals being interviewed as part of this evaluation, it is possible that you may be identifiable as the source of information included in our reports.

Finally, although every effort will be made to keep research records private, there may be times when federal or state law requires the disclosure of such records, including personal information. This is very unlikely, but if disclosure is ever required, NORC will take all steps allowable by law to protect the privacy of personal information.

I would like to audio-record this interview; this is very helpful in ensuring the accuracy of reports we will prepare using this data.

Before we go any further, would you please confirm you are willing to participate in this study? And do I have your permission to tape-record this interview?

IF INTERVIEWEE HAS GIVEN PERMISSION TO RECORD:

Turn on recorder & MAKE SURE IT IS RECORDING

Thank you again for participating in this interview. For the record, I wanted to confirm that I have turned on the recorder and to ask you to confirm (a) that I have explained the purposes of this interview to you and how this information will be used, and (b) that I have your consent to record our conversation. [Pause for interviewee to confirm consent to recording.] Thank you.

II. About the interviewee

Q1. To start, could you tell me a bit about the nature of your role in [INSERT NAME OF INTERVIEWEE’S ORGANIZATION]?

[PROBE:]

Q1a. How long have you been working with [INSERT NAME OF INTERVIEWEE’S ORGANIZATION]?

Q1b. What is your current position?

Q2. What are your roles and responsibilities for the Growth Model Pilot Program under the No Child Left Behind Act of 2001?

Q3. Are you familiar with efforts to implement the pilot growth model in your state?

□ Yes  CONTINUE with Q4

□ No  SKIP to Section IV

III. Implementation

Q4. Pilot states are at different points in their implementation of the growth model they are using for NCLB. Where is [STATE] in the process of implementing and reporting on its NCLB growth model?

□ Have not yet implemented the model

□ Have implemented the model; use the growth model to analyze and report AYP data …

□ … for one year

□ … for two years

□ … for three years

Q5. Is this on-track with where you expected [STATE] to be at this point?

□ Yes [PROBE:]

Q5a. What has helped you stay on-track?

 CONTINUE with Q6

□ No [PROBE:]

Q5b. To what extent has [STATE] had to vary from your original implementation plan?

Q5c. Do you have a sense of why this is the case? What led to the changes from the original schedule?

 CONTINUE with Q6

Q6. Have there been any surprises along the way? [PROBE:] Did [STATE] face any challenges?

□ Yes

Q6a. What kinds of challenges did you face? [PROBE:] Were any of the challenges associated with:

Q6a1. Resources? (e.g., financial,

staffing/expertise?)

Q6a2. Organizational constraints? (e.g., legal

restrictions, workplace culture?)

Q6a3. Technical limitations (e.g., obtaining data,

matching students, adequate scales?)

Q6a4. Local buy-in (e.g., from districts, school boards, principals, teachers, the business community?)

Q6b. How did you address them?

Q6c. From your perspective, do you expect these issues will have any effect on the results of [STATE’s] growth model pilot?

[PROBE:] Could you tell me more about that?

□ No  CONTINUE with Q7

Q7. Has [STATE] made any changes from its original growth model pilot plan in how growth is calculated or AYP is determined?

□ Yes

Q7a. Could you tell me more about that? What kinds of changes has [STATE] had to make?

Q7b. What made these changes necessary?

Q7c. How – if at all – are these changes likely to affect [STATE’s] AYP results, or how they are used?

□ No  CONTINUE with Q8

Q8. Does [STATE] plan to continue using a growth model to calculate AYP?

□ Yes

Q8a. What changes will you make, if any?  CONTINUE with Q9

□ No

Q8b. Could you tell me more about that?  SKIP TO Q10.

Q9. From your perspective, do you anticipate [STATE] will face any challenges in sustaining the program?

□ Yes

Q9a. What kinds of challenges do you anticipate? [PROBE:] Do you anticipate any challenges associated with:

Q9a1. Resources? (e.g., financial,

staffing/expertise?)

Q9a2. Organizational constraints? (e.g., changes in contractors?)

Q9a3. Local buy-in (e.g., resistance from policymakers, schools, teachers?)

□ No  CONTINUE with Section IV

IV. Comparison to other state growth models

[NOTE: Please check the following list to determine whether the interviewee’s state employs a growth model for state accountability purposes. If so, please cover the topics in this section with the interviewee. If not, please skip to section VI.]

□ Alaska → Continue with Q10

□ Arizona → Continue with Q10

□ Arkansas → Continue with Q10

□ Delaware → Skip to Section V

□ Florida → Continue with Q10

□ Iowa → Skip to Section V

□ North Carolina → Continue with Q10

□ Ohio → Continue with Q10

□ Tennessee → Continue with Q10

Q10. Some states participating in the Growth Model Pilot Program also use growth models for state accountability purposes. I understand [STATE] has such a model, [INSERT NAME OF MODEL]. Are you familiar with this model?

□ Yes  CONTINUE with Q11

□ No  SKIP TO SECTION V

Q11. How does [STATE’s] NCLB growth model compare to the state accountability growth model? [PROBE:]

Q11a. Do the two models use different student or test data?

Q11b. Do the technical specifications of the two models differ? (e.g. controls for

student or school characteristics)

Q11c. Do the models set different expectations for student growth?

Q11d. Do the models generate similar results in terms of identifying high-growth schools? [PROBE:] How do you account for the similarity or difference in results (e.g. technical features)?

Q12. How are the data from the two models used? [PROBE:]

Q12a. Which model is used to report student progress to teachers? To parents?

Q12b. Which model is used to allocate resources for school improvement?

Q13. Earlier I asked if you are aware of any problems that may have arisen in implementing [STATE’s] NCLB growth model. Do you know if any problems like this have ever occurred in connection with the implementation of the [INSERT NAME OF STATE GROWTH MODEL]?

□ Yes [PROBE:] Could you tell me more about that?

Q13a What kind(s) of challenges arose?

Q13b Were these similar to or different to any challenges [STATE] encountered in implementing the NCLB growth model?

□ No  CONTINUE with Q14

V. Reporting

Q14. Are you familiar with how results from the NCLB growth model are or will be reported in your state?

□ Yes  CONTINUE with Q15

□ No  SKIP to Section VI

Q15. Have any results from [STATE’s] NCLB growth model been reported to date?

□ Yes

Q15a. For which years have results been reported?

Q15b. Which month are growth results reported?

Q15c. Are growth results reported at the same time as status AYP results?

□ No  SKIP to Q23

Q16. I’d like to get a sense of what results from the NCLB growth model have been reported, how they have been reported, and who they have been reported to.

Q16a. Could you tell me about information reported by the state education agency? [IF YES, CONTINUE WITH QUESTIONS IN TABLE BELOW]

Q16b. Could you tell me about information reported by school districts? [IF YES, CONTINUE WITH QUESTIONS IN TABLE BELOW]

Q16c. Could you tell me about information reported by schools? [IF YES, CONTINUE WITH QUESTIONS IN TABLE BELOW]

Q16d. Could you tell me about information reported by other organizations? [IF YES, CONTINUE WITH QUESTIONS IN TABLE BELOW]

1	2	3
a. What information about the growth model has been reported? [PROBE: Do reports indicate □ which schools, districts, or subgroups made AYP under growth? □ how growth results varied by school/district? □ the progress of individual students (by teacher)?] b. How is the difference between status and status-plus-growth reported? c. Do reports indicate that a new method of determining AYP is being used?	How was the information being reported? [PROBE: In □ published report or brochure? □ written correspondence? □ oral presentations? □ online? IF YES, PLEASE ASK IF INTERVIEWEE CAN PROVIDE MATERIALS OR LOCATION]	To whom was the information reported? [PROBE: □ federal government? □ school districts? □ schools? □ teachers? □ parents? □ other (who)? IF YES, PLEASE ASK FOR DETAILS OF THE INFORMATION PROVIDED]

Q17. I’d like to understand how the reporting on [STATE’S] NCLB growth model data compares to the reporting on [STATE’S] status model data. What are the key differences between how information on student proficiency or school AYP is reported for each model? [PROBE:] Can you tell me more about this? Are there differences in who receives the information about each model?

Q18. We understand that [STATE’s] original plans for reporting were described in [INSERT SOURCE OF DATA PROVIDED IN PRE-INTERVIEW BRIEFING FROM Task 7]. Have those original plans changed?

□ No  SKIP to Q20

□ Don’t know  Who else do you suggest we

contact about this?

□ Yes  CONTINUE with Q19

Q19. Can you tell me about those changes? [PROBE:]

Q19a. What changed?

Q19b. What led to those changes?

Q19c. Is there a document that describes the current reporting plan?

□ No  CONTINUE with Q20

□ Don’t know  Is there anyone else you suggest we

contact about this?

□ Yes  Could you send us a copy?

Q20. Now let’s talk about how the reporting on your NCLB growth model data compares with the reporting on [NAME OF STATE ACCOUNTABILITY SYSTEM]. What are the key differences between how information on student proficiency or school AYP are reported for each model? Can you tell me more about this? Are there differences in who receives the information from each model?

Q21. What has been the reaction to the growth model information [STATE] has reported to date? [PROBE:]

Q21a. Did people have any questions about the NCLB growth model? [PROBE re who had what types of questions – e.g., SEA officials, district officials, schools, parents, other members of the public]

Q21b. Did they perceive any benefits to the NCLB growth model? [PROBE re who perceived what types of benefits]

Q21c. Have any concerns been expressed? [PROBE re who expressed what types of concerns]

Q21d. Are you aware of any confusion or misperceptions related to the information arising from the NCLB growth model? [PROBE re any confusions or misperceptions encountered, and how these may have been addressed]

Q22. Going forward, are you aware of any changes that will be made in how your state reports information from the NCLB growth model? [PROBE: Can you describe those changes and the thinking behind them?]  Please SKIP to Section VI

Q23. I’d like to understand what information [STATE] plans to report on your NCLB growth model, how it will be reported, and to whom.

Q23a. Could you tell me about information to be provided by the state education agency? [IF YES, CONTINUE WITH QUESTIONS IN TABLE BELOW]

Q23b. Could you tell me about information to be provided by school districts? [IF YES, CONTINUE WITH QUESTIONS IN TABLE BELOW]

Q23c. Could you tell me about information to be provided by schools? [IF YES, CONTINUE WITH QUESTIONS IN TABLE BELOW]

Q23d. Could you tell me about information to be provided by other organizations? [IF YES, CONTINUE WITH QUESTIONS IN TABLE BELOW]

1	2	3
a. What information about the growth model will be reported? [PROBE: Will reports indicate □ which schools, districts, or subgroups made AYP under growth? □ how growth results varied by school/district? □ the progress of individual students (by teacher)?] b. How will the difference between status and status-plus-growth be reported? c. Will reports indicate that a new method of determining AYP is being used?	How will the information be reported? [PROBE: In □ published report or brochure? □ written correspondence? □ oral presentations? □ online? IF YES, PLEASE ASK IF INTERVIEWEE CAN PROVIDE MATERIALS OR LOCATION]	To whom will the information be reported? [PROBE: □ federal government? □ school districts? □ schools? □ teachers? □ parents? □ other (who)? IF YES, PLEASE ASK FOR DETAILS OF THE INFORMATION PROVIDED]

Q24. Are these plans any different from the way information on student proficiency or school AYP based on status models are currently reported? [IF “YES,” PROBE:] Can you tell me more about this? How will these two kinds of information reported? What will be different about the way the results from the NCLB growth model are reported?

Q25. Are these plans any different from the way information on student proficiency or school AYP based on state accountability systems are currently reported? [IF “YES,” PROBE:] Can you tell me more about this? How will these two kinds of information be reported? What will be different about the way the results from the NCLB growth model are reported?

Q26. We understand that [STATE’s] original plans for reporting were described in [INSERT SOURCE OF DATA PROVIDED IN PRE-INTERVIEW BRIEFING FROM Task 7]. Have those original plans changed?

□ No  SKIP to Section VI

□ Don’t know  Who else you suggest we

contact about this?

□ Yes  CONTINUE with Q27

Q27. Can you tell me about those changes? [PROBE:]

Q27a. What is changing?

Q27b. What led to those changes?

Q27c. Is there a document that describes the current reporting plan?

□ No  CONTINUE with Section VI

□ Don’t know  Who else you suggest we

contact about this?

□ Yes  Could you send us a copy?

VI. Results

Q28. From your perspective, has [STATE] benefited from participating in the Growth Model Pilot Program? [PROBE:] Could you tell me more about that? How has the state benefited from participating in this program?

Q29. From your perspective, are there any drawbacks associated with [STATE’s] participation in the Growth Model Pilot Program? [PROBE:] Could you tell me more about that? What sorts of drawbacks are associated with the state’s participation in this program?

Q30. From your perspective, are there any changes to the state’s NCLB growth model, or to the pilot program itself, that would be helpful to the state, or enhance the program? [PROBE:] Are there any factors, internal or external to the growth model, that may have prevented high-growth schools from making AYP? That may have allowed low-growth schools to make AYP? How might these issues be addressed in future growth models?

Q31. Does [STATE] have plans to evaluate its NCLB growth model? [PROBE:] What is (was) the timeline for this evaluation? What were the outcomes/conclusions? How will they shape any modifications to the model? May we have a copy of the report?

VII. Use and Impact

Q32. Are you familiar with how growth model results have been or will be used to inform state policy or school improvement plans?

□ Yes  CONTINUE with Q33

□ No  SKIP to Section VIII

Q33. Have growth model results been used (or will they be used) any differently than [STATE] uses status-only results?

□ Yes  CONTINUE with Q34

□ No  SKIP to Section VIII after: [PROBE:] Why not?

Q34. How is (or how will be) NCLB growth model information used at the state level? [PROBE:]

□ To better identify what districts or schools need assistance?

□ To better identify what type(s) of assistance might be most useful?

□ For district or school corrective actions?

□ To target incentives?

□ To target supports? (e.g. funds for school improvement or teacher professional development)

□ For some other purpose?

Q35. How is (or how will be) NCLB growth model information used at the district level? [PROBE:]

□ To better identify what schools, students, or classrooms need assistance?

□ To better identify what type(s) of assistance might be most useful?

□ For school corrective actions?

□ To target incentives?

□ To target supports? (e.g. funds for school improvement or teacher professional development)

□ For some other purpose?

Q36. How is (or how will be) NCLB growth model information used by individual schools? [PROBE:]

□ To better identify what students or classrooms need assistance?

□ To better identify what type(s) of assistance might be most useful?

□ For teacher removal or incentives?

□ To target supports? (e.g. funds for teacher professional development)

□ For some other purpose?

[IN SECTION V, DID INTERVIEWEE INDICATE THAT THIS STATE HAD ALREADY BEGUN REPORTING RESULTS FROM ITS NCLB GROWTH MODEL?]

□ Yes  CONTINUE with Q37

□ No  SKIP to Section VIII

Q37. Earlier you indicated [STATE] had already begun reporting results from its NCLB growth model. At this point, does it seem that schools and districts are using the NCLB growth model information in the ways the state had envisioned, or are they using it in some other way? [PROBE:] Can you tell me more about that? How are schools and districts using this information?

Q38. From your perspective, are there any other ways districts or schools could be using the information resulting from [STATE’s] NCLB growth model? [PROBE:] Could you tell me more about that? How else could schools and districts use this information? Could growth and status results be used as separate accountability standards?

VIII. Lessons Learned

Q39. From your perspective, is there anything you think it would have been desirable for [STATE] to do differently in planning, implementing, or reporting information from its NCLB growth model? [PROBE:]

Q39a. Would you change the decision to participate in the Growth Model Pilot Program?

Q39b. Would you recommend a change in the way growth is calculated?

Q39c. Would you recommend any changes in the way the current NCLB model is implemented?

Q39d. Would you recommend any changes in the way information resulting from the NCLB growth model is reported to districts? Schools? Teachers? Parents? Any other groups or individuals with an interest in student or school performance?

Q40. What advice would you offer to someone in your position in another state considering using an NCLB growth model? [PROBE:] Do you have any advice to offer with respect to how growth models are implemented? How information from growth models is reported? How information from growth models is used?

Q41. As you may know, the U.S. Department of Education is developing regulations to allow all states to apply to amend their accountability workbooks to include an approved growth model. Based on your state’s experience, what lessons have you learned that you think it would be important for the Department to consider as growth model regulations are developed?

IX. Recommendations regarding other individuals to contact re other aspects of the evaluation study

Q42. Are there any other individuals you recommend we contact for input on the issues we have been discussing? These may be from your agency, contractors, or others who may have different experience or a different point of view. [PROBE for names, positions, contact information]

Thank you. We may not get in touch with everyone on this list, but if we do contact some of these individuals, may we explain that you suggested it would be helpful for us to contact them to learn more about the issues of implementation, reporting, and use that we have been discussing?

□ Yes

□ No

Those are all the questions we wanted to ask. Is there anything else you would like to add on these issues of implementation, reporting, and use that we have been discussing?

Thank you for taking the time to speak with me today. Your insights are extremely helpful, and we greatly appreciate your thoughtfulness and your time.

OMB Number ########

Expiration date: ## / ## / ####