OMB Memo requiring changes to case studies

Memorandum

To: Nicole Cafarella

From: Daphne Kaplan

cc: Kathy Axt, Katrina Ingalls

Subject: National Technology Activities Task Order (OMB Clearance

Number: 1875-0241) OMB Change Request

Date: August 10, 2007

We are requesting permission to submit a change worksheet and expedited review for proposed modifications to the National Technology Activities Task Order (NTA) project data collection (OMB Control Number: 1875-0241). We are making this request based on findings from our spring 07 case study site visits, results we did not have when we submitted our most recent clearance package for a district survey. Based on those results, we would like to conduct a new set of site visits this fall which collect information from groups of school staff rather than individual teachers. We would then like to re-visit our original (spring 07) case study sites in the spring of 08 to collect information using this group data collection method. We would like to be able to include the fall site visit results in our interim study report. The changes proposed are necessary to inform the administration’s efforts to provide timely technical assistance to schools and districts to encourage the effective use of educational data systems. Information gathered through these case studies will help illuminate promising practices associated with effective data use, which will be used to inform outreach and technical assistance activities associated with the project. We are therefore requesting approval of these changes by August 27, 2007. More details follow below.

The study currently provides for 10 district case studies and three schools per district, for a total of 30 schools. As part of the school site visits, six teachers at each school are participating in an assessment scenario interview, designed to provide insight into teachers’ understanding of some of the basic assessment and statistical concepts needed to make sense of student data. As the data collection unfolded this spring, it became apparent that there are very large differences across teachers and that some teachers struggle even to “read” a data table or graph. An important difference across districts and schools is the extent to which data analysis and reflection are a group as opposed to an individual process. We have been asking individual teachers to respond to our assessment scenarios, and many of them make many misinterpretations. If teachers usually explore data in groups, it may be that groups arrive at sounder conclusions and consider more variables than an individual teacher does when working alone. Thus, our assessment scenario interviews may under-estimate the quality of the conclusions actually made from data in schools where this is a group activity.

The fall 2007 site visits will provide an opportunity to evaluate the differences in teacher responses in group versus individual settings, to better inform the spring 2008 visits. Since we would like to include the results from the fall site visits in the interim report (the first draft is due October 2007), these visits must begin no later than September 2007.

In addition, we have become aware that even though our 10 case study districts were nominated as leaders in the use of student data systems to inform instruction, their learning and implementation in this area are still underway. We expect that many of the sites will make significant progress between spring 2007 and spring 2008 and we would like to go back to these districts in the spring of 2008. We propose to expand our case study work by making a second visit to willing case study schools at which time we can (1) interview the school leader or onsite data coach to update the description of their activities and accomplishments and (2) re-administer the assessment scenarios to small groups of teachers. We would propose using two kinds of groups: (1) three teachers working together and (2) a group of two teachers and the principal or data coach. With permission, we would propose to videotape a subset of these interactions so that we could do a detailed examination of the collaboration process in addition to the estimation of the quality of data inferences derived. With this additional data, we could compare the quality of teachers thinking and conclusions about data (1) when working alone, (2) when working with other teachers, and (3) when working with another teacher and a coach. These findings would have direct implications for strategies of Data Driven Decision Making implementation.

The district representative for the 9 districts visited during spring 2007 will be contacted via phone to discuss the willingness of the district and 3 schools to participate in another round of data collection in spring 2008. SRI staff will explain the reasons for this additional data collection activity and answer any questions that the respondent might have. We propose to make these calls in mid-August to determine how many districts will consent to the visits. (The site visit to the 10^th district in the original sample will be conducted in September 2007 using the revised data collection approach and, therefore, would not be revisited in spring 2008.) The number of districts included in the second round of site visits will determine how many additional districts will need to be recruited for site visits in fall 2007.

To obtain additional analytic power and anticipate potential attrition from the case study sample (i.e., the possibility that some schools will choose not to participate in the second data collection activity), we proposed adding up to five additional districts and three schools in each (total of 15 schools) which would have first site visits in fall 2007. These districts would be selected based on the same criteria utilized to obtain the initial 10 case study sites: districts already identified through polling of leaders in educational technology and TWG members, and interactions with district leaders active in data-driven decision making through the study’s various outreach activities. We propose to make these calls starting no later than August 20 so that there is sufficient time to initiate site visits by September 2007. For each district selected, project staff will contact the district representative by phone to explain the study, answer any questions, and to gain recommendations for 3 site visit schools at either the elementary or middle school leve1 based on the following criteria:

1. One school that the district considers exemplary in its data use policies.

2. One school that has shown dramatic improvement in its use of data to improve instruction and student outcomes.

3. One school that is typical of the district with respect to use of data systems.

To the extent possible, each district will be asked to recommend schools that serve demographically similar student populations at the same grade levels. Priority will also be given to schools serving large numbers of low-income student and schools that have experienced improved student achievement. Project staff will work with the district representative to establish contact with school staff and coordinate the site visit schedule (as soon as possible after the initial contact).

The original and proposed school data collections are summarized in Exhibit 1 below.

Exhibit 1
Original and Proposed Sample Sizes for Site Visit Data Collections

	Original Sample				Additional Sample
	LEAs	Schools	Teachers	Principals	LEAs	Schools	Teachers+	Principals
First Interviews	10	30	180	30	5	15	120	15
District staff (3/LEA)					15
Solitary Data Interpretations (3 tchrs/school)		30	180				45
Small-Group Data Interpretation (5 tchrs & coach/school)							90
Follow-up Interviews*					7	21	126	21
Solitary Data Interpretations (3 tchrs/school)						21	63
Small-Group Data Interpretation (2 tchrs & coach/school)						21	63

Note: We were unable to schedule one district site visit prior to the end of the 2006-07 school year so this visit will be made in the fall along with those to the 5 districts proposed here as additions to the sample.

*Follow-up visits would be conducted with all schools in our original sample wiling to host a second visit; we have assumed that 7 districts (and 21 schools) would agree.

+ Includes data coach.

The estimates in Exhibit 2 reflect the burden for both the original set of case studies and district survey data collection activity previously approved, as well as the additional case study data collection activity for which we are seeking approval. Based on the information in Exhibit 1, 360 respondents and 292.5 hours of burden will be added through the additional case study activity. The additional estimated burden will be $11,700 based on a $40/hour salary cost.

• District personnel—time associated with reviewing study information, and if required, reviewing study proposals submitted to the district research committee, and preparing a list of schools that are active data users (e.g., marking up existing list of schools); time associated with asking questions about the study and answering questions about the district’s use of data systems.

• School personnel—time associated with asking questions about the study and time associated with answering interview questions and conducting data scenarios with teachers.

Exhibit 2
Estimated Burden for Study

Group	Participants	Prior Total No.	New Total No.	No. of Hours per Participant	Prior Total No. of Hours	New Total No. of Hours	New Estimated Burden
District Personnel (survey)	Superintendent (notification) District staff (survey)	534 500*	534 500*	0.5 1.0	267 500	267 500	$10,680 20,000
Subtotal		1034	1034		767	767
District Personnel (case studies)	Superintendent (notification) District staff (interviews)	10 30	22 45	0.5 1.0	5 30	11 45	$ 440 1,800
Subtotal		40	67		35	56
School Personnel (case studies)	School principal (notification) School principal (interview) Teachers (interviews & focus groups) Teachers (solitary interviews) Teachers (group)	30 30 360	66 66 360 108 153	0.5 1.0 1.0 1.0 1.5	15 30 360	33 66 240 108 229.5	$ 1,320 2,640 9,600 4,320 9,180
Subtotal		420	753		405	676.5
	Totals	1,494	1,854		1,207	1,499.5	$59,980

*Anticipate some non-response hence only 500 respondents to survey.

There is no change in start up costs resulting from the increase in school sample size. Additional costs to the federal government associated with increasing the sample size is estimated at $634,329 for costs associated with additional travel and increased labor time to collect, analyze and report additional data.