ED's response to OMB comments

 

200903-1875-001: Evaluation of the Teaching American History Grants Program: Data Collection Instruments

OMB Questions and Comments 07/24/09

With ED’s ANSWERS:

1.      The introduction of the evaluation describes the collection as being the fifth and final component of the study. Please provide OMB with the status and results of the first 4 components or if not yet complete a status update and analysis plan.

The Feasibility Study (Task 2) was completed in July of 2008 and recommended that the State Data Analysis (Task 3) be conducted with multi-year student history assessment data collected from the states that had responded positively to data access questions during the feasibility study research. The Department accepted the final recommendations of the Feasibility Study, choosing to exercise the State Data Analysis Optional Task. This task is currently underway, including two analysis designs a regression discontinuity design, and an interrupted time series design. A final report for this analysis will be produced in September of 2009.

State data collected for this Task 3 have also been used to compare grantee outcomes and select case studies for Task 4, which focus on program practices associated with positive student outcomes.

The Annual Performance Report Review (Task 6) was completed in April 2009 and focused on teacher outcomes of 2006 grants as reported in the 2008 APRs. The study team reviewed evaluation methods and outcomes of over one hundred 2006 grantees that had submitted APRs. The results of this review were the basis of the selection of the Task 5 Case Studies, which focus on identifying teacher practices associated with positive outcomes

A meta-analysis (Task 7) is underway and will be completed in September 2009. In order to prepare for the meta-analysis the study team reviewed the available, final grantee evaluation reports and identified reports that were suitable for inclusion in a meta-analysis. The purpose of the meta-analysis is to use grantee-level evaluations in order to estimate an effect of the Teaching American History (TAH) program on student learning and to also estimate the effects of certain components of the TAH project activities on student learning. The team will weight the studies in the meta-analysis according to: (1) the within-study sample size (i.e., large-sample studies will be weighted more heavily than small studies); 2) the reliability and validity of the learning assessments based on the available coefficients; and 3) study quality, in particular, weighting the studies in terms of aspects of design that facilitate causal inference (e.g., random assignment of units to conditions).

2.      Is participation in evaluation mentioned on TAH grant applications?

No, participation in the national evaluation was not mentioned in the grant announcement and was not mentioned in the TAH grant applications.

3.      Please cite the relevant confidentiality statute on page 8 of the supporting statement and align all confidentiality references appropriately.  RIMS can assist with this.

Responses to this data collection will be used to summarize findings in an aggregate manner (across all sites), or will be used to provide examples of program implementation in a manner that does not associate responses with a specific site or individual. Pseudonyms will be used for each site. The study team may refer to the generic title of an individual in their study report, but the site where that individual works will not be used. (That is, they may refer to a respondent as a “project director” but they will never use an individual’s name.) The study team will not provide information that identifies a subject or site to anyone outside of the study team, except as required by law.

The study team will disassociate names and addresses from the data as they are entered into the database and they will use the names and addresses for data collection purposes only. As the study team gathers information on individuals or sites, they will assign each a unique identification number, which will be used for raw data, print-out listings that display data and analysis files.

If the answer above is acceptable, we will revise Section 10 of Part A of the Supporting Statement and the draft correspondence that we are sending to selected case study sites accordingly. We can provide those to you at your request.

4.      How is the evaluation locating teachers that participate in TAH?

Research staff will ask grant directors, in advance of the site visits, to suggest a list of participating teachers who are diverse with respect to teaching experience, teaching levels and types of grant activities in which they have participated. Site visitors will work with the grant directors to determine the best method of contacting the teachers and inviting them to participate in interviews. The grant directors and their staff may choose to contact the teachers, or the site visitors may contact the teachers to schedule the interviews, as preferred.

5.      Page 13 of Section B discusses grouping of grantees for sampling by teacher performance separately from student performance. Will there be some utilization of teacher-student crosswalks within the case studies where data is available to examine teacher and student performance jointly?

We do not have, and will not be collecting, teacher-student linked data, so we will not be able to conduct an actual crosswalk. However, during the course of the case study interviews, we will be asking all case study grantees about any available evidence of both teacher and student outcomes. We will incorporate any such evidence into our qualitative analysis of the case studies, including the analysis of which program practices are associated with positive teacher and student outcomes.

6.      Outside of answering the research questions, is there any plan to make data available to the program staff? For example, on a grantee basis?

We will share results of the case study findings with the Teaching American History Program Office staff in a conference call arranged for this purpose. In addition, we will assure all grantees participating in the case studies that they will receive the final evaluation reports upon approval and release of the reports by the Department of Education. The evaluation team will email the final reports to these sixteen grantees as soon as Department approval is granted.