ED Response to OMB Qs

1. The supporting statement describes how the new electronic system will make managing the data easier for ED.  Please explain how the revised APR makes reporting easier for the reporting entities.

The revised system will be significantly easier for the reporting entities in several ways: (1) Currently, the reporting entities are required to provide information in several different formats. They provide descriptive information on the OMB-approved form that is currently used; they provide a narrative summary of the progress they are making towards accomplishing the goals and objectives of the project; and, they are asked to provide a copy of an evaluator’s report, if there is one available. The projects have been somewhat confused about how much information to provide and in what format, and this on-line form will simplify that. The proposed form describes how much information to provide and pulls together all of the information in one format. (2) The proposed APR system will allow the State coordinators to receive and review the projects’ APRs before they are submitted to ED. This step will make it easier for the projects because the State coordinators will be able to verify the information in the APR form before it is submitted and then the State coordinator will send it on to ED. For example, entities are sometimes confused about the fiscal year from which their funding comes. The APR, web-based form should eliminate some of the back and forth with the projects that have been necessary up to this point because missing or inaccurate data. (3) The MSP program office has worked with the MSP State coordinators to develop this form. A national meeting of State directors was held before the APR form was submitted for OMB clearance. The State coordinators had a significant role in helping to develop and refine the instrument. They said that the revised APR simplifies the reporting process for their projects. In addition, the MSP program shared the draft APR with the reporting entities in a series of regional meetings around the country over the past three months, attended by about 600 people in all. The feedback was very positive, and no one expressed concern about the burden, but rather participants were pleased with the format and process for reporting in the revised APR and expressed that this would be easier for them to complete.

2. Why is ED changing the APR now (in what appears to be the third year of data collection) for a three-year grant program?  Why not wait a year and start anew?

The MSP program is a formula grant program that provides funds to the States. The States, in turn, are required to make competitive awards to projects within their State. The State-funded projects may be awarded for up to three years. These local projects are required to report on an annual basis to ED. As with most things, the States have taken very different approaches to how they disperse these funds. Some States have made one-year awards and have held competitions each year and made new awards. Other States have made 18 month or two-year awards that are renewable if the State is happy with their work. Other States have made three-year awards, but because they have received increased funding every year of this program, they have projects in various stages of implementation. Some States made their awards quickly upon receiving their appropriation in July of each year, other States wait until the end of the eligible period (17 months later) to make their awards for these funds. Given this diversity in the timing and scope of the projects, there is no pattern that would apply to the majority of projects, this is as good a time as any to implement a new data collection form.

3. For section D (on teachers who showed significant gains in content knowledge), ED will send a separate Excel spreadsheet and have the respondent enter the information into the spreadsheet and then upload into the system. Why not embed this functionality into the system rather than require the respondent to take a separate step outside of the system? Also, the Excel spreadsheet should be part of the submitted documents - please send to us now.

The main reason we chose not to embed the spreadsheet to determine significant gains in content knowledge into the online MSP APR system is that most projects will need to run the spreadsheet multiple times (once for each separate test they administer). If the spreadsheet were to be embedded, we anticipated that running and re-running the spreadsheet would have made the online system overly cumbersome. A second reason is we sensed reluctance among grantees to provide the actual teacher test scores to outside organizations, including ED. The APR asks grantees to report the number of teachers who made significant gains, and not their test scores. If the spreadsheet was embedded, grantees may be reluctant to enter the test score data, increasing the amount of missing data for GPRA

4. The supporting statement (#5) states that small entities will not be affected because the respondents are all LEAs.  In certain circumstances (e.g., a small rural school district), LEAs can be considered small entities.  Please revise the supporting statement to describe the steps you are taking to alleviate burden on small entities and resubmit to us.

MSP requires that there be partnerships between institutions of higher education and local education agencies. By statute, these are never stand-alone projects. Other entities, such as non-profit or for-profit organizations, intermediate education agencies, etc. may also be a part of the partnerships. Therefore, small, rural districts with limited administrative capacity are supported by the other organizations in the partnership. In most cases with rural projects, the IHEs or other entities handle the APR reporting.

In addition, the staff in the State educational agency responsible for managing the program provides support, and is responsible for making sure that the local projects have the resources they need to manage their projects. For example, in one largely rural State, the SEA used some if its administrative funds to support a statewide evaluation design so that smaller, more rural projects wouldn’t have to use their resources for this purpose. The SEA staff also review the APR before it is submitted to ED so that any problems with putting the APR information together can be handled within the State.

5. ED submitted major revisions to this data collection after submitting for OMB review. Please explain why these revisions are necessary and what steps you have taken to solicit public feedback on the revisions.

We apologize for submitting major changes to the data collection so late in the process. The MSP program office only recently began working with the Data Quality Initiative (DQI), which is a Budget Service and IES effort to improve the quality of program performance data for targeted ED programs. OMB asked the Department to create an initiative to improve the data quality of programs in ED and DQI is a result of that effort. MSP is the first program in ED with which the DQI is working. The DQI needs assessment for MSP, conducted in late fall, revealed: 1) Some of the MSP GPRA indicators were out of date and had to be changed to more accurately reflect the activities supported by the States with these funds. The DQI process allowed the conversations to take place with Budget Service and OMB to revise the GPRA indicators for this program. For example, several of the original indicators were based on the need to help teachers become “highly-qualified” as defined in NCLB. According to law, however, all teachers should be “highly qualified” by 2006 so this issue no longer applies. 2) The data requests needed to inform the revised GPRA indicators had not been included in the APR that was in the paperwork clearance process. There was agreement that both of these issues needed to be revised. The DQI staff worked with the MSP program office and the Budget Service to revise the GPRA indicators, which were subsequently approved by OMB. Then, DQI staff developed the APR sections to collect the GPRA information.

The MSP program is holding a series of regional meetings with grant recipients. One of the regional meetings was held in December, 2006 in DC and focused on program evaluation. DQI staff were invited to conduct a series of workshops at this meeting to discuss the proposed changes. The three sessions were attended by a total of about 125 people (with about 45 people in each session). During the sessions, the new APR sections were presented as “proposals,” and the presenters explicitly asked for and received feedback about them. Almost all of the feedback was positive and constructive, and we made several changes to the new APR sections based on the feedback. At that point, we revised the APR instrument and the supporting statement and resent the package.

6. Please explain how you developed the 14-hour burden estimate.  Doesn't this estimate need to be adjusted per the newest revisions.

This is a good point. The 14-hour burden was arrived at by estimating the amount of time it has taken projects to prepare summaries of the progress they made in carrying out the goals and objectives of the projects. This estimate came from the projects that completed their annual reports in the first year, calibrated to the demands on an on-line system. In the many feedback sessions we held at the regional meetings, participants felt that the revised APR tool will simplify the reporting process. However, we didn’t revise the estimate of the burden for completing the APR to include the proposed revisions. Staff for the Data Quality Initiative (DQI) have “field tested” this data tool and estimate that it will take each project an additional 15 minutes to enter the necessary information into the tool. We expect about 500 projects to submit APRs, therefore the overall burden should be changed to add 125 hours per year.

7. Please explain why this data cannot be collected via the Eden survey tool (especially since Eden/ EDFacts have recently been expanded to allow LEA reporting in certain circumstances).

The MSP program office met with EDFacts staff in March of 2006 during the 2006-2007 EDEN data definition process, and at that time it was determined that the MSP APR information was not “EDENable.” We followed up with EDFacts staff in January 2007 in response to this question and got additional information. Currently, EDFacts is not set up to acquire data from consortia, and most of the MSP grantees are consortia of institutions of higher education and multiple local educational agencies.

8. Question 28 asks respondents to report on years for which they were funded and the initial award date.  Don't you already have that data?  If so, why are you again requesting it from the respondent?

 

The States make the awards for these projects, not ED, so it is important for ED to be able to identify the start date and the fiscal year from which the projects were funded. The States make awards at different times during the year, and make the awards for various lengths of time. The APR allows us to confirm and verify the information about the period of time the project is reporting on, and from which fiscal year it was funded.