OMB Submission Part A

Supporting Statement for Paperwork Reduction Act Submissions

REL-NW Regional Needs Assessment Survey of Superintendents, Principals, and Teachers in a Five-State Area

OMB Control No: 1850-New, EDICS# 3237

Request for OMB Review – Part A

OMB Form 83-I and Supporting Statement for Data Collection

Submitted by:

Regional Educational Laboratory-Northwest

Portland, Oregon

January 2007

INTRODUCTION

The Regional Educational Laboratory Northwest (REL-Northwest) is requesting approval to collect information from a random sample of educators in the Pacific Northwest Region, including educators in the states of Alaska, Idaho, Montana, Oregon, and Washington. We are requesting a clearance to collect initial survey information in Year 2 of our contract (2007) and again in Year 4 (2009).

The goal of the initial needs assessment survey is to establish a road map by which to plan programs and set a meaningful research agenda to address state and regional educational needs. The goal of the follow-up survey in Year 4 is to assess similar questions from the same population, but with the express purpose of re-evaluating changes in the priorities and to plan future research and action. Therefore, the supporting statement below, describes the overarching process we will use for the project and details the survey planned for 2007, with reference to commonalities between it and the survey planned for Year 4 (2009).

A. Justification

A1. Rationale for Collecting Information

The proposed information collection is mandated in the Education Sciences Reform Act of 2002 (ESRA), Public Law 107-279, Section 174(f)(3) which states that “each regional educational laboratory awarded a contract under this section shall support applied research, development, wide dissemination, and technical assistance activities by developing a plan for identifying and serving the needs of the region by conducting a continuing survey of the educational needs, strengths, and weaknesses within the region.”

Further justification is found in Section 1.1—Regional Education Needs Analysis, Training and Technical Assistance Response Unit of the Institute for Education Sciences’ 2005 Statement of Work/Scope of Work for the Regional Educational Laboratory contracts (ED-05-R-0006) implementing the above quoted section of ESRA which states that “the contractor shall develop a Needs Analysis, Training and Technical Assistance Response Unit responsible for collecting information on the region’s needs for school improvement. The contractor shall assess regional needs regularly through surveys and contractor-developed outreach strategies to solicit comments from teachers and from district and state policymakers and administrators for pressing concerns that need attention.”

The proposed surveys described in this application are the strategy for soliciting comments from educators. Additional strategies other than the survey are being employed to obtain feedback from policy-makers and administrators.

And, from the REL Contract implementation (as stated in the “Regional needs survey in Years 2 & 4” section of the Revised Fast Response Plan, Task 1.1 – April 12, 2006: “REL-Northwest will conduct a regional needs assessment survey in Years 2 and 4 for a random, representative sample of the region’s educators – including teachers, principals, and superintendents – to provide a systematic, quantitative set of perceived needs for evidence that can be prioritized on the basis of their ratings.” This information will be used as a component of the needs analysis and the survey results will help to describe regional needs.

A2. Purpose and Use of Information Collected

This will be a new data collection. Respondents will be district superintendents, principals, and teachers from Alaska, Idaho, Montana, Oregon and Washington. The survey is designed to elicit information from Northwest educators regarding the key challenges their schools face in improving student learning and the types of research evidence that will assist them to best address these challenges. Surveying will take place in Years 2 and 4 to gather information to follow-up discussions with administrators and policy-makers held previously in each of the five states REL-Northwest serves.

REL-Northwest will use the findings to: 1) Identify issues which call for rigorous policy analyses that can guide states in developing sound educational policies to address the major challenges schools face, 2) Identify promising practices that can be evaluated to determine their impact on student learning and the extent to which they can be successfully adapted for diverse educational environments, and 3) Inform policy discussions among state and local education leaders and state policy makers. The information will also be used to help set the research agenda and program planning for the REL-Northwest.

A3. Use of Electronic Data Collection Techniques

The educators selected for the survey will be issued a printed survey and provided a link to a secure website should they wish to complete the survey electronically rather than on paper. The content of the survey instruments for either method of responding will be identical. Printed surveys will be serially numbered. Educators opting to complete the survey online will enter the serial number to access the survey database. Serial numbers will NOT be tracked to individual responders. Serial numbers will be used only to determine whether respondents completed both the online and the paper survey. In this case, only the first survey will received will be entered into the final database.

Parallel surveys for teachers, principals, and district superintendents have been developed. Time estimates were based on the amount of time it took respondents to complete a paper and pencil version of the study. Because the surveys are identical, it was estimated the online version will take about the same amount of time to complete, estimated to be between 15 and 20 minutes.

Giving respondents the option of completing the survey by mail or electronically via the web allows them to choose the method that will be most convenient for them to use and thus potentially lowers the burden and maximizes the potential for each respondent to complete the survey. Moreover, data collected electronically reduces the amount of time needed for data entry and minimizes error associated with data entry. The electronic version of the survey will have fillable fileable capability, meaning surveys can be completed online and submitted electronically.

We estimate between 25% and 50% of surveys completed will be completed online.

A4. Efforts to Identify Duplication

Secondary research conducted by REL-Northwest has revealed that minimal research has been done to accurately identify the information needs of Northwest educators and/or project these needs to the population of educators in this region.¹ Thus, we will not be duplicating information that federally funded programs are already reporting. We know of no other surveys that have asked the kinds of targeted questions that we are proposing.

A5. Minimizing Burden for Small Schools

Information will be collected from educational professionals, including teachers, principals, and district superintendents working in various school districts in a five-state region. We recognize that participating in a survey can place a burden on small schools and districts because of few personnel and many demands on time. Since we are using a random sampling technique the impact on teaching staff will be proportionate to school size, so as not to place more proportionately more burden on staff of smaller schools.

We have designed the surveys to reduce respondent burden by developing a parsimonious questionnaire focused only on the essential information needed, and easy-to-use questions that ask respondents to prioritize, by assigning “points,” to four items within each of 11 topic areas. We have also limited open-ended questions to short answers to reduce respondent burden and yet allow for information or ideas the educators have that are not covered by the questions in the 11 topic areas.

A6. Consequences of Not Collecting the Information as Proposed

If the proposed information is not collected it will be difficult, if not impossible, to determine the priorities Northwest educators have for evidentiary needs and thus difficult to set a research agenda for developing information targeted most effectively to improving educational programs that will ultimately improve student achievement. In addition, it is important to survey each group of educators—teachers principals, and superintendents—to achieve a balanced picture of priorities and needs.

A7. Special Circumstances

No special circumstances exist for this data collection.

A8. Soliciting Public Comments

The Institute for Education Sciences (IES) placed the announcement concerning this information collection in the Federal Register for public comments. Public comments on the design of the collection, the survey questions used, and the collection process were solicited; and no public comments have been received.

There were hearings with educators and education policy makers in each of the five states before the survey was designed. Educators were encouraged to share and discuss their needs and experiences on a variety of educational topics unique to their state. The hearings were not designed to be Q & A between participants and moderators as is normal in a focus group setting, nor were structured questions used as is typical in a survey project. A review of notes from the five hearings revealed patterns of information common to all states REL-Northwest serves. These patterns were used to develop the initial set of items that were then cognitively tested.

A9. Payment or Gifts to Respondents

Educators selected for the survey will receive a $5 incentive at the time they receive the mail survey. According to Don Dillman, a nationally recognized expert in the field of mail and web surveys:

“Research has convincingly shown that token financial incentives, enclosed with the request to complete a questionnaire significantly boos incentives, and inevitable outperform promises to send a larger payment after a completed questionnaire is received. The providing of a tangible incentive, even a token one, is effective because it invokes a sense of reciprocal obligation which can easily by discharged by returning the completed questionnaire.”²

Incentives are a proven strategy for increasing survey response rates and are recommended to meet the OMB Guideline of 80% response.³

A10. Assurance of Confidentiality

Responses to this data collection will be used only for statistical purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific district or individual. We will not provide information that identifies respondents to anyone outside the study team, except as required by law.

Paper surveys will be returned directly to the Seattle offices of the Gilmore Research Group. Only those staff specifically assigned to this study will have access to the data. Data will be entered into a password protected database by staff who have signed a confidentiality agreement specific to this project. All paper surveys will be kept in a locked cabinet to which only authorized staff have access.

Data collected online will be entered via a secure website. Respondents will enter the serial number from the paper survey when they first log in to the survey site. This number will give them access only to their own survey records.

A11. Sensitive Questions

The questions asked on this survey are not considered to be of a sensitive nature. Since participation is completely voluntary and respondents are randomly selected, respondent risk is very slight. The study asks only for opinions, so there are no right or wrong answers. No one other than the respondent knows he/she has been selected for the survey so there is neither risk of retaliation for responses provided or risk of pressure to participate or not participate in the study. This is further exemplified by receiving an exemption from IRB approval.

A12. Estimates of Hour Burden for Collecting Data

This survey seeks to collect data from teachers of core subjects, principals, and school district superintendents in Alaska, Idaho, Montana, Oregon and Washington. The estimated number of survey returns is based on prior REL-Northwest experience conducting paper surveys with these populations. The survey is designed to meet OMB Standard 1.3 which states:

“Agencies must design the survey to achieve the highest practical rates of response, commensurate with the importance of survey uses, respondent burden, and data collection costs, to ensure that survey results are representative of the target population so that they can be used with confidence to inform decisions.” ⁴

Estimated response rates are 80% for each population. The hourly burden shown in the table below is based on the expected number of responses. Hourly burden was estimated based on a small paper & pencil pretest with 3 principals and 4 teachers. A maximum of 20 minutes is required to complete the survey from start to finish including reading instructions and placing the completed survey in the pre-paid business reply envelope. We estimate the time it will take the respondent to complete the survey online will be equivalent to the paper and pencil version.

Separate, but parallel surveys have been designed for teachers, principals, and superintendents. Estimates of total hours burden for all surveys are in the following table.

	Total Study Population	Sample Size	Expected Number of Responses (80%)	Number of Collections	Burden Hours per Respondent	Total Annual Burden Hours	Estimated Cost Burden*
Teachers of core Subjects	56,404	6,000	4,800	1	.333	1,598	$ 46,342
Principals	5,000	3,326	2,661	1	.333	886	$ 34,554
Superintendents	825	825	660	1	.333	220	$ 8,800
Total	62,229	10,151	8,121			2,704	$ 89,696
*Based estimated hourly rates derived from annual median salary information provided by CNNmoney.com

A13. Cost Burden for Respondents

There are no start-up costs or operation or maintenance costs to respondents and their school districts associated with this information collection.

A14. Annualized Cost to Federal Government

The total annualized cost of $340,381 includes the costs of developing survey items and methodology, drawing the sample of 7,651 educators and collecting and analyzing the data. Additionally, costs include project administration and oversight by REL-Northwest senior staff and an estimate of the hourly wages for respondents to complete the survey based on the estimated burden hours. Cost estimations are based on prior experience fielding similar self-administered surveys with the targeted survey populations.

COST CATEGORY
Senior REL-Northwest staff time: Dr James Leffler, Lucy Barnett and Richard Greenough	$36,200
Respondent hourly wages*	$89,696
Total capital/startup cost including printing and mailing pre-survey notification letters, surveys and cover letters, incentives, web survey programming, data entry, coding, cleaning, tabulation and analysis	$238,000
TOTAL SURVEY COST	$364,169
*Hourly rates assumed = $40 for superintendents, $39 for principals, $29 for teachers

A15. Program Changes or Adjustments

No changes or adjustments were noted.

A16. Plans for Data Analysis and Reporting

The number of teachers of core subjects in the population, by state from SY2004-05 (source: NCES, Common Core of Data, preliminary release) is as follows: Alaska: 7,756, Idaho: 14,269, Montana: 10,224, Oregon: 27,431, Washington: 53,125, and the regional total: 112,805. The number of schools in the population, by state, to estimate the number of principals in the population based on SY2004-05 (source: NCES, Common Core of Data, preliminary release) is as follows: Alaska: 526, Idaho: 736, Montana: 868, Oregon: 1,290, Washington: 2,340, and the regional total: 5,760. The estimated total without double counting of shared principals and excluding schools without principals is approximately 5,000. The number of districts with administrative superintendents (source: SEA websites) is as follows: Alaska: 53, Idaho: 114, Montana: 196, Oregon: 198, Washington: 296, for a total of 857. The estimated total without double counting of shared superintendents is 825.

We will use a stratified random sampling procedure to draw a sample of 1,200 teachers of core subjects and 700 principals from each state in the five-state region. (In Alaska there are less than 500 principals and in Idaho, principals number approximately 700, so surveys will be mailed to all principals in those two states.) In the case of district superintendents, since the numbers are relatively small, we are sending surveys to all superintendents in each state. Thus, given our sampling design, we anticipate being able to generalize to teachers, principals, and superintendents within the region and within each state.

In terms of general data analysis, we will examine the perceived importance of various types of evidence to inform educational practices measured through use of a point allocation system. Data from the surveys will be collated and variable distributions will be examined to identify outliers and inform the choice of analytic techniques. Mean scores will be tallied for each item that receives a point allocation and items will be rank ordered within each group and overall, based on mean scores.  The scores will provide not only a rank order, but also give some idea of the magnitude of importance educators in each role group (teachers, principals, and superintendents) place on each item. 

Items rated on a 1 to 10 scale will be reported as frequencies, mean ratings and collated into groups of high/medium/low. Cross-tabulation tables will be generated using WinCross software.  Anticipated descriptive statistics include frequencies, mean, median, mode, and standard deviations, and to make comparisons among groups we will use chi-square, student-t, z-tests of proportions, cluster analysis, and correlation techniques as appropriate to address our research questions and planning goals. Examples of tables that might be used are provided on the following pages.

Research products include a draft research report with separate analysis for each survey population for review and comment by REL-Northwest staff and a final report incorporating comments.

TEACHER SURVEY Q1a. Identifying the problem areas that impact student learning the most
	TOTAL	STATE					RURALITY		School Level			School Academic Performance
		AK	ID	MT	OR	WA	Urban	Rural	Elem.	Middle	High	Low	Intermediate	High
Total Responding	1,500	300	300	300	300	300
Top 3 (Net)	53%
Major problem (10)	16%
9	22%
8	14%
Middle 4 (Net)	38%
7	14%
6	9%
5	11%
4	4%
Bottom 3 (Net)	9%
3	4%
2	5%
Not a problem (1)	0%
Mean	8.23%
Standard Deviation	1.17%
Standard error	0.46%
Z-tests for proportions will be used to test for significant differences between subgroups.

TEACHER SURVEY Q3a. Improving School Attendance

Average

STATE

RURALITY

School Level

School Academic Performance

AK

ID

MT

OR

WA

Urban

Rural

Elem.

Middle

High

Low

Intermediate

High

Total Responding

300

300

300

300

300

Strengthening connections between home and school

6.39

Dealing with the effects of family culture on school attendance

4.72

Creating school as a welcoming place for students and families

2.74

Developing a safety net of support systems for students that promote improved attendance

3.91

Student T-tests will be used to compare means across subgroups

Timeline for Data Collection and Reporting
Contract Month	Project Task
December 8, 2006	Publish project documents on Federal Register (60 day)
December – February 2007	Federal Register survey posting duration
February – March 2007	Federal Register survey posting (30 day)
April 2007	OMB Approval
September 2007	Prepare for data collection by validating mailing addresses, printing pre-notification letters, survey instruments and preparing the mail out
October 15, 2007	Mail pre-notification letters
November 1, 2007	Mail out surveys and cover letters
November 2, 2007	Begin receiving data
November 16, 2007	Send 1st reminder postcard to everyone
December 1, 2007	Mail out survey and cover letter to samples with low response rates.
December 15, 2007	Send final reminder postcard
January 15, 2008	Close data collection field period
February, 2008	Edit, enter and clean data from paper surveys, generate descriptive statistics and other analyses
March – April, 2008	Prepare initial draft of reports and publications
May 2008	Submit final reports and publications

A17. Display of OMB Certificate

The OMB clearance number will be displayed on all surveys.

A18. Exceptions to Certification Statement

No exceptions were noted.

1 Main sources of data other than surveys are: SEA data on Adequate Yearly Progress and School/District Improvement determinations; SEA data on achievement results; NCES Common Core of Data statistics on student demographic trends (free-reduced lunch rates, minority enrollment rates, LEP enrollment rates); state forums and Sentinel Surveillance System for feedback from education leaders on emerging issues.

2 Dillman, Don A. Mail and Internet Surveys: The Tailored Design Method, 2007 Update with New Internet, Visual and Mixed-Mode Guide. 2^nd Edition. John Wiley & Sons, Inc., Hoboken, NJ, 2007. p. 16.

3 Office of Management and Budget. “Standards and Guidelines for Statistical Surveys” September 2006. Section 1.3. p8.

4 Ibid.

8