Supporting Statement B

Contract No.: ED-IES-12-C-0004

Supporting Justification

for OMB Clearance of a

Needs Sensing Survey

under the Regional

Educational Laboratory

Program (REL)

Section B

June, 2012

Submitted to: Submitted by:

U.S. Department of Education American Institutes for Research

Institute of Education Sciences 1120 E. Diehl Road, Suite 200

555 New Jersey Ave., NW, Rm. 308 Naperville, IL 60563

Washington, DC 20208 (630) 649-6563

(202) 208-7078

Project Officer: Project Director:

Christopher Boccanfuso Matt Dawson, Ph.D.

CONTENTS

Page

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL

METHODS 1

1. Respondent Universe and Sampling Methods 1

2. Statistical Methods for Sample Selection and Degree of Accuracy

Needed 2

3. Methods to Maximize Response Rates and to Deal with Nonresponse 3

4. Tests of Procedures and Methods to Be Undertaken 4

5. Individuals Consulted on Statistical Aspects of the Design 4

REFERENCES 5

TABLES AND FIGURES

Tables Page

1 Potential Respondent Universe 1

Supporting Statement

Request for OMB Clearance of a Needs Sensing Survey under the Regional Educational Laboratory Program (REL)

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

NORC at the University of Chicago considered a number of sampling approaches, including the Probability Proportional to Size (PPS) sampling approach it has used successfully in previous similar projects. However, since REL Midwest specified sampling targets and because the structure involves specific regions and subgroups which would not be well served by a multi-stage sampling approach, a different plan for data sampling is being employed. NORC will purchase updated population lists from a vendor who specializes in collecting information on the education community and will draw samples of 400 individuals from each of the seven states specified. These 400 individuals will be chosen to reflect as far as possible equal N from each of the four specified subgroups (teachers, district administrators, principals, school board members). Samples will be adjusted based on availability within subgroup categories.

Table 1: Potential Respondent Universe

State	Teachers*	District Administrators*	Principals*	School Board Members**
IA	35,842	871	1,792	119
IL	138,482	2,653	7,451	248
IN	62,258	1,132	3,187	76
MI	92,691	3,304	4,868	424
MN	52,839	2,058	2,094	248
OH	111,377	2,079	5,057	433
WI	58,425	963	2,477	102
Total	551,916	13,060	26,926	1,650

Source: *Common Core of Data (2009-10); **Agile Education Data

The database of educators being leased provides over 100 characteristic variables describing sample members. These are of varying importance across the four subpopulations. NORC will work with REL Midwest to determine the best sets of characteristics to use to stratify each subsample. The database will be divided into the four subsets, then sorted by the chosen characteristics within subset so that the randomly sampled respondents from each subset will be implicitly stratified by the characteristics chosen for each subset, resulting in a sample in which the characteristics are present in the proportions in which they are present in the underlying population. Recognizing the importance of rural education to the study, NORC can assess the possibility of “oversampling” rural respondents, realizing that: (a) the potential for larger rural samples may be more feasible in some states than in others and that, (b) such oversampling on rural urbanicity will require a smaller sample on some other characteristic.

2. Statistical Methods for Sample Selection and Degree of Accuracy Needed

The power of a statistical test is its probability of correctly rejecting a null hypothesis; it indicates the likelihood that one will be able to detect a difference when the difference exists between groups. Power analysis is often done prior to a study in order to understand the likelihood of making a Type II error (failing to detect a difference that exists). Full response from this net sample of (100 per subgroup) x (4 subgroups) x (7 states) = 2,800 was assessed for the resulting power to detect independent differences in proportions between subgroups using the techniques set forth in Cohen’s Statistical Power Analysis for the Behavioral Sciences (1988). The power to detect a moderate effect size would be approximately 85% with a response rate between 80-100% but would drop to as low as 75% with a 60% response rate. Similar estimates of power would correspond to the detection of differences between correlation coefficients.

Because power fluctuates with sample size, NORC will conduct a power analysis with the actual resulting data and report the magnitude and significance of any observed differences in survey responses. Furthermore, the results of all analyses, both on the overall sample and on subgroups, will be reported in compliance with NCES Statistical Standard 5-1. If the sampling constraints in the subgroups within states described above lead to reduced sample sizes and the power analyses confirm that caution in interpretation is imperative, the guidelines described in NCES Statistical Standard 5-1-5.5 will apply. Statistical design consultant for this data collection is NORC Senior Survey Methodologist Bernard Dugoni, PhD (773-256-6193).

3. Methods to Maximize Response Rates and to Deal with Nonresponse

NORC will employ a modified version of Dillman’s (2007) tailored design method in an effort to optimize response rates. Members of each sample will first receive an advance letter explaining the goals of the survey and the process by which individuals were selected. Respondents will be notified that participation is voluntary but will also be assured that neither individual identifiers nor individually identifiable results will be disclosed. Members of each sample will receive instructions for accessing and completing the survey or for opting out of the study, and contact information for NORC staff who can answer questions about and, if necessary, assist respondents who experience difficulty completing the survey. Individuals who refuse to participate on initial contact will be replaced by a randomly selected case with the same demographic profile as the original case where possible. Please note that we do not intend to refresh the sample for replacement of those who refuse after non-response follow-up or do not respond.

Two weeks after the start of data collection, NORC will prompt survey non-respondents by mailing a postcard urging them to complete the survey at their earliest convenience. In addition, NORC will send a total of up to four (4) e-mail prompts to non-respondents, encouraging them to participate. Each e-mail prompt will consist of an embedded link to the survey as well as the respondent’s personalized PIN and password. Response rates to Web surveys vary widely. AIR has indicated that prior REL Midwest need sensing activities (mixed mode Web and telephone surveys) achieved final response rates as high as 65%. The response rate for this needs assessment survey is expected to be higher since the number of items has been reduced by approximately 1/3 and since the number of responses per item has been lowered by having the majority of questions use binary versus Likert scaled choices. In addition to reducing respondent burden, NORC devoted much of the first quarter of the contract collaborating with REL Midwest on instrument development to ensure a maximum response rate by focusing survey questions on issues of keen interest to educators in the Midwest, which the literature suggests influence decisions to participate and to complete a survey. REL Midwest staff at AIR will also collaborate with NORC prior to and during administration of the survey in a variety of initiatives to elicit the target 80% rate of cooperation. For example, REL Midwest will identify opportunities to provide respondent populations with information about the importance of the survey and the value placed on respondents’ assessments of regional needs and the supports REL Midwest can provide. This may include notices in REL Midwest’s electronic newsletter and providing SEAs and state-level professional associations with brief announcement language to include in their correspondence with practitioners. Similar methods will be used to support NORC nonresponse follow-up.

4. Tests of Procedures and Methods to Be Undertaken

No pretests were conducted with the survey instrument.

5. Individuals Consulted on Statistical Aspects of the Design

The following individuals were consulted on the statistical, data collection, and analytic aspects of the Needs Sensing Survey:

Dr. Bernard Dugoni, Senior Survey Methodologist, NORC at the University of Chicago

Dr. Raymond Lodato, Senior Survey Director, NORC at the University of Chicago

Dr. Eric C. Hedberg, Senior Research Scientist, NORC at the University of Chicago

Dr. Kevin L. Brown, Senior Research Scientist, NORC at the University of Chicago

REFERENCES

Cohen, Jacob. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd Edition). Hillsdale, NJ: Lawrence Earlbaum Associates

Dillman, Don A. (2007). Mail and Internet Surveys: The Tailored Design Method -- 2007 Update with New Internet, Visual, and Mixed-Mode Guide (2nd Edition). John Wiley: Hoboken, NJ.

National Center for Education Statistics. (2003). NCES Statistical Standards. U.S. Department of Education, Washington, DC. Retrieved March 2012 from http://nces.ed.gov/pubs2003/2003601.pdf.