NAEP Charter School Pilot Questionnaire Study Memo

MEMORANDUM OMB # 1850-0803 v.52

DATE: July 21, 2011

TO: Shelly Martinez

Office of Information and Regulatory Affairs, Office of Management and Budget

FROM: Patricia Etienne

National Center for Education Statistics

THROUGH: Kashka Kubzdela

National Center for Education Statistics

SUBJECT: Response to OMB passback on NAEP 2012 Charter School Pilot Questionnaire Study

1. Please provide an analytical justification for why 120 respondents are required.

NAEP’s general practice is to have a minimum of 100 respondents for any pilot questionnaire, to achieve sufficient stability and reliability of the results in order to evaluate the items for use in future operational assessments.  For the charter school pilot questionnaire, we need variability across grades, type of location (i.e., large city or not), and breadth of responses.  As such, NCES has decided to include 20 more schools than the minimum 100, in order to secure adequate sample across these dimensions, while minimizing the burden of the number of respondents required.

2. Is NCES saying that the CCD is not timely enough to support drawing a representative sample of charter schools?

NCES uses CCD to draw its nationally-representative operational NAEP sample, and is doing so for 2012.  However, this nationally-representative operational NAEP sample is not stratified by charter school classification, so the charter school questionnaire is normally completed by those schools in the sample that happen to be a charter school. 

NAEP typically pilots new pilot sections of a school questionnaire embedded as part of the operational school questionnaire. The relatively small sample required for the 2012 operational NAEP, however, means that there will not be a sufficient number of charter schools in the operational sample, to pilot the charter school questionnaire, since only approximately 100 schools will be included at each grade.  In order to achieve a more robust sample, NAEP proposes to use a purposeful sample selection method for the 2012 pilot phone administration of the charter school questionnaire.  Charter schools not selected for the operational sample will be eligible for the pilot administration of the questionnaire.  We will attempt to include charter schools from both large and non-large cities and, if possible, to stratify on “recently identified” and “well established” charter schools.  

The sample will provide adequate and necessary information to evaluate the quality of the items in the pilot questionnaire. The purpose of the pilot is to improve the quality of the questionnaire and not to characterize charter schools in general. Only the operational school questionnaire data will be used to make inferences about the population of schools.  NCES understands that the convenience sample selected for piloting a new charter school questionnaire cannot in any way be used to generalize the results to the population of charter schools or sets of subpopulations.  The results of this feasibility and viability pilot will be characterized and used accordingly. NCES has used purposeful samples for NAEP pilot assessments successfully in the past.

3. We are unsure why a “pilot” is more useful than one-on-one think aloud interviews using the “Pilot Charter School QxQs” as guides for probing respondents. 

1. We can see the value of using response frequencies and the “other” category in questions 3, 6, and 9 to help determine the right response options for these questions.  However, what will you learn from knowing the unweighted percentage of a convenience sample’s response to closed questions with a small number of response categories without probing? 

2. And why would you disaggregate this “sample” by large city and other? 

3. And what would you do with “unexpected results” since you did not do any probing and since the sample is not representative of any particular universe?

One-on-one think aloud interviews were conducted earlier in the development cycle (please see the attached School Core and Charter School Questionnaire Cognitive Interview Study report, starting on page 31, summarizing those interviews; based on the NAEP Core Background Questions Cognitive Interviews for Students, Teachers, and School Administrators study, OMB# 1850-0803 v.40, approved on 1/7/2011).  In those interviews, probing did occur to learn more about the responses and to understand any unexpected results.  The purpose of the pilot is to try-out the questions (revised after the one-on-one interviews) with a large sample to ensure that the items are functioning as expected.  A typical pilot for NAEP would be conducted via paper-and-pencil or computer administration.  However, given the small number of schools sampled in 2012 and, thus, the very small number of charter schools, the pilot phone administration of the charter school questionnaire to a separate sample of schools was proposed.  The data from pilot test will be evaluated to determine if the items are functioning as expected and suitable for future operational assessments, or if further revisions are needed.

In regards to the question about disaggregating by large city, charter schools are very different in terms of SES and racial makeup in large cities and outside of large cities. Therefore, having schools in each category is beneficial in evaluating the effectiveness of the pilot.

2. What is the source of the charter school definition, as it does not match the ESEA definition?

The description of charter schools that was used in this submission was taken from the NAEP report on charter schools, America’s Charter Schools, located at http://nces.ed.gov/nationsreportcard/studies/charter/.