PARENT
INFORMATION AND SCHOOL
CHOICE EVALUATION
Request
for OMB Clearance
OMB#
Supporting Statement Part B
National Center for Education Evaluation
U.S. Department of Education
Institute of Education Evaluation
Washington, DC
January 2016
Preface iii
B. Collection of Information Employing Statistical Methods 1
B.1 Respondent universe and sampling methods 1
B.2 Procedures for the collection of information 2
B.3 Methods to maximize response rates and deal with nonresponse 5
B.4 Tests of procedures or methods to be undertaken 5
B.5 Individuals consulted on statistical aspects of the design, data collection, and analysis 5
TABLES
Table Preface.1. Factors to be tested, mapped to research questions vi
Table B.1. Factors to be tested, mapped to research questions 3
Table B.2. Minimum detectable effects (MDE) of alternative designs 4
Sponsored by the Institute of Education Sciences (IES), U.S. Department of Education, the Parent Information and School Choice Evaluation (PISCE) is an important first step toward filling the wide gap in knowledge about how to present school choice information to parents. This research is needed to provide guidance to districts where school choice is expanding. PISCE seeks to identify the format, amount, and organization of information that is most comprehensible and usable to parents. The study will target low-income parents of school-age children and will evaluate perceptions of different presentations of school information. The results of the study will be used to create a reader-friendly guide for school districts.1
IES has contracted with Mathematica Policy Research to conduct the needed research. Most of the experiment will be conducted with members of a standing panel who already complete surveys on a regular basis for a variety of purposes. This approach provides a low-cost and quick turnaround method to obtain findings related to the understandability of school choice information, which does not require respondents to be making actual school choices for their children. To enhance what can be learned from the standing panel, the research team also intends to recruit a sample of low-income parents of school-age children from locations where a public school choice marketplace with unified enrollment has been active for at least two years. Parents who have experienced public school choice or are at least exposed to open enrollment in their district may experience the experiment differently than the standing panel members, for whom considering schools other than one’s default neighborhood school may be unfamiliar. This augmented sample of, presumably, less survey-savvy low-income parents will be used to provide a sensitivity check of the findings based on the standing panel alone. IES is submitting this clearance package which requests approval for the study’s recruitment and survey activities.
School choice has increased dramatically in recent years through the expansion of charter schools and open enrollment in traditional districts. School choice can only be effective policy if parents are able to navigate school choice systems, follow application procedures, and process large amounts of complex information to make informed choices about the best schools for their children. The rise of new technology and data systems has led to an explosion of such information, but the school choice marketplace has yet to determine the best ways to curate and present this information to parents. In particular, there is scant research-based guidance that school districts and related entities can rely on when making school choice information available to parents, and each new district that enters this policy arena has had to muddle through the process using trial and error. Further, ED’s school choice programs supporting magnets, charter schools, and vouchers have identified parent information as a barrier to greater participation in these programs and potentially to improved student outcomes.
The proposed experiment and data collection will be an important first step toward addressing this need for evidence-based guidance. The comprehensibility of school choice information is a particular concern for low-income parents and those who might struggle to navigate the technical systems used to display this information. The experiment will present parents with school information in various formats, amounts, and organizational layouts and allow us to evaluate how well parents understand and use the information.
Within the field of education, researchers have mostly focused on discovering what parents value in schools, rather than on how best to organize and present information on school attributes. The most common approach has been to conduct focus groups or surveys, asking participants about the factors that drive their school choices (Fossey 1992; Armor and Peiser 1998; Collins and Snell 2000; Klute 2012; Kelly and Scafidi 2013; Great Schools 2013; Jochim et al. 2014). However, this approach has been criticized for eliciting socially desirable responses and failing to capture the role of race, class, and other demographics (Stein et al. 2010). Other researchers have used Internet search terms (Schneider and Buckley 2002) or conducted statistical analyses of actual rankings submitted by families in real-world school choice settings (Glazerman 1997; Hastings et al. 2008; and Harris and Larsen 2015, using data from Minneapolis; Charlotte-Mecklenburg, NC; and New Orleans, respectively). These studies, which provide estimates on the relative importance of various school attributes to parents, are useful because they highlight the dimensions along which choice information might be more important, such as academic achievement of the school, demographics of the student body, distance and convenience of the school location; and school safety and climate.
Very little research has been done on how best to organize and present information about school attributes. Among the few studies available, Jacobsen et al. (2014) studied the effect of information formats on parents’ perceptions of schools. Other researchers have estimated the impact that information presentation has on school choice attitudes and behavior. For example, Valant (2014) used quick-turnaround online experiments and a regression discontinuity design to examine how parents update their opinions of local public schools after receiving various types of information. Valant and Loeb (2014) also conducted field experiments with families choosing schools in Milwaukee; Washington, DC; and Philadelphia to test how information affects school choosers’ attitudes, beliefs, and behaviors. Although directly relevant to the proposed study, Valant and Loeb’s experiments did not explore as many different types of information presentations as proposed here, and they focused on parents’ ratings of schools rather than on whether they actually understood the information and found it easy to use.
The study for which clearance is being sought will collect and analyze data to address three specific questions:
What is the optimal way to present school choice information?
What is the right amount of school choice information to present?
How is school choice information best organized?
To answer these questions, Mathematica will conduct an online experiment with low-income parents of school-age children. The sample will mostly be drawn from a market research standing panel that is commercially available. To address concerns about the ability to generalize from such a panel, Mathematica will augment the panel with members of the public recruited from targeted locations within cities such as Washington, DC, and New Orleans, where low-income families are immersed in an open-enrollment school marketplace with many public schools to choose from, including traditional district schools and charter schools.
Both the panel sample and the augmented sample members will participate in a web-based survey that will collect basic demographic information and an endline survey measuring how well the participants understood, used, and perceived the ease of use of school choice information. After completing the baseline survey, respondents will each be randomly assigned to one of several different ways to present school choice information (treatment arms). Random assignment allows us to assume that differences in responses to the endline survey, on average, are attributable to differences in the ways information was presented to respondents.
To generate most of the experimental data, Mathematica will work with a market research firm—the leading candidate is Survey Sampling International (SSI)—who will identify 3,300 parents of school-age children in the U.S. who are low income, defined as having an annual household income below $40,000. To check the sensitivity of the standing panel findings with a less internet savvy group, Mathematica will recruit 150 volunteers who are low-income parents of school-age children in low-income areas where school choice is particularly salient (the augmented sample). We plan to focus our recruiting efforts on Washington, DC, and New Orleans. However, we may consider other cities if necessary.
All eligible study participants will be asked to complete a 10-minute baseline survey. Then they will be randomly assigned to one of 72 different variations on a school information website and asked to complete a 20 to 30 minute endline survey, for a total of 30 to 40 minutes. The baseline will measure demographic characteristics, such as income and whether or not the respondents have school-aged children, as well as digital literacy. The endline will measure how respondents use the information, how well they understand the information, and how easy or difficult it was to use.
The treatments being studied consist of different ways to present information about a set of fictitious schools. The information will be presented in one of 72 different ways for each respondent, with the 72 variations being constructed by crossing five factors, each with two or three levels (3 x 2 x 2 x 3 x 2 = 72). Table Preface.1 lists each of the five factors and maps them to the study’s research questions. Table Preface.2 lists the information domains, the specific attributes that will be presented, and the presentation variations by format and source of information. For example, to address the question of presentation, we will test the format (factor A.1) in which discipline and safety information is presented and the source of that information. Discipline and safety information will be presented as a numerical rating, a graphical presentation, such as a bar chart, or an icon, such as a letter grade of “A” to indicate that the school has high safety ratings. Sources of the information will vary in that one source will be a more objective indicator, the number of school suspensions per year and another source would be a more subjective indicator, results from a parent survey on the school’s safety. The specific factors, domains, and attributes that make up the experiment (treatment arms) have been selected based on a review of research on information presentation across several fields, including health and marketing.
Table Preface.1. Factors to be tested, mapped to research questions
Research Question |
Factor |
Level 1 |
Level 2 |
Level 3 |
A. Presentation |
1. Format |
Numbers |
Graphs |
Icons |
2. Source of information |
Objective indicator (e.g. suspension rate) only |
Both objective and subjective indicators |
n.a. |
|
B. Amount |
1. Reference point |
No reference point |
District reference point |
n.a. |
2. Attributes and Disclosure |
Low information (one attribute per domain) |
High information, multiple attributes per domain all shown at once |
Progressive disclosure: high information, with drawer closed by default |
|
C. Organization |
1. Sort |
Default = distance |
Default = academic rating |
n.a. |
n.a. = not applicable
Table Preface.2. Types of School Choice Information to Display
Domain |
Attributes |
Variations in format (graphics or icons to be shown in addition to numbers) |
Variations in source (subjective indicator to be shown in addition to main attributes) |
Distance |
Straight-line distance from home to school *Walking time *Driving time |
No variation |
No variation |
Academics |
% proficient on 2016 achievement test *% proficient on the 2016 math test *% proficient on the 2016 reading test *Average 2015-2016 academic growth, 0-100 index *Average 2015-2016 academic growth in math *Average 2015-2016 academic growth in reading |
|
Percentage of parents agreeing with statement that they are highly satisfied with the school’s academic quality |
Safety |
% of students with no suspensions *Attendance rate *Yes/no: School won a blue-ribbon award for anti-bullying efforts |
|
Percentage of parents agreeing with statement that the school is a safe place for their child |
Resources |
Number of laptops or tablets per 100 students *Year of most recent school renovation *Yes/no on 4 items: school has dedicated art studio, library, computer lab, music program |
No variation |
No variation |
* Attribute will only appear in the open drawer
The respondent universe includes the entire population of low-information parents living in jurisdictions that offer the option of selecting schools from a menu of alternatives, such as charter schools or open-enrollment district schools. Low-information parents are those who may have limited English-language skills, may have limited educational attainment, or may face barriers to accessing information on school options.
Such a population is difficult to identify and list. Furthermore, the set of relevant jurisdictions is always changing. Therefore, it is not practical to seek a statistically representative sample, nor is it necessary given the goal of this study, which is to provide information to school districts and third parties that is timely, to address the rapid growth of choice systems nationwide. The study will primarily use a convenience sample of 3,300 parents drawn from a standing panel sample. This sample will be screened to identify if they meet key criteria necessary for study participation, namely that they are low-income parents of school-aged children. For simplicity, the study will use low-income status as a proxy for low information. A brief digital literacy module the digital literacy module will provide a set of covariates to use as statistical controls in the impact analysis, improving the statistical precision of the estimates. t.
To address concerns about the ability to generalize from such a panel, the study will recruit another 150 participants from the public, focusing on targeted neighborhoods in selected cities in which parents most likely to meet the screening criteria. For example, in Washington, DC, we would likely target neighborhoods in Ward 8, since it has the lowest median household income of any wards in the city. This sample will be used as a sensitivity check for the standing panel findings and may help us to distinguish between the best approaches if two or more contrasts look equally promising. We have determined adding 150 cases is sufficient to conduct a simple test of the difference in outcomes between the standing panel and the validation sample.
The one-time data collection will involve “authentic” participants who comprise a convenience sample, recruited by using a two-pronged approach:
Collaboration with community organizations. The study team will coordinate with organizations that have a strong presence in the selected cities. Such organizations include the YMCA, Boys & Girls Clubs, and, in Washington, DC, My School DC and DC School Reform Now (DCSRN). The experiment will be advertised through flyers and by word of mouth via community organizations. The in-person data collection will be held at a public library, city recreation center, or similar site within the community that is easily accessible and familiar. Conducting the in-person data collection in convenient locations frequented by the targeted families, at convenient times and with sufficient compensation, will make it possible to recruit a sufficient sample.
Recruitment at community events. We will also explore the possibility of conducting additional recruiting at local events for families with school-age children. Parents could participate in the experiment before, during, or after the event at the same location as the local event and would have the opportunity to sign up for, be given informed consent, and participate in the experiment on the spot.
Statistical methods for sample selection. The study will not rely on statistical methods to select either the communities where the experiment will take place or the respondents. Instead, the study team will select one or two communities with a high percentage of families choosing schools. In this respect, New Orleans and Washington, DC, are the two most attractive cities in which to recruit. The study team will work with local authority and community groups to target the neighborhoods with the greatest amount of choice or the need for choice among suspected low-information families. The augmented part of the study data will be based on a convenience sample of 150 parents within these communities. This study is a first step in understanding how parents consume school choice data based on organization and presentation. Before conducting a field study with a statistically representative sample, it is important to test a large number of factors in a laboratory setting, for which it would be infeasible to identify, list, and sample from a universe of low-information families poised to make school choices.
Statistical methods for impact estimation. The study will address three questions: (1) What is the optimal way to present school choice information?, (2) What is the right amount of school choice information to present?, and (3) How is school choice information best organized? Our study will test a total of 72 factors, selected from all possible factor combinations. PISCE will address the three questions by conducting a Bayesian factorial experiment. In such an experiment, a single, unified factorial design can efficiently take the place of many independent experiments and provide the experimenter with a framework for increasing statistical precision via efficient fractional factorials, sometimes known as orthogonal designs or orthogonal contrasts (Zurovac and Brown 2012).
The Bayesian model looks much like a classical linear regression model with main effects for each of the factors and interaction effects that predict the effectiveness of each examined factor combination (for additional precision, the model also controls for respondents’ demographic characteristics). The study team will analyze the model’s main effects to determine the overall effectiveness of each presentation factor, on average, for the study sample. In addition, the interaction effects will reveal additional insights regarding which combination of factors is most effective or whether particular factors are effective in some contexts but not in others. The model permits significance tests that estimate whether each factor combination produces an outcome that differs from the average in the experimental sample and allows pairwise tests for differences between every factor combination.
In the Bayesian approach, all model parameters are estimated simultaneously. That way, if a high correlation between the effectiveness of all the treatment arms (factor combinations) shares a given factor level, the main effect of the common factor level receives a greater weight than the interaction terms involving that factor. An example is an attempt to estimate the effect of a factor combination that uses stoplight icons to represent academic performance for a large number of schools (20). Even with a sample size of only 30 choosers with that factor combination, the effect estimate is weighted toward the average effect of stoplight icons based on every sample member who had stoplights, regardless of the number of schools shown (5, 10, or 20). Given that the estimate for this factor combination uses weighted data from 90 choosers, the study is more precise than a conventional design examining each factor combination independently.
The study will use Stan software developed by Andrew Gelman and colleagues (2015) for the analysis. Stan fits complex, high-dimensional Bayesian models faster than the limited systems traditionally used for such analyses.
Degree of accuracy needed. PISCE’s Bayesian framework accounts for the sample size requirements of a multiarm study design and permits the examination of a large number of factors by eliminating the need for post hoc multiple comparison adjustments that reduce power (Gelman et al. 2012). As a result, the design delivers large gains in precision and permits a much larger number of contrasts to be tested as compared with a design that considers each possible combination of factors in isolation (Finucane et al. 2015; Gelman et al. 2006; Ghitza and Gelman 2013). In Table B.2, we summarize the scope and statistical power of the PISCE design for various numbers of factor combinations. For each design, we summarize the MDE for a test of whether a given factor combination has a statistically significant effect relative to the average outcome among all study participants. We include a comparison of the standing panel sample to the “authentic” sample, which we will use to understand whether there are any key differences between the samples in terms of their understanding and use of the school choice information under various presentations and will provide evidence for the external validity for the experiment as a whole.
Table B.2. Minimum detectable effects (MDE) of alternative designs
Sample size |
Number
of factor |
Configuration
|
MDE |
Online experiment with between-respondent comparison of factors using a web-based interface |
|||
2,500 |
16 |
2 x 2 x 2 x 2 |
0.20 |
2,500 |
180 |
2 x 3 x 3 x10 |
0.28 |
2,500 |
72 |
3 x 3 x 2 x 2 x 2 |
0.21 |
3,000 |
72 |
3 x 3 x 2 x 2 x 2 |
0.19 |
3,500 |
72 |
3 x 3 x 2 x 2 x 2 |
0.17 |
Note: Each table row represents a different factorial design with between-respondent comparisons. The first and second rows show designs with four groups of factors (but a different number of factor levels) whereas our preferred design (shown for varying sample sizes in rows 3, 4 and 5) has five groups of factors. MDE represents the MDE size for a test that a given factor combination has an effect that is statistically distinguishable from the average in the study sample at a two-tailed significance level of 0.05 and 80 percent power. We assume an R2 of 0.12 from covariates. Power calculations were performed using 500 Monte Carlo simulations, with a different real effect for each factor combination in each simulation round. For each simulation, the variance of effects for each factor and pairwise interaction of factors were drawn from a half-Cauchy distribution as recommended by Gelman (2006); effects were randomly drawn given these variance parameters; and then effects were scaled such that the standard deviation across the effects of all factor combinations was equal to 0.25.
As the table shows, the study’s design makes it possible to test a large number of factors with a given sample size. By including 72 factor combinations, the study will provide an MDE of 0.17 with a sample of 3,000 study participants.2 A comparison test between the mean outcomes (and digital literacy scores) of the standing sample of panelists and those of the 150 respondents recruited in person will achieve an MDE of 0.21.
As described in B.1, it is not practical to seek a statistically representative sample, nor is it necessary given the goal of this study (for a discussion of why see Mullinix, Leeper Druckman & Freese, 2016). Instead, recruitment will continue until a sample size that will provide adequate statistical power (3,300 parents) is attained. The panel sample will be drawn from an opt-in survey panel. It is not possible to calculate opt-in rates for this kind of sample because the total number of those who had the opportunity to opt-in but did not is unknown (for a discussion see Callegaro & DiSogra, 2008). Survey response rates (the proportion of qualified survey respondents who accept the invitation to participate and then completed the survey) are likely to range from 80% to 95% based on prior studies using similar methods.
To encourage study participation and ensure high recruitment rates, we will provide participants of the in-person experiment with a monetary incentive of $30 after the survey is completed. Panel participants are incentivized in various ways by the panel management organization (e.g., they receive credit for survey participation towards online purchases).
As stated in B.1, we will recruit and conduct the in-person experiment in a setting that is convenient and familiar to participants, such as a public library or city recreation center. We will also offer times that are convenient to parents, such as weekend mornings, and during events that they plan to attend, such as school choice fairs.
We will test the data collection forms and procedures in a pretest involving nine or fewer individuals. After the forms are completed, members of the study team will debrief each participant by using a standard debriefing protocol to determine whether any words or questions are difficult to understand or answer.
The following individuals are responsible for the statistical aspects of the design, data collection, and analysis:
Meredith Bachman, IES |
(202) 219-2014 |
Marsha Silverberg , IES |
(202) 208-7178 |
Steven Glazerman, Mathematica Policy Research |
(202) 484-4834 |
Jon Valant, Education Research Alliance for New Orleans |
(504) 274-3617 |
Lisbeth Goble, Mathematica Policy Research |
(312) 994-1016 |
Ira Nichols-Barrer, Mathematica Policy Research |
(617) 674-8364 |
Jesse Chandler, Mathematica Policy Research |
(734) 205-3088 |
Mariel Finucane, Mathematica Policy Research |
(617) 715-6935 |
References
Callegaro, M. and DiSogra, C. “Computing response metrics for online panels.” Public Opinion Quarterly, vol. 72, 2008, pp. 1008–1032.
Finucane, Mariel, Ignacio Martinez, and Scott Cody. “What Works for Whom? A Bayesian Approach to Channeling Big Data Streams for Policy Analysis.” Mathematica Policy Research Working Paper Series, no. 40. Princeton, NJ: Mathematica Policy Research, June 2015.
Gelman, Andrew. “Prior Distributions for Variance Parameters in Hierarchical Models.” Bayesian Analysis, vol. 1, no. 3, 2006, pp. 515–533.
Gelman, Andrew. “Why We (Usually) Don’t Have to Worry about Multiple Comparisons.” Journal of Research on Educational Effectiveness, vol. 5, 2012, pp. 189–211.
Gelman, Andrew, Bob Carpenter, Matt Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, Allen Riddell, Marco Inacio, Jeffrey Arnold, Rob J. Goedman, Brian Lau, Mitzi Morris, Rob Trangucci, Jonah Sol Gabry, Alp Kucukelbir, and Robert L. Grant. Stan Version 2.7.0, 2015.
Ghitza, Yair, and Andrew Gelman. “Deep Interactions with MRP: Election Turnout and Voting Patterns Among Small Electoral Subgroups.” American Journal of Political Science, vol. 57, no. 3, July 2013, pp. 762–776.
Jacobsen, Rebecca, Jeffrey Snyder, and Andrew Saultz. “Information or Shaping Public Opinion? The Influence of School Accountability Data Format on Public Perceptions of School Quality.” American Journal of Education, vol. 121, no. 1, November 2014, pp. 1–27.
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. “The generalizability of survey experiments.” Journal of Experimental Political Science, vol. 2, no. 2, 2015, pp. 109–138.
Zurovac, Jelena, and Randy Brown. “Orthogonal Design: A Powerful Method for Comparative Effectiveness Research with Multiple Interventions.” Issue Brief. Princeton, NJ: Mathematica Policy Research, April 2012.
1 ED is also interested in the effects of providing better information on actual school choices and student outcomes. The Department will consider a field trial of strategies to disseminate school choice information after the current study has narrowed down the way information is best presented.
2 The study was powered to detect effects as small as 0.25 standard deviations. This threshold is based on past research showing that varying the style and format of school performance information can produce effects on parent survey outcomes that are of this size or greater (Jacobsen et al. 2014).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | PCI OMB Supporting Statement-Part B |
| Subject | PISCE Request for OMB Clearance OMB# |
| Author | Mathematica Staff |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |