SSCCA Supporting Part B

2. Collection of Information Employing Statistical Methods

1. Universe and Respondent Selection

This data collection project employs sampling methods. The population of inference for the 2010 SSCCA will be all direct criminal appeals disposed in intermediate appellate courts or courts of last resort. Specifically, the sample will be designed to examine a national sample of appeals adjudicated in intermediate appellate courts and courts of last resort separately because these courts perform important and different functions. Appellate courts have two primary functions: “error correction” to ensure that law is interpreted correctly and consistently and “law declaring” to provide a means for the development of law through their decisions and explanations of decisions. In states with two levels of appellate courts, the intermediate appellate courts are often assigned the error correcting function while the courts of last resort are primarily concerned with the development of law. Hence, the different roles of intermediate and state supreme courts necessitates a sampling framework capable of producing national level estimates on criminal cases disposed at both levels of appellate review.

State appellate courts will provide lists of criminal appeals disposed during calendar year 2010, from which the sample of cases will be drawn. The universe is based on appeals disposed, rather than appeals filed, to limit the study period required for prospective tracking of criminal appeals, and to prevent censoring of any appeals filed that did not reach disposition during the study period.

BJS used the number of criminal appeals filed as a proxy for the universe of criminal appeals disposed to estimate the appropriate sample size and sampling approach. There are currently no estimates of the number of cases disposed by appellate courts; however BJS’ initial design and development work did not identify any reasons to believe that the number appeals filed in a given year would be an inappropriate proxy for the number of criminal appeals disposed. Based on NCSC data from the 2008 Court Statistics Project¹, there were approximately 27,043 criminal appeals filed in courts of last resort and 70,113 criminal appeals filed in the intermediate appellate courts, for a total of 97,156 filed criminal appeals.

Based on this estimated number of criminal appeals, the SSCCA sample size will be approximately 4,860 cases. The 4,860 number was derived from a power analysis to determine the optimal sample needed to produce national level estimates and consideration of the costs associated with data collection. About 3,060 criminal appeals cases will be drawn from the 91 intermediate appellate courts and a minimum of 1,800 appeals will be selected from the 52 courts of last resort. The following table summarizes the population and sample size information.

Count of courts and cases by type of appellate court

Type of court	Courts	Cases	Case sample size
Intermediate appellate court	91	70,113	3,060
Court of last resort	52	27,043	1,800
Total	143	97,156	4,860

The proposed sample size of 4,860 cases will be selected across the 52 courts of last resort and 91 intermediate appellate courts, yielding an average sample size of about 34 cases per court.² Cases will be sampled within courts at an overall rate by type of court with a preliminary estimate of about 4.5% in intermediate appellate courts and 6.7% in courts of last resort. Since caseload volumes differ appreciably across the various appellate courts, the actual sample sizes will vary by court and depend on the total number of cases decided in these courts. For the most part, the cases selected from each court will be a random sample of criminal appeals disposed in 2010. The exception to the random sample will include death penalty appeals. Although the sample will allow for national inferences of appellate court case processing, the relatively small number of cases sampled per court means that inferences of appellate case processing at the court level will not be possible. For example, the SSCCA data will not be capable of providing estimates of appellate case processing for the individual appellate court(s) of a particular state or region within that state.

Another important component of the sampling framework is that the 4,860 number will encompass an oversample of two principle case types: death penalty and cases heard in both intermediate appellate courts and courts of last resort. All death penalty cases will be included in the study. Typically, these cases account for about 250 criminal appellate filings per year and are important to examine because of the stakes involved in addition to the substantial amount of time and resources courts expend on these cases. The death penalty cases will be identified by case type, particular filing requirements, or the familiarity of court staff with such cases because of their severity. Some courts will have no such cases for statutory reasons.

In addition, some court of last resort cases will have passed through an intermediates appellate court initially. Such cases are interesting analytically because they have gone through two levels of appellate review and also present an opportunity to collect data on a single case from both kinds of courts with some economy. A random sample of 300 such cases will be selected as part of the 1,800 COLR sample and their IAC data will be obtained and coded.

To reiterate, BJS plans to make estimates for four major domains, as follows:

1) Death penalty cases (expected sample size = 234 cases; a census)

2) Intermediate appellate courts (IAC) cases (expected sample size = 3,060 cases)

3) Courts of last resort (COLR) cases (expected sample size = 1,800 cases)

4) Heard by both courts - COLR / IAC cases (expected sample size = 300 cases)

These four kinds of cases will be sampled at four distinct rates (i.e., 100%, 4.25%, 5.39% and 27.67% (i.e., conditional on selection into the COLR sample) respective to 1…4 above) within the two kinds of courts (COLR, IAC).

Since the death penalty cases are intended to be a census, no precision table will be given below for that specific domain.

The precision offered by an overall responding sample size of 4,860 cases is, as follows:

Precision offered by 4,860 responding cases

Percentage	N	Standard Error	LCI	UCI
30.0%	4,860	0.00657	28.71%	31.29%
40.0%	4,860	0.00703	38.62%	41.38%
50.0%	4,860	0.00717	48.59%	51.41%
60.0%	4,860	0.00703	58.62%	61.38%
70.0%	4,860	0.00657	68.71%	71.29%

The standard error of the percentage is based on the percentage itself and the sample size. The lower and upper confidence interval end points (LCI, UCI) give the confidence interval for the estimate of the percentage.

The precision offered by a responding sample size of 3,060 (i.e., the IAC) cases, is as follows:

Precision offered by 3,060 responding cases

Percentage	N	Standard Error	LCI	UCI
30.0%	3,060	0.00828	28.38%	31.62%
40.0%	3,060	0.00886	38.26%	41.74%
50.0%	3,060	0.00904	48.23%	51.77%
60.0%	3,060	0.00886	58.26%	61.74%
70.0%	3,060	0.00828	68.38%	71.62%

The precision offered by a responding sample size of 1,800 (i.e., the COLR) cases is, as follows:

Precision offered by 1,800 responding cases

Percentage	N	Standard Error	LCI	UCI
30.0%	1,800	0.01080	27.88%	32.12%
40.0%	1,800	0.01155	37.74%	42.26%
50.0%	1,800	0.01179	47.69%	52.31%
60.0%	1,800	0.01155	57.74%	62.26%
70.0%	1,800	0.01080	67.88%	72.12%

The precision offered by a responding sample size of 300 (i.e., the COLR / IAC) cases is, as follows:

Precision offered by 300 responding cases

Percentage	N	Standard Error	LCI	UCI
30.0%	300	0.02646	24.81%	35.19%
40.0%	300	0.02828	34.46%	45.54%
50.0%	300	0.02887	44.34%	55.66%
60.0%	300	0.02828	54.46%	65.54%
70.0%	300	0.02646	64.81%	75.19%

BJS will also aim to test whether certain types of criminal cases (e.g., person vs. property) are more likely to result in reversals or whether the percent of reversed cases varies by whether the appeal resulted from a sentence or conviction. The minimum detectable effects for the person vs. property reversal analysis, assuming an overall responding sample size of 4,860 cases, alpha =.05, a desired 80% power, a person v/s property v/s other distribution of 46%, 24% and 30% (Time on Appeal, Roger Hanson (1996). National Center for State Courts: Williamsburg, VA), and an underlying base reversal rate of 50% for person cases (n = 2,236 or 46% of 4,860) is +-5.1%. Hence, the reversal rate for property cases (n = 1,166 or 24% of 4,860) would have to be greater than or equal to 55.1%, or less than or equal to 44.9% for there to be a statistically detectable difference between the reversal rates for person and property cases.

The criminal appeals study proposes to use a stratified, single stage probability sample of appeals within all 91 intermediate appellate courts (IACs) and all 52 courts of last resort (COLRs), yielding a total responding sample size of 4,860 cases (3,060 cases within the IACs, 1,800 cases within the COLRs). Given this sample design, the usual design effects due to clustering do not apply; however, a differential weighting effect does apply given the relative oversampling of COLR cases (which would be 1,353 cases under strictly proportional allocation) and under-sampling of IAC cases (which in turn would be 3,507 cases under strictly proportional allocation). We estimate this differential weighting effect to be approximately 1.06, which implies an overall effective sample size of approximately 4,590 (i.e., 4,860 / 1.058).

Note that within the four major domains listed above, each sample is an equal probability sample, and thus no differential weighting effect (and therefore no design effect) applies

It is anticipated that 95% or more of the nation’s appellate courts will participate in the SSCCA project. A national level data collection effort of criminal appeals data, however should anticipate some appellate courts refusing to participate because of time and cost constraints. The required sample size of 4,860, therefore, will be increased to 5,115 to compensate for this expected non-response.

Since the 2010 SSCCA estimates will be based on data obtained from a probability sample, each sample case requires the assignment of a weight in order to provide unbiased estimates with measurable precision. The overall weight for each case will be calculated as the product of a base weight, reflecting the overall probability of selection, a non-response adjustment factor to compensate for any missing cases (due mostly or entirely to courts that cannot or do not participate), and a post-stratification or raking factor to match sums of weights with known frame or population totals. The base weight of each case will be the reciprocal of its probability of selection, which may vary across cases due to the oversampling of death penalty cases and cases in both kinds of courts (courts of last resort and intermediate appellate courts).

A non-response adjustment factor will be calculated at the case level as the ratio of the sum of weights for all sampled cases to the sum of weights for all responding cases. A non-response adjusted weight will be calculated as the product of the base weight and the non-response adjustment factor for each case. A post-stratification or raking factor will be calculated for each case as the ratio of frame or population totals to the sum of non-response adjusted weights, and a post stratified or raked weight will be calculated for each case as the product of the post-stratification or raking factor and the non-response adjusted weight. Weights and factors will be reviewed at each stage of adjustment to control the additional variability introduced through assigning weights.

We expect very little non-response (due mostly or entirely to courts that cannot or do not participate) given the National Center for State Courts’ (NCSC) excellent working relationship with the state courts. Nonetheless, in such instances an attempt should be made to identify other, similar courts from which additional cases can be selected, or within which non-response adjustments can be made to existing sampled cases in order to compensate for the non-responding court(s). Similar courts can be grouped into non-response adjustment cells or otherwise identified by type of court (COLR, IAC), state or other criteria (e.g., states with no IAC) which affect the type and volume of appeals cases.

The proposed design will be a single stage probability sample of 5,115 (4,860 responding) cases. It is anticipated that jackknife replication methods for variance estimation will be utilized, and a set of jackknife replicate weights will be calculated along with each full sample weight and adjustment in order to estimate variances, standard errors and confidence intervals.

2. Procedures for Collecting Information

As stated in previous sections, the appellate court staff will not be asked to code the sampled appeals; rather, staff from the NCSC will code the data elements and case documents used to gather the information required for this project. Even though NCSC will collect these data essentially in-house, there are several steps that need to be undertaken to ensure the data collected are accurate, valid, and submitted in a timely matter. The first step will involve identifying contacts within the appellate courts to secure participation in the project. To accomplish this task, the project manager will correspond with the appellate courts (both intermediate appellate courts and courts of last resort), describing the purpose of the project and requesting that a member of the court staff be made available to act as the NCSC’s liaison to the project. Once a court liaison has been identified, NCSC project staff will request a list of appeals that meet the characteristics of a criminal appeal as defined by the project (e.g., a direct appeal from a criminal case in a trial court to the intermediate appellate court, the court of last resort, or both, disposed in 2010).

Once the sample lists are completed and a probability sample of approximately 5,000 appeals drawn from that sample, NCSC project staff will code the data elements and case documents used to gather the information required for this project. As previously described, NCSC staff will gather data from several major sources including the legal briefs submitted by the parties, the court docketing information, and the opinions produced by the courts. When possible this information will be accessed online; however, for courts with none or limited online access, NCSC will request that the proper documentation be mailed for in-house coding or will travel onsite to collect the data.

Having project staff engaged in the tasks of collecting and coding the data rather than hiring contract labor offers several advantages. Specifically, it reduces the amount of time needed to recruit and train on-site contractors; reduces the amount of time for data collection (fulltime NCSC employees can devote more time and attention to data collection than most part-time, temporary workers); improves the ability of NCSC project staff to oversee the quality and timeliness of data collection; and most importantly, enhances the accuracy, consistency, and intra-reliability of the coding.

Regardless of whether coding is done in-house, all coders will be required to undergo extensive training on the coding process. Coders will be required to pass a coding test after reviewing the coding instructions and will also be required to submit a select number of test cases to the project manager for review prior to being given the opportunity to code appeals on their own. By using in-house coders, answers that arise from coding questions can be easily shared to ensure a reliable and consistent coding strategy. In addition, systematic collaboration on coding questions provides immeasurable added value to the data quality.

As the last step in the collection of appeal-level data, all coding information will be reviewed by a project staff member who did not code the appeal. Following this secondary review, the data will be entered through a data entry screen by another trained staff member. The data entry program will use the web-based software developed by the project team and will incorporate extensive logic checks restricting erroneous data. Once the data have been entered, it will be transferred through the web based software into an SPSS database in which additional cleaning checks will be run to eliminate any remaining errors.

3. Methods to Maximize Response Rates

The universe of the nation’s 143 appellate courts are known to the data collection agent (NCSC) from its work on the Court Statistics Project as well as its relationships with various appellate court affiliated organizations including the National Conference of Appellate Court Clerks (NCACC), the Conference of Chief Justices (CCJ), Council of Chief Judges of the State Courts of Appeal (CCJSCA), and the Conference of Appellate Technology Officers (CATO). NCSC staff can take advantage of longstanding relationships with the offices of various court leadership organizations and their staff to further ease data collection. In addition, letters of support offered by the abovementioned organizations will be followed, during the project, by letters from those organizations to their members soliciting their full participation and cooperation. The annual meetings of those organizations as well as the e-newsletters of those organizations will be used to reinforce the importance of this effort to collect state appellate court data.

4. Testing of Procedures

The SSCCA data collection forms were pre-tested prior to data collection. Several respondents completed the data collection forms and provide feedback in terms of clarity and accuracy. In addition, an advisory board met to consult on the overall substance of information collected and the format in which questions are asked on the data collection form. Appropriate revisions and modifications were made to these data collection forms based on this feedback.

One of the major revisions concerned the types of appeal being coded for the SSCCA. The results of the pilot test showed that both post-conviction and interlocutory appeals were extremely difficult to code. The coding forms developed for the pilot test proved insufficient to capture the complexity of these cases, and coders found that it took approximately twice as long to code them as compared to direct appeals. Due to this difficulty, the coding form was changed so that only direct criminal appeals would be coded.

The obtainability of another data variable, whether or not an “Anders” brief was filed, was also tested during the design and development project. An Anders brief is a request filed by a court-appointed attorney to withdraw from the appeal of a criminal case because of his/her belief that the grounds for the appeal are frivolous (named after Anders v. California, 1967). Following the pilot test of the data collection forms it was determined that, while the majority of states have Anders-like briefs, not all of those states identify the brief as such, requiring the coder to determine if the brief that was filed contained the characteristics of an Anders brief. Due to the subjective nature of coding this data element, the data collection forms and instructions were revised to reflect the removal of the Anders brief question.

5. Contacts for Statistical Aspects and Data Collection

The prosecution and adjudications staff at the Bureau of Justice Statistics, along with staff from the National Center for State Courts, take responsibility for the overall design and management of the SSCCA data collection, including the development of the questionnaire spreadsheets and the analysis and publication of the data.

1. BJS contacts include

Duren Banks, Chief

Prosecution and Adjudications Statistics Unit

Bureau of Justice Statistics

810 7^th Street, NW

Washington, D.C. 20531

(202) 307 – 0765

Thomas H. Cohen, Statistician

Prosecution and Adjudications Statistics Unit

Bureau of Justice Statistics

810 7^th Street, NW

Washington, D.C. 20531

(202) 514 – 8344

2. Persons consulted on statistical methodology:

James Green, Senior Statistician

Westat

Main Campus
1650 Research Boulevard
Rockville, MD 20850

(301) 251-1500

3. Persons consulted on data collection and analysis:

National Center for State Courts staff

Nicole Waters

Senior Court Research Associate

National Center for State Courts

300 Newport Avenue

Williamsburg, VA 23185

(757) 259-1516

Shauna M. Strickland

Senior Court Research Analyst

National Center for State Courts

300 Newport Avenue,

Williamsburg, VA 23185

(757) 259-1511

State appellate courts experts

Thomas E. Baker

Professor of law

Florida International University College of Law

University Park, RDB 2038

Miami, Florida 33199

John Doerner

Principal Court Management Consultant

National Center for State Courts (previously Clerk of Court at Colorado Court of Appeals)

707 17th St
Denver, Colorado 80202

Peter Haas

Director of Technology

Supreme Court of Louisiana, (representing the Conference of Appellate Technology Officers)

400 Royal Street

New Orleans, LA 70130

Thomas D. Hall

Clerk of Court

Supreme Court of Florida (representing the National Conference of Appellate Court Clerks)

500 South Duval Street
Tallahassee, FL 32399-1925

John Olivier

Clerk of Court

Supreme Court of Louisiana (representing the National Conference of Appellate Court Clerks)

400 Royal Street

New Orleans, LA 70130

1 See footnote 2 for information about the types of aggregate count data available from the Court Statistics Project.

2 It should be noted that there is no direct link between the number of states and appellate courts, especially at the intermediate level. While smaller states might have only one intermediate appellate court for the entire state, many large states have several intermediate appellate courts that cover various parts of a particular state. For more information about the organizational characteristics of appellate courts, please see State Court Organization 2004 at http://bjs.ojp.usdoj.gov/index.cfm?ty=pbdetail&iid=1204.

10