OMB Passback Questions

Questions about the Survey of Stat Court Criminal Cases

1. There is a reference to Attachment 1 in the first paragraph of A.2, but this does not appear to be in ROCIS. Please upload a copy of this document. Also, please provide a copy of the design and development report referenced in footnote 12 on page 6.

The National Center for State Courts implementation plan for the “Survey of State Court Criminal Appeals Design and Development” project has been uploaded into ROCIS as requested.

2. Please update A.8 to include the dates of the publication of the notices in the Federal Register, and note that an attachment contains the comments and responses.

The supporting statement has been modified to include the following language: “BJS published the 60 day notice in the Federal Register on January 31, 2011 and published the 30 day notice in the Federal Register on April 4, 2011. The 60-day notice elicited comments which were addressed in detail. The comments and responses are included in an attached document which has been uploaded in ROCIS.”

3. In A.10, please clarify the second sentence by changing the phrase will be “kept confidential” to “anonymized.” Also, the last paragraph on ICPSR refers to respondent confidentiality, which does not appear to apply in this case since you are drawing information from public records.

The supporting statement has been revised as requested. In addition, the language referencing ICPSR has been deleted from the supporting statement.

4. In A.12, please include in the burden estimate the time that court personnel would be taking to provide access to the needed information when NCSC staff will be traveling to the courts; this is currently listed as zero.

The supporting statement has been revised to account for the additional burden hours required to complete data collection for courts requiring onsite travel. The revised burden hour section of the supporting statement is included in the response below.

Estimate of Respondent Burden (Section #12)

The SSCCA will collect data on a national sample of about 5,000 criminal appeals concluded in all of the nation’s 143 intermediate appellate courts and courts of last resort in 2010. Given the current budget climate and limited resources available to the appellate courts, appellate court staff will not be asked to code the sampled appeals. As such, project staff from the NCSC will code the data elements and case documents used to gather the information required for this project. Rather than coding appeals, the respondent burden will involve providing the necessary materials to NCSC staff for the purposes of data coding.

The burden hour computation involves both sample list generation and case level data collection. Each of the nation’s 143 intermediate appellate courts and courts of last resort will be asked to generate a sample of all their direct criminal appeals disposed in 2010 from which a probability sample can be drawn for the SSCCA. It is estimated that it should take 3 hours for each of the nation’s 143 appellate courts to generate an appropriate sample list for a total of 429 hours (143 appellate courts * 3 hours per court = 429 hours).¹

Once the sample lists are completed, the burden hour component will involve case level data collection. As previously stated, the burden hours will not be associated with coding data collection instruments but with providing the necessary materials for NCSC staff to engage in data collection and coding. There are three major sources of information NCSC staff will use to code information on criminal appeals including the legal briefs submitted by the parties, the court docketing information, and the opinions produced by the courts. The appellate court briefs are the legal documents submitted by the defense attorneys and prosecutors arguing why a particular case should or shouldn’t be overturned on appeal; the docketing information includes key case dates such as the dates of filing, oral argument, and appellate court disposition; and the opinion includes the appellant’s court decision to affirm, modify, or reverse a case. The legal briefs, dockets, and opinions are crucial components of the SSCCA data collection and the degree that courts make this information accessible online are essential to burden hour computation.

According to the criminal appeals design and development project, the nation’s 143 appellate courts fall into three broad categories in terms of online data availability. About a quarter of appellate courts either have all or none of their legal briefs, docketing, and opinion data available online. For these courts, NCSC staff will either utilize the internet to collect all requisite SSCCA data or will travel to these courts and code the cases remotely. Estimates from the criminal appeals design and development project showed that about 36 of the nation’s 143 appellate courts fall into the categories of either having all (26 courts) or none (10 courts) of their data available online for data collection. No burden hours will be involved for the 26 courts in which all data will be collected through online systems. For those 10 courts where onsite travel is required, it is estimated that court staff will expend an average of 2 hours assisting NCSC personnel in pulling case files or providing training on the operation of their case management systems. Thus, the 10 onsite data collection courts will require an estimated 2 hours per court for a total of 20 hours to complete data collection.

The remaining 107 appellate courts vary in terms of online data availability and will require a combination of online data collection and submission of copied case materials to NCSC for complete data collection. For example, some of these courts provide online access to opinions and legal briefs, but not to docketing information, while others offer online accessibility for dockets and opinions but provide no online access for submitted briefs. Among these appellate courts, the burden will involve either copying the necessary information in the legal briefs and opinions for submission to NCSC or in cases where docketing information is being requested, submitting the docketing information to NCSC through administrative data transfer.² According to the criminal appeals design and development project, an estimated 90 courts will provide copies of submitted legal briefs and 14 will provide copies of court opinions. Assuming 35 appeals per court (5,000 appeals / 143 courts = 35 appeals) and 10 minutes to copy each legal brief or opinion for mailing to NCSC, the total burden hours to copy these paper documents for each court should be about 6 hours for the legal briefs and 6 hours for the opinions (35 appeals * .17 hours per opinion/brief = 6 hours). In addition to providing copies of legal briefs and opinions, an estimated 57 courts will provide data extracts of docketing information and it is estimated that each of these courts will require 3 hours to provide the necessary docketing information.³

Therefore, it is estimated that a total of 795 hours will be needed for the 107 appellate courts with limited internet accessibility to provide the documentation in the form of mailed legal briefs/opinions or docket extracts to complete the SSCCA data collection. The 795 number is calculated by first computing the total burden hours appellate courts need to provide copies of submitted legal briefs (90 courts * 6 hours per court to provide copies of submitted legal briefs = 540 hours); and secondly, by computing the total burden hours for providing data extracts of docketing information (57 courts * 3 hours per court to provide extracts of docketing information = 171 hours); and thirdly, by computing the total burden hours for providing copies of court opinions (14 courts * 6 hours per court to provide copies of court opinions = 84 hours). Hence, 540 hours for providing copies of submitted legal briefs + 171 hours for providing data extracts of docketing information + 84 hours for providing copies of court opinions = 795 hours.

When the burden hours for sample list generation is added, the total burden hours for the SSCCA project sums to 1,244 hours (20 hours for onsite data collection + 795 hours to provide necessary case documentation + 429 hours for sample list generation = 1,244 hours).

The burden hours for the SSCCA 2010 project are further summarized in the following table.

5. The table on page 2 of part B is very helpful; however, it seems that BJS will be making separate estimates for COLR and IAC, and even subgroup of 300 cases in both the COLR and IAC samples. If so, please provide the precision for these estimates. Also, the last paragraph on page 2 refers to analyses comparing certain types of criminal cases (e.g., person versus property). Please provide the minimum detectable effects for these and similar analyses you plan to conduct with 80% power and alpha =.05.

The following discussion has been added to Part B of the supporting statement for the purpose of providing further details about sampling precision.

To reiterate, BJS plans to make estimates for four major domains, as follows:

1) Death penalty cases (expected sample size = 234 cases; a census)

2) Intermediate appellate courts (IAC) cases (expected sample size = 3,060 cases)

3) Courts of last resort (COLR) cases (expected sample size = 1,800 cases)

4) Heard by both courts - COLR / IAC cases (expected sample size = 300 cases)

These four kinds of cases will be sampled at four distinct rates (i.e., 100%, 4.25%, 5.39% and 27.67% (i.e., conditional on selection into the COLR sample) respective to 1…4 above) within the two kinds of courts (COLR, IAC).

Since the death penalty cases are intended to be a census, no precision table will be given below for that specific domain.

The precision offered by an overall responding sample size of 4,860 cases is, as follows:

Precision offered by 4,860 responding cases

Percentage	N	Standard Error	LCI	UCI
30.0%	4,860	0.00657	28.71%	31.29%
40.0%	4,860	0.00703	38.62%	41.38%
50.0%	4,860	0.00717	48.59%	51.41%
60.0%	4,860	0.00703	58.62%	61.38%
70.0%	4,860	0.00657	68.71%	71.29%

The standard error of the percentage is based on the percentage itself and the sample size. The lower and upper confidence interval end points (LCI, UCI) give the confidence interval for the estimate of the percentage.

The precision offered by a responding sample size of 3,060 (i.e., the IAC) cases, is as follows:

Precision offered by 3,060 responding cases

Percentage	N	Standard Error	LCI	UCI
30.0%	3,060	0.00828	28.38%	31.62%
40.0%	3,060	0.00886	38.26%	41.74%
50.0%	3,060	0.00904	48.23%	51.77%
60.0%	3,060	0.00886	58.26%	61.74%
70.0%	3,060	0.00828	68.38%	71.62%

The precision offered by a responding sample size of 1,800 (i.e., the COLR) cases is, as follows:

Precision offered by 1,800 responding cases

Percentage	N	Standard Error	LCI	UCI
30.0%	1,800	0.01080	27.88%	32.12%
40.0%	1,800	0.01155	37.74%	42.26%
50.0%	1,800	0.01179	47.69%	52.31%
60.0%	1,800	0.01155	57.74%	62.26%
70.0%	1,800	0.01080	67.88%	72.12%

The precision offered by a responding sample size of 300 (i.e., the COLR / IAC) cases is, as follows:

Precision offered by 300 responding cases

Percentage	N	Standard Error	LCI	UCI
30.0%	300	0.02646	24.81%	35.19%
40.0%	300	0.02828	34.46%	45.54%
50.0%	300	0.02887	44.34%	55.66%
60.0%	300	0.02828	54.46%	65.54%
70.0%	300	0.02646	64.81%	75.19%

BJS will also aim to test whether certain types of criminal cases (e.g., person vs. property) are more likely to result in reversals or whether the percent of reversed cases varies by whether the appeal resulted from a sentence or conviction. The minimum detectable effects for the person vs. property reversal analysis, assuming an overall responding sample size of 4,860 cases, alpha =.05, a desired 80% power, a person v/s property v/s other distribution of 46%, 24% and 30% (Time on Appeal, Roger Hanson (1996). National Center for State Courts: Williamsburg, VA), and an underlying base reversal rate of 50% for person cases (n = 2,236 or 46% of 4,860) is +-5.1%. Hence, the reversal rate for property cases (n = 1,166 or 24% of 4,860) would have to be greater than or equal to 55.1%, or less than or equal to 44.9% for there to be a statistically detectable difference between the reversal rates for person and property cases.

6. Do you have an estimate of the design effect for this sample design?

The following discussion has been added to Part B of the supporting statement for the purpose of providing further details about design effects.

The criminal appeals study proposes to use a stratified, single stage probability sample of appeals within all 91 intermediate appellate courts (IACs) and all 52 courts of last resort (COLRs), yielding a total responding sample size of 4,860 cases (3,060 cases within the IACs, 1,800 cases within the COLRs). Given this sample design, the usual design effects due to clustering do not apply; however, a differential weighting effect does apply given the relative oversampling of COLR cases (which would be 1,353 cases under strictly proportional allocation) and under-sampling of IAC cases (which in turn would be 3,507 cases under strictly proportional allocation). We estimate this differential weighting effect to be approximately 1.06, which implies an overall effective sample size of approximately 4,590 (i.e., 4,860 / 1.058).

Note that within the four major domains listed in Question #5 above, each sample is an equal probability sample, and thus no differential weighting effect (and therefore no design effect) applies.

7. Please clarify how you will be defining the nonresponse weighting cells described on page 3 of Part B.

The following discussion has been added to Part B of the supporting statement for the purpose of providing further details about non-response weighing.

We expect very little non-response (due mostly or entirely to courts that cannot or do not participate) given the National Center for State Courts’ (NCSC) excellent working relationship with the state courts. Nonetheless, in such instances an attempt should be made to identify other, similar courts from which additional cases can be selected, or within which non-response adjustments can be made to existing sampled cases in order to compensate for the non-responding court(s). Similar courts can be grouped into non-response adjustment cells or otherwise identified by type of court (COLR, IAC), state or other criteria (e.g., states with no IAC) which affect the type and volume of appeals cases.

8. In B.3, there are references to support letters from various organizations. Can you please provide copies of these letters?

The letters of support have been uploaded into ROCIS as requested.

9. Is there a report from the pretesting that was described in B.4?

Results and lessons learned from the pilot test were described in the design and development report. Please see pages 5-7 of the design and development report for a summary of the pilot test results.

10. In the “letter of cooperation,” there is a reference to an electronic template for the list of cases, but this does not appear to be in ROCIS; please upload it. Also, the first paragraph on the second page of this letter refers to a database on NCSC’s website that NCSC staff may contact someone to obtain additional information or to verify information obtained from public documents; however, it was not clear whether this was included in the burden in A.12.

The attached template for the appeals list requests has been uploaded into ROCIS.  Please note that there is a separate tab for each of the four types of appellate courts including intermediate appellate court (IAC) with no death penalty review, intermediate appellate court (IAC-DP) with death penalty review, court of last resort (COLR) without death penalty review, and court of last resort (COLR-DP) with death penalty review.

The reference to a database on NCSC’s website was from an earlier draft of the contact letter and should not have been included in the ROCIS documents. A revised contact letter has been uploaded into ROCIS which does not mention the verification of pubic information. NCSC has plans on creating a profile database of appellate courts; however, since this information can be collected and verified online without appellate court involvement, it was decided that the appellate courts would not be contacted and no burden hours requested for this part of the project.

1 All burden hour estimates were derived from the pretests and associated project activities of the criminal appeals design and development project.

2 NCSC will pay all expenses associated with copying and mailing the necessary materials. An administrative data transfer includes transmission of documents in several ways including a data export into a spreadsheet/ASCI format or screen shots either printed/saved electronically in PDF.

3 The number of courts with partial online accessibility will not sum to 107 because these courts overlap in terms of the types of data being requested.