ASPP2017 Supporting Statement Part B

B. Collection of Information Employing Statistical Methods

1. Universe and Respondent Selection

The Annual Surveys of Probation and Parole (ASPP) are designed to collect all probation and parole data from community-supervising jurisdictions within each state. The universe includes all federal, state, and locally administered probation and parole departments. Information is collected from central reporters within each state wherever possible, to reduce the burden on individual agencies. For parole, there are 52 respondents: 50 central state reports, the District of Columbia and the federal system. For probation, there are approximately 470 respondents: 33 central state reporters and about 435 separate city, county, or court reporters. The state agency in Pennsylvania also reports data for 65 counties, the District of Columbia self-reports, and the data for the federal system are obtained indirectly from the Administrative Office of the U.S. Courts through BJS’ Federal Justice Statistics Program.

In an effort to review the coverage of the Annual Probation Survey, in 2014 and 2015 collection years, three questions were added. One question asked respondents to specify the probation agencies for which they provide information. To facilitate the process, a preliminary list based on information collected through the 2014 Census of Adult Probation Supervising Agencies (OMB No. 1121-0339; Expiration 01/31/2016) was provided of all independent probation agencies in their state known to supervise adult probationers. Respondents were asked to delete agencies that were no longer in operation. The second question asked respondents to include the name and location of any probation agency that was not listed.

The third item asked the respondent to mark a checklist to indicate the level(s) of court responsible for placing adults on probation in the agencies for which they reported. A state-specific checklist of all levels of state courts responsible for criminal proceedings which might result in adults being placed on probation was provided, based on charts published by the Court Statistics Project of the National Center for State Courts (http://www.courtstatistics.org/Other-Pages/State_Court_Structure_Charts.aspx). These charts were last updated in 2010, but have been very stable over time.

BJS used this information to determine whether any probation agency defined as eligible was erroneously excluded and if the population reported by an agency was also reported by any other agencies. In addition, the information was used to check whether each level of court responsible for placing adults on probation supervision in each state was acknowledged by respondents. Court types that were unmarked could inform on probation supervising agencies that may be missing from the survey in each state.

Following the review, over 5,000 additional agencies were identified as potentially eligible for inclusion on the ASPP frame. Extensive research is needed to determine whether these agencies should be included and BJS anticipates that additions to the frame will happen over the coming years. By the fall of 2018, BJS plans to have a list of agencies by state to add to the frame. BJS will submit a modified burden estimate to OMB to reflect changes from the frame expansion as necessary.

BJS continues to use additional methods to ensure the accuracy and the completeness of the population frame for each survey:

• Agency staff provide information about newly formed, merged, and closed supervising agencies while data collection is ongoing via data retrieval phone calls and emails. This information is used to update the frame prior to the start of each data collection year.

• Close attention is paid to unexplained changes in the total population that occur from the end of one year to the beginning of the next, and large increases or decreases in the total population during the current reporting year. During survey administration, a comparison is made between the previous yearend population to the reported beginning current year population and, if there is a difference of 10% or greater, respondents are prompted to review their data. They are then asked to enter a reason for the discrepancy between the populations over two days.

Following data submission, all data are reviewed. Probation agencies with populations of 100 or more and parole agencies of any size, whose previous yearend population differs by more than 5% from that of their reported beginning year population, are flagged for review and potential follow up. This also occurs for probation agencies with a population less than 100 a 10% or greater population difference. RTI also reviews the information provided by agencies when January 1 to December 31 growth for the current reporting year exceeds 10%.

During follow-up, RTI uses open-ended probes to determine the reasons for differences in yearend to beginning year population or in the current reporting year. Differences may be explained by a variety of reasons, such as a data entry error, a reporting method change, a change in the agency’s responsibility (e.g., an agency has taken responsibility for probationers or parolees who were previously supervised by another agency), or, in the case of within-reporting year change, to genuine growth or decline of the population.

Over the past 2 years of the ASPP surveys, these methods have enabled BJS to achieve a minimum survey response rate of 90%. In 2015, the response rate for the Annual Probation Survey was 92% of surveyed agencies (representing 99% of the 2015 yearend probation population) and the response rate for the Annual Parole Survey was 100% (table 1).

2. Procedures for Collecting Information

Collection Procedure

BJS emphasizes the web as the primary mode of data collection. Hardcopy forms are sent to respondents upon request only. To draw attention to the ASPP collection in advance of the formal request to participate, a pre-notification letter is mailed and emailed to agencies in early November (Attachment 12). The letter provides information about the purpose and importance of the surveys as well as the type of information to be requested so they can plan to retain the yearend information that they will need. A designation form is included so that the agency head can select the most appropriate person to respond to the survey.

In December, all agencies receive a survey invitation letter requesting that they complete the survey on the web (Attachment 11). The letter explains the importance of the survey and provides a link to the most recent BJS Probation and Parole in the United States bulletin, states that participation is voluntary and thanks them for their involvement. Each agency is provided with a unique user ID and password to securely access the survey website to complete the questionnaire.

After this invitation, other communications inform respondents of the status of data collection or serve to remind them to respond. These include the following:

• Automatic thank-you emails are sent to those that have submitted their web survey (Attachment 15).

• Three reminder messages are sent to non-respondents throughout the data collection period.

• The first is sent via email to alert respondents of the impending survey due date (Attachment 14).

• The second is sent via USPS and e-mail a week after the survey due date (Attachment 16). BJS will investigate the effects on response rates and response speed depending on mode in RY2017 only. Agencies will be randomly assigned to one of two modes: half of the agencies will receive paper forms in the USPS mailing while the other half will only receive an email.

• The third is sent three weeks before the final cutoff of data collection, around the third week of April (Attachment 17).

• Telephone calls, as a reminder to non-respondents, are made to non-respondents immediately following the survey due date. The scripts are tailored to the size, type, and reporting history of the agency (Attachment 18).

• BJS instituted a practice of sending a closeout letter towards the end of data collection. This letter describes the status of the agency’s submission. There are three versions of the closeout letter: no data, partial data, or data that required clarification (Attachments 22, 23, 24).

• Additional follow-up is conducted as needed with non-respondents that indicate they need more time to provide data. Follow-up contact by telephone is attempted to resolve data discrepancies and obtain answers to items left unanswered in the survey (Attachment 18).

Within 2 weeks of survey submission follow-up activities begin. If critical items are missing or inconsistent, such as the beginning year or yearend population or the number of entries to or exits from supervision, staff contact respondents to determine if they can provide estimates or explanations for inconsistencies (see Attachment 19). Staff work with the respondents to estimate missing information if it cannot be easily provided, making sure to obtain agreement from the respondent before disseminating data containing any revisions (see Attachment 20).

Within the first four weeks of the start of the data collection period, preliminary analysis begins. RTI staff check the data for out-of-range values, missing data, and other types of responses that need data editing/cleaning. These preliminary analyses are undertaken while data collection is still in progress to provide adequate time for follow-up clarification calls.

Imputation Procedures

BJS has developed several imputation methods to estimate January 1 and December 31 populations as well as entries and exits if respondents are unable to provide any of the key information. For unit non-response, a combination of the population, entry and exit imputation methods are applied. When the January 1 probation population is missing, the December 31 population from the prior year is carried over. When the December 31 probation population is missing and January 1, exits, and entries from the current year are also missing, December 31 is populated with the last reported December 31 number.

BJS uses three methods of ratio estimation to impute probation entries for agencies not reporting these data. The first method is used to estimate entries for probation agencies that do not report entries in the current year but did report in the prior year. BJS estimates probation entries in the current year by using the ratio of entries in the prior year to the agency’s prior year probation population on January 1, and applying that ratio to the agency’s current year January 1 population.

A second method is used to estimate current year probation entries for agencies that do not report entries in either current or prior years and are in states with agencies of similar size. The ratio of prior year entries to the prior year January 1 population among reporting agencies of similar size within the state is applied to the January 1 population used to estimate the number of entries for non-reporting agencies.

A third method is used to estimate probation entries for agencies that do not report entries in either current or prior year and do not have similar sized agencies in their state. The ratio of prior year imputed entries to the prior year’s January 1 probation population and applying that ratio to the agency’s current year January 1 population.

To estimate parole entries for parole agencies that do not report entries in the current year but were able to report in the prior year, BJS calculates the ratio of entries in prior year to the agency’s prior year parole population on January 1, and applies that ratio to the agency’s current year January 1 population.

A single method is used to estimate probation and parole exits. For both probation and parole, BJS adds the agency’s current year estimated entries to the agency’s population on January 1 of the current year and subtracts that estimate from the population on December 31 of the current year.

The specific methods detailed above, and the jurisdictions to which they apply, are documented in the “Methodology” section of reports in the series Probation and Parole in the United States¹. The imputed values are used for all analyses and reports published by BJS. Imputed values are flagged as such in the files that are sent to NACJD (http://www.icpsr.umich.edu/icpsrweb/NACJD/index.jsp).

3. Methods to Maximize Response

BJS employs several techniques to maximize response rates. They include—

• Contacting the agencies prior to the start of data collection and making frequent contacts during the data collection period to solicit participation.

• Sending web survey invitations which include login instructions for the web survey in both hard-copy through the USPS and in electronic format.

• Making it easy for agencies to participate by providing technical support and other help with the survey as needed, offering a response mode other than web if requested, and providing respondents with real-time online data checks to add efficiency to the response process.

• Engaging respondents in the data collection process by highlighting BJS reports that provide information central to agency needs (see http://www.bjs.gov/index.cfm?ty=tp&tid=15#pubs).

• Analyzing response patterns to determine the most effective methods for contacting and following up with agencies.

• Providing the CJ-8A as a data collection option to smaller probation agencies. This has been shown to improve overall data quality (see section A, item 5, “Impact on Small Businesses or Entities/Efforts to Minimize Burden” for more information).

BJS monitors the progress of the data collection by reviewing data about respondents and non-respondents, their response characteristics, and their communications throughout the data collection period to inform and enhance non-response follow-up. BJS currently has real-time access to the following data:

• Agency contact information (e.g., names of agency heads and designated respondents, street and email addresses, telephone numbers);

• Individual files containing the image of each submitted survey from the current year, including notes provided by the respondents;

• Mode and date of survey submission;

• Notes describing contacts with agencies as well as follow-up efforts; and

• Statistics on the current year’s overall response rates and the response rates for each survey type.

During the collection cycle, the data are analyzed to assess response patterns (e.g., whether the same respondents are consistently late responders) and missing data on submitted forms, and to develop strategies to address the timeliness and completeness of data submissions.

Tailored follow-up timelines using the response patterns are put into place to not needlessly contact a respondent when their time of submission can be predicted. For example, data collected labels respondents as on time, up to 1 month late, 1 to 3 months late, and more than 3 months late.

Unit Non-Response

As seen in table 1, survey response rates for the past 2 years are 100% for parole respondents and at least 90% for probation respondents. To publish national totals of people supervised on probation BJS developed strategies to impute missing data for key items for agencies that do not respond. These items include the beginning of the year count, total entries, total exits, and the end of year count. Imputation for the end of year population count continues to be very low and concentrated in agencies who supervise very small populations. In 2015, only about 36,000 people in 44 agencies were imputed in a population of over 3.7 million.

Item Non-Response

Rates of item nonresponse on the parole survey vary, with maximum sentence continuing to be the largest nonresponse (table 2). Item nonresponse rates for the probation surveys have been higher than for the parole surveys. Type of entry to probation and status of supervision continue to have the largest non-response rates on the probation survey.

In 2015, BJS received OMB approval through a generic clearance (OMB control number 1121-0339) for questionnaire development. BJS conducted two rounds of debriefing interviews with 28 data providers for the 2014 ASPP and 15 heads of probation and parole agencies. The goal of the interviews was to evaluate the content and focus of the ASPP to ensure that the survey continues to accurately measure and describe the size, composition, and changes in the probation populations over time, while assessing the availability of data from respondents. Using the results of these interviews, BJS is working to redesign the CJ-7 and CJ-8 questionnaires for the 2018 collection to reduce item-nonresponse and burden on respondents while still being able to produce national statistics of the probation and parole populations. Findings from these interviews and from preliminary cognitive work in 2017 will be combined to support changes to the current surveys. BJS plans to submit a request for clearance under the generic clearance agreement OMB Number 1121-0339 to conduct a pilot test for the revised surveys in 2018.

BJS will continue to work to address both unit and item nonresponse by working with respondents to obtain more timely data submissions and to identify the reasons for unit non-response. In the last 15 years, nonresponse rates have remained flat across all types of data. During survey redesign, BJS will eliminate questions with high nonresponse rates and work to develop a survey that provides reliable statistics available for collection from respondents.

4. Testing of Procedures

During the 2016 collection, BJS performed non-response follow-up using a stratified approached based on the date of submission in the previous year. If a respondent had a very late response in 2015, they were not contacted right after the due date. This resulted in fewer phone calls, but did not affect the response-rate. After the conclusion of each year of data collection, BJS reviews the effectiveness of each method of contact with the respondents.

In fall 2017, BJS plans to submit a request for clearance under the generic clearance agreement OMB Number 1121-0339 to conduct outreach to assess the eligibility of missing probation supervising agencies in the ASPP frame. These 5,000 agencies were discovered through information gathered from the CAPSA collection, the added coverage questions from 2014 and 2015, and research into the structure of probation in each state.

5. Contacts for Statistical Aspects and Data Collection

The Correction Statistics Unit at BJS takes responsibility for the overall design and management of the activities described in this submission, including fielding of the survey, data cleaning, and data analysis. BJS contacts include:

Danielle Kaeble, Statistician

Bureau of Justice Statistics

U.S. Department of Justice

810 Seventh Street, NW

Washington, DC 20531

[email protected]

202/305.2017

E. Ann Carson, Ph.D., Acting Chief

Bureau of Justice Statistics

U.S. Department of Justice

810 Seventh Street, NW

Washington, DC 20531

[email protected]

202/616.3496

1 See Attachment 4, Probation and Parole in the United States, 2015; other reports in the series are available on the BJS website at http://www.bjs.gov/index.cfm?ty=pbse&sid=42.