Supporting Statement Part B final

Part B. Collection of Information Employing Statistical Methods

1. Universe and Respondent Selection

Scope of CAPSA

The Census of Adult Probation Supervising Agencies (CAPSA) is designed to identify all public federal, state, and local adult probation agencies that supervise adult probationers and/or have some discretion in the development of policy and defining of procedures for adult probation supervision. The following criteria are used to define an adult probation supervising agency: 1) has responsibility for activities related to the supervision of probationers disposed or sentenced by an adult criminal court; 2) supervises adult probationers who are: a) required to regularly report to the agency either in person, via telephone, mail, or electronic means, and b) felons; and 3) is independent, meaning the agency has authority and/or operational responsibility to administer adult probation. CAPSA is also designed to identify all private companies that supervise: 1) probationers disposed or sentenced by an adult criminal court; and 2) adult probationers who are required to regularly report to the company either in person, via telephone, mail, or electronic means.

When developing the CAPSA eligibility criteria, BJS consulted with community corrections practitioners and researchers through open-discussion workshops at conferences, meetings, and working group sessions. The field expressed concern about restricting the CAPSA criteria of private companies to those that supervise felons. Most of the community corrections field and the limited literature on this topic claim that private companies do not supervise felons on probation.^1,2,3 If CAPSA were to implement that criterion for private companies, CAPSA would collect very limited information about them. Stakeholders have expressed the need to understand more about private probation (e.g., prevalence, practices, and governmental oversight). The field has asserted that the use of private providers has increased over time as agencies have been faced with shrinking budgets and resources while caseloads continue to remain high. However, there is limited empirical information to provide insight into the current state of private probation in the United States.

The concept of independence will be used to define a public “agency” in CAPSA. One of the goals of CAPSA is to describe the variation in how adult probation is administered. This variation occurs at the level at which decisions about adult probation are determined and this will be the unit of analysis for CAPSA. However, identifying this unit of analysis is difficult to accomplish by simply using the limited amount of information on the roster of potential probation agencies, especially given the complexity of the organization of adult probation (among public agencies).⁴ (See below for more information about the roster.) For example, an agency could be nested within a larger entity and while the larger entity may have some authority/operational responsibility over some aspects of that particular agency, the agency itself may possess a level of independence, such as to establish policies and define procedures for probation supervision, that is critical to the administration of probation and may vary across other agencies in that jurisdiction that may also be nested within the larger entity. The CAPSA approach defines an “agency” as an entity with the authority and/or operational responsibility to make decisions about budgets, staffing, and policies or procedures of adult probation supervision. Agencies need to possess at least one of these characteristics to be classified as “independent”, meaning that CAPSA will identify the lowest level of probation administration in each jurisdiction that has at least some independence. This standardized approach will ensure that the variation in independence is measured consistently across all public entities. Another advantage is that this approach will permit the CAPSA data to be analyzed by level of independence to determine whether the characteristics of agencies and/or their supervision practices are associated with the extent to which they are independent.

Developing a Valid Frame

A “triangulation method” will be used by BJS and Westat to develop a comprehensive list of all entities that fall within the scope of CAPSA. Entities on the roster and those missing from the roster can be identified and validated through cross-referencing/verification and methodological techniques such as triangulation and the referral technique, also known as the snowball method.^5,6 Snowball sampling is a non-probability sampling technique in which existing study subjects identify (or recruit) future subjects among their acquaintances. This technique is typically used in hidden populations for which no population frame exists and subjects are difficult for researchers to locate. CAPSA efforts will rely on a snowball approach as one method to develop the CAPSA frame, as respondents and informants will be used to identify potential agencies missing from the roster, as well as to validate the ones on the roster.

The triangulation method combines several (preferably three or more) research methodologies and sources of information to identify entities for which no frame exists. This allows researchers to be more confident about the results of each method if multiple methods identify the same entity as in or out of scope. To mitigate the risk of any biases associated with each particular method, such as the snowball technique, the literature on this topic suggests that it is important to maximize the sources of referral or reference. BJS and Westat have proposed three distinct methods to deriving a valid and complete census. Combined, these methods utilized about 15 sources of information.

CAPSA Strategy

BJS and Westat will use three methods to develop and validate the list of all public and private adult probation supervising agencies/companies: 1) compile an initial roster of potential eligible agencies/companies using various sources; 2) screen all entities during the data collection process to determine eligibility; and 3) use a snowball approach during the data collection process and calls with an informant in each of the 52 jurisdictions (i.e., 50 states, District of Columbia, and the Federal system) to identify groups of agencies/companies or specific agencies/companies that are not on the initial roster and screen them. (See step “3. Snowball Method” below for more information on the calls with informants in each jurisdiction.) Only those agencies/companies that are found to meet the criteria described above will be eligible for CAPSA.

1. Development of Preliminary Roster Using Multiple Sources

BJS and Westat have worked to develop a preliminary roster of potential CAPSA-eligible entities using available resources. The following sources have been used to develop the roster.

Public agencies and private companies

• Annual Probation Survey (APS) frame⁷

• Commercial databases and directories, such as the American Correctional Association’s (ACA) 2010-2011 Directory of Probation and Parole⁸ and 2011 Directory of Adult and Juvenile Correctional Departments, Institutions, Agencies, and Probation and Parole Authorities

• Federal, state and local government websites, publications, and other sources

• Membership lists and websites of professional associations, such as APPA, Michigan Association of District Court Probation Officers, and the association of Chief Probation Officers of California

• Krauth, B., & Linke, L. (1999). State Organizational Structures for Delivering Adult Probation Services. Longmont, CO: NIC Information Center, National Institute of Corrections, U. S. Department of Justice.

• Information collected through the CAPSA pilot test (2013)

Public agencies only

• Federal government databases and directories such as:

  • Originating Agency Identifier (ORI) database of governmental agencies that are defined as criminal justice agencies

  • U.S. Census Bureau’s Government Integrated Directory

• Subramaniah, R. and Tublitz, R. (2012). Realigning Justice Resources: A Review of Population and Spending Shifts in Prison and Community Corrections. New York, NY: Vera Institute of Justice.

• Rottman, D.B. and Strickland, S.M. (2006). State Court Organization, 2004. Washington, DC: Bureau of Justice Statistics.

Private companies only

• Schloss, C.S. and Alarid, L.F. (2007). Standards in the Privatization of Probation Services: A Statutory Analysis. Criminal Justice Review 32 (3): 233-245.

• Information maintained by APPA about private companies engaged in adult probation

• Private provider websites, such as PRIDE Integrated Services, BI Incorporated, and Salvation Army Correctional Services

At this time, there are approximately 1,800 public agencies and 170 private companies on the CAPSA roster. The roster file was created by first comparing NIC’s report State Organizational Structures for Delivering Adult Probation Services with BJS’s publication State Court Organization, 2004 as well as the Vera Institute of Justice’s publication Realigning Justice Resources: A Review of Population and Spending Shifts in Prison and Community Corrections. The goal was to gain a better understanding, to the extent possible given the resources available, of how adult probation is administered and organized in different jurisdictions. Information from federal, state, and local websites and publications was used to resolve any discrepancies between the three sources, as well as to supplement the three sources. All of these findings were summarized. Then the APS frame was used to start building the roster file. The information gleaned from the reviews and websites was used to identify groups of agencies missing from the APS frame for the purposes of CAPSA. Once they were identified, the NIC report was used as a guide to determine an estimated number of potential eligible agencies within the missing groups.

The roster was then cross-referenced with other electronic files available (e.g. ACA directories, federal government databases and directories) to identify the overlap of entities and new entities. The newly created roster was de-duplicated by examining the various sources for identical or nearly identical addresses, phone numbers, or names of agency/company officials. A conservative approach was used when identifying entities to include on the roster; if any entity appeared on at least one of the files and could not be definitively determined to be a duplicate or out of scope, it was retained on the roster. Federal, state and local websites were searched if groups of agencies still appeared to be missing. The information from these websites was also used to verify the information on the roster, and obtain contact information if necessary and available. This step was also conducted using professional association websites.

To supplement the roster with private companies, BJS and Westat started by reviewing the literature, specifically the NIC report and Standards in the Privatization of Probation Services: A Statutory Analysis. The goal was to learn more about which states are currently known to use private probation providers for adult probation supervision. Developing the roster started with a list of private providers maintained by APPA. It was compared to the literature to determine the gaps. The list was then cross-referenced with other electronic files (e.g., APS frame, ACA directory, and CAPSA pilot data) that included some information about private companies. The same approach used for the public agency roster to identify duplicate entities to omit and entities to retain was also used for the private company roster. If the available information still indicated gaps in coverage, government and private provider websites were searched to attempt to identify the missing companies and contact information, as well as obtain missing contact information for other companies on the roster.

2. Screening during Data Collection

As explained earlier, the determination of whether the entities on the preliminary roster fall within the scope of CAPSA will be based on the responses to a few items on the questionnaire that measure the eligibility criteria described earlier in this section.

Public agencies that have responsibility for adult probation supervision (criterion 1) and are independent (criterion 3) will be eligible for CAPSA. In some states, particular agencies have some oversight of adult probation (criterion 3) and may perform various adult probation functions (criterion 1), but not necessarily supervise adult probationers (criterion 2). Not only may these types of entities establish policies or define procedures for adult probation supervision, they may have administrative (e.g., personnel management and budget preparation) or reporting functions (e.g., data collection and preparation of reports) of adult probation, may conduct training or certification of supervision officers, may issue standards for adult probation for which local supervising agencies must comply, or may distribute funding for adult probation throughout the state. Therefore, these types of entities may have some degree of authority and/or operational responsibility for adult probation in a state, even though the supervision of probationers is decentralized at the local level for which the local entities may also have some level of independence (e.g., also establish policies and/or define procedures for adult probation). These types of state entities are important to understand and it is important to describe this type of state structure given that the level of independence of the supervising agencies in these states may be different (i.e., less) compared to other states where local level supervising agencies are completely independent. This variation in independence may impact how adult probation is administered in one state compared to another state, for example the sources of funding for adult probation or whether specialized caseloads such as sex offenders exist and receive specialized programs or services, and a critical goal of CAPSA is to measure this variation.

Public agencies that meet criteria 1 and 3 and also supervise adult probationers who report regularly and directly to their agency (criterion 2A) and felons (criterion 2B) will be identified as a probation supervising agency in CAPSA, and will be asked additional questions on CAPSA to better understand their policies and practices of adult probation. Private companies that have responsibility for adult probation (criterion 1) and supervise adult probationers who report regularly and directly to their company (criterion 2A) will be eligible for CAPSA.⁹

Those agencies/companies that meet the eligibility criteria will remain on the final frame while those that do not will be flagged as out of scope for the purposes of CAPSA. The analysis of the screener data collected from the ineligible entities will help gain a better understanding about them. In the report that BJS will publish about CAPSA, a description about the ineligible entities will be provided in the context of describing the scope of CAPSA so that the population covered is clearly defined and understood.

While the criteria described above will be used to determine eligibility for the final frame for which BJS will report the CAPSA statistics, different groups of entities will still be asked some of the CAPSA questions even if they do not meet the CAPSA criteria that apply to them. For example, a juvenile probation agency that was added to the preliminary roster of public agencies and during screening is determined to not have any responsibility for adult probation, will still be asked the questions that are intended to enhance and validate the roster (e.g., other probation agencies not on the roster that supervise adult probationers).

Agencies that report they supervise adults on probation but not necessarily felons will be asked to complete the CAPSA questionnaire, but will also be flagged as ineligible on the final frame. As described above, the population of interest for public supervising agencies under CAPSA is the group of agencies that supervise adults on probation for a felony (some of which also supervise misdemeanants). The CAPSA efforts have focused on developing a systematic approach to identifying all of the felony supervising agencies in the United States and that information will be used to assess the coverage of these agencies in APS. However, as explained above, in the course of developing the roster, agencies were added if they could not be definitively deemed out of scope. Sometimes this was because there was conflicting information across sources. For example, for some groups of agencies in states, the NIC report indicated they do not supervise felons but the APS data suggested otherwise; this could be a reflection of a change in structure since the NIC report was published in 1999. Because we adopted a conservative approach in developing the roster, we expect that some agencies on the roster may in fact be misdemeanant-only probation supervising agencies. As described in Part A, Section 2 though, CAPSA is an opportunity to inform the APS efforts by providing more insight, albeit limited, into the organization and administration of adult misdemeanant probation, which is more complex than felony probation supervision in particular states. For example, based on some of the information available, some states are known to be partially centralized in administering adult probation in that adult felony probation is centralized at the state level while misdemeanant probation is decentralized at the local level and occurs at various levels of government such as the county and municipal levels.

When CAPSA began, efforts were devoted to trying to develop a roster of misdemeanant agencies but it became apparent that under the CAPSA design, more current knowledge about the various complex structures of misdemeanant probation would be critical to efficiently develop an initial roster to screen through data collection. Information from CAPSA about adult misdemeanant probation agencies, along with other data available such as through BJS’s Community Corrections Statistics Program (CCSP), will be analyzed to potentially develop systematic approaches in the future to identify the specific agencies that are conducting misdemeanant supervision, and determine how efficient and effective the approaches will be at achieving that goal. The information gleaned through the CAPSA efforts, including the data from a subset of the population of misdemeanant-only supervising agencies, will supplement the efforts through APS to assess coverage of the misdemeanant probation population; this work is currently in the developmental stages.^10,11

3. Snowball Method

The CAPSA questionnaire asks public agency respondents to review a list of probation agencies in their jurisdiction that are on the CAPSA roster and identify additional agencies that might be eligible but are missing from the list. (See Part B, Section 2 for more information on the collection procedures.) Respondents will also be asked if they are aware of private companies that supervise adult probationers in their jurisdiction; positive responses will be followed up by a question asking for contact information (name and county). Those additional entities identified by the respondents will be compared to the roster to determine if they are in fact missing or if they are ineligible. Entities that cannot be confirmed will be added to the roster and asked to participate in CAPSA to screen them for eligibility.

In addition to the sources described above and the CAPSA data collection, BJS and Westat will utilize another source to develop and validate the roster. An informant in each of the 52 jurisdictions will be contacted and asked to supplement and/or confirm the information gleaned from the various sources listed above that were used to develop the preliminary roster. (See Part B, Section 2 for more information on the informant calls.) A critical aspect to developing and validating the roster is a thorough understanding about how adult probation is currently organized and the variation across and within jurisdictions. A firm understanding of the structure is important to determine if particular types of probation supervising agencies may still be missing from the roster. BJS and Westat have worked together, and in consultation with APPA and Dr. Edward Latessa who is a prominent expert in the community corrections field and is serving as a consultant on the CAPSA project, to identify a contact in each jurisdiction that is extremely knowledgeable about the structure and administration of adult probation in their jurisdiction.

Based on the notes resulting from the informant interviews and any supplemental information obtained from the informants, a jurisdiction-specific summary will be developed. The summaries will follow a standardized template to facilitate aggregation of the data nationwide and comparisons across jurisdictions. The findings will then be compared to the contents of the CAPSA roster to help identify types of agencies that might be missing from the roster and to confirm that those agencies contained on the roster are appropriate. If needed, the roster will be revised and any newly identified agencies that are not on the roster and for which eligibility cannot be determined will be added to the pool of agencies that will be screened.

For the purposes of estimating burden associated with this collection and data collection costs, it is important to be able to estimate, to the extent possible given the information available, the maximum potential increase in the roster due to newly identified agencies, which could include newly opened agencies discovered through data collection (step 2 above) or the referral approach (step 3 above). Based on the overlap identified when cross-referencing the APS frame with other available electronic files and supplemental information described above, BJS and Westat estimate that a maximum of an additional 10% of public agencies may need to be screened as a result of the informant calls and the CAPSA data collection to definitively determine their eligibility status. This is equivalent to 180 additional public entities on the roster for a total of 1,980 public agencies.

Based on the information available and the overlap discovered when cross-referencing the private provider list provided by APPA and the other sources, BJS and Westat estimate that a maximum of an additional 20% of potential private probation companies will be identified through the informant calls and data collection. This is equivalent to 34 additional companies to be added to the roster, amounting to a total of approximately 204 private companies to screen through CAPSA. We expect to have to screen a larger proportion of potential private companies compared to the additional potential public agencies because fewer sources were available to develop the preliminary roster of private companies. In addition, private companies are likely to be more volatile than public agencies, meaning that the rate at which there are closures and openings of private companies is likely to be higher than that of public agencies. New private companies may exist by the time CAPSA is fielded.

A Census Compared to a Survey

CAPSA will be a census rather than a sample survey. The reasons for this decision include the following:

• Data are not available regarding the roster from which to draw a sample. And moving to a sample survey with a universe the size of CAPSA’s would not result in significant cost savings given the stratification dimensions needed to capture critical aspects of the universe, such as agency type (e.g., level and branch of government; sector; centralized or decentralized system) and size.

• A census provides BJS with an opportunity to demonstrate how probation supervision varies across and within jurisdictions. Being able to compare agencies is particularly important in light of the variability that exists among these organizations in terms of their structure, methods of supervision, sources of funding, policies, etc.

• Eventually data flowing from this collection will supplement the data from APS. Analyzing the APS data in combination with the CAPSA data would provide a way to better understand patterns and trends in the probation population as they relate to the practices of the supervising agencies.

• A census is preferred because it provides an opportunity to build a foundation for conducting future surveys of probation agencies. Completing the census, for example, will provide the necessary information for producing samples based on a more informed and fuller understanding of how each state and locality administers probation (e.g., agencies that conduct intensive supervision or offer specialized services for mentally ill offenders on probation). Attempts to generate samples of agencies without this crucial information would be incomplete as well as more time intensive and costly.

• While not essential for national estimates, the small increase in effort to conduct a census instead of a sample allows BJS to report on agencies in all 52 jurisdictions. The community corrections field has stressed to BJS that the utility of the CAPSA data would be extremely limited if jurisdiction-level estimates were not possible.

2. Procedures for Collecting Information

Telephone Interviews with Informants

Within the first two weeks after receiving clearance from OMB, Westat interviewers (those who will be assigned to conduct nonresponse follow-up for the CAPSA survey request) will telephone each informant. When possible, Westat will conduct the interview during the initial contact. Otherwise, interviews will be scheduled for a day/time when it would be convenient for the informant. It is anticipated that all interviews will be completed within six weeks after they begin.

During the interview call, the Westat interviewer will use the semi-structured protocol to guide the discussion (see Attachment 8 – State Informant Telephone Interview). The protocol will be tailored for each jurisdiction based on information found in the sources listed in Part B, Section 1. The interviewer will take detailed notes during the discussion. Follow-up questions and probes will be tailored to the unique characteristics of the discussion, but will always focus on the general objectives outlined in the protocol. When appropriate, the interviewer will encourage informants to share information that already exists in electronic form via email; this will minimize the burden placed on informants during the calls. After the call, Westat will send an email message thanking the informant for participating in the call and reminding him/her to provide any promised documentation at his/her earliest convenience. The interviewer will review the notes immediately following each call to ensure that they are complete and accurately reflect the discussion. If necessary, the interviewer will email the informant to confirm or clarify the notes.

CAPSA Survey Request

CAPSA data collection will take place in three phases: Pre-notification and Respondent Identification, Screener and Survey Data Collection, Data Retrieval/Follow-up. The activities associated with each phase are described below.

Phase 1: Pre-notification and Respondent Identification. A survey packet will be mailed via the U.S. Postal Service to the head of each public agency and private company on the CAPSA roster in mid-June 2014, approximately two weeks prior to the questionnaire reference date of June 30. The packet sent to public agency heads will include a cover letter (on BJS letterhead) explaining the goals of CAPSA, survey content, and the voluntary nature of participation (see Attachment 9 – Pre-notification Letter to Public Agency Heads). The packet will also include a letter of support from APPA (on APPA letterhead) and will contain an Agency Information Form (AIF) that the head can use to designate a respondent for CAPSA (see Attachment 10 – APPA Endorsement Letter and Attachment 11 – Public Agency Information Form). A list of survey topics will be included so that the agency head can assign an appropriate staff person to serve as the respondent (see Attachment 12 – List of Survey Topics for Public Agencies). The AIF will ask for contact information for the designee and whether the designee can complete the questionnaire online.¹² Agency heads will be asked to send the completed AIF to Westat via email or fax or provide the information by telephone.

Similar procedures will be used with private company heads. Packets will include a cover letter from BJS describing the goals, survey content, and the voluntary nature of participation (see Attachment 13 – Pre-notification Letter to Private Company Heads). The packet will also include a letter of support from APPA and the Company Information Form (CIF) that collects the same information as that collected on the AIF except that it does not include the question about Internet access (see Attachment 14 – Private Company Information Form). Since the private company questionnaire is brief and provided in hard-copy, the survey packet sent to the company head does not include a list of survey topics.

Agencies and companies that do not return an AIF or CIF in response to the initial correspondence will receive a second packet that includes a modified cover letter and a duplicate AIF/CIF (see Attachment 15 – AIF/CIF Reminder Letter to Agency/Company Heads). Additional prompting of nonrespondents will be done by telephone as needed (see Attachment 16 – Telephone Contact Guide: AIF/CIF Nonresponse Prompt). Telephone prompting of public agency and private company nonrespondents will be performed by Westat or APPA depending on the circumstances. APPA is a subcontractor to Westat for CAPSA and a recognized advocate for the community corrections field. Due to its status as the national association for community corrections professionals, APPA will be a resource for garnering support and encouraging participation in the study.

Phase 2: Screener and Survey Data Collection. Data collection will follow one of three tracks depending on the type of agency.

• Public agencies that indicate the ability to respond online. A survey packet will be mailed via the U.S. Postal Service to each designated respondent. The packet will include a cover letter (on BJS letterhead) explaining the goals of CAPSA and note that the recipient has been designated by the agency head to respond to the survey (see Attachment 17 – Invitation Letter to Public Agency Designees, Web Mode). The letter will also note the voluntary nature of participation and ask participants to provide their responses by August 31, 2014. Instructions for logging into the survey website, using a unique PIN, will appear in the letter. Also contained in the survey packet will be a letter of support from APPA (on APPA letterhead), a list of questionnaire topics, and study definitions (see Attachment 18 – Survey Definitions for Public Agencies). The cover letter will encourage the designee to review these materials prior to logging into the survey in order to simplify/shorten the response task. The fourth and final document included in the packet will be a state-specific listing of public probation supervising agencies included on the roster at the time of mailing (see Attachment 19 – State Example of Listing of Public Probation Supervising Agencies). One of the survey questions asks respondents to report the name and county of any agency that is missing from the roster. Having this list prior to beginning the survey will allow respondents an opportunity to review the contents and collect any information that will be needed to respond to the survey question. Nonrespondents will be contacted by telephone to prompt their participation (via the web) (see Attachment 20 – Telephone Contact Guide: Public Agency Web Survey Nonresponse Prompt). If a designee refuses to participate, Westat or APPA, depending on the circumstances, will contact the agency head to determine whether a different person can be designated. Four weeks prior to the end of data collection, all remaining nonresponse cases will be assigned to the telephone interviewing staff of Westat and attempts will be made to complete the survey by telephone.

• Public agencies that indicate a need to respond by telephone. A survey packet will be mailed via the U.S. Postal Service to each designee. The packet will include the same documents as the packets sent to the other public agency designees. The only difference will be that the cover letter sent to this group will not include instructions for logging into the survey website; rather, these designees will be informed that Westat will contact them to complete the interview by telephone (see Attachment 21 – Invitation Letter to Public Agency Designees, Telephone Mode and Attachment 22 – Telephone Contact Guide: Public Agency Telephone Survey Prompt). If a designee refuses to participate, Westat or APPA, depending on the circumstances, will contact the agency head to determine whether a different person can be designated as the respondent.

• Private companies. Data collection from private companies will be done using a mail questionnaire, with telephone follow-up for nonresponse as necessary. Designees will be sent a study packet containing a cover letter, questionnaire, and business reply envelope (see Attachment 23 – Invitation Letter to Private Company Designees). The cover letter (on BJS letterhead) will explain the goals of CAPSA and note that the recipient has been designated by the company head to respond to the survey. The letter will also note the voluntary nature of participation and ask participants to provide their responses by August 31, 2014. Similar to public agencies, survey nonresponse follow-up will be conducted either by Westat or APPA depending on the circumstance (see Attachment 24 – Telephone Contact Guide: Private Company Paper Survey Nonresponse Prompt).

Phase 3: Data Retrieval/Follow-up. As agencies submit data, Westat will perform quality checks to identify questions that should have been answered and were left unanswered, inconsistencies in reported data across items, and other instances where follow-up is needed. If a case does not present any issues needing follow-up, the agency will be flagged to receive a final thank-you letter (see Attachment 25 – Final Thank You Letter). If the only type of issue presented is missing data, an email message will be sent to the respondent asking that the data be reported through the web survey or by calling Westat (see Attachment 26 – Data Retrieval Email, Initial Contact for Item Nonresponse). A similar email will be sent if the only type of issue presented is a missing list, meaning that the agency responded to a survey question indicating that they will provide a list(s) of probation agencies/companies in their state and Westat has not received the list(s) (see Attachment 27 – Public Agency Data Retrieval Email, Initial Contact for Missing Lists). If an agency submits data that have any internal inconsistencies, the case will be assigned for personal follow-up by a member of the help desk staff. Follow-up will be initiated by email and completed by telephone (see Attachment 28 – Data Retrieval Email, Initial Contact for Other Issues). Using personal contacts in these situations will help ensure that the inconsistency is clearly articulated, that the study staff can assist respondents immediately, and that any problem with the survey itself (e.g., inappropriate question or lack of appropriate response option) can be documented without placing further burden on the respondent. Once all data issues have been resolved, help desk staff will email the respondent a summary of any changes that were discussed and made to their data (see Attachment 29 – Data Retrieval Email, Confirming Changes to Data). A final thank you letter will be mailed to each agency head. Similar procedures will be used with private companies to address item nonresponse and internal inconsistencies.

Agencies that are unresponsive to follow-up efforts will be flagged to receive a closeout letter (see Attachment 30 – Alternative Closeout Letter, Submission but Incomplete Data Retrieval). This letter will be sent to agency heads informing them that we were unable to reach their agency to clarify particular elements of their survey data and that the data collection period has closed. The letter asks agency heads to contact the CAPSA help desk or BJS project manager. Agencies that do not participate in data collection at all, that is, they do not submit any survey data, will be flagged to receive a closeout letter informing them that data collection has ended and to contact the BJS project manager if they are able to submit data (see Attachment 31 – Alternative Closeout Letter, No Participation). Similar procedures will be used with private companies at the end of the data collection period.

3. Methods to Maximize Response Rates

Marketing

BJS and Westat have started and will continue to market CAPSA to garner support and generate interest among stakeholders to maximize response to CAPSA.

• Four documents have been developed by BJS and Westat that are being used to publicize CAPSA and generate interest among key stakeholders. APPA also contributed to the production of these documents to help ensure that they will have the desired effect among both public agencies and private companies.

  • Study announcement. This one-page document describes the motivation for and goals of CAPSA, and includes contact information for BJS and Westat staff that can provide readers with additional information about the study.

  • Study brochure. This single sheet tri-fold brochure includes information about the goals of CAPSA, the types of agencies/companies that will be invited to participate, and the nature of participation (e.g., completion of a survey). The brochure also describes the survey content and potential uses of the data. Again, contact information is provided as well as an overview of the project schedule.

  • Study summary. This two-page document provides more detailed information than the CAPSA announcement or brochure. The summary includes an overview of the project goals, scope, and schedule. In addition, the questionnaire content and data products are described.

  • CAPSA article. The seven-page article includes the most detailed information about many of the topics addressed in the other outreach materials. In addition, the article presents information on the source of the frame and delineates the research questions that will be addressed by the study.

A stakeholder outreach plan has been developed that describes venues that have and will be used to distribute each of these resources. The plan targets international, national, federal, and state associations and groups with the largest number of likely CAPSA respondents. For example, the National Association of Probation Executives will publish the CAPSA article in their April 2014 edition of the journal Executive Exchange. The Florida Association of Community Corrections will include the study summary in email blasts to its officers and distributed the study brochure to its board members in February 2014. The Chief Probations Officers of California will provide the study brochures to its membership at two upcoming meetings in the spring of 2014. The International Community Corrections Association will publish the study announcement twice in June 2014 in its weekly e-newsletter Doing What Works.

• APPA is strongly encouraging participation of its membership in CAPSA. APPA has provided BJS and Westat with a letter of endorsement that will be sent to agency/company heads at the start of the data collection period. In addition, the Association included the study announcement in the program distributed at its winter conference in January 2014; attendees were also provided with the CAPSA study brochure in their registration materials. APPA will also include the study announcement in their electronic newsletter and publish the article on CAPSA in their journal Perspectives.

• BJS will have a page on its website where respondents can find out more information about CAPSA. The CAPSA web page will provide a full description of the project, including the goals and importance of the collection, scope, questionnaire content, a brief project schedule, and planned publications. The web page will also include the materials (e.g., study brochure, study announcement, and article) that BJS and Westat have developed to market the project to the community corrections field.

• BJS and Westat have given presentations and conducted open-discussion workshops about CAPSA with a variety of stakeholders at various venues over the past few years, including workshops at the APPA summer and winter annual conferences, working groups of community corrections professionals at the APPA annual conferences, meetings of APPA’s Board of Directors, workshops at the American Society of Criminology annual conferences, meetings of the Community Corrections Research Network of the National Institute of Justice, and meetings of the State Executives of Probation and Parole Network of NIC. The purpose of these presentations and discussions has been to obtain input and feedback about the content of CAPSA to generate interest among stakeholders, inform future respondents about the key aspects of CAPSA to familiarize them with the collection, demonstrate the importance of the information, and give them an opportunity to provide input and ask questions. In addition, these discussions also led BJS to scale back the CAPSA questionnaires over time to minimize burden and maximize response.

Administration

Westat will undertake various procedures during the administration of CAPSA to ensure high response rates are achieved.

• Survey materials will be provided to respondents in advance of any attempt to complete the questionnaire so that they can be better prepared to answer the survey questions.

• CAPSA participants will be notified about the CAPSA web page on BJS’s website in the survey letters and through the online survey at the login screen.

• The online survey will allow respondents to save their data and exit/re-enter the application at any time, thereby permitting completion at the convenience of the respondents. Similarly, telephone surveys will be scheduled for times that are convenient to respondents and can be done in segments to meet the needs of respondents.

• The online survey includes pop-up instructions for selected questions to assist respondents as they complete the questionnaire, and a list of survey definitions.

• APPA will contact nonrespondents as needed to encourage participation during the data collection effort.

• A help desk will be available to provide both substantive and technical assistance. Respondents can communicate with the help desk by telephone or by email.

In order to promote 100% item completion, Westat will monitor item responses on all surveys as they are completed and submitted. Westat will have a management system linked to the web-based/telephone data collection system that will be designed to flag instances of item nonresponse and invalid responses as surveys are completed. Westat will also flag such instances that appear on paper questionnaires on a rolling basis during the data collection period. The data collection manager will oversee personal telephone or email contacts with individual respondents to clarify missing or invalid responses and to take corrective action. These changes will also be tracked for follow-up, if necessary. Two full-time staff members at Westat will have primary responsibility for the response follow-up. Their efforts will be supplemented on an as-needed basis by the Westat data collection manager, project manager, and project director, by the BJS project manager, and by APPA.

Nonresponse Adjustments

For CAPSA, we anticipate a response rate of at least 90%.¹³ With any data collection, it is typically the case that some of the selected subjects will not respond to the survey request (i.e., unit nonresponse) and some will not respond to particular questions (i.e., item nonresponse), despite the best efforts made to collect all of the data. Nonresponse can occur for various reasons. For example, nonrespondent agencies might choose not to expend the staff resources needed to assemble data and report on the survey. And some respondents may leave items unanswered because their agency or company does not track the information needed to respond or they do not believe that the question or response options can be applied to their unique circumstances. Imputation and weighting adjustments will be used to address concerns of nonresponse bias in the estimates.

To address item nonresponse, imputation offers a number of advantages, including potentially the reduction of bias. We will use an imputation method that exploits available auxiliary data (external to the survey) and the relationships between survey variables through regression modeling. The method first develops a regression model for a variable to be imputed using all available covariates including the survey variables, and the predicted values by the regression model are used to create imputation cells, where hot-deck imputation is carried out cell by cell. Westat’s proprietary software will be used to implement this method. Such a method not only improves the quality of imputation but also helps maintain multivariate relationships of the survey variables. For CAPSA, the public agency frame does not include much auxiliary information except some geographic information (i.e., state) and some state-specific information about the organization and structure of probation supervision (e.g., level and branch of government responsible for adult probation). However, it may be possible to supplement the frame information with information gleaned from respondents in the same state through the CAPSA survey (e.g., proportion with operational responsibility for staffing and proportion using various methods to conduct supervision). And for those individual agencies that participate in APS, we will include data from that survey to impute variables on CAPSA. Auxiliary data for private companies is more limited, since the companies may operate in multiple states and operate in ways unlike public agencies in the same state. Therefore, we will rely exclusively on available survey data (e.g. number of states in which the company operates or use of various methods to conduct supervision) to impute for item nonresponse in this population. Use of such variables for both public agency and private company imputation is advantageous since these data are likely correlated with other key characteristics including types of populations supervised, authority/operational responsibility, and funding sources.

We will use weighting, and more specifically weighting classes, to adjust for unit nonresponse. While we did consider imputation as an option since it is the preferred method for a census such as CAPSA, the appropriateness of using this method depends heavily on the availability of good auxiliary data because imputation of the whole case solely depends on the auxiliary data as no survey items are present. Since such data are not available for CAPSA, weighting to the base weight (which is one for a census) is the better option. Because we have limited auxiliary data, we will form weighting cells by state and sector (i.e., public or private) and adjust the weights of respondents so that they represent the nonrespondents within the same weighting class. This adjusted weight will be further adjusted to benchmark to the known probation population total from APS through jurisdiction-level post-stratification.

4. Final Testing of Procedures

The CAPSA questionnaires and methodology have been developed with input from numerous individual stakeholders and probation agencies. For the past several years, BJS has consulted with a working group of community corrections practitioners at the APPA summer and winter annual conferences about ways to enhance and expand the CCSP. The group is comprised of staff from several state and local probation agencies from across the country, private consultants, researchers, and others working in the community corrections field. BJS has solicited input from members about the topics and constructs that would contribute to a better understanding of how probation supervision is conducted, and asked participants to identify or discuss possible topics or measures that should be included on the CAPSA questionnaires. For example, topics such as organization, risk assessments, funding sources, number of supervision officers, supervision authority, and special populations were identified by the field as critical to covering in CAPSA.

In addition to the working group, development of the CAPSA questionnaire has relied on feedback from public agencies and private companies that participated in a pretest and pilot test (see Attachment 32 – Pilot Test Report). As part of survey development, six public probation agencies participated in a pretest in October – November 2012 and responded to draft questions and provided comments about survey content, terminology, and question wording. Key findings included the following:

• Most study definitions (e.g., probation and supervision) were well understood. However, some terms such as “top-administrative body,” “central office,” and asking if the budget was “federally-funded,” or “state-funded,” posed challenges to respondents.

• Respondents were often unclear about the definition of the focal agency. Issues associated with centralized authority and responsibility for direct supervision made it difficult to answer some questions without a more clear and applicable definition.

• Use of a June 30 reference date worked well for all participants. Agencies maintain the population data needed to answer the questions using that date because it corresponds to the end of their fiscal year and/or because it is a midyear point.

Having incorporated the lessons from the discussions with working group participants and the pretest, a pilot test of both the public agency and private company surveys was conducted from July – September 2013.¹⁴ Thirty-seven public agencies responded to the request to participate. These agencies included state and local level agencies, as well as those from both the executive and judicial branches of government. Seven private companies also responded to the request. Some of the findings from the pilot test indicated:

• Mean questionnaire completion time was 65 minutes for the public agencies and 30 minutes for the private companies.

• Focal agency: Data and follow-up efforts revealed that some respondents experienced confusion about the focal agency/reporting unit for reasons such as partial oversight of the responding agency by larger entities and differences between the responding agency and the larger entity in terms of direct responsibility for supervision activities.

• Questions about agency functions needed to be tailored for both agencies with partial oversight of lower-level agencies and agencies that have some (but not complete) discretion over their own probation supervision procedures. Particular follow-up questions about populations and supervision needed to be based on responses to the agency function questions to ensure the appropriate questions are asked of particular types of agencies and to minimize confusion about the focal agency/reporting unit.

• Questions on branch of government and functions (i.e., administrative, reporting, supervisory, other) seemed to be understood by respondents. The question about level of government posed challenges to respondents in states where the supervision occurs at the local level but the staff are state employees.

• Questions on status of subsidiary agency appeared to pose problems in some cases because respondents were unclear how to interpret “subsidiary”.

• All agencies reported having some level of independence (i.e., decision-making authority or operational responsibility over budgets, staffing, or policies/procedures).

• Methods of supervision: Questions demonstrated the ability to identify a variety of methods used by agencies (e.g., in-person, telephone, email, text, electronic monitoring, and intensive supervision).

• Risk/Needs assessment: List of types of assessment tools (i.e., standardized, agency-developed, staff judgment) was insufficient for public agencies but sufficient for private companies.

• Populations supervised: Examples or more detailed definitions were needed to help respondents accurately report their populations.

• Population counts: Additional instructions for providing population counts were necessary to ensure respondents reported the correct types of probationers based on the CAPSA definitions and goals.

Staff from BJS and Westat reviewed the results of the pilot test and made appropriate modifications to the questionnaires as needed.

5. Contacts for Statistical Aspects and Data Collection

The Correction Statistics Program at BJS is responsible for the overall design and management of the activities described in this submission, including the fielding of the census, data cleaning, and data analysis. BJS contacts include:

Lauren Glaze

Statistician and CAPSA Project Manager

Corrections Statistics Program

[email protected]

(202) 305-9628

Daniela Golinelli, Ph.D.

Chief

Corrections Statistics Program

[email protected]

(202) 616-5164

During the development and design of CAPSA, Westat staff provided input to BJS, specifically knowledge and expertise in the areas of statistical methodology, data collection, and analysis. Westat will continue to provide that type of input throughout the course of CAPSA and will also manage and coordinate the collection of all data. Contacts at Westat include:

Tim Smith

Senior Study Director (for CAPSA)

[email protected]

(240) 314-2305

Monica Basena

Senior Study Director

[email protected]

(301) 294-3805

Heather Tubman-Carbone

Research Associate

[email protected]

(240) 314-2575

David Cantor

Vice President

[email protected]

(301) 294-2080

Roger Tourangeau

Vice President

[email protected]

(301) 294-2828

1 Schloss, C.S. and Alarid, L.F. (2007). Standards in the Privatization of Probation Services: A Statutory Analysis. Criminal Justice Review 32 (3): 233-245.

2 Krauth, B., & Linke, L. (1999). State Organizational Structures for Delivering Adult Probation Services. Longmont, CO: NIC Information Center, National Institute of Corrections, U. S. Department of Justice.

3 The CAPSA pilot data (n=7 for private companies) suggest that some private providers are supervising felons on probation, which could be a recent change. This is an example of how the CAPSA data will provide the field with new, critical information, some of which may refute previous claims.

4 This is not an issue for private companies which is why the “independence” criterion is not necessary to measure.

5 Heckathorn, D.D. (1997). Respondent-Driven Sampling: A New Approach to the Study of Hidden Populations. Social Problems 44 (2): 174-199.

6 Salganik, M.J. and D.D. Heckathorn (2004). Sampling and Estimation in Hidden Populations Using Respondent-Driven Sampling. Sociological Methodology 34 (1): 193-239.

7 Because the goal of APS is to collect aggregate counts of the adult probation population, if a central reporter exists in a state and is capable of providing the population counts for the entire state then BJS collects the information from that one respondent to minimize burden, including in states where the administration of adult probation is decentralized. Therefore, the APS frame does not include all independent supervising agencies with the authority and/or operational responsibility to administer adult probation, which is the unit of analysis for CAPSA.

8 This directory is different than ACA’s annual directory of adult and juvenile correctional departments/institutions and probation and parole authorities, which are not necessarily the same entities conducting community supervision in all of the states.

9 See the discussion in the section “Scope of CAPSA” above about why criteria 2B and 3 do not apply to private companies.

10 In addition to the information collected directly from some misdemeanant-only probation supervising agencies, the CAPSA data can be used to estimate the size of the misdemeanant probation population that is supervised by felony probation agencies, as a subset of these also supervise misdemeanants (CAPSA will ask for aggregate counts of felons and misdemeanants). These CAPSA estimates can be compared to the APS population data of misdemeanants to determine how large the misdemeanant probation population is that is supervised by misdemeanant-only agencies. The information will be available by jurisdiction and can be used to assist in the development of strategies to target particular jurisdictions where misdemeanant-only probation agencies appear to be most prevalent.

11 BJS is finalizing this developmental work through APS and is preparing to seek approval from OMB to implement additional strategies to assess coverage of the APS misdemeanant population.

12 During Phase 2, public agency designees without the ability to respond online will be asked to complete the survey by telephone.

13 The response rates for APS ranged from 93-100% over the past three years of data collection. While CAPSA will be contacting more respondents than APS, CAPSA respondents will be similar to the APS respondents (i.e., probation agencies) and some will even be the same respondent.

14 The CAPSA pilot test was approved under a generic clearance provided to BJS with OMB No. 1121-0339 and expiration date 1/31/2016.

12