OMB memo on terms of clearance

MEMORANDUM TO: Robert Sivinski

Office of Statistical and Science Policy

Office of Management and Budget

THROUGH: Melody Braswell, Clearance Officer, Justice Management Division

Jeffrey H. Anderson, Director, BJS

Devon Adams, Acting Principal Deputy Director, BJS

E. Ann Carson, Acting Unit Chief, Corrections Unit, BJS

FROM: Tracy L. Snell, Statistician, BJS

SUBJECT: Summary of usability evaluation of web-based data collection for Capital Punishment Report of Inmates Under Sentence of Death (NPS-8) collection

DATE: November 14, 2019

The Terms of Clearance for NPS-8 (Capital Punishment Report of Inmates Under Sentence of Death), issued December 19, 2016, stated that the Bureau of Justice Statistics (BJS) must prepare a plan to evaluate the usability of the web-based data collection tool before the next request for OMB clearance. In response, BJS conducted a systematic outreach effort with respondents to identify concerns raised by OMB related to the web-based data collection systems.

The current version of the web-based collection tool for the NPS-8 has been in use since the 2011 data collection cycle (see Appendix 1 for screenshots). During this time, BJS has informally assessed data quality related to unit and item nonresponse and has not identified any significant technical or substantive issues. Both the BJS project manager and the program manager at the Census Bureau (the data collection agent for NPS-8) routinely contact respondents to discuss inconsistencies in their data and the overall submission process. We have not been notified of any concerns related to the web-based system. However, BJS has found it beneficial to directly engage respondents to identify possible areas for updates or improvements related to data entry, data collection, and response burden. This memo summarizes BJS’s efforts as of September 1, 2019.

Summary Findings

• Since launching the current web-based data collection tool, the usage by respondents has increased. Just over 50% of all responses for the NPS-8/8A forms were submitted electronically in 2011, and this has increased to nearly 100% in recent years.

• Feedback from respondents regarding their use of the web-based data submission option have been generally positive. Reported problems were related to log-in issues, which were easily resolved by contacting the data collection agent.

Respondents Submitting Data Electronically

The NPS-8 collection has 2 components, each with 2 associated forms.

The first component collects individual-level data (such as such as name, demographic characteristics, criminal history, and dates of conviction and sentence for the capital offense) from the departments of correction (DOC) for all inmates under sentence of death during the reference year:

 The NPS-8 form is submitted by a DOC respondent when an inmate is newly received under sentence of death during the previous calendar year.

The NPS-8A form is sent to respondents for each inmate reported by the respondent as being under sentence of death on December 31 of the previous calendar year. This form is pre-filled with the previously-submitted data for each inmate. Respondents are asked to update any information about the status the inmate and the status of the death sentence on the reference date.

The second component is designed to capture information about statutes authorizing the imposition and implementation of the death penalty and changes in the jurisdiction’s death penalty laws that have occurred during the reference year:

 The NPS-8B form is sent to staff in the office of the attorney general in jurisdictions with no capital statute in force on December 31 of the previous reference year.

 The NPS-8C form is sent to staff in the office of the attorney general in jurisdictions with a capital statute in force on December 31 of the previous reference year.

Table 1. Percentage of NPS-8 Respondents Who Submitted Electronically

	2011	2012	2013	2014	2015	2016	2017	2018
NPS-8/8A Respondents	71%	73%	73%	95%	92%	98%	95%	99%
NPS-8B/8C Respondents	50%	46%	60%	62%	73%	73%	90%	73%

 

Table 2. Electronic Submissions for NPS-8 Data Collection by Form

	2011	2012	2013	2014	2015	2016	2017	2018
NPS-8 Forms	57%	73%	58%	78%	100%	87%	100%	100%
NPS-8A Forms	53%	58%	57%	74%	98%	95%	95%	99%
NPS-8B Forms	14%	7%	33%	38%	25%	50%	82%	71%
NPS-8C Forms	66%	66%	70%	72%	81%	83%	94%	74%

Description of activities

2016 usability study

Initial activities were conducted during the 2016 data collection cycle (January through April 2017). BJS developed a set of questions to solicit feedback about their preferences and experiences regarding data submission (see below). Census Bureau staff asked seven respondents to provide feedback to these questions. In order to minimize burden, the questions were administered to respondents at the same time as response follow-up activities were conducted.

During the 2016 data collection cycle, the following questions were asked:

1. For final data submission, do you prefer to use the web collection instrument, or paper form? Why?

2. Do you have any problems accessing the website?

3. Have you had any problems using the website? If yes, what were the problems?

4. What is the overall process you use for responding to this data collection?

5. How do you research/collate the information you will submit for the data collection? Does this process affect your preference for [web/paper - insert the previously stated preference] form submission? If so, how does it affect the preference?

6. NPS-8B/8C respondents only: Have you ever uploaded/attached supplemental materials? If yes, were there any difficulties?

7. Where do you seek help when you need it?

8. Are there any features of the web tool you feel should be changed, enhanced or added?

Results of 2016 usability survey

Four NPS8/8A respondents were asked to provide feedback. All indicated that they preferred to submit data via web, although sometimes they did submitted paper forms instead due to time constraints. None of these respondents reported problems.

Three NPS-8C respondents were asked to provide feedback. Two indicated a preference for the web-based data submission while one indicated that they preferred paper due to the portability. None reported having specific issues with the web-based tool.

2018 data collection cycle

For the 2018 data collection cycle (January through April 2019), BJS revised the data submission follow-up protocol. Each year, respondents contacted for clarifications on their data submissions will also be asked to provide feedback about their experiences with the web-based data collection tool. During the 2018 cycle, six NPS-8/8A respondents and five NPS-8B/8C respondents were asked for feedback. The set of questions asked and the responses received appear below:

1. Approximately how long did it take you to complete the form? Do you agree with the estimated burden time (NPS-8/8A: 30 minute per inmate; NPS-8B/8C: 15 minutes)?

NPS-8/8A respondents:

• “It only took me less than 10 minutes per inmate to complete the form.”

• “For the NPS-8 form - This is a good amount of time filling out info for the first time per person.”

• “About 15 minutes but I have very few on death row. 30 minutes per inmate is more than enough time.”

• “It took no more than 30 minutes per inmate.“

• “Are you referring to the form when entering a new inmate into your system? If so, I would say approximately 15 minutes. I feel that 30 is considerably more than I needed to complete the form.”

NPS-8B respondents:

• “Because we are a non-death penalty state, it takes very little time to complete the survey.”

• “Form took less than 15 minutes. Yes, agree with estimated time burden.”

NPS-8C respondents:

• “It just took me a couple minutes.”

• “I believe 15 minutes was an accurate assessment of how long it took to complete the survey.“

• “10 minutes max.”

2. Did you encounter any problems using the online tool? For instance, were the instructions for use clear enough, did you encounter any difficulty or confusion, etc?

NPS-8/8A respondents:

• “I did not encounter any problems but I have been doing this for a long time.”

• “I could not initially sign in with the info provided. Once told to copy and paste after reaching out via e-mail it worked.”

• “I had no problems and instructions were very clear. Thank you for the reminder email to get it done!”

• “I did not encounter any issues this year.”

• “No, I thought everything was very clear and easy to maneuver.“

• “I completed the survey, with a co-worker’s help. It wasn’t hard to use at all.”

NPS-8B respondents:

• “This year, there was a brief issue with accessing online. But it was immediately addressed, and no issues in prior years. Very easy to do.”

NPS-8C respondents:

• “There weren’t any problems or anything I see that needs changing.”

• “I did not have any problems using the online tool. The only concern I had was based more on whether my response to the appeal question was accurate enough for the yes/no portion, but I explained that in the Notes section at the end. I would have had the same issue with the paper form.”

3. Do you have any suggestions for improvement? For instance, are there any places where the instrument could be improved or could directions be communicated better during the mail out process.

NPS-8/8A respondents:

• “Any suggestions – Put in BOLD letters to print the attachments for people like me when you send out the original instructions.“

• “I did the online version which was just updating and that was easy. Looking through the envelope sent I was a little unsure on the difference between the card and the paper when just updating info. Maybe be a little more clear about what to do when nothing needs to change or be added to the current file. Maybe just one box to check off is there are no changes.”

• “Maybe a reminder earlier than the day it is due.”

• “I do not make any recommendations for changes at this time.”

• “So far, I haven’t noticed anything. It seems to run very smoothly and is user friendly.”

NPS-8B respondents:

• No suggestions

NPS-8C respondents:

• “No problems, & I think the instructions are very clear.”

• “I can’t think of any suggestions for the future, as I think it was a very simple process.”

• “I cannot think of any suggestions right now. I think the whole process was straightforward and easy to follow.”

Summary

This report details two efforts by BJS to document to document issues of usability of the NPS-8 web data collection tool. Overall, respondents had very few problems with or suggestions for the webtool. BJS and the collection agent for NPS-8 will continue to solicit feedback from respondents during the follow-up response activities regarding usability issues and potential improvements for the web-based data collection tool.