Usability Evaluation Plan

Usability Evaluation Plan for 2019 National Survey of Psychiatrists

Latest version: May 3, 2019

Last revision:

Version 1.0

Usability Study Team

Erica Olmsted-Hawala and Dawn Nelson

Project Management

Elizabeth Nichols

Sponsors

National Survey of Psychiatrists (NSP)

Center for Behavioral Science Methods

Associate Directorate for Research and Methodology

U.S. Census Bureau

1. Introduction

The 2019 National Survey Psychiatrists (NSP) is a new, nationally representative survey of psychiatrists in the U.S. administered by the United States Census Bureau. The National Survey of Psychiatrists (NSP) is conducted to assist in fulfilling the congressional mandates of Programmatic Authority 42 U.S.C. 290aa.

The NSP is designed to obtain the necessary data to better understand the characteristics and distribution of psychiatrists throughout the United States, as well as emerging patterns in their employment characteristics. These data will provide the means for the evaluation and assessment of the evolving demographics, career employment patterns, and populations served. Such data aids in better understanding the psychiatry workforce demands given the recent transformation of the healthcare system. The survey collects data on psychiatrists in United States and the type of practices they work in, and the types of work and patient care they are involved with including such things as information on medicine they dispense, patients they work with, and their opinion about their work practices,

The NSP will use a multi-mode approach to data collection. There will be an initial mailed invitation to respond to the online questionnaire. This will be followed by additional reminder mailings and then a paper questionnaire for nonresponders. There will also be an inbound telephone operation where respondents can call and report their data. The Census Bureau will conduct the survey in 2019. CBSM will conduct usability testing of the 2019 NSP Web survey in English and provide recommendations for enhancements where needed, with the intent to improve on the user interface design of the survey.

2. Project Objectives

This project will identify usability problems during the Web questionnaire development phase for the 2019 NSP for both PC and mobile phone devices by watching participants who are psychiatrists attempt to complete the survey. See Table 1.

3. Project Scope

There is one round of usability testing of the questionnaire. Ten usability sessions will be conducted in English. The NSP has 6 sections.

A. Eligibility

B. Current appointment in direct clinical care

C. Previous appointment in direct clinical care

D. Current employment not in direct clinical care

E. Education and credentials

F. Demographics

Participants will answer questions in each section that pertains to them. Their answers will reflect their own experiences.

Table 1. Proposed distribution of participants across NSP questionnaire, devices and languages for 2019 usability testing

	English
	NSP survey
PC	5
Smartphone	5

Table 2 outlines the proposed project schedule. Products include emails to ADDP including a short email summary of each session; a quick report in a PowerPoint format; and a final meeting discussing the usability findings.

Table 2. Estimated Project Schedule for 2019 NSP Usability Testing

Activity	Start date	End date	Duration
Project Planning	3/01/2019	6/17/2018	76 days
OMB Approval	6/17/2019	8/5/2019	40 days
Respondent Recruitment	8/1//2019	9/6/2019	30+ days
Usability Testing / Data Collection	9/3/2019	9/17/2019	10 days
Data Analysis / Quick Report Preparation	9/17/2019	10/2/2019	10 days
Quick Report Delivery	10/3/2019	10/3/2019	1 day
Quick Report Debriefing	10/3/2019	10/3/2019	1 day

4. Assumptions

1. Changes based on the testing can be made to the questionnaire before production deployment.

2. DSMD will provide CBSM with potential English-speaking respondents from the Washington/Baltimore area who contacted the Census Bureau to participate in previous cognitive testing for the project, but who were not used for that project. DSMD will also provide their screener questionnaire for this project and their OMB letter to review.

3. An external link to the online instrument will be available ten work days prior to the start of testing.

4. All changes to the instrument will have been finalized six work days prior to the start of testing and the survey must be operational, that is, all major issues which cause crashing are eliminated.

5. Mailed survey notifications will be available to use during the testing.

6. Sponsors will provide CBSM with 25 user ID codes (if codes are required to access the survey).

7. We might collect eye-tracking data for CBSM research purposes on the PC-versions of the survey.

8. Sponsors will provide a list of individuals to notify about each planned session. These individuals are welcome to attend any session at headquarters and will be notified prior to the beginning of each session. One observer can also attend sessions offsite. Who attends needs to be coordinated ahead of time with CBSM staff. Video recordings of sessions will be available, but observers must come to the usability lab at Census Bureau headquarters to watch them.

5. Usability Testing of the 2019 National Survey of Psychiatrists

After the majority of the instrument development is complete, a formative usability test will be conducted. During the test, the participants’ performance will be investigated using a think aloud protocol, in order to identify usability problems and to better understand the causes of the problems.

Overview of methodology.

In the usability testing, we will observe participants receiving mocked-up mailing materials about the survey and then accessing the survey using a Census-owned laptop or their own mobile devices. We will ask participants to provide their answers in the questionnaire – as they pertain to their real life. During this time, the participant will be asked to think aloud (verbalizing what he/she is thinking) and will be probed as needed. We will pause the participant half way through the survey and ask them to exit the survey and then re-enter the survey.

After the completion of the survey, the participant will answer a satisfaction questionnaire. We will conduct the debriefing by asking the participant to review selected screen shots of the survey and answer questions about those screens. Concurrent probes will be used to focus on any questions the sponsor would like additional feedback on as well as any questions the usability lab team notice is confusing to participants

Participants. All total, 10 individuals will participate in the testing: 5 on laptop and 5 on mobile devices. Participants will be recruited from the metropolitan DC area. Participants will need to be trained psychiatrists.

The following approaches will be used in recruitment:

1. Word of mouth, craigslist ads, flyers posted in relevant local doctor’s offices.

2. Posting a BOC announcement

3. Social media, such as meet up groups, listservs, local medical organizations, etc.

4. Medical schools with psychiatry residency programs

5. Personal social and professional networks

We will notify the sponsor of any difficulty recruiting by the end of the first week of recruiting.

Data collection methods. The following methods will be used to collect participants’ performance data:

1. Demographic questionnaire

2. Think-aloud protocol with minimal probing such as “Keep Talking;” “What are you thinking?” and acknowledgement tokens (linguists refer to this as backchannels) such as “Um-hum?”

3. Observation notes

4. Satisfaction questionnaire

5. Debriefing Probing

6. Vignettes: Such as exit and re-enter data

7. Audio and video recording

8. (Potentially) Eye tracking

Data analysis.

Qualitative

1. Review behavioral observations, spontaneous verbalizations and answers to debriefing questions in order to identify problems.

Quantitative

1. Compute the overall satisfaction ratings

Testing facility and procedure. Session will occur in the usability lab at the Census Bureau’s headquarters, community location such as a library or community center, or in a location more convenient to the participant. The session will be conducted one-on-one, i.e., one participant and one test administrator (TA), with all sessions including a note taker. The test will be carried out in the following sequence:

1. Participant’s consent

2. TA start audio and video recording

3. Demographic Questionnaire

4. Provided NSP mailing material and completes the Web questionnaire while thinking-aloud. Participant is stopped half-way through the questionnaire to administer the exit and re-entry task.

5. Participant completed satisfaction questionnaire.

6. Participant is asked to complete vignettes

7. TA conducts retrospective debriefing.

Schedule. Each session is estimated to last about 60 minutes. The schedule is available in Table 2 above.

9. Sponsor’s Role

• Coordinate meetings both during development and for results presentations

• Provide a list of people to include on the session invite list

• Watch sessions when available and debrief with task administrator immediately after session to identify problems

• Attend a results presentation meeting to discuss recommendations

• Provide written comments on task plan, protocol, and quick reports.

• Provide responses to each recommendation.

10. Remuneration

$100 (as approved by OMB) for a 60-minute session

11. Instrument

External site

Website: https://respond.census.gov/____________________

XX minute survey