OMB Memo

Usability testing of the National Survey of Psychiatrists

The Census Bureau plans to conduct additional survey work under the generic clearance for Data User Evaluation Surveys (OMB number 0607-0725).

Purpose of the Study

The Census Bureau’s Center for Behavioral Science Methods Usability Lab is planning a study to understand user interface design and cognitive issues with the National Survey of Psychiatrists (NSP). The NSP is designed to obtain the necessary data to better understand the characteristics and distribution of psychiatrists throughout the United States, as well as emerging patterns in their employment characteristics. These data will provide the means for the evaluation and assessment of the evolving demographics, career employment patterns, and populations served. Such data aids in better understanding the psychiatry workforce demands given the recent transformation of the healthcare system. The survey collects data on psychiatrists in United States and the type of practices they work in, and the types of work and patient care they are involved with including such things as information on medicine they dispense, patients they work with, and their opinion about their work practices.

This is a new survey that has never been fielded before. This is also the first time that usability testing will be conducted on the instrument. We would like to ensure that the online instrument is usable for participants as well as ensures high data quality for the sponsor. The study will identify areas of the instrument that may need to be tweaked or changed for a better user experience and data quality outcome.

The feedback from this study will help to better understand how participants will interact with the online survey and what changes need to be made to the survey prior to its being fielded.

The key research questions or substantive issues being examined are to 1) find usability problems which interfere with a respondent’s ability to answer the NSP accurately, efficiently and with satisfaction while they are using a smartphone or laptop. 2) find cognitive issues with the instructions, questions, or response categories as the participants interacts with the online instrument.

Methodology

• Population of Interest: The usability and cognitive evaluation will focus on assessing and improving the user experience for practicing psychiatrists.

• Timeline: Testing will be from September 3 through September 30^th 2019. Recruiting will start beginning of August, once OMB approval is received.

• Language: Testing will be conducted in English.

• Devices: Respondents will participate using either smartphones or laptops. Devices may be provided by Census Bureau or may be respondents’ own devices.

• Sample: Participants will be trained psychiatrists. Participants will not have participated in previous testing of a census online instrument. Participants will have at least one year’s experience with using the Internet for things other than email. For those participants using their own smartphone or tablet, they will know how to use that device to access the Internet.

• Respondents: Recruitment and Incentive: Participants will be recruited using a combination of word-of-mouth, in-person visits to offices to drop off letters/flyers, online & paper advertisements, and social media, such as meet up groups, listservs, reaching out to medical organizations, reaching out to medical schools with psychiatry residency programs, using personal social and professional networks, etc. Participants will be recruited in the DC metro area. This is considered a challenging recruit as we already know that during the cognitive testing portion of the study, the recruiter(s) encountered challenges in finding trained doctors of psychiatry with enough time, availability and interest to participate in this study. For this research, we request an increase in the typical incentive. The Census Bureau typically uses an incentive of $40 per one-hour usability session. Because the NSP target population (psychiatrists) is highly educated and therefore may be hard to recruit for a research study, we propose offering up to $100 for their participation. There is a precedence for researchers (at the National Center for Health Statistics and U.S. Census Bureau) giving a $100 incentive for cognitive interviews of medical professionals. As well as the cognitive testing portion of the NSP used an incentive of $100. Based on the cognitive testing experience the usability testing portion of the study would also like to use an increased incentive of $100 as this is a very specialized user group. We have additional screener questions that will be used to obtain the needed participants. These are included in the package.

• Researchers from Center for Behavioral Science Methods (CBSM) will conduct the interviews. We will ask participants to provide their answers in the questionnaire – as they pertain to his/her real life. During this time, the participant will be asked to think aloud (verbalizing what he/she is thinking) and will be probed as needed. We will pause the participant half way through the survey and ask them to exit the survey and then re-enter the survey. After the completion of the survey, the participant will answer a satisfaction questionnaire. We will use vignettes (pretend situations) to get information on specific screens that may not be answered by all or by the majority of participants but that the sponsor would like to get feedback on. Finally, we will conduct a debriefing of the survey by asking the participant to review selected screen shots of the survey and answer questions about those screens. Concurrent probes will be used to focus on any questions the sponsor would like additional feedback on as well as any questions the usability lab team notice is confusing to participants. The sponsor is invited to observe any of the session in the usability testing observation room. For off -site sessions, we can bring one to two additional observers, as space permits.

The session will be audio recorded and the computer screen will be video recorded. We may also employ eye-tracking technology.

• Informed Consent: We will inform participants that their response is voluntary and that the information they provide is confidential and will be accessed only by employees involved in the research project. We will used the standard consent form used at the Center for Behavioral Science Methods. This consent form indicates that the respondent agrees that the interview can be audio-taped to facilitate analysis of the results Participants who do not consent to be video or audio-taped will still be allowed to participate.

• Burden: We expect to screen 50 potential respondents at approximately 10 minutes per call for 10 successful scheduled user sessions. Each usability session should last 60 minutes and there will be 10 sessions. The total respondent burden is expected to be 18.3 hours. See Table 1.

Table 1. Total Estimated Burden

Category of Respondent	No. of Respondents	Participation Time	Burden
Screening	50	10 minutes	8.3 hours
Cognitive/Usability Interviews	10	60 minutes	10 hours
Totals			18.3 hours

Attachments

1. Protocol used for the study (Enclosure 1) including possible debriefing probes and vignettes

2. Mailing materials (Enclosure 2)

3. Paper questionnaire (Enclosure 3) (screenshots of the live instrument are not available yet)

4. Demographic questionnaire (Enclosure 4) (Previously approved by OMB from earlier usability studies)

5. Mobile experience questionnaire (Enclosure 5) (Previously approved OMB from earlier usability studies)

6. Satisfaction questionnaire (Enclosure 6) (Previously approved by OMB from earlier usability studies)

7. Additional screener questions for Psychiatrists (Enclosure 7)

8. NSP test plan (Enclosure 8)

For further information about this study, please contact Erica Olmsted-Hawala at 301-763-4893 or [email protected]. If you have questions on the package when Erica is away on extended leave (June 17 through August 1), please contact Elizabeth Nichols at [email protected].