Chatbot personality unmoderated usability testing

Study questions: Chatbot personality unmoderated usability testing

Starting instructions:

Thank you for participating in this study! This survey asks about your communication preferences and your thoughts on the Internal Revenue Service (IRS) chatbot. It won’t ask you to share any personal or identifying information. This survey should take no more than 15 minutes to complete, and your participation is voluntary. Your feedback will help the IRS improve its services for taxpayers like yourself!

[* indicates a required question. Questions without * are optional.]

Starting questions:

What is the main reason you might choose to use a chatbot over another contact method? If you never use chatbots, please write “N/A”.*

• Open-ended text box

Wireframe instructions:

In this section, you’ll see several possible conversations with the IRS chatbot. Focusing on the content of the conversations—rather than their formatting—please read these conversations and answer the corresponding questions.

[Participants will evaluate 10 sample conversations with the IRS chatbot. These conversations will address common taxpayer issues (e.g. refund tracking, getting help with a payment, clarifying a notice). No sensitive topics or information will be discussed in these conversations. We’ll present these conversations as simplified wireframe designs. Each conversation will have a different personality, varying along 5 dimensions:

• Name vs. no name

• Formal vs. casual

• Expert vs. novice

• Warm vs. cool

• Excited vs. calm

We’ll add the finalized wireframes to the study in the IRS Labs online platform. After looking at each wireframe, participants will answer the following 3 questions.]

Questions for each wireframe:

How trustworthy do you find this chatbot?*

• 1-5 scale: Not at all trustworthy – Extremely trustworthy

How helpful do you find this chatbot?*

• 1-5 scale: Not at all helpful – Extremely helpful

How likely are you to recommend this chatbot to a friend for help with tax-related questions or issues?*

• 1-5 scale: Not at all likely – Extremely likely

Summary instructions:

Before finishing this survey, please reflect on the chatbot conversations that you just saw. These conversations are repeated here with number labels for you to reference.

[The wireframes of the 10 sample conversations with the IRS chatbot will be repeated here, labeled with numbers for participants to reference in their responses.]

Summary questions:

What aspects of these conversations did you like (if any)?

• Open-ended text box

What aspects of these conversations did you not like (if any)?

• Open-ended text box

Which version of the IRS chatbot do you like the most?*

• Multiple choice with all 10 sample conversations

Closeout instructions/question:

Thank you again for participating in this study! Your feedback is crucial for improving IRS customer service.

In the email with the link to this survey, you should have received a unique participant code. Please enter your code here, so that you can be paid for your participation.*

• Open-ended text box

Thanks! Your responses have been recorded. Have a nice day!