USAJOBS and USA Staffing Joint usability testing session

USAJOBS/USA Staffing Usability Test Script

Facilitator: ____________________

Date: ______________

Hello! Thank you for taking the time to participate in a usability test of USAJOBS and an applicant tracking system called USA Staffing. I am going to give you this consent form to review and sign while I ask you a few questions. These questions are to gain a better understanding of our test participant profile. They will allow us to identify trends in our data.

Introductory Questions & Tasks:

• First, can you tell me if you are a…. (read options)

• Federal employee

• Not a federal employee

• Student

• Other:

• What is your occupation?

• Are you a veteran of a United States Armed Forces?

• Today I am going to ask you to interact with USAJOBS.gov. Have you been to this site before?

  • Yes

  • No

  • Not Sure

• (If yes) How often have you used USAJOBS in the past year?

  • Once a day

  • Once a week

  • Once a month

  • Once a year or less

  • Other:

• (If Yes) Have you applied for a job through USAJOBS?

  • Yes

  • No

• (If Yes) Do you remember the agency that you applied to?

  • Yes:

  • No

• (If no) Have you heard of this site before?

  • Yes

  • No

• Have you heard of Application Manager before?

• Yes

• No

Welcome and Purpose: (READ THE FOLLOWING OUT LOUD)

Thank you for agreeing to participate in this evaluation of our systems.

1. I’m here to guide you through a series of tasks that pertain to USAJOBS.gov and USA Staffing. One of my colleagues is helping me take notes and will observe your interaction with the materials I present to you. (Observer says “hi”)

2. Our goal is to see how easy or difficult you find the site to use. There aren’t any right or wrong answers, so don’t worry about making mistakes. We will record your reactions and opinions, so, we may ask you to clarify statements that you make from time to time.

3. I will ask you to think aloud to learn if what we cover works well for you.

4. If you have any questions, comments or areas of confusion while you are working, please tell me. I may not always be able to help you, but it is very important for us to understand where you have questions.

Do you have any questions before we begin?

Do you need anything before we begin? Water? Restroom?

Scenario 1: Add missing documents to USAJOBS Profile

(Will be delivered to half of participants)

Great, now we are ready to begin. We have created a mock account for you to use for this test. We are going to start by allowing you to review this job description on the screen in front of you. In this test, you will be asked to apply for this job. It details the requirements of the position and what you need to do to submit a complete application. Once you have reviewed the job description and are ready to move on just let me know.

(Allow user a maximum of 3 minutes to review the JOA)

Are you ready to continue?

Question #1

After reviewing the job description, do you have everything you need and are you ready to apply?

(Observation Goal: Does the user realize they may be missing a required document?)

Follow-up Questions:

• Why would you include those documents?

• If participant realizes that some required documents are missing: How do you know you are missing a required document?

Question #2

Please show me how you would apply for this position?

(Observation Goal: See if user notices that required document is missing at this stage? See if user forgets any required checkboxes?)

Question #3

(While Application Manager is loading)

Please read this page as quickly as possible and tell me what it details?

Question #4

(Once USA Staffing Application Manager has loaded)

Can you please tell me where you currently are in the application process? Can you tell me the name of the system you are using?

(Observation Goal: See if user understands that they have moved from USAJOBS to USA Staffing and that they are separate systems)

“Please continue to complete your job application for this opportunity. Answer the questions however you think an applicant with your experience and background would respond. We will ask questions where we see fit.”

(General observation time to see how user works through initial steps of Application Manager)

Question #5

(Wait for participant to get to the Supporting Documents page.)

Have you submitted everything you need for this job?

Follow-up Questions:

• If participant does not match all USAJOBS docs with a document type: Which documents are you submitting? [See if he/she understands that only the matched ones will be submitted.]

• If participant realizes that some required documents are missing: How do you know you are missing a required document?

Question #6:

(Once user realizes they are missing a required document)

Is there a way to add a required document?

Follow-up Questions:

• Are there multiple ways to add a required document?

Question #7

Before going any further, how would you like to add the missing required document?

(Don’t let the participant click anything yet. Just ask what they would like to do.)

Follow-up Questions:

• What other options do you have?

• Why are you more interested in that option over others?

“Please select the option to return to your USAJOBS profile and proceed to add the missing documents.”

Question #8

(Once user selects Return to USAJOBS)

Do you recognize this page? Where are you in the application process now?

Question #9

How would you add the missing required document to your application?

Follow-up Questions:

• Are there multiple ways to do this?

• Where did the documents section tell you to go?

Question #10

Now that you are in the Saved Documents section, how would you add your missing required document?

(Provide user with missing file locations)

Now that you have added the missing document(s), please return to and complete the application you have already started.

Question #11

What is the best way to return to the documents page you were on previously?

Follow-up Questions:

• Is there more than one way to return to the documents page?

Please submit your application.

Question #12

Now that you have successfully submitted your application:

Follow-up Questions:

• What did you think about the entire application process?

• Where did you experience the most difficulty?

• Was it difficult to upload a required document using this application?

• Do you have any recommendations how the process can be improved?

Scenario 2: Add missing documents to USAJOBS Profile

Great, now we are ready to begin. We have created a mock account for you to use for this test. We are going to start by allowing you to review this job description on the screen in front of you. In this test, you will be asked to apply for this job. It details the requirements of the position and what you need to do to submit a complete application. Once you have reviewed the job description and are ready to move on just let me know.

(Allow user a maximum of 3 minutes to review the JOA)

Are you ready to continue?

Question #1

After reviewing the job description do you have everything you need and are you ready to apply?

(Observation Goal: Does the user realize they may be missing a required document?)

Follow-up Questions:

• Why would you include those documents?

• If participant realizes that some required documents are missing: How do you know you are missing a required document?

Question #2

Please show me how you would apply for this position?

(Observation Goal: See if user notices that required document is missing at this stage? See if user forgets any required checkboxes?)

Question #3

(While Application Manager is loading)

Please read this page as quickly as possible and tell me what it details?

Question #4

(Once USA Staffing Application Manager has loaded)

Can you please tell me where you currently are in the application process? Can you tell me the name of the system you are using?

(Observation Goal: See if user understands that they have moved from USAJOBS to USA Staffing and that they are separate systems)

“Please continue to complete your job application for this opportunity. Answer the questions however you think an applicant with your experience and background would respond. We will ask questions where we see fit.”

(General observation time to see how user works through initial steps of Application Manager)

Question #5

(Wait for participant to get to the Supporting Documents page.)

Have you submitted everything you need for this job?

Follow-up Questions:

• If participant does not match all USAJOBS docs with a document type: Which documents are you submitting? [See if he/she understands that only the matched ones will be submitted.]

• If participant realizes that some required documents are missing: How do you know you are missing a required document?

Question #6:

(Once user realizes they are missing a required document)

Is there a way to add a required document?

Follow-up Questions:

• Are there multiple ways to add a required document?

Question #7

Before going any further, how would you like to add the missing required document?

(Don’t let the participant click anything yet. Just ask what they would like to do.)

Follow-up Questions:

• What other options do you have?

• Why are you more interested in that option over others?

“Please select the option to directly upload your document using this application.”

Question #8

Please attempt to upload your missing required document for this job application.

(Provide user with missing file locations once they select browse)

Question #9

(Once user select “Document Type” in upload pop-up)

Do you know which “Document Type” to save this uploaded document under?

Question #10

Now that you have uploaded your missing document, is it attached to your application?

Follow-up Questions:

• How would you attach the document to your application?

Please submit your application.

Question #11

Now that you have successfully submitted your application:

Follow-up Questions:

• What did you think about the entire application process?

• Where did you experience the most difficulty?

• Was it difficult to upload a required document using this application?

• Do you have any recommendations how the process can be improved?