2015 Annual Survey of Entrepreneurs
Usability Testing Protocol
(This protocol is a guide – the questions presented here may not be asked exactly as worded in the protocol or in this order. It also is important to note that not all questions will be asked in every interview.)
Introduce observers and their background
Purpose of visit:
Let me start by telling you a little about what we will be doing today. The Annual Survey of Entrepreneurs (formerly Survey of Business Owners) is given to business owners annually to collect data on the characteristics, decisions, and roles of the individuals that own businesses.
This survey is conducted electronically, and we want to get your feedback in order to make revisions before it is formally released. We are looking for feedback in regards to anything that you experience while using the survey system today, including both the functionality of the system and any content issues that you notice.
Permission to audio-record discussion? Have R sign consent form.
Before we get started, I would like to audio-record this interview so I don't have to rely on my memory later. Only those working specifically on this project will hear the recordings. If that is all right with you, please sign this consent form.
Filling out the form
Ok, what I’d like you to do now is go ahead and fill out the survey. (Have respondent access the website address). Go ahead and work through the survey. If you need to get information from records or others within the company in order to answer a question, just let us know what you would need to do. If at any point something isn’t clear or you have questions about something, please ask and we’ll talk about it.
I’ll periodically stop you and ask you questions abut some of the pages and get your feedback on them.
Note respondent’s behavior as they go through the questionnaire:
Use of previous and next buttons
Answering all questions
Responding to error messages
Scrolling
Navigating the questionnaire
Note any issues with browser settings, monitor settings, etc., that affect visibility of the instrument.
Login Screen Page
First, I’d like to know what you think of this screen?
Is it clear/obvious where you start the questionnaire?
Is there any other information you want on this screen to help you get started?
Do you think that the information given on this screen was sufficient?
What are your overall reactions to the process for logging into the system?
Is there any other information you’d like to see on any of the screens you saw while logging in?
Next, I’d like you to walk me through how you would log into the survey. (R will have a User ID and password combination)
How easy or difficult was it to log in? Would there be a better way to log in to the survey? If yes, tell me more about that…
How did you decide what information to use?
Main Menu Page
In your own words, describe what this screen is showing you.
What is being provided in each of the columns? Is this information helpful to you?
Is it clear where you need to go in order to start answering your survey? If not, what could be done to make the start of the form clearer?
If you had questions about definitions or the survey items, where would you go?
Is there any other information that you need on this screen before you would begin answering your survey? Why?
Did you notice the toolbar on this screen? What do you think each of the options will do?
Contact Information Page
Do you have any recommendations for how we could improve the layout of the items on these screens?
Is there any other information that you believe we should request here?
Is it clear how to enter your information?
General Probe as Respondents Navigate through Survey
(Give respondent time to look over the pages and complete them)
Do you have any recommendations for improving this screen?
Observe and note any problems with navigation
Observe and note any problems interacting with screen layouts
Instructions – Note whether R appears to read instructions
Do you recall whether you read any instructions on this screen?
Edit failure messages – If the R has not yet triggered an edit, have them do so on purpose
Do the messages on the screen clearly indicate to you that there is an issue, or are they not clear?
Can you explain to me what the problem(s) are?
Is there a better way that we could notify you about the problem(s) on this screen?
What would you need to do if you wanted to ignore the problem message you were receiving and move on to the next questions?
Is it clear to you what action you need to take?
Navigation
If you needed to return to a prior question, how would you do that?
Did you have any problems navigating back to a prior screen/section? Do you have any recommendations for improving the navigation?
If you needed to log out of the survey and then return to it, how would you do that?
Have R log out. Message should pop up saying “data on page will not be saved.” What do you think of this message? Is it clear or unclear? Is this what you’d expect?
Have R log back in and continue with the survey
Items with “Select only one” Instructions
Note whether Rs attend to the “select only one” instructions
Do you recall seeing the instruction to select only one answer?
(If R attempted to select more than one answer) Can you tell me about how you decided to select more than one answer in this question?
(If there is a None Of The Above option) What would you expect to happen if you select one of the choices, along with None Of The Above?
Items with Other
(If Other – Specify box) What would you type here?
(If NO Other – Specify box) Would it be helpful to specify?
Items Requiring Special Attention:
Number of Owners
Which of these options would you select for your situation?
How would you choose that option?
What does the option “business is owned by a parent company, estate, trust, or other entity” mean to you?
Did you notice the instructions?
What do the instructions mean to you?
Did you see the note on this page about estimates?
Business Owners
How would you answer questions about Owner 1?
Percent Ownership
Is it clear to you how to answer this question?
(If R has less than 4 owners) Is it clear to you what to do with the extra spaces?
(If R has more than 4 owners) Is it clear to you how to proceed?
(If R has more than 4 owners) Is it clear to you which owners to report?
Initial Ownership
Do you think you are able to select more than one option here? Why or why not?
What would you do if you wanted to select more than one option?
Initial Acquisition Year
Do you know the answer to this?
What would you do if you only knew an approximate year?
Job Functions
What would you do if an owner had multiple functions, but one of them was something that was not listed here?
Reasons for Owning Business
What do you think about the layout of this question?
What would you do if you had more than one response for Other?
New Funding Relationships
What would you do if you had more than one response for Other?
Profits or Losses
What do you think about the answer choices for this question?
Percent of clients by location
What relationship do these three categories have with one another and with the total?
How would you verify that these numbers add up to 100%?
Business Outside the U.S.
What would you do if you did not know the answer to this question?
R&D Funding Sources Percentages
What relationship do you think these categories have with each other and with the total?
Review Your Responses Page
What is your initial reaction to this page?
Is it clear what this screen is telling you? (Note: there will be an indication that there is a problem with one of the sections)
Do you find this information helpful?
Does it help you figure out what you need to do?
What does it mean to you for the status to be “ok”?
How would you go about correcting the issues? (Have R try and correct problems)
(When R has fixed problems) What do you need to do next? (observe if R can navigate back to the review page)
How would you go about submitting your data?
What do you expect to happen after you click the “Submit Data” button?
If you weren’t ready to submit your data at this point, what would you do?
What other types of review would you be interested in performing prior to submission?
Submission Confirmation Page
What is your initial reaction to this page?
What would you want to do from this page?
If you wanted to print out a copy of your answers to this form for your own records, what would you do?
Do you find this feature helpful?
Would you save a copy of this information for your records?
Is there any other information that you would like to have for your records?
Wrap Up
Now that you have gotten to see the online form, what do you think about it?
If you were completing this form electronically, would you still want/need the paper form? If so, what purpose would the paper form serve?
(If needed) Would you complete this on paper before completing online?
Would it be necessary for someone besides yourself to provide answers to any of these questions?
(If needed) If so, how would you handle that?
Do you think that you would complete the online form in one sitting?
Now that you have gotten a good feel for the online form, what is your overall impression of it? Was it easy, hard?
(If needed) Did you have any issues with the navigation on the site?
What, if anything, did you like the most about the online form? Least?
Do you have any suggestions for how to make this online form easier for you to use?
Technical Variables:
Is it a Mac of PC? ______________
Brower Type ______________
Version ______________
Is this the browser you normally use?
What other browsers do you use?
(Observe Only) Demographic Variables:
Male/Female
18-29, 30-44, 45-64, 65+
Thank you so much for your assistance today.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Jennifer Beck |
| File Modified | 0000-00-00 |
| File Created | 2021-01-28 |