2022 Census of Agriculture
Usability Testing of the Electronic Data Reporting Web Survey
10/24/2019
Interviewer:
POID:
State:
Date:
Type of Farm
This protocol is a guide – the questions presented here won’t necessarily be asked exactly as worded in the protocol or in this order. In addition, not all questions will be asked in every interview.
Moderator – Please READ before continuing:
In general, follow the guide and write findings below each section.
Unless indicated, use retrospective probing with respondents before they proceed to the next screen.
If a respondent is stuck on a particular screen, let them struggle a little before helping them. This may be a good time for an emergent probe (i.e., what would you do if I wasn’t here?)
Observe nonverbal cues and use emergent probes as necessary.
Introduce observers and their background
Purpose of visit:
Let me start by telling you a little about what we will be doing today. The Census of Agriculture is the leading source of data about American agriculture. Conducted every five years, the Census provides a detailed picture of U.S. farms and ranches and the people who operate them. It is the only source of uniform, comprehensive agricultural data for every state and county in the United States. Participation by every farmer and rancher, regardless of the size or type of operation, is vitally important.
For the 2022 Census of Agriculture, respondents will have the option to respond using a web survey through the Internet. Since we have updated the web survey, we want to get your feedback on it in order to assess how it works for respondents. We are more interested in your ability to navigate and use the web survey – we want to make sure it is intuitive. So at this point, feel free to just estimate your data as you go through it. Do you have any questions before we begin?
Can you tell me a little about your farming operation?
(If needed) What is your role?
Have you completed any other NASS surveys online? If so, which ones?
What made you choose to report electronically?
What was your overall experience?
(If needed) Did anything stand out as being very helpful?
(If needed) Was there anything that you didn’t like?
In your opinion, what benefits would there be for you to report electronically to the Census of Agriculture? What drawbacks?
Now, I’m going to give you the envelope you would receive in the mail from us during the actual Census. I want you to handle these materials the same way you would if I wasn’t here. [Hand the envelope to the respondent and observe/record their behaviors below.]
3. Survey letter
What stands out to you in the survey letter/postcard?
What information would you expect to be in the survey letter/postcard, but don’t see?
In your own words, how would you begin the Census of Agriculture?
(If needed) Is it clear? If not, what can we do to make it clearer?
(Observe: Did respondent read the letter, scan it, or not read it at all? Circle one: Read the letter/scanned it/did not read at all)
Now, I would like you to go through the web survey and complete it as you normally would. This is not a test of your ability or Internet savvy, so just proceed in whatever way makes the most sense to you. If you need to get information from records in order to answer a question, just let us know what you would need to do.
Don’t hesitate to tell me anything you find surprising in the web survey, or if you have questions about the material on the site, just ask. We’ll try our best to answer your questions.
(Note: The primary interview technique is retrospective; respondent is asked about each question and/or group of questions after s/he has reported. Concurrent “think-aloud” probes are used when respondent indicates some confusion or other problem, e.g., by frowning, re-reading a question, looking away in thought, saying “Hmm”, etc.)
If you have a Smartphone/Tablet users, please read to yourself before continuing. Note throughout your notes the following:
Which way the respondent is holding their device – i.e., portrait (vertical) or landscape (horizontal).
If tables being displayed vertically (on smartphone and tablet in portrait) mode seem to make a difference. I.e., does it lead to more item nonresponse? This is especially true in the crops/livestock sections.
If the respondent checks a “none” box in a table does it confuse them? (I think this is more confusing on the smartphone because it looks like a standalone question.)
If size of response boxes seems to matter, and if so, which screens? Anecdotally, is the respondent writing more information when the response box is larger?
If color impacts response/navigation? For example, on smartphones the first header item in a table is blue and the rest are not. Note that on the iPad this only occurs if you hold the tablet in portrait (vertical) mode.
3. Online Survey Response Screen
What do you think about this screen?
In your opinion, is it clear what you need to do? If not, how can we improve it?
(Observe: Did the respondent click the right button to start the survey (instead of the question mark)? Yes/No)
4. Login screen
What do you think about the process of going to the survey’s login screen?
Now that you’ve seen the login screen, do you think that the information given on it was sufficient?
Is there any other information you’d like to see on this page that you currently don’t see?
Did you read the “helpful hints for online response?”
(If no) Why not? What can we do to encourage people to read it?
(If yes) What did you think of this information provided? Is it easy or difficult to read?
If you had additional questions, what would you do?
If you wanted to contact NASS, where would you find the information to do this?
Would you call, fax, or email NASS?
Did you notice the green bars at the right-hand top of the screen? What do you think those green bars are for?
What are your overall reactions to the process for logging into the system?
(Observe: Did the respondent have difficulty typing in the survey code? Yes/No)
(If respondent receives error message when logging in) Did you notice the error message at the top of the screen? What do you think of this information?
5. Operator’s Survey List
What do you think of this screen?
Under the second column (labeled status)…
If you saw the term “available”, what would that mean to you?
What about the term “started”?
What about the term “submitted”?
What do you think about the order of the columns?
What would you expect to do on this screen?
6. Change of address (Probe Concurrently)
(Observe: Did the respondent read this page or immediately click “continue to...” button? Yes/No)
Can you describe to me what you see on this screen?
Where do you need to go in order to start completing your form?
Is this clear? If not, what could be done to make the start of the form clearer?
If you had questions about this survey, what would you do?
Is there anything else that you would need at this point in order to start completing the form?
Anything other comments on this screen?
7. Introduction (First Screen: Welcome to the 2016 Census of Agriculture Online Test)
(Observe: Did respondent read the first screen (welcome screen)? Yes/No)
8. Introduction (Second Screen – Question 1)
(After respondent has indicated “yes” or “no”) What do you think of this screen?
(If respondent clicks “no” for Q1) What is your reaction to the new item that just appeared on the screen?
Did you notice the “for example” statement?
(If needed) How would you navigate to the next screen?
If previously reported data are used, a statement will appear near this screen. Ask “what does this sentence mean to you? What do you expect to happen?
9. Acreage in 2016 (First Screen- Questions 1-4 or Land Owned)
(Observe: Did the respondent read the instructions at the top of the page? Yes/No)
(Confirm if needed) Did you read the instructions at the top of the screen?
(Observe: Did respondent use any of the “none” box functionality? Yes/No)
Do you think you can change this number in Question 4?
Do you have any suggestions for improving this screen?
If previously reported are used, ask “what do you think the numbers shown here are? Did you expect them? Were they helpful?”
10. Acreage in 2016 (Second Screen- Question 5 or Fee per Head)
(Observe: Did the respondent read the instructions at the top of the page? Yes/No)
(If respondent answers “Yes” to Q5) In your own words, what is Question 5a asking you to do?
What would you EXPECT to happen here?
11. Acreage in 2016 (Third Screen- Question 6 or County)
What do you think about the layout of this screen?
Do you have any suggestions on how to improve this screen?
12. Land use in 2016 (Cropland)
(Observe: Did the respondent read the instructions at the top of the page? Yes/No)?
Do you have any suggestions for improving this screen?
If you wanted to save and continue the survey later, what would you do? (Observe if they find and click the save & return later button. Once respondent clicks on the link, have them click “stay” and continue completing the survey.)
Is it clear how the number showed up in the response box underneath Question 5 (Total Acres)?
Do you think you can change this number in Question 5?
(If respondent’s total acres in Land Use in 2016 Section do NOT EQUAL total acres in the Acres in 2016 Section, observe: Did the respondent notice the edit message? Yes/No)
(If respondent received edit message) What do you think of this edit message?
In your own words, what is this edit message telling you to do?
(If respondent’s total acres in Land Use in 2016 Section DOES EQUAL total acres in the Acres in 2016 Section, have them change their number in Q5 to trigger edit.) What do you think of this edit message?
In your own words, what is this edit message telling you to do?
13. Land use practices (Drained by tile)
Do you have any suggestions for improving this screen?
14. Crop insurance and government programs
Do you have any suggestions for improving this page?
What do you think about the dollar signs used for Questions 3-6? (Observe if they use the left-hand panel or the back button)
15. Type of organization
(Question 3: If respondent selects answer 2, 3, or 4 and DOES NOT answer subquestions (Q3a, 3b, or specify type) Did you notice the question that popped-up below Question 3?
Why didn’t you answer the question that popped-up?
16. Personal characteristics (First screen- Question 1 or Men & Women)
(Observe: Did the respondent have any issues with this page? Use emergent probes as necessary.)
(If respondent doesn’t answer this question and receives warning message): What do you think about his message you received?
Is it clear? What would you do?
17. Personal characteristics (Second screen- Question 2a or Names)
(Observe: Did the respondent have any issues with this page? Use emergent probes as necessary.)
18. Personal characteristics (Third screen – Question 2b-2h or who’s completing the form)
(Observe: Did the respondent fill out these questions for all persons on this page? Yes/No)
(Observe: How is the respondent navigating the personal characteristics section? Next button or Tab at top?)
(Observe: If applicable, did the respondent have a difficult time reporting correctly when it switched from one person to another? Yes/No)
What do you think about the amount of questions on this page? Is it too little, too many, or just about enough?
If previously reported are used, ask “what do you think the numbers shown here are? Did you expect them? Were they helpful?”
19. Personal characteristics (Fourth screen Question 2i-2o or days worked off farm)
(Observe: Did the respondent fills out these questions for all persons on this page? Yes/No)
(Observe: How did the respondent navigate to the other persons’ tabs? Next button/Tab at top)
(Observe: If applicable, did the respondent have a difficult time reporting correctly when it switched from one person to another? Yes/No)
What did you think about the format?
Do you have any suggestions on how to improve this existing format?
What do you think about the amount of questions on this page? Is it too little, too many, or just about enough?
Did you notice the faded grey text in K and L? If yes, what do you think of them? Is it clear that we need a four digit year?
20. Personal characteristics (Fifth screen – Questions 3, 4, and 5 P.O. and decisions as of Dec. 31st)
(Observe if respondent has any difficulties on this screen. Use emergent probes as necessary.)
21. Crops (Screening)
(If R clicks “no” to any question on this screen) Did you notice anything when you clicked the “no” boxes for Question X? If yes, tell me about it…
Did you notice the “include” and “exclude” instructions throughout the page?
22. (If applicable) Hay and forage crops (Third screen – Question 7 – this table contains OR)
(Observe: Did the respondent have problems with the “or” option in the table? Yes/No; If yes, walk me through how you answered this question.)
(Third screen only) What do you think about the layout of this screen? Do you like it?
Smartphone/Tablet – Does respondent have any problems with the name appearing as a part of the header under the field crop name? Did the respondent see it?
Do you have any suggestions for us on how to the format of this table?
If previously reported are used, ask “what do you think the numbers shown here are? Did you expect them? Were they helpful?”
23. (If applicable) Cultivated Christmas trees, short rotation woody crops, and maple syrup
(Observe: Did the respondent have any problems with the formatting on this screen? Yes/No)
24. (If applicable) Field crops (Second screen – Question 2 or select all field crops)
(When they see list) What do you think about the organization of these field crops on this screen? Does it make sense? Is there a better way to organize it?
Observe – did respondent report hay here or in the hay section?
25. (If applicable) Field crops (Third screen – Questions 3a and 3b or any other field)
Observe how respondents fill out this screen. Do they skip to 3b? Do they fixate on 3a?
Do you have any other field crops that you harvested that are not listed in 3b? If so, why didn’t you list them?
How would you ADD a field crop not listed?
Can you show me how you’d do that?
How would you go about REMOVING field crops on this screen?
Can you show me how you’d do that?
Do you have any suggestions for us on how to improve this screen?
26. (If applicable) Nursery, greenhouse, floriculture, SOD, mushrooms, vegetable seeds, and propagative materials (Second screen – Question 3 or select)
What do you think of the layout of this screen?
Do you have any suggestions to improve it?
27. (If applicable) Nursery, greenhouse, floriculture, SOD, mushrooms, vegetable seeds, and propagative materials (Third screen – Questions 4a and 4b or any other)
Observe how respondents fill out this screen. Do they skip to 4b? Do they fixate on 4a?
Do you have any other nursery, greenhouse, floriculture, mushrooms, vegetable seeds, and propagative materials that you harvested that are not listed in 3b? If so, why didn’t you list them?
How would you ADD a crop not listed?
Can you show me how you’d do that?
How would you go about REMOVING crops on this screen?
Can you go ahead and do it?
Do you have any suggestions for us on how to improve this question?
28. (If applicable) Vegetables, potatoes, and melons (Second screen – Question 3 or select)
(When they see list) What do you think about the organization of these crops on this screen?
Does it make sense? Is there a better way to organize it?
29. (If applicable) Vegetables, potatoes, and melons (Third screen- Questions 4a and 4b or any other)
Observe how respondents fill out this screen. Do they skip to 4b? Do they fixate on 4a?
Do you have any other vegetables, potatoes or melons that you harvested that are not listed in 4b? If so, why didn’t you list them?
How would you ADD a crop not listed?
Can you show me how you’d do that?
How would you go about REMOVING crops on this screen?
Can you go ahead and do it?
Do you have any suggestions for us on how to improve this question?
30. (If applicable) Fruits, nut and berries (Second screen – Question 3 or select)
(When they see list) What do you think about the organization of these crops on this screen?
Does it make sense? Is there a better way to organize it?
Do you have any suggestions for us on how to improve this question?
31. (If applicable) Fruits, nut and berries (Third screen- Questions 4a and 4b or any other)
Observe how respondents fill out this screen. Do they skip to 4b? Do they fixate on 4a?
Do you have any other fruits, nuts or berries that you harvested that are not listed in 4b? If so, why didn’t you list them?
How would you ADD a crop not listed? Can you show me how you’d do that?
How would you go about REMOVING crops on this screen? Can you go ahead and do it?
Do you have any suggestions for us on how to improve this question?
32. Livestock
Did you read the “include” and “exclude” instructions on this screen?
If not, is there anything that we could do to make it easier to read?
(If R clicks “no” to any of the screening questions) Did you notice anything when you clicked the “no” boxes for Question X? If yes, tell me about it…
Overall, do you have any suggestions for improving this page?
33. Functionality – Navigation / Save / Help / Hamburger Button on Smartphones and Tablets
(Observe: Has the respondent used the left-side navigation panel at all? Yes/No)
(Observe and/or probe: How is the respondent navigating to previous sections? Left-hand panel navigation or previous button?)
Smartphone only - Observe if respondent knows how to/or is using the hamburger button in the upper-right hand corner, then ask:
In your opinion, what do those three lines in the upper-right hand corner do?
If the respondent doesn’t know its function, explain purpose: Is that clear? Why or why not?
What do you think of this functionality?
What do you think about the left-side of the screen?
Do you like it? Dislike it?
Do you have any suggestions on how to improve it?
In your opinion, what does the orange check box mean?
In your opinion, why are some of the rows greyed out while others are not?
(Observe and/or probe: If the respondent wanted to save their information, what would they do?)
(Observe and/or probe: If the respondent wanted to exit the survey, how would they do that?)
(Observe and/or probe: If the respondent needed help, either in general or for a question, what would they do?)
Smartphone/Tablet only – Observe if it seems to make a difference that the hover-overs aren’t appearing on the smartphone screen. Does the respondent seem to intuitively understand the function of each symbol?
34. (If applicable) Cattle and calves
(Observe if respondent has any difficulties on this screen. Use emergent probes as necessary.)
What do you think of the layout of this screen?
Do you have any suggestions to improve it?
If previously reported are used, ask “what do you think the numbers shown here are? Did you expect them? Were they helpful?”
35. (If applicable) Hogs and pigs
(Observe if respondent has any difficulties on this screen. Use emergent probes as necessary.)
What do you think of the layout of this screen?
Do you have any suggestions to improve it?
36. (If applicable) Horses, ponies, mules, burros, and donkeys
(Observe if respondent has any difficulties on this screen. Use emergent probes as necessary.)
(Observe if other types of equine are being reported in this section? After cognitive testing we changed the name of this section because of misreporting.)
What do you think of the layout of this screen? Do you have any suggestions to improve it?
37. (If applicable) Sheep and goats
(Observe if respondent has any difficulties on this screen. Use emergent probes as necessary.)
What do you think of the layout of this screen?
Do you have any suggestions to improve it?
38. (If applicable) Aquaculture (Second screen – Questions 2a and 2b or any other)
Observe how respondents fill out this screen. Do they skip to 2b? Do they fixate on 2a?
Do you have any other aquaculture that you did not list in 2b? If so, why didn’t you list them?
How would you ADD aquaculture not listed?
Can you show me how you’d do that?
How would you go about REMOVING aquaculture on this screen? Can you go ahead and do it?
Observe: did the respondent report for both pounds and number sold? If yes, ask them:
Did you see the “or” statement in the table? If so, what do you think that means? Note that the OR is being displayed differently in this section on the smartphone than the hay section.
Regardless if the respondent reports for both pounds and number sold…ask the following probe.
Is it clear that you can report pounds or number sold? If not, what can we do to make it clearer?
Observe: Did respondent report gross value of sale? If not, why did you not report gross value of sales?
Overall, what do you think of the layout of this screen?
Do you have any suggestions to improve it?
39. (If applicable) Poultry (All screens)
(Observe if respondent has any difficulties on these four screens. Use emergent probes as necessary.)
Observe how respondents fill out this screen. Do they skip to 5b? Do they fixate on 5a?
Do you have any other poultry that you did not list in 5b? If so, why didn’t you list them?
How would you ADD poultry not listed?
Can you show me how you’d do that?
How would you go about REMOVING poultry on this screen?
Can you go ahead and do it?
Observe respondents reaction to 6b (a popup depending on answer to 6a) – how does the respondent react to this?
40. (If applicable) Colonies of honey bees
(Observe if respondent has any difficulties on this screen. Use emergent probes as necessary.)
41. (If applicable) Other livestock (Second – Question 2a and 2b and Fourth –Question 4a and 4b)
Observe how respondents fill out this screen. Do they skip to 2b? Do they fixate on 2a?
Do you have any other kinds of other livestock that you did not in 2b? If so, why didn’t you list them?
How would you ADD a livestock or livestock product not listed?
Can you show me how you’d do that?
How would you go about REMOVING a livestock or livestock product on this screen?
Can you go ahead and do it?
Do you have any suggestions for us on how to improve this question?
42. Production contracts and custom feeding (Second screen – Question 2)
(Observe if respondent has any difficulties on this screen. Pay particular attention to the second table where they have to select their unit of measurement. Use emergent probes as necessary.)
(Observe if respondent has difficulty with questions 2i, 2k and 2l, do respondents notice that they need to select a unit? On the smartphone, the unit options are greyed out. Does this make a difference?
What do you think of the layout of this screen?
Do you have any suggestions to improve it?
43. Organic agriculture
(Observe if respondent has any difficulties on this screen. Use emergent probes as necessary.)
44. Practices
(Observe if respondent has any difficulties on this screen. Use emergent probes as necessary.)
45. Food marketing practices
(Observe if respondent has any difficulties on this screen. Use emergent probes as necessary.)
46. Fertilizers, chemicals, and soil conditioners applied
(Observe if respondent has any difficulties on this screen. Use emergent probes as necessary.)
47. Renewable energy
(Observe if respondent has any difficulties on this screen. Use emergent probes as necessary.)
48. Machinery and equipment
(Observe if respondent has any difficulties on this screen. Use emergent probes as necessary.)
49. Farm labor
(Observe if respondent has any difficulties on this screen. Use emergent probes as necessary.)
50. Production expenses
(Observe if respondent has any difficulties on this screen. Use emergent probes as necessary.)
51. Market value of land, buildings, machinery, and equipment
((Observe if respondent has any difficulties on this screen. Use emergent probes as necessary.)
52. Income from farm-related sources
(Observe if respondent has any difficulties on this screen. Use emergent probes as necessary.)
53. Agricultural activity within the borders of American Indian reservations, pueblos, and service areas
(Observe if respondent has any difficulties on this screen. Use emergent probes as necessary.)
54. Conclusion
(Observe if respondent has any difficulties on this screen. Use emergent probes as necessary.)
Before you submitted your survey, would you want to print your answers for your records?
If yes, how would you do that? (See if the respondent can find the print button.)
How would you go about submitting the survey?
Can you go ahead and do it?
Would you expect anything else at this point?
Do you think that you would complete the online form in one sitting?
If you were completing this form electronically, would you still want/need the paper form?
If so, what purpose would the paper form serve?
What are some of the things that you liked the MOST about the online form?
What are some of the things that you liked the LEAST about the online form?
Are there any other options that we should provide in the online form?
Do you have any suggestions for how to make this online form easier for you to use?
If previously reported are used, ask “was the data that was pre-filled helpful? Was is accurate? Did you like having it available in the answer cells? Would like to have this information available for more questions? Where do you think this information came from?”
Document the following based off observation:
Device Type: |
(Circle one) Desktop Laptop Smartphone Tablet Other:__________ |
If respondent used smartphone or tablet, how was it held? |
Horizontal Vertical Both Horizontal and Vertical – elaborate below:
|
Browser Type: |
(Circle one) Internet Explorer Safari Firefox Chrome Other:__________ |
Sex: |
(Circle one) Male Female |
Age Range: |
(Circle one) 18-29 30-44 45-64 65+ |
Self-Administered Post-Interview Feedback Survey
Give to respondent at the end of the interview.
1. I think that I would like to use this web survey when completing the Census of Agriculture instead of a paper survey. |
Strongly Disagree |
|
|
|
Strongly Agree |
|
|
|
|
|
|
1 |
2 |
3 |
4 |
5 |
|
2. I found this web survey unnecessarily complex.
|
Strongly Disagree |
|
|
|
Strongly Agree |
|
|
|
|
|
|
1 |
2 |
3 |
4 |
5 |
|
3. I thought the web survey was easy to use.
|
Strongly Disagree |
|
|
|
Strongly Agree |
|
|
|
|
|
|
1 |
2 |
3 |
4 |
5 |
|
4. I think that I would need the support of a technical person to be able to use this web survey.
|
Strongly Disagree |
|
|
|
Strongly Agree |
|
|
|
|
|
|
1 |
2 |
3 |
4 |
5 |
|
5. I found the various functions in this web survey were well integrated.
|
Strongly Disagree |
|
|
|
Strongly Agree |
|
|
|
|
|
|
1 |
2 |
3 |
4 |
5 |
|
6. I thought there was too much inconsistency in this web survey.
|
Strongly Disagree |
|
|
|
Strongly Agree |
|
|
|
|
|
|
1 |
2 |
3 |
4 |
5 |
|
Strongly Disagree |
|
|
|
Strongly Agree |
|
|
|
|
|
|
|
1 |
2 |
3 |
4 |
5 |
|
8. I found the web survey very cumbersome to use.
|
Strongly Disagree |
|
|
|
Strongly Agree |
|
|
|
|
|
|
1 |
2 |
3 |
4 |
5 |
|
9. I felt very confident using the web survey.
|
Strongly Disagree |
|
|
|
Strongly Agree |
|
|
|
|
|
|
1 |
2 |
3 |
4 |
5 |
|
10. I needed to learn a lot of things before I could get going with this web survey.
|
Strongly Disagree |
|
|
|
Strongly Agree |
|
|
|
|
|
|
1 |
2 |
3 |
4 |
5 |
11. How many minutes did it take you to complete this web survey?
____________ Minutes
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Pick, Kenneth - NASS |
| File Modified | 0000-00-00 |
| File Created | 2021-01-14 |