Protocol

2017 Economic Census Instrument Debriefings Protocol Draft

1. Research Questions

• Respondent Portal (eCorr):

  • Were respondents able to…

    • …create an account?

    • …link a survey using the authentication code?

  • Did respondents use the delegation system? If so, did it function as expected?

  • Did respondents look for survey information? If so, were they able to find it?

  • Did respondents attempt to contact Census through the website? If so, did this function as expected?

• Instrument (Centurion):

  • Does the flow of the instrument screens support respondents’ informational and decision-making needs with regard to reporting options and instrument functionality? If not, what other information did they need, and what are the optimal locations and formats for presenting such information?

  • Do button labels and descriptions make sense to respondents and inform respondents of the correct functions of the buttons? If not, what terms do respondents suggest?

  • Are respondents able to discern the functions of individual screens? If not, what are the sources of confusion, and how do they suggest making them clearer?

  • What features did respondents use when completing the survey, and did these features meet their needs?

  • What additional features do respondents want in the instrument?

• If respondent reported on the spreadsheet (MUs only):

  • Was the visual design of the spreadsheet template effective in promoting readability of questions and instructions?

  • What additional features or information do respondents want in the spreadsheet template?

2. Before Beginning

• Introductions (if necessary)

• Email with screenshots – so that you don’t have to rely entirely on your memory and to ensure that it’s clear what I’m asking about, I’d like to send you an email with some screenshots of the instrument. It will be coming from “…. dot ……. @census.gov” so you will know that it’s legitimate. I’m not going to be asking about any of your specific data, rather, I’m going to use these generic screenshots to discuss features and functions of the web instrument. Is your email address …?

• If respondent is willing to go through the debriefing questions at the time of the recruiting call, go ahead and complete the interview. If not, schedule the interview and get the respondent’s email address.

• Audio taping – would only be done if interviews take place in a conference room.

  • Permission to audio record discussion?

  • Before we get started: I'd like to audio record this interview, so I don't have to rely on my memory later. This session is confidential. Only people connected with this project will have direct access to your recording. Is this all right with you?

3. Participant Background

• How long have you been with the company/in your current position?

• What is your title/role?

• What are your major responsibilities?

• What are your government reporting responsibilities?

• What other government surveys or filings, if any, do you also handle?

4. General Probes (Use these whenever necessary)

• In your own words, what would you say is the purpose of this page?

• In your opinion, is the information on this page helpful or not helpful?

• Can you tell me what you are thinking about?

• Can you tell me more about that?

• I want to make sure I understand; can you explain that again?

• Reflect back on R’s answer (“I want to make sure I have it right. I think you said, “…?”)

5. Survey Background

• Can you tell me about your process for filling out Economic Census?

  • Additional probes (if necessary):

    • Was this your first time filling out this survey?

    • What was your role completing the Economic Census? Were others involved in helping gather the data? If so, how many and what were their responsibilities?

    • How do you typically gather the data?

    • Did someone else have to review the data before you submitted it?

Now I would like to ask you specific questions about your experience using the Census website to report. Please open the attachment that I sent you via e-mail. The document is in PDF form and each page is numbered. As I ask you questions, I will indicate which page you should be looking at.

6. eCorr Respondent Portal

• Did you have to create a new account in order to complete the Economic Census electronically?

  • (If yes) How did you feel about having to create an account?

  • How was the account creation process for you? Did you find it easy or difficult?

• You were also required to enter an authentication code to access your survey. Was that process clear or unclear to you?

• The survey card that appears for your survey contains some summary information: the name of the survey, due date, and company name and ID. Did you find this information helpful? Is there other information you would like to see here?

• Did you access the options drop-down menu? (located next to the ‘Report Now’ button)

  • (If yes): Which options did you use?

  • (If no): Were you aware that there were actions you could take on the survey under the options menu?

• This website has a function that allows you to share your survey with coworkers.

  • Were you aware of this feature?

    • (If yes): Did you use this feature?

      • (If yes): What did you think of this feature? Was it helpful or not helpful?

      • (If no): Was there a reason you decided not to use it?

    • (If no): Do you think you would have used this feature if you had known about it?

7. Reporting Dashboard/Location List

I want to ask you about the main Reporting Dashboard.

• In your own words, what was the purpose of this screen?

• Can you tell me about any of the different options that you used on this table?

• Did you use any options to help you organize your list of locations? Can you tell me what you did and why?

• Were there any other ways that you would have wanted help organizing your list of locations?

• Are there any improvements that we could make to this screen?

• Are the instructions at the top of the screen useful?

• Dashboard Overview – Did you notice/use the Help Site, How-To Videos, About, or Contact?

8. Preview Survey Questions

• Did you use the question preview to help you report?

  • (If yes):

    • Why did you want to use the questions preview?

    • Did you find it to be useful, or not useful?

    • Did you print the questions and instructions? Why (not)?

      • Did you share the preview with anyone when gathering your data? Only ask if they had other people help gather data.

    • Do you have any suggestions that would make the preview more useful to you?

  • (If no): Did you know there was a printable version of the questions and instructions?

    • (If yes): Is there a reason you chose not to use it?

    • (If no): Would you have used it if you had known about it?

9. Add/Delete Locations (Multi-unit companies only)

• Did you need to add any new locations for 2017?

  • (If yes): How did you add it/them? Did you use the ‘Add Location’ button on the Dashboard?

    • (If yes): Was the ‘Add Locations’ function clear or unclear? Were you able to successfully add the location(s) you needed to?

    • (If no): Did you know about the ‘Add Locations’ button? If so, is there a reason you chose not to use it? If not, would you have used it if you had known about it?

• Did you need to delete any extra locations you might have added?

  • If so, did you use the ‘Delete Locations’ button on the dashboard, and did it work like you expected it to?

10. Download/upload Spreadsheet (MUs only)

• Did you download a spreadsheet from the website?

• (If yes):

  • Please tell me about your experience downloading the spreadsheet.

    • (If needed): Did you find the process easy or difficult?

    • You are given an option to download two spreadsheets. They are the regular spreadsheet and the All Locations spreadsheet. Which did you download?

    • (If they downloaded All Locations) - Why did you download the ‘all locations’ spreadsheet? How did you use the spreadsheet? Was it helpful?

    • (If they didn’t download all locations) –What did you expect the ‘All Locations’ spreadsheet to show? Why wouldn’t this have been useful for you?

  • What was your overall experience using the spreadsheet?

  • Is there any other information you would like to see in the spreadsheet or other features we should consider?

  • Did you upload your spreadsheet?

    • (If yes), Did you encounter any issues when uploading the spreadsheet?

11. Notifications (Messages)

• Notifications are a new feature; did you use the ‘Messages’ page in any way?

• If yes, do you remember the types of notifications that you received?

• Were the notifications helpful or unhelpful?

12. Question View

• MUs: Did you use the individual location form screens (question-by-question) in any way (to report or review your data)?

• SUs: In general, how did you feel about the layout of the question-and-answer fields?

13. Review Dashboard

Now let’s talk about the review step.

• Can you tell me about your review process?

• Did you use the review step on the website?

• Did you find the review step on the website to be helpful or not helpful?

• Is there other information you would like to see in a review step for this survey?

14. Summary Data

• Did you use the summary data page? If so, how?

  • (If yes): Did you find this information to be useful or not useful?

    • Are there other types of information or features you would like to see on a page like this?

  • (If no): Did you know about the ‘Summary Data’ button? If so, is there a reason you chose not to use it? If not, would you have used it if you had known about it?

15. Submit Dashboard

Let’s talk about the submission step now.

• What was your overall impression of the submission process? Did you run into any issues?

• MUs: Did you submit the data for all of your locations at once, or submit each location once completed?

• Did you save a record of your response?

  • (If necessary): Did you know that the View Response pdf on the Submit screen had populated with your data? If so, did you print or attempt to save this for your records? If not, would you have printed or saved this for your records had you known?

• Did you see the record of your submission in the Messages area?

16. How-To Videos

I have a couple questions on the How-To videos that I’d like to ask.

• Did respondents use the how-to videos?

  • (If yes), which ones? Were they helpful? Recommendations for improving?

  • (If no), why not? Did you know that they were available? Is this something that you would have used if you had known it was available?

17. Computer Use and Internet Experience

Now I’d like to ask you a few questions regarding your computer use and Internet experience.

• On average, about how many hours do you spend on the Internet per day?

• On a scale from 1 to 10 with 1 being “No experience” and 10 being “Very experienced,” please rate your overall experience with the following:

  • Computers

  • Internet

• On a scale from 1 to 10 with 1 being “Not comfortable” and 10 being “Very comfortable,” how comfortable are you using and navigating through the Internet?

18. Demographics

I have a couple questions about you that I’d like to ask.

• Which age range best describes you: 18-25, 26-35, 36-45, 46-50, or 51+ years old?

• What is the highest grade or year of school you have completed: less than high school, completed high school, some college (no degree), associate’s degree (AA/AS), bachelor’s degree (BA/BS), or post-bachelor’s degree?

  • If R responds some college (no degree), associate’s degree (AA/AS), bachelor’s degree (BA/BS), or post-bachelor’s degree, then ask for area of study and degree.

Wrap-up

• Are there any other comments or suggestions you would like to make about anything we have gone over today?

This concludes our meeting. Thank you for your time and valuable feedback.

3/28/2018