2016 ASM-COS Debriefings Protocol
Research Questions
Respondent Portal:
Were respondents able to…
…register for an account?
…link a survey using the authentication code?
Did respondents use the delegation system? If so, did it function as expected?
Were respondents able to find the survey information, if needed?
Did respondents attempt to contact Census through the website? If so, did this function as expected?
Multi-unit (MU) Instrument (Centurion):
Does the flow of the instrument screens support respondents’ informational and decision-making needs with regard to reporting options and instrument functionality? If not, what other information did they need, and what are the optimal locations and formats for presenting such information?
Do button labels and descriptions make sense to respondents and inform respondents of the correct functions of the buttons? If not, what terms do respondents suggest?
Are respondents able to discern the functions of individual screens? If not, what are the sources of confusion, and how do they suggest making them clearer?
What features did respondents use when completing the survey, and did these features meet their needs?
What additional features do respondents want in the instrument?
If respondent reported on the spreadsheet:
Was the visual design of the spreadsheet template effective in promoting readability of questions and instructions?
What additional features or information do respondents want in the spreadsheet template?
Before beginning
Introductions (if necessary)
Audio taping – would only be done if interviews take place in a conference room.
Permission to audio- record discussion?
Before we get started: I'd like to audio record this interview, so I don't have to rely on my memory later. This session is confidential. Only people connected with this project will have direct access to your recording. Is this all right with you?
Participant Background
How long have you been with the company?
What is your title/role?
What are your major responsibilities?
What are your government reporting responsibilities?
What other government surveys or filings, if any, do you also handle?
Now I would like ask you specific questions about your experience using the Census website to report. If possible [convenient] please log into the Census website to refresh your memory.
General Probes (Use these whenever necessary)
In your own words, what would you say is the purpose of this page?
In your opinion, is the information on this page helpful or not helpful?
Can you tell me what you are thinking about?
Can you tell me more about that?
I want to make sure I understand; can you explain that again?
Reflect back on R’s answer (“I want to make sure I have it right. I think you said, “…?”)
Survey Background
Can you tell me about your process for filling out the Annual Survey of Manufactures/Report of Organization?
Additional probes (if necessary):
Was this your first time filling out this survey?
How many locations did you report for?
What is your role? How many people are involved?
How do you typically gather the data?
How did enter the data? (Form view/Spreadsheet)
Did someone else have to review the data before you submitted it?
Respondent Portal
Historically, the Census Bureau sends out an ID and a password for you to log into our website. This year we asked you to create an account on a new website to report your data.
How did you feel about having to create an account?
How was the account creation process for you? Did you find it easy or difficult?
You were also required to enter an authentication code to access your survey. Was that process clear or unclear to you?
This website has a function that allows you to share your survey with coworkers.
Were you aware of this feature?
(If yes): Did you use this feature?
(If yes): What did you think of this feature? Was it helpful or not helpful?
(If no): Was there a reason you decided not to use it?
(If no): Do you think you would have used this feature if you had known about it?
Reporting Dashboard/Location List (COS)
I want to ask you about the main Reporting Dashboard.
Did you find the purpose of this screen to be clear or unclear?
How did you use this screen?
For companies with COS locations:
Were you able to find the list of your Report of Organization locations from this page? (If needed: Under the ‘Num. Estabs in Survey’ column, there would have been a blue link labeled ‘View All’ next to the number)
(If yes): Was it clear to you where to find this list of locations?
Did you find the Location List to be useful or not useful?
(If no): Did you look for a list of your Report of Organization locations? If so, where did you expect to find such a list?
There was a list of locations you could access through the dashboard; would you have used this list if you had known about it?
Add/Delete Locations
Did you need to add any new locations this year?
(If yes): How did you add it/them? Did you use the ‘Add Location’ button on the Dashboard, or some other way?
(If yes): Was the ‘Add Locations’ widget clear or unclear? Were you able to successfully add the location(s) you needed to?
(If no): Did you know about the ‘Add Locations’ button? If so, is there a reason you chose not to use it? If not, would you have used it if you had known about it?
Did you need to delete any extra locations you might have added?
If so, did you use the ‘Delete Locations’ button on the dashboard, and did it work like you expected it to?
Question Preview
Did you use the question preview to help you report?
(If yes):
Did you find it to be useful, or not useful?
Did you print the questions and instructions? Why (not)?
Did you share the preview with anyone when gathering your data?
Do you have any suggestions that would make the preview more useful to you?
(If no): Did you know there was a printable version of the questions and instructions?
(If yes): Is there a reason you chose not to use it?
(If no): Would you have used it if you had known about it?
Export/Import wizard (MUs only)
If respondent reported on spreadsheet:
Please tell me about your experience downloading the spreadsheet.
If needed: Did you find the process easy or difficult?
Did you download the ‘all locations’ spreadsheet and the ‘full survey’ spreadsheet?
Was the spreadsheet itself easy to use or difficult to use?
Is there any other information you would like to see in the spreadsheet or other features we should consider?
Did you encounter any issues when uploading the spreadsheet?
Notifications (Messages):
Notifications are a new feature; did you use the ‘Messages’ page in any way?
Were the notifications helpful or unhelpful?
Form view
MUs: Did you use the individual location form screens (question-by-question) in any way (to report or review your data)?
SUs: In general, how did you feel about the layout of the question and answer fields?
Summary Data
Did you use the summary data page? If so, how?
(If yes): Did you find this information to be useful or not useful?
Are there other types of information or features would you would like to see on a page like this?
(If no): Did you know about the ‘Summary Data’ button? If so, is there a reason you chose not to use it? If not, would you have used it if you had known about it?
Review Dashboard:
Now let’s talk about the review step.
Can you tell me about your review process?
Did you find the review step on the website to be helpful or not helpful?
Is there other information you would like to see in a review step for this survey?
Submit Dashboard:
Let’s talk about the submission step now.
Did you find the submission process to be clear or unclear?
MUs: Did you submit the data for all of your locations at once, or submit each location once completed?
Did you save a record of your response?
If necessary: Did you know that the question preview on the report dashboard populated with your data as you entered it? If so, did you print or attempt to save this for your records? If not, would you have printed or saved this for your records had you known?
ASM Content Questions (For ASM companies only)
Now I’d like to ask you a couple of questions about some specific questions from the Annual Survey of Manufactures that we are considering revising.
In your records, are 1st line supervisors counted with the production workers, or are they counted separately?
In the Selected Expenses section (Items 16.A.5) you are asked to report the total production cost of electricity in thousands of dollars. We then ask for the quantity of purchased electricity, generated electricity, and electricity sold or transferred to other establishments (Item 16.B). We currently ask for these quantities in thousands of Kilowatt hours. We are considering revising the way we ask this question. We would still ask for the total production cost of electricity in thousands of dollars, but we are considering asking for the quantity of purchased electricity, generated electricity, and electricity sold or transferred to other establishments as percentages of that total cost. Would it be easier for you to report in thousands of Kilowatt hours or to report percentages of the total cost?
Wrap-up
Are there any other comments or suggestions you would like to make about anything we have gone over today?
Thank the respondent for their time.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Aryn Hernandez (CENSUS/ESMD FED) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |