BRDIS Census Taker Usability Testing Protocol
4/13/09
Background:
Please tell me a little about your company’s activities, and your role in the company.
Have you completed the R&D survey in the past? On paper or electronically?
What do you think are advantages and disadvantages of online reporting?
Please tell me a little about your company’s R&D activities.
Have you reported electronically to any surveys in the past?
What made you choose to report electronically?
What was your overall experience? Did anything stand out as being very helpful? Was there anything you didn’t like?
Login Page:
What do you think about the information presented on this screen?
Is there any other information that you would like to see on this screen?
Do the three bulleted instructions make sense to you?
How did you locate the username and password for this screen?
Form Menu:
Does the information displayed on this screen make sense to you?
What did you do here? Where did you go in order to start filling out information?
In your own words, what are the other options on this page? What would you get if you clicked on them?
If you wanted to glance at the help instructions before starting the form, where would you go?
Is there anything else that you would need at this point in order to start completing the form? Anything else that should be included on this screen?
Section 1: General Instructions
What do you notice on this screen?
What did you do here?
What do you think about the amount of information presented on this screen?
Do you have any suggestions about the layout of this screen?
Section 2: Company Information
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 3: (1.1) Survey Coordinator
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 4: (1.2) Company Operations
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 5: (1.3) Company Ownership
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 6: (1.3 continued) Reporting Instructions
What do you notice on this screen?
What did you do here?
What do you think about the amount of information presented on this screen?
Do you have any suggestions about the layout of this screen?
Section 7: (1.4) Company R&D Activity
What do you think about this screen? What did you do here? Do you have any suggestions?
What does “incur R&D expense” mean to you?
What does the second bullet mean to you?
Section 8: (1.5) Business Codes
Does the layout of information on this screen make sense to you?
What did you do here?
Did you read the questions and answers below the question?
Section 9: (1.5) Business Code Verification
Does the layout of information on this screen make sense to you?
What did you do here?
Did you click on the code list from the link or refer to the copy in the questionnaire?
Did you have any difficulties adding additional codes?
Section 10: (1.5 continued) Updated Business Codes
Does the layout of information on this screen make sense to you?
What did you do here?
If you had to make corrections, how did you do that?
Section 11: (1.6-7) Nets Sales and Operating Revenues
Does the layout of information on this screen make sense to you?
What did you do here?
Did you read the include/exclude lists?
What does “from domestic operations” mean to you?
Section 12: (1.8) Sales and Operating Revenues by Business Code
Does the layout of information on this screen make sense to you?
What did you do here?
If the business codes listed weren’t correct, how would you correct them?
Please tell me in your own words what columns 1 and 2 mean.
Section 13: Instructions for Questions 2.1-2.17: R&D Paid for by Your Company (R&D Expense)
What do you notice on this screen?
What did you do here? Did you read all the text?
What do you think about the amount of information presented on this screen?
Do you have any suggestions about the layout of this screen?
Section 14: (2.1) Worldwide R&D Expense
In your own words, what is this question asking for?
Did you read the include and exclude lists?
Does our definition of R&D make sense to you? Does it match how your company keeps its records?
Is it clear what to include and exclude?
How did you come up with an answer to this question? Did the data come from records or did you estimate?
Section 15: (2.3) Worldwide R&D Performed by Your Company
Does the question and layout of information make sense to you?
In your own words, please tell me what rows a and b are asking for.
What did you do here? Where did you get your information?
Section 16: (2.4) Payroll
Does the question and layout of information make sense to you?
What did you do here?
Section 17: (2.5) R&D Performed by Others
What does R&D performed by others mean to you in this question?
What did you do here?
Do the categories listed match how your company tracks this information?
How did you come up with an answer to this question? Did the data come from records or did you estimate?
Section 18: (2.6) Worldwide R&D Performed by Your Company by Country
Does the question and layout of information make sense to you?
Did you have any R&D in row Q? If so, which countries were included?
How did you come up with an answer to this question? Did the data come from records or did you estimate?
Section 19: (2.7) Domestic R&D Performed by Your Company Paid for by Your Company’s Foreign Operations
Does the question and layout of information make sense to you?
What did you do here?
In your own words, what is this question asking for.
Section 20: (2.8) Domestic R&D by Type of Cost
What do you think about this screen? What did you do here? Do you have any suggestions?
Do the categories listed match how your company tracks this information?
Section 21: (2.9) Domestic R&D by State
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 22: (2.10-11) Largest Domestic R&D Location
What do you think about this screen? What did you do here? Do you have any suggestions?
Was 2.11 easy or difficult to provide?
Section 23: (2.12) Worldwide R&D Expense by Business Code
What do you think about this screen? What did you do here? Do you have any suggestions?
How did you come up with an answer to this question? Did the data come from records or did you estimate?
Section 24: (2.13) Domestic R&D Expense
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 25: (2.14) R&D by Business Code
What do you think about this screen? What did you do here? Do you have any suggestions?
Please tell me in your own words what columns 1 and 2 are asking for?
How did you come up with answers to this question? Did the data come from records or did you estimate?
Section 26: (2.15) Domestic Capital Expenditures
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 27: (2.16) Domestic Capital Expenditures for R&D
What do you think about this screen? What did you do here? Do you have any suggestions?
Please tell me in your own words what rows a-d are asking for.
Section 28: (2.17) Projected R&D Expense
What do you think about this screen? What did you do here? Do you have any suggestions?
How did you come up with answers to this question? Did the data come from records or did you estimate?
Section 29: Instructions for Questions 3.1-3.41 Management and Strategy of R&D Expense
What do you notice on this screen?
What did you do here?
What do you think about the amount of information presented on this screen?
Do you have any suggestions about the layout of this screen?
Section 30: (3.4) Worldwide R&D Expense
What do you think about this screen? What did you do here? Do you have any suggestions?
In your own words, what is this question asking for?
How did you come up with an answer to this question? Did the data come from records or did you estimate?
Section 31: (3.5) Worldwide R&D Expense
What do you think about this screen? What did you do here? Do you have any suggestions?
In your own words, what is this question asking for?
How did you come up with an answer to this question? Did the data come from records or did you estimate?
Section 32: (3.6) Worldwide R&D Expense
What do you think about this screen? What did you do here? Do you have any suggestions?
In your own words, what is this question asking for?
How did you come up with an answer to this question? Did the data come from records or did you estimate?
Section 33: (3.7) Worldwide R&D Expense
What do you think about this screen? What did you do here? Do you have any suggestions?
In your own words, what is this question asking for?
How did you come up with an answer to this question? Did the data come from records or did you estimate?
Section 34: (3.8) R&D Agreements
What do you think about this screen? What did you do here? Do you have any suggestions?
In your own words, what is this question asking for?
How did you come up with an answer to this question? Did the data come from records or did you estimate?
Does the definition of “R&D agreements” match how you think of them?
Section 35: (3.9) R&D Agreements
What do you think about this screen? What did you do here? Do you have any suggestions?
In your own words, what is this question asking for?
How did you come up with an answer to this question? Did the data come from records or did you estimate?
Section 36: (3.10) R&D Agreements
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 37: (3.11) R&D Agreements
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 38: (3.12) R&D Agreements
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 39: (3.13) New R&D Agreements
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 40: (3.14) New R&D Agreements
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 41: (3.15) New R&D Agreements
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 42: (3.16) Characteristics of Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
In your own words, what do “research” and “development” mean to you?
Section 43: (3.17) Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
In your own words, what do “applied research” and “basic research” mean to you?
Section 44: (3.18) Characteristics of Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 45: (3.19) Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 46: (3.20) Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 47: (3.21) Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 48: (3.22) Areas of Application for Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 49: (3.23) Areas of Application for Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 50: (3.24) Areas of Application for Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 51: (3.25) Areas of Application for Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 52: (3.26) Areas of Application for Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 53: (3.27) Technology Focus of Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 54: (3.28) Technology Focus of Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 55: (3.29) Technology Focus of Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 56: (3.30) Technology Focus of Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 57: (3.31) Technology Focus of Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 58: (3.32) Technology Focus of Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 59: (3.33) Technology Focus of Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 60: (3.34) Technology Focus of Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 61: (3.35) Technology Focus of Domestic R&D Performed by Your Company
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 62: (3.36) R&D Performed by Others
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 63: (3.37) R&D Performed by Others
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 64: (3.38) R&D Performed by Others
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 65: (3.39) R&D Performed by Others
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 66: (3.40) Activities with Academia
What do you think about this screen? What did you do here? Do you have any suggestions?
In your own words, please tell me what rows a-e are asking for.
Section 67: (3.41) Company Characteristics
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 68: Instructions for Questions 4.1-4.27: R&D Funded or Paid for by Others
What do you notice on this screen?
What did you do here?
What do you think about the amount of information presented on this screen?
Do you have any suggestions about the layout of this screen?
Section 69: (4.1) Worldwide R&D That was Funded by Others
In your own words, what is this question asking for?
Did you read the include and exclude lists?
Does our definition of funded R&D make sense to you? Does it match how your company keeps its records?
Is it clear what to include and exclude?
How did you come up with an answer to this question? Did the data come from records or did you estimate?
Section 70: (4.2) Worldwide R&D That was Funded by Others
What do you think about this screen? What did you do here? Do you have any suggestions?
In row b, please tell me in your own words what this question is asking for. What does “such as subcontractors or subgrantees” mean to you?
How did you come up with an answer to this question? Did the data come from records or did you estimate?
Section 71: (4.4) Worldwide R&D Performed by Your Company Funded by Others by Country
What do you think about this screen? What did you do here? Do you have any suggestions?
Did you have any R&D in row Q? If so, which countries were included?
How did you come up with an answer to this question? Did the data come from records or did you estimate?
Section 72: (4.5) Domestic R&D Performed by Your Company Funded by Others by Business Code
What do you think about this screen? What did you do here? Do you have any suggestions?
How did you come up with an answer to this question? Did the data come from records or did you estimate?
Section 73: (4.6) Domestic R&D Performed by Your Company Funded by Others by Type of Costs
What do you think about this screen? What did you do here? Do you have any suggestions?
How did you come up with an answer to this question? Did the data come from records or did you estimate?
Section 74: (4.7) Domestic R&D Performed by Your Company Funded by Others by Type of Funding
What do you think about this screen? What did you do here? Do you have any suggestions?
What do you think rows a and b are asking for here?
Section 75: (4.8) Domestic R&D Performed by Your Company Funded by Others by Source of Funding
What do you think about this screen? What did you do here? Do you have any suggestions?
Do the categories listed match how your company tracks this information?
Section 76: (4.9) Domestic R&D Performed by Your Company Funded by Others
What do you think about this screen? What did you do here? Do you have any suggestions?
Do the categories listed match how your company tracks this information?
Section 77: (4.10) Domestic R&D Performed by Your Company Funded by the U.S. Federal Government
What do you think about this screen? What did you do here? Do you have any suggestions?
Do the categories listed match how your company tracks this information?
Section 78: (4.11) Worldwide R&D Performed by Your Company Funded by Others
What do you think about this screen? What did you do here? Do you have any suggestions?
What does “independent research and development (IR&D)” mean to you?
How did you come up with answers to this question? Did the data come from records or did you estimate?
Section 79: (4.12) Domestic R&D Performed by Your Company Funded by Others by State
What do you think about this screen? What did you do here? Do you have any suggestions?
In your own words, please tell me what columns 1-3 are asking for.
How did you come up with answers to this question? Did the data come from records or did you estimate?
Section 80: (4.13) Projected R&D Funded or Paid for by Others
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 81: (4.14) Projected R&D Funded or Paid for by Others
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 82: (4.15) Projected R&D Funded or Paid for by Others
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 83: (4.16) Projected R&D Funded or Paid for by Others
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 84: (4.17) Characteristics of Domestic R&D Performed by Your Company Funded by Others
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 85: (4.18) Domestic R&D Performed by Your Company Funded by Others
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 86: (4.19) Domestic R&D Performed by Your Company Funded by Others
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 87: (4.20) Domestic R&D Performed by Your Company Funded by Others
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 88: (4.21) Domestic R&D Performed by Your Company Funded by Others
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 89: (4.22) Domestic R&D Performed by Your Company Funded by Others
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 90: (4.23) Domestic R&D Performed by Your Company Funded by Others
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 91: (4.24) Domestic R&D Performed by Your Company Funded by Others
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 92: Instructions for Questions 5.1-5.11: R&D Human Resources
What do you notice on this screen?
What did you do here?
What do you think about the amount of information presented on this screen?
Do you have any suggestions about the layout of this screen?
Section 93: (5.1) Worldwide Employees (Including Domestic)
What do you think about this screen? What did you do here? Do you have any suggestions?
Did you read the include and exclude lists?
How did you come up with answers to this question? Did the data come from records or did you estimate?
Section 94: (5.2) Worldwide R&D Employees
What do you think about this screen? What did you do here? Do you have any suggestions?
Did you read the include and exclude lists?
How did you come up with answers to this question? Did the data come from records or did you estimate?
Section 95: (5.3) Worldwide R&D Employment by Occupation and Gender
What do you think about this screen? What did you do here? Do you have any suggestions?
Do the categories listed in the rows and columns match how your company tracks this information?
How did you come up with answers to this question? Did the data come from records or did you estimate?
Section 96: (5.4) Worldwide R&D Employment by Educational Attainment
What do you think about this screen? What did you do here? Do you have any suggestions?
Do the categories listed match how your company tracks this information?
How did you come up with answers to this question? Did the data come from records or did you estimate?
Section 97: (5.5) Domestic Employees
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 98: (5.6) Domestic R&D Employees
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 99: (5.7) Domestic R&D Employment by Occupation and Gender
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 100: (5.8) Domestic R&D Employment by Educational Attainment
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 101: (5.10) Domestic Full-time Equivalents (FTEs) for R&D Scientists and Engineers
What do you think about this screen? What did you do here? Do you have any suggestions?
Did you read the definition of “full-time equivalents”?
How does that match how your company tracks this information?
How did you come up with answers to this question? Did the data come from records or did you estimate?
Section 102: (5.11) Domestic R&D Full-time Equivalents (FTEs)
What do you think about this screen? What did you do here? Do you have any suggestions?
How easy or difficult was it for you to answer this question?
Section 103: Instructions for Questions 6.1-6.10: Intellectual Property, Technology Transfer, and Innovation
What do you notice on this screen?
What did you do here?
What do you think about the amount of information presented on this screen?
Do you have any suggestions about the layout of this screen?
Section 104: (6.1) Innovation
What do you think about this screen? What did you do here? Do you have any suggestions?
In your own words, please tell me what you think each row (a-e) is asking for.
How did you come up with answers to this question? Did the data come from records or did you estimate?
Section 105: (6.2) Patents
What do you think about this screen? What did you do here? Do you have any suggestions?
How did you come up with answers to this question? Did the data come from records or did you estimate?
Section 106: (6.3) Patents
What do you think about this screen? What did you do here? Do you have any suggestions?
In your own words, please tell me what this question is asking for.
How did you come up with answers to this question? Did the data come from records or did you estimate?
Section 107: (6.4) Patents
What do you think about this screen? What did you do here? Do you have any suggestions?
In your own words, please tell me what this question is asking for.
How did you come up with answers to this question? Did the data come from records or did you estimate?
Section 108: (6.5-6) Patents
What do you think about this screen? What did you do here? Do you have any suggestions?
In your own words, please tell me what these questions are asking for.
How did you come up with answers to these questions? Did the data come from records or did you estimate?
Section 109: (6.7) Patent Licensing
What do you think about this screen? What did you do here? Do you have any suggestions?
In your own words, please tell me what this question is asking for.
How did you come up with answers to this question? Did the data come from records or did you estimate?
Section 110: (6.8) Patent Licensing
What do you think about this screen? What did you do here? Do you have any suggestions?
In your own words, please tell me what this question is asking for.
How did you come up with answers to this question? Did the data come from records or did you estimate?
Section 111: (6.9) Intellectual Property Transfer Activities
What do you think about this screen? What did you do here? Do you have any suggestions?
In your own words, please tell me what each row is asking for.
How did you come up with answers to these questions? Did the data come from records or did you estimate?
Section 112: (6.10) Intellectual Property Protection
What do you think about this screen? What did you do here? Do you have any suggestions?
In your own words, please tell me what each row is asking for.
How did you come up with answers to these questions? Did the data come from records or did you estimate?
Section 113: Reporting Period
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 114: Reporting Information
What do you think about this screen? What did you do here? Do you have any suggestions?
Section 115: Remarks
Do you typically make many comments on the forms that you fill out?
What do you think about the layout of this page?
Do you have any concerns with this page being at the end of the form?
If you had a comment on an item earlier in the form, how did you provide the comment?
Check / Finish
What are your first thoughts when seeing this screen?
Can you explain to me what is expected of you? What is this screen telling you?
(If there are edits) How did you go about fixing this/these problem(s)? When you finished fixing the problems, what did you do next? How did you submit the form?
Do the instructions for parts A and B make sense?
What do you think of the title of this screen? Is it a logical title?
If you were to click on ‘Form Not Finished’ where would you expect to be taken from here? What would you expect to see?
If you were to click on ‘Form Finished’ where would you expect to be taken from here? What would you expect to see?
Now that you have completed the survey, what do you think about the different options on this screen? What will each option give you?
What is the function of the dropdown list at the top of the screen? Is this helpful? Are the options in the list self-explanatory? How could we improve this?
Are there any options on the screen that you don’t think are necessary? Are there any options that we don’t offer that you would like to see?
What do you think of the title of this screen? Does it make sense to you? Do you have recommendations for renaming it?
When filling out this type of survey, do you want to keep a copy of it for your records? How would you like to do that in this system? What do you think about the options for doing this in Census Taker?
What do each of the print/save options mean? Which do you prefer? Why?
Is the meaning of the word “Certification” clear? Is there a better way to word this option?
If you had a question or problem while completing the survey, how did you contact the Census Bureau? Would you be interested in contacting us electronically with a question? If so, where on this site would you do that?
Did you complete the online form in one sitting?
Did you still use the paper form? If so, how?
What is your overall impression of the online form? Was it easy, hard?
What do you think about how the online form itself works, the way it is set up?
Were you able to navigate easily through the form?
Was there any information about the online form that would have been helpful before starting? How would you like to have been given this information?
What are some of the things that you liked the most about the online form?
What are some of the things that you liked the least about the online form?
Do you have any suggestions for how to make this online form easier for you to use?
Thank you for your time and input.
| File Type | application/msword |
| File Title | R&D Census Taker Cognitive Testing Protocol – 03/28/08 |
| Author | Elvin J. Thomas |
| Last Modified By | Bureau Of The Census |
| File Modified | 2009-04-13 |
| File Created | 2009-04-13 |