2015 Management and Organizational Practices Survey (MOPS)
Cognitive Testing Protocol
Ken Pick
Michael Brennan
Research Questions:
Do respondents interpret the new questions and instructions in a manner that meets the questionnaire’s intent?
Do respondents comprehend the terminology used in the 2015 MOPS? (If not, what terminology are they using?)
Overall, is the requested information available in respondent’s records? (Furthermore, is it available for the time frames requested in the questionnaire?)
Did respondents need to go to other people or departments within their organization to report the data requested on the 2015 MOPS?
Expected Length of Interview: 60 minutes
Materials Needed:
2-3 copies of the draft 2015 MOPS
Consent forms
Digital recorder
Business cards
Pen and/or pencils
Introduction:
People around the table: how long at company, position/title and responsibilities
Explain purpose of meeting: to understand the process used to answer questions
Explain that we are not testing the respondent – we only want information
Structure of meeting: general and specific questions about the form itself, as well as the organization
Permission to record discussion? Have R sign consent form.
General probes to use throughout the interview:
Reflect back R’s specific answer: “you said…”
How did you arrive at this number/answer this question? / Can you tell me more about this?
What records (if any) would you look at? What line or line items would be of interest?
Specifically, what would you include in this number? What would you exclude?
Before discussing the 2015 MOPS questionnaire:
To begin, can you please tell me a little about your plant?
(If needed) What sorts of activities is it involved in?
What is your role at this plant?
(If needed) Have you filled out government forms in the past? Which ones?
(If needed) Generally speaking, how do government forms get completed at your plant?
(If needed) Have you completed the MOPS before? (If yes) What was your overall impression of the MOPS?
Now, I’m going to hand you a copy of the 2015 MOPS questionnaire. Since we are only interested in feedback on some of the questions in the 2015 MOPS, we’ll only go through some of the questions on some of the pages. Feel free to write in the questionnaire and think aloud as to how you would complete these questions. After you complete a question, I’ll ask you follow-up questions about the process you took to formulate an answer.
To begin, let’s turn to Page 7 and look at Question 17. Please take a moment to read the question and answer it.
In your own words, what is this question asking you?
(If needed) How would you go about answering this question?
Did you notice the skip pattern?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you in the answer that you would provide to this question for 2010? What about 2015?
(Data Driven Decisions)
Next, let’s go to Question 24. Again, please take a moment to read the question and answer it.
In your own words, what is this question asking you?
(If needed) How would you go about answering this question?
(If needed) What kind of data was available in 2010? What kind of data is available in 2015?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you in the answer that you would provide to this question for 2010? What about 2015?
Question 25:
How did you go about completing this question?
(If needed) What does the phrase “use of data” mean to you?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you in the answer that you would provide to this question for 2010? What about 2015?
Question 26:
How did you go about completing this question?
Are there any other response options that would make sense to you?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you in the answers that you would provide to this question for 2010? What about 2015?
Question 27:
In your own words, what is this question asking you?
(If needed) How would you go about answering this question?
What does the phrase “production performance indicators and instruments” mean to you? Can you give me some examples?
What does the phrase “employee-specific performance indicators” mean to you? Can you give me some examples?
What does the term “controlled experiments” mean to you? Can you give me some examples?
How would you answer this question if one of the sources of data did not apply to you establishment?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you in the answers that you would provide to this question for 2010? What about 2015?
Question 28:
In your own words, what is this question asking you?
(If needed) How would you go about answering this question?
What does the term “demand forecasting” mean to you?
What does the term “supply chain management” mean to you?
Are there other activities at your plant influenced by data analysis that aren’t currently listed as response options?
How would you answer this question if one of the activities did not apply to you establishment?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you in the answers that you would provide to this question for 2010? What about 2015?
Question 29:
How did you go about completing this question?
How would you answer this question if one of the activities listed did not apply to your establishment? What would you select?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you in the answers that you would provide to this question?
Question 30:
In your own words, what is this question asking you?
(If needed) How would you go about answering this question?
What does the term “predictive analytics” mean to you?
How would you answer this question if your establishment did not use predictive analytics?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you in the answers that you would provide to this question for 2010? What about
2015?
(Uncertainty)
Question 31:
How did you go about completing this question?
How would you know the approximate dollar value of products shipped for the 2016 calendar year?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you in the answer that you would provide to this question?
Question 32:
In your own words, what is this question asking you?
(If needed) How would you go about answering this question?
(If needed) Looking at the left column, can you tell me in your own words what this column is asking you to do? What is the difference between lowest and low? How about low and medium?
(If needed) How would you know the approximate dollar value of shipments in 2017?
(If needed) Looking at the right column, can you tell me in your own words what this column is asking you to do?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you in the answers that you would provide to this question?
Question 33:
How did you go about completing this question?
(If needed) What does the term “capital expenditures” mean to you?
How would you know the approximate dollar value of capital expenditures for the 2016 calendar year?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you in the answer that you would provide to this question?
Question 34:
In your own words, what is this question asking you?
(If needed) How would you go about answering this question?
(If needed) Looking at the left column, can you tell me in your own words what this column is asking you to do? What is the difference between lowest and low? How about low and medium?
(If needed) How would you know the approximate dollar value of capital expenditures in 2017?
(If needed) Looking at the right column, can you tell me in your own words what this column is asking you to do?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you in the answers that you would provide to this question?
Question 35:
How did you go about completing this question?
(If needed) What does the term “employee” mean to you? Does the term include or exclude contractors?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you in the answer that you would provide to this question?
Question 36:
In your own words, what is this question asking you?
(If needed) How would you go about answering this question?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you in the answers that you would provide to this question?
Question 37:
How did you go about completing this question?
How would you know the approximate dollar value of payroll (before deductions) for the 2016 calendar year?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you in the answer that you would provide to this question?
Question 38:
In your own words, what is this question asking you?
(If needed) How would you go about answering this question?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you in the answers that you would provide to this question?
Question 39:
How did you go about completing Question 39?
What does the term “intermediate inputs” mean to you?
How would you know the approximate dollar value of intermediate inputs for the 2016 calendar year?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you in the answer that you would provide to this question?
Question 40:
In your own words, what is this question asking you?
(If needed) How would you go about answering this question?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you in the answers that you would provide to this question?
(Background Characteristics)
Okay, let’s skip to Question 48 on Page 14:
What is Question 48 asking you?
(If needed) How did you answer this question?
What does the term “family firm” mean to you?
Wrapping Up:
Overall, what do you think of these questions that we looked at today?
Just thinking about these questions that you answered today, how much time do you think it would take you to complete them? How likely would you be to answer these questions if we were not here?
Finally, I want to briefly change topics and discuss with you the idea of transferring files between businesses.
Have you heard of the term file transfers before today? (Yes/No)
Is this a term that is used within your organization? (Yes/No)
(If no) What term is used?
Have you ever provided financial or other data to another party outside of your organization in the form of a file transfer?
(If yes) Can you briefly tell us about your experiences with file transfers?
Probe if needed: what type of organization did you transfer the data to?
Probe if needed: what type of data did you send?
(If no) Are you familiar with the concept of a file transfer?
Do you know of anyone in your company who is involved with file transfers?
On scale of 1 to 5, with 5 being very willing and 1 being very unwilling, how willing would your company be to provide data to the Census Bureau in this manner, in lieu of completing a survey?
What software does your company use for your accounting and payroll needs?
Would you be willing to talk about file transfers with us in more detail at a later time?
Thank you for your time today!
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Kenneth M Pick |
| File Modified | 0000-00-00 |
| File Created | 2021-01-28 |