Cog Testing Round 1

Round One AIES Cognitive Interview Findings

December 9, 2021

Rebecca Keegan, Economic Statistical Methods Division

Aryn Hernandez (ESMD)

Kristin Stettler (ESMD)

 

Cognitive Interviewing Methodology

What is Cognitive Interviewing?

<number>

A methodology used to test and improve survey questions by asking participants open-ended questions. Cognitive testing ensures that survey questions successfully capture the intent of the questions and, at the same time, make sense to respondents.

• •Do respondents understand the questions?  

• • oCan they provide accurate answers?  

• •What is the burden associated with obtaining the data? 

---

 

Cognitive Interviewing Methodology

Cognitive Interviewing for AIES

<number>

For this research cognitive testing will be more of an investigation of respondent process, rather than the typical function of cognitive testing which is usually to test the questions themselves.

• •Through previous research we already have a good sense of how respondents interact with the survey questions and content of the annuals.  

• •Especially in round one, the research is more geared towards understanding how the groupings of the modules and various parts of the survey work together in a holistic sense, and to understand if changes are needed in this sense.  

 

Cognitive Interviewing Methodology

<number>

 

Cognitive Interviewing Methodology

<number>

• •Understand respondents’ preferences regarding organization of the modules and their content 

• • •How do the modules relate in a holistic sense? 

• •Understand how the module topics align with respondents’ record keeping practices 

• •Investigate further the relationship between company and establishment record keeping practices  

• •Learn about respondents’ strategies for compiling the data for the questionnaire, with an eye towards identifying difficulties and mismatches between recorded and requested data, along with how those mismatches are resolved 

• •Assess the capabilities and preferences of respondents answering via spreadsheet upload 

• •Assess how a reporting change like this might affect respondent burden 

• •Linkage not just between the modules but back to the consolidated figures (totals) 

 

Demographics

Job Titles

• •Financial Planning and Analysis Manager 

• •Manager of Corporate Finance for U.S. Operations, 

• •Manager of External Financial Reporting 

• •Manager of Finance Reporting 

• •Manager of Financial Planning and Analysis 

• •Manager of Regulatory Reporting 

<number>

 

Findings

• •Overview  

• •Reporting Options 

• •Question Content: Location ID & Primary Business Activity  

• •NAICS & Company Summary Page 

• •Module 2 

• •Module 3 

• •Take Aways 

 

<number>

 

Overview

• •Seemed clear, R’s thought the overview page gave them a good idea of what to expect from the survey 

Interpreting “Company-Level Data”

• •Generally well understood, many R’s interpreted this to include the “nature of the company”, “basic hierarchy” or “call report like address etc.”  

• • • • oSome assumed it would include “how many owners- public or private; how many employees you have.” 

• •“Company-Wide Data” was seen as synonymous, some had a slight preference to this. 

<number>

 

 

Reporting Options

Thoughts on instructions

• •Some had questions as to what “locations” entails, and wanted more detail as to what kind of locations were going to be asked about 

• • • • o“Locations meaning what? manufacturing? Admin? Some aren't active. Maybe say what KIND of locations, or OPERATIONAL locations” 

• •Some R’s mentioned appreciating the note that they can change decision if you need to 

  • • • oNeed to include a note about whether data would be lost after changing paths 

• •Several mentioned the recommendations being useful  

• • •The PDF link was mentioned multiple times as a feature they would use.  

Preference towards spreadsheet for almost all:

• •R’s often wondered if you need to fill out 400 rows if they have 400 locations 

<number>

 

Question Content: Location ID

Instructions

• •It was unclear to some R’s whether the Location ID is something to be matched to a Census code, or whether it is internal. Some wondered if this would be assigned to them. 

• • • • oBe clear in instructions state that this can be a nickname that is a meaningful identifier to you, or the ID you refer to each location as. Note that they can change a prefilled cell to their own preferred nickname. 

<number>

 

Question Content: Location ID

Location ID as proper indicator in Mods 2 & 3:

• •Most R’s stated that the location ID alone would be sufficient identifier, but that providing the location ID in  addition to the address would leave no room for confusion 

• • • • o“Location ID is clear- straightforward. I know what to input, we have NUMBERED each location. Really meaningful in [company] basically go by that SOLEY in case you need a second reference also use the address. Some places might use that same ID for offsite showroom” 

<number>

 

Question Content: Primary Business Activities

<number>

Familiarity with NAICS

• •There is variability. Some R’s know their industry and NAICS code off the top of their head, others will need to look it up. 

• • • • oOne person who recently had to look up the code said the process was arduous 

      • oThe broader the NAICS category (2 digit versus 4) the more confident the R will feel in their response. “Familiar, not sure what our designation is, would have to look it up. Food manufacturing [is the] broad category, then there’s a bakery” 

• • •Respondents do not consistently accurately differentiate between ‘sector’ and ‘industry’ as terms.  

    • • o“Industry is larger scope than sector which is subset.” ; “I think they’re the same. Sector more your government terminology, but industry is more conversational. Without example we assume it's the same. We do not think those are different.” 

  • •For most R’s there will be little to no variation between establishment level industry codes, but the more detail required , the more difficult it will be for the respondent to provide, i.e., retail is simple but the kind of retail which may vary by establishment is more difficult.   

 

 

NAICS & Company Summary Page

<number>

 

Module 2

Groupings Thoughts

• •In general R’s felt that the topics were well aligned and would be accessible in their records. 

• •Some respondents stated that  pulling for establishment level data may be easier than at some other level such as state or industry.  

• •Some exceptions included:  

• • • • oSome concern over data related to e-commerce some companies do not maintain this data in an easily accessible way 

• • • • • E-commerce as a concept can be difficult without clear instructions/examples  

      • oWhat level of detail for sales/receipts they would need to pull at the establishment level . Some R’s were concerned about the actual quantity of data they would need to pull  

        • “to pull those reports, they're like a gigabyte”.  

      • oPotential overlap between accounting topics (would ‘detailed sales’ include electronic sales previously reported? Are sales and revenue treated differently?) 

      • oSome were unfamiliar with unfilled orders  

      • oContracting agencies sometime do not maintain their records by location making this task difficult. One R mentioned to gather data by location she would need to reach out to each establishment.  

<number>

 

Module 3

Grouping Thoughts

• •R’s seemed to think that the topics made sense together 

• • • • oThere is variation regarding whether or not the topics will be stored in one system/report, but R’s seemed to think the data would be easily accessible.  

• •Concern about pulling data at the establishment level for a multitude of companies was brought up again 

  • • • o“This is like a balance sheet- is this for company or location based? [A couple hundred] balance sheets would need to run a lot of reports for each location” 

• • •R&D data is often stored separately or may require reaching out to locations 

    • • oSeveral R’s had questions about robotics because they can be a confusing concept. Some mentioned they would need to reach out to individual locations to learn more.  

<number>

 

Take Aways

Overall, participants seemed enthused by the idea of merging the annual surveys.

• •Many mentioned that it seemed like a “good idea”. The general flow of the modules seemed to make sense to them. 

Totals from Mod One

• •Several R’s liked the idea of providing the totals from Mod one. The consistent question was if there would be guidance as to whether their detailed accounts should add up to those totals or not.  

• • • • o“Difficult to reconcile there could be variance for different reasons. gross receipt for each location might not add up to consolidated. “ 

      • o“Mod one would be great to refer back to - what you're reporting on a detail, scales back to totals- like quality check. Tell us if you want it to match back- if it doesn't match they can go back and adjust so it matches. As accountant like it to match.” 

<number>

 

Take Aways

Burden

• •Burden was generally anticipated to be about the same in terms of pulling/collecting data. But the time savings would be in terms of lack of redundancy between previously separated surveys. 

• •Another person mentioned that it was coordinating the various responses to the surveys that arrived at numerous locations and with different due dates  

• •For many, annual data is much easier to pull than quarterly.  

• • • • o“pay by quarter time consuming” column M limited information about NEW OWNER we would know it was sold. Anything ANNUAL we can pull much easier than quarterly to pull. 

Filter Questions & Skip Logic

• •Where possible, utilize filter questions & skip Logic. Templates should reflect their answers. This will majorly reduce respondent burden. 

  • • • oR’s sometimes commented that in filling out surveys they have to manually skip past whole sections, such as robotics, or e-commerce. 

Prefilling

• •Participants generally expect that data such as addresses and other consistent data will be prefilled.  

<number>

 

Take Aways

Spreadsheets

• •Almost all respondents indicated they would prefer to provide their data via spreadsheet.  

• • • • oOne qualification is that spreadsheets are most useful when there are a multitude of rows (estabs) to fill out data for. For this reason, if Module one does not ask for estab level data, it should only be offered as questions within the instrument.  

Many R’s anticipated utilizing PDFs

• •        whether to get a see the questions to make a plan for data collection 

• •        or to email relevant questions to other departments/locations 

Delegating Questions

• •Some respondents thought breaking out Payroll (and potentially employee count)  would make sense 

  • • • oPayroll is often stored separately from other accounting data  

      • oPayroll also tends to be the most sensitive.  

• •R’s often do not realize that e-corr has a delegate feature and usually report either email specific questions or send a listing of the questions via pdf or excel sheet.  

<number>

 

Next Steps

<number>

 

<number>

 

File Type	application/vnd.openxmlformats-officedocument.presentationml.presentation
File Modified	0000-00-00
File Created	0000-00-00

---

© 2024 OMB.report | Privacy Policy

---