Response to Dr. Andrew Reamer’s comments to the MIST OMB Clearance Package

NCSES Memorandum

October 10, 2014

To: Shelly Wilkie Martinez

Office of Management and Budget

Via: John Gawalt, Division Director

Jeri Mulrow, Deputy Division Director

National Center for Science and Engineering Statistics

National Science Foundation

From: John Jankowski, RDS Program Director

Audrey Kindlon, Survey Statistician

National Center for Science and Engineering Statistics

National Science Foundation

Subject: Response to Dr. Andrew Reamer’s comments to the MIST OMB Clearance Package

NCSES very much appreciates Dr. Andrew Reamer’s (George Washington University) support of the Microbusiness, Innovation, Science and Technology (MIST) survey testing and for his thoughtful consideration of the MIST OMB Clearance Package. After careful review of his memo below is the NCSES response.

As noted in the Clearance Package our request is to continue our testing with a larger sample. We have made several changes to our approach to address areas needing improvement. As we move forward with testing we anticipate making changes when necessary and conducting a full evaluation of all methods and of the sample once the data collection is complete.

Survey Instrument

Up to this point, the majority of MIST work has been on the development of the questionnaire. To date, MIST focused on determining the relevant research questions and designing the MIST questionnaire. This work included 48 cognitive interviews, consultation with an expert panel specially created for MIST, a pretest of 74 businesses and debriefing interviews with 20 of those respondents. NCSES is now requesting clearance for additional testing and follow up debriefing interviews to develop and test the survey methods and investigate ways to encourage response. As part of this effort, MIST will continue to evaluate the questionnaire.

• Regarding Dr. Reamer’s comments on question 5, we believe providing some background on the origins of the question will be helpful. In February 2012, NCSES conducted an Expert Panel to solicit advice from both Federal and nonfederal data users. Participants in the Microbusiness Expert Panel (MEP) are included in Attachment A. Question 5 was developed in response to those recommendations. Specifically the MEP requested a question on those outside the company structure who provide advice, coaching or mentoring. MEP members wanted to learn about individuals that also provide input to the business and are not employees or contractors. We tested and revised question 5 through two rounds of cognitive testing. Our debriefings found there were no issues with the wording.

• Dr. Reamer suggested adding the word ‘new’ to question 6. During prior rounds of testing, the questions were revised. As worded, Question 6 did not appear to cause confusion for respondents. We will re-evaluate during the debriefing interviews.

• Regarding Dr. Reamer’s comments about questions 28 – 30, we tested multiple versions of the Research and Development (R&D) questions making modifications to the questions between each round of cognitive interviewing. These questions were then administered in the pretest and further explored during the debriefing interviews. Respondents did not appear to have problems understanding the R&D questions.

The suggestion to combine questions 28-30 into one question and then subsequently ask for percentages of the amount reported in Questions 27 seems somewhat problematic. Question 30 requires a dollar figure (the cost of R&D services performed by the company – Question 29) to be broken out into salaries, wages, equipment, software, and other costs to be reported as a percent of the dollar total. If we revise the question as suggested we would be asking the respondent to break down a percentage into further percentages. It is unclear if the intent is to have the figures in the breakdown add up to 100% as they do in current version of Question 30 or if the intent is for the figures in the breakdown to add up to the percentage provided in the cost of R&D services performed by the company.

The current structure of the questions 28-30 is employed in BRDIS and appears to work for larger companies. We will evaluate during the debriefing interviews to see if it works for microbusinesses.

• Dr. Reamer suggests changing Question 31 to identify funding sources for all R&D. The only portion of funding excluded from Question 31 is the costs of R&D services purchased from others asked in Question 28. Question 31 is intentionally limited to sources for microbusiness’ performance, excluding purchased R&D services, to address the data needs for our National Patterns (U.S. total) R&D performance tabulations. Purchased R&D services represent performance that is captured on other R&D surveys about which the performer is asked to report the original funding sources for such activity.

Sampling Frame Source

Dr. Reamer noted that there are several options for developing the MIST sample. NCSES considered several sources for the frame, including the Census Bureau’s Business Register. Each source has its own advantages and disadvantages. The potential advantages of using IRS data as the source for the MIST sample are numerous. IRS data provide good coverage of U.S. businesses and allows us to create a nationally representative sample of U.S. microbusinesses.

Tax data are relatively accurate given the enforcement penalties for non-compliance and the monetary advantages of participating in the tax system. We will be allowed to use tax data for editing, imputing, and supplementing survey responses. We may also be able to use it to look retrospectively and if possible prospectively, to study firm performance over time. IRS data provides us flexibility in studying the microbusiness population. Using the IRS data will allow us to explore the richness of the IRS data in a test environment.

A subset of IRS data is used to develop the Census Bureau’s Business Register. The decision to use the IRS data as the sampling source was seen as an opportunity to use a data source that is not frequently used for conducting surveys and to evaluate it for this purpose. It was not a repudiation of the Business Register. Based on our understanding of the Business Register, coverage is similar to IRS data, but it does not have a primary focus on smaller businesses and is not updated as frequently as it is for larger businesses. Additionally, the full range of tax data are not available (e.g., information return data, income data, additional schedules, and forms) at Census.

Following receipt of the memo, we examined the University of Wisconsin’s Institute of Exception Growth Companies (National Establishment Time Series – NETS). The NETS data have a 21-month lag: for example data from January 2012 is not available until approximately October of 2013. IRS data are current and are updated as soon as returns are filed and posted.

NETS is developed from Dun and Bradstreet data and while it is difficult for any database to have all established businesses, IRS data is more comprehensive. In June of 2011, an analysis was conducted on the NETS data. That study (http://exceptionalgrowth.org/insights/NETSvsES-202.pdf) found that NETS may be considered unsuitable for research (the study specifically cites regional employment research), because the data are prepared for commercial purposes rather than as a tool for statistical analysis. The study found births and younger smaller businesses are underreported. Since microbusinesses are the smallest of businesses, NETS does not seem promising. The study also found discrepancies in the number of employees compared to unemployment data, especially among the smallest businesses.

NCSES will examine the quality of the sample frame and the appropriateness of the IRS data for this endeavor as part of the continued testing efforts.

Respondent Incentives

MIST is mindful of the potential for low response rates. During the prior debriefing interviews, we asked respondents what could be done to make the survey more appealing to potential respondents. Respondents noted that the NSF name was helpful so the NSF logo and name have been made more prominent on the survey materials. They requested that the survey be easy to complete, so a web survey has been developed. When probed about what else could be done to increase response, respondents mentioned monetary incentives. We are testing the use of monetary incentives. NCSES worked extensively with researchers from the University of Nebraska to develop our experiments regarding incentives. Prepaid incentives as a small token of appreciation have been shown to increase response rates in surveys of individuals when compared to postpaid incentives or offering no incentive.

As noted earlier, we will use the proposed debriefing interviews to help us understand general reactions to the survey, what motivated businesses to respond and for the non-respondents why they did not respond, and to evaluate the content of the questionnaire and effectiveness of the contact strategies. As suggested in the memo, these will provide opportunity for us to continue our discussion with respondents. It is important to note that during this testing phase, our procedures will be reviewed during and after data collection.

Regarding the suggestion to provide respondents with the opportunity to participate in a web-based conversation with NCSES regarding the potential usefulness of MIST findings for microbusinesses, NCSES will explore this suggestion in our respondent debriefings.

Value of MIST to Federal Policy

Dr. Reamer would like more detail on how MIST will contribute to public policy and he seems to suggest we engage in discussions with the Small Business Administration and the Small Business Innovation Research program.

The MIST survey was originally conceived to fill a data gap in the Business Research and Development Innovation Survey (BRDIS). BRDIS examines R&D and innovation at companies with five or more employees. The National Academy of Sciences’ Committee on National Statistics (CNSTAT) recommended that NCSES, then Science Resources Statistics, explore ways to measure firm innovation and investigate the incidence of R&D activities in growing sectors such as small businesses. The MIST survey was developed to provide data on these areas.

One member of the MEP stated that standard R&D indicators do not necessarily fit well with the innovative activity of small firms and that a significant share of innovation does not come from formal R&D activity. Therefore, instead of the broad questions included in BRDIS, the MIST survey allows us to disaggregate the components of innovation and ask respondents about those components.

MIST also consulted with data users during the development phase, and we plan to present our testing results to a panel of data users to help us better understand the extent to which we are meeting data user needs. We will invite members from the SBA and the SBIR to attend and provide input.

We look forward to conducting this additional testing as there are still many areas to work through regarding MIST.

Attachment A – Microbusiness Expert Panel Participants

PANEL MEMBERS

Ana Aizcorbe

Chief Economist

Bureau of Economic Analysis

Aaron Chatterji

Associate Professor

Duke University

Waverly Ding

Assistant Professor

University of Maryland

Jonathan Eckhardt

Associate Professor

University of Wisconsin - Madison

Leonard Gaines

Program Research Specialist

Empire State Development

Brent Goldfarb

Associate Professor

University of Maryland

Brian Headd

Economist

Small Business Administration

Alicia Robb

Senior Research Fellow

Ewing Marion Kauffman Foundation

Tony Stanco

Executive Director

Angel Investors of Greater Washington

Jeff Stanton

Associate Vice President for Research

Syracuse University

John Walsh

Professor

Georgia Institute of Technology