MIST Part B July 2014

Section B. Description of Statistical Methodology

As noted in Part A, the primary purpose of the additional testing is to build on what was learned during the pretest and debriefing interviews. The questionnaire (found in Attachment A) has already been examined in detail through cognitive interviews and a pretest, though its performance will continue to be monitored. With regard to sampling, the design effect will be measured, and the extent to which the stratification approach helps or hinders the quality of the survey estimates will be examined. With regard to data collection, several experiments will be employed to understand what steps may be taken to encourage a high response rate. The additional testing also will examine how the sample design interacts with data collection: e.g., whether certain types of businesses are more difficult to reach or obtain responses from, or show different responses to the incentives.

Furthermore, additional testing serves as a trial run for many of the procedures expected to be used in a full survey, allowing a chance to verify that the procedures work effectively and to modify them as needed.

• An online survey will be utilized.

• A brochure will be provided to further explain the survey. Respondents will be asked about the impact of the brochure in encouraging responses during the debriefings.

• Different contact strategies will be used and examined to understand which may work best for microbusinesses.

B.1. Respondent Universe and Sample Design

Respondent Universe

The population for MIST is businesses operating in the U.S. in 2012 delimited by four criteria:

• Business size (1 to 9 employees) as defined by number of employees,¹

• Revenues and expenses,

• Business structure as defined by legal form of organization (C corporations, S corporations, partnerships, and sole proprietorships), and

• Industry (16 NAICS codes defined primarily at the three- or four-digit level).

These criteria are defined in greater detail below.

Business size (as determined by number of employees)

The population for the additional testing is limited to businesses with between 1 and 9 employees. (While microbusinesses include only businesses with fewer than five employees businesses with between 5 and 9 employees will be canvassed to facilitate comparisons with BRDIS, which collects data on businesses with five or more employees. When testing the Microbusiness questionnaire, some questions, in particular the questions on innovation and R&D, needed adjustment for the microbusiness population when utilizing the standard BRDIS methodology. Therefore comparisons will be made using questions that are the same across the two survey populations to test data comparability. In addition, we will continue to examine the validity of the questions that differ slightly between the two questionnaires.) The number of employees at a business will be determined by using businesses’ quarterly filing of Form 941, which reports the number of employees subject to federal income tax withholding. The first quarter tax year 2013 Form 941 will be matched to the income tax return of tax year 2012 to determine the number of employees.

Revenues and Expenses

In 2009 there were almost 35 million businesses regardless of the type of business, industry or number of employees.² Microbusinesses are a large portion of that number. To eliminate businesses that may, in many cases, be considered diversions or hobbies, only businesses with revenues of at least $10,000 and expenditures of at least $5,000 (as reported on the corresponding tax return) will be included in the frame. A study conducted for the Office of Tax Analysis suggested that these are sufficient baselines for a business and that it “eliminates some entities like pure labor suppliers, misclassified employees, pure conduits, and independent contractors”³. In the pretest this requirement was applied only to sole proprietorships. Measures of income and expenses are based on the businesses’ annual tax filings using the most recent data available (Tax Year 2012).

Business structure

The MIST population will include four types of business organizations defined by their income tax return: C-Corporations (Form 1120), S-Corporations (Form 1120S), partnerships (Form 1065), and sole proprietorships (Form 1040 Schedule C).⁴ To focus our testing on microbusinesses that are most likely conducting R&D and other innovation related activities the following form types have been eliminated: property and casualty insurance companies (Form 1120-PC), regulated investment companies (Form 1120-RIC), real estate investment trusts (Form 1120-REIT), and cooperative associations (Form 1120-C). Individuals paying a self-employment tax (Form 1040 Schedule SE) might be comparable to sole proprietorships, but no IRS data are available on the number of employees and industry (two key criteria for identifying eligible businesses) and therefore those individuals who do not also file a Schedule C have been eliminated from the sample frame.

Table B.1 displays the frequency of microbusinesses in 2011 based on IRS tax data, showing the number of employees by IRS form type used. This table includes only businesses with 1 to 4 employees with $10,000 or more in revenues and $5,000 or more in expenditures.

Table B.1. Number of Microbusinesses* by Number of Employees and IRS Form Type, for Tax Year 2011

Number of employees	IRS form type
	1065	1120	1120S	Sch. C	Total	Percent
1	35,608	78,326	318,697	111,196	543,827	38.0%
2	32,663	68,694	213,773	79,899	395,029	27.6%
3	26,546	50,623	143,184	55,809	276,162	19.3%
4	21,264	41,027	114,980	38,483	215,754	15.1%
Total	116,081	238,670	790,634	285,387	1,430,772	100.0%
Percent	8.1%	16.7%	55.3%	19.9%	100.0%

* Includes only microbusinesses with more than $10,000 in revenues and $5,000 or more in expenses.

Industry

NSF’s primary interest in MIST is measuring participation in innovation and R&D. Industry in the IRS Compliance Data Warehouse (CDW) data is identified by the Primary Business Activity (PBA) code that appears on the income tax return. The PBA is very similar to the North American Industry Classification System (NAICS). Using data from BRDIS, NSF selected industries most likely to participate in R&D, as shown in Table B.2. Table B.2 also represents the sampling frame used for the additional testing. The data presented in Table B.2 is for tax year 2011. It is anticipated that tax year 2012 will be used for the microbusiness sample.

Table B.2. Microbusiness* Sample Frame by Principal Activity Code by Number of Employees for Tax Year 2011

		Total
2012 PBA code	2012 Principal Business Activity description	1 - 4 employees	5 - 9 employees
3254	Pharmaceutical and Medicine Manufacturing	233	158
3255	Paint, Coating, and Adhesive Manufacturing	278	244
3259	Other Chemical Product and Preparation Manufacturing	549	362
332	Fabricated Metal Product Manufacturing	10,826	8,383
3336	Engine, Turbine, and Power Transmission Equipment Manufacturing	138	83
3344	Semiconductor and Other Electronic Component Manufacturing	530	356
3345	Navigational, Measuring, Electromedical, and Control Instruments Manufacturing	314	205
335	Electrical Equipment, Appliance, and Component Manufacturing	1,648	1,112
3391	Medical Equipment and Supplies Manufacturing	1,653	769
5112	Software Publishers	856	332
519	Other Information Services	5,068	1,254
5413	Architectural, Engineering, and Related Services	31,789	11,125
5415	Computer Systems Design and Related Services	42,895	8,614
5416	Management, Scientific, and Technical Consulting Services	32,156	5,289
5417	Research and Development	2,328	879
5419	Other Professional, Scientific, and Technical Services	74,835	22,275
	Total	206,096	61,440

* Includes only microbusinesses with more than $10,000 in revenues and $5,000 or more in expenses.

Sample Design

Sample size

For additional testing including methodological testing, a sample of 4,000 will be selected to meet reliability targets described in section B.2, while also retaining supplemental cases that could be released in waves if necessary. Microbusinesses have a substantial rate of attrition: estimates from the pretest phase were in the 20 – 30% range annually. Data from the IRS CDW will be the source of our sample. CDW data have the advantage of being comprehensive (i.e., all businesses with taxable earnings are required to file), frequently updated (with both annual returns and quarterly returns to cover employee withholding), and providing key data on company characteristics (e.g., revenues, primary industry, and number of employees).

Stratification

The sample will be stratified by three variables: industry, company organization structure/IRS form type, and number of employees. As part of this phase, each of these variables will be used to determine eligibility for the sample. Subgroups within these variables may be important for input to any future full survey.

• Industry. Table B.2 presents the composition of the sample into 16 categories based on the Principal Business Activity (PBA). Close to 90 percent of the businesses fall within four of the selected PBA categories, all of which provide services (rather than manufacturing), so stratification will be used to obtain a sufficient mixture of business types.

• Business structure as defined by legal form of the organization. Businesses can be easily distinguished based on which tax forms they submit (forms 1120, 1120S, 1065, and Schedule C, as described above), and their organizational structures may affect how they manage employees and conduct research. A far higher proportion of microbusinesses are sole proprietorships (filing Schedule C). Therefore, stratification will be employed so that there is a sufficient amount of each type of business structure.

• Employment size groups. The number of employees will be divided into five categories: (1) 1 employee, (2) 2 employees, (3) 3 employees, (4) 4 employees, and (5) 5-9 employees. Because the smallest businesses outnumber the larger ones, stratification will be used to ensure sufficient numbers of larger businesses are selected in the sample.

The study has competing priorities which impact the sample design. To develop efficient overall estimates of prevalence rates, for example, it is best to minimize the variation in weights. To compare different groups (e.g., compare C-corporations with sole proprietorships), it is best to sample roughly equal numbers (unweighted) from each group. Given this a testing phase only, there is a greater interest in comparing groups (see table B.8 for examples of comparisons to determine what types of distinctions are most important) than in developing efficient overall estimates. Therefore, the sample design will equalize the number of businesses across groups.

• Equal numbers of businesses using forms 1065, 1120, 1120S, and Schedule C will be sought.

• Microbusinesses will be divided into five size categories (1 employee, 2 employees, 3 employees, 4 employees, and 5-9 employees), with equal sample sizes of 850 for the first four categories, and 600 for the last group. This is roughly equivalent to sampling the businesses with between 1 and 4 employees with probability proportional to the square root of size (PPS), which is a useful approach when one wants to produce both counts or percentages and means or totals. Sampling businesses across the first four size classes with equal probability would result in 38 percent of the sampled microbusinesses having only a single employee (a sample size of 1,520, with an estimated 1,216 respondents—80% response), and in 15.1 percent of the sampled microbusinesses having 4 employees (a sample size of 604, with an estimated 483 respondents). Such an allocation would not be efficient for making comparisons by size class. (See Table B.3)

Table B.3. Number of Microbusinesses by Number of Employees for Tax Year 2011

Number of employees	Number of businesses	Percent	Number of employees	Percent
1	543,827	38.0%	543,827	18.0%
2	395,029	27.6%	790,058	26.1%
3	276,162	19.3%	828,486	27.4%
4	215,754	15.1%	863,016	28.5%
Total	1,430,772	100.0%	3,025,387	100.0%

NOTE: This table shows only the distribution of microbusinesses with between 1 and 4 employees. Businesses with between 5 and 9 employees will be sampled separately, without regard to the number of businesses and number of employees.

• R&D and innovation are likely to manifest themselves differently in the service industries than in manufacturing, so an ideal distribution from a theoretical viewpoint might be to split the sample evenly between the two groups. Service industries comprise roughly nine-tenths of the firms in the 16 categories with between 1 and 4 employees, while manufacturing industries make up only 8 percent.⁵ A proportional representation would give us little information about the manufacturing industries, while an equal division between services and manufacturing would focus substantial resources on only a small segment of the population; it also would result in large variations in the weights, affecting the stability of the estimates. Instead, approximately one-third of the sample (1,300 of 4,000) will come from the manufacturing industries (split into two segments—code 332 and all others—because code 332 would otherwise tend to dominate the other manufacturing categories). The four largest industries (services) will be divided equally (600 each) and the remaining sample will come from the other services category (300). This allocation should provide sufficient sample size for comparisons across different groups. (See Table B.4)

Table B.4. Industry Strata by Percentage of Employees and Proposed Sample Size

2012 PBA Code	2012 Principal Business Activity description	Percentage (for 1 – 4 employees)*	Percentage (for 5 - 9 employees)*	Proposed Sample size
5415	Computer Systems Design and Related Services	20.8%	14.0%	600
5416	Management, Scientific, and Technical Consulting Services	15.6%	8.6%	600
5413	Architectural, Engineering, and Related Services	15.4%	18.1%	600
5419	Other Professional, Scientific, and Technical Services	36.3%	36.3%	600
5112, 5417 & 519	Other services (Software Publishers, Research and Development, and Other Information Services)	4.0%	4.0%	300
332	Fabricated Metal Product Manufacturing	5.3%	13.6%	500
	All others manufacturing categories	2.6%	5.4%	800
	Total	100%	100%	4,000

* Includes only microbusinesses with more than $10,000 in revenues and $5,000 or more in expenses.

Anticipated response rate

One of the main objectives of this testing is to determine how to maximize response rates, with a goal of obtaining at least 80%. Data collection for the pretest was extremely difficult and illustrated the need for additional testing before moving on to the full data collection. As a result, experiments will be conducted to compare different data collection approaches. The experiments detailed below include incentives and a one-page questionnaire as alternative approaches to increase response rates.

B.2. Statistical Methodology

Statistical Methodology for Stratification and Sample Selection and Expected Levels of Precision

The survey strata are described in section B. 1. Based on a desired 80% response rate, the 3,200 anticipated responses are presented in Table B.5 by key subgroups and anticipated standard errors, based on a standard error which assumes a design effect of 2 due to large variations in the final weights.

Table B.5. Expected number of completed questionnaires and corresponding standard errors for MIST, by stratum (80% desired response rate)

Survey strata	Expected number of respondents	Standard error of an estimated proportion equal to:
		P=0.10	P=0.25	P=0.50
Total sample	3,200	0.008	0.011	0.013
IRS form type
1065 (partnerships)	800	0.015	0.022	0.025
1120 (C-corporations)	800	0.015	0.022	0.025
1120S (S-corporations)	800	0.015	0.022	0.025
Sch. C. (sole proprietorships)	800	0.015	0.022	0.025
Number of employees
1	680	0.016	0.023	0.027
2	680	0.016	0.023	0.027
3	680	0.016	0.023	0.027
4	680	0.016	0.023	0.027
5-9	480	0.019	0.028	0.032
Industry
5419 ­ Other Professional, Scientific, and Technical Services	480	0.019	0.028	0.032
5415 ­ Computer Systems Design and Related Services	480	0.019	0.028	0.032
5416 ­ Management, Scientific, and Technical Consulting Services	480	0.019	0.028	0.032
5413 ­ Architectural, Engineering, and Related Services	480	0.019	0.028	0.032
5112, 5417 & 519 - Other services (Software Publishers, Research and Development, and Other Information Services)	240	0.027	0.040	0.046
332 – Fabricated Metal Product Manufacturing	400	0.021	0.031	0.035
Other (manufacturing)	640	0.017	0.024	0.028

B.3. Methods for Maximizing the Response Rate

Our experience with the cognitive interviews and pretest indicate that obtaining a high response rate will be very difficult, so one of the primary objectives of this additional testing is to test which strategies might be effective in increasing response rates. An experimental design will be used to examine specific approaches to data collection. See section B.4 for more description. Other data collection approaches will be evaluated less formally, without an experimental design; these are discussed here.

Considerable work has already been performed to develop the questionnaire, including cognitive interviews, a pretest, and debriefings of selected pretest respondents. Good questionnaire design, while partly aimed at data quality, can affect the response rate by helping to ensure that the questions are salient and understandable to the population being measured.

Data Collection Modes

The data collection implementation plan draws on the social exchange theory concept that people will be more likely to respond to a survey request if they trust that expected rewards will outweigh the perceived costs associated with responding (Dillman, Smyth, and Christian 2009). This design examines specific features to reduce the perceived costs of responding to a survey, increase the benefits of responding, and establishing trust: mode, incentives and questionnaire length. It is designed on current mixed mode literature that shows that mail recruitment for web surveys can be quite effective (Messer & Dillman, 2012; Millar & Dillman 2011; Smyth, Dillman, Christian, & O’Neill 2010) and even more effective than email contacts (Dillman, Smyth, & Christian 2009; Millar & Dillman 2011). In addition, it takes advantage of the social exchange principles described in Dillman et al. (2009).

The research design fully crosses incentive, questionnaire length, and mode treatments. The design randomly assigns sample members (n=4,000) to one of three incentive treatments ($0 versus $2 versus $5), one of two questionnaire lengths (short versus long), and one of two modes (mail versus web).

The initial contact for all sampled microbusinesses will be by mail with one-half of the sample receiving a one-page (two sided) shorter questionnaire (Attachment B) that screens respondents for survey eligibility. The brevity of the shorter questionnaire is expected to aid in increasing response rates (for the overall data collection). The other half of the sample will be further broken down and equally divided with each group receiving either a paper version of the questionnaire or a postal mail letter containing the URL, instructions for using the web version, and access code for the web version of the questionnaire. All respondents will receive in the initial mailing a brochure (Attachment C) explaining the purpose of the survey.

Two weeks after the initial mailing, respondents who received the one-page questionnaire will then receive either the paper questionnaire or a postal mail letter containing the URL, instructions for using the web version, and access code for the web version of the questionnaire. Respondents who returned the short questionnaire and are declared ineligible (based on their responses to questions 1 and 2) will receive no further mailings. Respondents who started with the one-page questionnaire would then receive one more mailing of the same treatment at a second two week interval. Two week intervals were chosen because they are short enough so that the businesses do not need to be reintroduced to the survey, but long enough so they will not feel they are being contacted excessively.

The other group will receive two more similar (i.e., if they initially received a paper questionnaire they will continue to do so) treatments at two week intervals receiving either a replacement paper questionnaire or a postal mail letter containing the URL and access code for the web version of the questionnaire.

This data collection approach leverages all types of contact information to keep costs low, increase benefits and trust, and decrease costs of responding for respondents. The postal mail contacts, for example, increase the chances of making contact compared to emails, which can be caught in spam filters and are more likely to be deleted without being read. The postal mail contact also enables the delivery of the incentive, which will increase benefits and trust. The email contact, which is timed to arrive at the same time or shortly after the postal mail letter, provides a clickable web link for convenience. This removes one of the barriers of responding to a web survey with only a postal mail request (i.e., the need to have the letter near a computer and transfer the URL from the letter to the computer). In other words, it reduces a cost of responding.

The research design is presented in table B.6.

Table B.6. Data Collection Design

Non-response Follow-up

A key to obtaining high response rates is follow-up of non-respondents. A comprehensive non-response analysis will be conducted based on data resource allocation for data collection and may include available paradata and, when appropriate, IRS data.

Research suggests that, in the initial survey contacts, it is best to offer a main or primary mode of data collection (de Leeuw 2005). As the survey progresses, however, offering additional modes can be beneficial. Following the first three treatments, a web-plus-mail design (e.g., Messer and Dillman 2011) will be employed for the sample members originally assigned to the web versions of MIST. In the web-plus-mail option, non-respondents who were originally assigned to the web mode will be offered a mail version of MIST in addition to the web URL and access code so that those who are unable or unwilling to respond by web can respond by paper.

Conversely, non-respondents who returned the one-page questionnaire but did not complete the full length questionnaire will receive email reminders with the web URL and access code to complete the questionnaire online. Telephone follow-up reminders to the non-respondents for whom telephone numbers are available will be used two weeks following the last treatment for all non-respondents. Telephone calls will be staggered. If contact is made over the telephone, but the questionnaire is not completed, additional calls will be made at specified intervals.

Regardless of whether or not the respondent receives the full questionnaire or the one-page questionnaire, all follow up procedures will be tracked and analyzed. The mode of follow-up contact will depend on the timing and on what contact information is available. If a business has not responded after the initial mailings demonstrated in Table B.6, the next contact will be by telephone (assuming telephone numbers are available, and using mail otherwise) to confirm that the mailings arrived and that the contact information is correct. If available, email (which facilitates sending a web link to the survey) will be the first choice, followed by telephone (if no email address is available or email contacts are unsuccessful) and, if no other data are available, mail. The choice of mode for making contacts will also depend on our past experience with a business—e.g., when the business indicates a preferred mode of contact. If a business partially completes a questionnaire without submitting it, the business will be contacted after two weeks to remind them of the survey and determine if there were any problems that prevented completion of the questionnaire.

For businesses that appear unwilling to complete the web survey, the option of responding by fax or mail will be offered. The web survey will also include the option of printing and mailing or faxing a survey. In the latter situation, the business would print out a blank generic questionnaire lacking the business ID; however, the name of the business should be available based on contact information in the questionnaire and the return address on the envelope. Figure B.1 presents non- response follow up procedures.

Figure B.1 Non-response Follow up Procedures

Design of Sampling Frame

During this testing phase, MIST is targeting industries that are most likely to be involved in R&D. The targeting is for two reasons. First, based on findings from BRDIS, involvement in R&D is very rare, and the precision of the R&D data will depend on the number of respondents that are actually involved in R&D. Second, we expect businesses will be more likely to participate in MIST if they perceive the survey as relevant, addressing concerns that are of interest to them. Selecting industries most likely to be involved in R&D, should yield a selection of businesses that are more likely to respond.

Tracing

The IRS-provided sample data will be incomplete (lacking telephone numbers and email addresses) and it may be outdated, given that businesses may move, change names, be purchased by other businesses, or close. Obtaining accurate and complete contact data is crucial both for obtaining a high response rate and for calculating the response rate; one must know whether a business has closed in order to know whether to count it in the response rate. In the pretest, due to IRS’ restrictions on use of the data, tracing efforts were limited to simple web searches, such as 411.com. Based on data collected during the pretest, it is estimated that at least 15 percent of the sample had no identifiable telephone number, an incorrect telephone number (i.e., a nonworking number, or a number leading to a different business or individual), or an incorrect address resulting in a postmaster return, and another nine percent of the businesses had closed.

For this round of testing an outside tracing service data will be used: (1) to obtain telephone numbers for the sampled businesses (since the IRS data do not include telephone numbers), (2) to obtain email addresses for the owners (if available), (3) to make use of National Change of Address (NCOA) and related services to identify businesses that have moved or closed (and to obtain updated contact information on businesses that have moved), and (4) to provide data on limited types of financial transactions to verify that the businesses remain active ( although this does not necessarily mean that the business is no longer in operation, but it will provide some additional context). This should result in improved contact information and fewer attempts to contact closed businesses, which should produce a higher response rate with reduced data collection costs.

To supplement the data from the outside tracing service data, and only when necessary, additional tracing such as using web searches to locate the businesses will be performed. Ideally, only a small number of businesses will require extra tracing, allowing more extensive search processes than were available for the pretest. The frequency of additional tracing and the usefulness of alternative search sources will be examined as part of the testing.

Brochure

A brochure that describes the survey, explains why it is important and how the data will be used, lists endorsing organizations, and answers frequently asked questions will be included in the initial mailing introducing the survey to the selected businesses. The brochure is included in Attachment C.

Use of NSF Name

The NSF name was helpful in recruiting cognitive interview participants and during the debriefing interviews some respondents stated they completed the survey because they were familiar with NSF. Therefore, the questionnaire and recruitment materials were designed to give prominence to the name and to describe NSF.

Attention to Microbusinesses

Some cognitive interview respondents were impressed that microbusinesses were receiving attention from the U.S. government. They would like their importance to be recognized and to know that the federal government is working to help them. Therefore, NSF’s interest in microbusinesses will be emphasized in both the introductory letter and survey introduction to encourage participation by microbusinesses.

B.4. Tests of Procedures and Methods

Additional testing of sampling and data collection approaches is needed in preparation of the full survey.

Sampling Approaches

To inform final decisions about the sample design for a potential full survey, data from this testing phase will be used to derive estimates of the expected levels of precision under alternative stratification schemes and for selected subgroups of interest. The statistics in this analysis include prevalence estimates (e.g., the proportion of microbusinesses that conduct R&D or apply innovation), estimates of means (e.g., mean R&D expenditures or mean number of employees engaged in R&D), and estimates of totals (e.g., total R&D expenditures or number of scientists engaged in R&D).

Experiments

Two experiments will be conducted to assess the efficacy and feasibility of using the one-page questionnaire and of using incentives for the full-scale survey. (See Attachment B for the one-page questionnaire.) After sampling and database cleaning to eliminate closed businesses, businesses will be randomly assigned to receive a one-page questionnaire or an immediate invitation to participate in the full survey, and to receive one of three incentive levels: no incentive, $2 (prepaid), or $5 (prepaid). The expected distribution of options, by contact approach and incentive level, is displayed in Table B.7.⁶

Table B.7. Expected distribution of Sample, by contact approach and incentive level

Contact approach	Incentive levels
	$0	$2 (prepaid)	$5 (prepaid)
One-page shorter questionnaire	667	667	666
Invitation to full survey	667	667	666

The above sampling distribution provides 2,000 cases for the one-page questionnaire or the invitation to the full survey, and roughly 1,234 cases for each incentive level. Based on the assumption of a design effect of 2, sample sizes of 2,000 and 1,234, combined with an 80 percent response rate (i.e., resulting in 1,600 responses and 987 responses, respectively), should produce standard errors no larger than 1.8 and 2.3 percentage points, respectively.⁷ A comparison of two subgroups each having 667 cases (i.e., resulting in 534 responses) should produce standard errors no larger than 3.1 percentage points, assuming that the two estimates are independent.

One-page (Two sided) Questionnaire

Making questionnaires short is one way to reduce the costs of participation because shorter questionnaires reduce the time respondents must commit to filling out a survey. Much evidence suggests that shorter questionnaires obtain higher response rates compared to longer questionnaires (Sahlqvist, et al. 2011; Vicente and Reis 2010; Edwards, et al. 2002; Bogen 1996; Dillman, Sinclair, and Clark 1993; Yammarino, Skinner, and Childers 1991; Heberlein and Baumgartner 1978).

The one-page questionnaire (request sent via USPS mail) contains two questions measuring eligibility for the full survey, a question asking about reasons for owning the business, eight questions on R&D, and one question asking for contact information. All of the questions on the one-page questionnaire are pulled from the full questionnaire. The one-page shorter questionnaire offers the following potential advantages:

• To the extent that survey responses depend on questionnaire length and perceived burden, many businesses might respond immediately to a one-page questionnaire while a longer questionnaire might be put aside and forgotten. The initial response to the one-page questionnaire will be compared with the initial response to the web survey to test the impact of this strategy.⁸

• The one-page questionnaire provides an inexpensive way of collecting email addresses (compared with telephoning the businesses), which will be facilitated to send links to the web survey when appropriate.

• The one-page questionnaire may immediately eliminate many ineligible businesses, lessening the effort required to contact and possibly trace ineligible businesses, and may allow data collection resources to be more focused on respondents who could complete the full questionnaire.

• The one-page questionnaire collects some of the key data desired from the survey (those on R&D) so that, even if the full survey is not completed, much of the data that are desired will already be collected.

Data from the one-page questionnaire would be prefilled into the web questionnaire so that respondents would not need to answer these questions a second time.

By contrast, those sampled businesses that do not get the shorter questionnaire will instead receive a request to complete the full survey on the web, with instructions on how to access and complete the web survey. The one-page questionnaire will only be used in the initial mailing, since it is intended to quickly obtain contact data and weed out ineligible businesses. All follow-up efforts will be directed at getting businesses to complete the full survey, rather than potentially creating two follow-up efforts, first to get the one-page questionnaire and then to get the full survey.

Incentives

The use of incentives has often been shown to encourage people to complete surveys, though sometimes only to a small degree. Through an experimental design, three research questions will be answered:

• How much do response rates increase (if at all) when microbusinesses are offered financial incentives?

• Whether or not response rates increase, could financial incentives be cost effective by reducing the number of follow-up attempts required?

• Does data quality (e.g., the amount of missing data or the number of edit flags generated) vary based on the incentive received?

Logistically, the $2 or $5 prepaid incentives will be included in the first mailing (either the one-page questionnaire or the invitation to the web survey or paper questionnaire); for small incentives, the actual receipt of the incentive appears more effective than the promise of a future incentive. The two levels of prepaid incentives are being offered to test which would be the most effective in maximizing response rates. Experience suggests that the initial incentive will continue to create positive attitudes toward the survey, rather than creating an expectation of an incentive at each stage.

Analysis

As a rule, one factor will be examined at a time. (Table B.8. presents the summary table shells.) For example, when looking at the impact of incentives, the response rates and number of contacts for the three incentive groups ($0, $2, and $5), without regard to the other groupings (e.g., sending a one-page questionnaire versus starting with the full questionnaire), will be compared. This will provide an overall measure of the effectiveness of a particular strategy across multiple data collection approaches and also provides the maximum number of cases for making comparisons. The stratification variables are included primarily to attain an adequate number of firms involved in R&D and to provide a trial run of the intended full survey methodology, but differences across the stratification categories will be examined.

Fifty debriefing interviews will be conducted with respondents and non-respondents using Webex. The purpose of these debriefings is to understand general reactions to the survey, evaluate content of the questionnaire, such as learning whether and why businesses made errors in some of their responses (e. g., based on question misinterpretation or lack of knowledge), effectiveness of the contact strategies, what motivated businesses to respond and for the non-respondents why they did not respond. A draft of the debriefing protocol is provided in Attachment E.

Table B.8. Illustrative summary table shells

Comparison of incentives

Data collection result	No incentive (N=1,334)	$2 prepaid (N=1,334)	$5 prepaid (N=1,332)
Mean response rate
Mean number of contacts
Average time until completion of full survey

Comparison of incentives by organization type

Data collection result	No incentive (N=1,334)	$2 prepaid (N=1,334)	$5 prepaid (N=1,332)
Mean response rate
Sole proprietors
C-Corporations
S-Corporations
Partnerships
Mean number of contacts
Sole proprietors
C-Corporations
S-Corporations
Partnerships
Average time (in days) until completion of full survey
Sole proprietors
C-Corporations
S-Corporations
Partnerships

Note: This table is exploratory only, based on the hypothesis that sole proprietors may be more like households than other small businesses, and thus may respond to incentives differently. Given the smaller Ns, differences would need to be relatively large to be statistically significant.

Potential value of one-page questionnaire

Data collection results	One-page questionnaire (N=2,000)	Full questionnaire (N=2,000)
Response rate after initial contact
Number of email addresses obtained		NA
Number of corrected telephone numbers obtained		NA
Number of corrected addresses obtained		NA
Number of ineligible businesses identified		NA
Response rate after first offering of full survey
Final response rate
Average number of contacts
Average time until completion of survey

Potential value of stratification by organization type

Key variables	Sole proprietors (N=1,000)	C-Corporations (N=1,000)	S-Corporations (N=1,000)	Partnerships (N=1,000)
Participate in R&D (Q25)
Engage in innovation (based on Q17-Q20)
Mean revenue (Q10)
First company started by owner (Q38)
Mean owner hours per week (Q37)
Reasons for owning own company (Q3)
Hired new employee (Q6)
Highest level of education (Q40)

Note: To the extent that differences are small between organization types, there may be less need for stratification by organization type in the full survey.

Potential value of stratification by number of employees

Key variables	1 employee (N=850)	2 employees (N=850)	3 employees (N=850)	4 employees (N=850)	5-9 employees (N=600)
Participate in R&D (Q25)
Engage in innovation (based on Q17-Q20)
Mean revenue (Q10)
First company started by owner (Q38)
Mean owner hours per week (Q37)
Reasons for owning own company (Q3)
Hired new employee (Q6)
Highest level of education (Q40)

Note: To the extent that differences are small based on the number of employees, there may be less need for stratification by this variable in the full survey.

B.5. Reviewing Statisticians

Dr. Adam Chu

Associate Director

Westat

(301) 251-4326

Dr. Hyunshik Lee

Senior Statistician

Westat

(301) 610-5112

Jock Black

Mathematical Statistician

National Science Foundation

(703) 292-7802

Audrey Kindlon

Survey Statistician

National Science Foundation

(703) 292-2332

John Jankowski

RDS Program Director

National Science Foundation

(703) 292-7781

Rebecca Morrison

Survey Statistician

National Science Foundation

(703) 292-7794

Jeri Mulrow

Deputy Division Director

National Science Foundation

(703)-292-4784

Dr. Kristen M. Olson

Associate Professor of Sociology and of Survey Research and Methodology

University of Nebraska - Lincoln

(402) 472-6057

Dr. Jolene D. Smyth

Associate Professor of Sociology, Associate Professor of Survey Research and Methodology, and Director of the Bureau of Sociological Research

University of Nebraska - Lincoln

(402) 472-7774

References

Armstrong, J. S. (1975). Monetary incentives in mail surveys. Public Opinion Quarterly, 39, 1,

111-116.

Asch, D. A., Christakis, N. A., & Ubel, P. A. (1998). Conducting physician mail surveys on a

limited budget: A randomized trial comparing $2 bill versus $5 bill incentives. Medical Care, 36,

1, 95-99.

Bogen, K. (1996). The effect of questionnaire length on response rates–A review of the

literature. Proceedings of the Survey Research Methods Section of the American Statistical

Association, 1020-1025.

Church, A. H. (1993). Estimating the effect of incentives on mail survey response rates: A metaanalysis.

Public Opinion Quarterly, 57, 1, 62-79.

Cook, S., LeBaron, P., Flicker, L., & Flanigan, T. S. (2009). Applying incentives to

establishment surveys: A review of the literature. Paper presented at the 2009 meeting of the

American Association of Public Opinion Research (AAPOR). Hollywood, FL.

de Leeuw, E. “To Mix or Not to Mix Data Collection Modes in Surveys.” Journal of Official Statistics, Volume 21, No. 2, pp. 233–255. (2005).

Dillman, D. A., Smyth, J.D., and Christian, L.M. (2009) Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method. New York: Wiley.

Dillman, D. A., Sinclair, M. D., & Clark, J. R. (1993). Effects of questionnaire length,

respondent-friendly design, and a difficult question on response rates for occupant-addressed

census mail surveys. Public Opinion Quarterly, 57, 289-304.

Edwards, P., Roberts, I., Clarke, M., DiGuiseppi, C., Pratap, S., Wentz, R., & Kwan, I. (2002).

Increasing response rates to postal questionnaires: Systematic review. British Journal of

Medicine, 324.

Fox, R. J., Crask, M. R., & Kim, J. (1988). Mail survey response rate: A meta-analysis of

selected techniques for inducing response. Public Opinion Quarterly, 52, 467-491.

Göritz, A. S (2006). Incentives in Web studies: Methodological issues and a review.

International Journal of Internet Science, 1, 58-70.

Heberlein, T. A., & Baumgartner, R. (1978). Factors affecting response rates to mailed

questionnaires: A quantitative analysis of the published literature. American Sociological

Review, 43, 447-462.

Knittel, M., Nelson, S., DeBacker, J., Kitchen, J., Pearce, J., and Prisinzano, R. “Methodology to Identify Small Businesses and Their Owners.” Office of Tax Analysis, Department of the Treasury. Technical Paper 4. August 2011.

Luo, A., and White, Jr., G. D. “Exploring a New Establishment Survey Incentive to Improve Response Rates.” In: Proceedings of the Survey Research Methods Section, American Statistical Association, pp. 3915–3918 (2005).

Messer, B. L., & Dillman, D. A. (2011). Surveying the general public over the internet using

address-based sampling and mail contact procedures. Public Opinion Quarterly, 75(3), 429–457.

12

Millar, M. M., & Dillman, D. A. (2011). Improving response to web and mixed-mode surveys.

Public Opinion Quarterly, 75(2), 249–269.

Sahlqvist, S., Song, Y., Bull, F., Adams, E., Preston, J., Ogilvie, D., & iConnect Consortium

(2011). Effect of questionnaire length, personalization, and reminder type on response rate to a

complex postal survey: Randomized controlled trial. BMC Medical Research Methodology,

11(62).

Shettle, C., & Mooney, G. (1999). Monetary incentives in US government surveys. Journal of

Official Statistics, 15(2), 231-250.

Singer, E. (2002). The use of incentives to reduce nonresponse in household surveys. In Survey

nonresponse, R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), 163-178. New

York, NY: Wiley.

Singer, E., & Ye, C. (2013). The use and effects of incentives in surveys. The ANNALS of the

American Academy of Political and Social Science, 645, 112-141.

Smyth, J. D., Dillman, D. A., Christian, L. M., & O’Neill, A. C. (2010). Using the Internet to

survey small towns and communities: Limitations and possibilities in the early 21st Century.

American Behavioral Scientist, 53, 1423–1448.

Trivedi, S. (2011, May 24). Treasury Seeking to Better Identify Small Business Owners. Tax Notes Today Retrieved August 20, 2013, from http://services.taxanalysts.com/taxbase/tnt3.nsf/

Trussell, N. and P. J. Lavrakas. (2004). The Influence of Incremental Increases in Token Cash

Incentives on Mail Survey Response: Is There an Optimal Amount?” Public Opinion Quarterly.

68(3):349-367.

Vicente, P., & Reis, E. (2010). Using questionnaire design to fight nonresponse bias in web

surveys. Social Science Computer Review, 28, 251-267.

Yammarino, F. J., Skinner, S. J., & Childers, T. L. (1991). Understanding mail survey response

behavior: A meta-analysis. Public Opinion Quarterly, 55, 613-639.

Attachments — Data Collection Instruments

1. MIST Questionnaire

2. MIST One-page questionnaire

3. Brochure

4. Correspondence

• Initial letter, One-page questionnaire (web survey follow up) — no incentive

• Initial letter, One-page questionnaire (web survey follow up) — $2 incentive

• Initial letter, One-page questionnaire (web survey follow up) — $5 incentive

• Initial letter, One-page questionnaire (paper questionnaire follow up) — no incentive

• Initial letter, One-page questionnaire (paper questionnaire follow up) — $2 incentive

• Initial letter, One-page questionnaire (paper questionnaire follow up) — $5 incentive

• Initial letter, Invitation to web survey — no incentive

• Initial letter, Invitation to web survey — $2 incentive

• Initial letter, Invitation to web survey — $5 incentive

• Initial letter, Invitation to paper questionnaire — no incentive

• Initial letter, Invitation to paper questionnaire — $2 incentive

• Initial letter, Invitation to paper questionnaire — $5 incentive

• Follow-up letter for web survey after receiving one-page questionnaire

• Follow-up letter for paper questionnaire after receiving one-page questionnaire

• Follow-up letter for web survey after not receiving one-page questionnaire

• Follow-up letter for paper questionnaire after not receiving one-page questionnaire

• Thank you email after completion of survey

5. Debriefing Interview Protocol

1 Microbusinesses are defined as businesses with fewer than five employees. Business with 5 to 9 employees are included as part of this testing to facilitate potential comparisons to the BRDIS data.

2 Businesses can be counted in a variety of ways. U.S. Census Bureau statistics show 5.7 million firms (or 7.4 million establishments) in 2010, with 93 percent of those firms in the four business categories sampled in MIST (see http://www.census.gov/econ/susb/). The Census Bureau statistics are limited to firms with paid employees. Most of the difference between the IRS based statistics and the Census statistics is from the inclusion of all Schedule C returns, with 25.3 million filers for tax year 2009 (by contrast, Census reports that 0.9 million firms were sole proprietorships). Excluding the Schedule C returns, there were 9.7 million returns filed using Forms 1120, 1120S, and 1065. Another source of data is Dun and Bradstreet, which tracks 29.5 million active companies in North America as of July 2013 (see http://www.dnb.com/company/our-data/data-quality-of-data-as-a-service.html), though that statistic includes non-U.S. businesses.

3 Trivedi, Shamik. (2011, May 24). Treasury Seeking to Better Identify Small Business Owners. Tax Notes Today Retrieved August 20, 2013, from http://services.taxanalysts.com/taxbase/tnt3.nsf/

4 The difference between an S- and C-corporations is mostly by how they are taxed. S-corporations elect to pass corporate income, losses, deductions and credit through to their shareholders for federal tax purposes resulting in what is called single taxation. C-corporations are taxed at the corporate and individual level. A partnership requires at least two individuals. The partnership reports income, deductions, gains, or losses, but it does not pay income tax as that is passed to the partners. A sole proprietor is someone who owns an unincorporated business by himself or herself. (Source: www.irs.gov)

5 Part of the reason may be that services are disproportionately represented among the smallest firms. U.S. Census data for 2010 indicate that 43 percent of manufacturing firms had less than 5 employees, and these firms had 2 percent of manufacturing employees; by contrast, 74 percent of firms providing professional, technical, and scientific services had less than 5 employees, and these firms had 11 percent of the employees in that industry. See http://www.census.gov/econ/susb/. Counts of manufacturing firms fail to reflect their economic importance; they comprised 5 percent of all firms but had 18 percent of all receipts (see http://www.sba.gov/advocacy/849/12162#susb).

6 Table B.7 presents the sample distribution. Analytic table shells to examine the survey results are presented later in this document.

7 These estimates assume the worst-case scenario: a proportion equal to 50 percent. Standard errors should be smaller for other proportions.

8 It is important to test the initial response rates to examine this issue. Later response rates will be affected by follow-up activities, and will not provide a clean measure of the impact of the one-page questionnaire.

10