3145-MIST Supporting Statement Part B

Section B. Description of Statistical Methodology

As noted in Part A, the primary purpose of the additional testing is to build on what was learned during the pretest and debriefing interviews. The questionnaire (found in Attachment A) has already been examined in detail through cognitive interviews and a pretest, though its performance will continue to be monitored. With regard to sampling, the design effect will be measured, and the extent to which the stratification approach helps or hinders the quality of the survey estimates will be examined. With regard to data collection, several experiments will be employed to understand what steps may be taken to encourage a high response rate. The additional testing also will examine how the sample design interacts with data collection: e.g., whether certain types of businesses are more difficult to reach or obtain responses from, or show different responses to the incentives.

Furthermore, additional testing serves as a trial run for many of the procedures expected to be used in a full survey, allowing a chance to verify that the procedures work effectively and to modify them as needed.

• It will be the first use of the online survey mode.

• It will include the first use by MIST of a brochure. Respondents will be asked about the impact of the brochure in encouraging responses during the debriefings.

• It will allow different contact strategies to be examined and to understand which may work best for the microbusinesses.

B.1. Respondent Universe and Sample Design

Respondent Universe

The population for MIST is businesses operating in the U.S. in 2012 delimited by four criteria:

• Business size (1 to 9 employees) as defined by number of employees,¹

• Revenues and expenses,

• Business structure as defined by legal form of organization (C corporations, S corporations, partnerships, and sole proprietorships), and

• Industry (16 NAICS codes defined primarily at the three- or four-digit level), and

These criteria are defined in greater detail below.

Business size (as determined by number of employees)

The population for the additional testing is limited to businesses with between 1 and 9 employees. (While microbusinesses include only businesses with fewer than five employees businesses with between 5 and 9 employees will be canvassed to facilitate comparisons with BRDIS, which collects data on businesses with five or more employees. When testing the Microbusiness questionnaire, some questions, in particular the questions on innovation and R&D, needed adjustment for the microbusiness population when utilizing the standard BRDIS methodology. Therefore comparisons will be made using questions that are the same across the two survey populations to test data comparability. In addition, we will continue to examine the validity of the questions that differ slightly between the two questionnaires. Finally it is important to note that no single business will receive both surveys.) The number of employees at a business will be determined by using businesses’ quarterly filing of Form 941, which reports the number of employees subject to federal income tax withholding. The first quarter tax year 2013 Form 941 will be matched to the income tax return of tax year 2012 to determine the number of employees.

Revenues and Expenses

In 2009 there were almost 35 million businesses regardless of the type of business, industry or number of employees.² Microbusinesses are a large portion of that number. To eliminate businesses that may, in many cases, be considered diversions or hobbies, only businesses with revenues of at least $10,000 and expenditures of at least $5,000 (as reported on the corresponding tax return) will be included in the frame. A study conducted for the Office of Tax Analysis suggested that these are sufficient baselines for a business and that it “eliminates some entities like pure labor suppliers, misclassified employees, pure conduits, and independent contractors”³. In the pretest this requirement was applied only to sole proprietorships. Measures of income and expenses are based on the businesses’ annual tax filings using the most recent data available (Tax Year 2012).

Business structure

The MIST population will include four types of business organizations defined by their income tax return: C-Corporations (Form 1120), S-Corporations (Form 1120S), partnerships (Form 1065), and sole proprietorships (Form 1040 Schedule C).⁴ To focus our testing on microbusinesses that are most likely conducting R&D and other innovation related activities the following form types have been eliminated: property and casualty insurance companies (Form 1120-PC), regulated investment companies (Form 1120-RIC), real estate investment trusts (Form 1120-REIT), and cooperative associations (Form 1120-C). Individuals paying a self-employment tax (Form 1040 Schedule SE) might be comparable to sole proprietorships, but no IRS data are available on the number of employees and industry (two key criteria for identifying eligible businesses) and therefore those individuals who do not also file a Schedule C have been eliminated from the sample frame.

Table B.1 displays the frequency of microbusinesses in 2011 based on IRS tax data, showing the number of employees by IRS form type used. This table includes only businesses with 1 to 4 employees with $10,000 or more in revenues and $5,000 or more in expenditures.

Table B.1. Number of Microbusinesses* by number of employees and IRS form type, for Tax Year 2011

Number of employees	IRS form type
	1065	1120	1120S	Sch. C	Total	Percent
1	35,608	78,326	318,697	111,196	543,827	38.0%
2	32,663	68,694	213,773	79,899	395,029	27.6%
3	26,546	50,623	143,184	55,809	276,162	19.3%
4	21,264	41,027	114,980	38,483	215,754	15.1%
Total	116,081	238,670	790,634	285,387	1,430,772	100.0%
Percent	8.1%	16.7%	55.3%	19.9%	100.0%

* Includes only microbusinesses with more than $10,000 in revenues and $5,000 or more in expenses.

Industry

NSF’s primary interest in MIST is measuring participation in innovation and R&D. Industry in the IRS Compliance Data Warehouse (CDW) data is identified by the Primary Business Activity (PBA) code that appears on the income tax return. The PBA is very similar to the North American Industry Classification System (NAICS). Using data from BRDIS, NSF selected industries most likely to participate in R&D, as shown in Table B.2. Table B.2 also represents the sampling frame used for the additional testing. The data presented in Table B.2 is for tax year 2011. It is anticipated that tax year 2012 will be used for the microbusiness sample.

Table B.2. Microbusiness* Sample Frame by Principal Activity Code by Number of Employees for Tax Year 2011

		Total
2012 PBA code	2012 Principal Business Activity description	1 - 4 employees	5 - 9 employees
3254	Pharmaceutical and Medicine Manufacturing	233	158
3255	Paint, Coating, and Adhesive Manufacturing	278	244
3259	Other Chemical Product and Preparation Manufacturing	549	362
332	Fabricated Metal Product Manufacturing	10,826	8,383
3336	Engine, Turbine, and Power Transmission Equipment Manufacturing	138	83
3344	Semiconductor and Other Electronic Component Manufacturing	530	356
3345	Navigational, Measuring, Electromedical, and Control Instruments Manufacturing	314	205
335	Electrical Equipment, Appliance, and Component Manufacturing	1,648	1,112
3391	Medical Equipment and Supplies Manufacturing	1,653	769
5112	Software Publishers	856	332
519	Other Information Services	5,068	1,254
5413	Architectural, Engineering, and Related Services	31,789	11,125
5415	Computer Systems Design and Related Services	42,895	8,614
5416	Management, Scientific, and Technical Consulting Services	32,156	5,289
5417	Research and Development	2,328	879
5419	Other Professional, Scientific, and Technical Services	74,835	22,275
	Total	206,096	61,440

* Includes only microbusinesses with more than $10,000 in revenues and $5,000 or more in expenses.

Sample Design

Sample size

For additional testing including methodological testing, a sample of 4,000 will be selected to meet reliability targets described in section B.2, while also retaining supplemental cases that could be released in waves if necessary. Microbusinesses have a substantial rate of attrition: estimates from the pretest phase were in the 20 – 30% range annually. Data from the IRS CDW will be the source of our sample. CDW data have the advantage of being comprehensive (i.e., all businesses with taxable earnings are required to file), frequently updated (with both annual returns and quarterly returns to cover employee withholding), and providing key data on company characteristics (e.g., revenues, primary industry, and number of employees).

Stratification

The sample will be stratified by three variables: industry, company organization structure/IRS form type, and number of employees. As part of this phase, each of these variables will be used to determine eligibility, but how subgroups within these variables vary may potentially be important for input for any future full survey as well.

• Industry. Table B.2 shows how the sample will be divided into 16 categories based on the Principal Business Activity. Close to 90 percent of the businesses fall within four categories, all of which provide services (rather than manufacturing), so stratification will be used to obtain a sufficient mixture of business types. Businesses will be divided into two groups: manufacturing and those providing services. The latter will be further subdivided because of the large numbers in four of the categories (PBA codes 5415, 5416, 5413, and 5419). PBA code 332 will be its own category as it makes up approximately two-thirds of the manufacturing codes.

• Business structure as defined by legal form of the organization. Businesses can be easily distinguished based on which tax forms they submit (forms 1120, 1120S, 1065, and Schedule C, as described above), and their organizational structures may affect how they manage employees and conduct research. The number of Sole proprietorships (filing Schedule C) is so much larger than the other groups that their sample size needs to be limited or they would overwhelm the other types of firms.

• Employment size groups. The number of employees will be divided into five categories: (1) 1 employee, (2) 2 employees, (3) 3 employees, (4) 4 employees, and (5) 5-9 employees. Because the smallest businesses outnumber the larger ones, stratification will be used to ensure sufficient numbers of larger businesses are selected in the sample.

The study has competing priorities which impact the sample design. To develop efficient overall estimates of prevalence rates, for example, it is best to minimize the variation in weights. To compare different groups (e.g., compare C-corporations with sole proprietorships), it is best to sample roughly equal numbers (unweighted) from each group. Given this a testing phase only, there is a greater interest in comparing groups (to determine what types of distinctions are most important) than in developing efficient overall estimates. Therefore, the sample design will equalize the number of businesses across groups.

• Equal numbers of businesses using forms 1065, 1120, 1120S, and Schedule C will be sought.

• Microbusinesses will be divided into five size categories (1 employee, 2 employees, 3 employees, 4 employees, and 5-9 employees), with equal sample sizes of 850 for the first four categories, and 600 for the last group. This is roughly equivalent to sampling the businesses with between 1 and 4 employees with probability proportional to the square root of size (PPS), which is a useful approach when one wants to produce both counts or percentages and means or totals. Sampling businesses across the first four size classes with equal probability would result in 38 percent of the sampled microbusinesses having only a single employee (a sample size of 1,520, with an estimated 1,216 respondents—80% response), and in 15.1 percent of the sampled microbusinesses having 4 employees (a sample size of 604, with an estimated 483 respondents). Such an allocation would not be efficient for making comparisons by size class. (See Table B.3)

Table B.3. Number of Microbusinesses by Number of Employees for Tax Year 2011

Number of employees	Number of businesses	Percent	Number of employees	Percent
1	543,827	38.0%	543,827	18.0%
2	395,029	27.6%	790,058	26.1%
3	276,162	19.3%	828,486	27.4%
4	215,754	15.1%	863,016	28.5%
Total	1,430,772	100.0%	3,025,387	100.0%

NOTE: This table shows only the distribution of microbusinesses with between 1 and 4 employees. Businesses with between 5 and 9 employees will be sampled separately, without regard to the number of businesses and number of employees.

• R&D and innovation are likely to manifest themselves differently in the service industries than in manufacturing, so an ideal distribution from a theoretical viewpoint might be to split the sample evenly between the two groups. Service industries comprise roughly nine-tenths of the firms in the 16 categories with between 1 and 4 employees, while manufacturing industries make up only 8 percent.⁵ A proportional representation would give us little information about the manufacturing industries, while an equal division between services and manufacturing would focus substantial resources on only a small segment of the population; it also would result in large variations in the weights, affecting the stability of the estimates. Instead, an allocation will be employed. Approximately one-third of the sample (1,300 of 4,000) will come from the manufacturing industries (split into two segments—code 332 and all others—because code 332 would otherwise tend to dominate the other manufacturing categories). The four largest industries (services) will be divided equally (600 each) and the remaining sample will come from the other services category (300). This allocation should provide sufficient sample size for comparisons across different groups. (See Table B.4)

Table B.4. Industry Strata by Percentage of Employees and Proposed Sample Size

2012 PBA Code	2012 Principal Business Activity description	Percentage (for 1 – 4 employees)*	Percentage (for 5 - 9 employees)*	Proposed Sample size
5415	Computer Systems Design and Related Services	20.8%	14.0%	600
5416	Management, Scientific, and Technical Consulting Services	15.6%	8.6%	600
5413	Architectural, Engineering, and Related Services	15.4%	18.1%	600
5419	Other Professional, Scientific, and Technical Services	36.3%	36.3%	600
5112, 5417 & 519	Other services (Software Publishers, Research and Development, and Other Information Services)	4.0%	4.0%	300
332	Fabricated Metal Product Manufacturing	5.3%	13.6%	500
	All others manufacturing categories	2.6%	5.4%	800
	Total	100%	100%	4,000

* Includes only microbusinesses with more than $10,000 in revenues and $5,000 or more in expenses.

Anticipated response rate

The goal of this phase of the testing is to determine how to maximize response rates. Data collection for the pretest was extremely difficult and illustrated the need for additional testing before moving on to the full data collection. As a result, experiments will be conducted to compare different data collection approaches. The experiments detailed below include incentives and a one page screening questionnaire as alternative approaches to increase response rates.

B.2. Statistical Methodology

Statistical Methodology for Stratification and Sample Selection and Expected Levels of Precision

The survey strata are described in section B. 1. Based on a desired 80% response rate, the 3,200 anticipated responses are presented in Table B.5 by key subgroups and anticipated standard errors, based standard error which assumes a design effect of 2 due to large variations in the final weights.

Table B.5. Expected number of completed questionnaires and corresponding standard errors for MIST, by strata (80% desired response rate)

Survey strata	Expected number of respondents	Standard error of an estimated proportion equal to:
		P=0.10	P=0.25	P=0.50
Total sample	3,200	0.008	0.011	0.013
IRS form type
1065 (partnerships)	800	0.015	0.022	0.025
1120 (C-corporations)	800	0.015	0.022	0.025
1120S (S-corporations)	800	0.015	0.022	0.025
Sch. C. (sole proprietorships)	800	0.015	0.022	0.025
Number of employees
1	680	0.016	0.023	0.027
2	680	0.016	0.023	0.027
3	680	0.016	0.023	0.027
4	680	0.016	0.023	0.027
5-9	480	0.019	0.028	0.032
Industry
5419 ­ Other Professional, Scientific, and Technical Services	480	0.019	0.028	0.032
5415 ­ Computer Systems Design and Related Services	480	0.019	0.028	0.032
5416 ­ Management, Scientific, and Technical Consulting Services	480	0.019	0.028	0.032
5413 ­ Architectural, Engineering, and Related Services	480	0.019	0.028	0.032
5112, 5417 & 519 - Other services (Software Publishers, Research and Development, and Other Information Services)	240	0.027	0.040	0.046
332 – Fabricated Metal Product Manufacturing	400	0.021	0.031	0.035
Other (manufacturing)	640	0.017	0.024	0.028

B.3. Methods for Maximizing the Response Rate

Our experience with the cognitive interviews and pretest indicate that obtaining a high response rate will be very difficult, so one of the primary objectives of this additional testing is to test which strategies might be effective in increasing response rates. An experimental design will be used to examine specific approaches to data collection. See section B.4 for more description. Other data collection approaches will be evaluated less formally, without an experimental design; these are discussed here.

Considerable work has already been performed to develop the questionnaire, including cognitive interviews, a pretest, and debriefings of selected pretest respondents. Good questionnaire design, while partly aimed at data quality, can affect the response rate by helping to ensure that the questions are salient and understandable to the population being measured.

Data Collection Modes

In the cognitive interviews, businesses indicated that they preferred to complete surveys over the web, therefore the web will be the primary data collection mode. This testing phase will provide the first full test of a web version of the survey.

Research suggests that, in the initial survey contacts, it is best to offer a main or primary mode of data collection (de Leeuw 2005). As the survey progresses, however, offering additional modes can be beneficial. Based on the research, a single mode will be offered at the start (mail when using the screening questionnaire, and web otherwise) and additional options will be offered later. However, because email addresses will not be available for the sampled businesses, the initial contact for all sampled microbusinesses will be by mail with information provided on how to complete the survey via the internet (Attachment D). Respondents who receive the one page screening questionnaire will receive a paper version of the one page questionnaire (Attachment B). All respondents will receive in the initial mailing a brochure (Attachment C) explaining the purpose of the survey.

The brevity of the one page screening questionnaire is expected to aid in increasing response rates. When mailing an invitation to the full survey, a web address and instructions for using the web version will be provided. For businesses that appear unwilling to complete the web survey, the option of responding by fax or mail will be offered. The web survey will also include the option of printing and mailing or faxing a survey. In the latter situation, the business would print out a blank generic questionnaire lacking the business ID; however, the name of the business should be available based on contact information in the questionnaire and the return address on the envelope.

Design of Sampling Frame

During this testing phase, MIST is targeting industries that are most likely to be involved in R&D. The targeting is for two reasons. First, based on findings from BRDIS, involvement in R&D is very rare, and the precision of the R&D data will depend on the number of respondents that are actually involved in R&D. Second, we expect businesses will be more likely to participate in MIST if they perceive the survey as relevant, addressing concerns that are of interest to them. Selecting industries most likely to be involved in R&D, should yield a selection of businesses that are more likely to respond.

Tracing

The IRS-provided sample data will be incomplete (lacking telephone numbers and email addresses) and it may be outdated, given that businesses may move, change names, be purchased by other businesses, or close. Obtaining accurate and complete contact data is crucial both for obtaining a high response rate and for calculating the response rate; one must know whether a business has closed in order to know whether to count it in the response rate. In the pretest, due to IRS’ restrictions on use of the data, tracing efforts were limited to simple web searches, such as 411.com. Based on data collected during the pretest, it is estimated that at least 15 percent of the sample had no identifiable telephone number, an incorrect telephone number (i.e., a nonworking number, or a number leading to a different business or individual), or an incorrect address resulting in a postmaster return, and another nine percent of the businesses had closed.

For this round of testing an outside tracing service data will be used: (1) to obtain telephone numbers for the sampled businesses (since the IRS data do not include telephone numbers), (2) to obtain email addresses for the owners (if available), (3) to make use of National Change of Address (NCOA) and related services to identify businesses that have moved or closed (and to obtain updated contact information on businesses that have moved), and (4) to provide data on limited types of financial transactions to verify that the businesses remain active ( although this does not necessarily mean that the business is no longer in operation, but it will provide some additional context). This should result in improved contact information and fewer attempts to contact closed businesses, which should produce a higher response rate with reduced data collection costs.

To supplement the data from the outside tracing service data, and only when necessary, additional tracing such as using web searches to locate the businesses will be performed. Ideally, only a small number of businesses will require extra tracing, allowing more extensive search processes than were available for the pretest. The frequency of additional tracing and the usefulness of alternative search sources will be examined as part of the testing.

Brochure

A brochure that describes the survey, explains why it is important and how the data will be used, lists endorsing organizations, and answers frequently asked questions will be included in the initial mailing introducing the survey to the selected businesses. The brochure is included in Attachment C.

Nonresponse Follow-up

A key to obtaining high response rates is follow-up of non respondents. Repeated efforts using multiple modes to contact the businesses will be made. The initial contact will be by mail, a reminder will be sent by mail after two weeks, and follow-up contacts will be made two weeks after the prior contact.⁶ Two weeks intervals of were chosen because they are short enough so that the businesses do not need to be reintroduced to the survey, but long enough so they will not feel they are being contacted excessively. A comprehensive non response analysis will be conducted based on data resource allocation for data collection and may include available paradata and, when appropriate, IRS data.

Regardless of whether or not the respondent receives the full questionnaire or the one page questionnaire, follow up procedures will be tracked and analyzed. The mode of follow-up contact will depend on the timing and on what contact information is available. If a business has not responded after the initial two mailings, the next contact will be by telephone (assuming telephone numbers are available, and using mail otherwise) to confirm that the mailings arrived and that the contact information is correct. If available, email (which facilitates sending a web link to the survey) will be the first choice, followed by telephone (if no email address is available or email contacts are unsuccessful) and, if no other data are available, mail. If there is evidence of incorrect contact information (such as returned mail or reaching nonworking telephone numbers) tracing will be used to correct the information. The choice of mode for making contacts will also depend on our past experience with a business—e.g., when the business indicates a preferred mode of contact. If a business partially completes a questionnaire without submitting it, the business will be contacted after two weeks to remind them of the survey and determine if there were any problems that prevented completion of the questionnaire. These steps are summarized in the figures on the next two pages.

Use of NSF Name

The NSF name was helpful in recruiting cognitive interview participants and during the debriefing interviews some respondents stated they completed the survey because they were familiar with NSF. Therefore, the questionnaire and recruitment materials were designed to give prominence to the name and to describe NSF.

Attention to Microbusinesses

Some cognitive interview respondents were impressed that microbusinesses were receiving attention from the U.S. government. They would like their importance to be recognized and to know that the federal government is working to help them. Therefore, NSF’s interest in microbusinesses will be emphasized in both the introductory letter and survey introduction to encourage participation by microbusinesses.

B.4. Tests of Procedures and Methods

Additional testing of sampling and data collection approaches is needed in preparation of the full survey.

Sampling Approaches

To inform final decisions about the sample design for a potential full survey, data from this testing phase will be used to derive estimates of the expected levels of precision under alternative stratification schemes and for selected subgroups of interest. The statistics in this analysis include prevalence estimates (e.g., the proportion of microbusinesses that conduct R&D or apply innovation), estimates of means (e.g., mean R&D expenditures or mean number of employees engaged in R&D), and estimates of totals (e.g., total R&D expenditures or number of scientists engaged in R&D).

Experiments

Two experiments will be conducted to assess the efficacy and feasibility of using the screening questionnaire and of using incentives for the full-scale survey. (See Attachment B for the one page questionnaire.) After sampling and database cleaning to eliminate closed businesses , businesses will be randomly assigned to receive a one page screening questionnaire or an immediate invitation to participate in the full survey, and to receive one of three incentive levels: no incentive, $2 (prepaid), or $20 (postpaid). The expected distribution of options, by contact approach and incentive level, is displayed in Table B.6.⁷

Table B.6. Expected distribution of Sample, by contact approach and incentive level

Contact approach	Incentive levels
	$0	$2 (prepaid)	$20 (promised)
One page screening questionnaire	667	667	666
Invitation to full survey	667	667	666

The above sampling distribution provides 2,000 cases for the one page screening questionnaire or the invitation to the full survey, and roughly 1,234 cases for each incentive level. Based on the assumption of a design effect of 2, sample sizes of 2,000 and 1,234, combined with an 80 percent response rate (i.e., resulting in 1,600 responses and 987 responses, respectively), should produce standard errors no larger than 1.8 and 2.3 percentage points, respectively.⁸ A comparison of two subgroups each having 667 cases (i.e., resulting in 534 responses) should produce standard errors no larger than 3.1 percentage points, assuming that the two estimates are independent.

One Page Screening Questionnaire

The one page screener (request sent via USPS mail) contains two questions measuring eligibility for the full survey, a question asking about reasons for owning the business, eight questions on R&D, and one question asking for contact information. All of the screening questions are pulled from the full questionnaire. The screening questionnaire offers the following potential advantages:

• To the extent that survey responses depend on questionnaire length and perceived burden, many businesses might respond immediately to the screener while a longer questionnaire might be put aside and forgotten. The initial response to the screener will be compared with the initial response to the web survey to test the impact of this strategy.⁹

• The screener provides an inexpensive way of collecting email addresses (compared with telephoning the businesses), which is the ideal way to send links to the web survey.

• The screener may immediately eliminate many ineligible businesses, lessening the effort required to contact and possibly trace ineligible businesses, and may allow data collection resources to be more focused on respondents who could complete the full questionnaire.

• The screener collects some of the key data desired from the survey (those on R&D) so that, even if the full survey is not completed, much of the data that are desired will already be collected.

Data from the screener would be prefilled into the web questionnaire so that respondents would not need to answer these questions a second time.

By contrast, those sampled businesses that do not get the one page screening questionnaire will instead receive a request to complete the full survey on the web, with instructions on how to access and complete the web survey. The screening questionnaire will only be used in the initial mailing, since it is intended to quickly obtain contact data and weed out ineligible businesses. All follow-up efforts will be directed at getting businesses to complete the full survey, rather than potentially creating two follow-up efforts, first to get the screening questionnaire and then to get the full survey.

Incentives

The use of incentives has often been shown to encourage people to complete surveys, though sometimes only to a small degree. Through an experimental design, four research questions will be answered:

• How much do response rates increase (if at all) when microbusinesses are offered financial incentives?

• Whether or not response rates increase, could financial incentives be cost effective by reducing the number of follow-up attempts required?

• Does data quality (e.g., the amount of missing data or the number of edit flags generated) vary based on the incentive received?

• What are the differences between a prepaid vs. a promised incentive? The $2 incentive will be prepaid while the $20 incentive will be postpaid. Some research indicates that prepaid incentives are more effective (Berk et al 1987). A postpaid incentive is being used for the $20 incentive because it is potentially wasteful to mail larger amounts of money when there is uncertainty regarding the contact information and in who will complete the survey. There is also a possibility that mailing an actual debit card, though not yet activated, may help to assure the respondent that the offer is real, as compared with promising to send something later; this may lessen the difference between a prepaid and postpaid incentive. This research design does not allow one to completely separate the impact of differing incentive amounts from the impact of prepaid versus postpaid methodologies; however, it is helped by the fact the expected impact of each approach is in opposite directions.

Logistically, the $2 prepaid incentive will be included in the first mailing (either the one-page screener or the invitation to the web survey); for small incentives, the actual receipt of the incentive appears more effective than the promise of a future incentive. The $20 incentive, by contrast, will be promised in the initial contact by including a debit card that will be activated upon completion of either the screener (for businesses in that group) or the web survey (for all other businesses, including those that fail to complete the screener but do complete the full survey). (The total cost of the debit card will be $21, including a $1 charge for activating the card.) For those businesses that are offered an incentive to complete the one-page screener, a second incentive to complete the full survey will not be offered. Experience suggests that the initial incentive will continue to create positive attitudes toward the survey, rather than creating an expectation of an incentive at each stage. The survey data will be used to confirm whether there is a residual benefit from the initial incentive. For businesses who were first asked to complete a screener, incentives will be paid immediately upon receipt of the screener rather than waiting for completion of the full survey; in general a mere promise of an incentive (rather than immediately sending an incentive) seems to decrease the perceived value of the incentive, so requiring a two-step procedure (completing both the screener and full questionnaire) seems likely to further diminish the perceived value of the incentive, and possibly antagonize respondents to the screener by seeming to continually add new conditions before paying the promised incentive.

In order to have the debit card activated, a sampled business will need to either mail back a completed (or partially completed) questionnaire, or click on the final “submit” button on the web questionnaire to indicate the respondent is finished with the survey. Since all questions are voluntary, respondents will receive the incentive even if some items are left blank. If a questionnaire is exited without submitting the data and left untouched for two weeks, the contractor will contact the business to determine the status of the questionnaire.

Analysis

As a rule, one factor will be examined at a time. (Table B.7. presents the summary table shells.) For example, when looking at the impact of incentives, the response rates and number of contacts for the three incentive groups ($0, $2, and $20), without regard to the other groupings (e.g., sending a one-page screener versus starting with the full questionnaire), will be compared. This will provide an overall measure of the effectiveness of a particular strategy across multiple data collection approaches and also provides the maximum number of cases for making comparisons. The stratification variables are included primarily to attain an adequate number of firms involved in R&D and to provide a trial run of the intended full survey methodology, but differences across the stratification categories will be examined.

Fifty debriefing interviews will be conducted with respondents and non-respondents using Webex. The purpose of these debriefings is to understand general reactions to the survey, evaluate content of the questionnaire, such as learning whether and why businesses made errors in some of their responses (e. g., based on question misinterpretation or lack of knowledge),effectiveness of the contact strategies, what motivated businesses to respond and for the non-respondents why they did not respond. A draft of the debriefing protocol is provided in Attachment E.

Table B.7. Illustrative summary table shells

Comparison of incentives

Data collection result	No incentive (N=1,334)	$2 prepaid (N=1,334)	$20 postpaid (N=1,332)
Mean response rate
Mean number of contacts
Average time until completion of full survey

Comparison of incentives by organization type

Data collection result	No incentive (N=1,334)	$2 prepaid (N=1,334)	$20 postpaid (N=1,332)
Mean response rate
Sole proprietors
C-Corporations
S-Corporations
Partnerships
Mean number of contacts
Sole proprietors
C-Corporations
S-Corporations
Partnerships
Average time (in days) until completion of full survey
Sole proprietors
C-Corporations
S-Corporations
Partnerships

Note: This table is exploratory only, based on the hypothesis that sole proprietors may be more like households than other small businesses, and thus may respond to incentives differently. Given the smaller Ns, differences would need to be relatively large to be statistically significant.

Potential value of screener

Data collection results	Screener (N=2,000)	Full questionnaire (N=2,000)
Response rate after initial contact
Number of email addresses obtained		NA
Number of corrected telephone numbers obtained		NA
Number of corrected addresses obtained		NA
Number of ineligible businesses identified		NA
Response rate after first offering of full survey
Final response rate
Average number of contacts
Average time until completion of survey

Potential value of stratification by organization type

Key variables	Sole proprietors (N=1,000)	C-Corporations (N=1,000)	S-Corporations (N=1,000)	Partnerships (N=1,000)
Participate in R&D (Q25)
Engage in innovation (based on Q17-Q20)
Mean revenue (Q10)
First company started by owner (Q38)
Mean owner hours per week (Q37)
Reasons for owning own company (Q3)
Hired new employee (Q6)
Highest level of education (Q40)

Note: To the extent that differences are small between organization types, there may be less need for stratification by organization type in the full survey.

Potential value of stratification by number of employees

Key variables	1 employee (N=850)	2 employees (N=850)	3 employees (N=850)	4 employees (N=850)	5-9 employees (N=600)
Participate in R&D (Q25)
Engage in innovation (based on Q17-Q20)
Mean revenue (Q10)
First company started by owner (Q38)
Mean owner hours per week (Q37)
Reasons for owning own company (Q3)
Hired new employee (Q6)
Highest level of education (Q40)

Note: To the extent that differences are small based on the number of employees, there may be less need for stratification by this variable in the full survey.

B.5. Reviewing Statisticians

Dr. Adam Chu

Associate Director

Westat

(301) 251-4326

Dr. Hyunshik Lee

Senior Statistician

Westat

(301) 610-5112

Jock Black

Mathematical Statistician

National Science Foundation

(703) 292-7802

Audrey Kindlon

Survey Statistician

National Science Foundation

(703) 292-2332

John Jankowski

RDS Program Director

National Science Foundation

(703) 292-7781

Dr. Stephen Cohen

Chief Statistician

National Science Foundation

(703) 292-7769

Rebecca Morrison

Survey Statistician

National Science Foundation

(703) 292-7794

Citations

Berk, Marc L., Nancy A. Mathiowetz, Edward P. Ward, and Andrew A. White. “The Effect of Prepaid and Promised Incentives: Results of a Controlled Experiment.” Journal of Official Statistics, Volume.3, No.4, pp. 449–457. (1987)

de Leeuw, Edith. “To Mix or Not to Mix Data Collection Modes in Surveys.” Journal of Official Statistics, Volume 21, No. 2, pp. 233–255. (2005).

Dillman, Don A., Jolene D. Smyth, and Leah Melani Christian. (2009) Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method. New York: Wiley.

Knittel, Matthew. Susan Nelson, Jason DeBacker, John Kitchen, James Pearce, and Richard Prisinzano. “Methodology to Identify Small Businesses and Their Owners.” Office of Tax Analysis, Department of the Treasury. Technical Paper 4. August 2011.

Luo, Amy and Glenn D. White, Jr. “Exploring a New Establishment Survey Incentive to Improve Response Rates.” In: Proceedings of the Survey Research Methods Section, American Statistical Association, pp. 3915–3918 (2005).

Attachments — Data Collection Instruments

1. MIST Questionnaire

2. MIST Screener (1 page reduced questionnaire)

3. Brochure

4. Correspondence

• 1 page screener — no incentive

• 1 page screener — $2 incentive

• 1 page screener — $20 incentive

• Invitation to full survey — no incentive

• Invitation to full survey — $2 incentive

• Invitation to full survey — $20 incentive

• Follow-up email invitation after getting one-page screener returned—No incentive or $2 incentive

• Follow-up email invitation after getting one-page screener returned—$20 incentive

• Follow-up mail invitation after not getting one-page screener returned—No incentive or $2 incentive

• Follow-up mail invitation after not getting one-page screener returned—$20 incentive

• Follow-up mail invitation after no response to initial invitation—No incentive or $2 incentive

• Follow-up mail invitation after no response to initial invitation—$20 incentive

• Thank you email if $20 incentive offered after completion of survey

• Thank you email after completion of survey if no incentive remains

5. Debriefing Interview Protocol

1 Microbusinesses are defined as businesses with fewer than five employees. Business with 5 to 9 employees are included as part of this testing to facilitate potential comparisons to the BRDIS data.

2 Businesses can be counted in a variety of ways. U.S. Census Bureau statistics show 5.7 million firms (or 7.4 million establishments) in 2010, with 93 percent of those firms in the four business categories sampled in MIST (see http://www.census.gov/econ/susb/). The Census Bureau statistics are limited to firms with paid employees. Most of the difference between the IRS based statistics and the Census statistics is from the inclusion of all Schedule C returns, with 25.3 million filers for tax year 2009 (by contrast, Census reports that 0.9 million firms were sole proprietorships). Excluding the Schedule C returns, there were 9.7 million returns filed using Forms 1120, 1120S, and 1065. Another source of data is Dun and Bradstreet, which tracks 29.5 million active companies in North America as of July 2013 (see http://www.dnb.com/company/our-data/data-quality-of-data-as-a-service.html), though that statistic includes non-U.S. businesses.

3 Trivedi, Shamik. (2011, May 24). Treasury Seeking to Better Identify Small Business Owners. Tax Notes Today Retrieved August 20, 2013, from http://services.taxanalysts.com/taxbase/tnt3.nsf/

4 The difference between an S- and C-corporations is mostly by how they are taxed. S-corporations elect to pass corporate income, losses, deductions and credit through to their shareholders for federal tax purposes resulting in what is called single taxation. C-corporations are taxed at the corporate and individual level. A partnership requires at least two individuals. The partnership reports income, deductions, gains, or losses, but it does not pay income tax as that is passed to the partners. A sole proprietor is someone who owns an unincorporated business by himself or herself. (Source: www.irs.gov)

5 Part of the reason may be that services are disproportionately represented among the smallest firms. U.S. Census data for 2010 indicate that 43 percent of manufacturing firms had less than 5 employees, and these firms had 2 percent of manufacturing employees; by contrast, 74 percent of firms providing professional, technical, and scientific services had less than 5 employees, and these firms had 11 percent of the employees in that industry. See http://www.census.gov/econ/susb/. Counts of manufacturing firms fail to reflect their economic importance; they comprised 5 percent of all firms but had 18 percent of all receipts (see http://www.sba.gov/advocacy/849/12162#susb).

6 All correspondence for MIST can be found in Attachment D.

7 Table B.6 presents the sample distribution. Analytic table shells to examine the survey results are presented later in this document.

8 These estimates assume the worst-case scenario: a proportion equal to 50 percent. Standard errors should be smaller for other proportions.

9 It is important to test the initial response rates to examine this issue. Later response rates will be affected by follow-up activities, and will not provide a clean measure of the impact of the screener.