Attachment V - Email Mode Sequence Test

Attachment V.

Mode Sequence for Soliciting Electronic Response in an Establishment Survey

The Occupational Employment Statistics (OES) survey collects data about business employment and wages electronically via a web instrument. Prior to the survey invitation, a pre-notification letter is sent to sampled establishments informing them of the upcoming survey. The pre-notification letter also requests the establishments to provide contact information, including an Email address, for the appropriate person within the business. For the businesses that respond to the pre-notification letter and include an Email address, it is now possible to solicit their survey response through an Email, thus saving cost and time compared to a mailed invite. In order to maximize the benefit of this cost saving mode, research is needed to evaluate response rates, costs and timeliness of response for various mode sequences of solicitation.

Previous research on mode sequence has focused on surveys of households and individuals. Medway and Fulton (2012) conducted a meta-analysis of surveys that offered concurrent web/mail options and examined the effect on response rates. Their review of previous experimental comparisons found that mail surveys that also incorporate a concurrent web option have a significantly lower response rate than those that do not. This suggests that modes should be offered in a sequence rather than providing respondents with a choice. Millar and Dillman (2011) investigated the effects of different sequences of web and mail on response. They found that following requests for web response with a final request to respond by mail produces an equivalent response rate to when mail is the only response option.

Given the previous results from household survey experimentation, it is possible that a similar effect of mode sequence will be realized in establishment surveys. However, as Willimack notes in Designing and Conducting Business Surveys (Snijkers et al., 2013), techniques commonly accepted and used in surveys of households and individuals may not be applicable for a survey of businesses. Therefore, it is important to conduct an experiment of mode sequences in an establishment survey to investigate and document mode sequence effects on response, costs and timeliness.

The proposed study will randomize sampled units with an available Email address into three groups while controlling for establishment size and industry. All groups will receive an initial solicitation to the survey through an Email. Non-responders will then be re-contacted through either an Email or a both an Email and a mail invite, see table 1 for sequences.

Table 1

Experimental Group	Initial contact	NRFU 1	NRFU 2
EEE	Email	Email	Email
EEM	Email	Email	Email + mail
EMM	Email	Email + mail	Email + mail

NRFU – Non-response follow-up

Hypothesis 1: The group receiving only emails (EEE) will have the lowest overall response rate.

Hypothesis 2: The two groups receiving a mailed invite (EEM, EMM) will have equivalent response rates.

Hypothesis 3: The group receiving only 1 mailed invite will have the lowest cost per response.

Hypothesis 4: The group receiving two mailed invites will have the fastest time to response.

Through controlling for industry and size class, we will also be able to investigate whether differential response occurs for various types of business. This may lead to a tailored mode sequence for certain industries or size classes.

It may also be possible to send the initial email solicitation to randomized establishments at various days of the week. This would inform survey directors of the potential benefits of sending emails on, say, Tuesdays as opposed to Mondays. To the best of our knowledge, no such research exists for establishment surveys.

References

Medway, R. L., and Fulton, J. (2012). “When More Gets You Less: a Meta-Analysis of the Effect of Concurrent Web Options on Mail Survey Response Rates.” Public Opinion Quarterly; nsf047.

Millar, Morgan M., and Don A. Dillman. (2011). "Improving Response to Web and Mixed-Mode Surveys." Public Opinion Quarterly; 75(2): 249-269.

Snijkers, G., Haraldsen, G., Jones, J., and Willimack, D. (2013). Designing and Conducting Business Surveys (Vol. 568). John Wiley & Sons.