Comments and responses about two questions

C1: How will the PIs distinguish between contracts that are canceled from those that were not renewed for reasons unrelated to Sandy?

C2: We presume that there is some baseline rate of cancellations, so how will processors distinguish cancellations from Sandy from baseline levels?

C3: What are the PIs thoughts about providing a definition to processors for what types of cancellations they should consider as a “result of Sandy”?

Response: These three comments addressed jointly by modifying Questions 4 and 5 to:

4: During the two months following Hurricane Sandy (Nov. & Dec. of 2012), how many of your sales contracts were canceled by your customers? What percentage of your sales volume does this represent? What is typical for this two-month period? When did your customers return and what did your firm do in response?

5: During the two months following Hurricane Sandy (Nov. & Dec. of 2012), how many of your purchase contracts were canceled by your suppliers? What percentage of your purchasing volume does this represent? What is typical for this two-month period? When did your suppliers return and what did your firm do in response?

Rationale:

We suspect that cancellations would have occurred quickly, which suggests bounding based on time (C3). Bounding based on time would also allow us to compare to a “non-Sandy” time period (C1, C2). However, the major drawback of this approach is that we may underestimate the effects of Hurricane Sandy if cancellations occurred over a longer period of time.

Other refinements considered:

• We could change the question to “…that you consider a result of Sandy.” This is not much better than the original version: each respondent may interpret this differently. We may over- or under-estimate the true effects of Hurricane Sandy.

• We could simply ask “How many of your usual [customers/suppliers] canceled [purchases/deliveries] or contracts and stated this was as a result of Hurricane Sandy?” A problem with this is that the processors probably are not given a reason for the contract cancellation, especially for suppliers. We may massively under-estimate the true effects of Hurricane Sandy.

C4: What are the PIs thoughts structuring as a multiple choice question with numerical ranges rather than asking for a specific number, given both the issues identified above as well as a potential for recall bias?

Response: We considered “closed-ended” or “open-ended” for this question (and all of our other questions). One of the major challenges described with closed-end questions is the “context” effect that might occur if the “scale” of the choices is incorrect (Smyth, Dillman, and Christian, 2007). Dillman et al (2009) suggest that open-ended might be better if:

“the surveyor does not want to influence respondent answers by providing a set of answer choices; when the goal is to collect rich, detailed information from respondents; and when the surveyor is questioning about topics for which little information is known ahead of time.” (p. 72)

Drawbacks include difficultly encoding responses, higher variability in answers, and non-response bias (in self-administered surveys only, not relevant for this survey).

Ultimately, we chose “open-ended” for the “influence” and “little information” reasons described by Dillman et al. (2009). This is partially due to the variability in size of our respondents: some might have just a small number of contracts while others have dozens.

References

Dillman, D.A., J.D. Smyth, and L.M. Christian. 2009. Internet, Mail and Mixed-Mode Surveys: The Tailored Design Method. 3^rd Edition. Hoboken, NJ: John D. Wiley & Sons.

Smyth, J.D., D.A. Dillman, and L.M. Christian. 2007. “Context Effects in Web Surveys: New Issues and Evidence. In The Oxford Handbook of Internet Psychology. Eds A. Joinson, K. McKenna, T. Postmes and U. Reips New York, NY: Oxford University Press.