NASS Comments & Forest Service Response

National Agricultural Statistical Service (NASS) Comments and Forest Service Responses to review of survey entitled:

Homeowner Risk Reduction Behaviors Concerning Wildfire Risks

April 2007

NASS Review conducted by:

Alexandra Riley

Statistics Division

Statistical Methods Branch

NASS

NASS review completed Summer 2006

• The respondent universe is not clearly defined; consequently, it is uncertain what population the study will ultimately draw conclusions on. The report [supporting statement] mentions that the survey will focus on targeted Home Owner Associations (HOAs) in three cities in the Western US that are high-risk areas for forest fires. The report did not specifically mention:

1. How were the three cities that the study will take place in chosen? How will the choice of these three cities affect the Forest Service's ability to generalize the results of the study to other cities in the Western US that did not have a chance to participate in the survey?

RESPONSE: The three locations where chosen based upon several criteria. First, experience with wildfire was considered. Each of the locations has a significant history with wildfire. Second, the three sites vary from a significantly urban setting in Boulder, Colorado to a rural setting at Camp Sherman, Oregon. Third, each of the locations has worked with a variety of organizations, NGOs and government agencies to address wildfire issues in their community. These three factors make the communities an ideal target for research in this area.

2. If an HOA chooses not to participate in the study, what methodology will be used to adjust for their non-response?

RESPONSE: Each of the HOAs identified for this study have been contacted to assure their willingness to participate in the study. The boards-of-directors for each have provided membership mailing lists to implement the survey.

3. If a household within an HOA does not respond, what methodology will be used to adjust for their non-response?

RESPONSE: Households within the HOA will be surveyed using the basic methodology prescribed by Dillman (2000, pp. 150-153). We believe that non-response will be minimized through the proper implementation of a mail survey and the support of the HOA Board of Directors. However, households that do not respond will be identified and a final contact made to determine if their characteristics are significantly different from the rest of the HOA surveyed (see supplemental statement B, item 3 for more detail). For example, if the response rate for part-time residents is significantly less than full-time residents then additional efforts will be used to increase the response from this group (i.e. contact the household via telephone to increase response).

4. Will homeowners that are not part of a community that has an HOA be given the opportunity to take part in the survey? If so, how will they be selected to participate? If not, how will that affect the interpretation of the results?

RESPONSE: Homeowners that are not a member of the targeted HOA will not be surveyed. This will limit the population that the results can be applied to; however, the objective of the research project is to reach a better understanding of the targeted HOAs and not the entire population living in the selected areas.

• If the purpose of the survey is to study the opinions and behaviors of a specific community so that Forest Service personnel can be better prepared to help that particular community, then the design of this survey is statistically defensible. However, the aggregated data from all the households that were surveyed should not be generalized beyond the communities that actually had a chance to participate in the study. Communities in other cities and other states may have different factors that would affect their responses to the survey.

RESPONSE: The objective of this data collection effort is to obtain internal validity for the statistical analysis of the selected communities. We do not intend to generalize beyond these three communities. However, comparing results across the three communities should allow us to achieve a better understanding of the factors that are more important in some situations versus others. This exploratory research will provide an important foundation to target future research that is more generalizable to a larger population.

• It is also unclear who within a selected household will actually complete the survey itself. Will it be all adults or the head of the household?

RESPONSE: The cover letter for the survey will indicate that an adult member of the household that is familiar with wildfire issues and the household response should complete the survey.

• The two survey instruments do not have any information identifying the survey, its purpose, who is conducting it, and any contact information, or basic instructions.

RESPONSE: There will only be one survey instrument.

• Each questionnaire needs a unique identifier for survey management purposes. There is also no space for the respondent to identify himself/herself to verify that the correct household responded. If the two survey instruments are independent and there is no identification on either survey, the management of the completed questionnaires will be difficult (e.g., the two surveys could become separated and the data may no longer be useable. Alternatively, the respondent may only return one of the questionnaires).

RESPONSE: Identification numbers will be added to the survey when we put the OMB number on the instrument.

Reference:

Dillman, Don A. (2000). Mail and Internet Surveys: The Tailored Design Method, 2^nd ed., New York: John Wiley & Sons, Inc.