2484 Appendix 2 Comments

Appendix 2: Invited Reviewer Comments

It should be noted that invited reviewer comments should not be taken as indication of support of methodological decisions made in this ICR proposal, nor were they aware at the time they made their comments that they might be included as part of a federal register notice. The first reviewer was Dr. V. Kerry Smith, Distinguished Sustainability Scientist at the Global Institute of Sustainability, and Regents' Professor in the Department of Economics at Arizona State University. Dr. Smith is a renowned expert in environmental economics techniques, including stated preference survey design. Dr. Smith’s made 10 points in his comments. Dr. Smith’s comments appear below:

June 10, 2013

To: Matt Weber

From: Kerry Smith

Subject: Arizona River Valuation Survey

As promised, I went through the survey. It is a very interesting project. I was really impressed by all of the documentation developed in planning the survey. There were a number of other aspects I liked in the research design. However, I will focus my comments on the issues where I felt there was some need for changes. I have listed them in order of importance:

1. The payment vehicle is not explained well or justified. Which taxes will go up and why? Why will the water be reduced in the “expected future”? You suggest on page 6 it is to reduce the amount of water used by the river ecosystem. What is the alternative use for the water that is not going to the ecosystem services? This is not explained. How will the proposed changes be maintained and why does it cost more to maintain a system that is the current baseline (without the water quality improvements)? These issues are not adequately explained. Until they are, as a reader I don’t know how or why this ends up increasing my taxes indefinitely. People know that taxes depend on income –how do you know what they will pay before you know their income? I realize in a mail question it cannot be dynamic in the way envisioned by this comment. Nonetheless it suggests the tax payment vehicle must be explained carefully.

2. There are too many choice panels. I would explain the number respondents get and have no more than two or three. Respondents are not told each panel is independent of the one before. Since it is expected to be mailed, you need to allow in your experimental design that they will read ahead. It is essential that they cannot find the same option or a better option at a cheaper cost. With most conventional experimental design software programs, the blocking process for selecting choice sequences will not include this constraint. I have had to manually inspect all design panels in previous studies to be sure “better” options are not included as part of the randomization process.

I realize that this affects the assumption of randomization but respondents read ahead. It is the bundle of choice sets you give them that have to be considered as an integrated whole.

3. I felt the choice set panels were too dense. I also felt “expected future” needs to be more carefully identified as what will happen if they do nothing. This is too implicit for me. Also, when is the expected future? What is the time horizon for this to happen? The timing and details on page 6 are too vague.

4. A key element is using these conjoint or CV questions is that respondents believe their answers will be consequential. This follows from the Carson-Groves argument in ERE and Vossler and others recently in the AEJ supporting it. Your discussion on page 1 gives the impression opinions will be used in selecting “best” options. I believe the statements need to be stronger – results based on respondents’ votes will be presented to policymakers and their decisions consider constituents desires. Explanation must be consistent with alternative uses for the water that are motivating the change from the status quo.

5. For me, the description of ecosystem services and situation with and without flow is inadequate. All one knows is the new miles of river and forest acres that is it. A little text that there are other “stuff” affected but little details.

6. The term “Full body contact” in my opinion is terrible. Explain what you mean in plain English. Water will never be deep enough for swimming, but one could wade, raft and not be concerned if you felt in and went underwater. You also need to be specific about current water quality, fish live there, some other species are supported, “no full body contact” is inadequate.

7. As you note reminding respondents what to consider on page 12 – telling them the questions display “different tradeoffs” is inappropriate. Questions display alternative choices – don’t tell them to make tradeoffs. We assume they do but that is an assumption on our part.

You say there are three voting questions. I saw four plus an example. None of the explanation on this page was adequate. It really needs to be re-worked.

8. You indicate in your discussion of recreation trips per year. You need to give the context in the text – trips in a typical year or trips last year; where do they go most often?

9. Question 14 is too direct – you are using economic explanations rather than what people might use to explain their motivations.

(10) I would want varialbes to characterize respondents:

• education

• employment status

• own a home

• boat ownership

• more income categories (and leave out “estimated”)

• define what you mean by income and then ask them to give their best assessment

I realize most of my suggestions are written in fairly direct terms. I don’t mean them to be critical of you—I just wanted to emphasize their importance. I also wrote this quickly.

Finally, I did not get into issues with the application of that conjoint framework for valuation and the implicit assumptions about estimating WTP versus MWTP for changes in an attribute. You did not have much detail on this in what you sent. It might be worth sketching what you hope to derive from the model for policy analysis before finalizing the survey.

A follow-up teleconference was held with Dr. Smith to discuss the points made and how the survey could be improved. Point 1) is in regards to an insufficiently described payment vehicle. Revisions were made to pg. 6 describing that there are competing demands for water in southern Arizona, and the payments would be increased to compensate for not selling the water to these competing demands. Potential further description of the taxes, such as how to address taxes typically being progressive (depending on income) was not added. Pre-tests did not yield a sense of doubt from respondents regarding the scenarios and the taxes posed, thus adding additional complexity may be counterproductive and may actually invite suspicion. Point 2) is that the draft had too many choice panels. In the best of cases, each respondent is only asked one question to maintain independence of responses. However, asking multiple questions, or “replications”, is standard in choice experiment design since so much investment has been made in each survey to describe management options. There are concerns that the quality of responses is reduced as the number of replications increases. Based on Dr. Smith’s suggestion the 5 panels were reduced to 4 per survey, the lower end of replications suggested by Bateman et al. (2002: pg 253). As Dr. Smith suggests, the experimental design will be inspected and modified as necessary to eliminate better, cheaper options within the same survey block, which can be confusing for respondents. Point 3) is in regards to the description of the “Expected Future”. This is similar to the complaint made by other commenters, including Pima County (see Public Comment section below). The justification for reduced river flows was poorly described. Revisions were made to pg. 6 describing that there are competing demands for water in southern Arizona besides supporting instream flows. Furthermore, as suggested by Dr. Smith a specific time horizon was added for a more complete description of the Expected Future. Finally, there is now a summary graphic on pg. 8 for the flow and forest options. Point 4) deals with consequentiality - respondents must believe their responses matter and that the scenarios are real. Dr. Smith recommended stronger consequentiality language. Throughout pre-tests, respondents conveyed a strong sense that the scenarios were real, thus edits in survey language were not made. Point 5) notes the minimal description of other ecological impacts of water besides the forest. Indeed, this is a weakness of the survey background, reflecting a general need for more robust “ecological production functions” describing specified and various impacts from environmental changes (USEPA 2009; e.g. pg 4). What makes this survey possible is the rare availability of a modeled relationship between water flow and forest change for the Santa Cruz River. Point 6) remarks on the awkwardness of the “full body contact” attribute name. In pre-tests ORD found subjects did understood this attribute as described and thus ORD chose not to “rebrand” so central a feature of the background despite the wordiness. This attribute is meant to capture preferences for more direct recreational contact in contrast to the ecological attribute of flow & forest (which does mention the presence of fish), thus fish are not mentioned in this attribute description. Point 7) suggests not to mention “tradeoffs” but rather “alternative choices” on pg 13, this change has been made. Point 8) suggests being more clear in the recreation question, the question has been modified to say “in the last 12 months”. Point 9) addresses question 13, that it is too direct and options may not be complete. This question was dropped, in favor of collecting more detail on river-related recreational behavior, pursuant to Dr. Hoehn’s comments (see below). Point 10) are suggestions for further sociodemographic variables. These were all incorporated, with the exception of home ownership and boat ownership, for which there was insufficient space. The last comment from Dr. Smith was to sketch how the results were designed to be meaningful for policy analysis, which is contained in other portions of this ICR supporting statement (Dr. Smith had not reviewed the supporting statement, only the survey itself).

The second invited reviewer was Dr. John Hoehn, Professor of Environmental and Natural Resource Economics in the Department of Agricultural Economics at Michigan State University. Dr. Hoehn is also a renowned expert in environmental economics including stated preference surveys. In particular, Dr. Hoehn has co-authored articles describing the crucial insights that arise during qualitative research during survey development (Kaplowitz and Hoehn 2001, and Hoehn et al. 2003). Dr. Hoehn’s comments appear below:

Arizona Stream Issue Questionnaire

Issues and Questions

John Hoehn

June 23, 2013

The questionnaire overall is very well put together. The text is readable and information. The sequencing

of sections and questions is logical. The diagrams and graphics are clear and helpful in understanding the

text and issues.

The key questionnaire is how your respondents react to and understand the issues in the questionnaire. I

assume that you’ve either tested or have plans to test the questionnaire with respondents like those in your

sample. With that assumption out of the way, I do have some comments on the questionnaire. None are

game changers but they are offered as issues to consider before testing and finalizing the questionnaire.

Here they are:

1. Page 3. What is treated wastewater? What’s meant by “this was just a method of disposing of”?

I think I understand, but I’m not sure. I suppose treated wastewater is sewage that’s been through some type of treatment facility? How much treatment? Is it discolored and smelly? Does it contain harmful bacteria, viruses or chemicals that may hurt plants, animals or me if I come into contact with it? Is there another term that could be used— recycled water maybe?

2. Pages 3 to 12 are a little much in terms of explanation overload. The 10%/90% information

seems unnecessary to selection the options. Page 5 doesn’t seem relevant either. Page 6 could be edited to reduce the amount of info and focus attention on options and issues in a choice table. The Expected Future lays out what you get for no additional cost. Options A and B are also possible if people are wtp. The Expected Future and the Options differ in the following:

a. Miles of streambed flooded

b. Acres of cottonwood/willow ecosystem

c. Whether the water is safe enough for full body contact

d. How the N and S are affected

e. Cost

The cost issue is somewhat confusing to me. How does one get more water by simply payingmore? Water has to come from someplace—will it raise water prices where I live? Can you offer a little more explanation?

3. Page 7. It seems like what’s represented is with and without water flow. Why not just label the

scenes accordingly—With and Without?

4. Page 9 Why is a sample table necessary? Respondents have probably already read ahead and

know what’s coming. Respondents typically skim through a questionnaire before deciding to take the time to answer it.

5. Page 13-15 Option B in page 15 seems like a real bargain compared to option A on page 16 when I’m indifferent to north or south flows.

This raises a question—is the N/S question issue really important enough for the added

complexity. Water for the south seems cheap which seems a little surprising with Tucson in the

N. Why is water for S so much cheaper than water in N?

Would it be better to focus the cost/service tradeoffs on total flow miles, forest acres and body contact while leaving the N/S question as a separate allocation question? Are there returns to

scope or scale in provision that make the N/S question significant? If not, it might be better to go

with a simpler structure by eliminating separate rows for N and S flows, acres and body contact.

A follow-up teleconference with Dr. Hoehn occurred to talk through his comments and to discuss how the survey could be improved. Dr. Hoehn’s comments are broken into 5 points, which he prefaces by noting the important role of pre-testing (he was at that point unaware of the pre-testing ORD had done). Point 1) is a series of questions about wastewater. These questions are in large part addressed by the survey background or the survey attributes. For example, that wastewater treatment would eliminate odor is mentioned in the survey, and there is an attribute that captures the safety of water contact. The term treated wastewater was widely understood in pre-tests, and ORD wished to be plain about the source of the water, rather than using a term such as “recycled water”. Point 2) notes that the background material is quite lengthy. Dr. Hoehn suggests some areas to cut back information; however in pre-tests subjects supported maintaining the level of information given in the survey, including information on infiltration since the state of the aquifer is a factor people considered in pre-tests. A problem with the draft implicit in Dr. Hoehn’s comments was the impression that one could pay to get more water into the river. Instead, the intention is to investigate willingness to pay to maintain different extents of flow. Revisions were made to pg. 6 and a summary graphic was added on pg. 8 to make this clear. Point 3) asks why the photos on pg 7 aren’t titled “with” and “without” water. The reason is that in pre-tests this led to an erroneous conclusion that a possible scenario was zero water in the river, which is not actually a scenario considered in the survey. Point 4) asks why a sample question is needed (pg 9). In initial pre-tests without a sample question we occasionally encountered respondent confusion on where/how to answer since choice experiments are not commonly found in surveys. Thus we opted to include a sample question. Point 5) notes that options in the South seem like a bargain compared with the North. We designed the sample survey price levels in this manner, since in pre-tests people seemed to greatly prefer maintaining conditions closer to Tucson. Making prices cheaper for the South allows us to better estimate the public values for ecological commodities in both locations. The more difficult the alternatives are to decide between, the more preference information ORD gains for the utility model. A further point raised in discussion was possibly changing question 13 to be more experientially-based, such as frequency of listed recreational activities, rather than a “rate the importance of...” question. Responses might then be ways to organize willingness to pay results, or even predictors of willingness to pay. Based on this suggestion a new experiential recreation question #11 was added, and prior question #13 was dropped (see Dr. Smith point 9 above).

Comments were also solicited from the Sonoran Institute, a southern Arizona non-profit with a specific program focused on the Santa Cruz River. The Sonoran Institute had also been consulted at various times earlier in the project, and had provided the draft cover photo sequence (which has been replaced in the revised survey, see response to Pima County Comments in Appendix 3). Sonoran Institute comments appear below:

Hi Matt,

As we discussed on the phone just now, Emily and I are very supportive of your work on the “Willingness to Pay” survey focused on the Santa Cruz River. The results of this work will be very helpful for Sonoran Institute’s Santa Cruz River Initiative. Understanding community values regarding natural resource management is critical for prioritizing on the ground actions, crafting policy, leveraging additional funds.

We found this version of the survey to be much improved from the earlier draft you shared. We have some specific comments/edits regarding the language in the survey that I list here in order by page:

• Page 2 – the italicized intro language makes it seem like these are real options that authorities are considering. We suggest a rewording of the third sentence. Instead of “Your responses will help authorities to select the best option.”…we suggest something more like: Your responses will help authorities to better understand public opinion regarding management of the river.

• Page 6 – the language is misleading regarding the cut back to river flows in the “expected future”. Seems river flows would be reduced because managers think the river uses too much. Rather flows will be reduced because effluent may be used for other purposes. Instead of “In the future, river flows are expected to be cut back to reduce the amount of water used by the river ecosystem.” ….perhaps something more like this: In the future, river flows are expected to be cut back as effluent is used for other purposes rather than released into the riverbed. To me this is more accurate.

• Page 6 – reference to voting in the Flow & Forest description implies that the outcome is really up to the survey respondents.

• Page 9 – provide an explanation as to why certain boxes within the questions are shaded. We assumed these were items that did not change from the expected future. This should be more explicitly stated.

• Page 12 – there is a typo in the first sentence. There are 5 voting questions, not 3.

• Page 12 – the language in the third bullet referencing voting again implies that these are real options that people are voting for. Perhaps reword to say something like: Review and answer each question carefully. Understanding your opinion will help authorities to best manage the river.

• Page 18 – minor formatting typo under question 9, first bullet. The text “Never chose” is not the same font as the subsequent questions.

Thank you for the opportunity to review and comment on this work. We would be happy to discuss this work further with you.

Claire and Emily

Claire Zugmeyer

Ecologist

A teleconference was held to discuss how the survey could be improved based on Sonoran Institute comments. Most of the comments are suggestions to reduce the impression that the survey presents real, immediate options that authorities are considering. ORD explained that the wording was a methodological issue to ensure consequentiality for respondents (see Dr. Smith’s point 4 above, which recommended even stronger consequentiality language). The less hypothetical the scenario appears to be, the less invitation there is for strategic behavior from respondents. An example of strategic behavior is a respondent claiming a willingness to pay larger than their actual willingness to pay. Like other reviewers, the Sonoran Institute noted the confusing language regarding why river flows would be cut back. This was addressed by changes to pg 6 and pg 8 as described above. Other minor errors noted by the Sonoran Institute were also corrected.