Attachment O NASS Review ERS Response

Attachment O Summary of NASS ICR Review and ERS Response

SSA Item 9:

NASS Comment: “Shouldn’t farmers only have a higher show-up bonus? Why is there a difference in the payments resulting from winning the auction? Their opportunity cost of time is represented in the show-up bonus. The only justification I could see for changing the value of the winning prizes is that you may be arguing that farmers will not give their offers sufficient thought if their numbers are not higher.”

ERS Response: The opportunity cost of time is captured in both the show-up bonus and the incentive payments. As OMB noted in earlier discussions about this project, there is a concern that the payments are too low for farmers if we are trying to have external validity. As for the idea that farmers “will not give their offers sufficient thought if their numbers are not higher,” yes, that is the assumption behind the concern over hypothetical bias. The salience of the reward is related to the opportunity cost of time. Note that the marginal bidding incentives, conditional on endowments, are the same for both populations.

Action: With the removal of the experiment student population no action is needed on this comment.

Attachment E2 EDAP Reviewer 2

NASS Comment: “This is a good point, and illustrates a question I still have after reading all the files in this folder.

I personally do not think this should matter provided each bidder assumes there is at least one other bidder in the game, because I do not see why anyone would bid shade in this structure. Since there is no outside option, winning the auction is strictly necessary to get any payout other than the show-up fee you’re getting anyway. Your opponents are in the same position. So won’t you all present the most generous offer possible?

If what I’m suggesting is not true, then the reviewer has a good point. If there is bid shading, then the # of opposing bids is important in determining optimal bid.

See other files for my concerns on this issue.”

ERS response: Part of the reason for not fixing the number of bidders in each round, as noted elsewhere, is for external validity. In the actual CRP, farmers do not know exactly how much competition they will be facing. As for the assertion that “won’t you all present the most generous offer possible?” We do not believe that is likely to occur. The three peer reviewers of the EDAP, who are all experts in experimental studies of conservation auctions, did not indicate that was a concern. The most generous offer possible would have zero expected gain, and so the participants would not receive any monetary benefit of participating. More importantly, the budget constraint adjusts across rounds, so the information given to the participants about the expected EBI cutoff, will be valid in sessions of different sizes.

Action: No change was made in the plan to conduct recruitment in waves that each clear separately. The comment implicitly suggests that the rounds should be the same population sizes, but also seems to suggest that farmers need different information about the competition they are facing. We believe that any effort to make changes along these lines would hurt the external validity by deviating more from the structure of CRP general signups.

NASS Comment: “As mentioned in the slideshow file, I am also skeptical of different setup structures between farmers and students. The only difference I think is justified is the show-up fee.”

ERS response: See response below on attachment I for additional detail. The only two differences in the farmer and student designs are the lower participation payment for students and the use of different acreage fields to scale up the farmer payments. This scaling ensures that the marginal auction incentives and Nash equilibrium offers are identical for farmers and students. Most importantly, the difference in payments is critical to the desire to test “typical” student-based conservation auction experiments against “typical” farmer-based conservation auction experiments. Eliminating the difference in payment levels would reduce the ability of the (originally proposed) study design to inform conservation auction research policy on the question of whether the benefit of running lower cost studies with student populations comes at a cost of producing bias results that lack external validity for inference about conservation auction design.

Action: With the removal of the experiment student population no action is needed on this comment.

Attachment F EDAP Response to Reviewers

NASS Comment: “Exactly – this is why you have different show-up fees. This logic does not lead to different experimental design/endowments/bids/payoffs.

Youll see that I independently came up with the same criticism in Attachment M, which I read before the reviewers’ reviews”

ERS Response: “The experimental design and endowments and bids and payoffs in per-acre terms are exactly the same for students and farmers. This comment suggests that scaling up the payments for farmers changes incentives, but as noted elsewhere the Nash Equilibrium is exactly the same for both populations. The study was explicitly designed to modify the scale of payments without modifying the marginal incentives by scaling all costs and payments proportionately.”

Action: With the removal of the experiment student population no action is needed on this comment.

Attachment I (H) CRP Experiment Screen Shot

NASS Comment: “The description for net income without CRP participation indicates that this is the money you would make from enrolling in CRP – you mean not enrolling, right? The same issue is in the next picture too.”

ERS Response: Yes, this needs to be edited.

Action: The text was edited.

NASS Comment: “I assume that sometimes the maximum allowable payment is a binding constraint, that is, max payment < net income without participation.”

ERS Response: In the actual auction, yes this can be a binding constraint. In designs where the extensive margin option is an important part of the research hypothesis, the design should include that. We explicitly designed the experiment to avoid such a case because defaults and information treatments can only be applied to people who have already decided to engage with their county office to submit an offer, meaning that they are extremely unlikely to be facing this binding constraint. To include such incentives in our design would be inappropriate for the desired context and would significantly reduce the statistical power of the study.”

Action: No change was required as this was primarily a clarification question. An outside, extensive margin was not added as this was a key aspect of the study design given major consideration by the design team early in the project development and was viewed as an appropriate design decision by the three external peer reviewers.

NASS Comment: “I presume the 45 points for requesting program payments less than the allowable maximum isn’t a flat amount dependent solely on whether the number is lower than the max. For example, if they were willing to accept $80 per acre instead of 91, maybe that 45 becomes 50.

If so, I suggest changing the description to something more generic like “Points based on program payment”. Right now, it looks like you get 45 points simply for being below $100.”

ERS response: You are correct that this is not a flat amount of points earned for any discount in rental payment. The screen shot shown illustrates the software using the both treatments, including the live score updating treatment. Under that treatment, this points value will change as the participant changes rental rates, which will avoid the suggested confusion. Under the control, participants will not see this score and will only see it on the final screen. The decision not to publish the exact points used for this “cost factor” is an intentional decision to maximize external validity. Under the actual CRP program, participants face uncertainty around the exact scale of the cost factor points. There is clear communication to CRP participants that lower rental rates will score higher, but the exact correspondence is not published by FSA.

Action: No change was made in how cost points are communicated in the software.

NASS Comment: “This frame again suggests that if they don’t win the auction, they don’t get paid. This is at odds with real-life CRP auctions in which the alternative use (most likely farming) provides income as well.

Of course, the flip side here is also true. In order to extract value from the alternative use, you need to work the land, and these participants know they will not be working the land.”

ERS Response: We don’t see how there is a “framing” here to suggest that if they don’t win they don’t get paid. They get the participation fee. The design of the experiment, to pay for the net gain over the assigned external rent is well documented in the proposal. The research team did consider a payment structure in which there would be a payment for the “absolute” outcome rather than based on net-gains. Under this design, even participants who submitted losing offers to the auction would get a payment, but they would also have the potential to do worse in the auction if they bid too aggressively and win the auction. A number of experimental studies in auction design and other choice environments use such a structure. In essence, this amounts to giving every participant an “endowment” fund from which they can lose money. Such designs are most appropriate when there is an expectation that in the study domain it is common for a meaningful subset of decision makers to incur losses based on their decisions. Since the CRP is a voluntary program, the research team did not feel that such outcomes are common.

Action: No action taken.

Attachment K Students as Models Whitepaper

Page 29

NASS Comment: “This is roughly in line with my priors. I would have expected farmers and students to mostly be the same.

I believe that your experiment is especially robust to differences, because the incentives of the two groups is aligned, and the experiments are almost the exact same. The only ways they could differ are:

1. Chance (always possible but unlikely at your high N number of participants)

2. Students not understanding the subject (unlikely)

3. Farmers just doing whatever they do in real life, at the cost of losing rewards in the experiment (unlikely)

4. Differences caused by different endowments and rewards in your experimental design (possible).”

ERS response: The NASS reviewer clearly agrees with Reviewer 3 of the EDAP that students and farmers will, in most cases, be comparable. The research team largely agrees with this perspective, and thus treats this as a null hypothesis. However, there are good reasons to be concerned that students might react differently beyond what is listed here. These include differences in experience with conservation auctions, differences in content area knowledge (such as the idea of trading off crop rental income versus program payments and having to net out planting costs), age and other sociodemographic factors, and possibly psychological factors, such as risk preferences, that influence approaches to auction strategy. All of these are reasons to take seriously the research hypothesis that there are systematic differences between students and farmers in completing the tasks required to compete in a simulated conservation auction. The NASS review is raising the payment difference (item 4) as a potential driver of differences. As noted in the response above, that is a source of differences that the (original) study design intended to retain.

Action: With the removal of the experiment student population no action is needed on this comment.

Attachment M Instructional Slides

“Participation details slide”

NASS Comment: “Presumably, their job is attempt to maximize the value they get out of their imaginary parcel, which may or may not entail getting in CRP.”

ERS Response: Yes, although this seems to be at odds with the earlier comment on attachment E2 in which the NASS reviewer argued that participants would just submit the most generous offer.

Action: No action take. This comment does not suggest any change.

“Practice round slide”

NASS Comment: should 'none of the above' be allowed? As in, the imaginary parcel is not submitted for any CRP practice.”

ERS Response: As noted in the response to the second item on attachment I, no, an extensive margin choice should not be allowed. The research team considered this design and it would reduce the statistical power.

Action: As noted above, no change was made. The study design was not changed to add an outside option.

“Program payment offer points” slide

NASS Comment: “this is what I'm referring to in my comment on Attachment I concerning the wording of points. You get more/extra points if you go further below that max. Ensure the participants see that.”

ERS Response: Participants will see this instruction before seeing the points on the software.

Action: No change was made as this comment seems to support our placing of this information in the instructional slides.

“If your offer is selected” slide

NASS Comment: “this is what I'm thinking about in my comment on Attachment I talking about non-CRP payments.

Shouldn't there be a payment to participants who DON'T win the auction?

I feel like the only rational move in this experiment as it's structured is to bid with the maximum points. You walk away with nothing other than a show-up payment you receive regardless unless you win the auction. This is the position everyone in the auction is in, regardless of what parcel they have. Since I know everyone is in that position with no outside option, we will all bid in a way such that we maximize our points because we have no outside option.

In the real world, there will be some separation and not everyone will want to maximize points, Because those with higher quality land would rather farm than participate in CRP, while those with low quality land (especially low quality AND sensitive) will find it makes sense to make a more competitive bid to CRP.

In this game, even if I have high-quality land that really should be farmed and not put into CRP, I know I have to try to win the auction, because I can't participate in that outside option.

My expectation is that this experiment will have a higher % of participants offer the maximum points possible than in real life.”

ERS Response: This comment does not accurately represent the incentives of the auction, None of the external peer reviewers, all experts in conservation auction experiments, thought that the proposed design would induce irrational bidding. We ran these comments by several other peers and asked whether they could envision the sort of offer strategy described here. None of them were convinced that this would be a problem.

Action: No change was made.

“Finally” slide

NASS Comment: “This is exactly what I'm concerned won't happen.”

ERS Response: As noted above, other reviewers do not share this concern.