Attachment X -- Anecdote Construction

Attachment X

Construction of Anecdotes

In the clinician choice experiment, participants in four of the experimental arms (#3, #4, #5, and #6) will be presented with “Patient Reviews” in the form of anecdotal comments, along with CAHPS information on Service Quality or both CAHPS and HEDIS information on Treatment Quality. To construct anecdotes that seem realistic to research participants and differ systematically in their emotional valence so that they could be assigned to physicians in a controlled manner, we conducted the formative work described below.

First, from a publicly available web site that rates physicians we downloaded 175 real patient comments about family practitioners in New Jersey, Georgia, Missouri, and Oregon. Using these comments as models, project staff constructed 124 patient comments that referred to similar aspects of physician performance as those addressed in real patient comments (accessibility, communication, caring about patients, staff quality) but that were also designed to fit into one of four valence categories: strongly negative, mildly negative, mildly positive, and strongly positive. In each valence category, we included in the patient comment a word with the appropriate valence from the ANEW word list (Bradley & Lang, 1999), which contains words with valences empirically derived from pleasantness ratings. For example, “distressed” and “useless” were rated as strongly negative words, “excuse” and “nuisance” as mildly negative, “trust” and “consoled” as mildly positive, and “capable” and “outstanding” as strongly positive. Variants of the ANEW words having the same root meaning were treated as equivalent in emotional valence (e.g., “distress” for “distressed”).

Each of these 124 fabricated comments was then rated on a) emotional valence and b) how informative the comment was about what it would be like to visit this doctor, by either 6 or 7 adults recruited from the community near Santa Monica. In a separate protocol, other adults recruited in the same manner were presented with a list of fabricated comments mixed together with real comments. They were told that some comments were real and others were made up by researchers, and were asked to judge each comment on a 5-point scale ranging from “definitely a real comment” to “definitely a made-up comment.” Each anecdote was rated on likelihood of being real by either 4 or 5 community members. Results from these judgment tasks, together with comments made by participants in a debriefing following each judgment task, were used to cull the list of fabricated comments, and modify some comments to make them more realistic. We also constructed new fabricated comments and tested the revised set by repeating the protocols described above. In the second round of testing, we obtained emotional valence and informativeness ratings on both fabricated comments and a set of 124 randomly selected real comments. Results of the second round were used to further revise the fabricated comments, which were then tested in a third round using the same protocol as in round 2.

The final set of comments to be used in the experiment consists of 150 fabricated comments, including 36 with strongly negative valence, 38 with mildly negative valence,

50 with mild positive valence, and 26 with strong positive valence. The comments range from 8 to 76 words in length.

In the health plan choice experiment, the patient comments from the clinician choice experiment will be modified through simple transformations to describe doctors in the health plan. For example, for comments referring to general practices or office staff, references to "Doctor B" will be changed to "Plan X." For comments referring to interpersonal skills, references to "Doctor B" will be changed to "Doctors in this plan" or "My doctor in this plan."

Reference:

Bradley, M. M., & Lang, P. J. (1999). Affective norms for English words (ANEW). Gainesville, FL. The NIMH Center for the Study of Emotion and Attention, University of Florida.