Pretest Report

`
MEMO
Date:

March 20, 2012

To:

Telba Irony and Martin Ho, US Food and Drug Administration

From:

Juan Marcos Gonzalez, F. Reed Johnson, and A. Brett Hauber, RTI-HS

Re:

Medical Device Decision Analysis: a Risk-Tolerance Pilot Study
Pretest Results Memorandum
RTI Health Solutions Project No. 0302907

This memorandum summarizes the findings from the pretest interviews conducted to evaluate the draft
survey instrument for the pilot study of patient preferences for features of weight-loss devices. Nine faceto-face pretest interviews were completed between February 13, 2012 and March 6, 2012, in Raleigh,
North Carolina. Respondents were invited to participate in the pretests if they had a calculated body mass
index of 30 or above, with at least three respondents having prior experience with bariatric surgery or
gastric banding.
In this memorandum, we present the characteristics of the pretest sample and a summary of respondents’
experience with managing their weight, including gastric banding. The memorandum summarizes the
issues with the survey instrument that were identified before and during the pretest interviews, and the
changes that we propose are made to the final survey instrument to address the identified issues. All the
proposed changes were tested during the pretest interviews.

DEMOGRAPHIC CHARACTERIS
TICS OF PRETEST RESPONDENTS
Although we attempted to recruit respondents with diverse backgrounds and experiences, the pretest
conducted with a small convenience sample; thus, any results should be interpreted with caution. Table 1
summarizes the characteristics of the pretest respondents.

Table 1.

Characteristics of the Pretest Respondents
Respondents
(N = 9)

Question
What is your gender?
Male

5

Female

4
45 (10.88)

Age, mean (SD)

What is your marital status?
Married

6

Widowed

0

Divorced or separated

2

Single

1

Other

0

Which of the following racial groups best describes you?
American Indian or Alaska native

0

Asian

0

Black or African American

4

Native Hawaiian or other Pacific Islander

1

Hispanic or Latino

0

White

4

Other (please specify)

0

Highest level of education you have completed
Less than high school

0

Some high school

0

High school or equivalent (e.g., GED)

1

Some college but no degree

1

Technical school

2

Associate’s degree (2-year college degree)

0

4-year college degree (e.g., BA, BS)

1

Some graduate school but no degree

3

Graduate or professional degree (e.g., MBA, MS, MD, PhD)

1

Height in feet and inches, mean (SD)

5’9”(4.29)

Weight in pounds, mean (SD)

264 (49.00)

Waist size in inches, mean (SD)
Hip size in inches, mean

a

b

45.5 (1.91)
51

SD = standard deviation.
a

Five respondents did not provide this information.

b

Eight respondents did not provide this information.

2

RESPONDENTS’ EXPERIENCE WITH WEIGHT MANAGEMENT
Table 2 presents a summary of respondents’ experience with weight changes and weight-loss efforts.
Also, after reading the descriptions of features of weight-loss devices, respondents were asked questions
to assess their understanding of the content of the descriptions and their experience with bariatric surgery
and gastric banding.
Table 2.

Experience With Weight Management

a

Respondents
(N = 9)

Question
How many pounds have you gained or lost in the last 12 months?
Gained more than 20 pounds

0

Gained between 10 and 19 pounds

2

Gained less than 10 pounds

2

No weight change

1

Lost less than 10 pounds

0

Lost between 10 and 19 pounds

0

Lost more than 20 pounds

2

Gained between 15 and 39 pounds

2

b

What motivated you to lose weight?
Improve problems of infertility

0

Improve my appearance in general

6

Improve my appearance for an upcoming event

1

Improve my high blood pressure

3

Improve my type 2 diabetes

1

Improve my high blood cholesterol

3

Improve my overall health

2

Other (please specify)

0

Don’t know

1

Doesn't apply

2

3

Respondents
(N = 9)

Question
How many pounds have you gained or lost since high school?
Gained more than 70 pounds

2

Gained between 40 and 69 pounds

2

Gained between 15 and 39 pounds

2

Gained less than 15 pounds

2

No weight change

2

Gained more than 20 pounds

5

Gained between 10 and 19 pounds

0

Gained less than 10 pounds

0

No weight change

0

Lost less than 10 pounds

0

Lost between 10 and 19 pounds

0

Lost more than 20 pounds

1

How many pounds would you like to lose?
I don’t want to lose any weight

0

Less than 5 pounds

0

Between 5 and 10 pounds

0

Between 11 and 20 pounds

1

Between 21 and 40 pounds

0

Between 41 and 60 pounds

3

Between 61 and 80 pounds
More than 80 pounds (average specified amount)

2
a

3 (122)

Don’t know

0
b

For which of the following are you currently taking a prescription medicine?
High blood pressure

4

Type 2 diabetes

0

High blood cholesterol

3

None of the above

2

Don’t know

1

Didn't answer

1

Which of the following conditions are you most concerned about?
High blood pressure

4

Type 2 diabetes

4

High blood cholesterol

1

Don’t know

0

Didn't answer

2

4

Respondents
(N = 9)

Question
b

Which of the following have you ever tried as a way to lose weight?
I have increased my physical activity

8

I have followed a diet (eat particular kinds of food, avoid particular kinds of food, or
limit the amount of food you eat)

8

I have had regular counseling, or I have joined a support group

3

I have taken over-the-counter drugs

4

I have taken herbal supplements

3

I have taken prescription drugs

1

I have had an operation to lose weight (e.g., bariatric bypass surgery)

3

I have had a gastric banding procedure (e.g., LAP-BAND®, REALISE®)

0

Other (please specify)

0

I have not done anything to lose weight

0

When did you have an operation (bariatric bypass surgery) or gastric banding
procedure to lose weight?
More than 4 years ago

2

2 to 4 years ago

0

1 to 2 years ago

0

Less than a year ago

1

Didn't answer

2

Does not apply

2

What kind of bariatric surgery did you have?
Endoscopic surgery

0

Laparoscopic surgery

2

Open surgery

1

Other (please specify)

0

Didn't answer

0

Does not apply
How satisfied were you with the information in the consent form you received before
your operation that explained possible benefits and risks?

2

Very satisfied

2

Somewhat satisfied

0

Neither satisfied nor dissatisfied

0

Somewhat dissatisfied

1

Very dissatisfied

0

I did not receive a consent form before my surgery

0

Don’t know

0

Didn't answer

2

Does not apply

2

5

Question

Respondents
(N = 9)

Did you have any serious complications after your surgery?
Yes

0

No

3

Don’t know

0

Didn't answer

2

Does not apply

2

Have you ever considered having bariatric surgery or getting a lap band to lose
weight?
Yes

2

No

3

Don’t know

0

Didn't answer

3

Have you ever had a major operation that required that you stayed at the hospital
overnight (for example, C-section, hip replacement, or gallbladder removal)?
Yes

3

No

2

Don’t know

0

Didn't answer

3

How many people die within a year after getting Device B?
1% (10 out of 1000)

0

Less than 1% (5 out of 1000)

5

Much less than 1% (1 out of 1000)

0

2% (20 out of 1000)

0

Didn't answer

2

Please indicate how difficult you think it would it be to eat ¼ cup of food at a time?
Extremely difficult

3

Very difficult

0

Difficult

2

Somewhat difficult

3

Not difficult

1

Please indicate how difficult you think it would it be to wait 4 hours between meals?
Extremely difficult

0

Very difficult

0

Difficult

0

Somewhat difficult

5

Not difficult

4

6

Respondents
(N = 9)

Question
Please indicate how difficult you think it would it be to not eat favorite foods?
Extremely difficult

2

Very difficult

2

Difficult

2

Somewhat difficult

2

Not difficult

1

Do you know anyone who got a weight loss device and gained back most or all of the
weight lost after the operation?
Yes

1

No

2

Don't know

0

Didn't answer

3

Didn’t ask

c

3

Which of the following symptoms do you think have the greatest limitation on doing
everyday work or social activities?
Difficulty swallowing

1

Vomiting

1

Pain

4

Not sure/don't know

0

Didn't answer

3

Have you ever experienced nausea or vomiting after a medical treatment?
Yes

3

No

0

Don't know

0

Didn't answer

3

Didn’t ask

c

3

Which would you be more concerned about?
20% chance of hospitalization with no operation

2

5% chance of hospitalization with operation

2

I'd be equally concerned with both

2

Not sure/don't know

0

Didn't answer

3

7

Respondents
(N = 9)

Question
About how much higher do you think your chance is of having kidney failure,
blindness, a heart attack, or amputation compared with people with normal weight?
About the same

0

About 10% higher

0

About 25% higher

1

About 50% higher

2

More than 50% higher

0

Not sure/don't know

0

Didn't answer

3

About how much higher do you think your chance is of having kidney failure,
blindness, a heart attack, or amputation compared with people with normal weight?
My chance is the about the same

2

Greater than the chance faced by people with normal weight, but lower than the
chance of people who are overweight

2

About the same as people who are overweight

2

More than most people who are overweight

2

Not sure or don't know

1

How likely is it that you actually would follow this rule?
I definitely would follow the rule

1

I probably would follow the rule

0

I might follow the rule

0

I probably would not follow the rule

0

I definitely would not follow the rule

0

I would not get this device

0

Didn't answer

5

Didn’t ask

c

3

How many people die within a year after getting device B?
1% (10 out of 100)

5

2% (2 out of 100)

5

5% (5 out of 100)

5

10% (10 out of 100)

5

a

Respondents provided weight-loss category and specific amount of weight loss desired.

b

More than one answer was provided by some respondents.

c

The “Didn’t ask” response category indicates the number of respondents who were assigned to versions of the survey instrument
that did not include the question.

8

COMPREHENSION OF RISK GRAPHICS
Before answering the choice questions in the survey, respondents read an explanation of the graphics
used to represent the change in the chance of dying from getting a weight-loss device, either because of
problems implanting the device or problems with the device itself. Respondents were asked a question to
test their comprehension of the probability grids. All respondents answered the risk-comprehension
question correctly without any help from the interviewers.

ISSUES WITH THE SURVEY INSTRUMENT
Before and during the pretest interviews, RTI Health Solutions (RTI-HS) identified specific issues that
could negatively affect the quality of the preference information collected with the survey instrument.
These issues and their potential impact on the collection of reliable preference information are
summarized below.

Survey Length
The length of a survey instrument affects the overall cognitive burden imposed on a respondent. Longer
surveys require more time and cognitive resources to answer the questions, and usually result in greater
levels of respondent fatigue. Fatigue increases measurement error, which in turn decreases the statistical
power of the study sample.
In a conjoint-analysis survey, length is largely driven by the number of attributes considered in the study.
Attributes included in a conjoint-analysis survey should be defined clearly to ensure that the preference
information obtained is comparable across respondents. The pretest survey instrument included 11
attributes that had to be carefully defined. Eleven is greater than the recommended number of six to eight
attributes per survey. (Marshall and colleagues [2010] report that less than 8% of published health
applications of conjoint analysis included more than 10 attributes.)
During the initial pretest interviews, respondents commented on the overall length of the instrument, and
most could not complete the survey during the interview. Also, respondents reacted to the amount of
information they were asked to consider in the choice questions by using simplifying heuristics that often
indicated they did not accurately recall important attribute details.

Survey Complexity
The complexity of surveys also affects respondents’ fatigue and the power of the sample in the study. The
complexity in a survey depends on several factors. These factors include poor understandability of the text
in the instrument and the layout of the information in the trade-off questions used to elicit preferences.
In addition, the language in a conjoint-analysis survey needs to be appropriate for most respondents
completing the survey. RTI-HS’s standard practice for survey development aims at a sixth grade reading-

9

comprehension level. Using this reading level reduces the risk of both increased variance and bias related
to educational attainment. We concluded that the length and complexity of the draft survey should be
reduced to ensure acceptable data quality.

Perceived Attribute Correlation
Conjoint-analysis surveys rely on the systematic variation of attributes associated with treatment profiles to
identify respondents’ relative preferences for the attributes in the study. A careful design of treatment
options provides the necessary statistical power to estimate preference weights for each study attribute.
Ideally, the attributes considered in a conjoint-analysis survey can be varied independently in the designed
choice questions. Independent variation of attributes is required for statistical identification of the choicemodel parameters. However, study attributes sometimes cannot be varied independently because
respondents perceive strong associations between particular attributes, which can lead to rejecting
specific combinations of attributes in the choice questions.
During the pretest interviews, respondents indicated that two attributes in the survey were correlated with
the amount of weight loss achieved after getting a weight-loss device. These attributes were (1) quality-oflife changes after losing weight; and (2) comorbidity risk reductions or medication requirements for treating
an existing comorbid condition.
Most respondents assumed the amount of weight loss achieved with a weight-loss device was a direct
effect of the device, while changes in quality of life and improvements in comorbidities generally were
perceived as a consequence of losing weight. Showing trade-off alternatives that were inconsistent with
that perception created confusion and rejection of the device descriptions. Specifically, respondents
generally were skeptical of devices that resulted in substantial weight loss without a corresponding
substantial reduction in comorbidity risk or treatment requirements, particularly when the alternative device
shown in the same question resulted in modest weight loss with large improvement in the comorbidity
attribute. Whether respondents’ perceptions are clinically accurate or not, we must limit the confusion
related to completely independent variation in attribute levels.

Recoding
Some study outcomes may be inherently under respondents’ control. When this happens, respondents
may be reluctant to evaluate the outcome profiles as described and instead mentally adjust the
descriptions to incorporate their perceived ability to control outcomes. That is, respondents select a
preferred device profile after adjusting or ignoring particular device features that they believe would not
apply to them. This adjustment of the treatment profiles is referred to as “recoding.”
When respondents recode, they are not responding to the designed stimuli prepared during the
development of the survey to ensure identification of preference parameters. Instead, respondents who
are recoding effectively create their own design, and their choices reflect their reaction to these alternative

10

stimuli. Without information on the alternative design that motivated respondents’ choices, it is not
possible to know the statistical properties of the preference data collected with the survey. The lack of
information can jeopardize the ability to identify statistically preference estimates for all study attributes.
Most pretest respondents recoded the amount and duration of the indicated weight loss in some or all of
the trade-off questions. Although respondents appeared to recode these attributes consistently, the
original draft survey instrument provided no method for determining how each respondent recoded the
attribute levels.

Attribute Levels
In a conjoint-analysis survey, the range of levels of each attribute should meet three criteria: (1) the range
of levels should span the clinically relevant range of outcomes that has been seen or might be expected to
be seen in clinical trials or clinical practice; (2) differences in levels should encompass the range of
improvements in efficacy outcomes or the range of increases in side-effect or risk outcomes that
potentially could be seen in clinical trials or clinical practice; and (3) the range of levels should encompass
the range over which respondents are willing to accept tradeoffs among attributes.
We checked respondents’ reactions to varying ranges of attribute levels in each interview. Several
attributes were found to have ranges that did not conform to the criteria presented above. These attributes
were: (1) amount of weight loss achieved with a device; (2) duration of weight loss; (3) duration of side
effects limiting daily activities; and (4) chance of dying from getting a weight-loss device. For the amount
of weight loss, respondents were not willing to accept a weight-loss device that produced only a 5% weight
loss. In the case of duration of weight loss, respondents reported that a 1-month weight-loss duration was
not reasonable given the invasiveness of the surgery to implant a weight-loss device. Also, respondents in
the pretest interviews did not think that experiencing side effects for 1 week was serious enough to
warrant foregoing any of the benefits of getting a weight-loss device. Finally, some respondents were
willing to accept a greater risk of death than the highest level of risk included in the draft survey to achieve
the benefit of weight loss. These respondents reported that their maximum willingness to accept a fatal
risk for weight loss was related to their perception that not losing weight conveyed a significant mortality
risk.

PROPOSED CHANGES TO THE SURVEY INSTRUMENT
We edited the draft instrument several times over the course of 3 days of interviews to address problems
as they arose. We then tested the changes with subsequent respondents. Below is a description of the
changes made to the survey and the impact of these changes.

11

Survey Length
To reduce the length of the survey, RTI-HS proposes eliminating three attributes from the final version of
the survey. These attributes are: (1) quality-of-life changes due to weight loss; (2) timing of weight loss;
and (3) duration of device inside the body. These changes significantly reduced the length of the survey.
We eliminated these three attributes after the first three interviews and evaluated the effect in the
remaining six interviews. All subsequent respondents were able to complete the survey during the
interview time, were more attentive to the specific features of the remaining attributes, and were less likely
to apply the simplifying decision heuristics observed during the first day of interviews.
In addition, RTI-HS also proposes eliminating the best-worst scaling questions from the survey. The
updated survey now includes only the choice questions and two contingent-behavior questions to collect
information on recoding. Eliminating the best-worst scaling questions significantly reduced the overall
length of the survey.

Survey Complexity
The smaller number of attributes significantly reduced the effort required to evaluate the trade-off
questions and thus improved the quality of the preference data obtained. Deleting the material required to
prepare respondents to answer the best-worst scaling questions and the questions themselves avoids
burdening respondents with a different question format.
In addition, RTI-HS proposes changing the labels used to define the attributes in the choice questions, the
levels used to characterize their likelihood or intensity, and the attribute descriptions in the survey. The
purpose of these changes is to lower the reading level for the survey and to emphasize important features
of the study attributes. When implemented during the pretests, these changes improved readability and
improved the consistency of respondents’ interpretation of the attributes and attribute levels.

Attribute Correlation
Eliminating the “quality-of-life” attribute limits the problems of perceived correlations among attribute levels
to the association between the amount of weight loss and the changes in the likelihood or severity of
comorbid conditions. As we indicated previously during survey development, attempting to vary weight
loss and comorbidity benefits independently invites scenario rejection and recoding. We observed this
problem in all nine interviews.
We considered several solutions to the problem of perceived comorbidity correlation during survey
development. RTI-HS pretested two changes we believe deal effectively with the problem. First, we
propose to restrict the experimental design to plausible combinations of weight loss and comorbidity
benefits. The restrictions would still allow for some degree of free variation between the amount of weight
loss and the severity or likelihood of a comorbidity.

12

RTI-HS also proposes to ask a separate choice question using the second practice question in the survey
to elicit preferences for changes in the severity or likelihood of a comorbidity that completely avoids
perceived confounding. Unlike previous solutions we considered, this does not add to the length or
complexity of the survey. The practice choice questions help familiarize respondents with the preferenceelicitation format and give them practice in evaluating simple tradeoffs before encountering the full set of
tradeoffs required later. Responses to these practice choice questions are not usually used to assess
respondents’ preferences. In this case, RTI-HS proposes taking advantage of having respondents
evaluate only a subset of the study attributes to evaluate changes in the severity or likelihood of a
comorbidity without presenting the amount of weight loss respondents would achieve with the devices in
the practice choice question. The levels in the practice choice question would be designed to maximize
the preference information obtained from respondents’ answers. Because the amount of weight loss
would not be included in the question, respondents would have no incentive to reject the scenario or
recode the levels shown because of perceived inconsistencies between the weight loss and comorbidity
benefit. We can analyze the data from this question separately or merge the data with the main trade-off
data. Below is an example of the revised second practice question.

13

Example #2
Which of these two weight-loss devices do you think would be better for people like
you?

Feature
Type of operation
Recommended diet
restriction

Device A

Device B

Endoscopic surgery

Open surgery

Can’t eat sweets or foods
that are hard to digest

Eat ¼ cup of food at a time

50% reduction

75% reduction

6 months

2 years

Average reduction in risk
of getting diabetes at the
lower weight
On average, how long
side effects last
(Remember that side effects will
limit your ability to do daily
activities several times a month.)

Which weight-loss device
do you think is better for
people like you?

Recoding
To discourage respondents from recoding the amount and duration of weight loss, RTI-HS proposes two
separate strategies. First, define attribute levels as clinical evidence gathered from other people who
received the device rather than outcomes that would be actually realized by a particular patient.
Respondents are told this information indicates what “people like them” could expect from each device.
This way of presenting the device outcomes is similar to the information respondents would have available
to them in evaluating actual device options.
Recall that the preference-elicitation question is “Which weight-loss device do you think is better for
people like you?” That question is immediately followed by “Would you get [this device] if it was available?
When framing the device profiles as clinical evidence, respondents thought of the profiles as average

14

results for “people like you” that combined the effects of the device with people’s own efforts. However,
recoding occurred when they answered the second question. Nearly everyone thought they would do
better than the average. The initial weight loss from the device was interpreted as “a jump start” or “getting
me over the hump” that would make achieving additional weight loss for longer durations easier. Our
second strategy collects information on that adjustment using two follow-up questions after – each
presented after one randomly selected choice question in the series) – that ask how much weight loss and
the duration of the weight loss as a result of the device and their own efforts.
We do not believe it is possible to eliminate recoding. However, the supplementary information will
facilitate statistically controlling for it in our analysis. Asking two questions will provide a check on whether
respondents answer the two questions the same or differently depending on the features of the chosen
device. Randomizing the presentation of the follow-up questions permits collecting recoding information
from all choice questions across respondents. The questions worked effectively in the last three
interviews.
Here are examples of the two follow-up questions.
Example follow-up question 1
On average, people who get Device B lose about 60 lbs. About how much weight do you think you would
lose with this device and your own effort?
Less than 60 lbs
60-70 lbs
70-80 lbs
More than 80 lbs Please specify how much weight you think you would lose. __________
Don’t know or not sure

Example follow-up question 2
On average, people who get Device B keep the weight off for about 1 year. About how long do you think
you would keep the weight off with this device and your own effort?
Less than 6 months
6-9 months
9-15 months
15 months-2 years
2-3 years
3-5 years
5-10 years
Rest of my life
Don’t know or not sure

15

Attribute Levels
RTI-HS recommends three changes in the attribute-level ranges. First, to ensure that respondents’
choices provide trade-off information among all levels of the study attributes, RTI-HS proposes eliminating
the 5% weight-loss level and the 1-month duration level. While we understand that there are actual
devices with these features, people who are considering lap-band alternatives are not also considering
devices of this type. It simply is not feasible to elicit preferences for both kinds of devices in a single
instrument.
Secondly, we recommend changing the maximum duration of side effects associated with getting a
weight-loss device from 30 years to “rest of life.” Although “rest of life” is an ambiguous level that varies by
respondent, we can approximate the interpretation given by each respondent by taking the difference
between respondents’ life expectancy and their current age.
Finally, we suggest increasing the maximum level of risk of dying from getting a weight-loss device to
10%. Although this would put the maximum risk of death associated with getting a weight-loss device
above the clinically relevant level of 2%, we will still get a preference-weight estimate for that level, and we
will improve our ability to identify respondents’ maximum willingness to accept fatal risks for given weight
loss.

REFERENCE
Marshall D, Bridges JFP, Hauber AB, Cameron R, Donnalley L, Fyie K, et al. Conjoint analysis
applications in health–how are studies being designed and reported? An update on current
practice in the published literature between 2005 and 2008. Patient. 2010 Dec 1;3(4):249-56.

16