A Tool for Measuring Graph Comprehension

Title: A Tool for Measuring Graph Comprehension

OMB# 0704-0553
04/30/2019

HSR Number: NRL.2016.0001
Authority
This research is conducted under authority delegated from the U.S. Secretary of the Navy to the U.S. Naval Research
Laboratory. Governing regulations Title 32 of the U.S. Code of Federal Regulations Part 219, (U.S.) Department
of Defense Instruction 3216.02, Secretary of the Navy Instruction 3900.39 (series), and Naval Research Laboratory
Instruction 3900.2 (series) provide that the researchers may be granted a waiver of the documentation of consent, if
this is the only connection between you and having participated in this study. Therefore, if you agree to take the
study after reading this information, you may continue to the next page (using the button below) to begin. If you
do not agree, then you should simply close your browser window (or tab).
Purpose of Study
This study involves research of information presentation through a variety of familiar forms that are collectively known
as statistical graphics (SGs). This includes depictions such as bar graphs, line graphs, pie charts, and extensions or
similar designs. We are evaluating a method to assess comprehension of SGs. Research information will be collected
to enhance basic scientific knowledge.
Procedures
1. We will first ask you to answer a few demographic questions. You may select “Prefer not to answer” to these
questions and still participate. You should be fluent in English to participate in this study.
2. You may use your normal web browsing method to access the survey. However, we recommend that you
be comfortably seated at a desktop workstation, with a high-resolution screen, reliable Internet access, and
freedom from interruptions.
3. If you agree and have a web cam, we will use the WebGazer software from Brown University to track your gaze
location. If you decline to permit this, you may still participate in the study without tracking your gaze.
4. You will next go through a series of trials. Each trial will consist of viewing a statistical graphic, viewing some
images, reading some text, and answering questions about the statistical graphic and the text.
5. When you have finished, we will ask your opinions of the system and ask you a couple of questions about your
experience with SGs.
Please remember that we are evaluating the techniques we use; we are not in any way evaluating you.
Expected Duration of Participation
Location of Study:

We expect each experimental session to last up to 60 minutes.

You will be at a location of your choosing from which you may browse the web.

Risks
The risks of the study are minimal – that is, equivalent to those of everyday life. The risks associated with use of
the web browsing technique you use apply. You will not need to press keys in rapid sequence, reducing the risk of
repetitive strain. You should adjust seating position to achieve a comfortable viewing position. If necessary, you may
take a break to stretch, stand, sit, or otherwise adjust position as often as necessary. There will be no cost incurred
by you except the time to participate.
There is a slight chance that you may experience eye strain, just as there would be if any computer user stares
intently at any computer monitor for a long time. The PI has not had these effects occur in similar, past sessions
with volunteers, however. If you become dizzy or feel eyestrain, please pause the experiment and rest. If you wish
to stop, you may do so at any time. We emphasize that there is no penalty for stopping.

1

Benefits
As a participant, you will help us learn about visualization methods and the measurement of comprehension of
statistical graphics. You will help make tools of this type easier and more effective to use. Benefits to society include
scientific knowledge of how SGs are interpreted and understood.
Compensation
There is no tangible reward associated with your participation in this study; you will not be paid for your participation.
Use and Confidentiality of Data
The data will be analyzed and the results may be published. The data and all related information will be held
in confidence. We do not collect your name or sufficient data to identify you. Data and related information will
remain confidential after the study ends. Data and documents will be stored at the U.S. Naval Research Laboratory,
Washington, D.C. The data may be used in anonymous form by the researchers or colleagues for comparison with
past, current, or future studies as may be scientifically appropriate.
Voluntary Participation
You can choose whether to be in this study or not. If you volunteer, you may withdraw at any time without
consequences.
Point of Contact
If you wish to ask questions about the study, you may contact the Principal Investigator. Please note that doing so
will break your anonymity.
Mark A. Livingston, Ph.D.
U.S. Naval Research Laboratory Code 5581

[email protected]
4555 Overlook Ave, Washington, DC 20375-5320, USA

For questions about the ethical aspects of this study, your rights as a volunteer, or any problem related to the
protection of study volunteers, please contact the NRL Human Subjects Institutional Review Board. You may reach
the Chair at . Please note that doing so will break your anonymity. If you wish to keep
a copy of this consent document, you should print one.
Statement of Consent
I have read the above information. I understand that I do not have to participate in this research, and that I will
not be paid. I further understand that this research has been granted a waiver of the documentation of consent,
and that my name will not be known to the research team or their support staff. I understand that my device’s IP
address will be known by the study server, but that this data will not be given to the research team and will be
discarded upon completion of the study or after 24 hours from the initial connection time. I voluntarily consent to
participate in the research.
A button will appear here, enabling the user who agrees to start the study. This button will be labeled with
the message “I agree. Start the study.”

2

Pre-session Questionnaire
Instructions: Indicate appropriate response. These will be formatted using typical entry forms for web pages.
1. Please select your gender: Female, Male, Prefer not to say
2. Please enter your age in years: (enter number)
OR
Please check the range which includes your age: 18-24, 25-29, 30-34, 35-39, 40-44, 45-49, 50-54, 55-59, 60 or
over, Prefer not to say
3. Please indicate your highest educational degree completed: Less than high school, High school diploma, Associate’s, Bachelor’s, Master’s, Professional Degree (MD, JD, MBA, et al.), Doctorate or higher, Prefer not to
say
4. Please indicate your native language or enter “n/a” if you prefer not to answer: (text entry)
5. If English is not your native language, how many years have you been speaking, reading, or writing in English?
If you prefer not to answer, please enter 0: (enter number)

3

Visualization Survey
Welcome and Introduction
Thank you for participating in our research!
This eye-tracking study involves a series of relatively simple line and bar graphs, a set of pictures, and a reading
assignment. The graphs show values for categorical data, the pictures are a random set of images, and the reading
material is excerpted from a short story. Your main tasks are to 1) examine what the graphs show 2) look at the
pictures, 3) read the story, which is presented as a series of installments, 4) confirm or challenge queries about the
data shown in each graph, and 5) confirm or challenge queries about the story.
The study will proceed as follows:
First, a “source” graph will appear. When you have finished looking at it, you will click a button labeled “Continue,”
and a picture will appear, then disappear after a few seconds. When you click “Continue” again, a second picture
will appear, then disappear. When you click “Continue” after the second picture, an installment of the story will
appear. After you read the installment, you will click “Continue” again, and a “query” about the graph will appear.
The query will also require reading a graph. It will be related to the source graph you saw four clicks back. Some of
the arbitrary features of the source graph - its color(s) and/or font(s), for instance - may look a bit different, but you
can ignore these changes. In many cases, the query will show only some of the information from the source graph.
More importantly, the information that is shown in the query may not agree with the gist of the corresponding
information in the source. If it does agree, you will confirm this by selecting a choice labeled “Stated.” If you
think it misrepresents the information it corresponds to in the source, you will challenge the query by selecting the
alternative choice labeled “Not Stated.” After you make your selection, you will click the “Continue” button. Then,
a written query about the reading material will appear. Somewhat like the graphical query, this will be a sentence
that is taken or derived from what you read. If it agrees with the gist of a sentence you read, you will select “Stated,”
and if it does not agree, you will select “Not Stated.” After clicking the “Continue” button on this screen, a new
graph/pictures/reading/graph-query/reading-query cycle will begin. You will continue in this fashion until this part
of the study is complete. You will then move on to a brief final questionnaire to finish the study.
About the queries
Both the graph queries and the sentence queries are based on a technique for verifying information that was developed
for educators in the late 1970s.
Queries in this technique fall into four categories: 1) “original” information, 2) “paraphrased” information, 3)
information that has a “meaning change,” and 4) “distractor” information that was not in the source material.
Queries in categories 1 and 2 (original and paraphrased information) are “Stated” queries because the information
they show agrees with the meaning of the information they are based on.
Conversely, queries in categories 3 and 4 (information whose meaning has been changed and information that was
not shown before) show information that was “Not Stated,” because they misrepresent original information or add
new information that was not part of the original material.
A written example
Here is how each query category works using a short text as the source information.
First, read the following fable.
The proud oak always claimed that he was much wiser than the humble reeds around him. He often
boasted, “I don’t bend my head in fear every time the wind roars.” But that very night a terrible storm
arose, and the oak was uprooted. The reeds murmured in sorrow and thought, “It is far better to bend
than it is to break.”
Now, read the following query, which falls into the “original” category.
4

The proud oak always claimed that he was much wiser than the humble reeds around him.
The information in this query is clearly “Stated” because it simply quotes one of the sentences in the fable.
Next, look at a query that falls into the “paraphrase” category.
He habitually bragged, “I don’t bow my crown in fright every time the wind howls.”
Although different words are used, the gist of this query is the same as the gist of the second sentence in the fable.
Consequently this query is also “Stated” because it corresponds to the intended meaning of the sentence it was
derived from in the fable, and most importantly, it does not misrepresent that meaning.
Now, look at a query that falls into the “meaning change” category.
But that very afternoon a terrible storm arose, and the oak was uprooted.
It is easy to see that this query corresponds to the third sentence in the fable because the wording is almost identical.
However, one word has been changed. Instead of reading, ”...that very night...,” the query reads, ”...that very
afternoon....” Changing “night” to “afternoon” factually contradicts the informational meaning of the corresponding
source material. Because of this contradiction, the query represents information that is “Not Stated.” To be clear,
note that the “broader” meaning of the fable is not altered by changing this one word. However, the meaning
change you must consider for a given query is the meaning of the source information it corresponds to. Here, that
information is the sentence the query is derived from.
Last, look at a query that falls into the “distractor” category.
Many reeds were crushed and the dying oak groaned, “What a broken fool I am.”
If you understood the fable as it is written, it should be easy to see that this last query is also information that is
“Not Stated.” With a little inspection, it is possible to discern that the query corresponds to the fable’s final sentence
- it is structured in much the same way and it ends with a pithy moral thought. But it should also be completely
clear that none of the information in this distractor was explicitly declared in the source material. Queries in the
distractor category are somewhat like queries in the meaning change category because they are intended to be on
topic and to add to, and even alter, the subject matter. However, they are much less subtle than one-word meaning
changes and, consequently, they should be easier to spot if you have internalized the source information.

On the next page of this introduction, we describe the sorts of graphic materials you will see in this study. We also
present a parallel example of graph queries.
When you are ready to go on, click the Continue button below.

5

Visualization Survey
Graphs
All of the “source” graphs in this study are relatively simple plots that show values for some sort of categorical data.
You will look at both line graphs and bar graphs. Some of the bar graphs use horizontal bars and others use vertical
bars. “Categorical data” simply means that the information in the graph falls into labeled categories. All of the
data in this study is fictional.
An example of the kind of graph you will see in the study is shown immediately below. Other graphs you will see
are a bit more complex, but many are simpler, and some are much simpler. Additionally, many of the graphs in
the study will have varying features, namely, the parts of the graph that help you to identify what the data is. For
instance, some may not have as many labels.
Perhaps the most important things to note in the example below are the data that are shown. Each bar represents
a specific datum and its length represents the datum’s specific value. The categorical identity of each datum is
collectively determined by referring to the graph’s labels, namely its title, its legend, the category names that are
given on the left in this example, and the label for the scale at the bottom. Similarly, the value of each datum and
the nature of the scale each value corresponds to (what the scale is, what portion of the scale is relevant, and what
level of granularity is depicted by the scale) is specified by reference to the graph’s scale and the scale’s label.

More to the point, there are four data in this graph. Each datum can (and should) be thought of as an independent
or standalone piece of information, but the graph has also been organized to convey a small number of relationships
that carry descriptive meaning. Collectively, the orange bars, which represent “Outbound” calls, are shorter than
the blue bars, which represent “Inbound” calls. Similarly, the categorical pairings of orange and blue bars allow the
values for each type of call to be descriptively compared across the “Corporate” and “Sales” categories - Corporate
had about the same number of Outbound calls but noticeably more Inbound calls in comparison to Sales.
To recap, each datum in a graph can be seen as an independent informational statement. However, in many cases,
the data in a graph is also organized to convey relational information. These two perspectives on what and how
information is shown in a graph concern the graph queries you will see in this study. Specifically, graph queries will
be much like the written queries you saw on the previous page, which were drawn from the sentences in a short
“source” passage of written material. That is, each graph query you see in this study will be a portion of the data
shown in the source graph. Because of the fact that individual data can be combined with other individual data to
make relational statements, this “portion” may be just one datum or some or all of the data shown in the source.

A graphical example
Remember that the information query technique in this study uses four categories: 1) “original” information, 2)
“paraphrased” information, 3) information that has a “meaning change”, and 4) “distractor” information that is not
in the source material.
Categories 1 and 2 correspond to information that is “Stated” in the source because these queries “agree” with the
6

intended meaning of the information they are based on. Conversely, categories 3 and 4 correspond to information
that is “Not Stated” in the source because they misrepresent source information or add new information that is not
explicitly present in the source material.
Here then, is how each query category works for graphs using a very simple bar graph as the information source. As
you read through the example, think about how it parallels the written queries you saw on the previous page.
First, look at the following “source” graph. Note that this graph only represents one datum. Unlike the written
example with queries based on successive sentences, each of the example queries that follow below will be derived
from this one datum.

Now, look at the following query, which falls into the “Original” category.

The information in this query is clearly “Stated” because it simply quotes the only datum in the source graph. You
can probably see that the query graph has been vertically squashed, but the important point is that the value and
the labeling that identifies the datum are unchanged.
Next, look at a query that falls into the “Paraphrase” category.

Note that this query and the source graph also do not look quite the same. Here, the graph and the bar are vertically
stretched, but again, these visual changes are only superficial. If you look at the value of the datum, you can see that
it is unchanged from its value in the source graph. What is not superficial in the query is the rewording of several
of the graph’s labels. For instance, the title has been changed to “Outgoing Calls by Division” (the source uses
“Outbound” and “Department”). Also, the category label has changed to “Law” (from “Legal”). The most subtle
change may be the use of numbers followed by a “k” along the bottom of the graph to indicate that the scale is in
thousands. Arguably, though, none of these word changes alters the gist of the datum in the source graph. The value
7

of the datum shown in the query is unchanged and its identity, which could be summarized as, “thousands of calls
made by the law division,” is a close paraphrase of the source material it was derived from. Consequently this query
is also “Stated,” and most importantly, it does not misrepresent the intended meaning of the original information.
Now, look at a query that falls into the “Meaning Change” category.

Like the original and paraphrase queries you have just seen, this query is also superficially different from the source
graph it is derived from. Essentially, the graph is smaller than the original, but other than this, it almost completely
matches the source. However, it is different in one key respect. Here, the value of the datum in the query is only a
little more than 10 thousand, whereas in the original, it is just over 20 thousand. Because of this one, intentionally
conspicuous change, the query misrepresents the datum it corresponds to in the source graph. Consequently, the
query clearly shows information that is “Not Stated.”
There are several other ways a single change could be made to misrepresent the datum this query corresponds to.
For example, instead of changing the datum’s value, its category label could be altered to contradict what the datum
represents in the source material - “Legal” could be changed to “Accounting,” for instance. Similarly, the label for
the horizontal axis could be altered to misrepresent the scale, the title could be altered to misrepresent the topic,
or the scale could be changed to misrepresent the value of the original length of the bar. As you proceed through
the study, be prepared to look for a single change in a given query that contradicts the meaning (or the value)
of corresponding data in the source graph. The pattern will always be that only one thing related to the original
meaning of the data is changed. When you spot a meaning change query, your response should be “Not Stated.”
Last, look at a query that falls into the “Distractor” category.

Once again, there are superficial differences between the “look” of this query and the source graph. The vertical
height of the graph is conspicuously reduced, the color of the bar is now mustard yellow, the bar itself is much
thinner, and the font used for all of the labels has changed. It should be plain however, that none of these “cosmetic”
changes interact with the value and the identity of the datum depicted in the query. In this particular instance, if
only these changes were made to the source graph, the statistical information it is intended to convey would still be
the same.
That said, there are three meaning-related differences that make this particular query a distractor and thus information that is “Not Stated.” First, the category of the datum depicted in the query is “Facilities;” in the source
graph, the category is “Legal.” Second, the length of the bar relative to the scale is noticeably shorter, and in fact,
the value shown is new. Additionally and third, the scale label now reads “hundreds” instead of “thousands.”
What is shown in the query is clearly not just a meaning change. It is an entirely new datum, whose identity and value,
while on topic, are not present in the source graph. Other distractors that combine multiple informational differences
can easily be imagined, providing they are on topic and represent new information. Like sentence distractors, graph
queries in the distractor category are much less subtle than queries in the meaning change category, wherein only
one element of the query contradicts the corresponding data in the source graph. Consequently, distractors should
8

be relatively easy to identify if you have studied the source information.

On the next page, we provide further clarification about the queries in the study and show an additional graphical
example.
When you are ready to continue, click the Continue button below.

9

Visualization Survey
An Important Clarification
The queries in this study are not intended to test your ability to remember precise details nor the accuracy of your
visual memory. Instead, they are designed to reflect your understanding of written and plotted information. The
idea is that when you read a paragraph or look at a graph, if you are able to “make sense” of the content, you will
likely be able to verify the gist of your understanding afterwards by respectively confirming or challenging queries
derived from the source material as “Stated” or “Not Stated.”
To be clear, in this study you are not expected to remember
• exact names, titles, numbers, times, or dates
• precise data values
• the order of unordered data or data series
• specific superficial visual details, such as font, layout, size, scale configuration, colors, and marks used for data
points
Additionally, the basic organization of source material will not be changed in any query. For sentences, this means
that the structure of a sentence you read in a source will not be swapped around as a “superficial” change in any of
the four query categories. Similarly for graphs, it means that a source graph’s use of axes and the type of plot that
is used (line or bar) will always be reflected in its queries.
This is not to say that you should simply skim through sources. However, there are no “trick” queries in the study.
Misspelled words, confusing or subtly different names, small inaccuracies, category reordering, changes in scale
endpoints, and so on will not be used as the basis for making any sentence or graph query “Not Stated.” Instead,
look for clear changes of meaning. “...bow my crown in fright...” is a reasonable paraphrase for “...bend my head
in fear...” but “afternoon” is not a substitute for “night.” “Outgoing Calls by Division” is a reasonable paraphrase
for “Outbound Calls by Department” but “Facilities” is clearly not the same category name as “Legal.” Similarly,
look for clear changes in numerical values. You are expected to make sense of a source graph’s scale and to note
the relative magnitude(s) and relationships of plotted data. But you are not expected to note very small differences
in the length of a bar or the position of a mark relative to their corresponding values in a source graph. However,
“...hundreds” are not “...thousands”, and a bit more than 10 thousand is not close to just over 20 thousand. In
summary, when names and/or values are changed to make a query represent information that is “Not Stated,” they
will be pointedly different from what is shown in the source material.

Another graphical example
Another example of a “source” graph is shown immediately below. A very simple “line” graph is used here to
acquaint you with queries for a different way of plotting categorical data. Note that this plot shows two data series.
Each series has only one datum, though, so there are no “lines” in this line graph, only “marks” for each datum. (If
there were more data in each series, lines would be used to connect the corresponding marks.) Note also that, like the
previous graphical example, there is only one category of data in this graph. In this plot, the category label appears
along the horizontal axis, and the vertical axis shows the scale that identifies the value of each datum. Unlike the
previous graphical example, the following queries are drawn from different combinations of the data shown below.

10

First, look at a query for the source graph that falls into the ”Original” category. Remember that queries in the
“Original” category represent information that is “Stated” in the source.

Only one of the two series in the source is shown in here, the one labeled “Basketball,” and there are several
superficial differences, most notably, the use of a different mark and color for the Basketball datum. Even so, the
datum’s value and identity are the same as the corresponding information in the source, so the query represents
“Stated” information.
Next, look at a ”Paraphrase” query. Queries in this category also correspond to information that is “Stated” in
the source graph.

Like the previous query, there are several superficial differences between this query and the source graph. One of
these differences - starting the scale on the vertical axis at 2 million dollars, as opposed to 0 in the source - might
seem to be an important change, but this has no effect on the meaning and value of the datum represented in the
query.
11

More conspicuously, all of the labels have been changed, but none of the paraphrased wording - the title, $ (mil),
NBA, and ’14 - contradicts the original labels’ respective meanings. An additional paraphrase involving the value of
the datum is also present. Although it may not be immediately apparent, the datum’s value has been rounded up
to $5 million, and thus the postion of its mark is very slightly higher than it is in the source graph.
Paraphrasing via rounding is a common practice in daily discourse involving numbers. We often say, for instance,
that an $18.65 meal cost “twenty bucks”, or that we got home “around 6,” when in fact we arrived at 6:12 pm.
More to the point, conceptual rounding - essentially, “paraphrasing” data values - is considered to be an important
part of grasping plotted information and being able to use it in broader contexts. Here, using a small amount of
rounding as a visual paraphrase, on and at the same scale as the source graph, illustrates the clarification about
numbers and precision that is stressed above. None of the modest meaning-related variations that are used in this
query can be construed as a clear meaning change. The information represented here is a reasonable paraphrase of
the corresponding datum in the source graph and thus should be identified as “Stated” information.
Now, look at a query that features a ”Meaning Change”.

Unlike the “original” and “paraphrase” queries, which represented the “Basketball” series in the source graph, this
query appears to represent the other series - and it does in every respect except for its label, which is “Baseball” in
the source graph and is “Soccer” here. Since only one meaning-related element is altered and there is no mention of
“Soccer” in the source graph, this query represents information that is “Not Stated.”
Last, look at a query that falls into the ”Distractor” category.

Unlike any of the queries you have seen up to this point, data from two series are shown here. Essentially, this reflects
what is shown in the source graph. Although a given datum, with its accompanying labels and scale, can be thought
of individually as a simple statement of information, data from different series can also be combined in various ways
12

to form informational statements that show more than just the identities and values of individual statistics (think
of the “Inbound vs. Outbound Calls” plot that you saw earlier, which had two series and four data points). A
combined statement can be made with the limited amount of data that is shown in the source graph in this example,
and correspondingly, that is what is done here. Specifically, the source graph and the plot in this query both show a
numerical relationship between two series’ marks in a single category. Additionally, in both graphs, one series’ value
is greater than the other’s. The dispositions of such relationships are informational statements in and of themselves.
More to the point, our ability to make sense of such representations and other sorts of visual comparisons among
data is a primary motivation for using graphical plots to convey data-driven information.
The query shown here, then, is much like the source graph, in the sense that it has two series and the same title,
and it uses exactly the same vertical scale. It does show new information, though. The category label at the bottom
of the query reads “2015,” instead of “2014,” and the two series are respectively named “NFL” and “NHL,” - not
“Basketball” and “Baseball.” Additionally, the data values in the query are noticeably lower and closer together
that the marks showing $4.9 and $3.8 million in the source graph - the NFL datum is at $2.6 million and the NHL
mark is at $2.1 million. (Remember, estimating graphed numbers to yourself, rather than trying to determine the
precise values that are plotted, is sufficient for the purposes of this study). So, while this query is derived from a
relational pattern of information that is shown in the source graph, several meaning-related differences between the
two add up to new information – information that is “Not Stated” in the source graph.

Next, you will have an opportunity to practice this information verification technique.
Also, please feel free to take a short break at this point if you need to or would simply like to pause for a moment.
When you are ready to continue, click the Continue button below.

13

Visualization Survey
Summary
To summarize, here is the task for the study. You will see
• a source graph, which you should study for at least 30 seconds so that you can answer a query about it later
• an image, which you should look at; it will disappear after a few seconds
• another image, which you should also look at; it too will disappear after a few seconds
• a short excerpt from a story, which you should read so that you can answer a query about it later
• the query about the graph
• the query about the story excerpt
You will repeat this cycle sixteen times during the study.
The first four times you do this will allow you to practice responding to this type of query. After you submit your
responses to both graph and text queries in each practice cycle, you will be given the correct answers.
After the practice cycles, you will no longer be given the correct answers.
When you finish, there will be a short questionnaire and a debriefing.
When you are ready to continue, click the Continue button below.

Note to proposal reviewers: The following questions represent the entire corpus of materials; the order of graph
practice queries is set: 0BP, 0LP, 1LP, 1BP. The order of the remaining graph queries is counterbalanced. The
selection of distraction images is counterbalanced. The order of prose queries is in the order given here. The
selection of which of the four query types for the questions after the practice is counterbalanced.

14

Question 0BP (Practice)
Please study the following graph. After 30 seconds, you may continue. After three minutes, the graph will disappear.

Each subject would be shown later the following graph and asked the question: Is the information you see in this
query graph information that was Stated or Not Stated in the source graph you most recently saw?

15

Question 1BP (Practice)
Please study the following graph. After 30 seconds, you may continue. After three minutes, the graph will disappear.

Each subject would be shown later the following graph and asked the question: Is the information you see in this
query graph information that was Stated or Not Stated in the source graph you most recently saw?

16

Question 1B
Please study the following graph. After 30 seconds, you may continue. After three minutes, the graph will disappear.

Each subject would be shown later exactly one of the following graphs and asked the question: Is the information you
see in this query graph information that was Stated or Not Stated in the source graph you most recently saw?

17

Question 2B
Please study the following graph. After 30 seconds, you may continue. After three minutes, the graph will disappear.

Each subject would be shown later exactly one of the following graphs and asked the question: Is the information you
see in this query graph information that was Stated or Not Stated in the source graph you most recently saw?

18

Question 3B
Please study the following graph. After 30 seconds, you may continue. After three minutes, the graph will disappear.

Each subject would be shown later exactly one of the following graphs and asked the question: Is the information you
see in this query graph information that was Stated or Not Stated in the source graph you most recently saw?

19

Question 4B
Please study the following graph. After 30 seconds, you may continue. After three minutes, the graph will disappear.

Each subject would be shown later exactly one of the following graphs and asked the question: Is the information you
see in this query graph information that was Stated or Not Stated in the source graph you most recently saw?

20

Question 5B
Please study the following graph. After 30 seconds, you may continue. After three minutes, the graph will disappear.

Each subject would be shown later exactly one of the following graphs and asked the question: Is the information you
see in this query graph information that was Stated or Not Stated in the source graph you most recently saw?

21

Question 6B
Please study the following graph. After 30 seconds, you may continue. After three minutes, the graph will disappear.

Each subject would be shown later exactly one of the following graphs and asked the question: Is the information you
see in this query graph information that was Stated or Not Stated in the source graph you most recently saw?

22

Question 0LP (Practice)
Please study the following graph. After 30 seconds, you may continue. After three minutes, the graph will disappear.

Each subject would be shown later the following graph and asked the question: Is the information you see in this
query graph information that was Stated or Not Stated in the source graph you most recently saw?

23

Question 1LP (Practice)
Please study the following graph. After 30 seconds, you may continue. After three minutes, the graph will disappear.

Each subject would be shown later the following graph and asked the question: Is the information you see in this
query graph information that was Stated or Not Stated in the source graph you most recently saw?

24

Question 1L
Please study the following graph. After 30 seconds, you may continue. After three minutes, the graph will disappear.

Each subject would be shown later exactly one of the following graphs and asked the question: Is the information you
see in this query graph information that was Stated or Not Stated in the source graph you most recently saw?

25

Question 2L
Please study the following graph. After 30 seconds, you may continue. After three minutes, the graph will disappear.

Each subject would be shown later exactly one of the following graphs and asked the question: Is the information you
see in this query graph information that was Stated or Not Stated in the source graph you most recently saw?

26

Question 3L
Please study the following graph. After 30 seconds, you may continue. After three minutes, the graph will disappear.

Each subject would be shown later exactly one of the following graphs and asked the question: Is the information you
see in this query graph information that was Stated or Not Stated in the source graph you most recently saw?

27

Question 4L
Please study the following graph. After 30 seconds, you may continue. After three minutes, the graph will disappear.

Each subject would be shown later exactly one of the following graphs and asked the question: Is the information you
see in this query graph information that was Stated or Not Stated in the source graph you most recently saw?

28

Question 5L
Please study the following graph. After 30 seconds, you may continue. After three minutes, the graph will disappear.

Each subject would be shown later exactly one of the following graphs and asked the question: Is the information you
see in this query graph information that was Stated or Not Stated in the source graph you most recently saw?

29

Question 6L
Please study the following graph. After 30 seconds, you may continue. After three minutes, the graph will disappear.

Each subject would be shown later exactly one of the following graphs and asked the question: Is the information you
see in this query graph information that was Stated or Not Stated in the source graph you most recently saw?

30

Prose Baseline Task
The next 16 pages give the prose sources and queries that users may see (depending on the counterbalancing scheme’s
assignment of queries to users).

Question 1P (Practice)
Please read the following. We will ask a query about this prose shortly.
Buffalo Jones was born in 1846 on the Illinois prairie, and he devoted practically all of his life to the
pursuit of wild animals. It has been a pursuit which owed its unflagging energy and indomitable purpose
to a singular passion, almost an obsession, to capture alive, not to kill. He has caught and broken the
will of every well-known wild beast native to western North America. Killing was repulsive to him. He
even disliked the sight of a sporting rifle, though for years necessity compelled him to earn his livelihood
by supplying the meat of buffalo to the caravans crossing the plains. At last, seeing that the extinction of
the noble beasts was inevitable, he smashed his rifle over a wagon wheel and vowed to save the species.
For ten years he labored, pursuing, capturing and taming buffalo, for which the West gave him fame, and
the name Preserver of the American Bison.
As civilization encroached upon the plains, Buffalo Jones ranged slowly westward; and to-day an isolated
desert-bound plateau on the north rim of the Grand Canyon of Arizona is his home. There his buffalo
browse with the mustang and deer, and are as free as ever they were on the rolling plains.
Each subject would be shown later exactly one of the following sentences and asked the question: Please
read the following and consider the information in this sentence. Is it information that was Stated or Not Stated
in the source passage you most recently saw?
• At last, seeing that the extinction of the noble beasts was inevitable, he smashed his rifle over a wagon wheel
and vowed to save the species.
• Finally, sensing that the destruction of the grand animals was certain, he broke his rifle over a wagon wheel
and promised to rescue the species.
• At last, seeing that the domestication of the noble beasts was inevitable, he smashed his rifle over a wagon
wheel and vowed to save the species.
• Upon realizing that civilization was eliminating the need for plainsmen, he traded his rifle for a bridle and
decided to drive stagecoaches.

31

Question 2P (Practice)
Please read the following. We will ask a query about this prose shortly.
In the spring of 1907, I was the fortunate companion of the old plainsman on a trip across the desert, and
a hunt in that wonderful country of yellow crags, deep canyons and giant pines. I want to tell about it.
I want to show the color and beauty of those painted cliffs and the long, brown-matted bluebell-dotted
aisles in the grand forests; I want to give a suggestion of the tang of the dry, cool air; and particularly
I want to throw a little light upon the life and nature of that strange character and remarkable man,
Buffalo Jones.
Happily in remembrance a writer can live over his experiences, and see once more the moon-blanched
silver mountain peaks against the dark blue sky; hear the lonely sigh of the night wind through the pines;
feel the dance of wild expectation in the quivering pulse; the stir, the thrill, the joy of hard action in
perilous moments; the mystery of man’s yearning for the unattainable.
I came to see the wonder, the tragedy of their lives, and to write about them. It has been my destiny to
live for a while in the fast-fading wild environment which produced these great men with the last of the
great plainsmen.
Each subject would be shown later exactly one of the following sentences and asked the question: Please
read the following and consider the information in this sentence. Is it information that was Stated or Not Stated
in the source passage you most recently saw?
• In the spring of 1907, I was the fortunate companion of the old plainsman on a trip across the desert.
• In the spring of 1907, I was the lucky partner of the aging plainsman on a journey through the arid wilderness.
• In the spring of 1907, I was the fortunate companion of the old plainsman on a trip across the mountains.
• In the spring of 1907, I became the intrepid biographer of the renowned plainsman as he recalled his wilderness
travels.

32

Question 3P (Practice)
Please read the following. We will ask a query about this prose shortly.
Jones, erect, rugged, brawny, stood in the full light of the campfire. He had a dark, bronzed, inscrutable
face; a stern mouth and square jaw, keen eyes, half-closed from years of searching the wide plains; and
deep furrows wrinkling his cheeks. A strange stillness enfolded his feature, the tranquility earned from a
long life of adventure.
Where I had once been interested in the old buffalo hunter, I was now fascinated. And now I was with
him in the desert and seeing him as he was, a simple, quiet man, who fitted the mountains and the
silences, and the long reaches of distance.
Once he had told me how, when a mere child, he had hazarded limb and neck to capture a fox squirrel,
how he had held on to the vicious little animal, though it bit his hand through; how he had never learned
to play the games of boyhood; that when the youths of the little Illinois village were at play, he roamed
the prairies, or the rolling, wooded hills, or watched a gopher hole. That boy was father of the man: for
sixty years an enduring passion for dominion over wild animals had possessed him, and made his life an
endless pursuit.
Each subject would be shown later exactly one of the following sentences and asked the question: Please
read the following and consider the information in this sentence. Is it information that was Stated or Not Stated
in the source passage you most recently saw?
• A strange stillness enfolded his feature, the tranquility earned from a long life of adventure.
• An odd serenity enveloped his face, the peaceful reward of a long life of quests.
• A strange stillness enfolded his feature, the fear earned from a long life of adventure.
• A steely glare crossed his face; the suspicion aroused from multiple battles.

33

Question 4P (Practice)
Please read the following. We will ask a query about this prose shortly.
Suddenly the hounds bristled, and old Moze, a surly and aggressive dog, rose and barked at some real or
imaginary desert prowler. A sharp command from Jones made Moze crouch down, and the other hounds
cowered close together.
“Better tie up the dogs,” suggested Jones. “Like as not coyotes run down here from the hills.” The
hounds were my especial delight. But Jones regarded them with considerable contempt. When all was
said, this was no small wonder, for that quintet of long-eared canines would have tried the patience of a
saint. Old Moze was a Missouri hound that Jones had procured in that State of uncertain qualities; and
the dog had grown old over coon-trails. He was black and white, grizzled and battle-scarred; and if ever
a dog had an evil eye, Moze was that dog. He had a way of wagging his tail–an indeterminate, equivocal
sort of wag, as if he realized his ugliness and knew he stood little chance of making friends, but was still
hopeful and willing. As for me, the first time he manifested this evidence of a good heart under a rough
coat, he won me forever.
Each subject would be shown later exactly one of the following sentences and asked the question: Please
read the following and consider the information in this sentence. Is it information that was Stated or Not Stated
in the source passage you most recently saw?
• Suddenly the hounds bristled, and old Moze, a surly and aggressive dog, rose and barked at some real or
imaginary desert prowler.
• Suddenly the dogs fretted, and old Moze, a sullen and fierce hound, got up and woofed at some true or imagined
desert predator.
• Suddenly the hounds bristled, and old Moze, a calm and docile dog, rose and barked at some real or imaginary
desert prowler.
• Suddenly the horses snorted, and old Moze, a normally deferential dog, confronted them and barked at the
team leader.

34

Question 5P
Please read the following. We will ask a query about this prose shortly.
Moze’s crowning feat, however, made even the stoical Jones open his mouth in amazement. We had taken
Moze to the El Tovar at the Grand Canyon, and finding it impossible to get over to the north rim, we
left him with one of Jones’s men, called Rust, who was working on the Canyon trail. Rust’s instructions
were to bring Moze to Flagstaff in two weeks. He brought the dog a little ahead of time, and roared his
appreciation of the relief it to get the responsibility off his hands. And he related many strange things,
most striking of which was how Moze had broken his chain and plunged into the raging Colorado River,
and tried to swim it just above the terrible Sockdolager Rapids. Rust and his fellow-workmen watched
the dog disappear in the yellow, wrestling, turbulent whirl of waters, and had heard his knell in the
booming roar of the falls. Nothing but a fish could live in that current; nothing but a bird could scale
those perpendicular marble walls. That night, however, when the men crossed on the tramway, Moze
met them with a wag of his tail. He had crossed the river, and he had come back!
Each subject would be shown later exactly one of the following sentences and asked the question: Please
read the following and consider the information in this sentence. Is it information that was Stated or Not Stated
in the source passage you most recently saw?
• Moze’s crowning feat, however, made even the stoical Jones open his mouth in amazement.
• Moze’s tour de force, however, caused even the unemotional Jones to drop his jaw in astonishment.
• Moze’s crowning feat, however, made even the stoical Jones open his mouth in rage.
• Moze’s latest trick, though, caused the plainsman to roar with laughter.

35

Question 6P
Please read the following. We will ask a query about this prose shortly.
To the four reddish-brown, high-framed bloodhounds I had given the names of Don, Tige, Jude and
Ranger; and by dint of persuasion, had succeeded in establishing some kind of family relation between
them and Moze. This night I tied up the bloodhounds, after bathing and salving their sore feet; and I
left Moze free, for he grew fretful and surly under restraint.
Jones was crawling into his bed. I walked a little way from the dying fire, and faced the north, where the
desert stretched, mysterious and illimitable. How solemn and still it was! I drew in a great breath of the
cold air, and thrilled with a nameless sensation. Something was there, away to the northward; it called
to me from out of the dark and gloom; I was going to meet it.
I lay down to sleep with the great blue expanse open to my eyes. The stars were very large, and
wonderfully bright, yet they seemed so much farther off than I had ever seen them. The wind softly sifted
the sand. I hearkened to the tinkle of the cowbells on the hobbled horses. The last thing I remembered
was old Moze creeping close to my side, seeking the warmth of my body.
Each subject would be shown later exactly one of the following sentences and asked the question: Please
read the following and consider the information in this sentence. Is it information that was Stated or Not Stated
in the source passage you most recently saw?
• The last thing I remembered was old Moze creeping close to my side, seeking the warmth of my body.
• The final action I recall was old Moze crawling next to me, searching for warmth from my body.
• The last thing I remembered was old Moze creeping close to the fire, seeking the warmth of the flames.
• One thing I noticed was old Moze patrolling around the camp, guarding against thieves in the night.

36

Question 7P
Please read the following. We will ask a query about this prose shortly.
When I awakened, a long, pale line showed out of the dun-colored clouds in the east. It slowly lengthened,
and tinged to red. Then the morning broke, and the slopes of snow on the San Francisco peaks behind
us glowed a delicate pink.
All morning we made good time, and as we descended into the desert, the air became warmer, the scrubby
cedar growth began to fail, and the bunches of sage were few and far between. I turned often to gaze back
at the San Francisco peaks. The snowcapped tips glistened and grew higher, and stood out in startling
relief. Someone said they could be seen two hundred miles across the desert, and were a landmark and
a fascination to all travelers thitherward. I never raised my eyes to the north that I did not draw my
breath quickly and grow chill with awe and bewilderment with the marvel of the desert. The scaly red
ground descended gradually; bare red knolls, like waves, rolled away northward; black buttes reared their
flat heads; long ranges of sand flowed between them like streams, and all sloped away to merge into gray,
shadowy obscurity, into wild and desolate, dreamy and misty nothingness.
Each subject would be shown later exactly one of the following sentences and asked the question: Please
read the following and consider the information in this sentence. Is it information that was Stated or Not Stated
in the source passage you most recently saw?
• Then the morning broke, and the slopes of snow on the San Francisco peaks behind us glowed a delicate pink.
• Then the dawn arose, and the banks of snow on the San Francisco summits behind us gleamed an exquisite
pink.
• Then the morning broke, and the slopes of snow on the San Francisco peaks ahead of us glowed a delicate pink.
• Towards the evening, the puffs of clouds over the San Francisco peaks behind us hinted of childhood summers.

37

Question 8P
Please read the following. We will ask a query about this prose shortly.
That afternoon, a hot wind blew in my face, carrying fine sand that cut and blinded. It filled my throat,
sending me to the water cask till I was ashamed. When I fell into my bed at night, I never turned. The
next day was hotter; the wind blew harder; the sand stung sharper.
About noon the following day, the horses whinnied, and the mules roused out of their tardy gait. “They
smell water,” said Emmett. And despite the heat, and the sand in my nostrils, I smelled it, too. The
dogs, poor foot-sore fellows, trotted on ahead down the trail. A few more miles of hot sand and gravel
and red stone brought us around a low mesa to the Little Colorado.
It was a wide stream of swiftly running, reddish-muddy water. In the channel, cut by floods, little streams
trickled and meandered in all directions. The dogs lolled in the water; the horses and mules tried to run
in, but were restrained; the men drank, and bathed their faces. This was one of the two drinks I would
get on the desert, so I availed myself heartily of the opportunity. The water was full of sand, but cold
and gratefully thirst-quenching.
Each subject would be shown later exactly one of the following sentences and asked the question: Please
read the following and consider the information in this sentence. Is it information that was Stated or Not Stated
in the source passage you most recently saw?
• It was a wide stream of swiftly running, reddish-muddy water.
• It was a broad stream of rapidly moving, red-tinged silty water.
• It was a wide stream of swiftly running, clear blue water.
• There was a sparse field of mostly brown, dying grass.

38

Question 9P
Please read the following. We will ask a query about this prose shortly.
“Doesn’t look bad, eh?” queried Emmett, who read my thought. “You’d be surprised to learn how many
men, horses, sheep and wagons are buried under that quicksand.”
The secret was out, and I wondered no more. At once the stream and wet bars of sand took on a different
color. I removed my boots, and waded out to a little bar. The sand seemed quite firm, but water oozed
out around my feet; and when I stepped, the whole bar shook like jelly. I pushed my foot through the
crust, and the cold, wet sand took hold, and tried to suck me down.
“How can you ford this stream with horses?” I asked Emmett. “We must take our chances,” replied he.
“We’ll hitch two teams to one wagon, and run the horses. I’ve forded here at worse stages than this. Once
a team got stuck, and I had to leave it; another time the water was high, and washed me downstream.”
Emmett sent his son into the stream on a mule. The rider lashed his mount, and plunging, splashing,
crossed at a pace near a gallop. He returned in the same manner, and reported one bad place near the
other side.
Each subject would be shown later exactly one of the following sentences and asked the question: Please
read the following and consider the information in this sentence. Is it information that was Stated or Not Stated
in the source passage you most recently saw?
• The sand seemed quite firm, but water oozed out around my feet.
• The sand looked quite solid, but water seeped out all around my feet.
• The logs seemed quite firm, but water oozed out around my feet.
• The river bed felt quite smooth, and the water cooled my feet.

39

Question 10P
Please read the following. We will ask a query about this prose shortly.
Jones and I got on the first wagon and tried to coax up the dogs, but they would not come. Emmett had
to lash the four horses to start them. The wagon bowled into the water with a tremendous splash. We
were wet through before we had gone twenty feet. The plunging horses were lost in yellow spray. Jones
yelled in my ear, but I could not hear what he said. Once, the wagon wheels struck a stone or log, almost
lurching us overboard. A muddy splash blinded me. I cried out in my excitement, and punched Jones
in the back. Next moment, the keen exhilaration of the ride gave way to horror. We seemed to drag,
and almost stop. Someone roared: “Horse down!” One instant of painful suspense, in which imagination
pictured another tragedy added to the record of this deceitful river–a moment filled with intense feeling,
and sensation of splash, and yell, and fury of action; then the three able horses dragged their comrade
out of the quicksand. He regained his feet, and plunged on. Spurred by fear, the horses increased their
efforts, and amid clouds of spray, galloped the remaining distance to the other side.
Each subject would be shown later exactly one of the following sentences and asked the question: Please
read the following and consider the information in this sentence. Is it information that was Stated or Not Stated
in the source passage you most recently saw?
• Once, the wagon wheels struck a stone or log, almost lurching us overboard.
• One time, the wagon wheels hit a rock or wood, nearly tossing us over the side.
• Once, the wagon wheels struck a stone or log, immediately lurching us overboard.
• Soon, the thinning path became a wash or gully, briefly slowing us down.

40

Question 11P
Please read the following. We will ask a query about this prose shortly.
Jones looked disgusted. Like all plainsmen, he hated water. Emmett and his men calmly unhitched. No
trace of alarm, or even of excitement showed in their bronzed faces.
“We made that fine and easy,” remarked Emmett.
So I sat down and wondered what Jones and Emmett, and these men would consider really hazardous.
I began to have a feeling that I would find out; that experience for me was but in its infancy; that far
across the desert the something which had called me would show hard, keen, perilous life. And I began
to think of reserve powers of fortitude and endurance. The other wagons were brought across without
mishap; but the dogs did not come with them. Jones called and called. The dogs howled and howled.
Finally I waded out over the wet bars and little streams to a point several hundred yards nearer the dogs.
Moze was lying down, but the others were whining and howling in a state of great perturbation. I called
and called. They answered, and even ran into the water, but did not start across. I yelled, losing my
patience.
“You’ve already swum the Big Colorado, and this is only a brook. Come on!”
Each subject would be shown later exactly one of the following sentences and asked the question: Please
read the following and consider the information in this sentence. Is it information that was Stated or Not Stated
in the source passage you most recently saw?
• Moze was lying down, but the others were whining and howling in a state of great perturbation.
• Moze was stretched out on the ground, but the others were whimpering and wailing in a state of great agitation.
• Moze was coming across, but the others were whining and howling in a state of great perturbation.
• Moze was wandering off, while the men were laughing and grinning on the shore in high spirits.

41

Question 12P
Please read the following. We will ask a query about this prose shortly.
This appeal evidently touched Moze, for he barked, and plunged in. He made the water fly, and when
carried off his feet, breasted the current with energy and power. He made shore almost even with me, and
wagged his tail. Not to be outdone, Jude, Tige and Don followed suit, and first one and then another was
swept off his feet and carried downstream. They landed below me. This left Ranger, the pup, alone on
the other shore. Of all the pitiful yelps ever uttered by a frightened and lonely puppy, his were the most
forlorn I had ever heard. I kept calling, and at last, hoping to make him come by a show of indifference,
I started away. This broke his heart. Putting up his head, he let out a long, melancholy wail, which for
aught I knew might have been a prayer, and then consigned himself to the yellow current. Ranger swam
like a boy learning. He seemed to be afraid to get wet. His forefeet were continually pawing the air in
front of his nose. He drifted far below, stranded on an island, crossed it, and plunged in again, to make
shore almost out of my sight.
Each subject would be shown later exactly one of the following sentences and asked the question: Please
read the following and consider the information in this sentence. Is it information that was Stated or Not Stated
in the source passage you most recently saw?
• I kept calling, and at last, hoping to make him come by a show of indifference, I started away.
• I continued yelling, and as a last resort, wishing to induce him to come by a show of complacence, I walked
away.
• I kept calling, and at last, hoping to make him come by a show of anger, I started away.
• I kept stomping, and at last, deciding to leave him alone in a show of anger, I stormed away.

42

Question 13P
Please read the following. We will ask a query about this prose shortly.
Imagination had pictured the desert for me as a vast, sandy plain, flat and monotonous. Reality showed
me desolate mountains gleaming bare in the sun, long lines of red bluffs, white sand dunes, and hills of
blue clay, areas of level ground–in all, a many-hued, boundless world in itself, wonderful and beautiful,
fading all around into the purple haze of deceiving distance.
Thin, clear, sweet, dry, the desert air carried a languor, a dreaminess, tidings of far-off things, and an
enthralling promise. The fragrance of flowers, the beauty and grace of women, the sweetness of music,
the mystery of life–all seemed to float on that promise. It was the air breathed by the lotus-eaters, when
they dreamed, and wandered no more.
Beyond the Little Colorado, we began to climb again. The sand was thick; the horses labored; the drivers
shielded their faces. The dogs began to limp and lag. Ranger had to be taken into a wagon; and then,
one by one, all of the other dogs, except Moze. He refused to ride, and trotted along with his head down.
Far to the front the pink cliffs, the ragged mesas, the dark, volcanic spurs of the Big Colorado stood up
and beckoned us onward.
Each subject would be shown later exactly one of the following sentences and asked the question: Please
read the following and consider the information in this sentence. Is it information that was Stated or Not Stated
in the source passage you most recently saw?
• Ranger had to be taken into a wagon; and then, one by one, all of the other dogs, except Moze.
• The pup had to be put in a wagon; and then, one at a time, all the other hounds, save for Moze.
• Ranger had to be taken into a wagon; and then, one by one, all of the other dogs, even Moze.
• Ranger had to be given water; and then, over time, some of the other dogs begged, but not Moze.

43

Question 14P
Please read the following. We will ask a query about this prose shortly.
We camped near another water hole, located in a deep, yellow-colored gorge, crumbling to pieces, a ruin
of rock, and silent as the grave. In the bottom of the canyon was a pool of water, covered with green scum.
My thirst was effectually quenched by the mere sight of it. I slept poorly, and lay for hours watching the
great stars. The silence was painfully oppressive. If Jones had not begun to give a respectable imitation
of the exhaust pipe on a steamboat, I should have been compelled to shout aloud, or get up; but this
snoring would have dispelled anything. The morning came gray and cheerless. I got up stiff and sore,
with a tongue like a rope.
All day long we ran the gauntlet of the hot, flying sand. Night came again, a cold, windy night. I slept
well until a mule stepped on my bed, which was conducive to restlessness. At dawn, cold, gray clouds
tried to blot out the rosy east. I could hardly get up. My lips were cracked; my tongue swollen to twice
its natural size; my eyes smarted and burned. The barrels and kegs of water were exhausted.
Each subject would be shown later exactly one of the following sentences and asked the question: Please
read the following and consider the information in this sentence. Is it information that was Stated or Not Stated
in the source passage you most recently saw?
• In the bottom of the canyon was a pool of water, covered with green scum.
• In the floor of the ravine was a water hole, blanketed by green algae.
• In the lip of the canyon was a pool of water, covered with green scum.
• On the sides of the canyon were patches of green, fed by trickles of water.

44

Question 15P
Please read the following. We will ask a query about this prose shortly.
Only twice that day did I rouse to anything resembling enthusiasm. We came to a stretch of country
showing the wonderful diversity of the desert land. A long range of beautifully rounded clay stones
bordered the trail. So symmetrical were they that I imagined them works of sculptors. Light blue, dark
blue, clay blue, marine blue, cobalt blue–every shade of blue was there, but no other color. The other
time that I awoke to sensations from without was when we came to the top of a ridge. We had been
passing through red-lands. Jones called the place a strong, specific word which really was illustrative of
the heat amid those scaling red ridges. We came out where the red changed abruptly to gray. I seemed
always to see things first, and I cried out: “Look! here are a red lake and trees!”
“No, lad, not a lake,” said old Jim, smiling at me; “that’s what haunts the desert traveler. It’s only
mirage!”
So I awoke to the realization of that illusive thing, the mirage, a beautiful lie, false as stairs of sand. For
a long moment it lay there, smiling in the sun, a thing almost tangible; and then it faded.
Each subject would be shown later exactly one of the following sentences and asked the question: Please
read the following and consider the information in this sentence. Is it information that was Stated or Not Stated
in the source passage you most recently saw?
• Only twice that day did I rouse to anything resembling enthusiasm.
• Only two times that day did I stir in any way resembling excitement.
• Only twice that day did I rouse to anything resembling panic.
• Only one time on the journey did I feel anything resembling tranquility.

45

Question 16P
Please read the following. We will ask a query about this prose shortly.
I felt a sense of actual loss. So real had been the illusion that I could not believe I was not soon to
drink and wade and dabble in the cool waters. Disappointment was keen. This is what maddens the
prospector or sheep-herder lost in the desert. Was it not a terrible thing to be dying of thirst, to see
sparkling water, almost to smell it and then realize suddenly that all was only a lying track of the desert,
a lure, a delusion? I ceased to wonder at the plainsmen, and their search for water, their talk of water.
But I had not realized its true significance. I had not known what water was. I had never appreciated
it. So it was my destiny to learn that water is the greatest thing on earth. I hung over a three-foot hole
in a dry stream-bed, and watched it ooze and seep through the sand, and fill up–oh, so slowly; and I felt
it loosen my parched tongue, and steal through all my dry body with strength and life. Water is said to
constitute three fourths of the universe. However that may be, on the desert it is the whole world, and
all of life.
Each subject would be shown later exactly one of the following sentences and asked the question: Please
read the following and consider the information in this sentence. Is it information that was Stated or Not Stated
in the source passage you most recently saw?
• So it was my destiny to learn that water is the greatest thing on earth.
• So I was destined to comprehend that water is the most important thing on earth.
• So it was my destiny to learn that friendship is the greatest thing on earth.
• So it was my fortune to experience that conquering fear is the greatest of man’s achievements.

46

Image Diversion
In order to make it more difficult for the participants to simply memorize the visual appearance, we intend to show
two of the following images to users immediately after they complete the study of the source graph. Which images
will be shown is governed by a counterbalancing scheme.

47

48

49

Post-session Questionnaire
Questions asked after example queries; all questions may be skipped
1. How often do you closely read statistical graphics (like the ones you just saw) for work or personal reasons?:
Daily, Weekly, Monthly, Less often, Prefer not to say
2. How often do you create statistical graphics (like the ones you just saw) for work or personal reasons?: Daily,
Weekly, Monthly, Less often, Prefer not to say
3. Do you believe that you understood what the queries about the text passages were asking? Yes, No, Prefer not
to answer
4. Of the 12 queries about the text passages, how many do you think you got right? Enter 0 if you prefer not to
answer. (numerical entry)
5. Do you believe that you understood what the queries about the graphs were asking? Yes, No, Prefer not to
answer
6. Of the 12 queries about the graphs, how many do you think you got right? Enter 0 if you prefer not to answer.
(numerical entry)

50

Debriefing Statement
Thank you for participating in our study!!
Your participation will help us determine the validity of queries about graph comprehension in the form that we described and used, fashioned after the Sentence Verification Technique. Ultimately, we hope to understand how various
visual representations of data are interpreted by users, so that we may improve the quality of these representations.
Your participation will someday help us design better visual representations of data.
If you have any questions about the research, you may contact the principal investigator (PI) at [email protected].
Please be aware that we have tried to protect your identity from the researchers conducting this work; if you send
email to the PI, you will lose your anonymity. If you have concerns about the conduct of this study, you may contact
the Chair of the Naval Research Laboratory Institutional Review Board at [email protected].

51