July 11, 2008
NOTE TO THE REVIEWER OF: |
OMB No. 1220-0141 Cognitive and Psychological Research
|
FROM: |
Jennifer Edgar Research Psychologist Office of Survey Methods Research (OSMR) |
SUBJECT: |
Submission of Materials for the Web Survey Design Eye Tracking Study |
Please accept the enclosed materials for approval under the OMB clearance package “Cognitive and Psychological Research.” In accordance with our agreement with OMB, I am submitting a brief description of the research, and the materials to be used in the research.
The total estimated respondent burden hours for the study are 15 hours.
If there are any questions regarding this project, please contact Jennifer Edgar at (202) 691-7528.
Introduction
This study has several research objectives, each will be investigated using eye tracking technology. The first objective is to investigate the impact of the layout of response options on respondent behavior in self-administered forms. Numerous studies have demonstrated context effects (Tourangeau, Rips & Raskinski, 2000; Couper, Conrad & Tourangeau, 2004, 2007; etc.), and it is clear that the way response options are presented can have an impact on response. We seek to explore these findings using eye tracking data, to the presentation of response options impacts the way respondents look at them; for example if they spend more time looking at the response options when they’re presented horizontally, or if they seem to skip some options when presented vertically.
The second research objective seeks to explore respondent eye movement in relation to a progress indicator. Although there are mixed results from past research on the impact of progress indicators (i.e., Conrad, Couper, Tourangeau & Baker, 2003; Heerweghk, 2004), they are generally recommended when designing web surveys (Dilman, Tortora & Bowker, 1999). We seek to explore this recommendation, to identify when and for how long respondents look at progress indicators, and if the presence of a progress indicator affects respondents’ perception of survey length.
Finally, we will investigate a few general survey designs, aimed at using respondents’ eye movements to add insight into survey design principles. The designs include a survey question formatted in a grid to see how respondents interact with the headings, questions and format in general; questions with similar stems and response options to see if respondents’ eye movements change as they move through the series, and questions with long lists of response options to see if there is evidence that respondents skim later items in the lists, or stop looking once they have found their response. We will also use questions with ‘other’ response options to see if respondents spend more or less time looking at response options when an “other” option is offered. A few questions with long lists of response options will be included, to investigate the possibility that respondents stop looking at the list of options once they’ve selected one. The last question investigates respondent’s eye behavior for a series of similar questions, looking to see if they spend the same amount of time looking at the earlier questions as later ones, or if it appears they start to skim the later questions.
In summary, using eye tracking data we will investigate the following questions:
What impact does formatting of response options (i.e. horizontal, vertical or drop down lists) have on respondent visual behavior?
How do respondents interact visually with progress indicators?
How do respondents scan and look at questions presented in a grid?
Does the presence of an ‘other’ response option change the amount of time respondents spend looking at the list of options?
For long list of response options, do respondents look at the entire list of options or do they stop looking once they’ve selected an option?
In a series of similar questions, do respondents spend the same amount of time looking at the early and late questions?
II. Data Collection Methodology
Procedure
Thirty respondents will be recruited from the OSMR database and brought into the cognitive lab. They will be randomly assigned one of six survey versions, as shown in Table 1. The survey versions are shown in Appendix A.
For each of the survey versions, respondents will be randomly assigned a version with or without an “other” response option to the relevant questions.
Table 1. Sample Sizes by Survey Design
|
Progress Indicator |
No Progress Indicator |
Horizontal responses |
5 |
5 |
Vertical responses |
5 |
5 |
Drop down Responses |
5 |
5 |
Respondents will be informed that they are in the lab to answer a survey about a range of general topics. They will also be told that we will be measuring their eye movements as they work on the computer, in order to evaluate the formatting of the survey. Respondents will be calibrated on the eye tracker and the survey website will then load automatically. The researcher will leave the room as the respondent completes the survey.
Once respondent have finished the survey, they will be given a debriefing questionnaire designed to evaluate their perceptions of the survey (Attachment B).
The survey questions were selected with the intention of being relevant to all respondents, as well as having the design or layout features needed to answer the research questions. Most questions were chosen from the 2008 General Social Survey, and the remaining were either written specifically for this project or adapted from on-going BLS surveys such as the Consumer Expenditure Survey or the Current Population Survey. In addition, a few demographic questions were included to gather background data on the respondents.
Eye tracking data will be analyzed in multiple ways to explore each of the research questions. Table 2 shows the list of research questions and the eye tracking data that will be used to investigate them.
Table 2. Eye Tracking Data to be used to evaluate research questions.
Research Question |
Eye Tracking Data |
|
|
|
|
|
|
|
|
|
|
|
|
This study will take place in the cognitive lab, and will involve up to 30 respondents, and they are expected to be in the lab completing both the survey and debriefing form for approximately thirty minutes. The total estimated respondent burden hours for the study are 15 hours.
V. Data Confidentiality
Respondents will be informed as to the voluntary nature of the study. Respondents will also be informed that the study will be used to add to general survey methodology knowledge. Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice. A copy of the consent form is included as Attachment C.
VI. Attachments
Attachment A: |
Survey Versions |
Attachment B: |
Debriefing Questionnaire |
Attachment C: |
Consent Form |
Attachment D |
References |
Attachment A: Survey Questions
(Note: Response options for questions 1 through 5 will be formatted either horizontally, vertically, or in a drop down box, depending on the treatment condition)
On the whole, how satisfied are you with the work you do – would you say you are very satisfied, moderately satisfied, a little dissatisfied, or very dissatisfied?
Very satisfied 1
Moderately satisfied 2
A little dissatisfied 3
Very dissatisfied 4
We are interested in how people are getting along financially these days. So far as you and your family are concerned, would you say that you are pretty well satisfied with your present financial situation, more or less satisfied, or not satisfied at all?
Pretty well satisfied 1
More or less satisfied 2
Not satisfied at all 3
During the last few years, has your financial situation been getting better, getting worse, or has it stayed the same?
Getting better 1
Getting worse 2
Stayed the same 3
Compared with American families in general, would you say your family income is – far below average, below average, average, above average, or far above average?
Far below average 1
Below average 2
Average 3
Above average 4
Far above average 5
Taken all together, how would you say things are these days – would you say that you are very happy, pretty happy, or not to happy?
Very happy 1
Pretty happy 2
Not too happy 3
Which answer comes closest to how often you do the following things…
|
Almost every day |
Once or twice a week |
Several times a month |
About once a month |
Several times a year |
Arrive late to work |
|
|
|
|
|
Arrive early to work |
|
|
|
|
|
Work longer than your scheduled shift |
|
|
|
|
|
Use earned vacation leave |
|
|
|
|
|
Use earned sick leave |
|
|
|
|
|
Thinking about your local government, would you say that it should take measures to improve traffic?
Yes
No
7a. Thinking about your local government, would you say that it should take measures to improve living conditions?
Yes
No
7b. Thinking about your local government, would you say that it should take measures to improve the health care system?
Yes
No
Thinking about your state government, would you say that it should take measures to improve traffic?
Yes
No
8a. Thinking about your state government, would you say that it should take measures to improve living conditions?
Yes
No
8b. Thinking about your state government, would you say that it should take measures to improve the health care system?
Yes
No
Thinking about your federal government, would you say that it should take measures to improve traffic?
Yes
No
9a. Thinking about your federal government, would you say that it should take measures to improve living conditions?
Yes
No
9b. Thinking about your federal government, would you say that it should take measures to improve the health care system?
Yes
No
Thinking about local private organizations, would you say that they should take measures to improve traffic?
Yes
No
10a. Thinking about your local private organizations, would you say that they should take measures to improve living conditions?
Yes
No
10b. Thinking about your local private organizations, would you say that they should take measures to improve the health care system?
Yes
No
(Note: the presence of an “other” options for questions 12 through 14 will vary depending on the treatment condition)
What is your marital status?
Married
Widowed
Divorced
Separated
Never married
Other specify
What is the highest level of school you have completed or the highest degree you have obtained?
Less than high school
High school
Some college
Associates degree
Bachelors degree
Master’s degree
Professional degree
Other
Which of the following categories best describes the job in which you received the most earnings during the last 12 months?
Teacher
Administrative support, including clerical
Sales, retail
Sales, business goods and services
Technician
Protective service
Private household service
Machine operator, assembler, inspector
Transportation operator
Handler, helper, laborer
Mechanic, repairer, precision production
Armed Forces
Other
How long have you been employed in your current position?
Less than one year
One to two years
Three to five years
Six to ten years
Eleven to twenty years
More than twenty years
Other
During the last year, which best describes your employment situation:
Employee of a private-for-profit company or business or of an individual, for wages, salary or commissions
Employee of a private-not-for-profit, tax exempt or charitable organization
Local government employee
State government employee
Federal government employee
Self employed in own not incorporated business, professional practice or farm
Self employed in own incorporated business, professional practice or farm
Working without pay in family business or farm
Which of the following types of health insurance or health coverage plans are you currently covered by:
Insurance through current or former employer or union
Insurance purchased directly through an insurance company
Medicare, for people 65 or older, or those with certain disabilities
Medicaid, Medical Assistance, or any kind of government-assistance plan for those with low incomes or a disability
VA
TRICARE or other military health care
Indian health service
Some other type of health insurance
Which best describes your primary residence?
A mobile home
A one-family house detached from any other house
A one-family house attached to one or more houses
A building with 2 apartments
A building with 3 or 4 apartments
A building with 5 to 9 apartments
A building with 10 to 19 apartments
A building with 20 to 49 apartments
A building with 50 or more apartments
Boat, RV, van, etc.
Do you own a home computer?
Yes
No (skip to end of survey)
How often do you use your home computer?
At least once a day
2-3 times a week
4-6 times a week
Once a week
Less than once a week
At home, what do you use the computer for (check all that apply)
Browsing the internet
Emailing
Word processing
Playing games
Other
Attachment B: Debriefing Questionnaire
Please answer the following questions about the survey you just completed.
1. How long do you feel this survey was?
1 |
2 |
3 |
4 |
5 |
Far too short |
Too short |
Just right |
Too long |
Far too long |
2. How long do you think it took you to complete the survey: _______________
3. How easy or difficult was the survey to complete?
1 |
2 |
3 |
4 |
5 |
Very easy |
Easy |
Neither easy nor difficult |
Difficult |
Very difficult |
4. How interesting were the survey questions to you?
1 |
2 |
3 |
4 |
5 |
Very interesting |
|
Neither interesting nor un-interesting |
|
Not at all interesting |
5. How useful did you find the progress indicator to be? [insert graphic]
1 |
2 |
3 |
4 |
5 |
Very useful |
Useful |
Neither useful nor non-useful |
Non-useful |
Very non-useful |
Attachment C: Consent Form
The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.
The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.
During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.
We estimate it will take you an average of 60 minutes to participate in this research (ranging from 50 minutes to 70 minutes).Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.
Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141, and expires 02/28/09.
------------------------------------------------------------------------------------------------------------
I have read and understand the statements above. I consent to participate in this study.
___________________________________ ___________________________
Participant's signature Date
___________________________________
Participant's printed name
___________________________________
Researcher's signature
OMB Control Number: 1220-0141
Expiration Date: 02/28/09
In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The BLS may release individually identifiable information to individuals designated as agents of the BLS in accordance with Public Law 107-347 to perform exclusively statistical activities. Individuals designated as agents of the BLS may be imprisoned for not more than 5 years or fined not more that $250,000 or both for any knowing and willful disclosure of respondent information to unauthorized persons. Such designated agents may include individuals from other sponsoring agencies; to contractors, grantees, and their employees or volunteers who are working on this study for the BLS and who need access to the information; or to the National Archives and Records Administration or the General Services Administration for records management purposes. Under written agreements to protect the confidentiality and security of individually identifiable information, the BLS may provide individually identifiable information to other researchers designated as agents of the BLS to conduct statistical research projects that further the mission and functions of the BLS.
Attachment D: References
Couper, M., Conrad, F., & Tourangeau, R., (2007). Visual context effects in web surveys. Public Opinion Quarterly, 71: 623-634.
Conrad, F., Couper, M., Tourangeau, R., & Baker, R. (2003). Use and non-use of clarification features in web surveys, Presented at the American Association for Public Opinion Research Annual Conference, Nashville TN.
Couper, M., Tourangeau, R., & Kenyon, K. (2004). Picture this! An analysis of visual effects in web surveys. Public Opinion Quarterly, 68(2): 255-266.
Dilman, D., Tortora, R. D., & Bowker D. (1999) Principles for conducting web surveys. Obtained April 17, 2008 from www.sesrc.wsu.edu/dillman/papers/websurveyppr.pdf
Heerweghk (2004). Using progress indicators in web surveys. Paper presented at the annual meeting of the American Association for Public Opinion Research, Phoenix, Arizona.
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The Psychology of Survey Response. Cambridge, UK: Cambridge University Press.
| File Type | application/msword |
| File Title | September 24, 2004 |
| Author | SHIELDS_J |
| Last Modified By | J. Edgar |
| File Modified | 2008-07-11 |
| File Created | 2008-06-10 |