5/3/07
NOTE TO THE REVIEWER OF: |
OMB No. 1220-0141 Cognitive and Psychological Research
|
FROM: |
Scott Fricker and Christine Rho Behavioral Sciences Research Center Office of Survey Methods Research (OSMR)
|
SUBJECT: |
Submission of Materials for the Testing of the BLS Public Website (www.bls.gov) |
Please accept the enclosed materials for approval under the OMB clearance package “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the research, and the materials to be used in the research at least two weeks prior to the beginning of the study.
Participants will be recruited from the OSMR participant database (i.e., general public), BLS staff, and BLS website users. The maximum number of burden hours is estimated to be 40.
If there are any questions regarding this project, please contact Scott Fricker (202) 691-7390.
I. Introduction and Purpose
The BLS Website (www.bls.gov) served an average of 1.5 million users each month in 2005. In its relatively short existence, this website has become the primary way for the overwhelming majority of our customers to obtain BLS data and for BLS to communicate with those customers.
The current design has been in place since 2001. Although it has served both the agency and the public well for the past five years, one reality of the Internet is that nothing can remain static for long. User demands, desires, and expectations constantly grow; BLS continues to post more and more information to the site and to expand the site’s uses; and technology continues to improve. Therefore, BLS has decided to redesign its website to better serve the needs of our customers and the mission of the agency.
The BLS Website Redesign Team has been organized to design, test, and implement a new navigation structure and look-and-feel for the BLS Website that betters serves our customers by making BLS information easier to find and easier to use. This redesign will concentrate on the organization of materials on the BLS Website, the navigation between pages, and the common structural elements of individual web pages.
The top priority for the redesign is an enhanced end-user experience. The user should be able to work quickly and effectively, without distraction, in a pleasant virtual environment.
In 2006, the Redesign Team completed an initial usability study of the existing BLS public website, and a card sorting study of the terms and concepts used on the BLS.gov homepage. Several prototypes for a revised site are being developed on the bases of these studies and input from design consultants and Team members. Two rounds of usability testing are currently planned for FY 2007, and this memorandum requests clearance for these usability tests.
II. Data Collection Methodology: Plan for Usability Testing
The objectives of these usability tests are to gather information on the ease and comfort level (satisfaction) of persons accessing the Bureau of Labor Statistics public website (www.bls.gov). In addition, demographic information about test participants will be collected (see Attachment 2).
Scenarios
Usability tests will be structured around tasks that users typically desire to accomplish when visiting the Bureau of Labor Statistics website. These tasks will be created by the website development team (draft of the scenarios can be found in Attachment 3).
Participants will be recruited from three sources: 1) the participant pool maintained by OSMR, 2) BLS staff and interns, and 3) current users of the BLS website.
Existing customer satisfaction surveys of the OOH website (Foresee) and feedback from helpdesk personnel (who answer questions from BLS customers) will be used to provide an approximate demographic breakdown of the user population. The OOH website apprises about 80% of the hits received on BLS Website. We will try to match the user population as closely as possible through our recruitment efforts. Although we have traditionally relied on 5-15 users in a usability test, recent research indicates that larger numbers are needed for reliable conclusions in diverse audiences. Therefore, we will attempt to recruit 20 participants for each of two tests—for a total of 40 participants.
OSMR will be using a new usability testing tool called UTE for Usability Testing Environment. This tool was developed through the combined support of several federal agencies (GSA, HHS, IRS, USDA, NRC). Its use allows for the conduct of highly structured, semi-automated, standardized usability tests (once they are set up). UTE will be used for this usability testing since it was specifically designed for website usability testing.
Participants will be given a series of tasks/scenario to complete on the BLS website. The tasks will be developed by OSMR and the BLS Website Redesign team to test key components of the interface (examples of scenarios are shown in Attachment 3). Scenarios will be specified in advance to determine the criteria for the ‘successful completion’ of each task. The scenarios and pre- and post-evaluation questions will take approximately an hour to complete. Participants will complete these tasks in the lab and be given an incentive of $40 for an hour session.
Respondents will be told that the purpose of the evaluation sessions is to examine how people use the BLS Website, and to identify ways in which the BLS might improve it.
In order to get a better idea of what respondents are thinking as they attempt each task, and to help delineate between errors of omission and commission, the usability protocol will employ the “think aloud” technique, which asks respondents to provide a running monologue of their thoughts as they attempt to complete tasks. The researcher will brief the respondent on how to provide this type of protocol. They will be encouraged to say whatever comes to mind, but told that the researchers are especially interested in hearing about any aspects of the system that the respondent finds difficult, why they are difficult, and how respondents are trying to resolve these difficulties.
Since the UTE tool controls the flow of the usability test, participants will be left to proceed at their own pace and largely without assistance from the researcher unless the user becomes stuck or overly frustrated. The UTE tool automatically records task completion times, system and user errors, and user comments. With the respondent’s permission, all lab sessions will be videotaped and audiotaped (see Attachment 1 for consent form).
In addition, we plan to utilize two technologies to extend the capabilities of these usability tests. First, to examine specific screen features, users’ eye movements will be recorded as they progress through the test protocol and the site. As part of the introduction to the study, these participants will be told that they will be testing the usability of the BLS website, and that part of this assessment will involve tracking of their eye movements. Second, when possible, we plan to conduct remote usability test sessions with actual users of the BLS public website. Users in different parts of the country will participate remotely in test sessions by using an on-line usability software called UserVue (http://www.techsmith.com/uservue.asp#). Remote users will be invited to join the test session via email and a simple and secure log on process, and the procedures for the remote test sessions will be identical to those used for local test sessions. No eye-tracking data will be collected from remote users.
User errors, time to completion, task failures, eye-gaze data, and post-task and post-test evaluations will be recorded for all users. In addition, user comments during the verbal protocol and post-test debriefing will be recorded. Results of these qualitative and quantitative measures will be summarized in written reports.
As noted, two rounds of testing are scheduled, with an average number of 20 persons in each round. It is estimated that the usability sessions will take an average of 60 minutes, for a maximum of 40 total burden hours (20 participant hours in each of 2 studies). This clearance is requested through September 2007.
IV. Data Confidentiality
Participants will be informed as to the voluntary nature of the study. Participants will be informed that the study will be used for internal purposes to improve the design and implementation of the BLS Website. No information related to this study will be released to the public in any way that would allow identification of individuals except as permitted by the Privacy Act (see Attachment 1).
V. Attachments
Attachment 1: Consent Form
Attachment 2: Pretest questions
Attachment 3: Scenarios for Usability Tests
ATTACHMENT 1 - Consent Form
The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.
The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.
During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.
We estimate it will take you an average of 60 minutes to participate in this research (ranging from 50 minutes to 70 minutes).Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.
Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141, and expires 02/28/09.
------------------------------------------------------------------------------------------------------------
I have read and understand the statements above. I consent to participate in this study.
___________________________________ ___________________________
Participant's signature Date
___________________________________
Participant's printed name
___________________________________
Researcher's signature
OMB Control Number: 1220-0141
Expiration Date: 02/28/09
In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The BLS may release individually identifiable information to individuals designated as agents of the BLS in accordance with Public Law 107-347 to perform exclusively statistical activities. Individuals designated as agents of the BLS may be imprisoned for not more than 5 years or fined not more that $250,000 or both for any knowing and willful disclosure of respondent information to unauthorized persons. Such designated agents may include individuals from other sponsoring agencies; to contractors, grantees, and their employees or volunteers who are working on this study for the BLS and who need access to the information; or to the National Archives and Records Administration or the General Services Administration for records management purposes. Under written agreements to protect the confidentiality and security of individually identifiable information, the BLS may provide individually identifiable information to other researchers designated as agents of the BLS to conduct statistical research projects that further the mission and functions of the BLS.
ATTACHMENT 2 – Pretest Questions
How many years have you used a computer?
Less than 1
1-4
5-9
10 or more
Weekly computer use?
Less than 1 hour a week
1-10 hours a week
11-20 hours a week
21 or more hours a week
How many years have you used the Internet for things other than just email?
Less than 1 year
1-4
5-9
10 or more years
Weekly Internet (Web) Use?
Less than 1 hour a week
1-10 hours a week
11-20 hours a week
21 or more hours a week
What is your gender?
Male
Female
What is your age?
18-39
40-59
60-75+
How much education have you completed?
High School or less
Some college
College degree
Advanced degree
ATTACHMENT 3 –– Usability Test Scenarios
Each person will attempt to complete 12 scenarios. The scenarios to be used in the usability test are shown below. The scenarios will be presented to participants in random order.
1. (Special Notices): You are interested in the Productivity and Costs indexes that BLS produces. You had heard that there were recent corrections made to the publication start date of some of the industries. Please check which industries are affected by this.
2. (Area & Occ Wage): You are opening a small pet supply store in Falls Church, VA and you would like to figure out how much to pay a cashier. Please find the average hourly pay for a cashier in private industry in northern Virginia.
3. (Prevail wage): Does BLS produce the prevailing wage data? Where can you find information about the prevailing wage data under the Davis-Bacon Act?
4. (CPI): You want to know about the 2006 scheduled news releases for the Consumer Price Index (CPI). Please find the page that contains this information.
5. (CEX): You're interested in statistics on household expenditures by region that BLS produces. Find which region of the country has the highest annual expenditure in household utilities.
6. (CPS): What was the national annual average unemployment rate for 1992?
7. (OOH): You are considering going to nursing school. What is the expected job growth for registered nurses?
8. (LAUS): Please find the unemployment rate (seasonally adjusted) in the District of Columbia for January 1996.
9. (QCEW): BLS publishes a quarterly count of employment and wages reported by employers covering 98 percent of U.S. jobs, available at the county, MSA, state and national levels by industry. Please find the page that contains this information.
10. (PPI): You heard that the prices received by producers of gasoline rose again in May. By what percentage did the producer price for gasoline increase in May?
11. (ECI): You're interested in finding out how rapidly labor costs are rising for employers. What was the percentage change in these costs (seasonally adjusted) for the first quarter of 2006 (that is, December 2005 to March 2006)?
12. (CES): You read that the manufacturing industry in Michigan has declined dramatically in recent years. How many manufacturing jobs did Michigan have in June 2000?
| File Type | application/msword |
| File Title | May 15, 2003 |
| Author | FRICKER_S |
| Last Modified By | FRICKER_S |
| File Modified | 2007-05-15 |
| File Created | 2007-05-15 |