Website Usability Testing

April 15, 2010

Ms. Christine Kymn

Department of Energy Desk Officer

Office of Information and Regulatory Affairs

Office of Management and Budget

Washington, DC 20503

SUBJECT: EIA-882T(50), “USE OF GENERIC CLEARANCE FOR THE ENERGY INFORMATION ADMINISTRATION’S WEBSITE USABILITY TESTING”

Dear Ms. Kymn:

The U.S. Energy Information Administration (EIA) plans to use the Office of Management and Budget (OMB) approved generic clearance, EIA-882T, “Use of Generic Clearance for Questionnaire Testing, Evaluation, and Research,” OMB No. 1905-0186, expiring 2/29/2012, to conduct online usability testing of EIA’s website. Your action is anticipated within two weeks; however, EIA will not conduct this testing without approval. Results of this testing will provide us with valuable insights to facilitate an upcoming website redesign and for a benchmark measurement from which to measure success.

The usability testing EIA is proposing will be conducted using an online testing tool called “Loop 11” (www.loop11.com) that administers the test by showing test questions superimposed on the target website (in this case, www.eia.gov). The Loop 11 tool records clicks through the target website, URLs visited, and answers users provide to the test questions. Loop 11 does not conduct analysis on the data; it is just a tool that collects and presents the data. EIA will then analyze how users navigated through the EIA website to answer the test questions, which will help us understand user behavior on the EIA website. The intended results are to improve the website’s usability and to guide the direction EIA takes to change and improve its navigation, web content and presentation, and electronic delivery of energy information.

The number of customers using EIA’s website continues to be very large. In 2009, EIA’s site had 27 million visitor sessions, for an average of 2.2 million customers per month. Customers include policy-makers, financial markets, the energy industry, commercial businesses, academia, the media, and private citizens. These customers rely on energy information from EIA that is relevant, easily accessible and comprehensible. With the move to nearly 100 percent electronic dissemination of information, EIA has responded to the growth in electronic customers and the increased focus on electronic government, and realizes that the design and functionality of a government statistical website has become more critical.

This is the first usability testing of this magnitude that EIA has undertaken. We expect to collect much valuable information and use it to make enhancements to our website in the upcoming redesign. Usability engineers recognize that website design enhancements often introduce new problems, and the only way to ensure this does not happen is through benchmarking and follow-up usability testing. Therefore, following the roll-out of the new design, we will conduct a follow-up usability test using the same protocol. We will compare the benchmark and follow-up data to ensure the redesign was successful in increasing access and usability.

The complete usability test package contains a total of 12 tasks, but each test participant will randomly receive only 2 tasks to complete. We present only 2 tasks to each participant to keep the test brief and encourage participation from our entire range of customers, including those with limited time such as congressional staff, upper management, and journalists. Some of the tasks are more difficult than others.

The tasks are in two forms: (1) a prompt to answer a question with a statistic or fact from the EIA’s website, or (2) a prompt to find the correct page on EIA’s website. The prompts were chosen because they represent EIA customers’ “top tasks” – they are among the most commonly asked questions of EIA’s energy information call center and are answered by referencing EIA’s most popular and important content and products.

Top tasks for the usability testing are:

Answer questions using energy statistics or facts from EIA’s website:

1. Which State produces the most coal?

2. What percent of U.S. electricity came from nuclear power in 1990?

3. Why doesn’t the United States use more renewable energy?

4. How much carbon dioxide does EIA project that developing nations will emit in 2030?

5. Did the amount of U.S. natural gas in storage rise or fall from last week, and by how much?

6. What does EIA project regular gasoline prices to average in 2011?

7. What percent of the energy used in the United States was used by homes in 2008?

8. What is the average price of gasoline in the United States that EIA has most recently reported?

Find the right page on EIA’s website:

9. Find the page where you can subscribe to receive weekly updates about oil markets.

10. Find the most recent testimony EIA gave to Congress.

11. EIA sometimes studies the expected impact of an energy bill before Congress. Find EIA’s most recent study.

12. Find a high-resolution map of offshore U.S. natural gas fields.

After each task, participants will be asked to rate the ease of completing the task on a six-step Likert scale (very hard, hard, somewhat hard, somewhat easy, easy, very easy).

Three demographic questions will be asked at the beginning of the survey to measure how the participant mix represents the EIA customer segments. The demographic questions to be asked are:

1. How often do you visit EIA’s website?

   1. Daily

   2. Weekly

   3. Monthly

   4. Less than once a month

   5. First-time visitor

2. Which category best describes you or your organization?

   1. Government (Federal, State, local, Congress)

   2. Finance (banking, investment)

   3. Business/Industry ­– Energy-related

   4. Business/Industry – NOT Energy-related

   5. Research/Consulting

   6. Media

   7. Nonprofit

   8. Teacher/Professor

   9. Student

   10. Library

   11. Private citizen

   12. Other

3. What is your highest level of education?

   1. High School or less

   2. Associate’s degree

   3. Bachelor degree

   4. Graduate degree or more

This test will be conducted on an opt-in participation basis. The test invitation will be presented to 10% (selected randomly) of EIA website users who visit at least two consecutive pages on our site. The invitation will be presented immediately upon visiting any second page on the EIA website. The code for displaying the invitation shows a hidden “div” tag; this is similar to the approach used by the American Customer Satisfaction Index (ACSI) survey, the survey tool broadly used across Government and cleared by OMB. Accepting the test invitation involves following a link shown in the invitation. Upon acceptance, users will input data for the test questions via the Loop 11 online tool.

EIA will field the test until 1,000 responses are received. The number of responses will be monitored each day. We anticipate it will not take longer than two weeks to reach this number of responses, because we receive several thousand responses to the annual web customer survey in two weeks.

The medium for administration of this test will be the Loop 11 (www.loop11.com) service, although the participants will only see the invitation and directions page as we set them up. The Loop 11 tool displays the test questions as only a thin masthead on top of the EIA website itself. To preview question #7, see: http://www.loop11.com/usability-test/3501/PREVIEW_DO_NOT_USE_THIS_LINK_FOR_COLLECTION/sO9ffAFbo8/

The information that Loop 11 provides us from each participant includes:

• Task completion rate

• Responses to “fill in the blank” and multiple choice style questions

• Time per task

• Detailed participant path analysis

• Number of page views to complete task

A simplified version of these analytics is demonstrated in a recent case study evaluating top tasks on commercial airline websites: http://www.loop11.com/blog/blog/2010/01/through-the-loop-case-study-airline-website-usability/

Because EIA does not have a list or frame of its website customers, EIA cannot designate a scientific sample. We do expect responses from a diverse group of customers due to the sheer volume of input seen from previous customer survey responses. Online testing is a convenient way to get users involved in improving the usability of our site while saving us and them time and money. While we cannot claim that responses to this test will be representative of all users, EIA anticipates gaining a much better understanding of how users navigate through the website and find our information in addition to an objective usability baseline against which to measure the success of our website redesign.

All participation in this testing is voluntary. No self-identification information will be requested. Results will be presented in aggregate form. Subsequent analysis of the data collected will limit any divulgence of individual customer responses.

Depending upon which tasks they receive, participants should take no more than five minutes to navigate to and answer each task. For the two tasks plus the three demographic questions, each participant is expected to be able to complete the entire usability test in ten minutes, based on tests with internal participants and because the instructions will ask participants to click “Abandon Task” after 5 minutes. The burden for the targeted customer group, which will mostly include users of the website external to EIA, should not exceed 167 hours (1,000 potential participants multiplied by 10 minutes each).

Colleen Blessing, EIA's User Experience Advisor, is the point of contact for questions and may be reached at 202-586-6482. Other questions should be directed to Grace Sutherland at 202-586-6264.

Sincerely,

Stephanie Brown

Director

Statistics and Methods Group

Energy Information Administration

.