Supplemental and Part B questions

NOAA National Weather Service

NWS 2016 Hurricane Products Usability Testing Project

Part A and B DOC/NOAA Customer Survey Clearance

OMB Control No. 0648-0342

Expiration Date: 05/31/2018

NATIONAL WEATHER SERVICE (NWS) 2016 HURRICANE PRODUCTS USABILITY TESTING PROJECT

1. Supplemental Questions for DOC/NOAA Customer Survey Clearance
   (OMB Control Number 0648-0342)

1. Explain who will be conducting this survey. What program office will be
   conducting the survey? What services does this program provide? Who are the customers? How are these services provided to the customer?

The National Weather Service (NWS) seeks to gather feedback from its partners on a new product, the Hurricane Threats and Impacts (HTI) website, through usability testing. The HTI website (http://preview.weather.gov/hti/) is an Internet-based decision-support service designed to help users quickly interface with local hazard information. During the Atlantic and Caribbean Hurricane Season (June 1-November 30), the information on the site will be generated and posted by 23 coastal NWS Weather Forecast Offices (WFOs) whenever tropical cyclone watches and/or warnings are in effect for their area. The site provides local threat assessments and corresponding potential impacts information about tropical storm or hurricane wind, storm surge, flooding rain, and tornadoes.

The target audience for the testing includes federal, state, regional, and local decision-makers and emergency management personnel; military, police, and transportation officials; school officials; private sector meteorologists; and representatives of disaster relief organizations. These individuals depend upon and work closely with local WFOs to get localized information whenever a tropical storm or hurricane threatens their area.

2. Explain how this survey was developed. With whom did you consult regarding content during the development of this survey? Statistics? What suggestions did you get about improving the survey?

NWS contracted with Eastern Research Group, Inc. (ERG) on the development of the usability testing guide. ERG has significant experience in conducting detailed interviews, focus groups, usability tests, and surveys for federal agencies that focus on customer satisfaction and outcome attainment. To develop the usability testing guide, ERG worked with Dr. Betty Morrow, a well-known and respected social scientist, who has conducted many research projects with NWS stakeholder groups related to communicating and interpreting tropical cyclone hazards, as well as evacuation studies and risk behavior/communication studies. ERG and Dr. Morrow also worked closely with Dr. Pablo Santos, the meteorologist-in-charge of the local Miami WFO, Dave Sharp the Science and Operations Officer for the Melbourne, Florida, WFO and Shannon White, with the NWS Advanced Weather Interactive Processing System (AWIPS) Support Branch.

The usability testing questions focus on completing a series of tasks to determine if users of the website can efficiently find information to help them do their jobs. The NOAA/NWS staff reviewed the tasks for accuracy and to ensure the phrasing of questions was appropriate.

3. Explain how the survey will be conducted. How will the customers be sampled (if fewer than all customers will be surveyed)? What percentage of customers asked to take the survey will respond? What actions are planned to increase the response rate? (Web-based surveys are not an acceptable method of sampling a broad population. Web-based surveys must be limited to services provided by Web.)

The usability testing will be conducted in person at the National Hurricane Center/Miami WFO. The testing will be in the form of one-on-one moderated sessions. A moderator will observe a participant as he or she completes a series of tasks using the HTI website. The website can be publicly viewed at (http://preview.weather.gov/hti/). The WFO in Miami provided a list of individuals to recruit from based on its knowledge of the partner community. The list includes 54 contacts: 16 emergency managers, 8 federal officials, 8 state officials, 8 school officials, 6 members of local community groups, and 8 members of the media.

ERG will contact all of individuals on this list and introduce the project, our role in the project, and request that the individual participate in the usability testing. We anticipate that roughly 50 percent (27 people) of the contacts will accept the invitation, given potential conflicts in schedules. The sample would therefore be approximately: 8 emergency managers, 4 federal officials, 4 state officials, 4 school officials, 3 members of local community groups, and 4 members of the media.

In contacting each individual, ERG will introduce ourselves, explain our affiliation, describe the testing logistics, and how long the testing will take:

My name is X. I am a consultant with Eastern Research Group (ERG), and I am are working under contract with NOAA’s NWS to conduct some usability testing on a new NOAA National Weather Service website. Would you be willing to spend about 50 minutes with us looking at the website and answering some questions about it? The dates are yet to be confirmed, but we are looking at either April 28 (Thursday) or April 29 (Friday) or May 19 (Thursday) and May 20 (Friday). Right now, we have slots open from around 8 a.m. to 5:30 p.m. We are looking at conducting the testing at the National Hurricane Center on the campus of the Florida International University at 11691 S.W. 17th Street, in Miami. If you are interested in helping us out, could you please reply to this email and let me know what date and timeframes you’d prefer? I will then be in contact with you to firm up the timing and location. Thank you so much in advance for your consideration. Your feedback would be very useful to the NWS!

ERG will generate detailed notes from the testing.

4. Describe how the results of this survey will be analyzed and used. If the customer population is sampled, what statistical techniques will be used to generalize the results to the entire customer population? Is this survey intended to measure a GPRA performance measure? (If so, please include an excerpt from the appropriate document.)

NOAA/NWS will use the information resulting from this testing to help guide refinements to the website and the user interface. The testing will also help NOAA/NWS determine if additional education, marketing, training, or partner coordination is warranted to ensure that organizations and individuals are aware of the website, know how to access it, and know how to use it.

NOAA/NWS will have ERG perform thematic analyses on the detailed testing and observation notes and recordings to identify key findings and recommendations.

The data do not directly contribute to a GPRA measure.

NWS/NHC is not using any statistical methods to select participants from the population and will select all population members in the interviews.

2. Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

The potential respondent universe includes an initial list of 54 individuals to recruit from based on the NWS’s knowledge of the partner community in the Miami area. The list includes 16 emergency managers, 8 federal officials, 8 state officials, 8 school officials, 6 members of local community groups, and 8 members of the media.

ERG will contact all of individuals on this list via email to introduce the project, our role in the project, and request that the individual participate in the usability testing. We anticipate that roughly 50 percent of the contacts (27 people) will accept the invitation, given potential conflicts in schedules. The sample would therefore be approximately: 8 emergency managers, 4 federal officials, 4 state officials, 4 school officials, 3 members of local community groups, and 4 members of the media.

No statistical methods are being used in the initial participant selection.

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

The testing will be conducted through a set of moderated one-on-one sessions, approximately 50 minutes long. Each testing session will pair one moderator with a participant. Participants will be asked to complete a set of tasks that correlate to the main types of information provided by the website. Each task will gauge the user’s ability to locate and interpret information available on the site. Moderators will observe the session and help keep the participant moving through the tasks and will prompt participants to narrate their thought processes as they complete each task. Moderators will not provide ancillary information about the site or answer participant questions about performing tasks. ERG will conduct the sessions over a two-day period on April 28-29 or May 19-20 in Miami at the National Hurricane Center (NHC)/WFO Miami.

Statistical Method for Stratification and Sample Selection

The NWS is not using statistical methods for collecting these data.

Estimation Procedure and Accuracy

The NWS does not need to extrapolate the results to the population and will therefore not need to estimate population parameters from the collected data. This also means that the accuracy of the estimates in not meaningful to calculate.

Unusual Problems Requiring Specialized Sampling Procedures

None are required.

Periodic Data Collection Cycles

This request is for a one-time data collection.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

ERG will contact all of contacts via email to introduce the project, our role in the project, and request their participation in a 50-minute usability testing session. NWS expects that 50 percent of the interviewees will agree to participate in the testing. Should ERG have difficulty recruiting participants, NWS staff in the Miami WFO who know the individuals on the contact list, will send or call them to request their participation and explain the importance of the testing to the NWS as the hurricane season approaches.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

The NWS consulted with Eastern Research Group, Inc. (ERG) on the development of the usability script. ERG has significant experience in conducting usability testing of websites, as well as in conducting detailed interviews, focus groups, stakeholder engagement, and surveys that focus on customer satisfaction with services. The usability script and guide development process was informed by discussions with NOAA and NWS staff.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The NWS has contracted with Eastern Research Group, Inc. (ERG) of Lexington, MA to design the interview guide and discussion questions and implement the data collections. ERG’s project manager for this work is Linda Girardi (703-841-0501; [email protected]).

3