8 to 10 SS Part A 072517

SUPPORTING STATEMENT

Day 8-10 Timeline Forecast Survey and Focus Groups

OMB CONTROL NO. 0648-xxxx

A. JUSTIFICATION

1. Explain the circumstances that make the collection of information necessary.

Increasingly, user demand is reflecting a need for probabilistic forecasts in the 8 to 10 day time frame. Improved forecast capability means the Weather Prediction Center (WPC) can begin to address this user need, but the task is complicated by the range of possible forecast format and delivery needs and options. WPC is seeking user-tested and informed guidance on how to present forecasts at the 8 to10 day time frame. WPC is looking to understand how probabilistic forecasts can improve core user decision-making in this range, and for guidance on the format and design of forecasts, based on iterative field-testing of core users. WPC requires a robust understanding of core user needs in this time frame, as well as explanation of the preferred web delivery methods and the optimal mix of design and delivery considerations. WPC also requires information about what products end-users are utilizing and how they are being used. To assess ways to improve guidance products for producing forecasts, the project will include meeting with forecasters, conducting phone interviews and web-based focus groups with core partners and posting an online public survey.

2. Explain how, by whom, how frequently, and for what purpose the information will be used. If the information collected will be disseminated to the public or used to support information that will be disseminated to the public, then explain how the collection complies with all applicable Information Quality Guidelines.

How will the information be used?

Staff will make assessments using the information from interviews and focus groups to help to better understand the decision-making and information needs of core partner users, and specifically, their needs for information from WPC products, including the need for and interest in forecasts in the 8 to 10 day range and how best to show the probability of long range forecasts. By doing this research, we hope to learn how probabilistic forecasts can improve decision-making in the 8 to 10 day timeframe.

A brief, web-based survey will be given that will guide end-users through a series of questions to identify what WPC products they use and how they would use 8 to 10 day probabilistic information from WPC. The WPC will use the information collected to conduct a complete evaluation of WPC prototype products and services and apply that information to creating better and more easily understandable weather forecast products.

By whom will the information be used?

The information will be used by WPC.

How frequently will the information be used?

The information provided by this data collection will be used on an ongoing basis to provide insight and guidance for developing probabilistic forecasts. Additionally, the data collected will be used to refine and enhance final WPC products and to provide insight to WPC when developing future products pertaining to day 8 to 10 timeline forecasts.

For what purpose will the information be used?

The information will provide WPC staff with the guidance needed to develop 8 to 10 day weather prediction products. Creating easier to use visualizations and communication products will provide the public and decision-makers accurate information that is understandable and will enhance value for a variety of weather related decision processes. Survey and focus group results will define how to present the probabilistic forecast tools clearly and concisely.

How does the data collection comply with all applicable Information Quality guidelines?

The WPC will retain control over the information and safeguard it from improper access, modification, and destruction, consistent with WPC standards for confidentiality, privacy, and electronic information. See response to Question 10 of this Supporting Statement for more information on confidentiality and privacy. The information collection is designed to yield data that meet all applicable information quality guidelines. Prior to dissemination, the information will be subjected to quality control measures and a pre-dissemination review pursuant to Section 515 of Public Law 106 554.

Survey Information

The survey to elicit public perceptions of and reactions to the variations in products will consist of three sets of questions: 1) demographic and locational characteristics of respondents;

2) current use and knowledge of WPC products; and 3) understanding of and reactions to the various representations of long range probabilistic forecast products.

For dissemination, the survey will be promoted widely nationwide through other outlets such as local weather channels via broadcast meteorologists and their social media accounts, WPC regional office social media accounts, and emergency management agency websites.

For the online survey, collection of demographic and locational characteristics information consists of multiple choice questions about age, gender, place of residence, employment position (self-employed, student, unemployed, government, private, etc.) and relationship to the National Weather Service (NWS) (member of the public, employee at local WFO, etc.).

To develop an understanding of current use and knowledge of WPC products, the respondents are asked multiple choice questions about where they get information about weather, how often they check the weather forecast, to what extent would an 8 to 10 day forecast assist decision-making, what are reasons for interest in an 8 to 10 day forecast, type of forecast preferred and usefulness of forecasts that provide probabilities.

The last section of the survey presents a series of graphics showing 8-10 day forecasts for temperature, precipitation, hazards and winter weather and asks the respondent multiple choice questions about what the product tells them, how likely they would be to use the product, what is their perception with respect to specific qualities (information included, format, understandability, graphics, text, and use of color). These questions are repeated for each graphic. The online survey then asks the respondent about their interest in seeing similar graphics for heat index and wind chill, how they would like to receive this type of forecast and two open ended questions pertaining to other types of weather information preferred and additional comments about 8 to 10 day probabilistic Weather Prediction Center products.

Interviews and Focus Group Information

To identify the core partner and end-user audiences and their potential use of these products, staff will use a two-pronged strategy involving web-based focus groups and phone interviews as well as a web-based survey. First, staff will assemble a list of core partner user groups, working in cooperation with WPC forecasters and building upon identified work through other social science research, web analytics and/or analysis completed by WPC, Climate Prediction Center or other offices. Staff will review existing lists and catalogs to develop a working list of core partner users, including groups such as emergency managers, water suppliers, transportation officials, and others.

From this list, staff will work to identify specific individuals, representing a national distribution, for participation in phone interviews and small web-based focus groups. Staff anticipate hosting 3 focus groups with 5-10 participants in each session representing different core partner groups. In addition, where necessary and helpful to reach further participants, staff will conduct in-depth interviews, conducted via phone and where possible, in person, with other core partners who are users of WPC products. These interviews and focus groups will help to better understand the decision-making and information needs of these core partner users, and specifically, their needs for information from WPC products, including probabilistic forecasts. The focus groups (1.5 to 2 hours in length) will ask core partner users to identify the critical stages of the decision-making processes, and the ways in which various core partner user groups use and/or would use WPC information throughout their decision-making, focusing on the 8 to 10 day time frame but inclusive of the 1-7 day period for a complete understanding of needs. Focus groups and interviews will be facilitated by the Senior Social Scientists, with members of the research team attending to assist in recording and analysis.

After the focus groups, interviews and web-based survey are complete, staff will analyze all qualitative and quantitative data to evaluate the WPC prototype products and services. Staff will develop an infographic that maps identified core partner users and their key decision-making needs for WPC forecast information in the 8 to 10 day timeframe. Then, drawing on the survey data from end users (which will include product-specific feedback), as well as the analysis of core partner user feedback, the staff will work to identify recommended improvements to WPC probabilistic prototype products. Specifically, staff will make changes to product design (e.g., including critical design and content factors, such as color, legends, balance of text and graphics, geographic specificity, clear definition of impacts, etc.)

Once prototype products appear ready for audience review, staff will reconvene with the identified core partner representatives for a second round of web-based focus groups and interviews, this time specifically focused on reviewing how the proposed prototypes meet the decision-making needs identified in the first round of discussion. These focus groups (i.e., 3) will guide participants through a series of product-focused questions related to their user group needs, and will query about how the products would be used, and how they could be further improved. These discussions will identify the range of decision-making contexts in which participants believe they would use the products, and will ask participants to specifically address the design elements (e.g., color, legends, balance of text, graphic presentation), as well as to provide feedback about preferred formats for information and the desired pathways for web-based information (i.e., where should information be located for most efficient access? Should it be pushed to users or pulled down as needed and in either case, how? What dissemination pathways are most helpful?) Questions will aim to identify those design factors that help distinguish low and high-impacts events, as well as to identify how the use of stories (including such elements as historical comparisons and description of variables affecting probabilities) in product development assists in decision-making.

From this second round of focus groups and interviews, staff will develop recommendations for further improvement to existing prototype probabilistic products, as well as recommend new products that may emerge as critical. Where it is deemed helpful, staff will re-assemble select members of the focus groups for an iterative review of prototype products, in order to give increasing levels of specificity on feedback to the product design.

Qualitative data from the focus groups, along with the quantitative findings from the web-based survey, will be assembled into a final visual infographic that provides an overview of the decision-making process for core partner and other end-user groups, including the information they need, the timing of that information, and findings on the design and delivery formats preferred. The infographic will identify the key challenges in end-user group access, use of 8 to 10 day forecasts, and a summary of the key recommendations for improving design and delivery formats.

3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological techniques or other forms of information technology.

The information collected via the online survey will make use of automated, electronic, mechanical and other technological techniques. The online survey will use a “responsive” web-based survey, meaning that its display will adapt to a desktop or mobile device screen allowing the public to take the survey whenever they have internet access, and will be built with a Microsoft SQL database back-end and ASP.net/C-sharp front-end. The web-based survey’s “landing” page will have a brief description of the survey, and a “Completely Automated Public Turning test to tell Computers and Humans Apart” (CAPTCHA) component which will be implemented to prevent “robots” from entering multiple copies of the survey. The web-based survey will also include a “completeness meter” to inform users of how close they are to completing the survey. The survey will display an error message when the user tries to navigate to the next web page without answering all the questions on the current page. Data collected through the web survey will be downloaded via Microsoft Excel and will be available in that format for data analysis.

The focus group data will be collected manually and entered directly into a desktop computer. Transcripts from each of the focus groups will also be provided.

4. Describe efforts to identify duplication.

Based on discussions with WPC staff and risk assessment social scientists, there is no significant duplication of effort within WPC or with other federal surveys.

5. If the collection of information involves small businesses or other small entities, describe the methods used to minimize burden.

The collection of this information does not involve small businesses or other small entities.

6. Describe the consequences to the Federal program or policy activities if the collection is not conducted or is conducted less frequently.

The WPC is tasked with providing weather, water, and climate data, forecasts and warnings for the protection of life and property and enhancement of the national economy. If this data is not collected, WPC will not be able to provide a public and understandable probabilistic weather information in the 8 to 10 day range. Survey and focus group results will define how to present the 8 to 10 day prediction graphics clearly and concisely to best elicit the appropriate response from society.

7. Explain any special circumstances that require the collection to be conducted in a manner inconsistent with OMB guidelines.

The collection will be conducted in a manner consistent with OMB Guidelines.

8. Provide information on the PRA Federal Register Notice that solicited public comments on the information collection prior to this submission. Summarize the public comments received in response to that notice and describe the actions taken by the agency in response to those comments. Describe the efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported.

A Federal Register Notice published on May 11, 2017 (82 FR 90) solicited public comments. No comments were received.

9. Explain any decisions to provide payments or gifts to respondents, other than remuneration of contractors or grantees.

No payments or gifts are made.

10. Describe any assurance of confidentiality provided to respondents and the basis for assurance in statute, regulation, or agency policy.

As stated on the survey and at the focus groups and during interviews, the data collected will not be released for public use except in aggregate statistical form.

11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private.

There are no questions of a sensitive nature.

12. Provide an estimate in hours of the burden of the collection of information.

Survey:

Total responses: 700 completed surveys

Completion time: 30 minutes

Total hours: 350 hours

Focus groups:

Total number: 60 participants

Completion time: 2 hours

Total hours: 120 hours

Interviews:

Total number: 15 participants

Completion time: 1 hour

Total hours: 15

Total respondents and responses: 775. Total hours: 485.

13. Provide an estimate of the total annual cost burden to the respondents or record-keepers resulting from the collection (excluding the value of the burden hours in Question 12 above).

There are no start-up, capital, or maintenance costs associated with this collection. No new or specialized equipment is needed to respond to this collection.

14. Provide estimates of annualized cost to the Federal government.

The estimated cost to the Government is a one-time cost of $44,882.45 for conducting the online survey and focus groups, developing the surveys and prototypes, entering the data from the survey and focus groups, analysis of the data and creation of a final report.

Any scientific publications will be developed by NWS staff as part of their regular federal duties.

15. Explain the reasons for any program changes or adjustments.

This is a new submission.

16. For collections whose results will be published, outline the plans for tabulation and publication.

All results from the survey and focus groups will be entered in a database using standard QA/QC procedures in survey research. The data will be analyzed using standard software (e.g. SPSS) and standard statistical procedures that are appropriate for survey data. Results from this collection may be used in scientific, management, technical or general informational publications, and would follow prescribed statistical tabulations and summary table formats.

Data from this survey may support research and analyses to be presented at appropriate professional meetings and may be submitted for publication in appropriate peer-reviewed journals.

17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons why display would be inappropriate.

NA.

18. Explain each exception to the certification statement.

NA.

7