Consultations with Stakeholders and Data Users

Appendix L: Consultations with Stakeholders and Data Users

The Office of Petroleum and Biofuels Statistics (PBS) has conducted meetings to obtain feedback from internal and external stakeholders and data users. This has involved qualitative and quantitative techniques to evaluate energy programs and survey processes. These evaluations use several techniques, including interviews, customer surveys, focus groups, and facilitated group techniques to evaluated needs of stakeholders and data users.

These outreach activities involved internal and external customers and involved both formally and informal processes. Each of these activities solicited feedback regarding the PMP and associated surveys. Numerous recommendations were compiled, reviewed and prioritized in advance of redesign the 2013 PMP. A team of program and survey managers, methodologists, and subject matter experts reviewed and prioritized these recommendations during the pre-clearance activities. As a result of these efforts, over 40 recommendations were adopted for surveys in the 2013 Petroleum Marketing Program (PMP). Below is a list of activities:

Year	Consultations with Internal and External Stakeholders and Data Users
2012	EIA released two Federal Register Notices soliciting public comments on proposed changes to surveys in the PMP.
2012	PBS Mathematical Statistician presented paper, Achieving Information Quality via Continuous Quality Improvement at the 2012 Federal Committee on Statistics and Methodology (FCSM) Conference.
2011	Preclearance meetings between PBS and internal stakeholders and data users, including EIA representatives from the AEO, the AER, FOKS, GDFU, HOPU, NEMS, PMM, SEDS, STEO, TWIP, and WPSR to discuss impact of changes in program to EIA publications.
2011	The Office of Survey Development and Statistical Integration conducted an early-scope assessment and provided over 100 recommendations. This assessment was conducted by survey methodologists in the Office of Survey Development and Statistical Integration, interviewed survey managers and data analysts. This evaluation involved early-stage scoping interviews that used semi-structure protocols to understand substantive issues from the subject matters’ point of view.

Year	Consultations with Internal and External Stakeholders and Data Users
2011	Director of PBS gave presentation, Tough Choices in U.S. EIA's Data Programs at EIA’s Annual Energy Conference on proposed changes to the Petroleum Marketing Program and other programs.
2011	Office of Communications issued Press Release indicating select EIA surveys (including EIA-782B, 821 and 863) suspended due to budget cuts.
2011	Presentation at Kauffman Forum on Establishment Surveys by PBS Mathematical Statistician on internal and external evaluations of program
2010	Director of PBS gave presentation to National Association of State Energy Officials (NASEO)
2009	EIA produced a report, State Energy Data Needs Assessment, as the result of the 2009 State Energy Data Workshops. Numerous representatives from the State Energy Offices attended this event.
2007	Another previous initiative was development of program and survey templates to document and evaluated the EIA programs and surveys. A survey template was developed and used by a contractor to document and evaluate the PMP and associated surveys. The survey template compiled information on survey objectives, survey preparations (including pretests), sample design, data collection, data processing, estimation, and errors, both sampling and nonsampling errors. A separate report was prepared for each of the surveys. Likewise a program template was developed and used to document the program, integration information from the survey methodology reports.6
2005	A quality assessment was conducted of all of surveys, including surveys in both the PMP and PSP in 2006. Survey methodologists from the Statistics and Methods Group developed and conducted interviews with survey managers to prepare these quality self-assessments. Each of these reports contained summary information regarding survey methods, select measures regarding data quality, and identified what the survey manager determined was going well and potential improvements. The quality measures compiled included frame activity (e.g. number of births, deaths, and mergers), coverage rate, percent by mode of data collection, response rate by unit and volume, percent of forms failing one or more edits, relative standard error (if applicable) for key variables, difference between release date of publication and end of reference period, and revision of published key aggregate variable (percent change from first published to final release).

Page | 0

Appendix L: Consultations with Stakeholders and Data Users