NCSES - User Interview and Survey for Developing a Digital Science and Engineering Indicators

Science and Engineering Indicators (SEI)

Protocol for Personal and Telephone Interviews with SEI Users

Thank you for agreeing to participate in this survey on the Science and Engineering Indicators (SEI) report. The purpose of this survey is to understand your experience with and perception of the biennial Science and Engineering Indicators report issued by the National Science Board (NSB). The report is produced by the National Center for Science and Engineering Statistics (NCSES) of the National Science Foundation under the guidance of the NSB.

NCSES is exploring options for transitioning the Indicators report from a print publication with a secondary digital version to a digital publication that may or may not include a print version. Your input will help NCSES and the NSB make design decisions about the report. We are trying to learn more about how you use the Indicators report, your experience as an active user of the SEI, and how it could be made better. Please keep in mind that this study is focused on the report, and not on other NCSES datasets or publications (except as they are integrated into the Indicators report). Do you have any questions about the purpose or why we are conducting this interview?

[ADDRESS QUESTIONS AS APPROPRIATE, THEN PROCEED TO THE NEXT PARAGRAPH]

To facilitate the note taking, I will be recording our conversations today. For your information, everything we say is completely confidential, and only researchers on the project will have access to the audio files. In addition, please sign our human subjects consent form. Essentially, this document states that: (1) all information will be held confidential, (2) your participation is voluntary and you may stop at any time if you feel uncomfortable. Thank you for your agreeing to participate.

[START TAPE RECORDER]

A. Background – User Profile

1. Please describe your current position and the value of the Science and Engineering Indicators report to your professional community?

2. What other policy indicator data do you usually use? Do you utilize these resources before, after, or in conjuction with using the SEI? How do these other indicator data products compare to the SEI?

B. Using the SEI

3. Approximately how often do you use the report in a given year?

4. When was the last time you used the Science and Engineering Indicators report?

   1. What question were you trying to answer?

   2. How easy or difficult was it to find the answer you were looking for? Once you found it, how easy or difficult was it to understand and apply what you had found?

   3. How did you use the information that you found in the SEI report? (to support a position, to track progress against a goal, to educate, etc)

   4. What aspect of the report was most useful for finding your answer and understanding the indicator data? (Textual narrative, data tables, figures, etc)

   5. Are there any features that could be added to the report that would have helped you find your answer faster?

5. Was this fairly typical of your use of the Indicators report? If not, could you describe another more typical experience? What types of questions are you usually trying to answer when you refer to the report?

6. Have you ever had difficulty finding what you were looking for in the Indicators report? If so, what were you looking for? Did you ultimately find what you needed in the SEI, or did you have to look elsewhere?

7. How important is the textual narrative of indicator data presented in the SEI?

8. How often do you refer to the data tables presented in the SEI?

9. How often do you go to the primary source surveys from NCSES for data?

10. What attributes of the report makes it particularly valuable for your work?

11. What is the most difficult aspect of working with the SEI today? Do you have any ideas for how to make that aspect less difficult?

C. SEI Digital Future

12. Based on your experience with other digital documents or your own impressions, in what ways would making Indicators a primarily-digital document improve its usefulness and value to you? To your professional community?

13. Are you aware of any other reports or online resources that could serve as a good model for the SEI? What makes these reports good examples for SEI?

14. What features would you like to see in a new digital version of the SEI report? How could these new features enhance your use of the SEI report?

D. Questions specific to NSF staff:

15. In your opinion or experience, who is the target audience for the Indicators report? What sorts of questions are SEI users trying to answer?

16. What is your current involvement in the production of the Indicators report?

17. What parts of the current SEI production process work well?

18. What are the most difficult aspects, or key constraints, of the current Indicators report production process? (timing, data collection, approval processes, etc)

19. Do you think moving to digital production might improve the Indicators production workflow, or make it worse? How so?

20. From your perspective, what are the most important or immediate things that should be addressed during the transition process? (technical, organizational, staffing, training, etc.)