Attachment 2. GSS COVID-19 Questions Cognitive Testing Report

Attachment 2. GSS COVID-19 Questions Cognitive Testing Report

1. BACKGROUND AND PURPOSE

This task served as a cognitive test for a set of newly-developed questions for the Survey of Graduate Students and Postdoctorates in Science and Engineering (GSS), sponsored by the National Center for Science and Engineering Statistics within the National Science Foundation and by the National Institutes of Health (NIH). The cognitive tests determine how well the questions work (i.e., are understood and are capturing intended data) when administered to institutional contacts familiar with the survey. This report summarizes results from the cognitive tests and, where appropriate, provides recommendations for changes to questions. An overview of participant recruitment strategies and interview procedures is presented first, followed by a presentation of findings and recommendations.

2. RECRUITMENT OF COGNITIVE INTERVIEW PARTICIPANTS

Twenty individuals participated in cognitive interviews. The participants were recruited from a group of 893 people who had provided data to the 2019 GSS survey. Recruitment included a two-step process, completed between September 8 and 25, 2020. Initially, a group of coordinators known to be knowledgeable, conscientious, and willing respondents were invited to participate. On September 16^th, a message was sent to all 893 coordinators on record regarding the start of the 2020 GSS survey cycle. Coordinators who responded to the message were added to a pool of possible participants; this list included 365 names when recruitment concluded. A total of 79 coordinators were invited to participate before the 20-participant threshold was reached. When selecting potential participants, staff worked to achieve an even distribution of the different coordinator roles, types of institutions, and geography. Invitations were distributed via email and, if necessary, a phone call and voicemail (see Appendix A for recruitment materials).

Table 1. Cognitive interview participant characteristics breakdown

Case #	Coordinator Type	Institution Type	Institution Region
P3290	Reports data on graduate students, postdocs, and NFR	Private not-for-profit, special focus four-year institution that includes a medical school or center. Offers postbaccalaureate degrees, including arts and sciences disciplines.	Northeast
P759	Reports data on graduate students, postdocs, and NFR	Public institution with very high research activity designation. Offers research doctoral degrees that include comprehensive programs as well as medical and veterinary degree programs.	Mountain West
P1093	Reports data on graduate students, postdocs, and NFR	Public institution with high research activity designation. Offers doctoral degrees which are dominated by professional degrees.	East Coast
P940	Reports data on graduate students, postdocs, and NFR	Public institution with very high research activity designation. Offers research doctoral degrees that include comprehensive programs as well as medical and veterinary degree programs.	Mid-West
P1302	Reports data on graduate students, postdocs, and NFR	Public institution with very high research activity designation. Offers research doctoral degrees that include comprehensive programs as well as medical and veterinary degree programs.	Mid-West
P1070	Reports data on graduate students, postdocs, and NFR	Public special focus institution that is a four-year medical school or center. Offers single program, research doctoral degrees as well as professional doctoral degrees.	Northeast
P1434	Reports data on graduate students, postdocs, and NFR	Public institution with very high research activity designation. Offers research doctoral degrees.	South
P1314	Reports data on graduate students, postdocs, and NFR	Public institution with very high research activity designation. Offers research doctoral degrees that include comprehensive programs as well as medical and veterinary degree programs.	Mid-West
P1380	Reports data on graduate students, postdocs, and NFR	Public institution with very high research activity designation. Offers research doctoral degrees that include comprehensive programs, but no medical or veterinary degrees.	South West
P1226	Reports data on graduate students, postdocs, and NFR	Public institution with very high research activity designation. Offers research doctoral degrees that include comprehensive programs as well as medical and veterinary degree programs.	East Coast
P877	Reports data on graduate students, postdocs, and NFR	Public institution that includes larger programs and awards master's and doctoral degrees. Only master's degrees are eligible for the GSS survey.	South
P959	Reports data on graduate students	Private institution with very high research activity designation. Offers research doctoral degrees that include comprehensive programs, including medical and veterinary degree programs.	West
P969	Reports data on graduate students	Public institution with very high research activity designation. Offers research doctoral degrees that include comprehensive programs as well as medical and veterinary degree programs.	West
P849	Reports data on graduate students	Private institution with very high research activity designation. Offers research doctoral degrees that include comprehensive programs, but no medical or veterinary degree programs.	Mid-West
P1097	Reports data on graduate students	Public institution with very high research activity designation. Offers research doctoral degrees that include comprehensive programs as well as medical and veterinary degree programs.	East Coast
P1503	Reports data on graduate students	Private not-for-profit, special focus four-year institution that includes a medical school or center. Offers postbaccalaureate degrees, including arts and sciences disciplines.	Mid-West
P1285	Reports data on postdocs and NFR	Private not-for-profit, special focus four-year institution that includes a medical school or center. Offers graduate degrees, including research doctoral degrees that are STEM dominant.	Mid-West
P932	Reports data on postdocs and NFR	Private not-for-profit special focus research and higher education institution.	Northeast
P1018	Reports data on postdocs and NFR	Public institution with very high research activity designation. Offers research doctoral degrees that include comprehensive programs, but no medical or veterinary degrees.	Northeast
P1119	Reports data on postdocs and NFR	Public institution with high research activity designation. Offers research doctoral degrees that are STEM-dominant.	West

3. INTERVIEW PROCEDURES

Zoom videoconferencing software was used to conduct all cognitive interviews. The interviewer provided a Zoom web link when the interview time and date was confirmed. At the start of the interview session, participants reviewed the purpose of the cognitive interview and signed an informed consent form. With consent given, participants logged in to the web instrument and began the survey. As participants completed the survey, they were instructed to read each question aloud, audibly think-through their answers, and then select the most appropriate response. Throughout this process, the interviewer noted responses and feedback as it was provided by the respondent.

Questions were divided into three sections: (a) expected capacity to complete the survey; (b) expected enrollment and financial support for graduate students; and (c) employment information regarding postdocs and doctorate-holding nonfaculty researchers. While all interviewees responded to section a, sections b and c were administered based on the type of data reported in the prior year. Table 1 notes the type of data reported by each coordinator.

As interviewees moved through the questions, the interviewer followed a protocol that contained a list of scripted probes and, when necessary, asked unscripted probes to inquire more deeply about issues raised by participants (e.g., can you tell me more about that?). See Appendix B for the cognitive interview protocol. After interviews were completed, all notes were compiled and analyzed to identify issues and suggest possible areas for improvement. The following sections provide a summary of findings and suggested recommendations.

4. GENERAL FINDINGS

Overall, the items worked well and as intended. Participants were able to understand and answer the questions posed. The timing of cognitive interviews did pose a challenge, as several interviewees had not yet finalized the data they plan to report. In such cases, the interviewee selected “don’t know” and then answered probes to confirm question comprehension as well as a more suitable timeline for data availability. The consensus among coordinators was that data would be available after November 1, 2020.

While the participants’ comprehension of the questions was consistent with the intended design, we recommend edits to eight of the questions. Recommendations include: the addition of clarifying text [A-2, C-2, C-3, C-6]; the adjustment of response options [B-2, B-3, C-3]; and the expansion [C1] as well as collapse [B-1] of items within a prompt. Note that we recommend two types of edits to question C-3.

5. ITEM SPECIFIC FINDINGS

Participants generally did not have issues responding to the questions. They were able to read through the items and respond with ease and little hesitation. When probed, participants asked for further clarification, though it was clear that their initial instincts were typically correct. That said, the review of comments and questions does note issues with some items. This section outlines findings for these questions and makes recommendations for improvement where appropriate. Tables 2 and 3 document the responses from each participant.

AVAILABLE WORKFORCE QUESTION A-1

Do you anticipate that the novel coronavirus (COVID-19) pandemic might impact your ability to complete the GSS 2020 survey because of decreased staff resources?

Response Options:

Choose one option – Yes, No, or Don’t know.

Findings:

The majority of participants answered “no” to this question. Three participants were unsure of the answer and selected “don’t know” [P759, P1302, P1434] and two answered “yes” [P969, P1285]. While many participants explained that the university has had furloughs, layoffs, and/or hiring freezes, the majority did not believe these would impact the ability to complete the GSS 2020 survey. Three participants confirmed that even staff working from home would have appropriate access to the data to complete the survey [P3290, P1503, P1285].

One participant wondered what “decreased staff resources” meant, but answered thinking about furloughs, layoffs, and people working from home [P1503]. Otherwise, there were no comprehension issues with the terms used in the question. Participants interpreted this question to be asking about staff resources that may have been impacted by furloughs, layoffs, and/or hiring freezes related to COVID19 who are directly responsible for completing or aiding with completing the GSS 2020 survey.

Those that answered “don’t know” are uncertain of the future and may not know about future events that would lead to decreased staff resources. P759 answered “don’t know” because she was thinking about future furloughs she may not be aware of, staff out sick, or staff pulled onto other projects that may take priority over the GSS 2020. P1302 was concerned about impacts on collecting the data due to staff working from home because of COVID19 and answered, “don’t know”. P1434 answered “don’t know” to be conservative because they were uncertain if COVID related initiatives may be prioritized over day to day work, including the GSS 2020 survey.

Recommendations:

There do not seem to be any comprehension issues or difficulties with this question, and we do not recommend any changes.

AVAILABLE WORKFORCE QUESTION A-2

Do you anticipate that the novel coronavirus (COVID-19) pandemic might impact your ability to complete the GSS 2020 survey because of issues regarding data access?

Response Options:

Choose one option – Yes, No, or Don’t Know.

Findings:

The majority of participants answered “no” to this question and indicated that there should be no issues regarding data access. Only two participants selected “don’t know” [P1302, P1434] and only one selected yes [P1285]. Those that answered “no” indicated that the data could be accessed remotely for staff working at home or staff were in the office and could access data as they normally would. The participants that selected “no” had no concerns about completing the GSS 2020 survey due to data access issues.

Two respondents included data quality concerns in their responses to the question. P1302, who answered “don’t know”, was uncertain about the quality of the data due to staffing issues in other departments because of a COVID hiring freeze. They are concerned that the data will be accessible, but not great quality due to less frequent updates to the database. P1285 answered “yes” because she has concerns about the quality of the data. P1285 confirmed the data will be accessible but is uncertain of the quality.

P1434 answered “don’t know, which was also selected for question A1, because of concerns about the prioritization of COVID related work over other tasks. P1434 selected “don’t know” in order to be conservative.

Findings:

A few participants are including data quality concerns in their answers for this question. To limit the number of respondents including data quality in their answers, it may be helpful to add an additional instruction after the question such as “Please think only about accessing the data and not the quality of the data”.

STUDENT DATA QUESTION B-1

This fall, did any graduate programs...

1. Intentionally reduce the number of newly admitted graduate students?

2. Suspend enrollment of new graduate students?

3. Eliminate new graduate enrollment?

4. If yes to any, please describe the changes.

Response Options:

For prompts A-C, choose one option – Yes, No, Don’t Know. Prompt D is an open-ended question where respondents can explain any changes to graduate program admission or enrollment policies.

Findings:

Of the sixteen participants who responded to this question, a majority responded “no” to each item, no respondent answered “yes” to any item. The item-specific breakdown is as follows: (A) ten “no” and 6 “don’t know; (B) eleven “no” and five “don’t know”; and (C) thirteen “no” and three “don’t know”.

Further questions suggest that participants chose “don’t know” due to the timing of the survey. Participants did not yet have access to relevant data [P959, P940] or had not yet had the time to review the available data and so could not confidently answer the questions [P3290, P849]. In each case, respondents expressed confidence in having answers to the questions by early-November 2020. Among interviewees that responded “no”, the consensus was that institutions had attempted to maintain, or increase, enrollment for Fall 2020 [P1434, P1503, P1302, P1093].

Recommendations:

Overall, the question appears to be well-understood by the participants. Because the responses to prompts A & B were paired in nearly every case, we recommend collapsing the two prompts. This will reduce respondent burden without impacting the quality of data collected by the question. The revised question prompt is:

This fall, did any graduate programs…

a. intentionally reduce, or suspend, the number of newly admitted graduate students? ___Yes __No __Don’t know

c. eliminate new graduate enrollment? __Yes __No __Don’t know

STUDENT DATA QUESTION B-2

Did your institution change the definition of full-time or part-time graduate enrollment this fall?

1. If yes, please describe the changes.

Response Options:

Choose one option – Yes or No. Prompt A is an open-ended question where respondents can explain any changes to the definition of student statuses.

Findings:

All sixteen responses to this question were “no”. In all but one case, participants clearly understood the question and were unambiguous in their response: “no, definition did not change” [P940, P1434, P1226, P959]. The conversation with respondent P1097 did uncover some uncertainty regarding this question. The participant expected that no changes had been made but was not certain. The question did not include a “don’t know” option, and so chose the “no” response option.

Recommendations:

While the question was widely understood, one interviewee was not certain of the answer. As this is not information that would be included in a data output, we recommend adding a “don’t know” option to this question.

STUDENT DATA QUESTION B-3

This fall, have any doctoral programs at your institution changed policies regarding...

1. Time-to-degree extensions for doctoral students?

2. Length of time doctoral students can receive financial support?

3. If yes to either, please describe the changes that were made.

Response Options:

For each prompt, choose one option – Yes, No, Don’t Know. Prompt C is an open-ended question where respondents can explain any changes to policies regarding time-to-degree or the length of time students can receive financial support.

Findings:

Fourteen of the interviewees report doctoral students to the GSS survey and thus were able to respond to this question. All but one set of responses to this question were paired (i.e., “no”/”yes”/”don’t know” to both). The lone exception was P1434, who reports a case-by-case time-to-degree extension, but no concurrent change in financial support. A majority of responses indicate that no decision has been made at the institutional level, instead any extension-related policies have been left to individual departments or academic units. Unfortunately, the participants selected each of the response options to communicate that such decisions were left to the unit, or department, level. Examples of the three response options capturing the same data include:

• P1203: “No policy change. Handle [extensions] on a case by case basis when they occur.”

• P1302: “Yes policy has changed. No single guideline applied to all students. Instead, departments have latitude to accommodate students.”

• P1314: “Don’t know about policy changes. Decisions occur at the department level and [departments] do not necessarily need to report the changes.”

Recommendations:

Respondents were clearly confused by this question, as the interview uncovered a shared set of motivations and explanations being associated with three different response options. In an attempt to add clarity, we recommend a change to the response format:

• Yes, formal change at the institution level

• Yes, informal change at the department level

• No

• Don’t know

STUDENT DATA QUESTION B-4

This fall, have enrollments increased, stayed about the same, or decreased for...

1. Master’s students who are US citizens or permanent residents?

2. Master’s students who are temporary visa holders?

3. Doctoral students who are US citizens or permanent residents?

4. Doctoral students who are temporary visa holders?

Response Options:

For each prompt, choose one option – Increased, Stayed about the same, Decreased, or Don’t know.

Findings:

Of the sixteen responses logged for this question, seven interviewees reported “don’t know” for at least three of the prompts. The two interviewees that only responded to items regarding master’s students both reported “don’t know” for both prompts [P3290, P1503].

Notes from cognitive interviewers indicate that “don’t know” is an artifact of timing. Participants who chose this option expect the ability to answer these questions later in the Fall 2020 semester [P3290, P849, P959]. In cases where respondents were able to confidently report that enrollment levels had “increased”, “decreased”, or “stayed the same” typically paired the answers by citizenship. For example, P969 reports decreased levels of international master’s and doctoral students due to travel restrictions and issues faced when obtaining student VISAs. In a similar manner, P1380 reports that enrollment stayed the same from 2019 to 2020 for both master’s and doctoral students who were either U.S. citizens or permanent residents.

Recommendations:

We do not recommend any changes or edits for this question. Responses suggest that interviewees understood the question as it was constructed. This understanding is manifest either through direct answers, or through explanations as to how the answers will be available later in the Fall 2020 semester.

STUDENT DATA QUESTION B-5

This fall, has financial support for graduate students increased, stayed about the same, or decreased for...

1. Master’s students?

2. Doctoral students?

Response Options:

For each prompt choose one option – Increased, Stayed about the same, Decreased, or Don’t know.

Findings:

Fourteen interviewees responded to both prompts, all but one of which paired their responses. The lone exception was P959, who reported that master’s support decreased while support for doctoral students increased. In this case, participant P959 could confidently report that financial resources are apportioned separately for the two-types of students.

The other thirteen responses suggest that participants considered graduate student financial support together, regardless of degree type. For example, P877 reports that the institutional budget has not changed and so expects funding to remain consistent for both types of students.

One participant, P969, reports that funding for both types of students decreased from Fall 2019 to Fall 2020. Seven report that financial support stayed the same for both types of students [P877, P1380, P1434, P1070, P1302, P1093, P759]. Six participants report that they don’t yet know the status of financial support [P940, P1226, P969, P849, P1097, P1503]. The participants who reported “don’t know” expected to have answers to the questions later in the Fall 2020 semester.

Recommendations:

We do not recommend any changes or edits for this question. Responses suggest that interviewees understood the question as it was constructed. This understanding is manifest either through direct answers, or through explanations as to how the answers will be available later in the Fall 2020 semester.

STUDENT DATA QUESTION B-6

Are there other novel coronavirus (COVID-19) pandemic impacts that may affect graduate student enrollment or financial support at your institution this fall?

1. If yes, what are they?

Response Options:

Choose one option – Yes or No. Prompt A is an open-ended question where respondents who answered “yes” can describe any other impacts that the survey did not capture.

Findings:

Eleven of the interviewees answered “yes” to this question, but all sixteen provided further feedback. Upon review of the comments, it became clear that respondents wanted to underscore the impact that VISA issues will have regarding international student enrollment [e.g., P3290, P940, P1314, P1380, P959, P969, P1503]. While outside the bounds of this question, feedback also underscores the range of staffing concerns that might impact the ability to collect and report data in a timely manner.

Recommendations:

We do not recommend any changes to this question or any changes to other questions as a result of this feedback. The open-ended responses available here underscores that the prior questions capture the main issues on the mind of interviewees – specifically complications surrounding study VISAs for international students. The question also has the capacity to capture enrollment issues unique to a particular institution.

POSTDOC AND NFR DATA QUESTION C-1

Have the number of postdocs at your institution this fall increased, stayed about the same, or decreased?

Response options:

Choose one option – Increased, Stayed about the same, Decreased, or Don’t know.

Findings:

Six participants answered “don’t know” to this question [P3290, P1093, P940, P1314, P1380, P1119]. Four of which indicated they would not know until after the census is complete [P3290, P940, P1380, P1119]. One participant does not have access to the data [P1093].

Another six participants answered “stayed about the same” [P759, P1302, P1070, P1434, P1226, P1018, P932]. One of which indicated that although they’ve been told the number of postdocs has ‘stayed about the same’, they don’t access the data themselves but are dependent on department representative to pull the data [P1070]. Two of these participants mentioned issues international students on VISAs have experienced. One indicated their answer of “stayed about the same” undermines issues international students on VISAs had with returning [P1434]. While another participant indicated they expect the number of postdocs to decrease due to VISA issues [P1018]. This same participant also mentioned NFRs were offered a separation package. Although this participant answered “stayed about the same” the comments imply the number of postdocs may have decreased.

One participant’s survey item response was “decreased” although commented that they expect the numbers have “stayed the same” [P932]. This participant cited postdocs ‘stayed on longer than expected due to limited opportunities available’ and ‘these extended positions filled holes caused by postdocs who were unable or unwilling to accept a position, typically due to VISA issues’.

One other participant answered “increased” citing Principal Investigators hiring former graduate students to provide them a job in a “bleak economy” [P1285].

Recommendations:

Comments captured by interviews demonstrate that participants understood the question and were able to identify a number of factors that may influence the number of postdocs at the institution. The factors often involved issues faced by employees, or would-be employees, who require temporary visas to work in the U.S. Participants note a similar situation among student enrollment, whereby U.S. citizens and permanent residents faced different circumstances than students who are temporary visa holders. For this reason, we recommend separating the response options to collect postdoc data by citizenship. The suggested revision is as follows:

Have the number of postdocs at your institution this fall increased, stayed about the same, or decreased?

a. postdocs who are US citizens or permanent residents?

__Increased __Stayed about the same __Decreased __Don’t know

b. postdocs who are temporary visa holders?

__Increased __Stayed about the same __Decreased __ Don’t know

POSTDOC AND NFR DATA QUESTION C-2

This fall, have any programs at your institution extended the number of years postdocs are allowed to receive funding?

Response Options:

Choose one option – Yes, No, or Don’t Know.

Findings:

Eight participants answered “don’t know” to this question [P3290, P1093, P940, P1302, P1314, P1380, P1285, P1119]. Two of these participants mentioned they would know more after the census [P3290, P1302]. Five of these participants indicated they either do not have access to the data or the informational discussions [P1093, P940, P1380, P1285, P1119]. One of these participants indicated the use of the word “allowed” is ambiguous because, “it’s not the institution level, instead decided by the Principal Investigator and projects and departments” and therefore this question isn’t easy to answer [P1119].

Two participants answered “No” to this question [P759, P1070]. One indicated hearing no mention from relevant officials/offices [P759]. Another participant mentioned there are ‘no formal limits from the institution on the number of years’ so there’s no change’ [P1070].

Four participants answered “Yes” to this question [P1434, P1226, P932, P1018]. One indicated although they answered “yes” this is handled on a ‘case by case basis’ [P1434]. One participant selected the survey response item “no” but commented “yes, the level had been extended”. This participant reported it’s handled on a ‘case by case basis’ and there have been ‘an additional 6-12 months made available to the postdocs already on staff’ [P932]. Another mentioned no limitation on postdoc funding [P1018].

Recommendations:

While interview responses indicate that participants were able to clearly understand this question, we do recommend a slight change to the question wording. Specifically, P1119 commented that the term “allowed” made the question ambiguous. We recommend changing the wording of this question as follows to avoid any ambiguity: This fall, have any programs at your institution extended the number of years postdocs can receive funding.

POSTDOC AND NFR DATA QUESTION C-3

This fall, has your institution extended appointments for postdocs in response to the novel coronavirus (COVID-19) pandemic?

Response Options:

Choose one option – Yes, No, or Don’t Know.

Findings:

Eight participants answered “don’t know” to this question [P3290, P1093, P940, P1302, P1070, P1314, P1380, P1119]. One of these participants indicated they would know after the census [P3290]. Five of these participants indicated not having access to this information, but would rely on others [P1093, P940, P1070, P1314, P1380]. One of these shared that even with information from others it ‘might be difficult to know because the motivation is unclear and would occur on a 1-1 basis determined by the project director or department’ [P1070]. Two participants indicated it would be determined by the ‘sponsored project’ or ‘department’ [P1070, P1119].

Two participants answered “no” to this question [P759, P932]. One participant indicated they cancelled offers ‘to most positions and assumes carry over to PDs and NFRs and new positions had to be deemed essential’ [759]. One respondent mentioned this question was closely related to question 2 with ‘no difference in the responses between the two questions’ [P932].

Four participants answered “yes” [P1434, P1226, P1285, P1018]. One of these participants indicated, ‘furloughs and involuntary reductions in time and layoffs did not affect the PDs’ [P1434]. Another mentioned, some foreign ‘PDs ended appointments early to leave the U.S. and return to their home countries when the pandemic started’ [P1226]. Another participant, although answered “yes” indicated they would need to follow-up with each department and expects variation by department. This participant also commented that it may be too early in the semester to know for sure (census) [P1285].

Recommendations:

The comments logged for this question indicate two points: (a) participants generally misunderstood the question; and (b) participants were unable to differentiate the act of extending appointments (question C-3) and the act of extending years of available funding (question C-2). With this feedback in mind, we suggest that further clarification is needed for this question. “Extended appointments” refers to the fact that postdocs are typically limited-term job opportunities, in which a single individual cannot serve for more than a designated number of years (i.e., 5-7). We recommend two actions for this question. First, the addition of some clarifying text before the question; and second the inclusion of a “does not apply” response option. The revised question is as follows:

Most institutions have policies that limit the number of years someone can serve as a postdoc. This fall, has your institution extended appointments for postdocs in response to the novel coronavirus (COVID-19) pandemic?

__Yes __No __Don’t know __Does not apply, length of postdoc appointments not limited

POSTDOC AND NFR DATA QUESTION C-4

This fall, has your institution had to hold off or cancel offers to any new postdocs?

Response Options:

Choose one option – Yes, No, or Don’t Know.

Findings:

Five participants answered “don’t know” to this question [P3290, P1093, P1070, P1380, P1119]. Two of these participants indicated they would know either after the census or if it becomes available [P3290, P1070]. Another participant doesn’t have access to the data [P1093]. Five participants answered “no” to this question [P759, P1314, P1434, P1285, P932]. One participant with access to the data but isn’t familiar with postdoc policies and decision and just enters the data for GSS [P1314]. Four participants answered “yes” to this question [P940, P1302, P1226, P1018]. Three of these participants indicated the institution had to hold off or cancel offers as a result of travel and VISA related issues [P940, P1226, P1018]. One participant reported a hiring freeze that’s impacted both PDs and NFR [P1302]. Four participants survey item responses were not consistent with comments and changed their initial survey item response [P1434, P1314, P932, P1018].

Recommendations:

We do not recommend any changes to this question or any changes to other questions as a result of this feedback. The participants’ comments and reactions to this question demonstrate an understanding of the prompt and how their data could be used to answer the question – whether at the time of the interview or a time in the future when relevant data would be available. While the “thinking aloud” process of the interview caused four participants to change their initial response, no single motivation or thought process was cited among the group regarding the change. For this reason, we do not have any recommendations to give regarding the question.

POSTDOC AND NFR DATA QUESTION C-5

This fall, have any of the following been affected by hiring freezes, furloughs, involuntary reductions in time, or layoffs?

1. Postdocs?

2. Doctorate-holding nonfaculty researchers (NFRs)?

Response Options:

For prompts A and B, choose one option – Yes, No, Don’t Know.

Findings:

Four participants answered “don’t know” to this question for postdocs [P3290, P1093, P1434, P1380] while one additional answered “no” for NFRs [P1285]. One of these participants indicated they would know after the census [P3290]. One participant doesn’t have access to the data [P1093].

Five participants answered “no” to this question for postdocs [P759, P1070, P1314, P932, P1018] while one additional answered “no” for NFRs [P1226].

Five participants answered “yes” to this question [P940, P1302, P1226, P1285, P1119] while one additional answered “yes” for NFRs [P1018]. Two participants indicated “yes” for postdocs only [P1226, P1285]. A ‘lot of employees furloughed, including postdoc positions’ but those willing and able were rehired and back at work’ [P1285]. Another participant indicated ‘12-month employees were furloughed’ while 9-month employees were not [P1119].

Recommendations:

No recommended changes or edits to this question. All participants seemed to understand the question being asked and clearly addressed it.

POSTDOC AND NFR DATA QUESTION C-6

Are there other novel coronavirus (COVID-19) pandemic impacts that may affect your institutions’ postdoc or NFR data?

1. If yes, what are they?

Response Options:

Choose one option – Yes or No. Prompt A is an open-ended question where respondents who answered “yes” can describe any other impacts that the survey did not capture.

Findings:

Two participants answered “don’t know” to this question [P3290, P1380]. One participant mentioned VISA issues would likely be an issue, while the other mentioned experiencing VISA issues prior to Covid-19 [P3290, P1380, respectively]. A third participant indicated the question does not explicitly ask about access to data or actual data [P3290]. One participant answered “yes” to this question [P759]. This participant commented that the report won’t be impacted but the ‘experience will look different’. One participant answered “no” to this question [P1119], suggesting instead that the prior questions captured the relevant impacts that the coronavirus pandemic may have on postdoc and NFR data.

Recommendations:

The interviews captured a bit of confusion for this question, as at least one respondent was unsure whether the question assessed access to data, or the quality of that data. The questions in Section A capture information about access to data, which means this question requires some clarification that it refers only to the quality and composition of postdoc and NFR data. Specifically, we recommend rewording the question to: Are there other novel coronavirus (COVID-19) pandemic impacts that may affect the quality of your institution’s postdoc or NFR data?

6. QUESTION RESPONSES BY PARTICIPANT

Table 2. Responses to Group A and Group C Questions.

Participant	Question		Group C Questions
	A1	A2	C1	C2	C3	C4	C5a	C5b	C6
P3290	No	No	Don't Know	Don't Know	Don't Know	Don't Know	Don't Know	Don't Know	Yes
P759	Don't Know	No	Stayed Same	No	No	Yes	Yes	Yes	No
P1093	No	No	Don't Know	Don't Know	Don't Know	Don't Know	Don't Know	Don't Know	Yes
P940	No	No	Don't Know	Don't Know	Don't Know	Don't Know	Don't Know	Don't Know	Yes
P1302	Don't Know	Don't Know	Stayed Same	Don't Know	Don't Know	Yes	Yes	Yes	No
P1070	No	No	Stayed Same	No	Don't Know	Don't Know	No	Don't Know	No
P1434	Don't Know	Don't Know	Stayed Same	Yes	Yes	Yes	Yes	Don't Know	No
P1314	No	No	Don’t Know	Don’t Know	Don’t Know	Don’t Know	Don’t Know	Don’t Know	No
P1380	No	No	Don't Know	Don't Know	Don't Know	Don't Know	Don't Know	Don't Know	No
P1226	No	No	Stayed Same	Yes	Yes	Yes	Yes	No	Yes
P877	No	No	-	-	-	-	-	-	-
P959	No	No	-	-	-	-	-	-	-
P969	Yes	No	-	-	-	-	-	-	-
P849	No	No	-	-	-	-	-	-	-
P1097	No	No	-	-	-	-	-	-	-
P1503	No	No	-	-	-	-	-	-	-
P1285	Yes	Yes	Increased	Don't Know	Yes	No	Yes	Don't Know	No
P932	No	No	Decreased	No	No	Yes	No	No	Yes
P1018	No	No	Stayed Same	Yes	Yes	No	No	Yes	Yes
P1119	No	No	Don't Know	Don't Know	Don't Know	Don't Know	Yes	Yes	No

Table 3: Responses to Group B Questions.

Participant	Group B Questions
	B1a	B1b	B1c	B2	B3a	B3b	B4a	B4b	B4c	B4d	B5a	B5b	B6
P3290	Don't Know	No	No	No	-	-	Don't Know	Don't Know	-	-	Don't Know	-	Yes
P759	No	No	No	No	No	No	Stayed Same	Stayed Same	Stayed Same	Stayed Same	Stayed Same	Stayed Same	Yes
P1093	No	No	No	No	No	No	Increased	Don't Know	Don't Know	Don't Know	Stayed Same	Stayed Same	Yes
P940	Don't Know	Don't Know	Don't Know	No	Don't Know	Don't Know	Don't Know	Don't Know	Don't Know	Decreased	Don't Know	Don't Know	Yes
P1302	No	No	No	No	Yes	Yes	Increased	Stayed Same	Increased	Stayed Same	Stayed Same	Stayed Same	No
P1070	No	No	No	No	No	No	Stayed Same	Don't Know	Decreased	Decreased	Stayed Same	Stayed Same	No
P1434	No	No	No	No	Yes	No	Increased	Don't Know	Don't Know	Don't Know	Stayed Same	Stayed Same	No
P1314	Don’t Know	Don’t Know	Don’t Know	No	Don’t Know	Don’t Know	Stayed Same	Decreased	Stayed Same	Decreased	Don’t Know	Don’t Know	Yes
P1380	No	No	No	No	Don't Know	Don't Know	Stayed Same	Decreased	Stayed Same	Stayed Same	Stayed Same	Stayed Same	No
P1226	Don't Know	Don't Know	No	No	Don't Know	Don't Know	Don't Know	Don't Know	Don't Know	Don't Know	Don't Know	Don't Know	Yes
P877	No	No	No	No	Don't Know	Don't Know	Stayed Same	Decreased	Stayed Same	Stayed Same	Stayed Same	Stayed Same	No
P959	Don't Know	Don't Know	No	No	No	No	Don't Know	Don't Know	Don't Know	Don't Know	Decreased	Increased	Yes
P969	No	No	No	No	Yes	Yes	Stayed Same	Decreased	Stayed Same	Decreased	Decreased	Decreased	Yes
P849	Don't Know	Don't Know	No	No	Yes	Yes	Don't Know	Don't Know	Don't Know	Don't Know	Don't Know	Don't Know	Yes
P1097	No	No	Don't Know	Don't Know	Yes	Yes	Don't Know	Don't Know	Don't Know	Don't Know	Don't Know	Don't Know	Yes
P1503	No	No	No	No	-	-	Don't Know	Don't Know	-	-	Don’t Know	Don’t Know	Yes
P1285	-	-	-	-	-	-	-	-	-	-	-	-	-
P932	-	-	-	-	-	-	-	-	-	-	-	-	-
P1018	-	-	-	-	-	-	-	-	-	-	-	-	-
P1119	-	-	-	-	-	-	-	-	-	-	-	-	-

Appendix A. Respondent Recruitment Materials

Email Recruitment Invitation

Subject Line: Invitation to Test Possible COVID-related Questions for NSF-NIH GSS

Dear <Coordinator>:

We are contacting you today to ask for your help in developing a few new items related to the impact of the coronavirus pandemic on your campus. These items may be added to the NSF-NIH Survey of Graduate Students and Postdoctorates in Science and Engineering (GSS) to help us understand changes at institutions and our coordinator’s ability to participate in the GSS. We are hoping that you will assist us in testing these questions prior to them being added to the 2020 GSS Data Collection. The testing of the questions will involve you not only answering the questions as you would during the actual data collection, but also answering questions from an interviewer about your understanding of items so that we can modify them to make them more relevant and meaningful. If you choose, we can retain these responses so you will not be required to complete them again when the survey officially launches.

We are hoping to conduct these interviews in the coming weeks. The interviews will take between 30-40 minutes to complete. If you are interested in participating, please reply to this email as soon as possible so we can set-up a day and time that is most convenient for you.

We appreciate your consideration and look forward to hearing from you.

Sincerely,

Jennifer Pauli
GSS Data Collection Manager
866-558-0781

Phone Call Recruitment Invitation Script

Hi this is ____ _______ calling on behalf of the NSF NIH Survey of Graduate Students and Postdocs in Science and Engineering. We have begun the process of adding questions to the 2020 survey – the additional questions will assess the impact of the coronavirus pandemic on institutions as well as coordinator’s ability to participate in the 2020 survey.

Would you be willing to participate in the testing of these questions? The testing process includes a 30‑40 minute interview during which you will review the questions and the interviewer will collect your answers. If so, could you please let me know your availability in the coming weeks?

Thank you.

Voicemail Invitation Recruitment Script

Hi this is ____ _____ calling on behalf of the NSF and NIH GSS survey. We have begun the process of adding questions to the 2020 survey – the additional questions will assess the impact of the coronavirus pandemic on institutions as well as coordinator’s ability to participate in the 2020 survey. We are asking for your help in testing these questions.

If you are willing to participate in this testing, please email us at [email protected] or call back at 866-558-0781 and we will work to schedule an appointment in the coming weeks.

Thank you.

Appendix B. Respondent Recruitment Materials

Protocol:

• We will interview a convenience sample of nine GSS School Coordinators. They will be selected to represent the diversity of the coordinators and the institutions such as both large and small schools and those who use and do not use unit coordinators.

• All cognitive interviews will be conducted virtually via a Zoom conference call. Participants will be asked to log in using a Zoom web link for the interview. The interviewer will read or paraphrase the introduction to the cognitive interview. Then, interviewer will share their screen with the informed consent PDF. they will be asked to go over the informed consent. Respondents will be asked to sign the informed consent. Upon getting the participant’s consent, we will start recording the session. Interviewers will ask once again for the participant’s consent for the survey and to be recorded. We want a record of their consent.

• Participants will be asked to read the instructions and questions aloud and then “think aloud” as they answer each question. We will use both concurrent think-aloud and retrospective probing to understand the response process and how participants understand the questions.

• We anticipate the interview process (including completion of the survey) to take 30-40 minutes.

• Survey link: https://gss2020stage.rti.org

• Coordinators login credentials (username and password). Refer to the first 2 worksheets: \\rtints23.rti.ns\gss\Data_Collection_2020\Communications\COVID_Onboarding_alternates.xlsx

INTRODUCTION

READ OR PARAPHRASE THE INTRODUCTION.

Hello, I’m [NAME] with [RTI], a survey research company, and I am talking to you on behalf of the National Center for Science and Engineering Statistics (NCSES) for the Survey of Graduate Students and Postdoctorates in Science and Engineering (GSS). Thanks for agreeing to help us today. Let me tell you a little about what we will be doing:

NCSES is considering the addition of new questions on the impact of the novel coronavirus 2019 (COVID-19) pandemic on institutions—whether there will be complications in completing the GSS survey, how the coronavirus pandemic is affecting enrollment and financial support for graduate students and hiring and financial support for postdocs and doctorate-level non-faculty researchers (NFRs). In order to produce the best information possible, NCSES is conducting this evaluation to ensure that the new questions and the associated response options adequately capture the experiences of coordinators and institutions. It’s important that the questions make sense, are easy to answer, and that everyone understands the questions in the same way.

If you agree to take part in this study, I will display a questionnaire for you to answer. I will ask you to read the instructions and questions aloud and then “think aloud” as you answer each question. This may feel a little unnatural, but it will help us understand how you think about and answer each question. Here’s an example:

Question: How many bedrooms are there in your house?

Answer: What do you mean by bedrooms—are these just where people sleep? If so, I have 2 bedrooms. But if we are counting anything with a closet and a window, we also have a study office. So perhaps, we have 3 bedrooms. I’m going with where people sleep, so 2 bedrooms.

At certain points in the survey, I may ask you to answer some additional questions about the survey question itself and/or your answer. I’d also like you to point out any questions or answer choices that might be confusing. The purpose of our conversation today is not for me to know the answers to the survey questions, but to learn from you whether the survey questions are worded clearly and, if they are not, what changes should be made. I want you to know that I didn’t write the survey questions, so I won’t be offended if you criticize them. Your honest feedback is very important. If something is confusing, it means that the writer didn’t do a good job writing the question and/or instructions.

Your participation in this interview is very important because it will help NCSES improve the GSS Survey and we will know more about the impact of the coronavirus pandemic on coordinators and institutions.

[SHARE INFORMED CONSENT FORM ON ZOOM AND GET RESPONDENT’S SIGNATURE]

COGNITIVE INTERVIEWING

[IF CONSENT WAS GIVEN, START RECORDING. COLLECT VERBAL CONSENT FOR RECORDING.]

COVID-19 Impact - Part A

The first two questions will help us understand the challenges you may face when completing the GSS this year. (All coordinators)

1. Do you anticipate that the novel coronavirus (COVID-19) pandemic might impact your ability to complete the GSS 2020 survey because of decreased staff resources?

__Yes __No __Don’t know

Probe A1a: What did you consider when you were answering this question?

Probe A1b: [IF RESPONSE TO PROBE01A DO NOT PROVIDE SUFFICIENT DETAIL, ASK] Can you tell me more about who or what you considered as part of the staff resources?

2. Do you anticipate that the novel coronavirus (COVID-19) pandemic might impact your ability to complete the GSS 2020 survey because of issues regarding data access?

__Yes __No __Don’t know

Probe A2a: What does “issues regarding data access” mean to you?

Probe A2b: [IF R ANSWERED NO AND IF RESPONSE TO PROBEA2A DO NOT PROVIDE SUFFICIENT DETAIL, ASK] Do you anticipate using a workaround to complete the GSS 2020? IF YES, please explain.

COVID-19 Impact – Part B (Students)

The remaining questions ask about possible impacts of the novel coronavirus (COVID-19) pandemic may have on your institution this fall.

In response to the novel coronavirus (COVID-19) pandemic developments, has your institution’s graduate student policies, enrollments or funding been impacted in any of the following ways?

1. This fall, did any graduate programs…

a. intentionally reduce the number of newly admitted graduate students? __Yes __No __Don’t know

b. suspend enrollment of new graduate students? __Yes __No __Don’t know

c. eliminate new graduate enrollment? __Yes __No __Don’t know

1d. If yes to any, please describe the changes

_____________________________________________________________________________________

Probe B1a: [IF RESPONSE TO ANY 1a-1c IS NOT DON’T KNOW, ASK] How did you determine the final tally of enrolled graduate students? That is, what did you use as a reference for your responses to questions 1a-1c.

Probe B1b: [IF RESPONSE TO PROBEB1a DO NOT PROVIDE SUFFICIENT DETAIL, ASK] Can you tell me more about the changes in enrollment?

Probe B1c: Are there any other enrollment issues that we have not captured?

2. Did your institution change the definition of full-time or part-time graduate enrollment this fall?

__Yes __No

2a. If yes, please describe the changes.

_____________________________________________________________________________________

Probe B2: [IF RESPONSE TO 2 is YES AND RESPONSE TO 2a IS NOT SUFFICIENT, ASK] How did the definition change?

3. This fall, have any doctoral programs at your institution changed policies regarding…

a. time-to-degree extensions for doctoral students? __Yes __No __Don’t know

b. length of time doctoral students can receive financial support? __Yes __No __Don’t know

3c. If yes to either, please describe the changes that were made.

_______________________________________________________________________________

Probe B3a: [IF RESPONSE TO 3a is YES AND RESPONSE TO 3c IS NOT SUFFICIENT, ASK] How did the time-to-degree extension policies change for the doctoral students?

Probe B3b: [IF RESPONSE TO 3b is YES AND RESPONSE TO 3c IS NOT SUFFICIENT, ASK] How did the policy regarding the length of time doctoral students can receive financial support change?

4. This fall, have enrollments increased, stayed about the same, or decreased for…

a. master’s students who are US citizens or permanent residents?

__Increased __Stayed about the same __Decreased __Don’t know

b. master’s students who are temporary visa holders?

__Increased __Stayed about the same __Decreased __ Don’t know

c. doctoral students who are US citizens or permanent residents?

__Increased __Stayed about the same __Decreased __ Don’t know

d. doctoral students who are temporary visa holders?

__Increased __Stayed about the same __Decreased __ Don’t know

Probe B4a: [IF RESPONSE TO ANY 4a-4d IS NOT DON’T KNOW, ASK] How did you determine the breakdown of the characteristics of the enrolled graduate students? That is, what did you use as a reference for your responses to questions 4a-4d.

Probe B4b: [IF RESPONSE TO ANY 4a-4d IS DON’T KNOW, ASK] You said “Don’t know”. Is that because enrollment data isn’t available yet or you don’t have access to it or for some other reason? When would enrollment data typically be available?

5. This fall, has financial support for graduate students increased, stayed about the same, or decreased for…

a. master’s students: __Increased __Stayed about the same __Decreased __Don’t know

b. doctoral students: __Increased __Stayed about the same __Decreased __Don’t know

Probe B5a: What did you consider as part of “financial support” for the graduate students?

Probe B5b: [IF RESPONSE TO 5a or 5b IS NOT DON’T KNOW, ASK] How did you determine the change in financial support for the graduate students? That is, what did you use as a reference for your responses to questions 5a-5b.

Probe B5c: [IF RESPONSE TO 5a or 5b IS DON’T KNOW, ASK] You said you don’t know the answer to this question. Is that because source data about financial support are not available yet or not final or some other reason? Can you tell me more about the reasons behind your answer?

6. Are there other novel coronavirus (COVID-19) pandemic impacts that may affect graduate student enrollment or financial support at your institution this fall?

__Yes __No

6a. If yes, what are they? _____________________________________________________________________________________

Probe B6a: What were you thinking about when you answered this question?

Probe B6b: [IF R ANSWERED YES AND IF NECESSARY] How prevalent would you say this other experience would be to other institutions?

COVID-19 Impact – Part B (Postdocs/NFRs)

The next set of questions concern impacts on postdocs and doctorate-level non-faculty researchers (NFRs) stemming from the novel coronavirus (COVID-19) pandemic this fall.

1. Have the number of postdocs at your institution this fall increased, stayed about the same, or decreased?

__Increased __Stayed about the same __Decreased __Don’t know

Probe C1a: [IF RESPONSE TO 1 IS NOT DON’T KNOW, ASK] How did you determine the final tally of postdocs? That is, what did you use as a reference for your responses to question 1.

Probe C1b: [IF RESPONSE TO 1 IS DON’T KNOW, ASK] You said you don’t know the answer to this question. Is that because source data about postdocs are not available yet or not final or some other reason? Can you tell me more about the reasons behind your answer?

2. This fall, have any programs at your institution extended the number of years postdocs are allowed to receive funding?

__Yes __No __ Don’t know

3. This fall, has your institution extended appointments for postdocs in response to the novel coronavirus (COVID-19) pandemic?

__Yes __No __Don’t know

Probe C2: [IF RESPONSE TO 2 or 3 IS DON’T KNOW, ASK] You said you don’t know the answer to this question. Is that because source data about post-docs are not available yet or not final or some other reason? Can you tell me more about the reasons behind your answer?

4. This fall, has your institution had to hold off or cancel offers to any new postdocs?

__Yes __No __Don’t know

5. This fall, have any of the following been affected by hiring freezes, furloughs, involuntary reductions in time, or layoffs?

a. Postdocs? __Yes __No __Don’t know

b. Doctorate-holding nonfaculty researchers (NFRs)? __Yes __No __Don’t know

Probe C3: [IF RESPONSE TO 4, 5a, or 5b IS DON’T KNOW, ASK] You said you don’t know the answer to this question. Is that because source data about hiring are not available yet or not final or some other reason? Can you tell me more about the reasons behind your answer?

6. Are there other novel coronavirus (COVID-19) pandemic impacts that may affect your institution’s postdoc or NFR data?

__Yes __No

6a. If yes, what are they? _____________________________________________________________________________________________________________________________________________________________________

Probe C4a: What were you thinking about when you answered this question?

Probe C4b: [IF R ANSWERED YES AND IF NECESSARY] How prevalent would you say this other experience would be to other institutions?

ProbeC5: Are there any other details about the novel coronavirus (COVID-19) pandemic in relation to the institution’s graduate students, postdocs, or doctorate-holding nonfaculty researchers (NFRs) you would like us to know about?