Noyce Evaluation SS B_draft2_to NSF_Clean

The Evaluation of the National Science Foundation’s (NSF) Robert Noyce Teacher Scholarship Program

Statement for Paperwork Reduction Act Submission

Supporting Statement B: Collections of Information Employing Statistical Methods

OMB No. 3145-New

DATE

Dr. Jennifer Ellis

Program Director

Division of Undergraduate Education

National Science Foundation

2415 Eisenhower Ave

Alexandria, Virginia 22314

Telephone: 703-292-8670

Email: [email protected]

Table of Contents

Part B: Collections of Information Employing Statistical Methods 1

B.1. Respondent Universe and Selection Methods 1

Table B1. Number Invited to Participate, Initial Sample Sizes, Expected Response Rates, and Final Sample Sizes 5

B.2. Procedures for the Collection of Information 9

B.3. Methods to Maximize Response Rates and the Issue of NonResponse 9

B.4. Tests of Procedures 12

B.5. Consultants 13

Table B2. Contact Information 13

Attachments

1. Demographic Questionnaire

2. Data Collector Confidentiality Agreement Form

3a. Former Scholar/Fellow Focus Group Protocol

3b. Evaluators Focus Group Protocol

3c. Faculty Focus Group Protocol

3d. K-12 District and School Leaders Focus Group Protocol

3e. Principal Investigator Interview Protocol

3f. Potential Principal Investigator Interview Protocol

3g. Program Staff Coordinators Focus Group Protocol

3h. NSF Program Staff Focus Group Protocol

3i. Current Scholars and Fellows Focus Group Protocol

3j. AAAS Staff Focus Group Protocol

4a. Email from NSF to Principal Investigators Introducing the Evaluation

4b. Email from NSF to Current/Former Scholars/Fellows

4c. Reminder for Partial Demographic Questionnaire (All Respondent Types)

4d. Final Reminder for Partial Demographic Questionnaire (All Respondent Types)

5a. Questionnaire Invitation Email for Evaluators, Faculty, K-12 Leaders, and Program Coordinators

5b. Questionnaire Invitation Email for Principal Investigators

5c. Questionnaire Invitation Email for Potential Principal Investigators (Referred)

5d. Questionnaire Invitation for Potential Principal Investigators (Original Contact - Non-Noyce Institutions)

6a. Reminder - Questionnaire Invitation for Evaluators, Faculty, K-12 Leaders, and Program Coordinators

6b. Reminder - Questionnaire Invitation for Principal Investigators

6c. Reminder - Questionnaire Invitation for Potential Principal Investigators (Referred)

6d. Reminder - Questionnaire Invitation for Potential Principal Investigators (Original Contact - Non-Noyce Institutions)

7a. Final Reminder - Questionnaire Invitation for Evaluators, Faculty, K-12 Leaders, and Program Coordinators

7b. Final Reminder - Questionnaire Invitation for Principal Investigators

7c. Final Reminder - Questionnaire Invitation for Potential Principal Investigators (Referred)

7d. Final Reminder - Questionnaire Invitation for Potential Principal Investigators (Original Contact - Non-Noyce Institutions)

8a. Request for Principal Investigator to Forward Invitation Email (PI is Participating)

8b. Request for Principal Investigator to Forward Invitation Email (PI is Not Participating)

8c. Reminder - Request for Principal Investigator to Forward Invitation Email (PI is Not Participating)

8d. Final Reminder - Request for Principal Investigator to Forward Invitation Email (PI is Not Participating)

8e. Form to Collect Potential Participant Contact Information from PIs

9a. Focus Group Invitation for Evaluators, Faculty, K-12 Leaders, Program Coordinators, and Current/Former Scholars/Fellows

9b. Interview Invitation for Principal Investigators and Potential Principal Investigators

9c. Focus Group Invitation for AAAS Staff

9d. Focus Group Invitation for NSF Staff

10a. Reminder - Focus Group Invitation for Evaluators, Faculty, K-12 Leaders, Program Coordinator, and Current/Former Scholars/Fellows

10b. Reminder - Interview Invitation for Principal Investigators and Potential Principal Investigators

10c. Reminder - Focus Group Invitation for AAAS Staff

10d. Reminder - Focus Group Invitation for NSF Staff

11a. Final Reminder - Focus Group Invitation for Evaluators, Faculty, K-12 Leaders, Program Coordinator, and Current/Former Scholars/Fellows

11b. Final Reminder - Interview Invitation for Principal Investigators and Potential Principal Investigators

11c. Final Reminder - Focus Group Invitation for AAAS Staff

11d. Final Reminder - Focus Group Invitation for NSF Staff

12a. Scheduling Confirmation (All Focus Group and Interview Participants)

12b. Reminder and Meeting Info (All FG and Interview Participants)

12c. Thank You Email (All FG and Interview Participants)

12d. No-Show Follow-Up Email (All Participants)

13a. Newsletter/Social Media Post Recruitment Text

13b. Recruitment Flyer - Print Version

13c. Recruitment Flyer - Digital Version

14a. America COMPETES Act (P.L. 110-69)

14b. Higher Education Act of 1965 (20 U.S.C. 1021)

14c. National Science Foundation Authorization Act of 2002 (P.L. 107-368)

15a. Noyce Evaluation Questions and Subquestions

15b. Federal Register 60-Day Notice

15c. Federal Register 30-Day Notice

15d. Federal Register 30-Day Notice Public Comments – if needed

15e. Federal Register 30-Day Notice NSF’s Response to Public Comments – if needed

15f. NSF Statistical Contact feedback on Public Comments

15g. Estimates of Respondent Burden and Annualized Cost

16a. Questionnaire Invitation Phone Script – Evaluators, Faculty, K-12 Leaders, Program Coordinators

16b. Questionnaire Invitation Phone Script – Principal Investigators

16c. Questionnaire Invitation Phone Script – Potential Principal Investigators (Referred)

16d. Questionnaire Invitation Phone Script – Potential Principal Investigators (Original Contact – Non-Noyce Institutions)

17a. Focus Group Invitation Phone Script – Evaluators, Faculty, K-12 Leaders, and Program Coordinators

17b. Interview Invitation Phone Script – Principal Investigators and Potential Principal Investigators

17c. Focus Group Invitation Phone Script – AAAS Staff

17d. Focus Group Invitation Phone Script – NSF Staff

Part B: Collections of Information Employing Statistical Methods

B.1. Respondent Universe and Selection Methods

The potential respondent universe is approximately 660 Noyce-eligible institutions and affiliated individuals who either directly or indirectly interact with the Noyce Program. The number of individuals within the respondent universe is currently unknown. However, the respondent universe for data collection is 1,185 for the demographic questionnaire (including pretesters), 120 for the interviews (excluding the pretesters), and 671 (excluding the pretesters) for the focus groups. This study does not require a nationally representative sample to identify what is working well and growth opportunities for the Noyce Program, particularly because the intention is to understand how to increase applications from those institutions that have not historically previously applied. Additionally, a truly random sample may underrepresent the vast diversity in experiences across Noyce-eligible institutions.

NSF is interested in gathering perceptions of the Noyce Program from individuals at Noyce-eligible institutions. Thus, given that the purpose of the information collection is for program improvement, the Contractor will use a purposive sampling design with components of stratified random sampling and oversampling with institutions being the sampling unit. Noyce-eligible institutions, for the purpose of this information collection, are accredited institutions of higher education offering teacher education or residency programs that allow STEM majors to obtain teaching certifications, as outlined by NSF. Noyce-eligible institutions that have received a Noyce award between FY2014 and FY2024 (“Noyce institutions”) will be carefully selected using quota sampling for award type (i.e., Track) as the primary selection stratum. Additionally, Noyce-eligible institutions that have not received a Noyce award between FY2014-FY2024 (“non-Noyce institutions”) will be sampled. Both types of institutions will be sampled based on an institution type stratum (e.g., MSI, HSI, HBCU), oversampling for MSIs and HBCUs to ensure adequate representation and ability to achieve saturation.¹ The institutions will then be randomly selected within the institution type stratum. One of the justifications for employing purposive sampling is to increase the coverage and depth of participants.^2,3 Using the track and institution type strata can ensure shared variation across the respondents.⁴ From the initially identified sample, the institutional characteristics listed below will be reviewed to ensure diversity in representation:

• Region (Desert and Pacific, Great Lakes, Great Plains, Interior Northeast, Mid-Atlantic, Mid-South, New York, Northeast, Southwest, West⁵),

• Rurality,

• Land grant status,

• EPSCoR state status,

• Percentage of undergraduates that complete an education degree,

• Percentage of student body with a Pell Grant,

• New to Noyce (for Noyce institutions only; an institution that has not had a new Noyce award in 5 years from the year of that award’s solicitation),

• Public, Private, not-for-profit, and

• Level (e.g., 2-year institution, 4-year institution).

These will not be proportional to the total number of Noyce institutions, but rather the Contractor will ensure that the institutions selected will include a range of characteristics to ensure the widest range of perspectives are captured.

To ensure a range of characteristics and experiences is represented, the full list of selected institutions, along with their institution-level characteristics for both the Noyce Program institutions and non-Noyce Program institutions will be reviewed with the external Evaluation Advisory Committee (EAC). Once the PIs agree to participate, they will be asked to identify program coordinators, faculty, K-12 district or school leaders, and external evaluators who are likewise involved in their Noyce project. They can either provide the contractor with contact information at the end of the demographic questionnaire (Attachment 1.) or at the end of the interview (Attachments 3e. and 8e.) or forward an email to these individuals (Attachment 8a.). NSF will send an initial email to current and former scholars/fellows alerting them to the evaluation (Attachment 4b.). PIs will also be asked to send their current and former scholars/fellows an informational email (Attachment 8a.) that provides information about the evaluation and requests that they reach out to the Contractor if they are interested in participating in the evaluation. All AAAS staff involved in supporting the Noyce Program and all current and former NSF staff will be invited to participate by the Contractor (Attachments 9c. and 9d., respectively).

Table B1. Number Invited to Participate, Initial Sample Sizes, Expected Response Rates, and Final Sample Sizes

Expected Response Rates

The expected responses rates will vary, depending on the data collection instrument and are detailed in Table B1 above and in the next sections. For the full data collection, the expected total number of individuals invited to participate in the evaluation includes 1360 for the demographic questionnaire and 861 for the interviews or focus groups. These figures include 827 demographic questionnaire respondents and 533 non-respondents; 70 interview respondents and 64 non-respondents; and 469 focus group respondents and 259 non-respondents.

Demographic Questionnaire. It is anticipated that the initial sample for the demographic questionnaire will include 122 PIs and 154 potential PIs, and up to 146 program coordinators, 293 faculty, 98 external evaluators, 195 current scholars/fellows, 156 former scholars/fellows, and 195 high need K-12 leaders. Assuming that 80% of PIs and 60% of potential PIs respond to the invitation email (Attachments 5a.-5d.), follow-up emails (Attachments 6a.-6d.; 7a.-7d.) or follow-up telephone calls (Attachments 16a.-16d.), 98 PIs and 93 potential PIs, and up to 117 program coordinators, 234 faculty, 78 external evaluators, 156 current scholars/fellows, 125 former scholars/fellows, and 156 high need K-12 school/district leaders will agree to complete the demographic questionnaire. Of those, 80% of all groups other than potential PIs (approximately 94, 188, 63, 125, 100, 125, respectively) and 60% of potential PIs (56) are expected to complete the demographic questionnaire (Attachment 1.).

PI Interviews and Potential PI Interviews. The initial sample of PIs and potential PIs will include the 78 and 56 PIs and potential PIs, respectively, who complete the demographic questionnaire. Of these, assuming an 80% and 60% response rate, respectively, it is anticipated that after receiving an invitation email (Attachment 9b.), follow-up emails (Attachments 10b. and 11b.), and/or follow-up telephone calls (Attachments 16b.-16d.), 63 PIs and 33 potential PIs will agree to complete interviews, and that of those who agree to participate, 50 PIs and 20 Potential PIs will complete an interview.

Focus Groups with Program Coordinators, Faculty, External Evaluators, and High Need K-12 School/District Leaders. As noted, PIs will be asked for contact information for program coordinators, faculty, external evaluators, and high need K-12 school leaders or will be asked to forward an email. Based on the expected response rate of 80% after sending invitation emails (Attachment 9a), follow-up emails (Attachment 10a and 11a), and/or follow-up telephone calls if phone numbers are available (Attachment 16a.), it is anticipated that 75, 150, 50, and 100 individuals, respectively, from the initial groups of 94, 188, 63, and 125 respondents who are anticipated to complete the demographic questionnaire will agree to participate in the focus group. After receiving a confirmation email about their focus group (Attachments 12a., for both virtual and in-person focus groups) and a reminder email with the meeting information (Attachment 12b., in the case of the virtual focus groups), it is expected that up to 60 program coordinators, 120 faculty, 40 external evaluators, and 80 high need K-12 school/district leaders will participate in a focus group.

Focus Groups with Current and Former Scholars/Fellows. The 125 current scholars/fellows and 100 former scholars/fellows expected to complete the demographic questionnaire will be invited to participate in a focus group. Of these 125 current scholars/fellows and 100 former scholars/fellows, and assuming an 80% response rate after an invitation email (Attachment 9a.) and follow-up emails (Attachments 10a. and 11a.), it is expected that 100 current scholars/fellows and 80 former scholars/fellows will agree to participate. Upon receipt of a confirmation email regarding the focus group (Attachments 12a., for both virtual and in-person focus groups) and a reminder email with the meeting information (Attachment 12b., in the case of the virtual focus groups), it is anticipated that a total of 80 current scholars/fellows and 64 former scholars/fellows will proceed to partake in the focus group.

Focus Groups with AAAS Staff and Current NSF Staff. For the focus groups, it is anticipated that 100% of the 3 AAAS staff and 6 current NSF staff who will be invited will agree to participate and successfully participate in their respective focus groups, after an invitation email (Attachments 9c and 9d, respectively), follow-up emails (Attachments 10c. and 10d.; 11c. and 11d.), and/or follow-up telephone calls (Attachments 17c. and 17d.).

Focus Groups with Former NSF Staff. Initially, 25 former NSF staff will be invited to participate in a focus group. Of these 25, assuming an 80% response rate after an invitation email (Attachment 9d.), follow-up emails (Attachment 10d. and 11d.), and/or follow-up telephone calls (Attachment 17d.), it is expected that 20 former NSF staff will agree to participate. Of these 20 former NSF staff who agree to participate, assuming an 80% response rate, it is anticipated that 16 will complete a focus group.

B.2. Procedures for the Collection of Information

This is not a nationally representative study; therefore, concern regarding the statistical methodology for both stratification and sample selection is not applicable.

B.3. Methods to Maximize Response Rates and the Issue of NonResponse

Overall response rate projections were presented previously in this document. Achieving the specified response rate involves locating the sample members to secure participation using procedures described below. It is estimated that 80%, on average, of the sampled respondents across all respondent types will complete both the demographic questionnaire and then on average, 80% of those who complete the demographic questionnaire will complete either an interview or focus group, as relevant.

Below are the procedures to be followed to maximize the number of sampled individuals who complete both the demographic questionnaire and either an individual interview or a focus group:

• Correspondence to potential participants will be constructed to relay important details of the study and how it will help NSF gain insight into respondents’ experiences with the Noyce Program

• The Contractor has tailored strategies to recruit specific respondent types to increase participation, recognizing that different respondent types will need to be recruited through different methods as follows:

  • PIs will be recruited directly by the Contractor via invitation emails

  • Program coordinators, faculty, K-12 district or school leaders, and external evaluators will be recruited by the Contractor after acquiring their contact information from PIs, then sending invitation emails

  • Current and former Scholars/fellows will be informed of the study by NSF and then contacted via their PIs with information about the study with a request to contact the Contractor regarding participation in the research

• The Contractor will work with participants to identify other potential respondents, using contact information to reach out to these individuals with pertinent information regarding participating in the project (e.g., the purpose of the evaluation, time commitment, etc.)

• Due to the number of respondent types, invitation and reminder emails will be tailored to participants to ensure they have the correct information when going through the recruitment process

• For potential participants who do not respond to the initial invite, team members will send a follow-up email up to two additional times, followed by a telephone call (if possible) to increase likelihood of participation

• The Evaluation Advisory Committee (EAC) has been consulted to gain insight on how to recruit specific respondent types, including timing, communication modality, and what context to provide, thus enhancing strategies to recruit participants across respondent types

• Participation correspondence throughout the recruitment process will be organized through a designated project-specific email address to enable better tracking of participants and communication with them, making the process more efficient for the Contractor

• The Contractor will have a designated phone number that potential respondents can use to help them navigate the process

• In-person and virtual data collection sessions will be scheduled (corresponding to times when potential participants are already gathered, for in-person) to increase participation by offering multiple ways respondents can participate

• The Contractor will keep a centralized database of potential participants and communication efforts to streamline correspondence and ensure respondents are accounted for

• The Contractor has developed standardized internal training for its staff to assist in recruitment efforts

• The Contractor has an evaluation-specific website that contains information regarding the project that can help facilitate participation by informing potential participants of the context of their engagement

B.4. Tests of Procedures

A total of 6 of the 11 data collection instruments were pretested with between 1 and 4 respondents of the appropriate type. The Contractor carefully selected pretest respondents. These respondents were contacted by email during the pretesting phase. The fifteen (15) pretest respondents evaluated assigned instruments for understandability, missing concepts, and length of time to answer. For the data collection instruments that were respondent type-specific interview or focus group protocol data collection instruments, five (5) PIs, three (3) Potential PIs, four (4) current Scholars/Fellows, one (1) Program Support Staff/Coordinator, and one (1) External Evaluator pretested their respective data collection instruments. Additionally, one (1) co-PI (included in the PI respondent type by definition and who also pretested the PI interview protocol data collection instrument), one (1) Potential PI, and one (1) External Evaluator (who also pretested the External Evaluator focus group protocol data collection instrument) pretested the demographic questionnaire. All instruments were revised in response to the feedback from the pretesting.

B.5. Consultants

The Contractor, WhitworthKee Consulting, LLC, will conduct this study.

Table B2. Contact Information

NAME	AFFILIATION	TELEPHONE NUMBER	EMAIL
Chad Kee	WhitworthKee Consulting, LLC	(202) 339-1473	[email protected]
Kevin Hylton	WhitworthKee Consulting, LLC	(202) 339-1473	[email protected]
Molly Matthews-Ewald	WhitworthKee Consulting, LLC	(202) 339-1473	[email protected]
Tatiana Patterson	WhitworthKee Consulting, LLC	(202) 339-1473	[email protected]
Rachel Messer	WhitworthKee Consulting, LLC	(202) 339-1473	[email protected]
Susan Ullrich	WhitworthKee Consulting, LLC	(202) 339-1473	[email protected]
Brittany Brewster	WhitworthKee Consulting, LLC	(202) 339-1473	[email protected]
Isabel Leamon	WhitworthKee Consulting, LLC	(202) 339-1473	[email protected]
Cameron Beatty	WhitworthKee Consulting, LLC	(202) 339-1473	cbeatty@whitworthkee.com
Allen Thomas	WhitworthKee Consulting, LLC	(202) 339-1473	[email protected]
Dia Sekayi	WhitworthKee Consulting, LLC	(202) 339-1473	[email protected]
Laura Monroe	WhitworthKee Consulting, LLC	(202) 339-1473	[email protected]
David Ayeni	WhitworthKee Consulting, LLC	(202) 339-1473	[email protected]

1 Ames, H., Glenton, C., & Lewin, S. (2019). Purposive sampling in a qualitative evidence synthesis: A worked example from a synthesis on parental perceptions of vaccination communication. BMC Medical Research Methodology, 19, 1-9. https://doi.org/10.1186/s12874-019-0665-4

2 Guest, G. (2015). Sampling and selecting participants in field research. In H.R. Bernard & C.C. Gravlee, (Eds.), Handbook of methods in cultural anthropology (2nd ed., pp. 215-250). London: Rowman & Littlefield.

3 Suen, L. J. W., Huang, H. M., & Lee, H. H. (2014). A comparison of convenience sampling and purposive sampling. Hu li za zhi, 61(3). https://doi.org/105. 10.6224/JN.61.3.105

4 Campbell, S., Greenwood, M., Prior, S., Shearer, T., Walkem, K., Young, S., ... & Walker, K. (2020). Purposive sampling: Complex or simple? Research case examples. Journal of Research in Nursing, 25(8), 652-661. https://doi.org/10.1177/1744987120927206

5 Informed by NSF's I-Corps Hub (National Science Foundation, https://new.nsf.gov/funding/initiatives/i-corps/view-hubsn)