1_NOAA.OMB.PartA_NOAAOEdSS_5.18.15

SUPPORTING STATEMENT

EVALUATION SUPPORT SERVICES

OMB CONTROL NO. 0648-XXXX

Introduction

This document outlines the National Oceanic and Atmospheric Administration’s (NOAA) justification for a new information collection. The Office of Education (OEd) at NOAA is requesting Office of Management and Budget (OMB) approval to collect information on the scholarship recipients and institutions that have received funding through two key NOAA scholarship programs to date: the Ernest F. Hollings Undergraduate Scholarship Program (HUSP) and the Educational Partnership Program (EPP). As part of the “Evaluation Support Services” study, NOAA OEd will sponsor a Web survey to a sample of 1,878 scholarship recipients and nonrecipients (nonrecipients will serve as a comparison group). In addition, NOAA will conduct 15 in-depth interviews with the faculty, staff, and students at each of four EPP Cooperative Science Center (CSC) institutions that have received support through the scholarship programs. This comprehensive mixed-methods evaluation is described in further detail below.

1. Justification

1. Explain the circumstances that make the collection of information necessary.

NOAA’s OEd serves a critical function as the nation’s primary educator on matters related to the ocean, coastal resources, the atmosphere, and climate. One of the ways NOAA fulfills its national duty is by providing educational resources and scholarship opportunities for future scholars. The NOAA OEd scholarship programs offer students the opportunity to study a wide range of NOAA mission-related fields. These include, but are not limited to, the following: oceanic, environmental, biological, and atmospheric sciences; mathematics; engineering; remote sensing technology; physical and social sciences including geography, physics, hydrology, and geomatics; and teacher education in a related area. This study will focus on two specific NOAA scholarship programs: HUSP and EPP. EPP is an umbrella program that includes the EPP Undergraduate Scholarship Program (USP), the EPP Graduate Sciences Program (GSP), and EPP-funded CSCs. Both HUSP and EPP offer undergraduate students tuition support and a paid internship opportunity in NOAA-related major fields of study. In addition, EPP targets underrepresented groups, and partners with several Minority Serving Institutions (MSIs) to serve postdoctoral, graduate, and undergraduate students, as well as faculty at CSCs.

The programs are intended to increase undergraduate training in science, technology, engineering, and mathematics (STEM) fields and encourage recipients to pursue a career in oceanic and atmospheric public service. Moreover, the underrepresentation of minorities in STEM fields has been a source of concern in recent years (Robinson, Rousseau, Mapp, Morris, & Laster, 2007). For example, Black and Hispanic scientists and engineers are unemployed at a rate two-thirds higher than that of their White counterparts and represent only 11 percent of employees working in science and engineering occupations (National Science Foundation, 2015). The focus of NOAA’s EPP is to address this issue by partnering with MSIs to support students from underrepresented communities as they gain relevant research experience and complete degrees. In turn, the program aims to produce well-trained scientists and enhance the NOAA-related workforce.

However, to ensure the goals of NOAA scholarship programs are met, the programs themselves must be assessed and improved consistently. Therefore, the first evaluation of NOAA’s OEd scholarship programs aims to address the following program-specific research questions:

1. HUSP:

1. To what extent do recipients gain knowledge and training in NOAA mission fields?

2. Does the scholarship shape recipients’ academic and career plans?

3. What do scholars value about their HUSP experience?

4. What challenges did the scholar experience while participating in the program?

5. What areas would be best targeted for future investment?

6. What are comparable programs that provide similar experiences for recipients?

2. EPP:

1. To what extent do recipients gain knowledge and training in NOAA mission fields?

2. Does the scholarship shape recipients’ academic and career plans?

3. What do scholars value about their USP experience?

4. What challenges did the scholar experience while participating in the program?

5. What areas would be best targeted for future investment?

6. What are comparable programs that provide similar experiences for recipients?

3. CSC:

1. To what extent does the CSC influence student outcomes?

2. To what extent does the CSC expand university outreach and partnership activities?

3. How does the CSC raise awareness at the home institution and partner institutions of NOAA-related opportunities and efforts?

4. To what extent do CSC faculty members generate new research or management products in NOAA mission fields?

5. What are participants’ (administrators, faculty, staff, and students) perceptions of the CSC?

6. What kinds of partnerships are most challenging or beneficial?

7. Which CSC activities are in need of greater time or financial resources?

To address these research questions, this comprehensive mixed-methods evaluation will include the following components:

• Reviews of extant data to understand the program and historical trends

• Web surveys of HUSP and EPP alumni with telephone follow-up to describe participant experiences and outcomes

• A regression discontinuity design evaluation of HUSP, EPP USP, and EPP GSP to compare scholarship recipients to similar applicants who did not receive scholarships

• Site visits – including 15 in-depth interviews with faculty, staff, and students at each of the four CSCs – to describe institution-level contexts and outcomes

2. Explain how, by whom, how frequently, and for what purpose the information will be used. If the information collected will be disseminated to the public or used to support information that will be disseminated to the public, then explain how the collection complies with the applicable Information Quality Guidelines.

The purpose of collecting information about the HUSP and EPP scholarship programs is to determine whether the NOAA scholarship programs increase opportunities and career outcomes for recipients versus nonrecipients. The educational and career outcomes of scholarship participants will be compared to those of individuals who applied for but did not receive the HUSP or EPP scholarships in an impact analysis using the a regression discontinuity design (RDD). If the HUSP and EPP scholarship programs successfully meet their goals, more students, including those from underrepresented groups, will graduate from university in STEM and NOAA-related majors, attend graduate school, and undertake careers in STEM and NOAA-related fields.

A secondary purpose of this information collection effort is to identify the strengths and weaknesses of the HUSP and EPP scholarship programs. By identifying how and in what ways the HUSP and EPP programs have helped serve student scholars and how these programs can be improved, NOAA can further refine its programming and target efforts for future investment.

The purpose of collecting information related to the CSCs is to evaluate these centers’ efficacy in training students in NOAA-related fields. To support the EPP’s goal to serve underrepresented groups in STEM, the four CSCs have formed partnerships with community organizations and other universities across the United States. One of the main functions of CSC partnerships is to educate, train, and disseminate valuable information in NOAA-related fields. Given this objective, a primary goal of this data collection effort is to understand how CSCs use their technological resources to educate, train, and otherwise support students at MSIs. It also aims to understand what kinds of partnerships are most challenging or beneficial. Based on this information, NOAA OEd can better allocate future funds and resources.

3. Describe whether, and to what extent, the collection of information involves the use of automated, electric, mechanical, or other technological techniques or other forms of information technology.

Information from current and former HUSP and EPP scholarship applicants and recipients will be collected via a Web-based survey administered over the course of 6 weeks. This Web-based survey will be the primary method of data collection, as it offers an efficient means of reaching HUSP and EPP alumni who may be widely dispersed. Using updated alumni contact information that is available to NOAA via the OEd’s Student and Performance Measures Tracking System and Undergraduate Scholarship Programs Application System, respondents will be emailed an invitation to participate that includes a link to the online survey.

The team will use a multi-tiered approach to obtain updated contact information for survey respondents whose information from NOAA is incomplete or out-of-date. First, for those with no listed email address, we will attempt to retrieve email addresses through an online search. We will also follow any possible connections to colleges or universities based on available information. When possible, we will try to contact the individual through social media sites such as LinkedIn or Facebook. Next, if we are unable to obtain email addresses through these means, or if the email addresses we retrieve prove to be invalid, we will obtain postal mailing addresses and/or telephone numbers through LexisNexis. We will send postcards to those individuals with postal mailing addresses to ask them to provide their email information by forwarding it to an email address set up for the study or by calling a toll-free number. We will call persons for whom LexisNexis obtains telephone numbers but no mailing addresses and similarly ask these individuals to provide us with their email addresses. For those whose postcards are returned due to incorrect address information or who do not respond to the postal mailing, we will follow up via telephone if we have valid telephone numbers for those individuals. Once we have obtained accurate current email addresses, we will provide each respondent with a secure individual passcode to log onto the survey.

Potential respondents who have not submitted a survey after several follow-up online reminders within the first 6 weeks will be contacted to schedule a telephone follow-up. Given the young age distribution of the respondents, we estimate that most will opt to take the online version of the survey rather than schedule a telephone interview. The online survey will be consistent with the attached paper version and will include the OMB Control Number and expiration date. NOAA anticipates that a minimum of 75 percent of these participants will respond to the email invitation to complete the Web-based survey. Insight Policy Research, Inc. (Insight), the contractor for this study, has experience with OMB approved data collections. For one recent survey, “Enhancing Completion Rates for SNAP (Supplemental Nutrition Assistance Program) Quality Control Reviews for the USDA” (OMB Control No. 0584-0590), Insight achieved a 91 per cent response rate. Based on this information and Insight’s expertise, we expect a high level of responsiveness from participants.

4. Describe efforts to identify duplication.

Currently, there is no similar data collection available. Every effort has been made to avoid duplication, and the Web survey includes questions not previously asked of NOAA scholars in relation to their individual outcomes and educational trajectories. The CSCs undergo an evaluation during the third year of a 5-year grant award. This information will be used whenever possible during CSC site visits rather than collecting similar metrics. Given that there has been no previous effort to examine all four CSCs systematically during a single evaluation, the likelihood of duplicating information collection is minimal.

5. If the collection of information involves small businesses or other small entities, describe the methods used to minimize burden.

NOAA has determined the requirements for this information collection do not adversely impact small businesses or other small entities. There are no small businesses used in this data collection. All new data collection is with HUSP and EPP applicants and alumni.

6. Describe the consequence to the Federal program or policy activities if the collection is not conducted or is conducted less frequently.

Obtaining information related to the outcomes and achievements of HUSP and EPP applicants and alumni are critical to the evaluation of the scholarship programs, to NOAA’s educational mission, and to supporting minority students in NOAA’s mission-related fields. If these data are not collected, critical information related to these scholarship programs’ successes and challenges cannot be evaluated for the betterment of the programs and NOAA’s educational missions.

7. Explain any special circumstances that require the collection to be conducted in a manner inconsistent with OMB guidelines.

There are no special circumstances.

8. Provide information on the PRA Federal Register Notice that solicited public comments on the information collection prior to this submission. Summarize the public comments received in response to that notice and describe the actions taken by the agency in response to those comments. Describe the efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported.

A Federal Register Notice published on April 29, 2015 (80 FR 23773) solicited comments on this request. No comments were received.

Our team consulted with each of the four CSC directors to ensure that we had a comprehensive understanding of the program data, including the directors’ perspectives on the creation of program measures and evaluation. The directors’ names and contact information are displayed in Table A.8.1.

Table A.8.1.
CSC Director Contact Information

Name	CSC Affiliation	Address	Phone Number
Vernon Morris, Ph.D.	NOAA Center for Atmospheric Science at Howard University (NCAS)	HURB-1 Building Room 302 1840 7th Street NW Washington, DC 20001	(202) 806-8678
Reza Khanbilvardi, Ph.D.	NOAA Cooperative Center Remote Sensing Science and Technology Center at the City College of the City University of New York (CREST)	Steinman Hall, Rm T-107 140th Street & Convent Avenue New York, NY 10031	(212) 650-8009
Michael Abazinge, Ph.D.	NOAA Environmental Cooperative Science Center at Florida A& M University (ECSC)	1515 S. Martin Luther King Jr. Blvd 305-D FSH Science Research Center Tallahassee, FL 32307	(850) 599-8553
Paulinus Chigbu, Ph.D.	NOAA Living Marine Resources Cooperative Science Center at the University of Maryland Eastern Shore (LMRCSC)	Carver Hall University of Maryland Eastern Shore Princess Anne, MD 21853	(404) 621-3034

9. Explain any decisions to provide payments or gifts to respondents, other than remuneration of contractors or grantees.

No payment or gift will be offered to interview or survey respondents.

10. Describe any assurance of confidentiality provided to respondents and the basis for assurance in statute, regulation, or agency policy.

Survey respondents will be assured that no individually identifiable information will be included with any response; every effort will be made during reporting to minimize the extent to which the identities of respondents can be inferred from the data. Furthermore, the link between any response and any individual will be secured in a locked (hardcopy) or encrypted (computer) file and destroyed at the conclusion of the study pursuant to applicable NOAA regulations. In adherence with FISMA Enclave Data Transmission Policies, all data transmitted must be encrypted via software using FIPS 140-2 standards to protect sensitive information for compliance purposes.

All data collection staff at Insight (who will design the online survey and analyze results) and IMPAQ interviewers and data analysis collection staff (who will and collect the survey data and conduct the CSC site visits) are required to sign a data confidentiality pledge (see Attachment D) associated specifically with this study. In this agreement, the staff pledges to maintain the privacy of all information collected from the respondents and to not disclose it to anyone other than authorized representatives of the study, except as otherwise required by law.

11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private.

The questions in the interview protocols and survey instruments are unlikely to be considered sensitive by respondents. Because one of the main goals of the EPP USP, GSP, and CSC programs is to serve underrepresented groups, the survey instrument will prompt respondents to provide information related to their race and ethnicity so that respondent outcomes can be disaggregated for analysis. Since enrollment in an MSI is a requirement for all EPP USP and GSP applicants, we anticipate that respondents will be comfortable disclosing this information on the survey. There are no questions regarding religious beliefs, sexual attitudes or behavior, or other matters commonly considered as private on the remainder of the survey instrument. As part of the consent process, respondents will be informed that they may choose not to answer any specific questions and, as noted in A.10 above, that responses will be treated as private.

12. Provide an estimate in hours of the burden of the collection of information.

Web Survey Hour Burden

The survey is expected to achieve at least a 75-percent response rate and will require approximately 25 minutes for an average current or past scholarship recipient respondent to complete, and approximately 15 minutes for respondents who have not participated in the scholarship programs. The non-scholarship respondent surveys are shorter because they are not asked to respond to questions about their experience with the scholarship programs. Therefore, an estimated minimum of 1,409 out of 1,878 potential respondents are expected to complete the survey for a total of 525 burden hours (1,034 respondents x 25 minutes and 375 respondents x 15 minutes). Because most potential survey respondents will have achieved at least a bachelor’s degree in a NOAA-related life, physical, or social science field, it is assumed that these individuals would meet the minimum qualification requirements for Life, Physical, and Social Science Technician positions. Based on Bureau of Labor Statistics estimates (BLS, 2013), the annual mean wage for this occupation is $45,770. Assuming a 40-hour workweek over the course of 52 weeks annually, the hourly wage for this occupation is approximately $22.00. Therefore, the overall cost to survey respondents would be approximately $11,550 (525 burden hours x $22.00 per hour).

CSC Site Visit Burden Hours

In addition to the surveys, site visits to the CSCs will consist of 20 student interviews of approximately 90 minutes each, plus 44 hour-long interviews with community partners, institution partners, CSC administrators, and CSC center directors, for a total of 74 burden hours. The estimated annualized hourly costs to interviewees are calculated in Table A.12.1, below. Annual wages for each type of interviewee are based on Bureau of Labor Statistics estimates for the most comparable occupation(s) (BLS, 2013). For example, we used the annual mean wage for Postsecondary Education Administrators as our best estimate for both CSC Administrator and Center Director interviewees. Because community partners tend to be field scientists working in community settings, and because there is no overall Life, Physical, and Social Scientist category, our best estimate was calculated as the average of several different types of scientists in life, physical, and social science fields. Because institution partners tend to be faculty members working at partner universities, and because there is no overall Postsecondary Life, Physical, and Social Science Teacher category, our best estimate was calculated as the average of several different types of post-secondary teachers in life, physical, and social science fields.

Table A.12.1.

Estimated Cost to Interviewees

Type of Interviewee	# of Interviews	Annual Wage	Hourly Wage	Total Cost
CSC Student	20	$0	$0.00	$0.00
Community Partner	20	$80,563	$38.73	$774.64
Institution Partner	12	$84,897	$40.82	$489.79
CSC Administrator	8	$100,600	$48.37	$386.96
CSC Center Director	4	$100,600	$48.37	$193.48
Total Cost to All Interviewees				$1,844.87

Taken together, surveys and interviews account for approximately 599 burden hours and $13,395 annualized hourly costs to participants.

13. Provide an estimate of the total annual cost burden to the respondents or record-keepers resulting from the collection (excluding the value of the burden hours in Question 12 above).

No capital/startup or ongoing operational/maintenance costs are associated with this information collection.

14. Provide estimates of annualized cost to the Federal government.

The annualized cost to the Federal Government for this contract is $262,534. This includes costs associated with the study design, instrument development, technical development of survey forms, information collection, analysis, reporting, and presentation/publication of the results. This is a 3-year contract with a period of performance from October 8, 2014 to August 25, 2017.

15. Explain the reasons for any program changes or adjustments.

This is a new collection of information effort for a program evaluation that will add 599 burden hours to the OMB inventory.

16. For collections whose results will be published, outline the plans for tabulation and publication.

The models for the local average treatment effects (receipt of the scholarship) will be estimated using local linear regressions or polynomial regressions in SAS or STATA software. Survey data will be analyzed with results presented in tabular form appropriate to the research questions outlined in the evaluation plan. Data from CSC site visits and in-depth interviews will be recorded, transcribed, and analyzed using NVivo 10 software. For in-depth interview analysis, the researchers will import a verbatim transcript of each interview into NVivo and will code the data using a standard coding scheme for each interview.

The schedule for data collection, analysis, and reporting is shown in Table A.16.1 below.

Table A.16.1

Project Time Schedule

Activity	Expected Activity Period
Develop data collection instruments	June­–September, 2015
Conduct surveys	September–December, 2015
Conduct CSC site visits	October–December, 2016
Data analysis	March–May, 2016
Submit final report to NOAA	August 2017

17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons why display would be inappropriate.

The agency plans to display the expiration date of OMB approval on all forms/questionnaires associated with this information collection.

18. Explain each exception to the certification statement.

There are no exceptions to the certification statement.

REFERENCES

Bureau of Labor and Statistics (2013). May 2013 National Occupational Employment and Wage Estimates United States [Occupational Employment Statistics]. Retrieved from http://www.bls.gov/oes/current/oes_nat.htm

National Science Foundation, National Center for Science and Engineering Statistics (2015). Women, Minorities, and Persons with Disabilities in Science and Engineering: 2015 (Special Report NSF 15-311). Retrieved from http://www.nsf.gov/statistics/wmpd/

Robinson, L., Rousseau, J., Mapp, D., Morris, V., & Laster, M. (2007). An Educational Partnership Program with Minority Serving Institutions: A Framework for Producing Minority Scientists in NOAA-Related Disciplines. Journal of Geoscience Education, 55 (6).

Page 7