2_NOAA OMB PartB_NOAAOEdSS

SUPPORTING STATEMENT

EVALUATION SUPPORT SERVICES

OMB CONTROL NO. 0648-XXXX

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

INTRODUCTION

This section documents the statistical procedures used for NOAA’s “Evaluation Support Services” study. The data collection process for this project includes two primary components: a Web-based survey and site visits. The Web-based survey will request quantitative data from a sample of 1,878 individuals. The site visits will collect qualitative data from a sample of 64 individuals through conducting 16 interviews at 4 academic institutions.

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

Web Survey

The quantitative component of this evaluation provides the data necessary for an impact analysis using a Regression Discontinuity Design (RDD) based on information obtained via Web-based surveys. The surveys will be administered to HUSP and EPP program alumni and a comparison group of unsuccessful scholarship applicants. For this portion of the study, the universe of scholarship alumni will be surveyed, which consists of 1,378 scholarship recipients. We will take an intent-to-treat group approach for those applicants who were awarded a scholarship but declined to accept it. Therefore, all scholarship awardees (including individuals who declined the scholarship) will be included in the alumni survey sampling frame. To provide a comparison group, a sample of 500 unsuccessful applicants will be taken from a universe of 1,981. Unsuccessful applicants will be sampled based on those whose average reviewer scores approximate the cutoff points for those applicants who were accepted. The response rate for both groups is anticipated at approximately 75 percent.

Site Visits

The qualitative component of this evaluation, the CSC site visits, will be conducted in person in the final year (year 3) of this evaluation. In order to ascertain institution-level effects of the EPP offerings, CSC site visits will include a series of 64 semi-structured interviews with institutional officials, program personnel, and faculty. Site visits will include focus groups with CSC students and alumni. NOAA estimates a universe of 2,228 students involved in CSC activities, 4 directors, 41 faculty, 24 institutional partners, and 44 community partners across all 4 CSCs. Of these individuals, NOAA intends to sample 20 students, 20 community partner teachers and administrators, 12 institutional partner administrators, 8 CSC faculty members, and 4 center directors. These respondents will be a purposive sample that will ensure that we capture the diverse characteristics of each group. . We will interview at least two members of each stakeholder group with the exception of CSC directors. The universe (n=4) of CSC directors will be interviewed.

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Web Survey

Procedures for the collection of data. IMPAQ designs integrated computer-assisted telephone interviewing (CATI) and Web surveys using the Voxco platform. The Voxco system provides the ability to deploy questionnaires across modes, which increases programming efficiency. The Web survey product, Acuity4, can handle sophisticated logic, multiple rotations, and advanced skip patterns. It has multiple language capability and automatic device detection, and it offers the capability to reach respondents via email or social media and to monitor open rates to enable adjustments to invitation and reminder strategies. In addition, Voxco’s reporting functions provide real-time results to monitor survey progress. Multimode results are stored on one database to simplify survey analysis and reporting. The following procedures will be used in the collection of data from NOAA scholarship recipients and nonrecipients:

• Pre-notification emails alerting respondents of the importance of the study for NOAA Office of Education objectives and continuation of scholarship opportunities. The text for these emails is included in Attachment B.

• Web survey, fielded for 6 weeks, with reminders for nonresponse and unique identifiers for each respondent.

• Follow-up CATI data collection for nonrespondents.

Methodology for stratification and sample selection. A major component of this evaluation will involve collecting data from HUSP and EPP scholars and unsuccessful applicants via online surveys. The scholar survey will be a census of the population. The sample of 500 unsuccessful applicants will be sampled based on a fixed threshold (the reviewer score) or cutoff point that is closest to that of the selected scholars.

Estimation procedure. The models for the local average treatment effects (receipt of the scholarship) will be estimated using local linear regressions or polynomial regressions in SAS or STATA software. Univariate and bivariate descriptive statistics of percentages and means will be produced prior to multivariate regressions. Next, the variables will be adjusted so that the intercept approximates the cutoff point for receiving the scholarships. We will have a two-equation system with an outcome equation and a treatment equation. Higher order terms will be specified to accommodate a curvilinear function, if necessary.

Degree of accuracy. For the population of younger, highly educated individuals, Web surveys are the most accurate mode of information gathering. Accuracy relies on securing the correct contact information from respondents. This is to be addressed through existing information collected by NOAA from individuals and institutions at the time of application as well as in subsequent database updates. Any additional contact information will be secured through a LexisNexis search. Results for the RDD are also dependent on the statistical power of the

sample. Assuming a response rate of 50 percent, the sample will enable us to detect a minimum effect size of 0.10 with power at 0.8 (assuming the baseline covariates explain 0.7 of the variance).

Unusual problems required specialized sampling procedures. NOAA does not anticipate the use of specialized sampling procedures.

Any use of periodic data collection cycles. The survey will only be administered at one time point.

Site Visits

Procedures for the collection of data. During site visits to the CSCs, a broad range of stakeholders will be interviewed, including center directors, participating faculty, collaborating institutions, and outreach partners. Focus groups will be held with graduate and undergraduate students as well as postdoctorate fellows. Each site visit will consist of five separate protocols, and protocols for these site visits are organized according to the particular stakeholder being interviewed. Protocols for the site visits are included in this submission. General topics for the site visit study protocol will reflect the key process evaluation questions (e.g., CSC contributions to home institutions, products generated by the CSCs, participants’ perceptions of the CSCs, activities in need of greater time or financial resources) as well as other issues that emerge through the analysis of data. Data from the interviews will be transcribed and coded across CSCs to develop listings of all activities and outcomes conducted. In preparation for this task, analysts will establish a codebook with a list of key concepts and areas of interest based on the questions asked during the interview and the document review. After compiling the list, each program lead will review a minimum of three interview transcripts to 1) determine whether the list is complete or whether additional variables are needed to capture every piece of data collected and 2) develop a list of response categories for some of the variables, where this makes sense. Information from each interview will be summarized across all interviews.

Methodology for stratification and sample selection. The sample selection process will be based on a purposive sample for each group of stakeholders at each CSC, with intentions to interview at least two members of each stakeholder group with the exception of CSC directors. The universe (n=4) of CSC directors will be interviewed.

Estimation procedure. There are no estimation procedures in the analysis of qualitative data.

Degree of accuracy. Findings from the site visits will be presented carefully to ensure that readers recognize the results are not generalizable beyond the findings of the specific sample of interviewees.

Unusual problems required specialized sampling procedures. NOAA does not anticipate the use of specialized sampling procedures.

Any use of periodic data collection cycles. The site visits will only occur at one point in time.

3. Describe the methods used to maximize response rates and to deal with nonresponse. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield "reliable" data that can be generalized to the universe studied.

Web Survey

The survey administration methods that will be used in this evaluation have been shown to yield high response rates when the survey is of reasonable length and when potential respondents consider the topic relevant. Surveys will be administered in multiple modes. These Web-based surveys have been efficiently designed with clear skip patterns and will take 25 minutes for scholarship recipients to complete and 15 minutes for the comparison group to complete. The most updated list of respondents and contact information will be obtained from the NOAA Student Performance Tracking System. Both email addresses and telephone numbers will be validated and updated as necessary using LexisNexis. The invitation sent out in advance of the survey will include language that communicates NOAA endorsement. This evaluation will use a multipronged strategy to ensure strong response rates. The first half (roughly 6 weeks) of data collection will focus on maximizing Web-based survey responses. The survey’s software has built-in tracking mechanisms to ensure that sample members can only complete the survey once; once a survey is complete, future attempts to access it will be denied. Those who prefer a phone interview, have no working email address, or have not responded to the online survey invitation will have the option of participating via the telephone (CATI) method, which will occur over a subsequent 6-week period following the online survey option. Each sampled individual will receive multiple call attempts in order to attempt a high response rate. All interviewing will be done from IMPAQ’s Survey Center, which houses a 55-station CATI facility. Given the expected response rate, the relatively small number of nonrespondents should have little impact on the expected results. Due diligence will be exercised to examine any potential differences between respondents, late responders, and nonrespondents to understand any potential bias introduced by non-response. We will examine the data for nonresponse bias by comparing the survey responses of on-time responders to those of late responders, on the premise that late responders are a proxy for nonrespondents. To do so, we will follow the recommendations of Lindner, Murphy, and Briers (2001) to identify the late responders. Those who complete the survey following the last wave of reminder contact will be identified as late responders. If this approach does not yield at least 30 responders for each survey, we will also include those who responded after the last two reminder contacts. Should this approach also fail to yield at least 30 responders, we will identify them as the latest 30 responses recorded in the database for each survey. We will use data elements such as the following to examine potential bias: program(s) applied for, year(s) applied, acceptance status, and application score. Results of this analysis will be included in the final report.

Data will be quality checked as they are received, and biweekly reports by respondent group will be created. When data collection is complete, the raw survey data will be archived, including the open-ended survey responses. Tables and graphs will be produced according to the analysts’ specifications following the research questions outlined in the evaluation plan. If the open-ended qualitative survey data are sufficiently rich and extensive, systematic thematic qualitative analysis will be conducted.

Site Visits

Site visits will be preceded by conversations with center directors identifying the directors’ suggestions for key metrics in the evaluation of the EPP program. These conversations will also ensure the site visit protocols address common issues across all four CSCs. In cooperation with CSC directors as well as other CSC staff members, our team will use constructed participant lists to purposively select participants for the site visit focus groups and interviews at each institution. We will interview at least two members of each stakeholder group with the exception of CSC directors. The universe (n=4) of CSC directors will be interviewed. Additionally, participants will be contacted with a letter of support from the NOAA OEd Director as well as endorsement from the CSC director (See Attachment B for the letter from the NOAA OEd to respondents.).

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections.

Web Survey

The survey instrument was programmed into Web-based survey software (Voxco). Multiple, iterative rounds of pre-testing were conducted by approximately ten staff members from Insight, IMPAQ, and NOAA. To provide perspective similar to that of the intended respondents, several former NOAA scholarship recipients also tested the instrument. Pre-testers thoroughly tested both survey instruments for any programming or logic inaccuracies. Based on the results of this testing, minor programming edits were made, followed by subsequent rounds of testing. After any minor changes were incorporated into the survey programming, timed testing indicated that the scholarship survey takes approximately 20 minutes to complete, and the comparison group survey takes approximately 10 minutes per respondent. NOAA will keep the slightly higher time estimates for completing the survey.

Site Visits

Each of the five site visit discussion guides (students, director, faculty members, community partners, and institutional partners) will be reviewed and revised in collaboration with NOAA staff members and then refined following the first site visit. These pretests will result in clarifications to the discussion guide and prompts as well as confirmed estimates of discussion length.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

NOAA has consulted with Insight and IMPAQ on matters of statistical design and analysis. The names and contact information for each team member appear in Table B.5.1 below.

Table B.5.1

Individuals Responsible for Statistical Aspects and Data Collection and Analysis

Name	Title (Project Role)	Organizational Affiliation and Address	Phone Number
Stéphane Baldi, Ph.D.	Vice President (Executive Project Director)	Insight Policy Research 1901 N. Moore Street, Suite 1100 Arlington, VA 22209	(703) 504-9486
Catharine Warner-Griffin, Ph.D.	Senior Researcher (Project Manager)	Insight Policy Research 1901 N. Moore Street, Suite 1100 Arlington, VA 22209	(703) 504-9493
Meg Trucano, Ph.D.	Research Analyst (Survey Design)	Insight Policy Research 1901 N. Moore Street, Suite 1100 Arlington, VA 22209	(571)758-5006
Amber Noel, M.S.	Research Analyst (Data Analyst)	Insight Policy Research 1901 N. Moore Street, Suite 1100 Arlington, VA 22209	(571)385-2460
Andrea Beesley, Ph.D.	Senior Research Associate (Task Lead)	IMPAQ International 1101 Vermont Avenue, NW 11th Floor Washington, DC 20005	(443) 832-2313

REFERENCES

Lindner, J. R., Murphy, T. H., & Briers, G. E. (2001). Handling nonresponse in social science research. Journal of Agricultural Education, 42(4), 43-53.

Page 5