SMART Scholar and SF Survey_SSB

SUPPORTING STATEMENT – PART B

B.  COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

If the collection of information employs statistical methods, the following information should be provided in this Supporting Statement:

1.  Description of the Activity

Background: The “SMART Scholars” and “SMART SF” surveys are part of an evaluation of the Science, Mathematics and Research for Transformation (SMART) Scholarship for Service Program. The purpose of the scholar survey is to gain a better understanding of scholars’ perspective on the program and its impact on the scholar. The purpose of the sponsoring facilities (SF) survey is to gain a better understanding of DoD SF use of the SMART Program.

Recruitment and Sampling: The universe of potential respondents for the scholar survey includes all participants in the SMART Program, from the award years of 2006 through 2021. There have been approximately 3783 such awards, with 104 of those award recipients receiving more than a single award, for a total number of unique recipients being approximately 3679. For this evaluation, IDA will ask the SMART Support Contractor to send a survey invitation email that contains a link to the online survey to the most recent address that is on file. Potential respondents will read the invitation and then decide whether they wish to participate by clicking a static link that takes them to the survey. We expect a response rate of approximately 50%.

Collection Round	Strata	Universe of Potential Respondents (estimated)	Enrolled Respondents (estimated)	Response Rate (estimated)
1 of 1	None	3679	1800	50%

The universe of potential respondents for the SF survey includes SFs that are currently using the SMART Program. While there have been almost 200 SFs that have had SMART scholars, many of those are not currently active with SMART. This survey will only be requested from SFs that have engaged with SMART in the past 3 years, which would be approximately 120. For this evaluation, IDA will ask the SMART Support Contractor to send a survey invitation email that contains a link to the online survey to the SF point of contact email address that they have on file. Potential respondents will read the invitation and then decide whether they wish to participate by clicking a static link that takes them to the survey. We expect a response rate of approximately 50%.

Collection Round	Strata	Universe of Potential Respondents (estimated)	Enrolled Respondents (estimated)	Response Rate (estimated)
1 of 1	None	120	60	50%

2.  Procedures for the Collection of Information

The scholar survey is drawing upon a volunteer sample from all participants in the SMART Program, so there are no statistical methodologies or estimation procedures employed to stratify or otherwise structure the sample selection. Potential respondents will decide whether they wish to voluntarily opt into taking the survey.

The SF survey is drawing upon a volunteer sample from SMART SFs, so there are no statistical methodologies or estimation procedures employed to stratify or otherwise structure the sample selection. Potential respondents will decide whether they wish to voluntarily opt into taking the survey.

Since this is a one-time collection event, there are no periodic or cyclical collection procedures planned to reduce respondent burden.

3.  Maximization of Response Rates, Non-response, and Reliability

IDA is using three techniques to maximize response rates.

1. The SMART Support Contractor will send the SMART scholars and SFs an email, describing the survey and request that scholars participate by clicking on a link. Respondents regularly receive emails from the SMART Support Contractor, so this is a known entity for respondents. Working through a trusted program support entity should increase the response rate as compared to if IDA would independently send the study invitation.

2. IDA is using a recruitment email designed to encourage response rates by being informative using non-technical language. The recruitment email provides information about who is doing the study, the types of questions will be asked, and how the information they choose to share may be used. The recruitment email also underscores the steps IDA is taking to protect respondents’ confidentiality and agency with respect to managing information, participation levels, and time.

3. The survey instrument was designed to maximize the acquisition of useful information gained on all of the program phases, while minimizing items that don’t apply to all respondents. A skip logic will be used to only provide survey items that are relevant to particular respondents. While the total printed survey may be rather long, there are many logic-determined points within the survey that will reduce the presentation of items to those that the questions does not apply. Also, most of the survey items are either discrete choice or closed, which makes them significantly easier and faster than open-ended questions. All questions are also optional, allowing respondents to determine if they want to respond or not.

Both surveys are using volunteer sampling, since the goal is to capture a variety of feedback across the SMART Program and SMART SFs, rather than test a specific hypothesis using a statistically representative sample. As such, IDA is seeking to learn from however many SMART scholars and SFs are willing to provide program feedback.

The surveys are SMART Program specific, so no broad established survey instrument could be used. The scholar survey does include items from a prior SMART Scholar survey (2016) so that a comparison of responses could be assessed as the program has evolved.

Since both surveys are SMART Program specific, it would be inappropriate to assume generalizability beyond SMART participants. The analysis primarily focuses on outcomes of the SMART Program, rather than comparing individual participants to each other.

4.  Tests of Procedures

Survey procedures will be pilot tested with 5 IDA employees/interns to determine the proper skip logic and presentation of survey items. The surveys were reviewed by the SMART Program Office and the DoD STEM Directorate to ensure that the items were relevant to information that will be valuable as part of the outcome evaluation.

5.  Statistical Consultation and Information Analysis

The following individuals were consulted on the statistical aspects of the design.

• Dr. James Belanich, Institute for Defense Analyses, 703-845-6606

• Dr. Sujeeta Bhatt, Institute for Defense Analyses, 703-578-2719

The following individuals and organizations will be involved in the collection and analysis of information.

• Dr. James Belanich, Institute for Defense Analyses

• Dr. Sujeeta Bhatt, Institute for Defense Analyses

2