Supporting Statement B

Status: Original and derived artifacts are available for this document.

Download: docx | pdf | html

SUPPORTING STATEMENT – PART B
    B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
If the collection of information employs statistical methods, the following information should be provided in this Supporting Statement:
    1. Description of the Activity
Background: The survey is designed to be part of an ongoing portfolio analysis of work experience programs, or “WEPs” (e.g., internships, apprenticeships, post-doctoral positions) that promote science, technology, engineering, and mathematics (STEM) capabilities and are affiliated with the Department of War (DoW). The purpose of the survey is to gain a better understanding of WEP participants’ educational and professional backgrounds, what they gain from completing WEPs, potential barriers to their success, and how programs may be improved for future cohorts.  
Recruitment and Sampling: The universe of potential respondents includes adults (age 18 and up) who, within the 12-24 months prior to the survey, attended any of the DoW STEM WEPs that have chosen to participate in the portfolio analysis. WEP sponsors will distribute a study invitation that contains a link to the online survey and a description of the effort to their most recent cohort of participants. Potential respondents will read the invitation and then decide whether they wish to provide feedback on the most recent program that they attended by clicking a static link that takes them to the survey.  The table below provides a summary of the projected recruitment efforts.  The initial collection was conducted in 2023, and the proposed update is to allow use of the survey instrument by WEP sponsors on an annual basis. 

Collection Round
Strata
Universe of Potential Respondents (estimated)
Enrolled Respondents (estimated)
Response Rate (estimated)
Annually for a 5-year period
None
1500 per year
500
33.3%

    2. Procedures for the Collection of Information
No statistical methodologies or estimation procedures are being employed to stratify or otherwise structure the sample selection.  Programs that voluntarily opt into the study each year are advised to invite all people who participated in their programs within the past 12-24 months to take the survey.  Potential respondents then decide whether they wish to voluntarily opt into taking the survey. 
This was initially proposed as a one-time collection event, but the current update requested is for this to be a generic request that can be used in subsequent years by STEM-oriented WEP affiliated with DoW. The survey participants will be a unique cohort each year so as not to increase the burden on individual participants.
    3. Maximization of Response Rates, Non-response, and Reliability
This guidance outlines baseline recommendations and proven practices for evaluation partners deploying the ongoing collection tool. Evaluation partners are encouraged to use the foundational strategies outlined below, originally established during the initial Institute for Defense Analyses (IDA) baseline evaluation, while maintaining the flexibility to tailor specific methodologies to their program environments.
    1. Participant recruitment should be routed through trusted program intermediaries (e.g., WEP representatives, mentors, and site coordinators) rather than cold outreach. This approach has potential to enhance perceived credibility and provides additional anonymity between evaluators and respondents.
    2. Recruitment messaging should use clear, nontechnical language that outlines the study purpose, data protection protocols, estimated time commitment, and voluntary participation. It is recommended to also customize email communication templates to match program specific branding. 
    3. The survey instrument was designed so that closed ended or discrete choice items comprise most questions (i.e., 91% of survey items) to reduce response fatigue. Interactive response elements like rating scales were integrated to ensure a completion time of 10-15 minutes. Additionally, all the survey questions are optional to provide respondents with agency over their time and participation levels.

The survey uses volunteer sampling to capture a variety of feedback about the programs rather than testing narrow statistical hypotheses. As such, evaluation partners should seek to learn from as many former participants as possible to capture subjective sentiments or perceptions of work experience programs. 

The survey instrument’s measurement reliability was developed by adapting questions from established validated survey instruments. For example, the questions about “goal jobs” were adapted from a long-standing survey used by the National Science Foundation. In addition, during the piloting phase, IDA refined confusing questions to ensure ease of use. 

If DoW Components and/or other defense agencies would like to assess participant perceptions of a collection of WEPs they sponsor, it is recommended that program evaluators contextualize findings by tracking program-level participation and evaluating potential non-response bias. However, since the survey is drawing upon volunteers, it is difficult to ensure the generalizability of the sample to the broader population.  In the prior collection, IDA took the following approach when examining a broad group of WEPs: 
    1. Calculate Program Response Rate: Divide the number of completed survey responses by the total number of individuals invited to respond to a specific program. 
    2. Calculate Average Response Rate: Calculate the mean response rate (i.e., Program Response Rate) across all participating programs.
    3. Response Categorization: Group programs using a standard deviation distribution to assess potential skew.  
Response Category
Criteria
Low
Program Response Rate is more than 1 standard deviation lower than the Average Response Rate
Medium
Program Response Rate is within +/- 1 standard deviation of the Average Response Rate
High
Program Response Rate is more than 1 standard deviation higher than the Average Response Rate

    4. The analysis primarily focuses on patterns across groups of programs, rather than comparing individual programs to each other in a more granular way.  After generating preliminary findings, the Response Category will be used as a sensitivity test to check whether findings are disproportionally skewed by programs that have particularly low response rates so appropriate measures can be taken.
    4. Tests of Procedures
The survey instrument was developed and validated through a multistage process including a pilot test and full-scale survey instrument, both conducted by IDA, before its adoption as an ongoing collection tool. This process ensures that evaluation partners are using a tested and reliable instrument.
    1. A baseline pilot test was conducted with a representative cohort of five IDA personnel who mirrored the characteristics of the target respondent population (i.e., participants in DoW-affiliated STEM work experience programs). The primary objectives of this initial test were to:
        ◦ Validate Completion Time: Confirm that the survey could be completed within the target 15-minute timeframe to minimize respondent burden.
        ◦ Refine Question Clarity: Solicit direct feedback to identify and revise any confusing or ambiguous wording, thereby improving the overall user experience.
    2. Following the pilot, the survey instrument was successfully deployed at scale as the primary data collection tool for the Portfolio Review of DoD's Work Experience Programs: Analysis of Program and Participants' Perspectives (IDA Product 3002369). This large-scale deployment, which gathered responses from 275 participants across at least 19 different programs, served as a comprehensive, real-world validation of the survey's procedures and effectiveness.

    5. Statistical Consultation and Information Analysis
The following IDA representatives were consulted on the statistical aspects of the design as well as the collection and analysis of information during the pilot phase and launch of the survey instrument:
    • Dr. James Belanich, Institute for Defense Analyses, 703-845-6606.
    • Dr. Alexis Pang, Institute for Defense Analyses, (703) 845-2054.