PSTAP Power Analysis

U.S. Department of Veterans Affairs

Office of Transition and Economic Development (TED)

Post-Separation Transition Assistance Program Assessment (PSTAP)

Power Analysis

March 13, 2019

Submitted by:

Economic Systems Inc. (Prime Contractor)

3120 Fairview Park Drive, Suite 500

Falls Church, VA 22042

703.642.5225

www.econsys.com

Westat, Inc. (Subcontractor)

1600 Research Boulevard

Rockville, MD 20850

301.251.1500
www.westat.com

Introduction

This document provides a power analysis of the proposed cross sectional survey conducted by Economic Systems Inc. (EconSys) for the U.S. Department of Veterans Affairs (VA) Office of Transition and Economic Development (TED). This analysis is being conducted to determine whether the proposed sample of transitioning servicemembers will provide sufficient data to identify meaningful differences in outcomes between transitioning servicemembers who did or did not receive Transition Assistance Program (TAP) training. This document first lays out our sampling assumptions. Next, the power analysis and conclusions are presented.

Assumptions

In order to conduct the analysis, the study team used the following assumptions based on research and discussions with TED personnel:

1. Total separations from Armed Forces, FY 2018 (Active and Reserve Component): 198,784

2. Our sampling approach is to take a census of all separating members within particular 2-month window. We will refer to each such census as a “cohort”. The estimated sample size of each cohort = (198,784 / 12) * 2, or 33,130

3. The cross-sectional surveys will be fielded to three distinct cohorts:

1. 5-6 months since separation (cohort 1)

2. 11-12 months since separation (cohort 2)

3. 35-36 months since separation (cohort 3)

4. For each cohort, we estimate a 20 percent response rate to the PSTAP cross-sectional survey.

5. Estimated number of completed surveys per cohort = 33,130 * .2, or 6,626
   Estimated number of completed surveys from all three cohorts combined = 6,626 * 3, or 19,878

Summary

Overall, the power analysis concludes that the projected response rates will produce the number of responses necessary to detect statistically significant differences in outcomes between service members who did and did not attend TAP training. The following section provides our findings of the power analysis showing that the minimal detectable differences calculated are acceptable.

Power Analysis

Two power analyses are provided in the tables below, the first (Table A) for a single cohort (e.g., 6,626 completes from cohort 1), and the second (Table B) for the combined sample from all three cohorts.

Statistical power in the tables is expressed using Minimal Detectable Differences (MDDs). The MDD defines the difference between the means of an outcome measure (e.g., employment) for the treatment group and the control group that must exist to detect a statistically significant relationship. The tables assume a 95 percent confidence level (alpha = .05), and a one-tailed significance test.

Each table provides MDDs (expressed as percentages) for two scenarios. In the first row (scenario 1) we assume 80 percent of cases are in the treatment group (i.e., participated in TAP) and 20 percent in the control (did not take TAP). In the second row (scenario 2) we assume 85 percent of cases in the treatment group and 15 percent in the control group. For both scenarios, we provide MDDs for response rates of 25 percent, 20 percent, and 15 percent.

Table A. Minimum Detectable Differences, 95% confidence level (alpha = .05) for one cohort*
Subgroup proportions (treatment vs. control)	Number of respondents, assuming 25%, 20%, and 15% response rates from a starting sample of 33,130
	N = 8,283	N = 6,626	N = 4,970
Scenario I: 80%, 20%	5.4%	6.0%	7.0%
Scenario II: 85%, 15%	6.2%	7.0%	8.0%

*MDDs are for one-tailed comparisons

Table A, center column shows that under scenario 1, at a 95 percent confidence level, a sample size of 6,626 completed surveys—which we would achieve assuming a 20 percent response rate—we would detect as statistically significant a difference of 6 percent in the means between treatment and control groups for a given outcome measure (e.g., employment). As an example, if 70 percent of non-TAP participants (the control group) report being employed and 76 percent of TAP participants (the treatment group) report being employed , this difference would meet the MDD and be statistically significant at the 95 percent confidence level.

Table B below presents a power analysis assuming that the survey data from the three cohorts are combined, resulting in a pooled sample.

Table B. Minimum Detectable Differences, 95% confidence level (alpha = .05) for three cohorts combined*
Subgroup proportions (treatment vs. control)	Number of respondents in pooled data set, assuming 25%, 20%, and 15% response rates and a starting sample of 33,130 per cohort
	N = 24,848	N = 19,878	N = 14,909
Scenario I: 80%, 20%	3.1%	3.5%	4.0%
Scenario II: 85%, 15%	3.6%	4.0%	4.6%

*MDDs are for one-tailed comparisons

Table B shows the increased statistical power—expressed in smaller MDDs as compared with Table A—resulting from the larger number of cases available when the survey results from three cohorts are pooled. Because of the larger sample size, a 3.5 percent difference in the employment rate between TAP participants and non-participants will be large enough to constitute a statistically significant difference. The tables indicate that regardless of whether cohorts are analyzed separately or pooled together we will have enough statistical power to detect significant effects associated with TAP, even if differences in outcomes between participants and non-participants are relatively small.