U. Site-specific MDI Tables

U. Site-specific MDI Tables.docx

Rapid Cycle Evaluation of Operational Improvements in Supplemental Nutrition Assistance Program (SNAP) Employment & Training (E&T) Programs

U. Site-specific MDI Tables

OMB: 0584-0680

Document [docx]
Download: docx | pdf


Appendix U. Site-specific MDI Tables





This page has been left blank for double-sided copying.

Appendix U. Table B2.1.a. Colorado minimum detectable impacts based on administrative data

Test

Assumption

Total sample size

MDI (percentage points)

Test 1 (Denver, Broomfield, Larimer)

100 percent of anticipated sample

9,000

2.7 pp.


75 percent of anticipated sample

6,750

3.2 pp.


50 percent of anticipated sample

4,500

3.9 pp.


15 percent of anticipated sample

1,350

7.1 pp.

Test 2 (Montrose)

100 percent of anticipated sample

2,500

4.0 pp.


75 percent of anticipated sample

1,875

4.6 pp.


50 percent of anticipated sample

1,250

5.7 pp.


15 percent of anticipated sample

375

10.4 pp.

Note: Test 1 includes 5 research groups, while Test 2 includes 3 research groups. MDIs are calculated for a two-tailed test with 80 percent power at a 0.05 significance level, and are based on the following assumptions: (1) R-squared (the proportion of variation in the outcome explained by covariates) is 0.05; and (2) average take-up of SNAP E&T in the control group is 10 percent. (Colorado Department of Human Services estimates between 2 and 7 percent of work registrants participate in SNAP E&T across the four counties; we assume a higher take-up rate to be conservative.) Because we are calculating MDIs for a binary outcome, the MDI is highest for an outcome with a mean of 0.5 (since this would maximize the variance of the outcome). Assuming a higher take-up rate in this context (in which take-up is below 0.5) therefore increases the MDI, which is conservative because it requires a larger impact of the intervention to detect significant effects.

MDIs calculated using the following formula:

Where is the significance level, is the probability of type II error, and is the inverse of the t distribution with degrees of freedom (df) equal to the total sample size minus 1. Deff is the assumed design effect, is the share of variation in the outcome that can be explained by baseline covariates, is the variance of the outcome, and and are the treatment and control group sample sizes, respectively. The MDI calculations use the following parameter values: , 80, , deff=1, and . and are as listed in each row of the table.



Appendix U. Table B2.1.b. Connecticut minimum detectable impacts based on administrative data

Assumptions


MDI (percentage points)




Total responding sample size

3 research groups

2 research groups

7-month enrollment period (July-January)

808

11.8

9.6

6-month enrollment period (July-December)

588

13.8

11.3

3-month enrollment period (July-September)

428

16.2

13.2

Note: MDIs are calculated for a two-tailed test with 80 percent power at a 0.05 significance level, and are based on the following assumptions: (1) an R-squared (the proportion of variation in the outcome explained by covariates) of 0.05, (2) a binary outcome with a mean value of 0.50, and (3) a design effect of 1.0 for outcomes based on administrative data due to the absence of weighting. Sample sizes are based on estimates provided by Connecticut Community College leadership of the number of community college students enrolled in SNAP E&T from July 2023 to January 2024.

MDIs calculated using the following formula:

Where is the significance level, is the probability of type II error, and is the inverse of the t distribution with degrees of freedom (df) equal to the total sample size minus 1. Deff is the assumed design effect, is the share of variation in the outcome that can be explained by baseline covariates, is the variance of the outcome, and and are the treatment and control group sample sizes, respectively. The MDI calculations use the following parameter values: , 80, , deff=1, and . and are as listed in each row of the table.

.

Appendix U. Table B2.1.c. District of Columbia minimum detectable impacts based on administrative data

Pre-post analysis

RCT analysis

Total number of individuals in pre-intervention group

Total number of individuals in post-intervention group

MDI (percentage points)

Size of treatment and control groups

MDI (percentage points)

300

240

11.8

120 and 120

17.7

200

160

14.5

80 and 80

21.7

175

140

15.5

70 and 70

23.2

150

120

16.8

60 and 60

25.1

Note: The pre-post analysis compares a percentage outcome among the pre- and post-intervention groups. The size of the post-intervention group assumes that 20 percent of individuals in the pre-intervention group continue to participate in SNAP E&T in the post-intervention period; these individuals will be excluded from the analyses of outcomes in the post-intervention period due to exposure to both case management approaches. MDIs are based on a 0.80 power level and assume the mean value of the binary outcome is 0.50, baseline variables explain 5 percent of the variation in the outcome, response rates of 100 percent for outcomes based on administrative data, as well as a design effect of 1.0 for outcomes based on administrative data due to the absence of weighting.

MDIs calculated using the following formula:

Where is the significance level, is the probability of type II error, and is the inverse of the t distribution with degrees of freedom (df) equal to the total sample size minus 1. Deff is the assumed design effect, is the share of variation in the outcome that can be explained by baseline covariates, is the variance of the outcome, and and are the treatment and control group sample sizes, respectively. The MDI calculations use the following parameter values: , 80, , deff=1, and . and are as listed in each row of the table.



Appendix U. Table B2.1.d. Kansas minimum detectable impacts based on administrative data


Based on starting
sample size of 1,200

Assumption

Sample size
for each group

MDI (percentage points)

All individuals (100 percent of starting sample)



Estimate effect of receiving behaviorally informed text message appointment reminders compared to not receiving them on percentage of individuals who remain engaged in SNAP E&T activities

300 and 300

11.2

Estimate effect of receiving behaviorally informed text message nudges compared to not receiving them on percentage of individuals who remain engaged in SNAP E&T activities

300 and 300

11.2

Estimate effect of receiving behaviorally informed text message appointment reminders and nudges compared to not receiving them on percentage of individuals who remain engaged in SNAP E&T activities

300 and 300

11.2

Estimate effect of receiving behaviorally informed text message reminders or nudges compared to not receiving them on percentage of individuals who remain engaged in SNAP E&T activities

900 and 300

9.1

Note: MDIs are calculated for a two-tailed test with 0.80 power level at a 0.05 significance level and assume: (1) the mean value of the binary outcome is 0.50, (2) baseline variables explain 5 percent of the variation in the outcome, (3) response rates are 100 percent for outcomes based on administrative data, as well as (4) the design effect is 1.0 for outcomes based on administrative data due to the absence of weighting.

MDIs calculated using the following formula:

Where is the significance level, is the probability of type II error, and is the inverse of the t distribution with degrees of freedom (df) equal to the total sample size minus 1. Deff is the assumed design effect, is the share of variation in the outcome that can be explained by baseline covariates, is the variance of the outcome, and and are the treatment and control group sample sizes, respectively. The MDI calculations use the following parameter values: , 80, , deff=1, and . and are as listed in each row of the table.



Appendix U. Table B2.1.e. Massachusetts minimum detectable impacts based on administrative data


Based on starting
sample size of 30,000

Assumption

Sample size for each group

MDI (percentage points)

Compare text message treatment group 1 to text message treatment group 2 to estimate the effect of message content on expressing interest in learning more about E&T services

12,000 and 12,000

1.8

Compare (1) the combination of individuals who do not pass the survey screener and those who pass the screener and are placed in the control group for the assessment and (2) the text message control group to estimate the effect of the outreach message on the percentage of individuals who enroll in SNAP E&T. (All originating from treatment group T1.)

9,000 and 6,000

2.3

Compare (1) the combination of individuals who do not pass the survey screener and those who pass the screener and are placed in the control group for the assessment and (2) the text message control group to estimate the effect of the outreach message on the percentage of individuals who enroll in SNAP E&T. (All originating from treatment group T2.)

9,000 and 6,000

2.3

Compare (1) the combination of individuals in the assessment treatment group who were deemed not to be work ready and those in the assessment treatment group who were deemed to be work ready but were assigned to the career center control group and (2) the assessment control group, to estimate the effect of the assessment on the percentage of individuals who enroll in SNAP E&T. (All originating from treatment group T1.)

2,100 and 3,000

3.9

Compare (1) the combination of individuals in the assessment treatment group who were deemed not to be work ready and those in the assessment treatment group who were deemed to be work ready but were assigned to the career center control group and (2) the assessment control group, to estimate the effect of the assessment on the percentage of individuals who enroll in SNAP E&T. (All originating from treatment group T2.)

2,100 and 3,000

3.9

Compare the career center treatment group and the career center control group to estimate the effect of the warm handoff referral on the percentage of individuals who enroll in SNAP E&T. (All originating from treatment group T1.)

900 and 900

6.4

Compare the career center treatment group and the career center control group to estimate the effect of the warm handoff referral on the percentage of individuals who enroll in SNAP E&T. (All originating from treatment group T2.)

900 and 900

6.4

Note: MDIs are calculated for a two-tailed test with 0.80 power level at a 0.05 significance level. MDIs assume: the mean value of the binary outcome is 0.50, baseline variables explain 5 percent of the variation in the outcome, response rates of 100 percent for outcomes based on administrative data, as well as a design effect of 1.0 for outcomes based on administrative data due to the absence of weighting.

MDIs calculated using the following formula:

Where is the significance level, is the probability of type II error, and is the inverse of the t distribution with degrees of freedom (df) equal to the total sample size minus 1. Deff is the assumed design effect, is the share of variation in the outcome that can be explained by baseline covariates, is the variance of the outcome, and and are the treatment and control group sample sizes, respectively. The MDI calculations use the following parameter values: , 80, , deff=1, and . and are as listed in each row of the table.

.

Appendix U. Table B2.1.f. Minnesota-Hennepin minimum detectable impacts based on administrative data

Assumption

Total sample size

MDI (percentage points)

100 percent of projected sample

4,700

5.6

75 percent of projected sample

3,525

6.5

50 percent of projected sample

2,350

8.0

Note: Power calculations do not adjust for multiple comparisons. MDIs are calculated for a two-tailed test with 80 percent power at a 0.05 significance level and are based on the following assumptions: (1) R-squared (the proportion of variation in the outcome explained by covariates) is 0.05; (2) Average take-up of SNAP E&T in the control group is 50 percent; (3) there is no design effect from weighting because these outcomes will be based on administrative data.

MDIs calculated using the following formula:

Where is the significance level, is the probability of type II error, and is the inverse of the t distribution with degrees of freedom (df) equal to the total sample size minus 1. Deff is the assumed design effect, is the share of variation in the outcome that can be explained by baseline covariates, is the variance of the outcome, and and are the treatment and control group sample sizes, respectively. The MDI calculations use the following parameter values: , 80, , deff=1, and . and are as listed in each row of the table.



Appendix U. Table B2.1.g. Minnesota-Rural minimum detectable impacts based on administrative data

Total sample size

MDI (percentage points)

4,500

4.1

3,375

4.7

2,250

5.8

1,125

8.1

Note: MDIs are calculated for a two-tailed test with 80 percent power at a 0.05 significance level, and are based on the following assumptions: (1) an equal number of individuals are randomly assigned to the treatment and control groups (2) R-squared (the proportion of variation in the outcome explained by covariates) is 0.05; (3) Average take-up of SNAP E&T in the control group is 50 percent; (4) there is no design effect from weighting because these outcomes will be based on administrative data.

MDIs calculated using the following formula:

Where is the significance level, is the probability of type II error, and is the inverse of the t distribution with degrees of freedom (df) equal to the total sample size minus 1. Deff is the assumed design effect, is the share of variation in the outcome that can be explained by baseline covariates, is the variance of the outcome, and and are the treatment and control group sample sizes, respectively. The MDI calculations use the following parameter values: , 80, , deff=1, and . and are as listed in each row of the table.



Appendix U. Table B2.1.h. Rhode Island minimum detectable impacts based on administrative data



Based on starting
sample size of 5,000

Comparison

Main outcomes

Sample size for each group

MDI (percentage points)

All individuals (100 percent of starting sample)




Compare Treatment Group 2 (email web link) to control group to estimate the effect of message content on expressing interest in learning more about E&T services

Percentage of individuals who view the website

Percentage of individuals who complete and submit contact form in website link

Percentage of individuals who enroll in SNAP E&T

750 and 1,000

6.6

Compare Treatment Group 1 (text web link) to Treatment Group 2 (email web link) to estimate the effect of type of outreach messaging on outcomes

Percentage of individuals who view the website

Percentage of individuals who complete and submit contact form in website link

Percentage of individuals who enroll in SNAP E&T

750 and 750

7.1

Compare (1) the combination of individuals in Treatment Group 3 (text) who do not reply and those who do reply but are assigned to receive the existing assessment and (2) Treatment Group 1 (text web link) to estimate the effect of replying to outreach messages on outcomes

Percentage of individuals who complete and submit contact form in website link in the email or who reply to the text; percentage that completes form or responds within 1 month of receiving the message

Percentage of individuals who enroll in SNAP E&T

700 and 750

7.2

Compare (1) the combination of individuals in Treatment Group 4 (email) who do not reply and those who do reply but are assigned to receive the existing assessment and (2) Treatment Group 2 (email web link) to estimate the effect of replying to outreach messages on outcomes

Percentage of individuals who complete and submit contact form in website link in the email or who reply to the text; percentage that completes form or responds within 1 month of receiving the message

Percentage of individuals who enroll in SNAP E&T

700 and 750

7.2

Compare Treatment Group 3A (text) who receives enhanced assessment and Treatment Group 3B who receives current assessment to estimate the effect of the enhanced assessment on outcomes measuring whether individuals are a “better fit” with providers. (Both groups initially received a text message inviting them to reply “Yes” to learn more about E&T.)

Percentage of individuals who finish an assessment

Percentage of individuals who are referred to a provider

Percentage of individuals who start intake at a provider

Percentage of individuals who stay with a provider

Percentage of individuals who get assigned to a component

Percentage of individuals who start a component

Percentage of individuals who remain in the component for a certain period of time

Percentage of individuals who experience an inter-provider referral

Percentage of individuals who are referred to another provider or back to DHS

450 and 450

9.1

Compare Treatment Group 4A (email) who receives the enhanced assessment and Treatment Group 4B who receives the current assessment to estimate the effect of the enhanced assessment on outcomes measuring whether individuals are a “better fit” with providers. (Both groups initially received an email inviting them to reply “Yes” to learn more about E&T.)

Percentage of individuals who finish an assessment

Percentage of individuals who are referred to a provider

Percentage of individuals who start intake at a provider

Percentage of individuals who stay with a provider

Percentage of individuals who get assigned to a component

Percentage of individuals who start a component

Percentage of individuals who remain in the component for a certain period of time

Percentage of individuals who experience an inter-provider referral

Percentage of individuals who are referred to another provider or back to DHS

450 and 450

9.1

Note: MDIs are calculated for a two-tailed test with 0.80 power level at a 0.05 significance level. MDIs assume the mean value of the binary outcome is 0.50, baseline variables explain 5 percent of the variation in the outcome, response rates of 100 percent for outcomes based on administrative data, as well as a design effect of 1.0 for outcomes based on administrative data due to the absence of weighting.

MDIs calculated using the following formula:

Where is the significance level, is the probability of type II error, and is the inverse of the t distribution with degrees of freedom (df) equal to the total sample size minus 1. Deff is the assumed design effect, is the share of variation in the outcome that can be explained by baseline covariates, is the variance of the outcome, and and are the treatment and control group sample sizes, respectively. The MDI calculations use the following parameter values: , 80, , deff=1, and . and are as listed in each row of the table.




File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File TitleMathematica Report
AuthorJames Mabli
File Modified0000-00-00
File Created2023-09-28

© 2024 OMB.report | Privacy Policy