Attach1_SFTXT Planning Matrix_12062012

SFTXT evaluation planning matrix

Attachment 1

Evaluation Planning Matrix

Evaluation questions	Category	Construct	Data sources
Were the eligibility criteria a barrier to participation among those indicating interest?	Methods: Recruitment	Eligibility criteria	Screener
Did those enrolled in the study differ from those who were not enrolled?	Methods: Recruitment	Non-response bias	Exit survey
Of those enrolled, how many complete each of the follow-up surveys? Are participants who drop out different than participants who do not? Are there particular questions participants do not answer? What proportion of incentives are claimed? Are follow-up strategies effective in converting non-responders to responders?	Methods: Attrition	Attrition	Baseline survey Quit-date survey 12week survey 24 week survey Incentive claims
What are the key intervention strategies/methods?	Intervention	Intervention components	Communication with NCI/MMG “Smokefree txt Theory Report” SMS message mapping document “Smokefree Teen A Theory-Informed Design” document
How engaged were participants with SMS content? How many times did they reply to messages? How many key word messages did they send? How many times did they click on a link in the text message? # of undeliverable messages # of declined messages (STOP) When did they opt out of the program?	Exposure/ Engagement	Engagement with SMS	SMS metrics
Do those who received the full program of text messages in (arm 3) recall receiving more messages than those in arms 1 (assessment only) and 2 (assessment + countdown messages)? Do those in arm 2 recall receiving more messages than those in arm 1?	Exposure/ Engagement	Recall of text messaging	1 week survey End of treatment survey

Evaluation Planning Matrix (continued)

Evaluation questions	Category	Construct	Data sources
How useful did respondents think Quittxt messages were in helping them stay smokefree, overcome cravings, etc. ? Were any specific types of messages perceived as more useful than others (quit date reminders; mood assessments; smokefree reminders, etc.)? Overall how useful did participants perceive the program to be?	Exposure/ Engagement	Perceived utility	1 week survey End of treatment survey
Does risk perception moderate the effects of the intervention?	Risk perception	Risk perceptions	Baseline
Are those who received the full program of text messaging (arm 3) less likely to predict that they will be smoking one year from now than those who only received assessment messages (arm 1) or assessment and countdown messages (arm 2)? Are those exposed to assessment only messages plus countdown messages (arm 2) less likely to predict that they will be smoking one year from now than those who received assessment messages only (arm 1)?	Proximal outcomes	Forecasted future smoking	Baseline End of treatment survey 12 week survey

Evaluation Planning Matrix (continued)

	Evaluation questions	Category	Construct	Data sources
Did exposure to the full text program of text messaging (arm 3) result in increased self-efficacy toward cessation initiation? Did exposure to assessment only messages plus countdown messages (arm 2) result in increased self-efficacy toward cessation initiation than assessment messages only (arm 1)? Did those who were more engaged with SMS have higher self-efficacy toward cessation initiation? Did exposure to the full program of text messaging (arm 3) result in increased self-efficacy toward cessation maintenance compared to the other groups)? Did exposure to assessment only messages plus countdown messages (arm 2) result in increased self-efficacy toward cessation maintenance than assessment messages only (arm 1)? Did those who were more engaged with the program have higher self-efficacy toward cessation maintenance than those who were less engaged?		Proximal outcomes	Self-efficacy	Baseline (most items) 6 week (1 item) 12 week survey (1 item) 24 week survey (1 item)
Did exposure to the intervention result in change in positive expectancies for smoking? Did the full program of messages (arm 3) result in greater changes in positive expectancies than assessment only messages (arm 1) or assessment and countdown messages (arm 2)? Did assessment and countdown messages (arm 2) result in greater changes in positive expectancies than assessment only messages (arm 1)?		Proximal outcomes	Smoking expectancies (i.e., costs and benefits of smoking)	Baseline End of treatment

Evaluation Planning Matrix (continued)

Evaluation questions	Construct	Data sources
Did the full program of text messaging (arm 3) result in higher motivation to quit than assessment only messages (arm 1) or assessment and countdown messages (arm 2)? Were those who were more engaged with SMS more likely to develop motivation? Did motivation/salience of cessation moderate the effects the intervention?	Motivation/salience of cessation	Baseline 6 week survey 12 week survey 24 week survey
Did full program of text messaging (arm 3) result in more skills development than assessment only messages (arm 1) or assessment and countdown messages (arm 2)? Did assessment and countdown messages (arm 2) result in more skills development than assessment only messages (arm 1)? Were those who were more engaged with SMS more likely to develop skills? Were those who were more engaged with web and SMS more likely to develop skills? Did skills/abilities moderate the effects the intervention?	Skills/abilities (manage cravings; emotional coping responses; develop quit plan)	6 week survey 12 week survey 24 week survey
Did those who received the full program of text messaging (arm 3) have more extra-treatment social support than those who received assessment only messages (arm 1) or assessment and countdown messages (arm 2)? Did assessment and countdown messages (arm 2) result in more extra-treatment social support than assessment only messages (arm 1)? Were those who were more engaged with SMS more likely to receive extra-treatment social support? Did extra-treatment social support moderate the effects the intervention?	Extra-treatment social support	Baseline 6 week survey 12 week survey 24 week survey

Evaluation Planning Matrix (continued)

Evaluation questions	Construct	Data sources
Are those who received the full program of text messaging (arm 3) more likely to quit than those who received assessment only messages (arm 1) or assessment and countdown messages (arm 2)? Are those exposed to assessment only messages plus countdown messages (arm 2) more likely to quit than those who received assessment messages only (arm 1)?	Current smoking behavior (cessation)	1 week survey 6 week survey 12 week survey 24 week survey
Are those who received the full program of text messaging (arm 3) more likely to reduce the number of cigarettes they smoke than those who received assessment only messages (arm 1) or assessment and countdown messages (arm 2)? Are those exposed to assessment only messages plus countdown messages (arm 2) more likely to reduce the number of cigarettes they smoke than those who received assessment messages only (arm 1)?	Reduction in rate/ frequency of smoking (# of non-smoking days, # of cpd)	Baseline 1 week survey 6 week survey 12 week survey 24 week survey Text message responses
Are those who received the full program of text messaging (arm 3) more likely to stay quit than those who received assessment only messages (arm 1) or assessment and countdown messages (arm 2)? Are those exposed to assessment only messages plus countdown messages (arm 2) more likely to stay quit than those who received assessment messages only (arm 1)?	Abstinence	Baseline 1 week survey 6 week survey 12 week survey 24 week survey Text message responses

Are those who received the full program of text messaging (arm 3) less likely to relapse than those who received assessment only messages (arm 1) or assessment and countdown messages (arm 2)? Are those exposed to assessment only messages plus countdown messages (arm 2) less likely to relapse than those who received assessment messages only (arm 1)?	Relapse (return to regular smoking after an established period of abstinence) Treatment Failure (failure to establish an initial period of abstinence)	1 week survey 6 week survey 12 week survey 24 week survey
Did environmental factors moderate the effects of the intervention?	Environment (household smoking ban; access in the home)	Baseline survey
Evaluation Planning Matrix (continued)
Evaluation questions	Construct	Data sources
Did socio-demographics moderate the effects of the intervention?	Socio-demographics (age, gender, race/ethnicity, SES)	Baseline survey
Did psychosocial factors moderate the effects of the intervention?	Psychosocial factors (mental health, negative mood regulation, perceived stress)	Baseline survey 6 week survey
Did smoking history moderate the effects of the intervention?	Smoking history (age at first cigarette, quit attempts, etc.)	Baseline survey
Did smoking patterns at baseline moderate the effects of the intervention?	Current smoking patterns	Baseline
Did use of technology moderate the effects of the intervention?	Technology use patterns	Baseline survey
Quit Process Measures	Withdrawal symptoms, cravings	1 week survey

1