Process Evaluation

Models of SNAP-Ed and Evaluation: Process Evaluation Instruments

Appendix B.

Data Collection Instruments
for Process Evaluation

Secondary Data Collection Instruments

Data Abstraction Form for IA Application to FNS and 2010 SNAP-Ed Plans [Pre-Implementation]

Data Abstraction Form for IA-Led Evaluation [Pre-Implementation]

Data Abstraction Form for IA-Led Evaluation [Post-Implementation]

Project Resource and Expense Tracking Form for Program Administrator [Post-Implementation]

Data Abstraction Form for IA Application to FNS and 2010 SNAP-Ed Plans

[Pre-Implementation]

IA: _______________________________________________________________

State: _______________________________________________________________

Program name: _______________________________________________________________

Data abstractor: _______________________________________________________________

Date of abstraction: __________________________________________________________

Other resources used (if applicable): ___________________________________________________

TOPIC AREA 1: Formative Research and Intervention Design

1. Target audience(s)

2. Reach or intended size of intervention

3. Behavioral goals and objectives of 2010 nutrition education effort /intervention:

____ A. Dietary Quality

Intended impact/change	Specific objectives	Related education messages
Knowledge
Attitudes
Behavior	Increase fruit and vegetable consumption	Fruit and vegetables are fun!

____ B. Physical Activity

Intended impact/change	Specific objectives	Related education messages
Knowledge
Attitudes
Behavior

____ C. Food Resource Management

Intended impact/change	Specific objectives	Related education messages
Knowledge
Attitudes
Behavior

____ D. Food Safety

Intended impact/change	Specific objectives	Related education messages
Knowledge
Attitudes
Behavior

____ E. Food Security

Intended impact/change	Specific objectives	Related education messages
Knowledge
Attitudes
Behavior

____ F. Other

Intended impact/change	Specific objectives	Related education messages
Knowledge
Attitudes
Behavior

4. Methods to be used/general form of nutrition education and anticipated dose and intensity of intervention

____ A. Direct education

Dose (# of contacts with each participant)
Intensity (# of contacts X length of contact)

____ B. Indirect education

Dose (# of contacts with each participant)
Intensity (# of contacts X length of contact)

____ C. Social marketing

Dose (# of contacts with each participant)
Intensity (# of contacts X length of contact)

____ D. Other

Dose (# of contacts with each participant)
Intensity (# of contacts X length of contact)

5. Nutrition education materials

____ A. Materials developed by FNS

If modified FNS materials, how and why?

____ B. Materials developed by other State SNAP-Ed programs

If modified these existing materials, how and why?

____ C. Materials developed by other public nutrition educations programs

If modified these existing materials, how and why?

____ D. Materials developed by private agencies

If modified these existing materials, how and why?

____ E. Materials developed by project

Justification for development?

____ F. Other

6. Theoretical underpinnings for nutrition education

6. Evidence that suggest the intervention will be successful (i.e., pilot project results, previously tested instruments, etc.)

6. Key players in the design of the intervention

1. Who were the key players from the implementing agency?

2. Were there any partnerships with other public or private organizations that were key to the design and implementation plan of the intervention?

3. If so, how were these partnerships formed?

4. Other key players?

TOPIC AREA 2: Operational Steps Involved in Intervention Implementation

1. Management and oversight structure

1. Who are the program administrators and coordinators?

2. Who is responsible for monitoring the nutrition education delivery?

2-2. Qualifications of nutrition educator trainer(s)

1. Level of education

2. On-the-job training

3. Years of experience

3. Qualifications of nutrition educator provider(s)

3. Level of education

4. Specialized training

5. Years of experience delivering nutrition education

3. Plans for training of nutrition education providers

6. Level of education

7. Specialized training

8. Years of experience delivering nutrition education

3. Recruitment of intervention sites/participants

1. How were intervention sites selected and recruited?

2. How were participants recruited?

3. Efforts to retain participants and ensure participants receive maximum dose

TOPIC AREA 3: Resources Devoted to the Intervention

1. Summarize human capital anticipated for each project

1. At the administrative, coordination, oversight level, trainer

Title of position	Brief description of responsibilities	FTEs	Average salary for this position	Salary range for this position

2. At the nutrition educator level (per intervention site), IF APPLICABLE

Intervention site/unit of analysis (i.e. ,school, single classroom):

Total number of “units” involved in this demonstration project:

Title of position	Brief description of responsibilities	FTEs	Average salary for this position	Salary range for this position

3. IT/Technical Staff, IF APPLICABLE

Title of position	Brief description of responsibilities	FTEs	Average salary for this position	Salary range for this position

4. Other

Title of position	Brief description of responsibilities	FTEs	Average salary for this position	Salary range for this position

2. Describe the physical capital required to implement project

1. Space

2. Audio/visual

3. Computer/software

4. Other

3. Provide the proposed budget for the SNAP-Ed project (excluding costs associated with IA-led evaluation, if possible)

Budget category	Line Item	Budgeted amount
Personnel	Salaries/benefits
Operational	Non-capital equipment and supplies
	Materials
	Travel
	Administrative
	Building/space
	Maintenance
	Equipment and other capital
TOTAL Direct Costs
Indirect costs
TOTAL Costs

Data Abstraction Form for IA-Led Evaluation

[Pre-Implementation]

IA: _______________________________________________________________

State: _______________________________________________________________

Program name: _______________________________________________________________

Data abstractor: _______________________________________________________________

Date of abstraction: __________________________________________________________

Resources used for data abstraction:

__ IA’s Evaluation Plan (if available)

__ IA’s Evaluation Instruments

__ IA’s 2010 SNAP-Ed Plan

__ Other (if applicable) ________________________________________________________________

1. What is the research design for the impact evaluation (e.g., pre/post- survey design, only post-intervention data collection, quasi-experimental with control group)?

2. What is the rationale for this evaluation approach and/or what research is available to support the use of such methods?

3. What are the key measures or indicators used to assess the intervention’s impact?

4. What information is provided on reliability (internal consistency (alpha), test-retest reliability, and/or reliability across raters) and construct validity of measures?

5. Are the measures used scales or single-item measures?

6. Are pre-existing evaluation instruments being used or modified or are new instruments being developed?

7. Have the evaluation instruments ever been tested?

8. Were the same tools used during the pilot phase (if applicable)? If yes, were modifications made for any reason and what were they?

9. Who will be evaluated (e.g., children, parents, and/or teachers)?

10. What are the sampling techniques and sample size of the population being assessed?

11. Was a power analysis conducted? If so, provide the details including whether it indicates the sample is sufficient to detect statistically significant differences in outcomes between treatment and control/comparison groups, and whether published literature or pilot work were used as the basis for an anticipated program effect size.

12. What data collection techniques will be used and what is the planned timing for these techniques?

13. What methods are planned for increasing the likelihood that members of the target population will agree to participate in the data collection?

14. What process measures and data collection methods, if any are being used to assess intervention effectiveness?

15. What methods are planned for ensuring privacy of the participant responses?

16. What training will the data collectors be required to have before beginning data collection (e.g. classroom education and/or format and content of training by intervention evaluation staff)?

Data Abstraction Form for IA-Led Evaluation

[Post-Implementation]

IA: _______________________________________________________________

State: _______________________________________________________________

Program name: _______________________________________________________________

Data abstractor: _______________________________________________________________

Date of abstraction: __________________________________________________________

Resources used for data abstraction:

__ IA’s State SNAP Ed Annual Final Report

__ Other (if applicable) ________________________________________________________________

This data collection form will be used to summarize information about how the IA ACTUALLY evaluated the success of its SNAP-Ed intervention (as opposed to what was planned, which was assessed via the earlier IA-led Evaluation Data Abstraction Form).

1. Identify each impact that was assessed by the evaluation.

2. Describe the study population being evaluated and its size.

3. Describe the unit of assignment to the intervention and control groups. 

4. Describe how assignment to intervention and control groups was carried out. 

5. Describe how many units and individuals were in the intervention and control groups at the start of the intervention.

6. Describe each impact measure and indicate if impact data were collected through observation, self-report, or another method.

7. Describe at what point the data were collected from intervention and control group participants (e.g., during and post, follow-up, etc.).

8. Results-- compare intervention and control groups at each measurement point, by individual measure, and the number of participants measured at each point.   Describe any tests of statistical significance and the results.

9. References—who is the contact person for additional details? Please provide a reference to any other report of the evaluation.

Project Resource and Expense Tracking Form for Program Administrator

[Post-Implementation]

This data collection form will be used to summarize information about ACTUAL resources used for and expenses related to your SNAP-Ed intervention. In Section 1 we are requesting information that is specific to the planning, design, and implementation of the intervention ONLY. In Section 2 we are requesting information that is specific only to the evaluation (IA-led assessment) component of your intervention.

SECTION 1. In the following tables, please provide the requested information as it relates to the planning, design, and implementation of your INTERVENTION ONLY (Please do not include resources or expenses related to your impact evaluation).

1. Summarize actual human capital used for each project

1. At the administrative, coordination, oversight level, trainer

Title of position	Brief description of responsibilities	FTEs	Average salary for this position	Salary range for this position

2. At the nutrition educator level (per intervention site), IF APPLICABLE

Intervention site/unit of analysis (i.e. school, single classroom):

Total number of “units” involved in this demonstration project:

Title of position	Brief description of responsibilities	FTEs	Average salary for this position	Salary range for this position

3. IT/Technical Staff, IF APPLICABLE

Title of position	Brief description of responsibilities	FTEs	Average salary for this position	Salary range for this position

4. Other

Title of position	Brief description of responsibilities	FTEs	Average salary for this position	Salary range for this position

2. Describe the ACTUAL physical capital required to implement project

5. Space

6. Audio/visual

7. Computer/software

8. Other

3. Please provide the following information for ACTUAL expenditures related to the implementation of your SNAP-Ed intervention only (NOT FOR IMPACT EVALUATION)

Budget category	Line Item	(a) Non-Federal Public Funds		(b) Non-Federal, Non-Public Funds	(c) Total Non-Federal Funds (a+b)	(d) Federal Funds	Total Funds (c+d)
		Cash	In-kind
Personnel	Salaries/benefits
Operational	Non-capital equipment and supplies
	Materials
	Travel
	Administrative
	Building/space
	Maintenance
	Equipment and other capital
TOTAL Direct Costs
Indirect costs
TOTAL Costs

SECTION 2. In the following tables, please provide the requested information as it relates to the planning, design, and implementation of the EVALUATION component, or IA-led assessment of your SNAP-Ed intervention (Please do not include resources or expenses related to the planning, design, or implementation of your intervention).

1. Summarize actual human capital used for your evaluation

5. At the administrative, coordination, oversight level

Title of position	Brief description of responsibilities	FTEs	Average salary for this position	Salary range for this position

6. At the evaluator level, IF APPLICABLE

Intervention site/unit of analysis (i.e. school, single classroom):

Total number of “units” involved in this demonstration project:

Title of position	Brief description of responsibilities	FTEs	Average salary for this position	Salary range for this position

7. IT/Technical Staff, IF APPLICABLE

Title of position	Brief description of responsibilities	FTEs	Average salary for this position	Salary range for this position

8. Other

Title of position	Brief description of responsibilities	FTEs	Average salary for this position	Salary range for this position

2. Describe the ACTUAL physical capital required to evaluate this project

1. Space

2. Audio/visual

3. Computer/software

4. Other

3. Please provide the following information for ACTUAL expenditures related to the evaluation of your SNAP-Ed intervention only (NOT FOR IMPLEMENTATION)

Budget category	Line Item	(a) Non-Federal Public Funds		(b) Non-Federal, Non-Public Funds	(c) Total Non-Federal Funds (a+b)	(d) Federal Funds	Total Funds (c+d)
		Cash	In-kind
Personnel	Salaries/benefits
Operational	Non-capital equipment and supplies
	Materials
	Travel
	Administrative
	Building/space
	Maintenance
	Equipment and other capital
TOTAL Direct Costs
Indirect costs
TOTAL Costs

Primary Data Collection Instruments

Respondent Type	Instrument Format	Data Collection Timing (relative to intervention)		Demonstration Projects Where Instrument Will be Used
		Pre	Post	Eagle Play	All 4 Kids	EWPHCCS	About Eating
Instruments to be used Across Projects
Evaluation Manager	Discussion Guide						
Project Administrators/ Planners	Discussion Guide					
Trainers of the On-site Educators	Discussion Guide					
Direct Educators	Discussion Guide					
Parents of Child Participants	Group Interview Guide (for on-site visits)					
School Principals or Child Care Center Directors	Discussion Guide (for on-site visits)					
Child Care Teachers	Discussion Guide (for on-site visits) and Mail Questionnaire (for teachers at sample sites where there will not be on-site visits)					
Instruments to be used only in About Eating
About Eating Program Administrator	Discussion Guide						
About Eating Web-Site Developer	Discussion Guide						
Recruiters of About Eating Participants	Discussion Guide						
About Eating Pilot Participants	Web Questionnaire						
About Eating Intervention Participants	Discussion Guide						

Table 1. Data Collection Instruments for Process Evaluation by Respondent Type, Timing, and Intervention.

Primary Data Collection Instruments for All Sites

Discussion Guide for Implementing Agency Evaluation Manager [Pre-Implementation]

Discussion Guide for Implementing Agency Evaluation Manager [Post-Implementation]

Discussion Guide for Implementing Agency Evaluation Manager

[Pre-Implementation]

State: ________________________ Interviewer: ______________

Respondent: ________________________ Date of Interview: ______________

Title: ________________________ Study ID No: ______________

Organization: ________________________

Address: ________________________

________________________

________________________

Phone: ________________________

Fax: ________________________

Email: ________________________

Thank you for taking the time for this interview. The U.S. Department of Agriculture’s Food and Nutrition Service has contracted with Altarum Institute to conduct a study of the [NAME OF INTERVENTION] that is offering information to children and their families about healthy foods to eat and the importance of being active. Altarum is a health and nutrition policy research and consulting institute and our work focuses on helping to improve the health and nutrition status of children, families, and adults. This study will include not only outcome evaluation information but also process information on how it is being implemented and how you are evaluating the intervention. All of this will be useful to both FNS and to other SNAP-Ed implementing agencies that are planning to evaluate their own SNAP-Ed interventions.

We will be using first names only today. Everything you say is private. After we conduct several of these interviews, we will write a report for the U.S. Department of Agriculture’s Food and Nutrition Service. Your name will not appear anywhere in the report. Nothing said today will be attached to your name at any point. Nothing that you say will affect your job or be shared with your employers.

I expect that our discussion today will take 30 minutes. Before I begin, do you have any questions?

Overview of IA-Led Evaluation Design, Budget and Staffing

Several weeks ago we reviewed the IA application submitted to FNS, 2010 SNAP-Ed Plan, and additional updated materials you have provided to us about your evaluation plans. We summarized this information and sent you a synopsis for your review. To begin our discussion today we would like to go over that summary document with you and give you the opportunity to comment or suggest revisions and provide additional information that we could not fill in from the written materials.

1. After reading the summary does any of the information reported appear to be incorrect or inaccurately describe your project in any way?

Please highlight any information that is incorrect or needs more clarification and make the necessary corrections or additions.

a) Is this information incorrect because your project has changed in some way since submitting your 2010 SNAP-Ed Plan or did we misunderstand or misinterpret something?

1. We want to be sure we understand your staffing plan for the evaluation. Which project staff or other staff will be responsible for conducting the evaluation? Please name staff and time allotted and if they will be involved in data collection only, data analysis only, in project implementation, or in any combination of these three activities.

2. Will any quality control or monitoring take place during data collection? If so, please describe.

Thank you clarifying this overview of your evaluation plan.

Evaluation Planning Phase

Now let’s briefly talk about your experiences in the design and planning phase for this evaluation.

1. What challenges, if any, have you faced during the design and planning phases of this evaluation?

3. What factors do you feel have contributed most to a successful design and planning phase?

4. What lessons have you learned during this key phase of the evaluation design? What would you do differently? What would you do the same?

Anticipated Challenges for Implementation

2. What challenges do you anticipate for this evaluation as you now approach your initial evaluation data collection phase?

Dissemination of Evaluation Results

5. When do you expect to complete data collection? When do you anticipate that you will complete data analysis?

6. How do you intend to use and/or disseminate your evaluation results?

That ends my formal interview questions. Do you have any information about your evaluation plans, comments or recommendations that you would like to add?

Thank you very much for your time and input on this very important project.

Discussion Guide for Implementing Agency Evaluation Manager

[Post-Implementation]

State: ________________________ Interviewer: ______________

Respondent: ________________________ Date of Interview: ______________

Title: ________________________ Study ID No: ______________

Organization: ________________________

Address: ________________________

________________________

________________________

Phone: ________________________

Fax: ________________________

Email: ________________________

Thank you for taking the time for this interview. As you know, the U.S. Department of Agriculture’s Food and Nutrition Service has contracted with Altarum Institute to conduct a study of the [NAME OF INTERVENTION] that is offering information to children and their families about healthy foods to eat and the importance of being active. Altarum is a health and nutrition policy research and consulting institute and our work focuses on helping to improve the health and nutrition status of children, families, and adults.

This study will include not only outcome evaluation information but also process information on how it is being implemented and how you are evaluating the intervention. All of this will be useful to both FNS and to other SNAP-Ed implementing agencies that are planning to evaluate their own SNAP-Ed interventions.

As I mentioned during our last meeting, we will be using first names only today. Everything you say is private.. After we conduct several of these interviews, we will write a report for the U.S. Department of Agriculture’s Food and Nutrition Service. Your name will not appear anywhere in the report. Nothing said today will be attached to your name at any point. Nothing that you say will affect your job or be shared with your employers.

I expect that our discussion will take about 30-45 minutes today. Before I begin, do you have any questions?

Specific Changes from Planned to Actual Evaluation

We would like to know about the specific aspects of your evaluation that might have changed along the way. We want to be able to describe any deviations from the evaluation plan you described to us during our first meeting, and also know why you had to make any specific changes from your plans.

1. Let’s start with the evaluation design. What changes, if any, occurred from your planned evaluation design? What caused these changes?

7. What changes, if any, occurred in your process measures, outcome measures, your data collection tools, and/or your planned data collection techniques? What caused these changes?

8. What changes, if any, did you make in the methods for protecting participant privacy? What caused these changes?

9. What changes, if any, did you make [or are you planning to make] in your data analysis plan? What caused these changes?

10. What changes if any did you make in the staffing for your data collection or staffing for your data analysis?

9. Did you need more or less time than budgeted for staff to spend on the data collection? On the data analysis?

b) Why do you think you needed more/less time than budgeted for these evaluation tasks?

11. Did you have or are you anticipating any increased non-personnel costs or resources required for the evaluation? If yes, what additional costs or resources have been or will be needed compared to what you planned for?

Questions to Obtain Information Not Provided in Final Report

1. With many programs, there are alternative explanations of program outcomes that need to be ruled out due to plausible threats to validity. If you saw changes in the program outcomes, what other factors could explain the changes you see? [Probe as needed on validity threats such as competing programs, concurrent media campaigns, and the effects of maturation among evaluation participants.

2. (If needed) What was the response rate for the pre- and post-survey data collection (by intervention and comparison group, if available)?

3. (If needed) Could you provide tables showing the demographic information and number of participants in the intervention and comparison groups, as well as the statistical tests you used to assess baseline comparability across treatment conditions?

4. (If needed) Can you describe the analyses and methods you used to handle attrition bias?

5. (If needed) What procedures did you use to account for missing data?

6. (If needed) What subgroup analyses were conducted for primary outcomes?

Lessons Learned

Next let’s talk about your overall experience in carrying out this evaluation and what you see as lessons learned and recommendations for the future.

3. Other than those we discussed above, what challenges, if any, have you faced during the implementation of this evaluation? [Refer back to the anticipated challenges cited by the interviewee prior to beginning the IA-led evaluation.]

12. What do you think worked very well in the implementation of this evaluation? What factors contributed to what worked well?

13. What do you think did not work well and what factors contributed to this?

14. What lessons have you learned from this evaluation design?

15. What would you do differently?

16. What would you be sure to do the same?

Dissemination Plans

17. How do you now plan to use and/or disseminate your evaluation results?

That ends my formal interview questions. Do you have any comments or recommendations that you would like to add?

Thank you very much for your time and input on this important project.

Primary Data Collection Instruments for:

CNNS, UNV, NYSDOH

Discussion Guide for Implementing Agency Program Administrator [Pre-Implementation]

Discussion Guide for Implementing Agency Program Administrator [Post-Implementation]

Discussion Guide for Trainers of On-Site Nutrition Educators [Pre-Implementation]

Discussion Guide for Trainers of On-Site Nutrition Educators [Post-Implementation]

Discussion Guide for On-Site Nutrition Educators [Pre-Implementation]

Discussion Guide for On-Site Nutrition Educators [Post-Implementation]

Discussion Guide for School Principals/Childcare Center Directors [Pre-Implementation]

Discussion Guide for School Principals/Childcare Center Directors [Post-Implementation]

Discussion Guide for Parents/Caregiver GROUP Discussions [Post-Implementation ONLY]

Discussion Guide for Implementing Agency Program Administrator

[Pre-Implementation]

State: ________________________ Interviewer: ______________

Respondent: ________________________ Date of Interview: ______________

Title: ________________________ Study ID No: ______________

Organization: ________________________

Address: ________________________

________________________

________________________

Phone: ________________________

Fax: ________________________

Email: ________________________

Thank you for taking the time for this interview. The U.S. Department of Agriculture’s Food and Nutrition Service has contracted with Altarum Institute to conduct a study of the [NAME OF INTERVENTION] that is offering information to children and their families about healthy foods to eat and the importance of being active. Altarum is a health and nutrition policy research and consulting institute and our work focuses on helping improve the health and nutrition status of children, families, and adults. The purpose of the study is to evaluate several SNAP (Supplemental Nutrition Education Program)-Education models around the country and to provide recommendations for how these interventions could be improved to better serve the children and families in your community. We also will be evaluating how the intervention might be replicated in other communities.

Although there are only a select number of programs participating in this evaluation, we will do our best to aggregate data wherever possible in order to avoid information being tied back to a particular respondent. Nothing said today will be attached to you, and nothing that you say will affect your job or be shared with your employers.

Today we will specifically be discussing the planning process and your expectations for the intervention. Once it has been implemented, we will follow up with you to find out whether the intervention met your expectations and how it might be improved. I expect that this interview will take about 45 minutes. Thank you for taking the time to speak with me.

Before I begin, do you have any questions?

[Send project summary to administrator(s) prior to pre-interview]

Review of Abstraction Summary:

Several weeks ago we reviewed your IA application that was submitted to FNS, your 2010 SNAP-Ed Plan, and your nutrition education materials, and sent you a synopsis of your project based on this information. To begin our discussion today we would like to review that summary with you and give you the opportunity to comment on and/or suggest revisions to the summary.

2. After reading the summary, does any of the information reported appear to be incorrect or inaccurately describe your project in any way?

Please highlight any information that is incorrect or needs more clarification and make the necessary corrections or additions.

a) Is this information incorrect because your project has changed in some way since submitting your 2010 SNAP-Ed Plan or did we misunderstand or misinterpret something?

Thank you for reviewing the project summary we created and providing this feedback. Now let’s briefly talk about the planning and design phase of your project.

18. What challenges, if any, have you faced during the design and planning phases of this nutrition education program?

1. What factors do you feel have contributed most to a successful design and planning phase (prompts: using education materials that were already developed, good communication between contributors, knowledgeable staff, establishment of strong partnerships, etc.)?

2. What lessons have you learned during this key phase of program development? What would you do differently? What would you do the same?

Okay, now I would like to shift our focus to the upcoming implementation of your SNAP-Ed project.

19. Now that you are ready to transition from the planning and design phase of your project to the implementation phase, what challenges, if any, are you anticipating?

20. Do you feel that the environment in which the intervention will take place will be able to support the intended change in behavior, knowledge, and/or attitudes?

1. For example, do you have any sense of the teacher’s buy-in and/or enthusiasm about the intervention and what impact this might have on the children?

2. Does the school/child care center offer the children healthy foods options and are healthy foods otherwise available?

3. What, if any, other nutrition education messages are the children in the intervention sites being exposed to (that you are aware of)?

21. Did the program have any difficulty recruiting adequate staff for the nutrition education delivery?

[IF YES]

a) What were the recruitment challenges/problems?

22. Please describe any quality control and monitoring efforts that will take place during implementation?

That ends my formal interview questions. Do you have any comments or recommendations that you would like to add?

Thank you very much for your time and input on this very important project. As I mentioned, we will follow up and talk with you after the intervention and evaluation period are over.

Discussion Guide for Implementing Agency Program Administrator

[Post-Implementation]

State: ________________________ Interviewer: ______________

Respondent: ________________________ Date of Interview: ______________

Title: ________________________ Study ID No: ______________

Organization: ________________________

Address: ________________________

________________________

________________________

Phone: ________________________

Fax: ________________________

Email: ________________________

Thank you for taking the time for this interview. As I told you during our last meeting, the U.S. Department of Agriculture’s Food and Nutrition Service has contracted with Altarum Institute to conduct a study of the [NAME OF INTERVENTION] that is offering information to children and their families about healthy foods to eat and the importance of being active. Altarum is a health and nutrition policy research and consulting institute and our work focuses on helping improve the health and nutrition status of children, families, and adults.

As mentioned during our last meeting, nothing said today will be attached to you, and nothing that you say will affect your job or be shared with your employers.

Today we will specifically discuss how the implementation of the program differed from your expectations. We also will discuss lessons learned and your feedback on how the program might be improved. I expect that this discussion will take about 45 minutes. I appreciate you taking the time to speak with me today.

Before I begin, do you have any questions?

Formative research and program design

I’d like to briefly discuss how, if at all, the implementation of your nutrition education intervention differed from what was originally planned. There are several aspects of implementation that I would like to cover.

1. Nutrition education messages. Were the nutrition education messages modified at any point during implementation?

[IF YES]

a) How and why were they modified?

23. Target audience. Did the target audience differ from what was originally planned?

[IF YES]

a) How and why did they differ?

24. Method of delivery. Were the methods of delivery (i.e., direct education, indirect education) modified during implementation for any reason?

[IF YES]

a) How and why were they changed?

25. Dose. Did the dose of nutrition education vary from what was originally planned (i.e., the number of lessons, the length of each lesson, etc.)?

[IF YES]

a) How and why did this vary from what was planned?

26. Reach. Were you able to implement the intervention at the originally proposed number of sites and do you feel that you reached the intended number of participants? Were there any factors that affected your ability to achieve the full, intended reach?

Nutrition education materials. Were the nutrition education materials modified at any point during implementation?

[IF YES]

a) How were the materials modified and why?

27. Timeline. To what extent were the original implementation timelines met?

a) What are the reasons for and implications of any departures from the original timelines?

Operational steps involved in program implementation

28. Did you find the level of staff, both in terms of qualifications and total number of staff (and types of staff), adequate for optimally delivering your nutrition education intervention?

29. What changes, if any, were made to planned key staff involvement and what were the reasons for any such changes?

30. Were any quality control and monitoring processes employed to maximize the fidelity/quality of the intervention delivery?

31. How effective were staff in delivering the intended nutrition education messages?

a) Why do you think these staff were effective/ineffective?

b) What could they have done differently to improve their effectiveness?

32. Do you think the nutrition educator training was sufficient?

a) What worked well?

b) What could have been improved?

33. Were planned recruitment (of participants/parents) efforts modified during implementation?

[IF YES]

a) How were recruitment efforts modified and for what reasons?

34. What recruitment methods did you find to be most effective/least effective?

35. In your opinion, how well was the program able to track participation?

36. Did previously identified partners remain engaged throughout the intervention?

37. Were these partnerships successful?

[IF YES]

a) What would you say contributed to their success?

[IF NO]

b) Why not?

Resources devoted to intervention

38. What were the actual time commitments for key staff (FTEs) if different than planned?

[IF YES]

a) Why did they differ?

39. How closely did the actual program cost components reflect the budgeted costs?

a) If there was a difference between budgeted and actual, what factors might have contributed to this?

40. Were the necessary type and quantity of materials, technology, etc. available to carry out the implementation as planned? If not, what else was needed?

Lessons learned for improvement and replicability

Next I’d like to talk about lessons learned during implementation of the study.

41. Overall, what factors were key to the success of this nutrition education program?

42. What factors hindered or limited the success of this nutrition education program?

43. Looking back over the past [NUMBER OF MONTHS] months, what lessons have you learned? What would be most valuable for another State or implementing agency to know if they were considering using this model?

44. In your opinion, are there any aspects of this SNAP-Ed program that would make it difficult to implement on a larger scale?

45. How did the FNS requirements for this demonstration project influence the design of your intervention project in ways that you had not anticipated when you applied for this demonstration project grant?

Assessment of IA-led evaluation

46. What methods were used to conduct the evaluation, if different than originally planned? If different, why?

47. Were the evaluation tools modified for any reason since the intervention began? If so, how and why?

48. Did the planned staff conduct the evaluation? If not, why not and who ended up conducting the evaluation?

49. Did the actual costs of the evaluation vary from what was planned? If so, how and why?

That ends my formal interview questions. Do you have any comments or recommendations that you would like to add?

Thank you very much for your time and input on this very important project.

Discussion Guide for Trainers of On-Site Nutrition Educators

[Pre-Implementation]

State: ________________________ Interviewer: ______________

Respondent: ________________________ Date of Interview: ______________

Title: ________________________ Study ID No: ______________

Organization: ________________________

Address: ________________________

________________________

________________________

Phone: ________________________

Fax: ________________________

Email: ________________________

Thank you for taking the time for this interview. The U.S. Department of Agriculture’s Food and Nutrition Service contracted with Altarum Institute to conduct a study of the [NAME OF INTERVENTION] that is offering information to children and their families about healthy foods to eat and the importance of being active. Altarum is a health and nutrition policy research and consulting institute and our work focuses on helping to improve the health and nutrition status of children, families, and adults.

The purpose of the study is to evaluate several SNAP-Education models around the country and provide recommendations for how these interventions could be improved to better serve the children and families in your community. We also will be evaluating how the intervention might be replicated in other communities.

We will be using first names only today. Everything you say is private. After we conduct several of these interviews, we will write a report for the U.S. Department of Agriculture’s Food and Nutrition Service. Your name will not appear anywhere in the report. Nothing said today will be attached to your name at any point. Nothing that you say will affect your job or be shared with your employers.

Today we will specifically be discussing the planning process and expectations for the intervention. Once it has been implemented, we will follow up with you to find out whether the intervention met your expectations and how it might be improved. I expected that our discussion will take about 30 minutes. I appreciate you taking the time to speak with me today.

Before I begin, do you have any questions?

Training Background:

1. First I’d like to ask you about the training that you are providing/have provided to the direct educators:

1. When will the training be provided?

2. Who designed this training? Was it the administrators, you, or someone else?

3. In terms of quantity, how much training will the instructors receive (in hours, units, etc.)?

4. What is the format for the training (e.g., group or one-on-one, written materials, lecture, etc.)?

5. Do you know why this particular format was chosen?

6. Do you think it will be effective? Why/why not?

7. What other aspects of the training do you think will work well? What would you change?

50. Will you be providing any ongoing training or assistance to the educators as they implement the program?

[IF YES] Please describe.

51. Will there be any monitoring of the educators as they implement the program?

[IF YES]

1. Who will be performing this monitoring?

2. What methods will be used (e.g., review of logs, site visits, etc.)?

3. What feedback will be provided to the instructors, and when?

52. Will there be any assessment of the trainers after they have finished the intervention?

[IF YES]

1. Who will be doing this assessment?

2. How will it be conducted?

3. What will be the outcome of the assessment (e.g., feedback given to instructors, etc.) ?

Formative Research and Intervention Design:

53. Finally I would just like to ask you whether you foresee any challenges in implementing the intervention as designed or planned?

[IF YES]

1. What challenges do you anticipate?

That ends my formal interview questions. Do you have any comments or recommendations that you would like to add?

Thank you very much for your time and input on this very important project. As I mentioned, we will follow up with your after the intervention has been implemented.

Discussion Guide for Trainers of On-Site Nutrition Educators

[Post-Implementation]

State: ________________________ Interviewer: ______________

Respondent: ________________________ Date of Interview: ______________

Title: ________________________ Study ID No: ______________

Organization: ________________________

Address: ________________________

________________________

________________________

Phone: ________________________

Fax: ________________________

Email: ________________________

Thank you for taking the time for this interview. As I told you during our first meeting, the U.S. Department of Agriculture’s Food and Nutrition Service has contracted Altarum Institute to conduct a study of the [NAME OF INTERVENTION] that is offering information to children and their families about healthy foods to eat and the importance of being active. Altarum is a health and nutrition policy research and consulting institute and our work focuses on helping to improve the health and nutrition status of children, families, and adults.

The purpose of the study is to evaluate several SNAP-Education models around the country and to provide recommendations for how these interventions could be improved to better serve the children and families in your community. We also will be evaluating how the intervention might be replicated in other communities.

Again, we will be using first names only today. Everything you say is privatel. After we conduct several of these interviews, we will write a report for the U.S. Department of Agriculture’s Food and Nutrition Service. Your name will not appear anywhere in the report. Nothing said today will be attached to your name at any point. Nothing that you say will affect your job or be shared with your employers.

Today I have just a few questions about your assessment of the intervention—whether it was effective, and what changes might be made. I expect that our discussion will take 30 to 40 minutes. I appreciate you taking the time to speak with me today.

Before I begin, do you have any questions?

Training Background:

First I would like to follow up with you on the training that you provided.

1. [IF THE TRAINING HAD NOT BEEN IMPLEMENTED BEFORE THE PRE-IMPLEMENTATION SURVEY]. First I would like to follow up with you on the training of the direct educators.

1. Was the format of the training successful? Why/why not? What would you change?

2. What other aspects of the training do you think worked well? What didn’t work well? What would you change?

54. [IF APPLICABLE] During our last interview, you indicated that you would be providing ongoing training or assistance to the educators as they implemented the program.

1. Do you feel that it was helpful? Why or why not?

2. What would you change about the way you provided ongoing training or technical assistance to the intervention educators?

55. (IF APPLICABLE) During our last interview, you indicated that there would be some performance monitoring of the educators as they implemented the program.

1. Who performed that monitoring?

2. What methods were used (e.g., review of logs, site visits, etc.)?

3. What feedback was provided to the instructors, and when?

4. Do you think this feedback was helpful to the instructors?

5. In the future, would you somehow incorporate what you learned from this monitoring into the initial training?

6. What could make the performance monitoring of the educators more effective or useful?

Formative research and Intervention Design:

Next I would like to ask about the success of certain aspects of the intervention.

56. Do you think that the direct classroom education for the children was effective for this target audience?

1. If not, why?

2. What format might have been more effective?

57. Do you think that the nutrition education materials designed for use in the classroom with children were effective?

1. Why or why not?

2. What might be more effective?

58. Do you think the take home nutrition education materials targeted for the parents were effective?

1. Why or why not?

2. What might make them more effective?

59. Do you think that the classes or family nights directed at the children’s parents were effective for this target audience?

1. Why or why not?

2. What might make them more effective?

60. Were the channels of communication effective?

1. Why or why not?

2. What might be more effective?

Lessons Learned for Improvement and Replicability:

Next I’d like to discuss some lessons learned with you.

61. When we interviewed you prior to the project’s start-up, you discussed some challenges you could foresee for effectively implementing it as planned. You said [FILL IN PRE-INTERVENTION RESPONSE]. Did these challenges in fact materialize and if so how did they change the implementation from what was planned?

62. Are there other ways the project implementation varied from what you had planned? [PROBE FOR ALL KEY DEVIATIONS FROM THE IMPLEMENTATION PLAN; FOR EACH DEVIATION, ASK THE FOLLOWING:]

1. What was the reason or cause for this change?

2. How was this change positive for the program and its potential impact on the target audience?

3. How was this change negative for the program and it potential impact on the target audience?

63. What particular aspects of the program do you think worked well?

64. What particular aspects of the program do you think did not work well?

65. Do you have any recommendations or suggestions for ways that the program could be revised and improved?

66. Do you have any other suggestions for how (schools or child care centers) in the target communities can encourage preschool children to eat more fruits and vegetables at home and encourage their parents to serve more fruits and vegetables?

That ends my formal interview questions. Do you have any comments or recommendations that you would like to add? Thank you very much for your time and input on this very important project.

Discussion Guide for On-Site Nutrition Educators

[Pre-Implementation]

State: ________________________ Interviewer: ______________

Respondent: ________________________ Date of Interview: ______________

Title: ________________________ Study ID No: ______________

Organization: ________________________

Address: ________________________

________________________

________________________

Phone: ________________________

Fax: ________________________

Email: ________________________

Thank you for taking the time for this interview. The U.S. Department of Agriculture’s Food and Nutrition Service has contracted with Altarum Institute to conduct a study of the [NAME OF INTERVENTION] that is offering information to children and their families about healthy foods to eat and the importance of being active. Altarum is a health and nutrition policy research and consulting institute and our work focuses on helping to improve the health and nutrition status of children, families, and adults. The purpose of the study is to evaluate several SNAP-Education models around the country and to provide recommendations for how these interventions could be improved to better serve the children and families in your community. We also will be evaluating how the intervention might be replicated in other communities.

We will be using first names only today. Everything you say is private. After we conduct several of these interviews, we will write a report for the U.S. Department of Agriculture’s Food and Nutrition Service. Your name will not appear anywhere in the report. Nothing said today will be attached to your name at any point. Nothing that you say will affect your job or be shared with your employers.

Today we will specifically be discussing your background and other qualifications as an educator for this education program, the planning process that has already begun with the intervention sites, and your expectations for the reach and design of the program. Once you have completed teaching one complete session of [NAME OF INTERVENTION], we will follow up with you for one more interview to find out how things may have changed from what you planned to do and to obtain your experiences and views on what worked well or not and why, and what you might change to improve the program.

I expect that our discussion today will take about 30 minutes. Before I begin, do you have any questions?

Educator’s Job Title, Qualifications and Capabilities

First I would like to ask you a few questions about your position and your background for this type of work.

1. What is your job title in this role as educator for [NAME OF INTERVENTION]?

67. Do you also provide nutrition education or community education for any other programs?

[IF YES]

a) Please tell me a little bit about your other related work?

[ASK THIS ONE QUESTION BELOW ONLY FOR EDUCATORS IN NY INTERVENTION]

68. For how many months or years have you been providing the Eat Well Play Hard in Child Care Settings classes?

69. Prior to this role as an educator for [NAME OF INTERVENTION] have you had any other job or volunteer experience in nutrition or health education for children and families?

[IF YES]

a) Please describe these job or volunteer experiences?

b) How many total years of experience in nutrition or health education for children and families did you have before you came to be an educator in [NAME OF INTERVENTION]?

70. What is the highest level of education you have completed to date?

1. [For those who have college or graduate school degrees] What subject was your major or degree in?

71. Outside of any formal education, have you had any specialized training or certification either in nutrition education or health education?

[IF YES]

a) Could you please describe this training for me?

72. What else from your life experience do you think makes you capable of being an effective educator for this SNAP Ed program? [NAME OF INTERVENTION]?

73. What are some of the challenges that you or others like you might face in being an effective educator for this intervention?

Recruitment and Implementation Plans

Next I would like to discuss what is being planned to recruit sites and participants for the intervention and how many sites, classes and students you plan to be working with.

74. Do you know yet how the sites will be recruited?

1. Who does the recruiting and how do they or did they reach out to enroll the sites?

2. Do you think this is an effective way to select the sites? Why or why not?

75. At how many sites do you plan to teach the [NAME OF INTERVENTION] classes?

76. How many classrooms or groups of children will you be working with at each of these sites?

1. How often will you be going out to these groups, and how long will each class or activity be?

2. How many children do you expect will be involved in each class?

[ASK NEXT 3 QUESTIONS ONLY FOR NY and NV INTERVENTIONS ONLY]

77. Will you be conducting parent/caregiver/family classes or events at the intervention sites?

[IF YES]

a) How many of these parent classes or events do you plan to hold at each site?

b) How often will they occur?

3. At what time of day and how long will each session be planned for?

78. How many adults do you expect will be attending each of the classes for parents/caregivers?

79. What efforts will be made to retain the parents in the classes so that they receive the entire intervention or attend as many classes as possible for them?

80. Aside from yourself as the nutrition educator, will there be anyone else involved teaching the [NAME OF INTERVENTION] curriculum at these sites?

[IF YES]

a) What are their roles?

81. What physical resources will you need at the sites to implement the intervention? (e.g. space, a/v equipment, computers)?

Perceived Facilitators and Challenges to Intervention Success

82. Based on what you know about the curriculum, materials and other aspects of the [NAME OF INTERVENTION], what aspects of the do you think will be most effective with the target audiences you are trying to reach?

83. Before we close, I would like to ask you whether you foresee any challenges in implementing the intervention as designed or planned?

[IF YES]

a) What are those potential challenges and how might they be overcome?

That ends my formal interview questions. Do you have any comments or recommendations that you would like to add?

Thank you very much for your time and input on this important project. My colleagues and I at Altarum will get be getting back in touch with you to schedule a follow-up interview after you finish teaching [NAME OF INTERVENTION]. I am looking forward to talking with you then.

Discussion Guide for On-Site Nutrition Educators

[Post-Implementation]

State: ________________________ Interviewer: ______________

Respondent: ________________________ Date of Interview: ______________

Title: ________________________ Study ID No: ______________

Organization: ________________________

Address: ________________________

________________________

________________________

Phone: ________________________

Fax: ________________________

Email: ________________________

Thank you for taking the time for this interview. As I told you during our first meeting, the U.S. Department of Agriculture’s Food and Nutrition Service has contracted with our Altarum Institute to conduct a study of the [NAME OF INTERVENTION] that is offering information to children and their families about healthy foods to eat and the importance of being active. Altarum is a health and nutrition policy research and consulting institute and our work focuses on helping to improve the health and nutrition status of children, families, and adults.

The purpose of the study is to describe how several SNAP-Education program models are being carried out across the country and evaluate their impact on nutrition behaviors. The study will also highlight recommendations for how to replicate and improve these SNAP-Education models –based on what we observe and learn from the program planners, from the people who are implementing these interventions—like yourselves—and from the intervention participants.

We will be using first names only today. Everything you say is private. After we conduct several of these interviews, we will write a report for the U.S. Department of Agriculture’s Food and Nutrition Service. Your name will not appear anywhere in the report. Nothing said today will be attached to your name at any point. Nothing that you say will affect your job or be shared with your employers.

Today we will talk first about the training and assistance you were provided, then about differences between what your planned implementation versus what actually happened. After we cover that information, I want to spend most of our today hearing what you think worked well and your suggestions for any revisions or improvements to [NAME OF INTERVENTION].

I expect that our discussion will take about 30 minutes today. Before I begin, do you have any questions?

Experience and Satisfaction with Training

Let’s start with your views on the he training you received from (NAME OF ADMINISTERING AGENCY) before you began implementing the [NAME OF INTERVENTION].

1. Please describe the format of initial training you received from [NAME OF INTERVENTION] staff to teach this curriculum. (e.g., was it in a large or small group or one-on-one, was observation of implementation involved, etc.)

84. How much training did you initially receive (number and length of sessions)?

85. What aspects of the training did you find most useful?

86. What additional information, tools, skills, or other training do you think should be provided in the initial training to help you or other educators like yourself be more effective in delivering [NAME OF INTERVENTION]?

87. Are there any other changes you would suggest to improve the content or format or other aspects of the educator trainings for [NAME OF INTERVENTION]?

Did you receive any ongoing training or assistance—in a structured or unstructured format— after your initial training for [NAME OF INTERVENTION]?
[IF YES]

1. What was the format?

2. What was the content?

3. How much of this assistance did you receive?

4. What was helpful about this follow-up training or assistance?

5. What other follow-up training or assistance could have helped your or other educators like you teach the nutrition education curriculum to this target population more effectively?

[IF NO]

6. What kind of follow-up training or technical assistance do you think could have helped you more effectively teach the [curriculum]?

Reach, Dosage and Intensity of Intervention- Actual Compared to Planned

88. How did the number of sites, groups, classes and activities you implemented with the children differ, if at all, from what you had planned?

89. How did the amount of time you spent in direct education with the children differ, if at all, from what you had planned?

[ASK NEXT TWO QUESTIONS FOR NY AND NV INTERVENTIONS ONLY]

90. How did the number of sites, groups, classes and activities you implemented with parents/caregivers/families differ, if at all, from what you had planned?

91. How did the amount of time you spent in direct education with the parents/caregivers/families differ, if at all, from what you had planned?

Recruitment and Retention [ASK THIS SECTION FOR NY AND NEVADA INTERVENTIONS ONLY.]

92. Who carried out the outreach/recruitment efforts to encourage adults to participate in intervention?

93. How effective do you think the recruitment efforts were and why?

94. What were barriers to recruiting parents to come to classes or events and how do you think that these barriers could be reduced?

95. What efforts were made to retain parents beyond their first class or activity night?

1. How effective do you think the retention efforts were and why?

96. What were barriers to retaining parents in the intervention that might be different from recruitment barriers and how do you think that these barriers could be reduced?

Differences between Actual and Planned Implementation

97. In addition to any changes in the number, size and length of your educational activities you mentioned earlier, were there other differences in how you implemented the [NAME OF INTERVENTION] compared to what you and the program planners had intended?

[IF YES]

a) In what ways was it implemented differently from what was planned?

2. Why did these changes from the original plan occur?

3. In what ways were the changes positive?

4. In what ways were the changes negative?

Lessons Learned for Improvement and Replicability

98. What do you think worked well about the direct education, in-classroom education format of [NAME OF INTERVENTION] for children and why do you think it worked well?

99. What could be improved about the direct education, in-classroom education format of [NAME OF INTERVENTION] for children and why would you suggest this change?

100. What do you think worked well about the nutrition education materials and lesson activities designed for the children and why do you think it worked well?

101. What could be improved about the nutrition education materials and lesson activities designed for the children and why would you suggest this change?

[ASK NEXT 4 QUESTIONS FOR NY AND NV INTERVENTIONS ONLY]

102. What do you think worked well about the direct education formats that are designed to engage parents or caregivers in [NAME OF INTERVENTION] and why do you think these worked well?

103. What could be improved about the direct education formats for parents or caregivers and why would you suggest this change?

104. What about the take-home nutrition education materials and lessons targeted to parents or caregivers worked well and why do think this worked well?

105. What could be improved about the take-home nutrition education materials and lessons targeted to parents or caregivers and why would you suggest this change?

106. [IF NOT ALREADY MENTIONED IN RESPONSE TO QUESTIONS 24-28] Do you think that the nutrition educational materials and lessons and other aspects of the [NAME OF INTERVENTION] are tailored to be culturally-appropriate to the racial and ethnic groups that are in the target audience?

[IF YES]

a) What features of the materials and lessons make them culturally-appropriate?

[IF NO]

b) What do you think specifically could be changed or tailored in the materials and/or lessons or class form to make them more culturally appropriate for the racial and ethnic groups that are in the target audience for [NAME OF INTERVENTION]?

107. In addition to what we have already talked about already, are there any other specific aspects of the [NAME OF INTERVENTION] that you think worked well?

108. Are there other particular aspects of the program do you think did not work well?

109. Do you have any other suggestions for ways that [NAME OF INTERVENTION] could be improved to be more effective in improving the nutrition behaviors of its target audiences?

That ends my formal interview questions. Do you have any comments or recommendations that you would like to add?

Thank you very much for your time and input on this important project.

Discussion Guide for School Principals or Childcare Center Directors

[Pre-Implementation]

State: ________________________ Interviewer: ______________

Respondent: ________________________ Date of Interview: ______________

Title: ________________________ Study ID No: ______________

Organization: ________________________

Address: ________________________

________________________

________________________

Phone: ________________________

Fax: ________________________

Email: ________________________

Thank you for taking the time to participate in this interview. The U.S. Department of Agriculture’s Food and Nutrition Service has contracted with Altarum Institute to conduct a study of the [NAME OF INTERVENTION] that is offering information to children and their families about healthy foods to eat and the importance of being active. Altarum is a health and nutrition policy research and consulting institute and our work focuses on helping to improve the health and nutrition status of children, families, and adults.

This study will provide information on how the [NAME OF INTERVENTION] works from the perspective of the people who planned the program, the program teachers, you and your staff and some of the parents whose children participated. We also will use what you tell us today to provide recommendations for how [NAME OF INTERVENTION] can be improved to better work with organizations like yours and the children and families you serve.

Any answers you provide for this study will be kept private and your name will not be identified with any answers you provide. The estimated amount of time required to complete this interview is 30 minutes. I want to thank you for taking the time today to speak with me.

Before I begin, do you have any questions?

Initial Engagement and Response to the Program

1. How did you find out about the [NAME OF INTERVENTION]?

110. Why did your child care center/school decide to participate in the [NAME OF INTERVENTION] at this time?

111. What do you see as the most important messages and goals of the [NAME OF INTERVENTION] for the children and families it is trying to reach?

112. How would you say that the messages and goals of [NAME OF INTERVENTION] will fit into other aspects of the curriculum you have for the children in the targeted classrooms?

113. What are your initial impressions of the educator who will be teaching the [NAME OF INTERVENTION] to the children (and their parents) at your center/school?

Implementation Plans

114. How many classrooms and children are planned to be involved in [NAME OF INTERVENTION] at your center/school over the next couple of months?

115. How will the [NAME OF INTERVENTION] take home materials be distributed to the parents of the children in these classrooms?

116. What mechanisms are in place to find out if the parents saw or used the materials?

[ASK NEXT 3 QUESTIONS FOR NY AND NV INTERVENTIONS ONLY]

117. When will classes for the parents and other caregivers be scheduled at your center?

118. Why was this time period selected?

119. What have or are you planning to do to invite and encourage parents or other caregivers to participate in these classes or other parent-focused activities of [NAME OF INTERVENTION]?

Implementation Challenges and Solutions

120. What do you see as the logistical challenges that your teachers or you as the principal/director may face in fitting [NAME OF INTERVENTION] into the daily schedule and activities that are already going on at the center/school for the children?

121. In addition to the in-classroom logistical issues we discussed early, do you anticipate any other challenges or issues that in implementing the [NAME OF INTERVENTION] as planned?

122. If any unanticipated challenges arise during the next couple of months while the intervention is going on, how do you think they can be addressed?

That ends my formal interview questions. Do you have any comments or recommendations that you would like to add?

Thank you very much for your time and input on this very important project. As I mentioned earlier, we will follow up with you after the intervention has been implemented to hear your experience and recommendations.

Discussion Guide for School Principals or Childcare Center Directors

[Post-Implementation]

State: ________________________ Interviewer: ______________

Respondent: ________________________ Date of Interview: ______________

Title: ________________________ Study ID No: ______________

Organization: ________________________

Address: ________________________

________________________

________________________

Phone: ________________________

Fax: ________________________

Email: ________________________

Thank you for taking the time for this interview. As I explained during our first meeting, the U.S. Department of Agriculture’s Food and Nutrition Service has contracted with Altarum to conduct a study of the (NAME OF INTERVENTION)that is offering information to children and their families about healthy foods to eat and importance of being active.

This study will provide information on how the [NAME OF INTERVENTION] works from the perspective of the people who planned the program, the program teachers, you and your staff and some of the parents whose children participated. We also will use what you tell us today to provide recommendations for how [NAME OF INTERVENTION] can be improved to better work with organizations like yours and the children and families you serve.

Again, everything you say is private. After we conduct several of these interviews, we will write a report for the U.S. Department of Agriculture’s Food and Nutrition Service. Your name will not appear anywhere in the report. Nothing said today will be attached to your name at any point.

Today I have just a few questions about how the [NAME OF INTERVENTION] was carried out at your (school or center), and your views on whether it was effective and how it could be improved.

The estimated amount of time required to complete this interview is 30 minutes. I want to thank you for taking the time today to speak with me.

Before I begin, do you have any questions?

1. First I would like to know how involved you have you been in overseeing the implementation of [NAME OF INTERVENTION]?

REQUIRED PROBES:

1. Have you observed any of the in-classroom activities for the children?

2. [For Chickasaw Nation only] Have you observed any of the garden activities?

3. [For Chickasaw Nation only] Have you observed the plays the children were involved in?

4. Have you read any of the [NAME OF INTERVENTION] materials that were sent home with children to their parents?

5. [For New York and Nevada only] Have you observed any of the classes tailored to engage parents or caregivers in the [NAME OF INTERVENTION]?

123. Now that the intervention is over what are your views on the educator who led the classes?

124. What would you say are the most useful aspects of [NAME OF INTERVENTION] overall for the age groups of children it is targeting?

125. How do you think the various strategies that were used by [NAME OF INTERVENTION] to encourage parent involvement (e.g. take home materials and/or classes/activities targeted to parents and caregivers) succeeded?

1. What worked well? Why?

2. What could be changed or improved to increase parent or other caregiver engagement in the program’s nutrition education components?

126. What challenges or issues did you face in implementing this program at you [school or center]?

1. How did you address these?

2. Did you need to communicate with the [NAME OF INTERVENTION] program staff to address any of these issues? If so what did you need to communicate to them about and how were those issues addressed?

127. What could be done to make [NAME OF INTERVENTION] more appealing to schools and child care centers like yours?

128. Do you have any other suggestions for ways that this educational program could be improved?

129. Thinking outside of the way [NAME OF INTERVENTION] works, do you have any suggestions for other ways that child care centers like yours can encourage preschool children to eat more fruits and vegetables at home and encourage their parents to serve more fruits and vegetables?

130. My final and very straightforward question for you today is: would you want the [NAME OF INTERVENTION] to come to your center/school next year?

[IF YES]

a) Why would you want this program back at your [center or school] again?

[IF NO]

b) Why not?

That ends my formal interview questions. Do you have any comments or recommendations that you would like to add?

Thank you very much for your time and input on this very important project. We have a small gift for the child care center classrooms to thank you for your time.

GROUP Discussion Guide for Parents/Caregivers

[Post-Implementation ONLY]

Date of discussion: ______________

Location: ________________________ Study ID No: ______________

Facilitator: ________________________ Note Taker: ______________

Number of participants: ____________

Start Time: ______________ End Time: ______________

Welcome! My name is _________, I am here with my co-worker _________. Thank you for taking the time for this group discussion. The U.S. Department of Agriculture’s Food and Nutrition Service has contracted with Altarum Institute to conduct a study of the [NAME OF INTERVENTION] that is offering information to children and their families about healthy foods to eat and the importance of being active. Altarum is a health and nutrition policy research consulting institute and our work focuses on helping improve the health and nutrition status of children, families, and adults.

This study will provide information on how the program in which your children participates works from the perspective of: the people who planned the program, the teachers, you and your child. The purpose of today’s group is to hear from you—about you and your child’s experiences and satisfaction with this program that recently took place at your child’s day care/school. We also will use what you tell us today provide recommendations for how [NAME OF INTERVENTION] can be improved to better serve the children and families in your community and those in other communities like yours.

We will be using first names only today. Everything you say is private. After we conduct several of these group discussions, we will write a report for the U.S. Department of Agriculture’s Food and Nutrition Service. Your name will not appear anywhere in the report. Nothing said today will be attached to your name at any point. Nothing that you say will affect the services you receive through any of the programs we talk about today.

Before we begin, I would like to review a few details about our discussion:

• First, your participation in today’s discussion is voluntary. You are free to leave at any time.

• There are no right or wrong answers. Remember that we don’t work for the child care center or with the educators, so please feel free to say whatever you think.

• Also, it is okay to have ideas or opinions that are different from each other. We want to hear everyone’s point of view.

It would be helpful to have only one person talking at a time. We are tape recording this session so that we don’t miss anything important. If two people talk at once, we can’t understand what anyone is saying. We may remind you of this during the group discussion.

• We would like everyone to participate. But, you each don’t have to answer every question. You don’t have to raise your hand either. If, however, some of you are shy or we really want to know what you think about a particular question, we may ask you what you think.

• We have a lot to talk about today. So, don’t be surprised if at some point we interrupt the discussion and move to another topic. But, don’t let us cut you off. If there is something important you want to say, let us know and you can add your thoughts before we change subjects.

• Finally, we just want to emphasize what we said earlier: we will be using first names only. Everything you say is private. What you say today will not be attached to your name at any point. Nothing that you say will affect the child care you receive at this site or any other services you receive from this or any other program.

The group will last no more than 2 hours. You will not get out any later than _______. We will not be taking a formal break, but if you need to leave for a restroom break, the bathrooms are _____________. And feel free to get snacks.

For this session, I will read a question and then listen to your responses. I also may ask follow up questions to get some more detail.

Let’s get started! I can’t wait to hear what you think of the [NAME OF INTERVENTION].

Do you have any questions before we begin?

Introductions/Icebreaker

Let’s go around the room for this one: Please introduce yourself, tell us how long you have been coming to this child care center with your child, and name one fun activity you like doing with your preschooler. [MODERATOR NOTE: it is helpful to go in order of how the group is sitting. This will allow the transcriptionist to label responses by person. Also for note taking you can then label person1, person2, person 3 etc- to be able to write comments]

Exposure and Accessibility of SNAP-Ed Intervention for Parents/Caregivers

Please raise your hand if you know that your child has been participating in a program at this school (or child care center) where they learn about what healthy foods and being active. [ASK FOLLOWING QUESTIONS FOR THOSE WHO RAISE HAND]

1. What did your children tell you about what they did in these classes or sessions?

PROBES: Food they tried? Activities they did? Games they played? What they learned?

131. Did you see any take home materials on food and physical activity recently provided for you by the [NAME OF INTERVENTION]? [MODERATOR SHOULD PROMPT RESPONSE BY SHOWING SOME SAMPLE TAKE-HOME MATERIALS USED IN THE INTERVENTION]

132. What were the most helpful aspects of these take home materials?

[ASK QUESTIONS ONLY FOR NY STATE AND LAS VEGAS]:

133. Did you hear about the parent classes/family activity nights that were offered at [NAME OF CHILD CARE CENTER]?

1. If yes, how did you hear about them?

134. Raise your hand if you went to at least one class?

1. If you raised your hand: what made you decide to go?

2. If you didn’t raise your hand: what were the reasons that you didn’t go? (e.g., barriers related to timing and location, other barriers related to accessibility, level or interest or perceived need)

135. Please think for a moment about what could be done to encourage more people like you to participate in these classes/family nights. I will hand out a pencil and paper if you want to write down your ideas before you answer out loud.

[AFTER ABOUT 2 MINUTES TAKE ANSWERS VIA ROUND ROBIN QUESTIONING]

136. If you went to any of these classes did you receive any handouts?

1. Which handouts were most helpful and why?

2. Which handouts were not helpful and why?

137. Do you think the educator who led the classes provided information in a way that was easy for the people in the class to understand?

138. Would you say that the educator who led the classes was a good teacher for you?

139. If yes, what made her a good teacher? If not, why not?

Satisfaction/Likes and Dislikes with Intervention

140. Tell me about the parts of the program overall—including the classes for your children, the take home materials, and any classes you may have participated in, that you liked the best and why you liked these parts?

141. Now, I would like to know what parts of the program you liked least and why?

142. What parts of the program do you think your child liked the best and why?

143. What parts of the program did your child like the least and why?

Perceptions of Goals and Relevancy of Intervention

We are interested in hearing more about what you thought about the purpose of the classes, whether they helped you and provided useful information to you.

144. What do you think the [NAME OF INTERVENTION] was trying to teach you and your child?

145. How useful was the information the program offered for parents like you with young children?

146. How well did the program suggestions and information fit with the ways that people of your racial or ethnic background live your life?

147. How well did the program suggestions and information fit with the challenges faced by people who do not have a lot of money?

Intervention Impacts

These next few questions are about how you think [NAME OF INTERVENTION] classes and materials may have helped you learn new information or other ways it may have changed things for you or your children.

148. What are the most important things that your child learned from this program?

149. What are the most important things that you learned from this program?

150. Now I would like to ask you a question that you probably need more time to think about: What are the most significant change or changes that have taken place in your household because of this program?

I am passing out pieces of paper again if you want to write down your response.

[AFTER ABOUT 2 MINUTES TAKE ANSWERS VIA ROUND ROBIN QUESTIONING]

OPTIONAL PROBES AS NEEDED:

• Changes in food parents serve to their children?

• Changes in the food children select?

• Changes in physical activity at home?

Factors Affecting Fruit and Vegetable Availability at Home and Ways of Addressing these Barriers

Now I would like to take a few moments to ask you about the difficulties that parents who live in your neighborhood might face in trying to buy, store, and prepare fruits and vegetables for your preschool child.

151. What makes it harder for you or other parents like you to buy and keep fruits and vegetables at home? (e.g., cost, access, and storage)

152. What makes it harder for you or other parents of young children like you to prepare and serve fruits and vegetables to your young children?

153. Did the information or take home materials provided to you by [NAME OF INTERVENTION] help you to address any of these difficulties or barriers?

1. For those of you that said “yes”, how was the information or materials helpful?

154. For those who said “no”, what could have been done to make the information or take home materials more helpful for parents?

Recommendations

155. Would you recommend this program to friends?

[IF YES]

a) Why?

[IF NO]

2. Why not?

156. If you could change anything about the classes or take home materials or other aspects of the (NAME OF INTERVENTION) program – what would it be?

157. Is there anything we haven’t asked that you would like to tell us about your experience with and opinions of the (NAME OF INTERVENTION) program?

158. Before we close, I would like you to help us by giving us your ideas for other ways that child care centers/schools could encourage children to eat more fruits and vegetables and encourage their parents to serve fruits and vegetables more often.

Thank you very much for participating in this discussion group today. We have learned a lot from your experiences and recommendations.

In appreciation of your time and trouble today, we have gift cards for each of you today. Before you leave, please take one of these and sign the form indicating you have received one of these cards.

Primary Data Collection Instruments for:

UNV and NYSDOH

Discussion Guide for Childcare Center Classroom Teachers [Post-Implementation]

Paper-Based Questionnaire for Childcare Center Classroom Teachers [Post-Implementation]

Discussion Guide for Childcare Center Classroom Teachers

[Post-Implementation]

State: ________________________ Interviewer: ______________

Respondent: ________________________ Date of Interview: ______________

Title: ________________________ Study ID No: ______________

Center name: ________________________

Address: ________________________

________________________

________________________

Phone: ________________________

Fax: ________________________

Email: ________________________

Thank you for taking the time for this interview. The U.S. Department of Agriculture’s Food and Nutrition Service has contracted with Altarum Institute to conduct a study of the [NAME OF INTERVENTION] that is offering information to children and their families about healthy foods to eat and the importance of being active. Altarum is a health and nutrition policy research and consulting institute and our work focuses on helping to improve the health and nutrition status of children, families, and adults. This study will include not only outcome evaluation information but also process information on how it is being implemented and how you are evaluating the intervention. All of this will be useful to both FNS and to other SNAP-Ed implementing agencies that are planning to evaluate their own SNAP-Ed interventions.

We will be using first names only today. Everything you say is private. After we conduct several of these interviews, we will write a report for the U.S. Department of Agriculture’s Food and Nutrition Service. Your name will not appear anywhere in the report. Nothing said today will be attached to your name at any point. Nothing that you say will affect your job or be shared with your employers.

The purpose of my interview today is primarily to ask you about your experiences with perceptions of the [NAME OF INTERVENTION] Program at your center. I will use what you tell us today to provide recommendations for how the [NAME OF INTERVENTION] Program could be improved to better serve the children and families in your community and those in other communities like yours.

I expect that our discussion today will take about 30 minutes. Before I begin, do you have any questions?

Background Information

1. First, I would like to confirm that you are a teacher who works in a preschool room at the child care center?

159. What is your current job title here at the center?

160. How long have you worked in this position at this center?

161. On a scale of 1 to 10, with 1 being totally unimportant and 10 being extremely important, how important would you say nutrition is for preschool children and their families?

1. Why do you think this?

Exposure and Satisfaction with Intervention Classes Targeted to Child Care Center Staff

Now I would like to ask you about your experience with the classes that [NAME OF INTERVENTION] held here for child care teachers and administrators. I understand that there were [FILL IN NUMBER] of classes/orientation sessions provided for staff at your center by the [NAME OF INTERVENTION] nutritionist.

162. Were you aware of these classes/orientation sessions?

[IF YES]

a) How many sessions did you attend?

163. [FOR RESPONDENTS WHO ANSWER ZERO OR LESS THAN THE MAXIMUM NUMBER OF CLASSES OFFERED] What could have made it easier for you to come to these classes? (e.g., scheduling issues, length of class, language barriers, etc.)

164. What were the most important things you learned from these classes? Why do you think they are the most important?

165. What were the least helpful or useful aspects of these classes? Why do you think this?

166. Did you incorporate any nutrition messages, sample activities or tools from the [NAME OF INTERVENTION] in the classroom?

[IF YES]

1. Please describe how you incorporated these in your classroom

167. How often would you estimate you used the new information you received from [NAME OF INTERVENTION] in your classroom? Would you say a couple of times, once every week, a few times a week, or more often?

168. What aspects of the [NAME OF INTERVENTION] content or design of the messages, sample activities or tools made it easier for you to incorporate these into your classroom activities? (e.g., ease of use in the classroom setting, cultural sensitivity, age appropriateness of the materials for the target audience)

169. What aspects prevented you from using these tools in your classroom? (e.g., lack of time, lack of money for supplies, lack of confidence)

Lessons Learned and Recommendations

Now, I would like to hear about any recommendations you have for the classes and other supports provided by [NAME OF INTERVENTION] and on any other aspects of program as you observed it being implemented at your center.

1. Classes Taught by RDs for Children

170. Have you stayed in the classroom and observed the activities that the [NAME OF INTERVENTION] RD/nutritionist conducts with the children?

[IF YES]

1. How many of the RD/nutritionist classes did you observe?

171. What was your role there? (e.g., to deal with emergencies, to assist, to meet mandated staffing levels in the classroom, to supervise the RD)

172. What do you think worked well about these in-classroom activities?

173. What do you think should be revised or improved about these in-classroom activities to be more appealing or interesting for the children in your classroom?

2. Classes for Parents/Family Members

174. Have you observed any of the classes that the [NAME OF INTERVENTION] conducts with parents or family members?

[IF YES]

1. How many of these classes did you observe?

175. What do you think worked well about these classes for parents and family members?

176. What do you think should be revised or improved about the classes for parents and family members?

3. Take-home Materials (informational materials and recipes)

177. Have you read any of the take-home materials sent home through children to their parents by the [NAME OF INTERVENTION] Program?

[IF YES]

1. Which ones have you reviewed? [In New York ask about both “Parent Pages” and “Recipes”; in Nevada ask about “Smart Snack Cards”, “Let’s Hunt for Healthy Foods” and the “Hungry Meter” take home materials]

178. What do you think were the most helpful aspects of these take-home materials?

179. What improvements could be made to these materials?

180. Do you have suggestions for the take-home materials provided by the program to better reinforce the program’s nutrition messages for children, families and other caregivers?

181. Do you have any other recommendations or suggestions for ways that the program could be improved?

182. Do you have any other suggestions for how child care centers like yours can encourage preschool children to eat more fruits and vegetables at home and encourage their parents to serve more fruits and vegetables?

That ends my formal interview questions. Do you have any comments or recommendations that you would like to add?

Thank you very much for your time and input on this very important project. We have a gift card to thank you for your time.

Paper-Based Questionnaire for Childcare Center Classroom Teachers

[Post-Implementation]

The U.S. Department of Agriculture’s Food and Nutrition Service wants to know about your experience with the [NAME OF INTERVENTION] Program. They have contracted with Altarum Institute to study how this program is being implemented at local child care centers. Please fill out the form below to provide your feedback and help improve this program for children and families in your community and those in other communities like yours.

This questionnaire is private. After we have received all of the completed questionnaires and conducted interviews with a number of sites, we will write a report for the U.S. Department of Agriculture’s Food and Nutrition Service. Your name will not appear anywhere in the report. Nothing that you write will be attached to your name at any point. None of your responses will affect your job or be shared with the child care center administrator where you work.

Instructions: Please fill out the form below. Answer each question honestly and thoughtfully. If you have any questions, please feel free to contact [INSERT CONTACT INFO]. When you have completed the form, please mail it to the address at the bottom.

Child Care Center Name: (optional)

Your Name: (optional)

Date:

Background Information

First, I would like to confirm that you are a teacher who works in a preschool room at a child care center that participates in the [NAME OF INTERVENTION]. If this is not correct, please indicate so here and return the form to us. Do NOT continue with the rest of the survey.

1. What is your current job title here at the center?

183. How long have you worked in this position at this center?

184. Please indicate below how important you would say nutrition is for preschool children and their families.

1— Not at all Important

2

3—Somewhat important

4

5—Moderately important

6

7— Quite important

8

9

10—Extremely important

Teacher Observations of the [NAME OF INTERVENTION] Classes

185. Have you stayed in the classroom and observed the activities that the [NAME OF INTERVENTION] RD/nutritionist conducts with the children?

[IF YES]

1. How many of the RD/nutritionist classes did you observe?

186. What was your role there?

187. What do you think worked well about these in-classroom activities?

188. What do you think should be revised or improved about these in-classroom activities to be more appealing or interesting for the children in your classroom?

189. Have you observed any of the classes that the [All-4 Kids or Eat Well Play Hard in Child Care Settings] conducts with parents or family members?

[IF YES]

1. How many of these classes did you observe?

190. What do you think worked well about these classes for parents and family members?

191. What do you think should be revised or improved about the classes for parents and family members?

192. Have you read any of the take-home materials sent home through children to their parents by the [NAME OF INTERVENTION] Program?

[IF YES]

1. Which ones have you reviewed?

193. What do you think were the most helpful aspects of these take-home materials?

194. What improvements would you suggest for the take home materials?

195. Do you have any other recommendations or suggestions for ways that the [NAME OF INTERVENTION] Program could be improved?

Thank you very much for your time and input on this very important project.

Please mail your private responses to:

Altarum Institute

1200 18^th Street N.W. Suite 700

Washington, DC 20036

Attn: SNAP-ED Evaluation Data Collection Manager

Primary Data Collection Instruments for:

PSU

Discussion Guide for Implementing Agency Program Administrator [Pre-Implementation]

Discussion Guide for Implementing Agency Program Administrator [Post-Implementation]

Telephone Discussion Guide for Recruiters [Post-Implementation]

Telephone Discussion Guide for Web-Developers [Post-Implementation]

Questionnaire for Pilot Participants [Post-Pilot]

Telephone Discussion Guide for Intervention Participants [Post-Implementation]

Discussion Guide for Implementing Agency Program Administrator [PSU only]

[Pre-Implementation]

State: ________________________ Interviewer: ______________

Respondent: ________________________ Date of Interview: ______________

Title: ________________________ Study ID No: ______________

Organization: ________________________

Address: ________________________

________________________

________________________

Phone: ________________________

Fax: ________________________

Email: ________________________

Thank you for taking the time for this interview. The U.S. Department of Agriculture’s Food and Nutrition Service has contracted with Altarum Institute to conduct a study of the [NAME OF INTERVENTION] that is offering information to children and their families about healthy foods to eat and the importance of being active. Altarum is a health and nutrition policy research and consulting institute and our work focuses on helping to improve the health and nutrition status of children, families, and adults. The purpose of the study is to evaluate several SNAP-Education models around the country and to provide recommendations for how these interventions could be improved to better serve the children and families in your community. We also will be evaluating how the intervention might be replicated in other communities.

We will be using first names only today. Everything you say is private. After we conduct several of these interviews, we will write a report for the U.S. Department of Agriculture’s Food and Nutrition Service. Your name will not appear anywhere in the report. Nothing said today will be attached to your name at any point. Nothing that you say will affect your job or be shared with your employers.

Today we will specifically be discussing the planning process and expectations for the intervention. Once it has been implemented, we will follow up with you to find out whether the intervention met your expectations and how it might be improved.

I expect that our discussion today will take about 30 minutes. Before I begin, do you have any questions?

Review of Abstraction Summary

Several weeks ago we reviewed your IA application (submitted to FNS), 2010 SNAP-Ed Plan, and nutrition education materials and sent you a synopsis of your project based on this information. To begin our discussion today we would like to review that summary with you and give you the opportunity to comment on and/or suggest revisions to the summary.

1. After reading the summary does any of the information reported appear to be incorrect or inaccurately describe your project in any way?

2. If so, what information is incorrect?

3. Is this information incorrect because your project has changed in some way since submitting your 2010 SNAP-Ed Plan or did we just misunderstand or misinterpret something?

Thank you for reviewing the project summary we created and providing this feedback. Now let’s briefly talk about the planning and design phase of your project.

196. What challenges, if any, have you faced during the design and planning phases of the About Eating program?

197. What factors do you feel have contributed most to a successful design and planning phase (prompts: using education messages that were already developed, good communication between contributors, knowledgeable staff, establishment of strong partnerships, etc.)?

198. What lessons have you learned during this key phase of program development? What would you do differently? What would you do the same?

Okay, now I would like to shift our focus to the upcoming implementation of your SNAP-Ed project.

199. Now that you are ready to transition from the planning and design phase of your project to the implementation phase, what challenges, if any, are you anticipating?

200. Do you feel that the environment in which the intervention will take place will be able to support the intended change in behavior, knowledge, and/or attitudes?

4. For example, do you have any sense of whether technical glitches or complications with the web-based application might influence the participants’ ability to become eating competent?

5. Are there any other nutrition education messages (that you are aware of) that participants might be exposed to during the intervention that would impact the outcome(s) of your study?

201. Did the program have any difficulty finding adequate staff for the recruitment of participants? If so, what were the challenges/problems in finding staff for recruitment of participants?

202. Will any quality control and monitoring take place during implementation? If so, please describe.

Okay, now I would like to shift our focus to specifics of the development of the About Eating Web site.

203. What type of formative research did you conduct on web applications and nutrition interventions prior to deciding to use this approach with the About Eating curriculum? Were other Web applications used as models for About Eating?

204. We understand that the initial web model for About Eating was designed and tested on students. How was the eating competency curriculum and its web-based application adapted to the low-income target audience?

6. Were there any aspects of “eating competence” content that did not fit well with either the low-income audience or the web applications? If so, what were they?

7. Were there aspects of the Web application that were difficult to construct to convey the learner-centered approach you desired? If so, what were they?

205. How many total iterations has the About Eating Web site been through and over what period of time?

That ends my formal interview questions. Do you have any comments or recommendations that you would like to add?

Thank you very much for your time and input on this very important project.

Discussion Guide for Implementing Agency Program Administrator [PSU only]

[Post-Implementation]

State: ________________________ Interviewer: ______________

Respondent: ________________________ Date of Interview: ______________

Title: ________________________ Study ID No: ______________

Organization: ________________________

Address: ________________________

________________________

________________________

Phone: ________________________

Fax: ________________________

Email: ________________________

Thank you for taking the time for this interview. As you know, the U.S. Department of Agriculture’s Food and Nutrition Service contracted with Altarum to conduct a study of the [NAME OF INTERVENTION]. The purpose of the study is to evaluate several SNAP-Education models around the country and to provide recommendations for how these interventions could be improved to better serve the children and families in your community. We also will be evaluating how the intervention might be replicated in other communities.

As mentioned during our last meeting, nothing said today will be attached to your, and nothing that you say will affect your job or be shared with your employers.

Today we will specifically be discussing how the implementation of the program differed from your expectations. We also will be discussing lessons learned and you feedback on how the program might be improved.

I expect that our discussion today will take 30 to 40 minutes. Before I begin, do you have any questions?

Formative Research and Program Design

I’d like to briefly discuss how, if at all, the implementation of your nutrition education intervention differed from what was originally planned. In order to guide this discussion, let’s think and talk about the following aspects of implementation:

1. Eating competency messages.

8. Were the eating competency messages modified at any point during implementation? If so, how and why?

9. Also, were the eating competency messages modified for the control group’s participation in the intervention? If so, how and why?

206. Target audience.

10. Did the target audience differ from what was originally planned? If so, how and why?

207. Method of delivery.

11. Were the methods of delivery modified during implementation for any reason? If so, how and why?

208. Dose.

12. Did the dose of nutrition education vary from what was originally planned, i.e. the number of modules, the length of each module, etc.?

209. Reach.

13. Were you able to implement the intervention with the originally proposed number of participants? Were there any factors that affected your ability to achieve the full, intended reach?

210. About Eating Web modules.

14. Were the Web modules modified at any point during implementation? If so, how and why?

211. Timeline.

15. To what extent were the original implementation timelines met?

What are the reasons for and implications of any departures from the original timelines?

Operational steps involved in program implementation

212. Did you find the level of staff both in terms of qualifications and total number of staff (and types of staff) adequate for optimally developing and delivering the About Eating intervention?

213. What changes, if any, were made to planned key staff involvement and what were the reasons for any such changes?

214. Who reviewed the Web site for the following: fidelity, usability (navigation, etc.), readability, content, and aesthetics? How helpful was this review?

215. Was there a protocol or other standard set of criteria for reviewing the Web site? If yes, what criteria were used? (prompts: fidelity, usability [navigation, etc.], readability, content, aesthetics, etc.)

216. Were any quality control and monitoring processes used to maximize the fidelity/quality of the intervention delivery?

217. How effective were the Web development staff in developing the intended eating competency messages?

16. Why do you think these staff were effective/ineffective?

17. What could they have done differently to improve their effectiveness?

218. How easy or difficult was it to work with the Web developers to design the current structure of the Web site? Had they worked on any similar types of applications prior to this?

219. Do you think the level of training for Web developers was sufficient?

220. Were planned recruitment (of participants) efforts modified during implementation? If so, how and for what reasons?

221. What recruitment methods did you find to be most effective/least effective?

222. In your opinion, how well was the program able to track participation?

223. Did previously identified partners remain engaged throughout the intervention? Were these partnerships successful? If no, why?

Resources devoted to intervention

224. What were the actual time commitments for key staff (FTEs) if different than planned? If they differed, why did they differ?

225. How closely did the actual program cost components reflect the budgeted costs?

226. What factors contributed to any differences between the budgeted and actual costs?

227. Were the necessary type and quantity of materials, technology, etc. available to carry out the implementation as planned? If not, what else was needed?

Lessons learned for improvement and replicability

228. What types of lessons learned came out of the Web site review and testing process that would help others that wish to replicate your approach?

18. Do you have any suggestions to make that would improve the Web site development and review process?

229. Overall, what factors were key to the success of this nutrition education program? What factors hindered or limited the success of this nutrition education program?

230. Looking back over the past [NUMBER] months, what lessons have you learned?

231. What would be most valuable for another State or implementing agency to know if they were considering using this model?

232. In your opinion, are there any aspects of this SNAP-Ed program that would make it difficult to implement at a larger scale?

Assessment of IA-led evaluation

233. What methods were used to conduct the evaluation, if different than originally planned? If different, why?

234. Were the evaluation tools modified for any reason since the intervention began? If so, how and why?

235. Did the planned staff conduct the evaluation? If not, why not and who ended up conducting the evaluation?

236. Did the actual costs of the evaluation vary from what was planned? If so, how and why?

That ends my formal interview questions. Do you have any comments or recommendations that you would like to add?

Thank you very much for your time and input on this very important project.

Telephone Discussion Guide for Recruiters [PSU only]

[Post-Implementation]

State: ________________________ Interviewer: ______________

Respondent: ________________________ Date of Interview: ______________

Title: ________________________ Study ID No: ______________

Organization: ________________________

Address: ________________________

________________________

________________________

Phone: ________________________

Fax: ________________________

Email: ________________________

Hello, my name is ___________________ and I am with Altarum Institute. Altarum is conducting an evaluation of About Eating Program for the U.S. Department of Agriculture’s Food & Nutrition Service.

The purpose of this study is to assess the degree to which the About Eating Program is reaching its program objectives, with emphasis also on the accessibility of a web-based nutrition intervention for SNAP-Ed recipients. An additional objective of the evaluation is to identify best practices for recruiting Program participants.

As part of this evaluation we are conducting key informant interviews. We are interviewing a sample of About Eating recruiting staff, and we will ask you about your experiences including the following:

• Training and instruction you received on the recruitment process

• Recruitment goals

• Recruitment process, including methods used

• Perceived effectiveness of recruitment methods

• How incentives were incorporated into the recruitment process

The data obtained from this evaluation will provide information to strengthen this Program and inform future decisions at the U.S. Department of Agriculture’s Food & Nutrition Service.

Any answers you provide for this study will be kept private and your name will not be identified with any answers you provide. The estimated amount of time required to complete this interview is 45 minutes. I want to thank you for taking the time today to speak with me.

Before I begin, do you have any questions?

Training

1. Did you receive training on how to recruit participants? (yes/no)

[IF YES}

19. How many hours?

20. In what format? (in person, Web-based, etc)

21. What did you learn?

237. Were there instructions to follow for recruitment efforts? (yes/no) [If yes, the protocol will be included in the secondary data.]

[IF YES]

22. Did you follow the instructions? (yes/no)

23. How did you feel about the instructions? Are there any changes that would strengthen the instructions?

238. What do you understand to be the goals of the recruitment process?

24. What did you learn in the training about the recruitment goals? What did the instructions say about the recruitment goals?

Do you feel that you were given enough information to understand the recruitment goals?

239. Do you have past experience recruiting participants?

Recruitment Process

Next I would like to talk to you about the recruitment process.

240. Please describe how you recruited participants. What did this process look like?

241. Where did you recruit participants? (e.g., Laundromat, WIC clinic, GED center, community center, career center, food stamp office, grocery store, discount stores, etc.)

242. How much time do you think you dedicated to the recruitment process?

243. How much effort do you feel that you dedicated to the recruitment process?

244. What were your methods for recruiting participants [SEE BELOW TABLE]?

	Posters/ fliers	One-on-one conversation with potential participants	Asked eligible women to help recruit	Asked administration at recruitment sites to recruit	Other:
Check all methods used for recruiting participants
Check the method most commonly used
Check the method that you believe was the most effective

Perceived Effectiveness of Recruiting Process

245. How well did your recruitment methods work?

Not at all

A little—not as well as expected

Average

Good—better than expected

Excellent—much better than expected

246. What were the barriers to recruitment (Check all that apply; Circle the greatest barrier)

• Potential participants did not seem interested

• Lack of administrative support in the settings in which I was trying to recruit

• Lack of support from the study staff

• Lack of resources (materials, finances)

• Not enough time in day

• Recruitment period too short

• Ads and flyers not catchy enough

• Didn’t feel adequately trained

• Other: ___________________

247. Did potential participants show more interest in some recruiting locations than in others? (yes/no)

25. If so, what were the more effective recruiting locations? What were the least effective recruiting locations? Why?

248. What other settings do you think would be useful for recruiting?

249. What do you think worked well about this recruitment process overall?

250. What do you think should be revised or improved for recruiting participants to a nutrition education intervention like this one? (e.g., to make it easier for recruiters to do their job more effectively? to reach more of the people you want to reach with this nutrition education intervention?)

Incentives

251. Did you recruit participants who would be getting an incentive? (incentive, no incentive, recruited both types of participants)

26. If you recruited individuals who would receive incentives for participating, how did you communicate about the incentive?

27. What were potential participants’ reactions to hearing about the incentive?

Possible Confounding factors:

I am going to read several statements now. For each statement, please tell me whether you strongly disagree, somewhat disagree, are neutral, somewhat agree or strongly agree.

252. Overall, I felt knowledgeable about the About Eating recruitment process.

Strongly disagree

Somewhat disagree

Neutral

Somewhat agree

Strongly agree

253. Overall, I think About Eating is a worthwhile program.

Strongly disagree

Somewhat disagree

Neutral

Somewhat agree

Strongly agree

254. I am confident in my ability to recruit participants for research studies, like About Eating.

Strongly disagree

Somewhat disagree

Neutral

Somewhat agree

Strongly agree

That ends my formal interview questions. Do you have any comments or recommendations that you would like to add?

Thank you very much for your time and input on this very important project.

Telephone Discussion Guide for Wed-developers [PSU only]

[Post-Implementation]

State: ________________________ Interviewer: ______________

Respondent: ________________________ Date of Interview: ______________

Title: ________________________ Study ID No: ______________

Organization: ________________________

Address: ________________________

________________________

________________________

Phone: ________________________

Fax: ________________________

Email: ________________________

Hello, my name is ___________________ and I am with Altarum Institute. Altarum is conducting an evaluation of About Eating Program for the U.S. Department of Agriculture’s Food & Nutrition Service.

The purpose of this evaluation is to assess the degree to which the About Eating Program is reaching its program objectives, with emphasis on the accessibility of a web-based nutrition intervention for SNAP-Ed recipients. An additional objective of the evaluation is to identify best practices for developing and delivering Web-based nutrition education.

As part of this evaluation we are conducting key informant interviews. We are interviewing all the About Eating Web development staff, and we will ask you about your experiences including the following:

• Development of the structure and navigation of the Web site

• Review of the usability of the Web site

• Testing and modification of the Web site

• Staffing devoted to this site’s development

• Cost of the Web site

The data obtained from this evaluation will provide information to strengthen this Program and inform future decisions at the U.S. Department of Agriculture’s Food & Nutrition Service.

Any answers you provide for this study will be kept private and your name will not be identified with any answers you provide. The estimated amount of time required to complete this interview is 30 minutes. I want to thank you for taking the time today to speak with me.

Before I begin, do you have any questions?

1. How did you develop the current structure of the Web site and navigation approach?

255. Had you worked on any similar types of applications prior to this?

256. We understand that the initial web model for About Eating was designed and tested on students. Was the web-based application adapted to the low-income target audience? If so, how?

257. Were there any aspects of the Eating Healthy curriculum that were difficult to program or convey in a web application? If so, what were these and how did you overcome these difficulties?

258. Who reviewed the Web site for the following: fidelity, usability (navigation, etc.), readability, content, and aesthetics?

259. Was there a protocol or other standard set of criteria for reviewing the Web site?

28. If yes, what criteria were used? (Probes: fidelity, usability (navigation, etc.), readability, content, aesthetics).

260. Did you test a prototype of the current Web site (for usability, navigation, readability, etc.) before deploying it?

[IF YES]

29. Please describe the testing process.

• Were any major modifications made and incorporated into the Web site?

• How many times were changes incorporated and the site retested?

• Is there a protocol for testing the Web site?

261. How many iterations has the About Eating Web site been through and over what period of time?

262. What types of lessons learned came out of the review and testing process that would help others that wish to replicate this approach? Do you have any suggestions to make that would improve the Web site development and review process?

Okay, now I would like to shift our focus to staffing and cost of the development, testing, and operation of the About Eating site.

263. Approximately how many total (cumulative) staff hours were devoted to the design and development of this version of the Web site?

264. Approximately, how many total (cumulative) staff hours were devoted to the creation of previous versions of the Web site?

265. How much did it cost to develop, test, and operationalize the Web site? What are the ongoing maintenance costs?

That ends my formal interview questions. Do you have any comments or recommendations that you would like to add?

Thank you very much for your time and input on this very important project.

Questionnaire for Pilot Participants [PSU only]

[Post-Pilot]

This study is being funded by the U.S. Department of Agriculture’s Food & Nutrition Service and conducted by RTI International and Altarum Institute to find out more about your experiences using the About Eating Program. Your responses and recommendations in this questionnaire will help improve this Program and other Web-based nutrition interventions for SNAP-Ed recipients. Your participation is completely voluntary, and you may skip any questions you do not want to answer. All of your answers will be kept private. The questionnaire will take about 10 minutes to complete. If you have any questions, please call (NAME TO BE ADDED) at Altarum Institute at (NUMBER TO BE ADDED).

Please answer the following questions for the About Eating Program (for all lessons combined).

1. How much do you agree or disagree with the following statements? (Select one answer for each statement.)

   	Strongly agree	Agree	Disagree	Strongly disagree	Not applicable
   It is easy for me to access the Internet.
   It was easy for me to move around the Web site.
   The instructions for each lesson were clear.
   I was able to jump to links of interest.
   The information provided on the web site was easy to read.
   The information provided on the Web site was easy to understand.
   The About Eating Program made me feel self-conscious.
   The About Eating Program was designed for someone like me.
   I thought the information provided on the Web site was interesting.
   I thought the information provided on the Web site was factual.
   I did NOT find the material in the lessons to be repetitive.
   I was able to print resources from the Web site (e.g. Food Shopping List).
   I prefer to learn about nutrition and eating online instead of in-person with a nutritionist.
   I prefer to go through the online lessons at a staggered pace instead of at my own pace.
   I would participate in a program like this again even if I did not receive a gift card.

2. What do you think about the amount of time it took to complete each lesson? (Select one answer only.)

1. Too long

2. Too short

3. Just right

3. Please rank the lessons in order of how well you liked them.

(1=best, 5=worst)

_____ Your Food Variety

_____ Enjoying Eating

_____ Hunger and Fullness

_____ Skills to Fuel Your Body

_____ Physical Activity

4. Please share any comments on the About Eating Web site.

Demographic Questions

5. How many people under 18 years of age live in your household? ____

6. Including yourself, how many people 18 years of age or older live in your household? ____

7. How do you usually access the Internet? (Select one answer only.)

1. Home

2. Work

3. Friend’s or neighbor’s house

4. Family member’s house

5. Library or community center

6. Other: (Please specify__________________________)

8. How often do you usually access the Internet? (Select one answer only.)

1. At least once per day

2. A few times per week

3. A few times per month

4. A few times per year

9. Which of the following categories best describes your age? (Select one answer only.)

1. 18 to 24

2. 25 to 34

3. 35 to 45

10. Are you Hispanic or Latino? (Select one answer only.)

1. Yes

2. No

11. What is your race? (Select all answers that apply.)

1. White

2. Black or African American

3. Asian

4. Native Hawaiian or other Pacific Islander

5. American Indian or Alaskan Native

6. Other (specify): ­­__________________

12. What is the highest level of schooling you have completed? (Select one answer only.)

1. Did not complete high school

2. High school graduate or GED

3. Some college or 2-year degree

4. College degree

266. What is your marital status? (Select one answer only.)

1. Married or living with a partner

2. Separate or divorced

3. Widowed

4. Never married

Thank you for your time and interest in helping us learn about your experiences with and recommendations for the Web-based About Eating Program.

Telephone Discussion Guide for Intervention Participants [PSU only]

[Post-Implementation]

Hello, my name is ___________________ and I work for Altarum Institute. Altarum is a health and nutrition policy research consulting institute and our work focuses on helping improve the health and nutrition status of children, families, and adults.

Thank you for taking the time to complete this interview. The U.S. Department of Agriculture’s Food and Nutrition Service has asked us to interview people about their experience with the About Eating program , that you were part of. I am calling today to hear from you—about what you thought about this program , how easy it was to use the website, and how useful you thought the information was. We will use what you tell us today to give suggestions for how the About Eating online program can be improved to help others in your community and people in other communities like yours.

Everything you say is private. After we conduct several of these interviews, we will write a report for the U.S. Department of Agriculture’s Food and Nutrition Service. Your name will not appear anywhere in the report. Nothing you say today will be attached to your name at any point. Nothing that you say will affect the services you receive through any of the programs we talk about today.

The estimated amount of time required to complete this interview is 30 minutes.

Before I begin, do you have any questions?

Access to the About Eating Web Site

The About Eating program uses the web/ internet to provide people with information. Sometimes computer or web/internet problems may get in the way of finding and using the information. Our first set of questions are about how easy or hard it was to use the About Eating Web site.

1. First, we would like to know if you had any trouble with a) finding the website, b) logging on, or c) trouble with the internet connection?

1a. If yes, what kind of trouble? How did you resolve the problem? (probe: switched computers, went to library or other computer site, tried different times of day, etc.)

267. Were there any connection problems with the About Eating web site while you were logged on? While you were using the website- did you have any problems?

2a. If yes, what were the problems

We also want to know if you logged into the About Eating program alone or if did anyone do the modules/lessons with you.

268. Did any family members or friends use the About Eating Web site with you?

3a. If yes, did they participate with you at the same time or at a different time?

3b. If yes, how did they participate?

3c. If yes, how was it working on the Web site with someone else?

269. Do you feel that you had enough time to finish the lessons on the About Eating Web site, or did you feel that you did not have enough time to complete the lessons?

4a. How much time did you expect you could spend on the Web ite?

4b. How much time did you actually spend?

270. Which of the four lessons (Your Food Variety, Enjoying Eating, Hunger and Fullness, Skills to Fuel Your Body) did you like the best? Why?

271. Which of the four lessons (Your Food Variety, Enjoying Eating, Hunger and Fullness, Skills to Fuel Your Body) did you like the least? Why?

272. How do you think you will use the information you learned from the Web site?

Questions about the About Eating Lessons

The next questions are about how you went through the lessons on the About Eating Web site. Think about all of the lessons together for the next questions. The lessons were: Your Food Variety, Enjoying Eating, Hunger and Fullness, Skills to fuel your body. I’ll ask you what you liked, what you didn’t like and how it was like to move around in the lessons.

273. Did you complete all or most of the lessons?

8a. If no, can you please tell me why?

([INTERVIEWER NOTE]If the respondent says ‘NO’ end the survey. If the respondent says ‘YES’ continue to question 9.)

274. Was the information provided in the lessons easy to understand?

9a. If not, what was difficult to understand?

275. What did you think of the graphs and charts used in the lessons?

(Probe: For example there was a chart that showed how Americans and French people responded to the following question: “Which would you prefer given the same cost? A luxury hotel with average food or a modest hotel with gourmet food?”)

10a. Did the chart help you understand the lesson better?

276. Tell us about your experience moving from lesson to lesson.

(Probe: Was it easy or hard, did you get lost or annoyed? Did the instructions help you move forward in the lesson? Did you find the links helpful?

11a) Tell us about moving from page to page

11b) Tell us about moving from section to section

277. Did you find any of the material in the lessons to be repetitive?

12a. If yes, can you give examples?

278. Do you feel these lessons were designed for someone like you?

13a.If yes, how so?

13b. If no, why not?

279. Did these lessons talk about foods and eating habits that were familiar to you? If not, why not?

280. If you could change anything about these lessons- what would that be?

281. Is there anything we haven’t asked about that you would like to comment on or tell us about your experience with and opinions about using the About Eating program?

Thank you for your time and interest in helping us learn about your experiences with and recommendations for the web-based About Eating Program.

0