0990-BodyWorks_ss_PartB_111711

Supporting Statement

for Paperwork Reduction Act Submissions

Section B

Multi-Component Evaluation of

the BodyWorks Program

Prepared for:

Office on Women’s Health

Prepared by:

Shattuck & Associates, Inc.

B) Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The respondent universe, sample and expected number of completed surveys are summarized in Table B1.

Table B1: Summary of Respondent Universe, Sample size and Number of Expected Completed Surveys
Survey	Universe	Sample	Expected Completed
Follow-up Study
Trainer Survey	1900*	1900	950
Parent Survey	2000*	2000	600
Trainer Telephone Interviews	10	10	10
Parent Focus Groups	18	18	18
Post Only Pilot Study
Participant Exit Survey	100	100	100
Trainer Telephone Interviews	10	10	10
Full Evaluation
Participant Pretests**	368	368	368
Participant Posttests**	368	368	265
Participant Follow-ups**	265	265	225
Comparison Pretests**	368	368	368
Comparison Posttests**	368	368	265
Comparison Follow-ups**	265	265	225
Participant Session Feedback Forms**	265	265	265
Trainer Feedback Forms	23	23	23

*Numbers reflect a query of the BodyWorks databases as of October 2011

**Includes parents and children

The respondent universe for the Trainer and Parent Follow-up Study will be the population who have participated in either a Train-the-Trainer program or Parent Program and have returned the BodyWorks Toolkit tags and therefore have been entered into the BodyWorks Database maintained by OWH. We expect a 50% response rate of the trainers and a 30% response rate of the parents. The trainers involved in the program, and therefore in the database, are very motivated to help provide feedback and make the project better. The program is a free resource for them and they typically are willing to provide feedback, especially as many are implementing BodyWorks as part of their workday. Given this motivation and buy in, it was felt that the response rate would be fairly high. Online surveys have been to have an average response rate of about 30-40% (University of Texas, 2011). Given the high buy in and motivation of the sample population, it was estimated that our response rates would be higher. The estimated response rate for parents is lower (30%) than for the trainers and is consistent with the general online response rates. The qualitative interviews with the trainers will be used to help explore the quantitative survey findings and help us to better understand the interpretations we draw from our analysis of the survey data. We will not use these interviews to draw any conclusions about trainers in general, but rather to provide some context and depth about the survey data.

For the Post Only Survey Pilot Study, ten sites will be recruited to pilot test the tool with program participants at the end of one of their BodyWorks programs. We expect approximately 10 program participants per site resulting in a respondent universe for the participant post only survey of 100 (10 sites x 10 participants).

For the full evaluation of BodyWorks, eight sites have been recruited to conduct a total of 23 series of BodyWorks sessions. Each series will have between 6 and 10 parent/caregivers and 6 to 10 adolescent participants. The final respondent universe of participants in the Full Evaluation of BodyWorks therefore will be between 138 (23x6) and 230 parents (23x10) and similarly, between 138 (23x6) and 230 (23x10) adolescents. Therefore, the total number of participants ranges from 276 to 460. We are estimating 368 total participants (adults and children). As participants will be asked to complete the pretest on entry in the program, we expect a 100% response rate for the pre test instrument. Based on the preliminary data collected on the original BodyWorks program, we expect a 72% response rate for the post test instrument as well as the Session Evaluation Forms. Based on the value of the group support that has been shown in the preliminary data collected on the BodyWorks program, and given the follow-up questionnaire will be collected at a 2-month post- program celebration, we expect 85% of those who complete the post test to complete the follow-up questionnaire. In addition, all BodyWorks trainers (n=23) are expected to complete the Facilitator Feedback Forms.

In the evaluation of the Best Bones Forever! BodyWorks program, medium to large effect sizes for changes in knowledge, attitudes, and behaviors were found. Based on this, a power analysis conducted with G-Power 3 software for repeated measures between-group differences using medium effect size, a conservative sample size of 225 adults or children, and alpha=.05 would yield a power to detect differences of .99.

To recruit evaluation sites, the Office on Women’s Health (OWH) released a nationwide request for proposals (RFP). The RFP was distributed broadly by the 10 HHS Regional Women’s Health Coordinators. Eligible applicants had to be located in the 50 states, the District of Columbia, the 6 U.S.-Affiliated Pacific Island Jurisdictions, Puerto Rico or the U.S. Virgin Islands. Eligible applicants could include: public and private non-profit organizations, community-based organizations, faith-based organizations, national organizations, colleges and universities, health care providing organizations, government agencies, tribal government agencies and tribal/urban Indian, Native American/Alaska Native organizations. Applicants had to have implemented at least two full programs of BodyWorks, in English or Spanish, within the last 12 months for parents/caregivers and their children ages 9 to 14. The distribution of this RFP used the same procedures as OWH uses for all of its regional RFP.

A panel that included the evaluation staff and program coordinators for Bodyworks reviewed the applications. We also sought guidance from the Regional Women’s Health Coordinators who often were familiar with the applicants. We received 10 applicants and selected 8. The sites are in Virginia, West Virginia, Texas (2 sites), Washington, Colorado, Nebraska, and California. They include a school system, university, nonprofits, and health care organizations. Each will receive up to $7,250 (average of $6,780) to complete 2-3 separate BodyWorks programs each within the current fiscal year, plus follow-up events, as well as facilitate the evaluation process. The funding covers program costs (such as trainers, facilities, and supplies for healthy cooking demonstrations) as well as costs associated with the evaluation (such as copying and mailing). Sites have different levels of support because some have costs covered by other sources. For example, some have room fees in their budget, whereas some have free access to space to hold the classes.

The comparison group will be recruited from this region (DC, VA, MD, PA, WV). In the last BodyWorks evaluation (2006 -- of the previous program), the sites selected a comparison, but that was unsuccessful.  We ended up with significantly heavier people in the experimental group.  We want to avoid site involvement in the comparison and think this broad area is diverse enough to find communities that are similar on the key demographics.  We have steps in place to ensure comparable groups. First, we will recruit from geographic areas with key demographic similarities to each evaluation site. In addition, comparison group participants will be recruited from existing organizations and groups in select areas where, ideally, motivation to change personal and family health habits is similar to BodyWorks participants. The second step taken to ensure that the comparison group is as comparable as possible to the experimental group will be to include items on the pretest for both the experimental and comparison group members that assess demographic information, prior health program experience, and motivation to change personal and family health habits. Inclusion of these items will allow for statistical controlling of the comparison group data as needed to ensure meaningful comparisons to the experimental group.

We know that that finding comparisons in each of the communities throughout the US is a stronger approach, but weighed this against the significantly higher costs. In addition, we have not selected sites with the capacity to recruit for the comparison group, as well as achieve the numbers of participants required for the evaluation. We are proposing the same method we used for the Best Bones Forever! BodyWorks evaluation funded under OMB control # 0990-0337), but understand this is should not be the model for future studies.

Full Evaluation of BodyWorks Questions Used to Ensure Comparison and Experimental Group Similarity
Demographic Information
Evaluation Form	Form Questions
BodyWorks Parent Pre Test	Q19-25, 27-48
BodyWorks Child Pre Test	Q21-25; Q27-28
Prior Program Exposure
Evaluation Form	Form Questions
BodyWorks Parent Pre Test	Q26
BodyWorks Child Pre Test	Q26
Motivation to Change
Evaluation Form	Form Questions
BodyWorks Parent Pre Test	Q3-6
BodyWorks Child Pre Test	Q3-6

The respondent universe will be all those who agree to participate in the comparison group. We anticipate a 100% response rate on the pre test questionnaire and a 72% response rate on the post test questionnaire. We expect 85% of those in the comparison group who complete the post test to complete the follow-up questionnaire.

2. Procedures for the Collection of Information

At the beginning of the Trainer and Parent Follow-up Study data collection period, an e-mail will be sent from OWH asking trainers and parents for their participation in the online surveys. The e-mail will contain a direct hyperlink to the parent or trainer follow-up survey. Parents will also be sent a postcard informing them of the survey. The postcard will contain the link to the survey as well as a toll-free number they can call if they prefer to have the survey mailed to them. Participants who prefer to complete a paper survey will also receive a postage-paid pre-addressed envelope in which to return the survey. This strategy will ensure that those who do not have Internet access can participate in the survey. Following the collection of this survey data, two groups of parents will be recruited to participate in a group discussion of their experiences with BodyWorks. In addition, phone interviews will be scheduled and conducted with recruited trainers. During both the focus groups and the interviews, extensive notes will be taken and the discussions will be digitally recorded.

Bodyworks trainers recruited to participate in the Post Only Survey Pilot Study will be informed and instructed on all study procedures during individual phone trainings. In addition, each participating trainer will receive written instructions to further support their data collection efforts. Participating trainers will proctor the pilot post only survey with BodyWorks participants after the completion of all sessions. Following the survey administration, phone interviews will be scheduled and conducted with recruited trainers. During the interviews, extensive notes will be taken and the discussions will be digitally recorded.

Trainers recruited to participate and support the Full Evaluation of Bodyworks will participate in training session that will detail all responsibilities and tasks. Participants in the Full Evaluation of BodyWorks will be informed during recruitment about the evaluation and their responsibilities will be outlined. In addition, each BodyWorks participant will receive a letter explaining the pilot program and evaluation components. Trainers will proctor the pre test with their respective groups of participants prior to the start of the first BodyWorks session and the post test after the completion of the sessions. Before the conclusion of the last session, trainers will also schedule a 2 month follow-up celebration. All participants will be invited back for this celebration and asked to complete the follow-up survey. Honest answers will be encouraged prior to follow-up survey completion. Trainers will be expected to contact any participants who don’t attend the follow-up celebration and obtain completed follow-up surveys. Trainers will also hand out and collect participant feedback forms at the end of each session. Likewise, facilitators themselves will complete a Facilitator Feedback Form after each session.

3. Methods to Maximize Response Rates and Deal with Nonresponse

Trainers and participants of the Post Only Survey Pilot Study will be informed during recruitment about the evaluation and responsibilities of all will be outlined. In addition, each BodyWorks participant will receive a letter from a senior OWH official explaining the evaluation responsibilities. Providing such a letter prior to data collection to inform respondents about the evaluation can lead to improved response rates (Graham, 2006). The post only survey will be given out during the final BodyWorks session. By asking the respondents to complete the questionnaires and return them prior to leaving the last session, we anticipate a higher response rate.

Similarly, trainers and participants of the Full Evaluation of BodyWorks will be informed during recruitment about the evaluation and the responsibilities of all will be outlined. In addition, each of these BodyWorks participants will also receive a letter from a senior OWH official explaining evaluation responsibilities. The BodyWorks pre test, post test and session evaluation forms will all be given out during appropriate BodyWorks sessions. By asking the respondents to complete the questionnaires and return them prior to leaving the session, we anticipate a higher response rate. Including the follow-up survey data collection as part of a larger follow-up celebration can also lead to a higher response rate. In addition, a small acknowledgement of time and trouble in the form of a monetary incentive ($25 cash) will be given to parent/caregiver and adolescent girl participants when they complete the pre, post and follow-up questionnaires in an effort to maximize response rates.

Efforts to maximize response rates for comparison group participants will include a letter from a senior OWH official explaining the comparison group responsibilities, the use of a monetary incentive ($60 cash distributed as follows: $25 after the pretest, $15 after the posttest, and $20 after the follow-up survey) and providing postcards that comparison group respondents can use to notify OWH of address changes.

4. Tests of Procedures or Methods to be Undertaken

Cognitive testing will be conducted with trainers, parents/caregivers as well as adolescents (n<10 ) on each data collection instrument. Feedback from this cognitive testing will be incorporated into the survey instruments.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data.

Shattuck and Associates

1313 Scotch Heather Avenue

Mt. Airy, MD 21771

866-829-1777

Works Cited

Graham, JD (2006). Questions and Answers When Designing Surveys For Information Collections. Washington, DC: Office of Information and Regulatory Affairs Office of Management and Budget.

University of Texas. (2011). Response Rates. Retrieved from http://www.utexas.edu/academic/ctl/assessment/iar/teaching/gather/method/survey-Response.php

7