SS-Part B 01 10 2012

Pre-Evaluation Assessments of Nutrition, Physical Activity and Obesity

Programs and Policies

Information Collection Request

New

Supporting Statement

Part B – Statistical Methods

October 27, 2011

Jan Jernigan, Ph.D.

Technical Monitor and Principal Investigator,

Division of Nutrition, Physical Activity and Obesity

Phone: 770.488.5224

E-mail: [email protected]

U.S. Department of Health and Human Services

Centers for Disease Control and Prevention

National Center for Chronic Disease Prevention and Health Promotion

Atlanta, GA

TABLE OF CONTENTS

B. Collections of Information Employing Statistical Methods

B.1 Respondent Universe and Sampling Methods

B.2 Procedures for the Collection of Information

B.3 Methods to Maximize Response Rates and Deal with Nonresponse

B.4 Tests of Procedures or Methods to be Undertaken

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

C. References

LIST OF ATTACHMENTS

1. Authorizing legislation, Sections 301 (a) and 317 (k) of the Public Health Service Act

2. FRN and Comments on FRN

   1. Federal Register Notice

   2. Summary of Public Comments and CDC Response

3. DNPAO “Recommended Community Strategies and Measurements to Prevent Obesity in the United States: Implementation and Measurement Guide”

4. Program Description Nomination Form

   1. Microsoft Word

   2. Web-based (screen shots from Zoomerang)

   3. Nomination Solicitation Email

5. Site Visit Forms:

   1. Site Visit Availability Calendar

   2. Suggested Interviewees Form

   3. Notification of Selection (email and letter)

   4. Site Visit FAQs

   5. Site Visit Schedule Instructions and Template

6. Lead Administrator Interview Guide

7. Evaluator Interview Guide

8. Program Staff Interview Guide

9. Public and Private Sector Partners and Other Stakeholders Interview Guide

10. Environmental Observation Form

11. Non-Disclosure agreement for data collection contractors

12. Oral Informed Consent Form

13. Expert Consultants Site Selection Guidance Document

Part B: Collections of Information Employing Statistical Methods

The proposed study will not utilize statistical methods for selection of study participants. As these are qualitative data, no statistical methods will be used in the analysis. This section is used to describe the data collection procedures.

B.1 Respondent Universe and Sampling Methods

Due to the exploratory nature of this project, a qualitative methodology has been selected. Data will be collected through (1) a paper or on-line nomination submission process for nutrition and physical activity programs and policies that will result in a program description, and (2) semi-structured interviews with key informants, and (3) environmental observations of the programs during site visits. The interviews will take place during 2.5-day site visits, approximately 15 sites in year one and eight sites in year two. Sites will have been implementing their program for varying lengths of time and may or may not receive funding from CDC. Criteria were established to allow for a variety of experiences in the obesity prevention programs’ implementation.

The pre-evaluation assessment expects to receive about 51 nominations from nutrition or physical activity programs. Nominated sites will be screened to determine whether they will be considered by the expert consultants as a candidate for the pre-evaluation assessments. Nominees first will be checked against our inclusion and exclusion criterion noted below. Each inclusion criterion response must be yes, and each exclusion criterion response must be no.

Inclusion criteria:

1. The obesity prevention program/intervention has been implemented for at least six months.

2. The program or intervention is suitable for implementation and replication in similar settings or populations.

Exclusion criterion:

1. The program or intervention is currently undergoing a rigorous evaluation or has in the past.

Once nominated programs are screened for eligibility, the program descriptions that are determined to be eligible will be reviewed by a group of paid expert consultants. The consultants are experts in the obesity prevention field. Some are paid external consultants to CDC, and others may be CDC federal employees. There will be 9 or fewer expert consultants paid to review nominations. The expert consultants will review program nomination descriptions using the selection criteria below. See Attachment 13, Expert Consultants Site Selection Guidance Document, for the complete guidance document that experts will use to make their selections.

Expert Consultants Selection Criteria:

1. Potential Impact—The initiative appears to have potential for impact on the social or physical environment pertinent to healthy eating and active living. Note: Assess based on your judgment the intervention’s conceptual logic and other pertinent characteristics such as intensity and duration. Interventions that are critical to building a whole environment that is conducive to physical activity (PA) should also be considered as these may result in average increases in PA.

2. Reach to Target Population—What percentage of the target population is “reached” or in some other way positively affected by the initiative? Note: Reach in this case should be defined in terms of the initiative achieving participation by the target population or of the target population being covered under the influence of the policy or environmental change.

3. Acceptability to Stakeholders—The initiative appears to be acceptable and even attractive to pertinent collaborators, gatekeepers, and other necessary groups, such as community groups, schools, workplaces, and government agencies?

4. Feasibility of Implementation—The likelihood that the initiative as designed can be or has been implemented fully. Note: For this question, consider the full implementation of the actual policy or environmental changes established by the AT planning group.

5. Feasibility of Adoption—The potential for other similar sites/entities to adopt the initiative—particularly in multiple states/regions.

6. Transportability/Generalizability—The degree to which the initiative has the potential to be adapted for other settings that differ in size, resources, and demographics.

7. Initiative Sustainability—The likelihood that the initiative can continue over time without special resources or extraordinary leadership.

8. Sustainability of Health Effect—Will the intended health effect of the initiative endure over time?

9. Staff/Organizational Capacity—Sponsoring organization and staff have the capacity to participate fully in a brief assessment, learn from it, and further develop the initiative. Note: For this question, consider staff’s reported capacity to organize interviews with relevant stakeholders and host a 2.5-day site visit, and learn from constructive assessments provided.

The expert consultants will meet to discuss the nominated sites and recommend their top sites to CDC. CDC’s DNPAO staff will make the final selection of sites to be visited based on the feedback provided by the expert panelists.

Interviews will be conducted with approximately 12 individuals at each selected site. Site visitors also will conduct environmental observations of program sites or other aspects evidencing their work during the site visits. Respondents will be identified by lead administrators at each of the sites, using the data tables and criteria set forth in the scheduling instructions, see Attachment 5e. Respondents will include:

• Lead administrators (1 per site)

• Evaluators (1 per site)

• Program staff (3 per site)

• Public and private sector partners/other stakeholders (7 per site)

B.2 Procedures for the Collection of Information

After their selection by the expert consultants and DNPAO, each site will be notified by mail and email (Attachment 5c) that they have been selected for participation in the pre-evaluation assessment site visits. This initial notification, via mail and email, will include: site visit FAQ sheet (Attachment 5d), calendar for noting preferred site visit dates (Attachment 5a), and a form for noting suggested interviewees (Attachment 5b). Sites selected for inclusion in the pre-evaluation assessment will be asked to help determine two or three possible dates for the site visit and set the site visit schedule.

Once the site visit dates have been determined, about six weeks in advance of the site visit, a list of interviewees will be confirmed with the site through the use of the site visit schedule instructions and template (Attachment 5e). Project staff will assist the data collection contractor, ICF Macro, by identifying and scheduling these interviews and a final site visit agenda returned to the ICF Macro site visit team lead. Any changes to the schedule or individuals selected for participation will be discussed with the site until a final schedule is agreed upon. Two weeks in advance of the site visit, the sites will provide any last minute information on travel logistics.

Teams of trained, experienced site visitors will conduct the site visits, including at least one senior staff member and one to two supporting staff. Site visitors will attend a one-day training where they will receive background information on the pre-evaluation assessment and training on specific content area by DNPAO staff. The training will consist of a thorough review of each interview guide to ensure that there is a shared understanding of the intent of the questions. Site visitors will also have time to practice with the guides and discuss and resolve any questions that might arise. A site visit manual will be developed for this training, including materials on the overall goals of the project, the intent of the pre-evaluation assessments, and copies of the interview guides.

Semi-structured individual interviews will be conducted with approximately 12 key informants at each selected site. To reduce burden and ensure that questions are tailored to each respondent type, customized interview guides have been created for Lead Administrator (Attachment 6); Evaluator (Attachment 7); Program Staff (Attachment 8); and Public and Private Sector Partners and Other Stakeholders (Attachment 9). In addition the site team will complete (Attachment 10) to document their environmental observations of the site. Since this form is completed by the site visitor team, it does not present a burden to respondents.

The lead administrator interview will take place first, before any other interviews are conducted, to set the context for the site visit. This interview is expected to last approximately two hours. Subsequent interviews will be conducted at a time convenient for the respondent. The remaining interviews are expected to last about one hour. In the event that any participants are unable to meet with a site visitor in-person, telephone interviews will be conducted when possible. We estimate conducting 12 interviews at each site.

B.3 Methods to Maximize Response Rates and Deal with Nonresponse

In order to maximize response rates, several strategies have been incorporated into the study design. The pre-evaluation assessment relies primarily on a self-nomination process, whereby programs nominate themselves for involvement in the study. Once nominees are notified of their selection, a letter will be sent to the lead administrator regarding their participation in the pre-evaluation assessment. Programs can voluntarily decide whether they would like to continue their participation in the pre-evaluation assessment or not. The site visits and interviews will be scheduled in conjunction with the program site. Additionally, the interviews will be conducted in participants’ offices, at easily accessible locations of their choosing, or via telephone so that participants do not have to travel long distances, eliminating concerns of transportation and reducing time burden. Appointments will be scheduled at a time most convenient to the respondent, and participation in the interviews is voluntary. Site visitors will follow-up with participants who are difficult to schedule or have to cancel to attempt to reschedule interviews on-site or via telephone shortly following the site visit.

B.4 Tests of Procedures or Methods to be Undertaken

Estimated burden hours are based on a prior study funded by RWJF, “Early Assessment of Programs and Policies to Prevent Childhood Obesity” ¹, which used a similar methodology. Existing instruments from the RWJF study were reviewed and in some cases adapted for use with the pre-evaluation assessments. A workgroup of DNPAO program staff and experts in the field have reviewed and provided feedback on the guides to minimize redundancy and ensure that the most useful questions will be asked. The workgroup carefully considered the content, appropriateness, and phrasing of the study questions so that they are brief, easy to use, and understandable.

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

No statistical sampling or estimation procedures are used in this data collection; therefore no individuals were consulted on the statistical aspects of the design.

Individuals Collecting and Analyzing the Data include:

Nicola Dawkins, PhD, MPH 3 Corporate Square, NE Suite 370 Atlanta, GA 30329 PH: 404-321-3211 Email: [email protected]	Michelle Revels, MPA 3 Corporate Square, NE Suite 370 Atlanta, GA 30329 PH: 404-321-3211 Email: [email protected]
Tracy Fox MPH, R.D. Food, Nutrition & Policy Consultants, LLC 1214 Constitution Ave. NE Washington, DC 20002 PH: 202-621-7697 Email: [email protected]	Lesley Cottrell, PhD 1369 Hunter Lane Morgantown, WV 26505 PH: 304-293-1149 Email: [email protected]
Marjorie A. Gutman, PhD Gutman Research Associates 42 North Main Street Cranbury, New Jersey 08512 PH: (609) 655-3524 Email: [email protected]