Supporting Statement Part B_12.28.09

Supporting Statement: Part B

Evaluation of Childhood

Obesity Prevention and Control Initiative:

New York City Health Bucks Program

December 28, 2009

Submitted by

Co-Technical Monitors:

Jan Jernigan and Gayle Payne

Division of Nutrition, Physical Activity and Obesity

National Center for Chronic Disease Prevention and Health Promotion

Centers for Disease Prevention and Control

4770 Buford Highway, NE, MS K-03

Atlanta, GA

Telephone (770) 488-5224

Contents

B. Collection of Information Employing Statistical Methods 1

B-1. Respondent Universe and Sampling Methods 1

B-2. Procedures for the Collection of Information 2

B-3. Methods to Maximize Response Rates and Deal with Non-response 5

B-4. Tests of Procedures or Methods to be Undertaken 6

Qualitative Data Analysis 6

Quantitative Analysis 6

B-5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 8

List of Appendices

Appendix A: Authorizing Legislation

Appendix B-1: Federal Register Notice

Appendix B-2: Summary of Public Comments and CDC Response

Appendix C: Local Community Organization Survey Instrument

Appendix D-1: Farmers’ Market Manager Survey Instrument

Appendix D-2: Fact Sheet for Distribution to Farmers’ Market Managers

Appendix E-1: Farmers’ Market Vendor Survey Instrument

Appendix E-2: Fact Sheet for Distribution to Farmers’ Market Vendors

Appendix F-1: Focus Group Moderator Guide – Farmers’ Market Vendors

Appendix F-2: Focus Group Consent Form – Farmers’ Market Vendors

Appendix G-1: Farmers’ Market Consumer Survey Instrument

Appendix G-2: Fact Sheet for Distribution to Farmers’ Market Consumers

Appendix H-1: Focus Group Moderator Guide – Farmers’ Market Consumers

Appendix H-2: Focus Group Consent Form – Farmers’ Market Consumers

Appendix I: NYC Health Bucks Neighborhood Resident Survey Instrument

Appendix J: IRB Approval Letter

Appendix K: List of Individuals Consulted During Project Development

B. Collection of Information Employing Statistical Methods

This information collection is being conducted as part of a mixed methods public health evaluation of the NYC Health Bucks program. In the remainder of this section, we provide an overview of the study and a discussion of the statistical methods to be employed in collecting and analyzing data.

B-1. Respondent Universe and Sampling Methods

Seven separate data collection activities (five surveys and two sets of focus groups) will be conducted in support of the planned process and outcome evaluation. Statistical methods will be employed as part of data collection for only four of these seven activities. Statistical methods will not be used to draw the respondent samples of farmers’ group vendors or consumers for focus groups because focus groups are a qualitative data collection technique. Surveys of local community organizations will be distributed to all organizations participating in the NYC Health Bucks program; this survey thus represents a complete census of eligible respondents and does not therefore employ statistical methods.

However, we will use a stratified random sampling approach to select non-Health Bucks farmers’ markets for our surveys of farmers’ market managers, vendors, and consumers. We will also draw a random-digit-dial (RDD) telephone sample of NYC Health Bucks neighborhoods.

Stratified sample of farmers’ markets. Because the focus of the study is to perform a comprehensive process and outcome evaluation the NYC Health Bucks program, we will survey managers, vendors, and consumers at all farmers’ markets that accept Health Bucks. However, we also wish to make comparisons to farmers’ markets that do not accept Health Bucks, so that we may better understand differences in managers, vendors, and consumers across Health Bucks and non-Health Bucks markets. In order to draw an appropriate sample for comparison while simultaneously minimizing burden on groups of respondents who do not currently stand to directly benefit from the Health Bucks program, we have therefore elected to draw a random stratified sample of farmers’ markets that do not accept Health Bucks rather than performing a complete census of these markets.

In order to ensure adequate representation of relevant farmers’ market characteristics across non-Health Bucks markets, we will stratify our sample by presence of wireless EBT technology, ownership by Greenmarket (the largest operator of farmers’ markets in New York City), and location in low-income neighborhoods, for a total of eight individual strata. Information on these market characteristics is readily available from public sources. Markets will be randomly chosen from each of the eight strata to reflect as closely as possible the distribution of these characteristics across the universe of all 55 non-Health Bucks farmers’ markets. Sampling weights will then be constructed as the inverse of the probability of selection for each market.

There are approximately 105 markets currently in operation across the five boroughs of New York City. Approximately 50 of these markets currently accept Health Bucks. Of the remaining 55 markets, we will use stratified random sampling techniques to select 40 non-Health Bucks markets to participate in our study. Surveys will be distributed to all vendors and market managers at each of the 90 total markets. Additionally, an average of 30 consumers will be interviewed at each of the 50 Health Bucks markets, and an average of 20 consumers at each of the 40 selected non-Health Bucks markets, for a total final consumer sample of 2,300: 1,500 Health Bucks market consumers and 800 non-Health Bucks market consumers. Each market will be visited twice during the 2010 farmers’ market season, with approximately half of the consumer interviews for that market completed during each visit. Sampling weights will be constructed to appropriately weight for the lower sampling probabilities in non-Health Bucks markets.

Note that this approach to sampling will produce a representative sample of market visits, as opposed to a representative sample of market consumers. Consumers who visit the farmers’ market more frequently will have a higher probability of selection. Consumer respondents will therefore be asked to report how often they shop at farmers’ markets during the regular farmers’ market season. We will also collect this information from residents of Health Bucks neighborhoods responding to our RDD survey. Responses to this visit frequency question will allow us to construct a weighting variable to be used to approximate a sample of consumers. This weighting strategy will allow us to compare responses across Health Bucks markets and our RDD sample of Health Bucks neighborhood residents.

For a specified power of 0.8 and significance level of α=0.05, farmers’ market visit sample sizes are sufficient to detect a difference of means of approximately 0.123 standard deviation units across Health Bucks and non-Health Bucks market visits, or a difference of means of approximately 0.145 standard deviation units across two equally-sized subgroups of Health Bucks market visits. In our judgment, these sample sizes are adequate both for detecting differences between Health Bucks and non-Health Bucks visits, and for drilling down to detect differences between sufficiently large subgroups of Health Bucks market visits.

RDD sample of NYC Health Bucks neighborhoods. The RDD sample of residents of NYC Health Bucks neighborhoods will be drawn from telephone exchanges that are known to be included in the ZIP codes that define the three DPHO areas in which NYC Health Bucks operates (the South Bronx, North and Central Brooklyn, and East and Central Harlem). Respondents will then be screened for residency in one of the ZIP codes or one of the directly adjacent ZIP codes.

In geographically-targeted RDD sampling, there is a tradeoff between how much of the target area is covered by exchanges included in the sampling frame and how efficiently households within the area can be targeted. As coverage improves, efficiency declines. Telephone exchanges to be included in the sampling frame for each of the three target neighborhoods will be selected in order to achieve an optimal balance between these two competing considerations. In the event that the RDD sampling approach yields insufficient coverage in selected ZIP code areas, we will consider using an Address-Based Sampling approach as an alternative.

B-2. Procedures for the Collection of Information

In this section, we describe procedures for each planned data collection activity. Consent will be obtained from all study participants prior to data collection. Survey design experts will review all instruments to ensure proper flow, clarity, and overall validity, as well as appropriateness for the target survey populations. Spanish-language versions of the vendor, consumer, and neighborhood resident survey instruments will also be developed and tested. In NYC Health Bucks neighborhoods, over 98% of all Community Health Survey interviews were conducted in either English or Spanish in 2008; we do not therefore anticipate a need to translate instruments into additional languages.

Focus groups. We will conduct focus groups with farmers’ market vendors and consumers targeted by the NYC Health Bucks intervention. Each focus group will be comprised of a maximum of eight participants. This number is sufficiently large to allow the expression of a diversity of viewpoints, yet sufficiently small so that participants may respond to each others’ comments, discuss topics in-depth, and allow the moderator to explore emerging themes. Potential participants will be identified by program contacts or recruited on site at farmers’ markets or other community locations. As recommended by the literature, we will ensure to the greatest extent possible that focus group participants are unknown to one another prior to participating in the focus group discussion, to reduce the likelihood of distracting dynamics within the group (Patton, 2002).

Group discussion questions for each set of focus groups have been compiled into focus group moderator guides (see Appendixes F-1 and H-1). The guides contain the script for conducting the focus group sessions and will ensure uniformity across the focus groups and allow for the orderly collection of participant self-report data. Each trained focus group moderator will facilitate one focus group for each designated group of participants. Each will observe and take notes for each others’ focus group facilitation.

The commitment for the focus groups will include participating in a 90 - 120 minute focus group to be scheduled at a mutually convenient time for all participants. Focus groups will take place in New York City at a convenient location for participants. Participants will receive a $35 cash incentive to compensate them for their time and for transportation to and from the focus group facility. In our experience, this size incentive will minimize the labor and cost involved in recruiting participants, keep attrition low, and yet it is not so large as to be an inappropriate influence on voluntary participation.

Surveys. All five groups of respondents will complete surveys; however, the mode of survey administration varies from group to group in order to balance varying considerations of feasibility, cost, and respondent burden.

Local community organizations. The local community organization survey will be administered online in order to minimize respondent burden. E-mail addresses for NYC Health Bucks contacts at each organization will be provided to the prime contractor by the NYC DOHMH. At the end of the New York City farmers’ market season (November 2010), these e-mail addresses will be used to send an invitation to complete the survey, including a link to the secure website through which the survey will be administered. Up to four additional reminder e-mails will be sent to each contact in the event that the survey is not completed. The survey will take each respondent approximately ten minutes to complete. See Appendix I for a copy of the local community organization survey instrument.

Farmers’ market managers. The farmers’ market manager survey will be administered via postal mailing, with in-person follow-up at farmers’ markets with managers that do not respond to the initial mailing. Mailing addresses for farmers’ market managers will be provided to the prime contractor by the Farmers’ Market Federation of New York (FMFNY), the implementation contractor for the NYC Health Bucks initiative. The mailing will include a cover letter explaining the study and a business reply envelope. In-person follow-up efforts for those managers who do not initially respond will be restricted to times of day when managers are experiencing down time, as with the vendor surveys described above. Efforts will be made to limit open-ended questions in order to reduce burden on market managers. The survey will take each respondent approximately eight minutes to complete. See Appendix D-1 for a copy of the farmers’ market manager survey instrument.

Farmers’ market vendors. A two-page written survey will be distributed in person on-site to all farmers’ market vendors operating in selected New York City farmers’ markets during the 2010 farmers’ market season. The survey is expected to take approximately seven minutes to complete.

The survey will be administered in person (as opposed to self-administered paper-based or web-based surveys) to ensure that vendors who may not have regular access to the internet are able to participate. By surveying vendors at the farmers’ market, we will ensure that they do not need to take time outside of their work days to complete the survey. In order to minimize vendor burden, trained survey staff will approach vendors when they are experiencing “down time” and are not serving customers. Vendors who do not wish to complete the survey immediately will be provided with business reply envelopes in order to allow them to complete and return the survey at their later convenience. See Appendix E-1 for a copy of the farmers’ market vendor survey instrument.

Farmers’ market consumers. Farmer’s market consumers will be surveyed on site at farmers’ markets in order to assess perceived access to, knowledge and attitudes about, purchases, and consumption of fresh fruits and vegetables among New York City farmers’ market consumers (including both Health Bucks and non-Health Bucks users), as well as consumer awareness of and attitudes about farmers’ markets and the NYC Health Bucks program. Teams of trained interviewers will administer interviews at each New York City farmers’ market in operation during the 2010 season. Each market will be visited twice over the course of the sampling period, once early in the season and once later in the season.

Field staff will observe entry and exit points at each market and will select one or more exit points based on observed consumer flow at which to implement our intercept sample. Selected locations will be situated so as to maximize access and visibility while avoiding interference with the normal flow of market activity. At each selected exit point, an intercept sample of individuals leaving the market will be implemented. We will implement a systematic random sample (skip sample) of individuals exiting the market based on expected flow of individuals through market on that day. Only consumers making purchases at the farmers’ market will be eligible to participate in the survey. However, interviewers will avoid interrupting consumers while they are shopping so as not to disrupt their shopping experience.

A round-trip MetroCard ($4.50 face value) will be provided to individuals as an incentive for completing the survey. The consumer survey instrument is two pages in length, and each interview is expected to take seven minutes to complete. See Appendix G-1 for a copy of the farmers’ market consumer survey instrument.

Telephone survey of NYC Health Bucks neighborhood residents. This telephone survey will be our primary source of information on individuals residing in neighborhoods in which the NYC Health Bucks operates who do not regularly shop at farmers’ markets. Since most residents of these target neighborhoods are low-income, we anticipate that a substantial proportion of respondents will be SNAP participants who are eligible to participate in the EBT incentive component of the NYC Health Bucks program. Though administered by telephone, content of this survey will be similar to that of the farmers’ market consumer survey, intended to assess knowledge and attitudes about, purchases of, access to, and consumption of fresh fruits and vegetables in this population. See Appendix I for a copy of the neighborhood resident survey instrument.

The random-digit-dial telephone survey will allow us to target households in NYC Health Bucks areas by telephone exchange. Households will then be screened to ensure they are within one of the target ZIP codes for our study, i.e., ZIP codes in or adjacent to NYC Health Bucks neighborhoods. If the household is in one of the target ZIP code areas, the interviewer will ask to speak to the household member over the age of 18 who is primarily responsible for food shopping in the household. Informed consent will be obtained verbally from that individual prior to completion of the telephone interview. We anticipate that each interview will take approximately nine minutes to complete.

B-3. Methods to Maximize Response Rates and Deal with Non-response

This section describes methods used to maximize response rates and deal with non-response issues encountered as part of our survey data collection procedures. As a general matter, every effort will be made to minimize respondent burden in order to maximize overall survey response rates.

For the local community organization, farmers’ market manager, and farmers’ market vendor surveys in particular, we will work closely with the DOHMH and the FMFNY to encourage individual respondents to complete and return their surveys. Because the DOHMH and the FMFNY currently require completion of post-season surveys for these groups as a condition of participation in the NYC Health Bucks program, we anticipate that non-response issues will be minimal for these three types of respondents. In-person follow-up at farmers’ markets with managers, on-site distribution to vendors, and follow-up email reminders to local community organizations will further reduce potential non-response. For farmers’ market vendors and managers, as described above, on-site distribution during market “down times” will additionally encourage higher response rates, as will the distribution of business reply envelopes to individuals not wishing to complete questionnaires immediately. For the farmers’ market manager and vendor surveys, all analyses will additionally incorporate appropriately-constructed sampling weights to account for differing probabilities of selection by market.

For the farmers’ market consumer survey, we anticipate that response rates will vary considerably with external conditions, including weather and traffic flow at the market. Completion rates are expected to range from roughly 10 to 30%, depending on these conditions, with completion rates at most markets expected to be near the lower end of that range. In-person administration by trained interviewers experienced with this type of skip-sample data collection will encourage consumer participation in the survey. Attempts will be made to recruit consumer participants at high-traffic times of day in order to increase overall participation in the survey. However, as with any intercept sample of this type, selection bias may be an issue for this group of respondents. We will track refusals and non-completions as part of our interview protocol. As described above, all analyses of data from the farmers’ market consumer survey will additionally incorporate weights constructed to account for (1) differing probabilities of selection due to our random stratified sampling strategy for non-Health Bucks markets and (2) higher probabilities of selection for consumers shopping at farmers’ markets more frequently.

Expected response rates for our RDD survey of Health Bucks neighborhood residents are calculated based on AAPOR’s Response Rate 3. We estimate an overall response rate of between 20 and 25% for this landline survey. Our RDD survey will also necessarily exclude households without telephones, which is likely to be a small source of non-coverage bias (~2% of households), as well as cell-phone-only households, which is likely to be a greater issue (~20% of households). We will use data from the 2009 New York City Community Health Survey (CHS) to assess potential bias from these two sources: non-coverage bias due to the exclusion of no-telephone and cell-phone-only households from our RDD sample, and non-response bias due to refusals, terminations, and failures to connect. The CHS includes both a landline and a cell-phone-only component. Key outcome measures and demographic questions from the 2009 CHS will be included in our landline-RDD survey for comparison purposes. Comparisons with cell-phone-only households in the CHS will allow us to assess the degree of non-coverage bias. Comparisons with landline households in the CHS will allow us to assess the degree of non-response bias.

B-4. Tests of Procedures or Methods to be Undertaken

Experts in survey design will review all instruments for consistency and flow, and individuals familiar with NYC Health Bucks and these respondent groups will be asked to review the language and content of these instruments to ensure their appropriateness for these populations. Additionally, to the extent possible, survey instruments have been adapted from existing, previously-validated instruments.

Because the local community organization survey, the farmers’ market manager survey, and the farmers’ market vendor survey will be administered to all eligible respondents, formal pretesting will not be possible for these three survey instruments. However, the farmers’ market consumer point-of-purchase survey and the telephone survey of NYC Health Bucks residents will each be pretested with no more than nine farmers’ market consumers early in the 2010 farmers’ market season.

B-5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The persons who designed the data collection and who will analyze the data are:

Lauren Olsho

Abt Associates Inc.

55 Wheeler Street

Cambridge, MA 02138

P: (617) 520-2326

F: (617) 386-7571

[email protected]

Deborah Klein Walker

Abt Associates Inc.

55 Wheeler Street

Cambridge, MA 02138

P: (617) 349-2390

F: (617) 386-7664

[email protected]

Yvonne Abel

Abt Associates Inc.

55 Wheeler Street

Cambridge, MA 02138

P: (617) 349-2451

F: (617) 386-8300

[email protected]

Andrew Weiss

Senior Vice President

Abt SRBI Inc.

8403 Colesville Road #820

Silver Spring, MD 20910

P: (301) 608-3883

F: (301) 608-3888

[email protected]

References

Anderson, J.V., Bybee, D.I., Brown, R.M., McLean, D.F., Garcia, E.M., Breer, M.L., and Schillo, B.A. (2001). 5 A Day fruit and vegetable intervention improves consumption in a low income population. Journal of the American Dietetic Association, 101(2), 195-202.

Carlton-Tohill, B. (2007). Fruits and vegetables consumption and body weight management. Acta Horticulturae, 744, 39-46.

Centers for Disease Control and Prevention. Obesity: Halting the Epidemic by Making Health Easier (At A Glance 2009). Accessed online at: http://www.cdc.gov/nccdphp/publications/AAG/pdf/obesity_aaa.pdf.

Centers for Disease Control and Prevention. (1999). Framework for program evaluation in public health. MMWR 48 (No. RR-11).

Dauchet, L., Amouyel, P., Hercberg, S., and Dallongeville, J. (2006). Fruit and vegetable consumption and risk of coronary heart disease: A meta-analysis of cohort studies. Journal of Nutrition 136(10), 2588-2593.

Gordon-Larsen, P., Adair, L.S., and Popkin, B.M. (2003). The relationship of ethnicity, socioeconomic factors, and overweight in US adolescents. Obesity Research, 11(1), 121-9.

Guenther, P.M., Dodd, K.W., Reedy, J., and Krebs-Smith, S.M. (2006). Most Americans eat much less than recommended amounts of fruits and vegetables. Journal of the American Dietetic Association, 106(9), 1371-1379.

He, K., Hu, F.B., Colditz, G.A., Manson, J.E., Willett, W.C., and Liu S (2004). Changes in intake of fruits and vegetables in relation to risk of obesity and weight gain among middle-aged women. International Journal of Obesity, 28, 1569-1574.

Herman, D.R., Harrison, G.G., and Jenks, E. (2006). Choices made by low-income women provided with an economic supplement for fresh fruit and vegetable purchase. Journal of the American Dietetic Association, 106(5), 740-744.

Jebb, S.A. (2005). Dietary strategies for the prevention of obesity. Proceedings of the Nutrition Society, 64(2), 217-227.

Krebs-Smith, S.M., Cook, A., Subar, A.F., Cleveland, L., Friday, J., and Kahle, L.L. (1996). Fruit and vegetable intakes of children and adolescents in the United States. Archives of Pediatric and Adolescent Medicine, 150(1), 81–86.

Mirza, N.M., Kadow, K., Palmer, M., et al (2004). Prevalence of overweight among inner city Hispanic-American children and adolescents. Obesity Research, 12, 1298-1310.

Mokdad, A.H., Marks, J.S., Stroup, D.F., et al (2004). Actual causes of death in the United States, 2000. JAMA, 291, 1238–1245.

Ogden, C.L., Carroll, M.D., and Flegal, K.M. (2008). High body mass index for age among US children and adolescents, 2003-2006. Journal of the American Medical Association, 299(20), 2401-2405.

Ogden, C.L., Carroll, M.D., Curtin, L.R., et al (2006). Prevalence of overweight and obesity in the United States, 1999-2004. JAMA, 295, 1549-1555.

Patton, M.Q. (2002). Qualitative Research and Evaluation Methods. Thousand Oaks, CA: Sage Publications.

Riboli, E., and Norat, T. (2003). Epidemiologic evidence of the protective effect of fruit and vegetables on cancer risk. American Journal of Clinical Nutrition, 78(3 Suppl), 559S-569S.

Rolls, B.J., Ello-Martin, J.A., and Carlton-Tohill, B. (2004). What can intervention studies tell us about the relationship between fruit and vegetable consumption and weight management? Nutrition Reviews, 62, 1-17.

Steinmetz, K.A., and Potter, J.D. (1996). Vegetables, fruit, and cancer prevention: a review. Journal of the American Dietetic Association, 96, 1027-1039.

U.S. Department of Health and Human Services. Healthy People 2010. 2nd ed. With Understanding and Improving Health and Objectives for Improving Health. 2 vols. Washington, DC: U.S. Government Printing Office, 2000.

U.S. Government Accountability Office. (2008). Food Stamp Program: Options For Delivering Financial Incentives to Participants for Purchasing Targeted Foods (GAO-08-415). Washington, DC: author.

Wang, Y (2001). Cross-national comparison of childhood obesity: the epidemic and the relationship between obesity and socioeconomic status. International Journal of Epidemiology, 30, 1129-1136.

Wang,Y., and Zhang, Q. (2006). Are American children and adolescents of low socioeconomic status at increased risk of obesity? Changes in the association between overweight and family income between 1971 and 2002. American Journal of Clinical Nutrition, 84(4), 707-716.