Wave II OMB_Supporting Statement Part B_final_012111_(2)

Table of Contents

B.1 Respondent Universe and Sampling Methods B1

B.2 Describe the procedures for the collection of information including: B6

B.3 Methods to Maximize Response Rates and Deal with Nonresponse B7

B.4 Tests of Procedures or Methods to Be Undertaken. B8

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data B9

Part B

Collections of Information Employing Statistical Methods

This section of the Supporting Statement addresses the five points outlined in Part B of the OMB guidelines and focuses on statistical methods related to the collection of information for the study. Because the process evaluation component of the study will primarily collect descriptive data from existing documentation, a purposeful selection of key informants, and a small group of participants, it will not require the use of statistical methods. For this reason, Part B of the Supporting Statement is for the impact evaluation component of the study only.

B.1—Describe (including a numerical estimate) the potential respondent universe and any sam­pling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corre­sponding sample are to be provided in tabular form for the uni­verse as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The impact evaluation for the three demonstration projects includes conducting pre- and post-surveys of the intervention group and a control or comparison group. Table A.1-1 in Part A provides a summary of the research design and data collection methods for each of the three demonstration projects. The respondent universe (study population) and sampling method for each of the three demonstration projects is described below. We also provide the expected overall response rates for each demonstration project. We anticipate that the overall response rates may be lower than 75 percent because of the multiple rounds of data collection (i.e., consent to provide parental contact information for the two school-based programs, pre-survey, and post-survey) and expected attrition between the pre- and post-surveys (e.g., participants move or change schools). As described in Section B.3, our analyses will use statistical models to correct for participant attrition.

For Wave I of the Models of SNAP-Ed and Evaluation study, data were collected from four demonstration projects. Three of the four projects were interventions implemented in elementary schools or child care centers. The evaluations included pre- and post-surveys of parents and caregivers similar to the proposed study. The anticipated and actual cooperation rate for each stage of data collection and the overall response rates for these three projects are shown in Table B.1-1. For the Chickasaw Nation Nutrition Services’ (CNNS) Eagle Adventure program, we nearly achieved the target response rate and completed the required number of surveys needed for our analysis based on the power calculations. For the University of Nevada’s (UNV) All 4 Kids Program, we exceeded the target response rate for the treatment group, but did not achieve the target response rate for the control group; however, we completed the required number of surveys needed for our analysis. For the New York State Department of Health’s (NYSDOH) Eat Well and Play Hard in Child Care Settings, we greatly exceeded the target response rate and required number of surveys for the treatment and control groups.

The CNNS study is most comparable to the INN and UKCES studies, since the target population is elementary school children. We anticipate that the overall response rates for these studies will be 42 percent, which is slightly lower than what we achieved for CNNS (46–49 percent). For the INN and UKCES studies, we are assuming a lower return rate for the contact card to consent for study participation compared to the CNNS study. For MSUE, the overall response rate is expected to be 46 percent, based on previous experience with this study population and similar survey procedures.

Table B.1-1. Response Rates for Wave 1 of the Models of SNAP-Ed and Evaluation Study

	Anticipated				Actual
	Return Rate for Contact Card (%)a	Cooperation Rate for Pre-survey (%)b	Cooperation Rate for Post-survey (%)c	Overall Response Rate (%)d	Return Rate for Contact Card (%)a	Cooperation Rate for Pre-survey (%)b	Cooperation Rate for Post-survey (%)c	Overall Response Rate (%)d
CNNS
Treatment	82	75	83	51	70	83	84	49
Comparison	82	75	83	51	63	85	85	46
UNV
Treatment	—e	83	80	66	—e	85	83	71
Control	—e	83	80	66	—e	54f	81	44
NYSDOH
Treatment	78	70	70	38	—g	76	80	61
Control	78	70	70	38	—g	74	78	58

^a Return rate for contact card = number of parents who returned card and consented to participate in study / eligible population.

^b Cooperation rate for pre-survey = number of parents who completed pre-survey / number of parents who returned contact card.

^c Cooperation rate for post-survey = number of parents who completed post-survey / number of parents who completed pre-survey.

^d Overall response rate = number of parents who completed post-survey / eligible population.

^e For the UNV study, the pre-survey was conducted in person, so it was not necessary to return the contact card.

^f The cooperation and overall response rate are lower than anticipated because study enrollment was open to the entire center, instead of specific classrooms, thus greatly increasing the size of the eligible population.

^g For the actual data collection for NYSDOH, parents returned the contact card consenting to participate in the study and the survey at the same time.

INN

The population of interest is third grade students attending eligible schools in four Iowa school districts (Council Bluffs, Waterloo, Des Moines, and Davenport). Data on this population will be gathered through surveys of children’s parents and/or guardians about dietary behavior in the family home. The evaluation of the INN BASICS and Pick a better snack™ nutrition education interventions is based on a quasi-experimental design. The study will include two active intervention conditions and one comparison condition, with 11 schools assigned to each condition. One of the active interventions includes the school-based BASICS curriculum (single channel); the other active intervention combines the school-based BASICS curriculum and the Pick a better snack™ social marketing campaign delivered in the community through grocery stores and supermarkets (multichannel). INN selected and purposively assigned school districts to treatment conditions: the Waterloo and Council Bluffs school districts will receive the single-channel intervention, the Des Moines will receive the multichannel intervention, and the Davenport school district will serve as the comparison condition.

Because of logistical considerations, the selection of schools for inclusion in the 2012 school year evaluation will not be possible until the end of the 2011 school year. Selection of schools for inclusion in the study will occur in two steps. In the first step, a list of schools that meet inclusion criteria will be generated in each district. Inclusion criteria will ensure that schools in the study meet FNS SNAP-Ed eligibility requirements and are large enough to meet the sampling needs. Second, the lists of available schools (i.e., the universe) and the available data on each will be reviewed to determine whether a matching or stratification approach is likely to be beneficial. Matching and stratification processes are employed to ensure that potential confounds are similarly distributed across study conditions. Table E-1 of Appendix E provides the sample design with information on response and attrition rates for each stage of data collection. The overall response rate is expected to be 42 percent.

UKCES

The population of interest is first through third grade students attending eligible schools in two Kentucky school districts. Ten schools in Laurel County and six schools in Perry County will be recruited by UKCES to participate in the study. Because of the sample size requirements detailed below, schools with fewer than 40 first- through third-grade students were removed from consideration prior to selection and randomization.

To control for potential differences between the two counties, schools were matched within county. One school from each pair was randomly selected to receive the intervention. Data provided by UKCES on school size (number of anticipated first- through third-grade students) and percentage of students receiving free and reduced-price meals (FARM) were used to create matched pairs. Appendix E describes the statistical methodology for sample selection. Tables E-2 and E-3 of Appendix E provides the assignment of schools to the treatment and control groups. Table E-4 provides the sample design with information on response and attrition rates for each stage of data collection. The overall response rate is expected to be 42 percent.

MSUE

The study population consists of older adults (aged 60 and older at the beginning of the intervention) who attend one of approximately 30 senior centers throughout the State of Michigan. For the purposes of this study, a senior center is defined as a facility that is open to the public and offers social services or support to seniors. Because of logistical considerations, the selection of centers for inclusion in the evaluation will not be possible until spring 2011. Selection of centers for inclusion in the study will occur in two steps. In the first step, a list of centers that meet inclusion criteria will be generated to ensure that centers in the study meet FNS SNAP-Ed eligibility requirements and are large enough to meet the sampling needs (i.e., serve at least 30 seniors). Centers that are housing or assisted living facilities or that provide two or more meals per day will be excluded from the sampling frame. Second, using a stratified assignment process, remaining centers will be randomly assigned from within strata to a study condition (treatment versus control). Appendix E describes the statistical methodology for stratification and sample selection. Table E-5 provides the sample design with information on response and attrition rates for each stage of data collection. The overall response rate is expected to be 46 percent.

B.2—Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

• Estimation procedure,

• Degree of accuracy needed for the pur­pose described in the justification,

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Appendix E describes the statistical methodology for stratification and sample selection, the estimation and analysis procedures, and the degree of accuracy needed for the purpose described in the justification for each of the three demonstration projects. Appendix F describes the data collection procedures for each of the three demonstration projects and Appendix G provides our assumptions for sample size estimation. Appendix H provides copies of advance letters, post cards, telephone scripts, and other materials used in contacting respondents for the impact evaluation and Appendix I provides similar materials used in contacting respondents for the process evaluation.

Unusual Problems Requiring Specialized Sampling Procedures

No specialized sampling procedures are involved.

Use of Periodic Data Collection Cycles to Reduce Burden

This is a one-time survey data collection effort.

B.3—Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reli­able” data that can be generalized to the universe studied.

The following methods will be employed to maximize the response rate for the study:

• Meet with school principals and teachers to establish buy-in for the study and enlist their cooperation (INN, UKCES).

• Closely coordinate the pre-survey data collection efforts with MSUE who will be the primary liaison with the senior centers.

• Deliver an informational packet to parents and caregivers explaining the purpose of the study, the importance of participation, and the contractor’s pledge of privacy (INN, UKCES).

• Design an instrument that minimizes respondent burden (short in length, written in easy-to-understand language, and using visual aides to facilitate understanding).

• Test the draft instrument using cognitive interviews to ensure respondents can properly understand the questions and response options are robust.

• Mail postcards to remind participants to complete the survey and/or to thank them for their participation.

• Contact nonrespondents and program dropouts by telephone to complete the survey.

• Provide a monetary incentive to participants who complete each survey.

• Provide survey materials in English and Spanish for INN, since the program materials are available in Spanish and 8–17 percent of the population is Hispanic in the targeted school districts. (Spanish translated version of the instruments and survey materials are provided in Appendix A and H, respectively).

When evaluating experimental and quasi-experimental designs, attrition can bias program effects, leading researchers to erroneous conclusions. Attrition refers to incomplete participation and occurs when individuals assigned to an experimental condition provide baseline information but fail to provide data at follow-up. To account for and correct for potential bias due to attrition, we will compare baseline values of key outcome variables, covariates, and demographic variables between study participants who remain in the study for all surveys and those who do not. Variables associated with attrition will be included as covariates in the main analysis to help remove any potential biases caused by differential attrition rates. Additionally, variables associated with attrition will be included in a propensity analysis that will allow us to assess the relationship between key predictor variables and failure to provide follow up data. This will help characterize study participants lost to follow-up, regardless of whether differences in attrition between the two experimental groups are observed. The propensity analysis is based on a logistic regression that quantifies the probability of noncompletion and provides a propensity weight that can be applied to statistical models to correct for participant attrition.

B.4—Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separate­ly or in combination with the main collection of information.

One-on-one cognitive interviews were used to evaluate whether the survey instruments were appropriate for the target audience in terms of comprehension and understanding and whether improvements to the instruments could be made. Nine interviews were conducted with parents to evaluate the instruments for the INN and UKCES studies and nine interviews were conducted with older adults to evaluate the instrument for the MSUE study. A “think-aloud” approach (Willis, 2004) was used to address cognitive dimensions and usability issues. Using this approach, the respondent completed the instrument, and then the interviewer asked the respondent specific questions about the survey questions and response options. Based on the cognitive interviews, revisions were made to the instruments, to clarify questions and response options and to delete questions that were redundant.

A Fry test (Fry, 1968) was conducted to assess the readability of the instruments. This test examines the proportion of syllables and sentence length and is a commonly used measure of reading level. Excluding the opening narrative, the INN draft instrument was between a fourth- and eighth-grade reading level, and the UKCES instrument was between a fifth- and seventh-grade reading level, and the MSUE instrument was between a third- and sixth-grade reading level.

Upon receiving OMB approval, cognitive interviews will be conducted with three Spanish-speaking individuals to test the Spanish-translated version of the instruments for the INN impact evaluation. Based on the interview findings, revisions will be made to improve understanding of the questions and response items for the Spanish-speaking population.

B.5—Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The contractors, Altarum Institute and RTI International, will collect the information and analyze the data on behalf of FNS. Altarum is leading the process evaluation, RTI is leading the impact evaluation, and both organizations will conduct the assessment of the demonstration-project led evaluations. Jonathan Blitstein, Ph.D., of RTI (919-541-7313) is the senior methodologist for the study. Sampling and statistical methodologies were reviewed by the National Agricultural Statistical Service (NASS).

References

Shadish, W. R., Cook, T.D., & Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin Company.

Fry, E. (1968). A readability formula that saves time. Journal of Reading, 11(7), 265–271.

Krueger, R. A., & Casey, M. A. (2009). Focus groups: A practical guide for applied research, 4^th ed. Thousand Oaks, CA: Sage Publications.

Willis, G. (2004). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage Publications.

B 1