Comments from NASS

APPENDIX G

COMMENTS FROM NASS REVIEWER

NASS Review of Revision to OMB 0584- XXXX

Supporting Justification for OMB Clearance for the

FNS - Community Eligibility Provision (CEP) Characteristics Study

May 2016

• The description in the research question and telephone interview scripts are very clear and follow Dillman’s principle. For example they:

  • Placed special instructions inside questions and not as freestanding entities.

  • Placed the instructions exactly where that information is needed and not at the beginning of the questionnaire.

  • Created clear navigational paths.

• The sampling frame for this survey is appropriate. They include three parts:

• All State Agencies (SAs) with at least one school eligible for CEP during school year (SY) 2016-17.

• Participating Local Education Agencies (LEA) are those with at least one school participating in CEP during school year (SY) 2016-17

• Non-participating LEAs are those with at least one school eligible for but none participating in CEP during school year (SY) 2016-17.

• The study design to generate equal participating group and control group sample sizes is ideal.

• The sampling method used for this survey design is appropriate. Probability Proportional Sampling methods provides equal or better precision than a simple random sample of the same size, and gains in precision accrue to all survey measures.

• The methods described to reduce non-response rates is thoughtful and data quality control is practicable. Sending inviting letters to LEA and CN directors, designating FNS regional staff to serve as regional study liaisons, and providing a toll free phone number and email address can all help reduce non-response and improve data quality.

• The data analysis method in this survey is appropriate.

• It is appropriate to employ the longitudinal study since data were collected in the different time period: SY2013 – 2014, 2014-2015, 2015-2016, and also will be collected in 2016 -2017. Data will also be analyzed across different schools or agencies at the same time point; thus, to conduct a cross section analysis is also proper.

• It is appropriate to use a difference-in-difference (DD) estimator for this study.

• The matched comparison group will be comprised of an equal number of eligible and nonparticipating LEAs, and Matching will be restricted of each treatment LEA to a single comparison LEA within the same State to improve the strength of the match. Thus, it may contain nonrandomized group design.

  • Nonrandomized group design is a quasi-experimental design that assigns identifiable groups of individuals to the intervention or comparison condition in some nonrandom way (Murray, 1998). Researchers often make study assignments based on characteristics of the groups for convenience (e.g., geographic location, such as within same state in this study) or other pragmatic reasons. It is very important to correct selection bias.

  • The DD estimator can control for a wide variety of selection mechanisms (e.g., selection decisions made by the director of an agency) if there is no observational distinction exists between the selection mechanism and the study condition of the group. The advantage of DD is that it relaxes the assumption of condition exogeneity or selection only on observed characteristics.

• In this study the differences in pre-program outcomes for both the treatment and a matched comparison group are compared to post-program outcomes. A popular approach for addressing this problem is the DD or double difference estimator, obtained by comparing the change in outcomes before and after the program for a treatment group to the change in outcomes over the same time period for a control group. Thus, to conduct a DD estimation for this study is appropriate.

Sample design, questionnaire design

• Consider adding questionnaire numbers for the State Agency Child Nutrition Director Web Survey (APPENDIX C1), so that CN directors will be able to follow the instructions. For example, the question directs the Director to question 1.8 which does not exist. This type of problem occurs in several sections of this questionnaire. (Questionnaire revised—Appendix C1)

• Consider changing “estimated proportion with an assumed population value” to “estimated proportion value” (last paragraph, page B-4). If this proportion is unknown, consider setting it to 0.50, as this produces a conservative estimate of variance. (Changed)

• For clarity, consider moving the purposeful sample explanation on Page B-6 to the purposeful sampling comment on page B-4. (Moved up) Also, consider listing the 12 SAs so the reader can determine if the selections are representative. (We do not know the States at this time.)

• Should 51 be 52 on page B-7? Or is the State that “will not involve sampling” omitted? (Changed to 52)

• The State CN Director Survey will be conducted as a census of all 51 State CN directors (including all States, the District of Columbia and Guam) and will not involve sampling.

• Consider using logistic regression for data analysis for research question RQ-5 and RQ-8, Table 2, in Appendix A. (This will be considered in analysis plan.)

• In this study the survey data from the participating and non-participating surveys will be weighted to reflect the probability of selection into the sample, but the Taylor approach does not fully capture the effects of different weighting adjustments (Sadeq R. Chowdhury. A Comparison of Taylor Linearization and Balanced Repeated Replication Methods for Variance Estimation in Medical Expenditure Panel Survey. Agency for Healthcare Research and Quality Working Paper No. 13004, July 2013, http://www.ahrq.gov). Thus, variance estimation results might be conservative for the first stage of sampling (i.e., the primary sampling units) when using the Taylor linearization approach. (We will also use replicate weights methods to estimate variances. Amended page B-7.)