Appendix F Expert Review Panel Comments and Responses

Appendix F. Expert Review Panel Comments and Responses

Date:

November 26, 2018

To:

Alexander Bush

From:

Thea Zimmerman

Subject:

Summary of External Reviewer Comments on Draft Study Plan for the WIC
Nutrition Risk Study

Three researchers agreed to serve as external reviewers for the FNS Process and Use of the WIC
Nutrition Risk Assessment Study (referred to as the WIC Nutrition Risk Study, or NRStudy):
 Angela Odoms-Young, Associate Professor, Kinesiology and Nutrition, University of
Illinois at Chicago
 Sarah Jane Schwarzenberg, Professor, Department of Pediatrics, University of Minnesota
Medical School
 Joni Geppert, Senior Nutrition Epidemiologist, WIC Director’s Office, Minnesota
Department of Health
This memo summarizes the feedback Westat received on the Draft Study Plan and provides
Westat response to the comments. As part of their review, Westat asked that the reviewers
consider three questions.
QUESTION: Do you have suggestions for ways to reduce burden on participants or make
the data collection efforts more efficient, while still addressing USDA’s goals?
Comment: One reviewer asked if we had considered focus groups.
Westat response: We did consider focus groups, but rejected that as likely to inhibit open
discussion of the nutrition risk assessment process by staff members, and also likely to
increase burden on the WIC site due to the need to take multiple staff members away
from their duties at the same time.
Comment: Reviewers asked that we consider using technology to collect interview data
with participants. One reviewer suggested that client satisfaction surveys can be conducted
via email and text, as over 90% of WIC participants have smart phones, and several states
have WIC apps that can send short questionnaires.
Westat response:
 Westat plans to offer WIC participants the option of completing the semiquantitative interviews via telephone at a time convenient to them.
 Because the in-depth interviews planned for the WIC site staff and the semiquantitative interviews with WIC participants include several open-ended

-2questions, a session with an interviewer is actually less burdensome than asking
respondents to type in their response, and more likely to elicit complete, detailed
responses.
Comment: One reviewer felt there was no need to survey all local agencies within the 10
selected states. A geographic sample would suffice. The reviewer suggested keeping the
LA survey to just the 30 LAs selected for key characteristics, where one of the
characteristics was geography.
Westat response:
 The census survey is essential to obtain the LA characteristics necessary for
selecting 30 LAs to represent the needed diversity.
 Quantitative study analyses will also utilize data from the broader group of LAs to
explore a fuller range of nutrition risk assessment procedure and policy
enhancements.
QUESTION: Do you foresee any difficulties/challenges with our plans for identifying a
sample of State Agencies or WIC clinics and encouraging their participation?
Comment: One reviewer asked how many SAs actually meet the criteria for the EBT system
that Westat proposed in the Study Plan.
Westat response: As of October 12, 2018, the WIC EBT Activities web page
(https://www.fns.usda.gov/wic/wic-ebt-activities) listed the following:
 45 SAs already have statewide implementation of their EBT system
 22 SAs are implementing their systems; 14 plan to roll out their systems before
October, 2019, which would make them eligible for recruitment in April 2020
(meeting the criteria to have EBT system in place at least six months prior to
recruitment).
 The remaining 44 are still in the planning phase.
If only the states currently planning to have their EBT systems in place before
October 2019 are eligible, there will be 59 SAs eligible. Of these, 51 are within the
continental US. This number of eligible SAs may be lower if any of the SAs with fullyimplemented EBT systems transition to a new EBT system within the six months
prior to April 2020.
Comment: One reviewer suggested that a pre-incentive for the LA may be very effective,
and has been used for interventions at other community-based sites.
Westat response: Westat is willing to propose pre-incentives, as we know they can be
very effective, but will need to work with FNS to ensure that the OMB will accept these
plans.
Comment: One reviewer suggested that LAs may be more willing to participate if we
assure them that they will receive a report.
Westat response: This is a part of our Study Plan.
QUESTION: Do you foresee any difficulties/challenges with any of our data collection
methods? How might these difficulties be overcome?
Comment: One reviewer suggested providing child care during the WIC participant
interview, as the client may have young children with them.

-3Westat response: If a WIC participant is unable to complete the interview onsite, Westat
will offer the option of completing it by telephone at the participant’s preferred time.
Comment: Two reviewers had concerns about the need for interpreters to help interview
diverse populations of WIC participants.
Westat response:
 Westat will have one bilingual (Spanish) site visit team, and will determine the
need for translation in conversations with the Site Director.
 We acknowledge that we may encounter the need for translation for additional
languages. We assume that either the WIC site will have a translator onsite for
the assessment interview or that the WIC participant will have a bilingual
companion. In either case, we will attempt to complete the interview with the
WIC participant onsite with the help of the WIC site translator or the bilingual
companion. If that is not possible, we will ask the WIC participant if they are
able to complete the interview via telephone with an interpreter assisting them.
We have added this option to the Draft Study Plan.
Comment: One reviewer praised the pretesting plan, as it should ensure that the site visit
team learns and uses the WIC language and terminology correctly.
Comment: One reviewer expressed concern about the time lapse between the clinic visit
and a telephone interview.
Westat response: Westat acknowledges that an immediate post-assessment interview is
ideal, but it is impractical to think that all WIC participants will be able to stay onsite for
that. Interviewing only those WIC participants who can do an onsite interview introduces
a bias of its own, as well. Our procedures to consent the WIC participant and schedule the
telephone interview prior to their leaving the WIC site will increase the likelihood that we
will be able to complete the interview.
Additional comment included:
Comment: Edit language that implies that all WIC participants will be women, as they
may be fathers, grandparents, etc.
Westat response: We have reviewed and edited the Study Plan accordingly.