Appendix AA. NASS comments and responses

Appendix AA

NASS comments and responses

WIC & FMNP Outreach, Innovation, and Modernization Evaluation

OMB Control Number 0584-[NEW]

Reviewed by Lindsay Drunasky

Mathematical Statistician

Summary, Estimation, and Disclosure Methodology Branch

USDA National Agricultural Statistics Service

202-690-8141

February 13, 2025

The study aims to assess the effectiveness of modernization efforts funded under the ARPA of 2021.

Three Areas of the Study

1. Implementation Study: Evaluates how WIC modernization efforts are being implemented across state agencies

2. Waiver Study: Examines how waivers granted under ARPA have been used by WIC state agencies

3. Impact Study: Measures the effect of modernization projects on WIC participation, retention, benefit redemption, and satisfaction.

Sampling Strategy

1. Implementation Study:

• Purposive selection method for WIC State Agency Staff interview, WIC local agency staff case study interview, WIC & FMNP vendor/outlet staff case study interview.

• Convenience selection method for WIC participant case study focus group in English or Spanish.

2. Waiver Study: rely on implementation study data

3. Impact Study:

• Random selection for WIC & FMNP vendor/ outlet staff experience survey, and WIC participant experience survey in English or Spanish.

The implementation and waiver studies rely on case study interview with 32 local agencies out of 2,000 and 10,000 clinics nationwide. Findings from these interviews are used to draw broader conclusions about WIC modernization, despite the small and non-random sample. Additional justification of the sample selection method is suggested.

RESPONSE: The participants in the case study interviews will be purposively selected to provide the most relevant information on priority topics for understanding recent modernization efforts and inform improvements of future modernization efforts. The case studies are intended to provide specific information that contextualizes specific modernization activities; their findings are not meant to be broadly generalizable.

WIC participants are selected using random sampling, but focus groups rely on convenience sampling. Convenience sampling for focus group (580 participants from 7 million) can introduce several types of research bias, such as selection bias and sampling bias. Also, findings may be skewed toward easily accessible participants. Consideration of other sampling methods are suggested to avoid bias.

RESPONSE: The case studies will be focused on specific modernization efforts; thus, convenience sampling will be conducted among WIC participants with experiences most relevant to a site’s case study focus. The focus groups are not meant to provide generalizable information; rather, they will provide rich contextual information on the specific topics of focus in a case study.

Nonresponse Bias

The expected 9% response rate for WIC participants based on the previous survey is low. Nonresponse bias is likely because those who respond to surveys may systematically differ from nonrespondents. Mitigation strategies such as oversampling WIC participants to account for the low response rate should be considered.

RESPONSE: The WIC participant experience surveys have a target of 200 respondents per State agency and the evaluation will oversample WIC participants at each State agency to reach that target. The evaluation team will weight the responses based on basic demographics to partially address response bias. Oversampling based on demographic characteristics would be hard to accomplish because we expect most, if not all, of the State agencies will choose to conduct the randomization of their participant lists and directly field the surveys so they wouldn’t have to share participant PII. In addition, the response rate may be higher with the survey coming from the WIC program, a trusted source. The study team would provide State agencies with simple computer code to conduct the randomization, which would be substantially complicated if it included oversampling of specific demographic groups.