Appendix D. Additional evaluation details_5-20-2025

Appendix D

Additional evaluation details

Additional information on: A.2. Purpose and use of the information

WIC agencies at the State and local levels can improve the success of modernization efforts by using information about the implementation experiences of other WIC agencies. FNS and WIC agencies will use information on the impacts of modernization efforts to continue and accelerate efforts that improve the program for participants.

The study will collect a broad range of data from all 88 WIC State agencies, as well as a convenience sample of WIC program staff, WIC participants, and WIC & FMNP vendors/outlets during FY 2025–2028. These data will provide needed information to assess implementation progress and key process outcomes of the modernization efforts across the State agencies, local agencies, and WIC & FMNP vendors and outlets, and to assess how the WIC State agencies leveraged waivers to aid in implementation. Using qualitative analysis methods, the study team will analyze the data to assess implementation, understand the projects’ reach and other implementation characteristics, challenges and their solutions, promising practices, waiver usage, and the conditions needed to replicate and sustain the modernization interventions.

This study also will use the data collected to estimate the impact of the modernization efforts on WIC participants through key short-term and intermediate-term outcome measures including enrollment, participation, retention, benefit redemption, participant experience, and program access and delivery. The impact analysis will use a longitudinal regression model, drawing on comprehensive information from the implementation and waiver studies (e.g., from document reviews and State agency interviews) about the projects that were implemented and timing of implementation to identify the set of State agency comparisons and time periods that can yield meaningful impact estimates. The impact analysis will rely on administrative data that FNS is collecting under another project to measure key outcomes: enrollment, participation, retention, and benefit redemption. The analysis will use surveys of WIC participants, staff, and vendors/outlets to measure outcomes related to experience and satisfaction with modernization efforts. The analysis will measure the impact of modernization on program access and delivery by examining differential outcomes for racial and ethnic groups.

Additional information on: A.2. How and by whom the information will be collected

A.2.2. Implementation study recruitment

Study	Instrument
Implementation Study	WIC State agency staff interview protocol (Appendix F.1)
	WIC & FMNP vendor/outlet staff case study interview protocol (Appendix F.2)
	WIC local agency staff case study interview protocol (Appendix F.3)
	WIC participant case study focus group guide in English or Spanish (Appendices F.4 and F.4a)

WIC State agency recruitment. Before data collection begins, FNS will notify the regional offices through email (Appendix G.1) and the regional offices will notify the WIC State agencies through email (Appendix G.2) of the upcoming data collection activities. Next, the study team will invite WIC State agency staff via email to attend a webinar hosted by the study team that describes the evaluation objectives, data sources, time burden, and how participation will benefit State agencies by informing program and policy changes (Appendix H). After the webinar, the study team will reach out to each WIC State agency and hold individual initial planning calls where the team will answer questions about the study, describe the study data collection processes, timelines, and recruitment needs in more detail.

In addition to the agency webinar, the following recruitment materials will be used for WIC State agencies: (1) a USDA endorsement letter for WIC State agencies (Appendix I), (2) a study description for WIC State agencies (Appendix J), and (3) the WIC State agency study recruitment email from the research study team (Appendix K.1). Using the State agency staff interview scheduling email (Appendix K.2), the study team will schedule each of the 60-minute State agency interviews at a time convenient for the State agency staff member most knowledgeable about the WIC modernization efforts. The interview scheduling email (K.2) includes information specific to each State agency regarding the topics to be covered in the interview. To ensure participation of staff from all 88 WIC State agencies, and reduce the likelihood of missed interviews, the study team will confirm the State agency interview date and time and send a WIC State agency staff interview reminder email (Appendix K.3) prior to the scheduled interview. Once the interview is complete, the study team will send the respondent the WIC State agency staff interview thank you email (Appendix K.4). If there are items that State agency staff prefer to cover in an email follow-up to the interview, the follow-up will be included in the thank you email (Appendix K.4).

Case study recruitment. For each WIC local agency site identified for the case study, the WIC State agency will send them the WIC local agency case study recruitment email from State agency (Appendix L.1) to let them know they have been recommended for participation. Then, the study team will follow up by sending them the WIC local agency case study planning call recruitment email (Appendix L.2) to schedule a time to discuss the case study process in more detail with the local agency staff. The assigned case study team, comprising of three trained data collectors, will work with an on-site coordinator (either State or local agency staff) to identify up to 7 local agency staff and up to 6 WIC & FMNP vendors/outlets to recruit for interviews (Appendices F.2 and F.3) about the case study’s focus modernization activities. The case study team will also work with the on-site coordinator to identify up to 29 WIC participants affected by the case study’s focus modernization activities to recruit for two focus groups in English or Spanish (Appendices F.4 and F.4a).

For the local agency staff case study interviews (Appendix F.3), the case study team will send the WIC local agency staff case study interview scheduling email (Appendix L.3) to the staff identified for these one-hour WIC local agency staff case study interviews (Appendix F.3), including a study description for WIC local agencies (Appendix L.4) as an attachment. As the site visit approaches, the study team will send the WIC local agency staff case study interview reminder email about the upcoming interview’s date and time (Appendix L.5). Once each local agency case study staff interview (Appendix F.3) is complete, the study team will send the respondent the WIC local agency staff case study interview thank you email (Appendix L.6).

For the WIC & FMNP vendor/outlet staff case study interviews (Appendix F.2), the State or local agency director will first send the WIC & FMNP vendor/outlet staff case study interview recruitment email from State or local agency (Appendix M.1) to selected WIC & FMNP vendors/outlets, notifying them of their nomination to participate in the WIC & FMNP vendor/outlet staff case study interview (Appendix F.2). The case study team will then send the selected WIC & FMNP vendors/outlet staff the WIC & FMNP vendor/outlet staff case study interview recruitment email (Appendix M.2) to provide more details on the study and begin the process of scheduling the interview. This email will include the study description for WIC & FMNP vendors/outlets (Appendix M.3) that provides an overview of the evaluation’s data collection activities. Once an interview date and time is confirmed, the study team will send the WIC & FMNP vendor/outlet staff case study interview confirmation email (Appendix M.4) and interview confirmation text (Appendix M.5). As the site visit approaches, the case study team will send the WIC & FMNP vendor/outlet staff case study interview reminder email (Appendix M.6) and interview reminder text (Appendix M.7) about the upcoming WIC & FMNP vendor/outlet case study interview (F.2). Once each WIC & FMNP vendor/outlet staff case study staff interview (Appendix F.2) is complete, the study team will send the respondent the WIC & FMNP vendor/outlet staff case study interview thank you email (Appendix M.8) and interview thank you text (Appendix M.9).

To prepare for the WIC participant case study focus groups in English or Spanish (Appendices F.4 and F.4a), about six to eight weeks before the scheduled site visit the study team will provide the local agency with the WIC participant case study focus group recruitment flyer (Appendices N.1 and N.1a), in both English and Spanish, to share with WIC participants to recruit volunteers. The study team aims to recruit an average of 9 WIC participants for each of the two 90-minute WIC participant case study focus groups in English or Spanish (Appendices F.4 and F.4a). The State or local agency staff will then send out the WIC participant case study focus group recruitment email in English or Spanish (Appendices N.2 and N.2a) to invite WIC participants to be part of a focus group in English or Spanish (Appendices F.4 and F.4a). The study team or local agency staff will follow up by sending the WIC participant case study focus group scheduling email in English or Spanish (Appendices N.3 and N.3a) to WIC participants who expressed interest in the focus group, providing them with details about possible dates and times and including the study description for WIC participants in English or Spanish (Appendices N.4 and N.4a) to provide more details about the study. Once WIC participants agree to participate in a focus group, the study team will send them the WIC participant case study focus group confirmation email in English or Spanish (Appendices N.5 and N.5a) and case study focus group confirmation text in English or Spanish (Appendices N.6 and N.6a) with details about the date, time, and location of the scheduled focus group. In the days leading up to the focus group, the study team will send participants the WIC participant case study focus group reminder email in English or Spanish (Appendices N.7 and N.7a) and case study focus group reminder text in English or Spanish (Appendices N.8 and N.8a) that reminds the participants of the date, time, and location of the scheduled focus group. Once each WIC participant case study focus group discussion in English or Spanish (Appendices F.4 and F.4a) is complete, the study team will send each participant the WIC participant case study focus group thank you email in English or Spanish (Appendices N.9 and N.9a) and case study focus group thank you text in English or Spanish (Appendices N.10 and N.10a).

A.2.3. Waiver study recruitment

Study	Instrument
Waiver Study	WIC State agency staff interview protocol (Appendix F.1)

The study team will gather information on waivers through the implementation study’s State agency staff interview (Appendix F.1). Recruiting and follow-up for the State agency staff interview is discussed with recruiting for the implementation study in Section A.2.2 (above).

A.2.4. Impact study recruitment

Study	Instrument
Impact Study	WIC program staff experience survey (Appendix O)
	WIC & FMNP vendor/outlet staff experience survey (Appendix P)
	WIC participant experience survey in English or Spanish (Appendices Q and Q.1)

The study team will work with FNS to reach out to WIC State agency directors asking them to work with the study team to field each of the experience surveys (with the WIC participant experience survey being in both English and Spanish) (Appendix O, Appendix P, and Appendices Q and Q.1).

Program staff experience survey. About three months before the field period for the WIC program staff experience survey (Appendix P), the study team will email to the WIC State agency director the FNS and Mathematica research study team request to WIC State agencies for WIC program staff contact list for experience survey (Appendix R.1). This email requests the email contact information for 25 randomly selected staff from each category: WIC State agency staff, local agency staff, and clinic staff. This email also contains a short description of what information the survey will collect from WIC staff. The State agency will subsequently share the contact information of the randomly selected WIC staff via the secure platform Box. The study team will subsequently send the randomly selected WIC State agency staff, local agency staff, and clinic staff the WIC program staff experience survey invitation email from the research study team (Appendix R.2), inviting them to complete the WIC program staff experience survey (Appendix O) on the web. The study team will send nonrespondents the survey link again, up to four times, via the WIC program staff experience survey reminder email (Appendix R.3). The study team will send all WIC program staff who complete their survey the WIC program staff experience survey thank you email (Appendix R.4).

WIC & FMNP vendor/outlet staff experience survey. For the WIC & FMNP vendor/outlet staff experience survey (Appendix P), the study team will offer State agencies the option to reach out to respondents directly or to provide the study team with respondent contact information; the study team anticipates that most State agencies will send the vendor contact information to the study team to conduct outreach and recruitment. About three months before the field period for the WIC & FMNP vendor/outlet staff experience survey (Appendix P), the study team will email the WIC State agency director the request to WIC State agencies for WIC & FMNP vendor/outlet staff contact list for experience survey (Appendix S.1) requesting the contact information (email and/or text) for 25 randomly selected WIC & FMNP vendors/outlets in each category: online vendors, in-store vendors, and farmers’ market vendors/roadside produce stands. This email also contains a short description of what information the survey will collect from the WIC vendors/authorized outlets. The WIC State agency director will subsequently send selected vendors/outlets staff the WIC & FMNP vendor/outlet staff experience survey recruitment email (Appendix S.2) notifying them of the upcoming experience survey and requesting their participation. Once the WIC State agency director has sent that email and shared the WIC & FMNP vendor/outlet contact information via Box with the study team, the study team will begin data collection activities. First, the study team will send the WIC & FMNP vendor/outlet staff experience survey invitation email (Appendix S.3) and WIC & FMNP vendor/outlet staff experience survey invitation text (Appendix S.4), each containing a link to the web-based WIC & FMNP vendor/outlet staff experience survey (Appendix P). The study team will send out the survey link to nonrespondents up to four times via the WIC & FMNP vendor/outlet staff experience survey reminder email (Appendix S.5) and WIC & FMNP vendor/outlet staff experience survey reminder text (Appendix S.6) to nonrespondents. For nonrespondents for whom the study team has a text number, the team will conduct one phone call reminder (same content as email reminder, Appendix S.5). Once respondents complete their survey, the study team will send them the WIC & FMNP vendor/outlet staff experience survey thank you email (Appendix S.7) and the WIC & FMNP vendor/outlet staff experience survey thank you text (Appendix S.8)

WIC participant experience survey. The study team anticipates that many State agencies (85 percent) will prefer to reach out directly to WIC participants to invite them to complete the WIC participant experience survey in English or Spanish (Appendices Q and Q.1) so that they do not need to share participant contact information with the study team. As a result, about three months before the field period for the WIC participant experience survey, the study team will email each WIC State agency director the research study team request to WIC State agencies for WIC participant contact list for experience surveys (Appendix T.1) to determine whether the WIC State agency prefers to field the participant experience survey or prefers that the study team does it. This email will also request that the WIC State agency start the process of randomly selecting up to 2,300 participants across different local WIC agencies for participation in the survey. Next, the State agency director will send the WIC participant experience survey recruitment email from WIC State agency in English or Spanish (Appendices T.2 and T.2a) to the randomly selected WIC participants, notifying them of their selection to complete the WIC participant experience survey in English or Spanish (Appendices Q and Q.1). The WIC State agency director will then send the WIC participant experience survey invitation email from WIC State agency in English or Spanish (Appendices T.3 and T.3a) and the WIC participant experience survey invitation text in English or Spanish (Appendices T.4 and T.4a) to selected WIC participants, each communication containing the web survey link. The WIC State agency will send nonrespondents up to four survey reminders using the WIC participant experience survey reminder email in English or Spanish (Appendices T.5 and T.5a) and the WIC participant experience survey reminder text in English or Spanish (Appendices T.6 and T.6a). Once respondents complete their WIC participant experience survey in English or Spanish (Appendices Q and Q.1), the WIC State agency director will send them the WIC participant experience survey thank you email in English or Spanish (Appendices T.7 and T.7a) and the WIC participant experience survey thank you text in English or Spanish (Appendices T.8 and T.8a). For State agencies that prefer to provide contact information (email and/or text) for WIC participants to the contractor team, the team will send the invitation email in English or Spanish (Appendices T.3 and T.3a) and text in English or Spanish (Appendices T.4 and T.4a), up to four reminders in English or Spanish (Appendices T.5, T.5a, T.6, and T.6a) and thank you communications in English or Spanish (Appendices T.7, T.7a, T.8, and T.8a). The team will also conduct one reminder phone call in English or Spanish to nonrespondents for whom the team has a text number (same content as email reminder, Appendices T.5 and T.5a). If the State agency provides a physical address for the WIC participant, the study team will send a reminder post card in English or Spanish (same content as email reminder, Appendices T.5 and T.5a). WIC participants who prefer to complete the survey by phone will have the option to do so.

Additional information on: A.2. Information shared with any other organizations inside or outside USDA or the government

In addition, annual reporting will update public audiences, State agencies, and FNS on modernization implementation and key findings of the evaluation. The study team will produce an annual evaluation brief for public audiences describing the progress and high-level findings for each modernization project, the evaluation overall, and any special topics determined in consultation with FNS. Annual State-by-State summaries (88 each year) will provide a description of activities in each State agency for FNS and for staff in each State agency. The study team will conduct an annual briefing for FNS and an annual public webinar. In addition, each year, the study team will provide a midyear memo and slide deck to update FNS approximately 6 months before the annual reporting.

A final summary report, available publicly, will describe the findings on the key outcomes the evaluation is studying. Final evaluation report briefs will be developed and will provide short descriptions of the findings for each focus area for all audiences. To increase the reach of the evaluation findings, dissemination may also include conference presentations, journal articles, short videos, and infographics for public audiences, and a briefing for WIC State agency leaders and staff. The study team will conduct a briefing for FNS of the final summary report.

Additional information on: A.9. Explain any decisions to provide any payment or gift to respondents.

The purpose of incentives is to (1) reduce nonresponse bias, (2) improve data quality, and (3) reduce data collection costs. Providing study participants with a monetary incentive reduces non-response bias and improves representativeness, especially in populations defined as being in poverty.^{1 2 3 4 5} Specifically, incentives can improve sample representativeness and reduce non-response bias ^{6 7} by encouraging those less interested in research to participate,⁸ including low-income respondents.⁹ Several studies provide evidence that offering incentives may improve representation for low-education, low-income, and ethnic minority subgroups. In a meta-analysis¹⁰, Mercer and colleagues estimated an improvement of 5 percentage points for surveys that promised $10 compared to no incentive. Importantly, Frederickson et al. (2005)¹¹ found a $10 contingent incentive to increase responses by 20 percentage points among Medicaid recipients, a similar population to those being recruited in this study. Minimizing the cost of completing study activities and maximizing the social reward for respondents can increase the likelihood that respondents will participate.^12,13 In addition, incentives treat respondents justly and with respect by recognizing and acknowledging the effort they expend to participate.^14,15

The population of interest in the current study is comparable to the populations discussed above, as WIC eligibility is based in part on participants having household income at or below an income level set by the State agency, or by participating in other assistance programs based on income. Based on the empirical evidence above, FNS feels strongly that the proposed incentives for the WIC participants are necessary to obtain enough completed data collection activities from a diverse group of respondents.

1 Singer E. (2002). The use of incentives to reduce non response in households surveys in: Groves R, Dillman D, Eltinge J, Little R (eds.) Survey Non Response. New York: Wiley, pp 163-177.

2 James T. (1996). Results of wave 1 incentive experiment in the 1996 survey of income and program participation. Proceedings of the Survey Research Section, American Statistical Association, 834-839.

3 Groves R, Fowler F, Couper M, Lepkowski J, Singer E. (2009) in: Survey methodology. John Wiley & Sons, pp 205-206.

4 Singer E. (2002).

5 Singer E and Ye C. (2013). The use and effectives of incentives in surveys. Annals of the American Academy of Political and Social Science, 645(1):112-141.

6 Groves, R., Singer, E., Corning, A. (2000). Leverage-saliency theory of survey participation: description and an illustration. Public Opinion Quarterly. 64(3): 299-308.

7 Messer B and Dillman D (2011). Surveying the general public over the internet using address-based sampling and mail contact procedures. Public Opinion Quarterly, 75:429-457.

8 Groves RM, Couper MP, Presser S, Singer E, Tourangeau R, Acosta G, Nelson L. (2006) Experiments in Producing Nonresponse bias. Public Opinion Quarterly. 70(5): 720-736

9 Singer, E., and R.A. Kulka. “Paying Respondents for Survey Participation.” In Studies of Welfare Populations: Data Collection and Research Issues. Panel on Data and Methods for Measuring the Effects of Changes in Social Welfare Programs, edited by Michele Ver Ploeg, Robert A. Moffitt, and Constance F. Citro. Committee on National Statistics, Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press, 2002, pp. 105–128.

10 Mercer A, Caporaso A, Cantor D, Townsend R (2015). How much gets you how much? Monetary incentives and response rates in household surveys. Public Opinion Quarterly, 79:105-129.

11 Fredrickson, D.D., Jones, T.I, Molgaard, C.A., Carman, C.G., Schukman, J., Dismuke, S.E. and E. Ablah (2005) Optimal Design Features for Surveying Low-Income Populations. Journal of Health Care for the Poor and Underserved 16: 677-690.

12 Dillman, D. Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method. John Wiley, 2009.

13 Porter, S. R. “Raising Response Rates: What Works?” New Dimensions for Institutional Research, vol. 2004, no. 121, 2004, pp. 5–21.

14 Halpen, S. D., J. H. Karlawish, D. Casarett, J.A. Berlin, and D.A. Asch. “Empirical Assessment of Whether Moderate Payments are Undue or Unjust Inducements for Participation in Clinical Trials.” Archives of Internal Medicine, vol. 164, no. 7, 2004, pp. 801–803.

15 Russell, M. L., D. G. Moralejo, and E. D. Burgess. “Participants’ Perspectives.” Journal of Medical Ethics, vol. 26, no. 2, 2000, pp. 126–130.