SUPPORTING STATEMENT - PART B for
OMB Control Number 0584-[XXXX]:
WIC & FMNP Outreach, Innovation, and Modernization Evaluation
March 2025
Project Officer: Carol Dreibelbis
Social Science Research Analyst
Evidence, Analysis, and Regulatory Affairs Office
USDA, Food and Nutrition Service
1320 Braddock Place
5th Floor
Alexandria, Virginia 22314
CONTENTS
B.1 Respondent universe and sampling methods 2
B.2 Procedures for the collection of information 6
B.3 Methods to maximize the response rates and to deal with nonresponse 14
B.4 Test of procedures or methods to be undertaken 19
B.5 Individuals consulted on statistical aspects and individuals collecting and/or analyzing data 19
APPENDICES
Appendix A Public Law 117-2, Sec. 1106 – WIC Program Modernization
Appendix B Public Law 111-296, Sec. 305
Appendix C Code of Federal Regulations Title 7, Section 246.26(k)
Appendix D Additional evaluation details
Appendix D.1 Data collection plan table
Appendix E IRB approval letter
Appendix F.1 WIC State agency staff interview protocol
Appendix F.2 WIC & FMNP vendor/outlet staff case study interview protocol
Appendix F.3 WIC local agency staff case study interview protocol
Appendix F.4 WIC participant case study focus group guide
Appendix F.4a. WIC participant case study focus group guide (Spanish)
Appendix G.1 Regional office study notification email from FNS
Appendix G.2 State study notification email from regional office
Appendix H WIC State agency webinar invitation email and webinar
Appendix I USDA endorsement letter for WIC State agencies
Appendix J Study description for WIC State agencies
Appendix K.1 WIC State agency recruitment email from the research study team
Appendix K.2 WIC State agency staff interview scheduling email
Appendix K.3 WIC State agency staff interview reminder email
Appendix K.4 WIC State agency staff interview thank you email
Appendix L.1 WIC local agency case study recruitment email from State agency
Appendix L.2 WIC local agency case study planning recruitment email from the research study team
Appendix L.3 WIC local agency staff case study interview scheduling email
Appendix L.4 Study description for WIC local agencies
Appendix L.5 WIC local agency staff case study interview reminder email
Appendix L.6 WIC local agency staff case study interview thank you email
Appendix M.1 WIC & FMNP vendor/outlet staff case study interview recruitment email from State or local agency
Appendix M.2 WIC & FMNP vendor/outlet staff case study interview recruitment email from the research study team
Appendix M.3 Study description for WIC & FMNP vendors/outlets
Appendix M.4 WIC & FMNP vendor/outlet staff case study interview confirmation email
Appendix M.5 WIC & FMNP vendor/outlet staff case study interview confirmation text
Appendix M.6 WIC & FMNP vendor/outlet staff case study interview reminder email
Appendix M.7 WIC & FMNP vendor/outlet staff case study interview reminder text
Appendix M.8 WIC & FMNP vendor/outlet staff case study interview thank you email
Appendix M.9 WIC & FMNP vendor/outlet staff case study interview thank you text
Appendix N.1 WIC participant case study focus group recruitment flyer
Appendix N.1a. WIC participant case study focus group recruitment flyer (Spanish)
Appendix N.2 WIC participant case study focus group recruitment email from State/local agency
Appendix N.2a. WIC participant case study focus group recruitment email from State/local agency (Spanish)
Appendix N.3 WIC participant case study focus group scheduling email
Appendix N.3a. WIC participant case study focus group scheduling email (Spanish)
Appendix N.4 Study description for WIC participants
Appendix N.4a. Study description for WIC participants (Spanish)
Appendix N.5 WIC participant case study focus group confirmation email
Appendix N.5a. WIC participant case study focus group confirmation email (Spanish)
Appendix N.6 WIC participant case study focus group confirmation text
Appendix N.6a. WIC participant case study focus group confirmation text (Spanish)
Appendix N.7 WIC participant case study focus group reminder email
Appendix N.7a WIC participant case study focus group reminder email (Spanish)
Appendix N.8 WIC participant case study focus group reminder text
Appendix N.8a. WIC participant case study focus group reminder text (Spanish)
Appendix N.9 WIC participant case study focus group thank you email
Appendix N.9a. WIC participant case study focus group thank you email (Spanish)
Appendix N.10 WIC participant case study focus group thank you text
Appendix N.10a.WIC participant case study focus group thank you text (Spanish)
Appendix O WIC program staff experience survey
Appendix P WIC & FMNP vendor/outlet staff experience survey
Appendix Q WIC participant experience survey
Appendix Q.1. WIC participant experience survey (Spanish)
Appendix R.1 Research study team request to WIC State agencies for WIC program staff contact list for experience survey
Appendix R.2 WIC program staff experience survey invitation email from the research study team
Appendix R.3 WIC program staff experience survey reminder email
Appendix R.4 WIC program staff experience survey thank you email
Appendix S.1 Research study team request to WIC State agencies for WIC & FMNP vendor/outlet staff contact list for experience survey
Appendix S.2 WIC & FMNP vendor/outlet staff experience survey recruitment email from WIC State agency
Appendix S.3 WIC & FMNP vendor/outlet staff experience survey invitation email from the research study team
Appendix S.4 WIC & FMNP vendor/outlet staff experience survey invitation text
Appendix S.5 WIC & FMNP vendor/outlet staff experience survey reminders
Appendix S.6 WIC & FMNP vendor/outlet staff experience survey reminder text
Appendix S.7 WIC & FMNP vendor/outlet staff experience survey thank you email
Appendix S.8 WIC & FMNP vendor/outlet staff experience survey thank you text
Appendix T.1 Research study team request to WIC State agencies for WIC participant contact list for experience survey
Appendix T.2 WIC participant experience survey recruitment email from WIC State agency
Appendix T.2a. WIC participant experience survey recruitment email from WIC State agency (Spanish)
Appendix T.3 WIC participant experience survey invitation email from WIC State agency
Appendix T.3a. WIC participant experience survey invitation email from WIC State agency (Spanish)
Appendix T.4 WIC participant experience survey invitation text
Appendix T.4a. WIC participant experience survey invitation text (Spanish)
Appendix T.5 WIC participant experience survey reminders
Appendix T.5a. WIC participant experience survey reminders (Spanish)
Appendix T.6 WIC participant experience survey reminder text
Appendix T.6a. WIC participant experience survey reminder text (Spanish)
Appendix T.7 WIC participant experience survey thank you email
Appendix T.7a. WIC participant experience survey thank you email (Spanish)
Appendix T.8 WIC participant experience survey thank you text
Appendix T.8a. WIC participant experience survey thank you text (Spanish)
Appendix U.1 First public comment
Appendix U.2 Second public comment
Appendix U.3 Third public comment
Appendix V.1 FNS response to the first public comment
Appendix V.2 FNS response to the second public comment
Appendix V.3 FNS response to the third public comment
Appendix W Incentive plan for each respondent type
Appendix X.1 Contractor staff confidentiality pledge – subcontractors
Appendix X.2 Mathematica confidentiality pledge
Appendix Y Burden table
Appendix Z WIC pretest memo
Appendix AA NASS comments and responses
Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (such as establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.
This new information collection request is for an evaluation with three components. Table B.1.1 summarizes data collection activities for each study component, including the potential respondent universe, selection method, number to be contacted and expected number of respondents, and expected response rate for each. The rest of this section describes the selection methods in detail.
Table B.1.1. Summary of data collection activities, respondent universe, selection methods, and expected response rates
Data collection activity |
Respondent universe |
Selection method |
Total to be contacted |
Expected number of respondents |
Expected response rate |
Implementation study |
|||||
WIC State agency staff interview (Appendix F.1) |
440 |
Purposive |
132 |
132 |
100% |
WIC local agency staff case study interview (Appendix F.3) |
110,000 |
Purposive |
222 |
160 |
72% |
WIC & FMNP vendor/ outlet staff case study interview (Appendix F.2) |
460,000 |
Purposive |
100 |
68 |
68% |
WIC participant case study focus group in English or Spanish (Appendices F.4 and F.4a) |
7,000,000 |
Convenience |
920 |
580 |
63% |
Waiver study |
|||||
Will rely on implementation study data |
|||||
Impact study |
|||||
WIC program staff experience survey (Appendix O) |
110,440 |
Random selection |
5,200 |
4,160 |
80% |
WIC & FMNP vendor/ outlet staff experience survey (Appendix P) |
460,000 |
Random selection |
4,100 |
3,280 |
80% |
WIC participant experience survey in English or Spanish (Appendices Q and Q.1) |
7,000,000 |
Random selection |
175,550 |
15,800 |
9% |
Total |
7,570,440 |
|
186,224 |
24,180 |
13% |
Note: The study team expects that half of the 88 State agencies will have up to two State agency staff participate in the WIC State agency staff interview (Appendix F.1).
The implementation study. The respondent universe for the WIC State agency staff interviews (Appendix F.1) is all 88 WIC State agencies; however, we expect approximately half of those State agencies will have up to two staff participate in the WIC State agency staff interview (Appendix F.1). There will be no sampling because the evaluation relies on a detailed understanding of implementation for all State agency-led modernization projects. The study team expects to interview the one or two most knowledgeable staff. The study team expects a 100 percent response rate.
The respondent universe for the WIC local agency staff case study interviews (Appendix F.3) includes staff in approximately 2,000 agencies and 10,000 clinics. The study team will purposively select up to 32 local agencies implementing modernization projects for the case studies, which are intended to provide a rich, contextualized understanding of specific modernization activities rather than generalizable or representative information about typical activities (see B.2.1 below). We plan to interview an average of 5 staff per case study for a total of 160 staff. With an anticipated response rate of about 72%, we would reach out to about 222 staff to invite them to complete an interview. The study team will consider the following factors for site selection: (1) sites where the implementation information suggests a State agency’s modernization work is impactful, the modernization projects are innovative, or the State agency is successfully implementing a particularly challenging activity; (2) sites with modernization efforts across various domains and program areas, and that vary in types of efforts within the same domain; and (3) sites representing a mix of urban and rural locations, serving a mix of populations, located on Tribal lands, and representing varying geographies (from different regions and States). The study team expects a 72 percent response rate.
The study team anticipates interviewing staff from WIC & FMNP vendors/outlets in about half of the 32 sites, with an average of 4 WIC & FMNP vendors/outlets participating at each of those sites, for a total of 68 vendors/authorized outlets completing the WIC & FMNP vendor/outlet staff case study interview (Appendix F.2). With an anticipated response rate of about 68%, the study team would contact approximately 100 WIC & FMNP vendors/outlets and will select a purposive sample of vendors/authorized outlets near the local sites selected for the case studies.
The study team will select a convenience sample of WIC participants served by the local sites selected for the above case studies and invite them to participate in the WIC participant case study focus group in English or Spanish (Appendices F.4 and F.4a). This convenience sampling will be conducted among WIC participants with experiences most relevant to a site’s case study focus. The study team will conduct two focus groups per site with an average of 9 participants per focus group for a total of 580 participants. With an anticipated response rate of about 63%, the study team would reach out to about 920 WIC participants to invite them to participate in a focus group.
The waiver study. The waiver study will rely on data collected for the implementation study, particularly the State agency staff interview.
The impact study. The impact study will include all 88 WIC State agencies; there will be no sampling of State agencies because the evaluation is examining the impact of modernization projects across all State agencies. Each State agency will participate once; the period for fielding the experience surveys within each State agency will depend on the timing of the State agency’s most salient modernization efforts and whether the effects are best measured as soon as possible or after some time has passed. The study team will work with State agencies to randomly select WIC & FMNP vendors/outlets of each type (online, in store, and farmers’ markets) to complete the WIC & FMNP vendor/outlet staff experience survey (Appendix P); randomly select WIC staff of each type (State, local, and clinic) and invite them to complete the WIC program staff experience survey (Appendix O); and randomly select WIC participants and invite them to complete the WIC participant experience survey in English or Spanish (Appendices Q and Q.1). The study team will use the experience surveys to measure outcomes related to knowledge, experience, and satisfaction with modernization efforts.
Respondents. The total number of estimated unique respondents for this study is 24,180. Members of the public affected by the data collection include individuals/households, business (profit, non-profit, or farm) and State, local, or tribal government agencies. Table B.1.2, shows the respondent universe, sample size, and expected response rate for each respondent type.
Table B.1.2. Summary of respondents and nonrespondents by respondent type
Data collection |
Affected public |
Respondent type |
Total to be contacted |
Expected number of respondents |
Expected number of nonrespondents |
Expected response rate |
WIC State agency interviews (Appendix F.1) |
State and Tribal government |
State/Territory agency staff |
132 |
132 |
0 |
100% |
Case study interviews (Appendices F.2 and F.3) and focus groups in English or Spanish (Appendices F.4 and F.4a) |
Local government, business (profit, non-profit, or farm), and individuals/ households |
Local agency/clinic staff |
222 |
160 |
62 |
72% |
WIC & FMNP Vendor/ outlet staff |
100 |
68 |
32 |
68% |
||
WIC participants |
920 |
580 |
340 |
63% |
||
Experience surveys (Appendices O, P, Q and Q.1) |
State, local, and Tribal government, business (profit, non-profit, or farm), and individuals/ households |
State and local/clinic agency staff |
5,200 |
4,160 |
1,040 |
80% |
WIC & FMNP vendor/ outlet staff |
4,100 |
3,280 |
820 |
80% |
||
WIC participants |
175,550 |
15,800 |
159,750 |
9% |
||
Total |
|
|
186,224 |
24,180 |
162,044 |
13% |
Describe the procedures for the collection of information, including the following:
Statistical methodology for stratification and sample selection
Estimation procedure
Degree of accuracy needed for the purpose described in the justification
Unusual problems requiring specialized sampling procedures
Any use of periodic (less frequent than annual) data collection cycles to reduce burden
B.2.1. Statistical methodology, estimation, and degree of accuracy
In this study, the procedures for collection of information or data will involve sampling and stratification; the study will use estimation procedures for the impact study. The implementation study interviews (Appendices F.1, F.2, and F.3) and focus groups in English or Spanish (Appendices F.4 and F.4a) are not intended to produce generalizable or representative information about typical WIC modernization activities. The collected information will be used to develop detailed case studies that provide examples of how State and local agencies are implementing the modernization activities to better serve WIC participants; give context to findings in the impact study; and provide the data needed to identify best practices and lessons learned for other WIC agencies undertaking similar modernization activities. For the waiver study, the study team will conduct quantitative analysis via Excel to describe the number and type of waivers and the waiver implementation timeline by State agency. For the impact study, the study team will estimate multivariate regression models of outcomes on measures of project implementation. Aside from the experience surveys (including the WIC participant experience survey in English or Spanish) (Appendices O, P, Q, and Q.1), the measures will be from WIC administrative data (not collected by this study). Below is a description of the data collection procedures for the study components.
Implementation study – State agency interviews. The study team will interview staff from all 88 WIC State agencies using the WIC State agency staff interview protocol (Appendix F.1). The study team will conduct the virtual interviews once per year for three years for modernization projects that are ongoing throughout the study period. However, if a project’s implementation ends before the evaluation’s data collection begins, the study team will administer the WIC State agency staff interview (Appendix F.1) once for a retroactive perspective. The field period for the first round of WIC State agency staff interviews (Appendix F.1) will start as soon as possible once the study receives clearance to begin data collection. The second round will take place in Summer 2026 and the third round will take place in Summer 2027.
Interviewers will tailor the WIC State agency staff interview protocol (Appendix F.1) based on the information learned from reviewing State agency documents (for example, progress and final reports and other project documentation related to implementing the modernization projects) to reduce burden and streamline the interview process. The interview will be conducted by a trained interviewer (Appendix F.1). The interview will be recorded, with the consent of the respondent, and transcribed. Once the interview is complete, the study team will send the State agency staff member who participated the WIC State agency staff interview thank you email (Appendix K.4).
Implementation study – case studies. The study team will conduct two-day, in-person site visits from Fall 2025 through November 2027 with up to 32 local sites implementing modernization projects. The number of case studies the team conducts each year will be determined by the stage of project implementation; to capture a more complete picture of the intervention, the study team will aim to collect case study data when the grant funding for the WIC local agencies is close to ending. Each visit will include in-person interviews with an average of five staff from local WIC agencies and clinics using the WIC local agency staff case study interview protocol (Appendix F.3); interviews with an average of four WIC & FMNP vendors/outlets in about half of the case study sites using the WIC & FMNP vendor/outlet staff case study interview protocol (Appendix F.2); and two focus groups with an average of nine WIC participants per focus group using the WIC participant case study focus group guide in English or Spanish (Appendices F.4 and F.4a).
The on-site coordinator will assist the case study team with logistics for conducting the interviews and will help find a convenient location to hold the site’s focus groups. Before each site visit, the case study team will tailor the data collection instruments for the site to reduce burden and streamline the interview process, focusing on information not already provided through the document review.
WIC program staff case study interviews. A trained data collector will conduct the WIC local agency staff case study interview (Appendix F.3) The interview will be recorded, with the respondent’s permission, for later transcription. Once the interview is complete, the case study team will send the program staff who participated the WIC local agency staff case study interview thank you email (Appendix L.6).
WIC & FMNP vendor/outlet staff case study interviews. A trained data collector will conduct the in-person interview using the WIC & FMNP vendor/outlet staff case study interview protocol (Appendix F.2) and will record the interview, with the respondent’s permission, for later transcription. Once the interview is complete, the case study team will send the WIC & FMNP vendor/outlet staff who participated the WIC & FMNP vendor/outlet staff case study interview thank you email (Appendix M.8) and WIC & FMNP vendor/outlet staff case study interview thank you text (Appendix M.9), depending on their communication preference, accompanied by a $50 e-gift card.
WIC participant case study focus groups. The study team will conduct the focus groups using the WIC participant case study focus group guide in English or Spanish (Appendices F.4 and F.4a). The study team will give participants a $50 gift card at the conclusion of the focus group and send participants the WIC participant case study focus group thank you email in English or Spanish (Appendices N.9 and N.9a) and WIC participant case study focus group thank you text in English or Spanish (Appendices N.10 and N.10a), depending on their communication preference.
Although the study team plans to hold the focus groups in person at locations convenient to the participants, they will be flexible and will conduct focus groups virtually if that is more convenient for participants. Focus groups will be conducted in English and in Spanish, as necessary. Before each focus group, the facilitators will tailor the WIC participant case study focus group guide in English or Spanish (Appendices F.4 and F.4a) to only ask questions that are relevant to the site’s modernization projects, to reduce burden and streamline the focus group discussion.
Waiver study. For the waiver study, the study team will collect data from all 88 WIC State agencies through the WIC State agency staff interview (Appendix F.1). The study team will also rely on data from documentation provided by FNS, including administrative data on waiver issuance and reporting documents from the FNS Regional offices and State agencies. The study team will use this information to tailor the waiver questions in the WIC State agency staff interview protocol (Appendix F.1). The study team will ask waiver questions from the WIC State agency staff interview protocol (Appendix F.1) once per year for three years, for modernization projects that are ongoing through the study period, to collect real-time information about waiver usage. However, if the projects that the waiver supports end before the evaluation’s data collection period begins, the study team will administer these questions once for a retroactive perspective. The field period for the first round of the WIC State agency staff interview (Appendix F.1) will begin as soon as possible once the study receives clearance to begin data collection ; the second round will take place in Summer 2026, and the third round will take place in Summer 2027.
Impact study. For the impact study, the study team will field a set of experience surveys (including the WIC participant experience survey in English or Spanish) (Appendices O, P, Q, and Q.1) to WIC program staff, WIC & FMNP vendors/outlet staff, and WIC participants to measure outcomes related to knowledge, experience, and satisfaction with the modernization efforts. The study team will field the surveys at two times with State agencies: first in Fall 2026, and then in Fall 2027 (as late as possible for results to be analyzed and included in final reporting). Each of the 88 State agencies will only participate once, and the survey period for each State agency will depend on the timing of their most salient modernization efforts and whether the effects are best measured as soon as possible or after some time has passed. These 10-minute experience surveys (including the WIC participant experience survey in English or Spanish) (Appendices O, P, Q, and Q.1) will be administered on the web, and they will be accessible on mobile devices. Please see the end of each survey appendix for screenshot examples of what the programmed web surveys will look like. The study team will use administrative data (collected by another study) to measure enrollment, participation, retention, and benefit redemption from FNS to supplement the information gathered through the experience surveys (including the WIC participant experience survey in English or Spanish) (Appendices O, P, Q, and Q.1). The study team will work closely with each of the WIC State agencies throughout the process of fielding the three experience surveys (including the WIC participant experience survey in English or Spanish) (Appendices O, P, Q, and Q.1).
WIC program staff experience survey. WIC program staff within the jurisdiction of each WIC State agency, which includes the State agency and local agencies/clinics, will complete the 10-minute WIC program staff experience survey (Appendix O) on the web once; the field period for each agency will depend on the timing of their most salient modernization efforts and whether the effects are best measured as soon as possible or after some time has passed. Once the study team receives the contact information of randomly selected WIC staff via the secure platform Box, the study team will send the selected staff the WIC program staff experience survey invitation email from the research study team (Appendix R.2) with the survey link to invite them to complete the WIC program staff experience survey (Appendix O). In addition, the invitation email will include either the study description for WIC State agencies (Appendix J) or the study description for WIC local agencies (Appendix L.4), depending on the respondent. The study team will send the WIC program staff experience survey reminder email (Appendix R.3) up to four times during the field period to WIC program staff who have not completed their survey, reminding them that study participation is mandatory for WIC staff. Once WIC staff complete their survey, the study team will send them the WIC program staff experience survey thank you email (Appendix R.4), thanking them for their participation. The study team expects to collect about 47 completed staff experience surveys per State agency. These are intended to be representative of each State and will be used descriptively, including as outcome measures in a descriptive regression. The study is not powered for precise estimates to test differences between States in staff experience measures.
WIC & FMNP vendor/outlet staff experience survey. WIC & FMNP vendor/outlet staff, including those from online vendors, in-store vendors, and farmers/markets, will be invited to complete the 10-minute WIC & FMNP vendor/outlet staff experience survey (Appendix P) on the web at two times: first in Fall 2026, and then in Fall 2027 (as late as possible for results to be analyzed and included in final reporting). Administration procedures are similar to those for the WIC program staff experience survey (Appendix O). Once the study team receives the contact information of randomly selected WIC & FMNP vendor/outlet staff via the secure platform Box, the study team will send selected WIC & FMNP vendors/outlet staff the WIC & FMNP vendor/outlet staff experience survey invitation email (Appendix S.3) or WIC & FMNP vendor/outlet staff experience survey invitation text (Appendix S.4), each containing a link to the WIC & FMNP vendor/outlet experience survey (Appendix P). The study team will send nonresponding WIC & FMNP vendors/outlet staff the WIC & FMNP vendor/outlet staff experience survey reminder emails (Appendix S.5) or WIC & FMNP vendor/outlet staff experience survey reminder texts (Appendix S.6) during the field period. Reminders will be sent up to four times. The mode of communication for the invitation and reminder texts will depend on respondent communication preference. For nonrespondents for whom the study team has a text number, the team will conduct one phone call reminder (same content as email reminder, Appendix S.5). Once a WIC & FMNP vendor/outlet staff completes the WIC & FMNP vendor/outlet staff experience survey (Appendix P), the study team will send them a $10 e-gift card and the WIC & FMNP vendor/outlet staff experience survey thank you email (Appendix S.7), or WIC & FMNP vendor/outlet staff experience survey thank you text (Appendix S.8) depending on their communication preference. The study team expects to collect about 37 completed WIC & FMNP vendor/outlet staff experience surveys (Appendix P) per State agency. These are intended to be representative of each State and will be used descriptively, including as outcome measures in a descriptive regression. The study is not powered for precise estimates to test differences between States in WIC & FMNP vendor/outlet staff experience measures.
WIC participant experience survey. If the WIC State agency has decided to field the WIC participant experience survey in English or Spanish (Appendices Q and Q.1) directly, rather than through the study team, the State agency will send the WIC participants the WIC participant experience survey invitation email from WIC State agency in English or Spanish (Appendices T.3 and T.3a) or the WIC participant experience survey invitation text in English or Spanish (Appendices T.4 and T.4a), which each contain a link to the web WIC participant experience survey in English or Spanish (Appendices Q and Q.1), depending on their communication preference. The WIC State agency will send out the WIC participant experience survey reminder email in English or Spanish (Appendices T.5 and T.5a) or WIC participant experience survey reminder texts in English or Spanish (Appendices T.6 and T.6a) to nonrespondents throughout the survey field period, depending on their communication preference. Reminders will be sent up to four times. Once a respondent has completed their WIC participant experience survey in English or Spanish (Appendices Q and Q.1), the WIC State agency will send them a $10 e-gift card and the WIC participant experience survey thank you email in English or Spanish (Appendices T.7 and T.7a) or WIC participant experience survey thank you text in English or Spanish (Appendices T.8 and T.8a), depending on their communication preference. If the WIC State agency prefers that Mathematica field the survey, the study team will send the invitation, reminder, and thank you emails/texts in English or Spanish (Appendices T.3, T.3a, T.4, T.4a, T.5, T.5a, T.6, T.6a, T.7, T.7a, T.8, and T.8a). The team will also conduct one reminder phone call in English or Spanish to nonrespondents for whom the team has a text number (same content as email reminder, Appendices T.5 and T.5a). If the State agency provides a physical address for the WIC participant, the study team will send a reminder post card in English or Spanish (same content as email reminder, Appendices T.5 & T.5a). WIC participants who prefer to complete the survey by phone will have the option to do so.
The study team expects to collect an average of 180 completed WIC participant experience surveys in English or Spanish (Appendices Q and Q.1) for each State agency. These are intended to be representative of each State and will be used descriptively and as an outcome in a causal impact model. Assuming variation in program bundles implemented across local agencies, this proposed sample size should result in a minimum detectable effect size of about 0.2 standard deviations, or about 10 percentage points.1
Training. After OMB approval, the data collection staff will participate in a training on the purpose of the study, the data collection instruments, data collection implementation and management, and analysis procedures.
The study has no unusual problems requiring specialized sampling procedures.
The impact study and the implementation study’s case studies each have only one cycle of data collection. For modernization projects that are ongoing throughout the study period, the study team will conduct three cycles of the implementation study’s WIC State agency staff interview (Appendix F.1). This data will be collected annually for three years. This will enable the study to collect real-time measures of reach and other implementation characteristics. The annual reporting from the study will inform ongoing FNS supports for modernization projects.
Describe methods to maximize response rates and to deal with issues of nonresponse. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.
The evaluation’s success will depend on WIC agencies’ participation and support of the various data collection activities. The study team will utilize several strategies before the start of data collection to obtain the support and buy-in of the WIC agencies for the data collection activities. This includes holding a webinar for all 88 WIC State agencies; a follow-up call by the study team’s Tribal liaison with 32 State agencies that are Indian Tribal Organizations (ITOs); and initial planning calls with each WIC State agency to answer additional questions and provide more details about study activities. Throughout data collection, the study team will employ a series of tactics to reduce burden, maximize response rates, and address nonresponse. These tactics are described in further detail below.
Implementation study – WIC State agency staff interviews. As stated earlier, the study team expects a 100 percent response rate for the WIC State agency staff interviews (Appendix F.1). For this type of data collection, a 100 percent response rate is a reasonable target that has been achieved or almost achieved on other, similar projects. To underscore the importance of the study, the study team will share with respondents the USDA endorsement letter for WIC State agencies (Appendix I); in addition, the recruitment materials for WIC state staff (Appendices K.1 and K.2) note that their participation in the study is mandatory. To encourage State agency staff to complete their WIC State agency staff interviews (Appendix F.1) each year, a highly skilled interviewer from the study team will make initial contact with the State agency staff to build rapport. The study team will highlight the flexible scheduling of interviews at a time that is convenient for respondents and the fact that interviews will take place virtually, adding an additional element of flexibility. The study team will send the respondent the WIC State agency staff interview reminder email (Appendix K.3) before the scheduled WIC State agency staff interview (Appendix F.1) in an effort to reduce the likelihood of a missed interview and allow for rescheduling in the case of any last-minute schedule conflicts.
Before each interview, the study team will conduct a thorough review of all reports and documentation to appropriately tailor the WIC State agency staff interview protocol (Appendix F.1) to focus only on information not already provided in the documents. In addition, the WIC State agency staff interview scheduling email (Appendix K2) will provide information about the topics that will be covered, so respondents can be fully prepared to participate.
Implementation study – case studies. The study team expects a 72 percent response rate among the WIC local agency/clinic staff for the WIC local agency staff case study interviews (Appendix F.3) and a 68 percent response rate among the WIC & FMNP vendor/outlet staff participating in the WIC & FMNP vendor/outlet staff case study interview (Appendix F.2). For this type of data collection, these are reasonable targets that have been achieved or almost achieved on other, similar projects. The study team will schedule the site visits for a time period that is most convenient for the WIC local agency/clinic staff and the WIC & FMNP vendor/outlet staff respondents. In addition, the study team will also work closely with the WIC local agency/clinic staff to recruit WIC participants for the two WIC participant case study focus groups at each site. To increase the likelihood of participation, the study team will offer WIC & FMNP vendor/outlet staff and WIC participants each $50 for participating in the WIC & FMNP vendor/outlet staff case study interview (Appendix F.2) and the WIC participant case study focus groups in English or Spanish (Appendices F.4 and F.4a), respectively.
The study team will highlight the flexible scheduling of interviews (Appendices F.2 and F.3) and focus groups in English or Spanish (Appendices F.4 and F.4a) at times that are convenient for respondents. If in-person data collection is not possible at a site, the study team will conduct virtual interviews (Appendices F.2 and F.3) or virtual focus groups in English or Spanish (Appendices F.4 and F.4a) instead. Before each interview (Appendices F.2 and F.3) and focus group in English or Spanish (Appendices F.4 and F.4a), the study team will thoroughly review of all reports and documentation to tailor the protocols to focus only on gathering information not already provided in the documents. In addition, invitation emails sent to WIC local agency/clinic staff (Appendix L.3) and WIC & FMNP vendor/outlet staff (Appendix M.2) about the interviews (Appendices F.2 and F.3) will provide information about the topics that will be covered, so respondents can be fully prepared to participate.
The case study interviews (Appendices F.2 and F.3) and focus groups in English or Spanish (Appendices F.4 and F.4a) are not intended to produce generalizable or representative information about typical WIC modernization activities. The information collected through the case study interviews (Appendixes F.2 and F.3) and focus groups in English or Spanish (Appendices F.4 and F.4a) will be used to develop detailed case studies that provide examples of how State and local agencies are implementing the modernization activities to better serve WIC participants; to give context to the experience survey findings; and to provide the data needed to identify best practices and lessons learned for other WIC agencies undertaking similar modernization activities.
Waiver study. The waiver study will rely on data collected for the implementation study, particularly the State agency staff interview (Appendix F.1; described above) and on extant data in FNS policy documents and State agency reporting documents.
Impact study. The study team expects an 80 percent response rate on the WIC program staff experience survey (Appendix O) and the WIC & FMNP vendor/outlet staff experience survey (Appendix P), and a 9 percent response rate for the WIC participant experience survey in English or Spanish (Appendices Q and Q.1). For this type of data collection with these populations, these response rates are a reasonable target that has been achieved or almost achieved on other, similar projects. For the WIC participant experience survey, the 9% response rate is a realistic estimate based on a similar survey conducted in 20232. This response rate is sufficient for the intended uses in this study. The study team will oversample WIC participants at each State agency to attain the targeted number of 200 respondents per State agency. It’s likely most State agencies will choose to conduct the randomization of their participant lists and field the surveys themselves, so they won’t need to share participant PII. The study team would provide State agencies with simple computer code to conduct this randomization; however it would be substantially more complicated to have State agencies oversample specific demographic groups. The survey will be weighted to be representative of the demographic characteristics of WIC participants at the State level (primarily race/ethnicity). The weighted survey results will be used to create statistical estimates of summary measures of participant satisfaction for each State agency. We will use a statistical model to examine the relationship between participant satisfaction and State agency modernization activities. Although the low response rate may allow for some response bias in the summary measures of participant satisfaction, the model results should be informative if the bias is not systematically related to modernization activities. We have no reason to believe that there would be a substantive, systematic relationship between any response bias and modernization activities. In addition, we will use the participant experience survey responses to identify areas of WIC participant experience for further exploration by State agencies. Specifically, State agencies will be encouraged to combine this information with other indicators of participant experience (such as participant and staff feedback) and explore opportunities to improve participant experience.
The State and local agency/clinic staff recruitment, invitation, and reminder materials (Appendices J, L.4, R.2, and R.3) for the WIC program staff experience survey (Appendix O) state that their participation in the study is mandatory. The study team plans to maximize participation by highlighting the web surveys’ low response burden of 10 minutes, the $10 incentive payment (for WIC & FMNP vendor/authorized outlet staff and WIC participants completing their experience surveys in English or Spanish (Appendices P, Q and Q.1)), and the flexibility of being able to complete the experience surveys at times convenient to respondents and across multiple sessions. Sending nonrespondents email reminders (in English or Spanish for WIC participants) (Appendices R.3, S.5, T.5, and T.5a) and text reminders (in English or Spanish for WIC participants) (Appendices S.6, T.6, and T.6a) increases the likelihood that individuals with different characteristics will complete their experience surveys, which decreases the risk of non-response bias.
Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.
Pre-test of the study’s data collection instruments. The study team pre-tested the WIC State agency staff interview protocol (Appendix F.1), the WIC local agency staff case study interview protocol (Appendix F.3), the WIC & FMNP vendor/outlet staff case study interview protocol (Appendix F.2), the WIC program staff experience survey (Appendix O), the WIC & FMNP vendor/outlet staff experience survey (Appendix P), and the WIC participant experience survey (Appendix Q) with 4 pretest volunteers for each instrument across four State agencies (Vermont, Michigan, Minnesota, and Mississippi) in June and July 2024. These pretests were meant to assess the length of time needed to complete each instrument and to determine the extent to which any changes were needed to the interview questions or survey items (refer to Appendix Z for a memo describing the pre-test and the findings).
Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
The following individuals from the USDA were involved in the design of this project:
Carol Dreibelbis, USDA, FNS, (612) 708-4268
Daniel Perez-Lopez, USDA, FNS, (703) 305-0913
Ruth Morgan, USDA, FNS, (703) 457-7759
Melissa Abelev, USDA, FNS, (703) 305-2209
Lindsay Drunasky, USDA, National Agricultural Statistics Service, (202) 690-8141
Mathematica will collect and analyze the information, in coordination with FNS. The following individuals consulted on the data collection instruments, procedures, or statistical aspects of the design:
Maria Boyle, Principal Researcher, Mathematica, (609) 297-4636
Debbie Reed, Senior Fellow, Mathematica, (510) 830-3702
Dallas Dotter, Lead Data Scientist, Mathematica, (510) 830-3719
Diana Gates, Senior Managing Consultant, Mathematica, (312) 260-2968
Lorrene Ritchie, Director, Nutrition Policy Institute, University of California, (510) 987-0523
Nicole Huret, Senior Research Associate, MEF Associates, (703) 278-2400
1This estimate of statistical power assumes (1) an outcome standard deviation of 0.5, and (2) a bundle of modernization implementations consisting of at least 10 percent of respondents and an appropriate comparison group of respondents within at least 50 State agencies, as well as an average of 10 local agencies per State agency. The analysis will be able to detect smaller differences in satisfaction percentages for program bundles of modernization projects implemented for larger percentages of the sample or across more agencies.
2 Lee, D.L., C. Felix, L. Ritchie, L. Kim, S. Whaley, and G. Machell. “2023 Multi-State WIC Participant Satisfaction Survey.” The National WIC Association, https://media.nwica.org, April 9, 2024
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Statement for OMB No |
| Author | USDA |
| File Modified | 0000-00-00 |
| File Created | 2025-06-13 |