SSB - WIC NATS Final

SUPPORTING STATEMENT - PART B for

OMB Control Number 0584-NEW

WIC Nutrition Assessment and Tailoring Study

November 23, 2020

Alexander Bush

Social Science Research Analyst

Office of Policy Support

USDA, Food and Nutrition Service

1320 Braddock Place

Alexandria, VA 22314

Email: [email protected]

Telephone: (703) 305-2127

Table of Contents

B.1 Respondent Universe and Sampling Methods 4

B.2 Procedures for the Collection of Information 12

B.3 Methods to Maximize Response Rates and to Deal With Nonresponse 14

B.4 Test of Procedures or Methods to be Undertaken 17

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 19

Tables

B1-1. Respondent Universe by Respondent Category 5

B1-2. FNS Regions and Respective State Agencies 6

B1-3. Strata for Selecting State Agencies 8

B1-4. The LA Selection Process 9

B1-5. Decision Criteria for Selection of Clinics 10

B2-1. Standard Error of the Estimates 14

B4-1. Pretested Instruments by Respondent 17

B5-1. Individuals Consulted on Data Collection or Analysis 21

Appendices

Appendix A: WIC NATS Supporting Documents

1. Authorizing Statutes

2. Relevant WIC Policy and Guidance Documents

3. WIC NATS Objectives and Research Questions

4. WIC NATS Recruitment Summary

Appendix B: WIC NATS OMB Burden Table

Appendix C: Information Collections and Instruments

1. Call with State Agency Director and Request Documents

2. Local Agency Director Survey

1. Local Agency Director Survey (Screenshots)

3. Call with Local Agency Director and Request Documents

4. Clinic Site Information Form

5. Clinic Observation Form

6. Site Director Interview Guide

7. Nutrition Services Observation Form

   1. Identified Risks Data Collection Form

8. Staff Interview Guide

9. WIC Participant Interview Guide

   1. WIC Participant Interview Guide-Spanish

Appendix D: State Agency Recruitment

1. Introductory Email from FNS to SA

2. Email to SA with Info Packet and Schedule Call

3. MOU for SA

Appendix E: Local Agency Director Survey Recruitment

1. Introductory Email from SA to all LAs with Info Packet

2. Email to LA with LA Director Survey Link

3. Reminder Emails to Complete LA Director Survey

Appendix F: Local Agencies Selected for Site Visits

F1. Email to SA with LAs Selected for Site Visit

F2. Site Visit Email from SA to Selected LA

F3. Site Visit Email to LA and Schedule Call

F4. Reminder Email to Submit Nutrition Risk Assessment Documents

Appendix G: WIC Clinic Site Visit and Participant Recruitment

1. Email to LA with Selected Site Name

2. Site Visit Email from LA to WIC Clinic

3. Site Visit Email to WIC Clinic and Schedule Call

4. Telephone Call with WIC Clinic

5. Informed Consent for Site Director Interview

6. Informed Consent for Observation and Staff Interview

7. Study Brochure for WIC Participant

G7a. Study Brochure for WIC Participant-Spanish

8. WIC Participant Screener

G8a. WIC Participant Screener-Spanish

9. Informed Consent for Observation and WIC Participant Interview

G9a. Informed Consent for Observation and WIC Participant Interview-Spanish

10. Reminder Text for WIC Participant Interview to be Conducted Over the Phone

G10a. Reminder Text for WIC Participant Interview to be Conducted Over the Phone-Spanish

11. Reminder Call for WIC Participant Interview to be Conducted Over the Phone

G11a. Reminder Call for WIC Participant Interview to be Conducted Over the Phone-Spanish

Appendix H: Comments

H1. Public Comment from National WIC Association

H2. Public Comment from 1,000 Days

H3. Public Comment from Academy of Nutrition and Dietetics

H4. Comments from NASS

H5. Expert Review Panel Comments and Responses

Appendix I: FNS Response to Comments

I1. FNS Response to National WIC Association

I2. FNS Response to 1,000 Days

I3. FNS Response to Academy of Nutrition and Dietetics

I4. FNS Response to NASS Comments

Appendix J: Pretest Memorandum

Appendix K: Confidentiality Agreements and IRB Documents

K1. Statement of Confidentiality and Non-Disclosure

K2. Westat IRB Approval Letter

B.1 Respondent Universe and Sampling Methods

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

B.1.1 Respondent Universe

The WIC Nutrition Assessment and Tailoring Study (WIC NATS) requires data collection from four respondent groups: (1) WIC State Agencies (SAs); (2) WIC Local Agencies (LAs); (3) WIC Clinics; and (4) WIC participants. We plan to select a diverse, non-probability national sample of SAs, LAs, WIC clinics, and WIC participants using a progressive sample selection approach. We will first select 10 SAs, then 30 LAs within those 10 SAs, and ultimately 1 clinic site per LA and approximately 17 WIC participants within those sites.

WIC is administered in 89 SAs – which includes all 50 States, 33 Indian Tribal Organizations (ITOs), the District of Columbia, and five territories (the Commonwealth of the Northern Mariana Islands, American Samoa, Guam, Puerto Rico, and the U.S. Virgin Islands). The universe of WIC SAs for this study includes all SAs expected to meet the following eligibility criteria: having a fully-operational Electronic Benefit Transfer (EBT) system for at least six months, not engaging in any major overhauls of their Management Information System (MIS) during the study data collection window, and located within the contiguous United States. Across these estimated 54 SAs, there is a universe of an estimated 1,533 LAs and an estimated 3,669 clinics associated with those LAs.¹ The universe of eligible WIC participants is estimated at 5,355,980 participants.² Table B1-1 summarizes the respondent universe and estimated response rates.

The Study Team will take many strategic steps to maximize response rates across respondent groups. This includes providing clear information about the study and flexible scheduling to all participants, using incentives for WIC participants, sending reminders to complete web-based surveys or phone interviews, and requesting assistance from FNS Regional Offices and WIC State agencies to encourage local participation. A full description of efforts to improve response rates can be found in Section B3 of this supporting statement.

Table B1-1. Respondent Universe by Respondent Category

Respondent Category	Size of Respondent Universe	Initial Sample1	Target completed cases	Response Rates
WIC SAs	54	13	10	77%
WIC LAs LA Survey Site Visits	1,533	370 36	306 30	83% 83%
WIC clinics2	3,669	36	30	83%
WIC participants Observation Interview	5,355,980	1,020 510	510 300	50% 59%
Overall Response Rate		1,439	856	59%

¹ Does not include pretest respondents.

² Initial sample for WIC clinics includes the approximately 36 clinics that will be contacted to successfully recruit 30 clinics into the study. It is expected that all 30 clinics will fully participate in all site director and staff interviews.

B.1.2 Sampling Methods

B.1.2.1 State Agencies

The study team will purposively select 10 SAs (and three alternates). Only if a contacted SA declines participation will the study team contact alternates. Given the high cooperation rate of SAs in similar, previously conducted studies³, it is expected that no more than 13 SAs will be contacted in order to obtain 10 willing to participate. Table B1-2 lists the expected frame of eligible SAs by FNS region, including 54 WIC SAs.

Table B1-2. FNS Regions and Respective State Agencies

FNS Region	State Agencies*
Mid-Atlantic Regional Office (MARO)	Delaware, Maryland, Pennsylvania, Virginia, West Virginia
Midwest Regional Office (MWRO)	Indiana, Iowa, Michigan, Minnesota, Ohio, Wisconsin
Mountain Plains Regional Office (MPRO)	Colorado, Kansas, Montana, Nebraska, South Dakota, Wyoming
Northeast Regional Office (NERO)	Connecticut, Massachusetts, New Hampshire, New York, Rhode Island, Vermont
Southeast Regional Office (SERO)	Alabama, Mississippi Band of Choctaw Indians, Eastern Band of Cherokee Indians, Florida, Kentucky, North Carolina, South Carolina, Tennessee
Southwest Regional Office (SWRO)	Arizona, Arkansas, Cherokee Nation of Oklahoma, Chickasaw Nation, Choctaw Nation of Oklahoma, Citizen Potawatomi Nation, Inter-Tribal Council of Arizona, Inter-Tribal Council of Oklahoma, Louisiana, Muscogee (Creek) Nation, New Mexico, Oklahoma, Osage Nation, Otoe-Missouria Tribe, Pueblo of Isleta, Texas, WCD Enterprises, Inc (Wichita, Caddo, and Delaware)
Western Regional Office (WRO)	Idaho, Inter-Tribal Council of Nevada, Navajo Nation, Nevada, Oregon, Washington

*Expected to be eligible at time of data collection

Using WIC participant caseload and State Plan data, the study team will stratify the eligible SAs by four stratification variables to capture the primary variation in WIC nutrition risk assessments at the SA level, while maintaining geographic diversity. The SA stratification variables are:

1. FNS region: One SA from each of the seven FNS regions plus an additional SA from the three regions with the largest WIC participant caseload using the most recent annual participation data available from the FNS’s WIC data tables: https://www.fns.usda.gov/pd/wic-program.

2. SA participation level (high or low): High and low participation is defined using the most recent annual participation data available from the FNS’s WIC data tables: https://www.fns.usda.gov/pd/wic-program. SAs above the median will be considered to be high participation level SAs and those below the median to be low participation level SAs.

3. Flexibility allowed in tailoring of food packages (high or low): High or low flexibility allowed in tailoring food packages, will be based on a review of State Plans to determine whether the SA allows LAs to develop specific individual food package tailoring guidelines (State Plan section IIB, question 2d). SAs that do not allow LAs to develop tailoring guidelines will be considered low flexibility SAs; SAs that do allow this will be considered high flexibility SAs.

4. MIS complexity (high or low): MIS complexity will also be determined from State Plan data. In section IIIC of the State Plan, SAs use a checklist of 25 WIC systems functional requirements and indicate which functions the SA system currently performs. High complexity/low complexity will reflect the number of items checked based on a “cutoff” between high and low complexity systems. SAs with the most functionality in their MIS will be considered high MIS complexity SAs.

To systematically select a diverse set of 10 SAs, the study team will perform the following steps:

Step 1: Place all eligible SAs into one of 8 stratum based on participation level (high/low), tailoring flexibility (high/low) and MIS complexity (high/low). The 8 strata are shown in Table B1-3.

Step 2: Order the strata by the number of SAs in each, lowest to highest.

Step 3: In the first stratum, randomly select one SA.

Step 4: If the selected SA’s region is a region from which we will select only one SA, remove the remaining SAs in that region from the sampling frame.

Step 5: In the next stratum, randomly select one SA.

Step 6: Remove the remaining SAs in that region if the region has met its desired quota.

Step 7: Continue until we have selected a SA from each strata. (We will have selected up to 8 SAs at this point.)

Step 8: Re-order strata that still have SAs in them by the number of SAs in each, lowest to highest.

Step 9: Continue selecting from each stratum until we have selected 10 SAs.

Table B1-3. Strata for Selecting State Agencies

Strata	Enrollment	Tailoring	MIS
1	Low	Low	Low
2	Low	Low	High
3	Low	High	Low
4	Low	High	High
5	High	Low	Low
6	High	Low	High
7	High	High	Low
8	High	High	High

B.1.2.2 Local Agencies and WIC Clinics

In order to capture important variation in the implementation of the nutrition risk assessment at the LA level, we will ask all LAs within each of the 10 participating SAs to complete an on-line survey (Appendix C2 and C2a) about their nutrition risk assessment process. Survey responses will inform the selection of 30 LAs that represent the variability observed based on the following key characteristics:

1. LA size (caseload), constructed from WIC Participant and Program Characteristics 2018 data, defined as large, medium, or small.

2. Urbanicity (from the survey), defined as primarily rural, primarily urban, and primarily suburban.

3. Language use (from the survey). Classified into two categories:

   1. Greater than 10% of participants are Spanish speakers (yes/no)

   2. Greater than 10% of participants speak an “other” language (yes/no)

4. Modes of nutrition education offered (from the survey), defined as offering more than X modes (yes/no), with X determined based on the distribution of responses to the associated survey question.

5. Use of technology for nutrition education (from the survey), (yes/no).

We will target a final sample of 30 LAs to participate in the site visit phase of the study, and expect we will need to approach 36 LAs before successfully recruiting 30 into the sample. The study team will take the following steps to systematically select the sample of LAs:

Step 1: For each of the five characteristics listed above (size, urbanicity, language use, modes of nutrition education, and technology use), the study team will determine the number of LAs with the characteristic. If each characteristic is a bucket with one or more LAs in it, then, given the list of characteristics, LAs can be in up to 6 buckets.

Step 2: Sort the characteristics by number of LAs with each from lowest to highest.

Step 3: Divide the characteristics into quartiles based on number of LAs with each characteristic.

Step 4: Randomly select LAs from each characteristic as follows:

• From the characteristics with the 1st quartile (those having the smallest number of LAs) randomly select 1 LA from each.

• From the characteristics in the 2nd quartile, randomly select 2 LAs from each.

• From the characteristics in the 3rd quartile, randomly select 3 LAs from each.

• From the characteristics in the 4th quartile, randomly select 4 LAs from each.

Table B.1-4 shows the total number of LAs selected from each quartile.

Table B1-4. The LA Selection Process

Quartile	Characteristics in Each	Number of LAs Selected from Each Characteristic	Total Number of LAs Selected
1st	3	1	3
2nd	3	2	6
3rd	3	3	9
4th	3	4	12
Total			30

Selected LAs that agree to participate will be asked to provide the following information in an Excel spreadsheet (Appendix C4) about each WIC clinic site operated by the LA:

• Monthly caseload for reference month

• Zip code

• Language use

• Percent of high risk participants

• Percent of participants with only one documented risk

• Days of the week usually open for new enrollments and recertification appointments 

From this list, one clinic per LA will be selected for a site visit. To ensure that the study team will be able to observe up to 17 nutrition risk assessments at each clinic site, only sites open at least three or more consecutive days a week will be eligible for selection. Selection of clinic sites will be based on similarity of their characteristics with those of their LA (i.e., for LAs with large caseloads, the clinic with the largest monthly caseload will be recruited). This ensures we maintain the characteristics for which we selected the LA as we select clinic sites.

Table B1-5 shows the decision criteria for the purposive selection of clinics with the LA selection based on size, urbanicity, and language use.

Table B1-5. Decision Criteria for Selection of Clinics

LA Selection Criteria	WIC Clinic Selected
Large caseload	Site with largest caseload in the LA
Medium caseload	Site with caseload closest to the median caseload across the LA
Small caseload	Site with smallest caseload in the LA
Primarily urban	Randomly selected urban site (if more than one site is in an urban area)
Primarily rural	Randomly selected rural site (if more than one site is in a rural area)
Primarily suburban	Randomly selected suburban site (if more than one site is in a suburban area)
Greater than 10% of participants are Spanish speakers	Site with the largest percentage of Spanish speakers
Greater than 10% of participants speak an “other” language	Site with the largest percentage of “other language” speakers

For LAs selected based on modes of education and use of technology in providing nutrition education, all their respective clinic sites are assumed to have the same characteristics. Therefore, within these LAs, clinics will be selected with varying percentages of high-risk participants and participants with only one documented risk. We will sort clinics by percentage of high-risk participants and will select those at the high and low end for inclusion. We will sort clinics in the remaining LAs by the percentage of participants with only one documented risk and will select those at the high and low end for inclusion. If a selected clinic cannot participate in the study, another clinic with similar characteristics will be selected.

Given an expected response rate of 83 percent⁴, an estimated 36 clinics will be approached (Appendix G2, G3) to successfully recruit 30 that will agree to participate in the study.

B.1.2.3 WIC Participants

During the site visits, the field researchers will screen (Appendix G8, G8a) an estimated 1,020 WIC participants in order to recruit approximately 510 WIC participants (about 17 per site) to provide informed consent (Appendix G9, G9a) and participate in the observation (estimated 50 percent participation rate).⁵ Upon arrival at each clinic, field researchers will review the WIC clinic schedule for the week of the site visit with the site director. To ensure that a diverse sample of participants, and nutrition risk assessments, are observed, the team will aim to recruit participants across appointment types (certification and recertification), and participant categories (pregnant women, postpartum women, infants, and children).

All observed participants will be asked to complete the follow-up interview; approximately 59 percent are estimated to consent.⁵ Thus, the study team expects to complete interviews with 300 of the 510 WIC participants whose nutrition assessments were observed. WIC participants will be given the option of either completing their interview at the clinic following their appointment or scheduling a later date to complete the interview over the phone. We expect approximately 153 participants will complete the interview at the clinic, while approximately 20 percent (102) decline to participate in the interview at the outset. We expect 255 participants will schedule a phone interview, but only anticipate 147 of those will complete the phone interview. Therefore, of 300 total completed interviews, we expect to complete 153 at the clinic and 147 by phone.

B.2 Procedures for the Collection of Information

Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

• Estimation procedure,

• Degree of accuracy needed for the purpose described in the justification,

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Information on how information will be collected is described in Section A.2 and in Appendix A4 of this request.

B.2.1 Statistical Methodology for Stratification and Sample Selection, Estimation Procedure, and Degree of Accuracy Needed for the Purpose Described in the Justification

As discussed in Section B.1, we plan to select a diverse, non-probability national sample of SAs, LAs, WIC clinics, and WIC participants. We will collect quantitative and qualitative information through the online survey with LA directors, the observations of clinic environment and nutrition services visits, and in-depth interviews with WIC clinic staff and WIC participants. Because we are collecting data from a purposive sample of each respondent group, and because the results will not be weighted to achieve national representation, no statistical methodology for stratification and sample selection or estimation procedures are warranted. We will use the following methods to ensure that our selection of SAs, LAs, and WIC clinics achieves diversity across the characteristics described in section B.1.

B.2.1.1 Selecting State Agencies

We expect a response rate of 76.9 percent for SAs. To select a diverse set of 10 SAs, we will stratify the sample by FNS region, participation, flexibility in food package tailoring, and MIS complexity. The selection process prioritizes region over the other stratification variables, ensuring selection of at least one SA from each FNS region plus an additional SA from the three regions with the largest WIC participant caseload aggregated across eligible SAs (using the latest available WIC participant data at the time of selection from WIC Data Tables page on the FNS website (https://www.fns.usda.gov/pd/wic-program).

B.2.1.2 Selecting Local Agencies and WIC Clinics

We expect an 83 percent response rate from LAs and WIC clinics. To select a diverse set of 30 LAs for site visits and ensure broad representation, a stratified random sample will be selected based on the following characteristics: size, urbanicity, language use, modes of nutrition education, and technology use. The selection process will ensure at least one and no more than three LAs are selected from every sampled SA. If an LA is unable to participate in the study for whatever reason, we will select an LA with similar characteristics from the same SA. As discussed in section B.1.2.2, we will use the spreadsheet of clinic information (Appendix C4) provided by the LA director to select one clinic within the LA for a site visit. WIC staff and WIC participants at this site will be recruited during the site visit. No sampling will be involved, as we are limited to recruiting WIC participants scheduled for certification or recertification visits during the site visit. Once a WIC participant agrees to participate in the study (Appendix G9, G9a), we will recruit the staff member conducting the assessment and/or benefit tailoring to participate in the study (Appendix G6).

We expect that 50 percent of the WIC participants approached will consent to participate in the observation, and that 59 percent of those who allow us to observe their appointment will complete the follow-up interview.

Table B.2-1 below shows the expected standard errors of the estimates for various levels of the proportion estimates (p) and sample sizes. These expected standard errors account for the nested design (i.e., the sampling of WIC participants within sites) and assume a small intraclass (within-site) correlation of 0.01⁶. With 510 appointment observations and 300 participant interviews, the expected standard errors range from about 0.014 to 0.030. For example, for a proportion estimate of 90 percent and a sample size of 300, a 95 percent confidence interval is between 86.0 and 93.2 percent.

Table B2-1. Standard Error of the Estimates

Proportion Estimate (p)	Sample sizes
	300	450	600
0.1 (or 0.90)	0.018	0.015	0.013
0.2 (or 0.80)	0.024	0.020	0.018
0.3 (or 0.70)	0.030	0.025	0.022

B.2.2 Unusual Problems Requiring Specialized Sampling Procedures

There are no unusual problems that require specialized sampling procedures.

B.2.3 Use of Periodic Data Collection Cycles to Reduce Burden

This is a one-time study; concern regarding the periodicity of data collection cycles is not applicable.

B.3 Methods to Maximize Response Rates and to Deal With Nonresponse

Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

As mentioned in previous sections, this study will select a diverse, non-probability national sample of SAs, LAs, WIC clinics, and WIC participants. The sample is not intended to produce nationally representative findings, but instead to represent key variations described in the explanation of the sampling criteria given in Section B.2.

The expected response rate for SAs is 77 percent. Advanced notification from FNS and the Regional Office Special Nutrition Programs Directors will help bolster participation by SA directors (Appendix D1). This is reflected in the initial sampling of 13 SAs, of which 10 SAs are ultimately expected to participate.

When recruiting LAs, the support of the SA directors to encourage their participation is expected to promote response. Specifically, the email sent to SA directors informing them of the study (Appendix D2) will include a template email for SA directors to send to all of their LAs encouraging their participation (Appendix E1). LA directors will be recruited through an email (Appendix E2) that includes a link to the on-line LA survey (Appendix C2, C2a); non-respondents will be sent two email reminders (Appendix E3). Of the approximately 370 LAs invited to complete the LA survey, we expect a response rate of 83 percent. Once we select the 30 LAs for the site visit phase, SA directors will be contacted via email informing them of the selection (Appendix F1). A template email for the SA directors to send to the selected LAs, informing them of the site visit plans and encouraging them to continue their participation, will also be provided (Appendix F2). We anticipate 83 percent of selected LAs will agree to participate in the site visit phase of this study, therefore we anticipate contacting no more than 36 LAs to recruit 30 for site visits.

One WIC clinic from each of the 30 selected LAs will be selected for the site visits. While some clinics may be unable or unwilling to participate, we anticipate contacting no more than 36 WIC clinics in order to enroll 30 for the site visits, based on an estimated response rate of 83 percent. To maximize clinic participation, we will first notify their respective LA with the name of the clinic selected for the study (Appendix G1). In this email to the LA, we will attach a template email that LAs may send to the clinic, informing the clinic of the study and encouraging their participation (Appendix G2). In addition, we will contact each selected clinic (Appendix G3) to schedule an informational call (Appendix G4). During this call, we will provide the clinic with an overview of the study and the data collection procedures, and allow the site to ask any questions they may have about the study. We will also explain during the call that participating WIC clinics will be offered a gift (valued at $40) of children’s books and games for their waiting area as a thank you for time spent accommodating the study’s activities.

Across all site visits with participating WIC clinics, trained field data collectors in teams of two will screen 1,020 WIC participants by administering a screener (Appendix G8, G8a) and sharing the study brochure (Appendix G7, G7a). Given an expected response rate of 50 percent, we will identify 510 participants who consent (Appendix G9, G9a) to having the data collectors observe their nutrition risk assessment and benefit tailoring appointment. We believe the following steps will be sufficient to reach response rate targets by giving adequate information about the study, offering incentives to encourage participation, and allowing for flexibility of when and how the interview is conducted to meet participant schedule restraints. In order to ensure the study is accurately conveyed to participants in recruitment, field researchers will be trained in the details of the study as a whole and will be able to respond to participants’ questions or concerns. The study team will offer a $20 incentive to WIC participants that both allow the observation (Appendix C7) and complete the follow-up interview (Appendix C9, C9a). For flexibility, participants will have the option of completing the interview via telephone, should they be unable to remain at the WIC clinic long enough to complete the interview in-person. Participants who elect to complete the survey by phone will be provided with a disposable cell phone pre-loaded with enough minutes to complete the interview, should they not have access to a telephone for the call. For all scheduled telephone interviews, we will send the WIC participant a reminder by text message (Appendix G10, G10a) and, if needed, a reminder phone call (Appendix G11, G11a). We will attempt to complete the interview within one week following the observation. We expect to complete phone interviews with 147 WIC participants which, combined with the estimated 153 interviews completed in person, are expected to result in 300 completed interviews out of the 510 WIC participants who allow their nutrition assessment appointment to be observed during the site visit (response rate of 59 percent).

B.4 Test of Procedures or Methods to be Undertaken

Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

All data collection instruments were pretested (Appendix J). The pretest involved a total of 9 individuals (3 LA directors, 2 clinic directors, 2 clinic staff members and 2 WIC participants), as seen in Table B4-1.

Table B4-1. Pretested Instruments by Respondent

Respondent Number	Pretest Respondent	Instruments Pretested by Respondent
1	LA director (State #1)	LA Director Survey (Appendix C2) and cognitive interview Clinic Site Information Form (Appendix C4)
2	LA director (State #2)	LA Director Survey (Appendix C2) and cognitive interview Clinic Site Information Form (Appendix C4)
3	LA director (State #3)	LA Director Survey (Appendix C2) and cognitive interview Clinic Site Information Form (Appendix C4)
4	Clinic director (State #1)	Site Director Interview Guide (Appendix C6)
5	Clinic director (State #2)	Site Director Interview Guide (Appendix C6)
6	WIC clinic staff member (State #1)	Staff Interview Guide (Appendix C8) Nutrition Services Observation Form (Appendix C7)
7	WIC clinic staff member (State #2)	Staff Interview Guide (Appendix C8) Nutrition Services Observation Form (Appendix C7)
8	WIC participant (State #1)	WIC Participant Interview Guide (Appendix C9)
9	WIC participant (State #2)	WIC Participant Interview Guide (Appendix C9)

To minimize cost, FNS suggested three different SAs for the pretest that were all located near the Washington, D.C. region, in consultation with the FNS Mid-Atlantic Regional Office. Three LA directors, each from different states, pretested the LA Director Survey (Appendix C2). LAs were selected to obtain input from agencies operating within different SAs. The LAs were suggested by their respective SAs. The LA directors completed a paper version of the LA Director Survey followed by a telephone cognitive interview. The three LA directors also completed and provided input on the Clinic Site Information Form (Appendix C4).

Two WIC clinics were nominated by their respective LAs and selected to pretest the remaining data collection instruments. At on-site visits with each of these clinics, the site visit staff pretested the Site Director Interview Guide (Appendix C6) with the clinic director. The site visit staff also pretested the Clinic Observation Form (Appendix C5), which did not require a respondent. The site visit also included pretesting of the Staff Interview Guide (Appendix C8) and Nutrition Services Observation Form (Appendix C7) with two staff members who conduct nutrition risk assessment and benefit tailoring activities. Additionally, during the site visits the WIC Participant Interview Guide (Appendix C9) was pretested with two WIC participants, one from each pretest clinic, who were present for certification and recertification visits. The WIC participants were recruited at random during the pretest site visit.

The LA Survey, Site Director, Staff, and WIC Participant Interview Guides were tested to ensure that the respondents interpreted the questions as intended and could easily respond, that the interview guide was easy for the interviewer to administer, and to verify the burden estimates. Trained interviewers reviewed the instruments question by question with the respondents, observed and documented any issues that arose for either respondents or interviewers, and discussed any points of difficulty with respondents. The Clinic Observation Guide and Nutrition Services Observation Form were tested to ensure that observers can complete the forms as intended and were not missing any key elements of the visits and clinic flow.

Following the pretests, analysts reviewed notes from each interview and observation, and summarized themes and patterns within the data. In particular, analysts focused on problems with the instruments, including areas where the respondents demonstrated confusion, hesitation, or uncertainty. Themes and patterns were organized, evaluated, synthesized, and summarized into the WIC NATS Pretest Memorandum (Appendix J). The attached WIC NATS Pretest Memorandum also describes specific comments and recommendations from the pretest respondents as well as how these were used to refine the survey, interview guides, and observation forms. The pretest verified the initial burden estimates for the survey and interviews.

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Table B5-1 presents a summary of individuals consulted on instrument design, data collection, and/or analysis. The protocol and instruments were developed and reviewed extensively by FNS and the Westat study team—which includes researchers that Westat sub-contracted with from Insight Policy Research, PHFE WIC, and Gabor and Associates. Westat, Insight Policy Research, PHFE WIC, and Gabor and Associates will also be responsible for the collection and analysis of the study’s data, in coordination with FNS. The three members of the Expert Review Panel reviewed the data collection procedures and the data analysis plan. The statistical procedures included in this information request have also been reviewed by Anthony Fischer with the USDA National Agricultual Statistics Service (NASS). Comments from NASS are included in Appendix H4, and responses are incorporated in this supporting statement and listed in Appendix I4.

Table B5-1. Individuals Consulted on Data Collection or Analysis

Staff	Title	Contact Information (phone or email)
Westat (contractor)
Laurie May, PhD	Vice President	301-517-4076
Mary Gabay, MS	Senior Study Director	301-294-2811
Christine Borger, PhD	Senior Study Director	301-294-2072
Thea Zimmerman, MS, RD	Senior Study Director	240-314-2413
Consultants and Sub-Contractors
Vivian Gabor, MPH	Principal, Gabor & Associates	202-841-3071
Shannon Whaley, PhD	Director of Research and Evaluation, PHFE WIC	626-233-0798
Betsy Thorn	Associate Director, Insight Policy Research	703-504-9488
Expert Review Panel
Joni Geppert, MPH, RD, LN	Senior Epidemiologist Minnesota Department of Health	651-201-3632
Angela Odoms-Young, PhD	Associate Professor, Kinesiology and Nutrition University of Illinois at Chicago	773-391-3358
Sarah Jane Schwarzenberg, MD	Professor, Department of Pediatrics University of Minnesota Masonic Children’s Hospital	612-624-1133
FNS Staff
Alexander Bush, MPH Courtney Paolicelli, DrPH	Social Science Research Analyst Lead Social Science Analyst	703-305-2127 703-605-4370
NASS Staff
Anthony Fischer	Mathematical Statistician, Methodology Division	202-720-0791

1 WICprograms.org, WIC Program and Office Directory. https://www.wicprograms.org/; accessed January 23, 2020.

2 U.S. Department of Agriculture, Food and Nutrition Service. WIC Data Tables, 2018 National level annual summary. https://www.fns.usda.gov/pd/wic-program; accessed January 23, 2020.

3 Anticipated response rate, based on experience from studies involving similar entities and burden.

4 Anticipated response rate, based on past experience from studies involving similar entities and burden.

5 Based on response rates seen in the WIC Infant and Toddler Feeding Practices Study-2: Infant Report (OMB Control No: 0584-0580; Expiration Date: 03/31/2022), available at https://fns-prod.azureedge.net/sites/default/files/ops/WIC-ITFPS2-Infant.pdf

6We assume a small intraclass correlation based on work previously conducted for the WIC program by Westat for the WIC Infant Toddler Feeding Practices Study 2 (OMB Control No: 0584-0580; Expiration Date: 03/31/2022). In that study, intraclass correlation estimates of 0.01 were calculated for the variables WC21 (which asks if the mother made a change due to something she learned at WIC, a question asked at 3, 13, and 24 months) and I7_KA28 (which asked if WIC allows the mother to buy the types of food she would usually feed her child).