QRS WIC Child Retention Memo

Memorandum

Date: November 13, 2019

To: James Crowe, OMB Desk Officer, Office of Information and Regulatory Affairs, Office of Management and Budget

Through: Ruth Brown, Desk Officer, U.S. Department of Agriculture, Office of the Chief Information Officer

From: Kelly Stewart, Chief, Planning & Regulatory Affairs Office, Food and Nutrition Service

Re: Under Approved Generic OMB Clearance No. 0584-0613 – Special Nutrition Programs Quick Response Surveys

The U.S. Department of Agriculture (USDA) Food and Nutrition Service (FNS) is requesting approval to conduct research under Approved Generic Office of Management and Budget (OMB) Clearance Number 0584-0613, Special Nutrition Programs (SNP) Quick Response Surveys (QRS), expiration date 02/28/2021.

This request is to acquire clearance to conduct a survey of clinic staff from a nationally representative sample of Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) Local Agencies. The study will examine reasons given when child participants leave the WIC Program and identify clinic-level staff recommendations for improving child retention. The following information is provided for your review:

1. Title of the Project: WIC Child Retention

2. Control Number: 0584-0613, Expires 02/28/2021

3. Public Affected by this Project:

State, Local, and Tribal Government

• WIC State Agencies

• WIC Local Agencies

• WIC Clinic Staff

Business or Other For-Profit and Nonprofit Institutions

• WIC Local Agencies

• WIC Clinic Staff

4. Number of Respondents and Research Activities:

Exhibit 1 below outlines the respondent types, research activities, and associated number of study participants.

Exhibit 1 | Assumptions on Total Number of Respondents

Respondent Group	Respondents	Research Activity	Number of Participants
State, Local, and Tribal Governments	WIC State Agencies	Email Notifications to WIC State Agency	90
	WIC Local Agencies	Pretest - Local Agencies	5
		Email Request to Local Agency	490
		Reminder Emails to Local Agency	392
		Reminder Phone Calls to Local Agency	314
	WIC Clinic Staff	Pretest - Clinic Staff	2
		Initial Survey Notification Email to Clinic-Level Staff Member	490
		Survey Reminder Emails to Clinic-Level Staff Member	343
		Survey Reminder Phone Calls to Clinic-Level Staff Member	240
		Survey - WIC Child Retention	490
		Post-Survey Response Clarification Phone Call	10
		Post-Survey Response Clarification Email	49
Business or Other For-Profit and Nonprofit Institutions	WIC Local Agencies	Pretest - Local Agencies	0
		Email Request to Local Agency	210
		Reminder Emails to Local Agency	168
		Reminder Phone Calls to Local Agency	134
	WIC Clinic Staff	Pretest - Clinic Staff	0
		Initial Survey Notification Email to Clinic-Level Staff Member	210
		Survey Reminder Emails to Clinic-Level Staff Member	168
		Survey Reminder Phone Calls to Clinic-Level Staff Member	118
		Survey - WIC Child Retention	210
		Post-Survey Response Clarification Phone Call	4
		Post-Survey Response Clarification Email	21
		TOTAL*	1497

The total unique entities contacted includes 90 State agencies for the main study, 5 Local agencies for pretest, 2 WIC clinic-level staff for pretest, 700 Local agencies for the main study, and 700 WIC clinic-level staff for the main study.

As reflected in Exhibit 1, approximately 30% of WIC LAs are operated by businesses or other non-governmental organizations (including private hospitals and community service organizations). For this study, the survey and respondent requirements will be the same for both government-run and non-government WIC LAs and the respondent in both cases will be a clinic-level staff member who is selected by the LA director based upon a set of defined criteria.

5. Time Needed Per Response:

Exhibit 2 shows the estimated time needed for the pretest, survey, and email notification/recruitment and follow-up reminders. The estimated times for the email and follow-up reminders are the same for all respondent types.

Exhibit 2 | Time Needed for the Pretest and Main Study Research Activities

Research Activity	Respondent	Time (minutes)	Time (hours)
Pretest
Pretest	WIC Local Agenciesa	17.5	0.29
Pretest	WIC Clinic Staffa	200.5	3.34
Main Study
Email Notifications to WIC State Agency	WIC State Agenciesb	21.5	0.36
Email Request to WIC Local Agency	WIC Local Agenciesa	17.5	0.29
Reminder Emails to WIC Local Agency	WIC Local Agenciesa	12	0.20
Reminder Phone Calls to WIC Local Agency	WIC Local Agenciesa	9.5	0.16
Initial Survey Notification Email to Clinic-Level Staff Member	WIC Local Agenciesa	11.4	0.19
Survey Reminder Emails to Clinic-Level Staff Member	WIC Clinic Staffa	12	0.20
Survey Reminder Phone Calls to Clinic-Level Staff Member	WIC Clinic Staffa	9.5	0.16
Survey - WIC Child Retention	WIC Clinic Staffa	20	0.33
Post-Survey Response Clarification Phone Call	WIC Clinic Staffa	7	0.12
Post-Survey Response Clarification Email	WIC Clinic Staffa	5	0.08

^a Includes entities that are both State, Local, and Tribal Governments, and Business or Other For-Profit and Nonprofit Institutions.

^b All State agencies are considered State, Local, and Tribal Governments.

Total Burden Hours on Public:

The estimated total burden in terms of hours is 1139.54 hours. The complete burden table is enclosed as Appendix A.1.

6. Project Purpose, Methodology, and Research Design:

Background

The Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) is a key part of the nation’s public health infrastructure, providing healthy foods, nutrition education, and referrals to health and social services for pregnant and postpartum women, infants, and children up to age 5 years. People applying for WIC (or their parents or guardians) must be residents of the state or territory; have family incomes less than or equal to 185 percent of the federal poverty level, or be enrolled in Medicaid; and have at least one nutritional risk. WIC provides supplemental food packages that are appropriate to the nutritional needs of eligible participants.

USDA FNS administers the WIC program at the Federal level. At the state level, WIC is administered by 90 State Agencies, including 50 States and the District of Columbia; 34 Inter-Tribal Organizations; and the 5 U.S. Territories of American Samoa, Guam, the Commonwealth of the Northern Mariana Islands, Puerto Rico, and the U.S. Virgin Islands. State Agencies contract with WIC Local Agencies to provide WIC services to participants through clinic sites staffed by professional and paraprofessional personnel.

From 2010 to 2016, average monthly participation in WIC decreased by approximately 1.5 million participants, with the most dramatic decrease observed around age 1 year for eligible child participants.¹ Additional information is needed to understand the key drivers of discontinued program participation and what policies and procedures at the State and local levels could potentially address the retention challenge. Gathering input from clinic staff who have firsthand experience conducting child certifications and no-show follow-ups is critical to better understanding the barriers to child participation in the program.

Purpose

The purpose of this study is to examine why child participants leave the WIC Program and to identify staff recommendations for improving child retention. This study will address the following three research objectives:

Objective 1: Identify and describe the reasons why parents, guardians, or caretakers of eligible child participants discontinue receiving WIC benefits before the child turns 5 years old.

Objective 2: Describe how WIC Local Agencies operationalize certification, scheduling, and other service-related policies and procedures that may promote continued participation among children ages 1 through 4 years.

Objective 3: Identify WIC Local Agency and clinic staff recommendations for programmatic policies and procedures in WIC service delivery that could promote continued participation among children ages 1 through 4 years.

The study will help FNS better understand the reasons why eligible child participants leave the WIC program and gather recommendations on improving child retention from WIC clinic staff. This data may inform FNS guidance related to policies and procedures for improving WIC child retention.

Methodology/Research Design

The Study Team will address the study’s research objectives by conducting a web-based survey with WIC clinic staff who provide direct services to WIC participants. The clinic staff respondents are affiliated with WIC Local Agencies nationwide. The web-based survey is designed to focus on identifying reasons child participants leave the program; current services and practices at the clinic level to retain children; and staff suggestions to help retain children in WIC.

Prior to the main study, the Study Team will pretest the study procedures and survey with a small sample of Local agencies and WIC clinic staff (Appendices A.2, A.3, and A.15). Section 7 describes pretesting procedures. After reviewing pretest results, the Study Team will finalize the study materials.

Upon OMB approval, FNS Office of Policy Support will notify the FNS Regional Offices about the study. The FNS Regional Offices and the Study Team will notify State Agencies about the study via email and ask for their support (Appendix A.4). Following State agency outreach, the Study Team will contact selected Local Agency directors (Appendix A.5) via email and invite them to nominate one clinic-level staff member per Local agency to participate in the survey. In the event that a Local Agency does not respond to the initial email outreach, the Study Team will send follow-up emails (Appendix A.6) and phone calls (Appendix A.7).

Once a Local Agency director identifies a clinic-level staff member to participate in the survey, the Study Team will email the staff member with an invitation (Appendix A.8) to complete the web-based Survey of WIC Child Retention (Appendix A.9 and A.14). In the event that a clinic staff member does not respond to the initial email invitation, the Study Team will send follow-up emails (Appendix A.10) and attempt to contact the staff member via phone (Appendix A.11).

Once the clinic staff member has completed the Survey of WIC Child Retention, the Study Team will review each respondent’s answers. If any of the responses are unclear, the Study Team will attempt clarify the answers by contacting the staff member via telephone (Appendix A.12) or email (Appendix A.13).

Once OMB approval is received, all data collection will occur over a 4-week period.

Sampling Procedures

The Study Team will select a nationally representative, stratified, systematic, random sample of 700 Local Agencies, for which strata are defined according to the seven FNS Regions. Prior to sampling, Local Agencies will be sorted within FNS Regions by ITO status (Yes/No), Local Agency caseload size², and EBT status (Planning, Implementing, Pilot/Rollout, Statewide Implementation). This approach ensures that enough Local Agencies are selected for each FNS Region (to ensure national representativeness and precision of estimates) and that a balanced number of Local Agencies based on ITO status, LA caseload, and EBT status are represented in the sample.

In each selected Local Agency, the Local Agency director will identify one clinic-level staff member to participate in the survey. An 80-percent response rate, designated by USDA/FNS, requires an analytical sample of approximately 560 clinic-level respondents (based on an original sample size of 700 Survey respondents). This satisfies the required precision targets—namely, a 5-percent level of precision with a 95-percent level of confidence for national estimates, and a 7- to 10-percent level of precision with 90-percent confidence level for subgroup estimates.

Recruitment and Consent

Recruitment for WEB Survey

In order to recruit WIC clinic-level staff for the Survey of WIC Child Retention, the Study Team will complete the following steps:

1. Notify WIC State Agencies with information regarding the study (Appendix A.4);

2. Notify WIC Local Agencies with information regarding the study, and ask Local Agencies to identify a qualified clinic-level staff member to complete the Survey of WIC Child Retention (Appendix A.5); and

3. Contact WIC clinic-level staff with an invitation to complete the web-based survey (Appendix A.8).

Data Collection Activities

Exhibit 3 details the timeline of data collection activities. The schedule allows a total of 4 weeks for data collection to reach the target number of completed surveys (N = 560). The Study Team will use reminder emails and phone calls to encourage participation.

Exhibit 3 | Schedule of Main Study Data Collection Activities

Activities	Pre- Data Collect-ion	Data Collection
		Week 1						Week 2					Week 3						Week 4
		M	T	W	R	F	M		T	W	R	F		M	T	W	R	F		M	T	W	R	F
Email notification to WIC State Agency (Appendix A.4)	X
Email request to Local agency (Appendix A.5)	X
Initial survey notification email to WIC clinic staff (Appendix A.8)	X	X	X
Survey reminder emails (Appendix A.10)					X					X					X					X
Survey reminder phone calls (Appendix A.11)							X		X	X	X	X		X	X	X	X	X		X	X	X	X	X

Survey support staff will immediately address any technical issues with the survey and will respond to substantive questions within a few hours. The Study Team will provide in-depth training to the survey support staff. The Study Team will review the web responses and identify any responses that require follow-up. These responses will be passed to the survey support staff to contact the corresponding respondents to clarify incomplete or unclear responses.

The Study Team will submit a weekly data collection memo to FNS, and the memo will summarize survey response rates (e.g., number of completed surveys, surveys in progress, or surveys not started/accessed/returned), recruitment and follow-up activities, and any challenges.

COMPENSATION

There will be no compensation given to the respondents in the main study. Due to the length of time needed to complete the pretest, pretest respondents will receive a $30 honorarium in the form of a Visa gift card. These honoraria are appropriate for these professionals as they are being requested to respond to the information collection in addition to their regular duties as administrators/managers and direct services staff. Given the level of demand already on their time, the proposed honoraria are important to ensure the respondents are compensated for the additional time to participate in the pretest portion of the study. 

Data Preparation, WEIGHTING, and Analysis

Preparation

Upon completion of data collection, the Study Team will review the survey data for inconsistent data and reporting errors.³ After reviewing the data, the dataset will be considered clean and ready for post-survey adjustments (nonresponse adjustments).

Weighting and Nonresponse Adjustment

The Study Team’s statisticians will weight the survey data to adjust for unit nonresponse, using a propensity modeling procedure to predict the probability of responding to the survey based on the available characteristics data collected on the sampling frame file. Under this approach, the responding cases will be weighted by the inverse of the predicted probability of response, using a weighting class methodology that divides the propensity scores into classes and assigns the average score within the class to each case. This approach helps eliminate large adjustments to the survey weights to increase the survey precision in the estimates.⁴

The Study Team will adjust for nonresponse bias according to the characteristics available if an 80-percent response rate is not achieved. To conduct this analysis, the Study Team will carry out the following steps:

1. Code sampled records as respondents or nonrespondents

2. Use a logistic regression model to identify subgroups that are significantly different between respondents and nonrespondents

3. Report model results and potential nonresponse bias to FNS

4. Use a jackknife variance replication method to simplify the statistical significance testing of the descriptive statistics and regression-adjusted estimates⁵

This methodology also will help account for the nonsampling errors associated with the nonresponse adjustment and any subsequent post-stratification or calibration.

Analysis

Descriptive findings from the survey will be presented in data tables with weighted percentages. Data tables will include weighted and unweighted counts of respondents. The Study Team will provide descriptive cross-tabulations by clinic-level characteristics, such as caseload size for survey questions, when appropriate.

Outcomes/Findings

The findings from this study will help FNS better understand why eligible child participants leave the WIC program and will identify program and policy initiatives that may improve child retention. By providing information on how to retain more children in the program, the results of this study will ultimately be used to enable FNS to improve the WIC program to ensure that a greater number of eligible children receive the nutrition support that they need for a healthy start.

Cognitive Pretesting

The Study Team pretested the study procedures with four WIC Local Agencies, and they pretested the survey with five WIC clinic staff. For the Local Agency pretest, the Study Team queried Local Agencies to examine how well they understood the WIC clinic-level staff selection criteria and why they chose the selected clinic respondent for the study (Appendix A.2). For the WIC clinic staff pretest, the Study Team had the staff member complete the survey and its associated cognitive interview (Appendix A.3). The cognitive interview questions focused on considerations such as whether respondents understood the meaning of the questions; whether they had difficulty answering the questions; and whether the response options were applicable, clear, and comprehensive. A detailed description of the pretest procedures,

7. Confidentiality:

The survey does not ask any sensitive questions. All information gathered from the survey is for research purposes only and will be kept private to the full extent allowed by law. Respondents will be assured that their information and responses will remain private and will only be used for research purposes and reported in the aggregate. In addition, a unique identifier will link the respondent information to the survey responses. The Study Team will ensure that the spreadsheet linking the unique identifiers with the respondent information is electronically stored, with access allowed only for the Study Team and kept separate from the survey responses. This file linking the respondent’s information to the unique identifier on the survey responses will only be transmitted using encryption.

Data confidentiality will be considered a continuous process during the life of the study and will be monitored and controlled by the Study Team. 2M will report any improper disclosure or unauthorized use of FNS data to the Contracting Officer’s Representative within 24 hours of discovery or loss of data.

FNS published a system of record notice entitled FNS-8 USDA/FNS Studies and Reports in the Federal Register on April 25, 1991, volume 56, pp. 19078–19080, that discusses the terms of protections that will be provided to respondents. FNS and the Study Team will comply with the requirements of the Privacy Act of 1974.

8. Federal Costs:

It is estimated that federal employees will spend approximately 277 hours overseeing this study in 2019. Using the hourly wage rate of $39.96 for a GS-12, step 1, federal employee from the 2019 Washington, DC, locality pay table, the estimated 2019 costs equal $11,068.92. In addition, the Study Team assumes 20 hours annually for the Branch Chief, for a GS-14, step 4, at $61.77 per hour for a total of $1,235.40.

Contractor costs to the Federal Government associated with this study will total $295,497.80 in 2019. When combining the federal employee and contractor costs, the total annual cost to the Federal Government for this information collection is estimated at $307,802.12.

8. Appendices (Including Research Tools/Instruments):

Appendix A.1 Respondent Burden Table

Materials Related to Recruitment for the Survey

Appendix A.4: Email Notifications to WIC State Agency

Appendix A.5: Email Request to WIC Local Agency

Appendix A.6: Reminder Emails to WIC Local Agency

Appendix A.7: Reminder Phone Calls to WIC Local Agency

Appendix A.8: Initial Survey Notification Email to Clinic-Level Staff Member

Appendix A.10: Survey Reminder Emails to Clinic-Level Staff Member

Appendix A.11: Survey Reminder Phone Calls to Clinic-Level Staff Member

Appendix A.12: Post-Survey Response Clarification Phone Call

Appendix A.13: Post-Survey Response Clarification Email

Web-based Survey Instrument

Appendix A.9: Survey - WIC Child Retention

Appendix A.14: Survey - WIC Child Retention (Screenshots)

Materials Related to Pretest:

Appendix A.2: Pretest-Local Agency

Appendix A.3: Pretest-Clinic Staff

Appendix A.15: Cognitive Pretest Memo

1 U.S. Department of Agriculture, Food and Nutrition Service. (2017). WIC Program: National level annual summary. Retrieved from: https://fns-prod.azureedge.net/sites/default/files/pd/wisummary.pdf.

2 The following are the LA size (caseloads, expressed as total participant enrollment for the most recent month) categories used for sampling: Small LAs: Caseload is fewer than 1,000; Medium LAs: Caseload is 1,000 to 4,999; and Large LAs: Caseload is 5,000 or greater.

3 Inconsistent reporting errors can occur for several reasons and in different forms. For example, a respondent might rush through questions and select the first response to each question.

4 Wun, L., Ezzati-Rice, T. M., Baskin, R., Greenblatt, J., Zodet, M., Potter, F., . . . Touzani, M. (2004). Using propensity scores to adjust weights to compensate for dwelling unit level nonresponse in the Medical Expenditure Panel Survey (Agency for Healthcare Research and Quality Working Paper No. 04004). Retrieved from https://meps.ahrq.gov/data_files/publications/workingpapers/wp_04004.pdf

5 Shao, J., & Wu, C. F. J. (1989). A general theory for jackknife variance estimation. Annals of Statistics, 17(3), 1176–1197.