OMB Memo - NSWP III Pre-testing

Memorandum

Date: September 20, 2016

To: Stephanie Tatham, OMB Desk Officer

Through: Ruth Brown, United States Department of Agriculture, Office of Chief Information Office

From: Lynnette Thomas

Food and Nutrition Service, Branch Chief, Planning & Regulatory Affairs

Re: Generic OMB Clearance No. 0584-0606 – Pretesting for the Third National Survey of WIC Participants (NSWP-III)

The Food and Nutrition Service (FNS) of the United States Department of Agriculture (USDA) is requesting approval for pretesting of data collection instruments under Approved Generic OMB Clearance No. 0584-0606.

This request is to acquire clearance to conduct pretesting with the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC)’s State agencies (SAs), local agencies (LAs), current and former WIC participants, and denied WIC applicants with the purpose of identifying potential data quality problems, such as high item nonresponse rates, and modification of flow, order, and timing of questions, as well as participant understanding of questions asked. The following information is provided for review:

1. Title of the Project: Pretesting for the Third National Survey of WIC Participants (NSWP-III)

2. Control Number: 0584-0606, Expires 03/31/2019

3. Public Affected by this Project:

State, Local, Tribal Governments:

• State WIC Agency Directors

• Local WIC Agency Directors

Individuals/Households:

• Current WIC Participants

• Former WIC Participants

• Denied WIC Applicants

4. Number of Respondents & Research Activities:

Table 1. Research Activities Target Audience

Respondent Type	Research Activity	# of Participants
State WIC Agency Director	State Agency Survey	9
Local WIC Agency Director	Local WIC Agency Survey	9
WIC Participants	WIC Participant Certification Survey	9
	Program Experiences Survey	9
	Former WIC Participants Case Study	9
Denied WIC Applicants	Denied Applicant Survey	9
Total		54

5. Time Needed Per Response:

Table 2. Estimated Time Needed Per Response by Research Activity

Respondent Type	Research Activity	Time (Hours)
State WIC Director	State Agency Survey	1.15
Local WIC Agency Director	Local WIC Agency Survey	0.72
WIC Participants	WIC Participant Certification Survey	0.50
	Program Experiences Survey	0.50
	Former WIC Participants Case Study	0.67
Denied WIC Applicants	Denied Applicant Survey	0.58

Table 3. Estimates of Respondent Burden

6. Project Purpose, Methodology, and Formative Research Design:

Background

A primary objective of the Third National Survey of WIC Participants (NSWP-III) is to provide FNS with nationally representative estimates of improper payments from the WIC program arising from errors in the certification or denial of WIC applicants, in order to fulfill the requirements of the Improper Payment Elimination and Recovery Act (IPERIA) of 2012. In addition, the study will investigate potential SA and LA characteristics that may correlate with these errors, and will assess WIC participants’ reasons for satisfaction or dissatisfaction with the program. NSWP-III builds on three previous studies and reports that span several decades.

To accomplish study objectives, the following data collections are planned for the full data collection: (1) a Certification Survey with up to 2,000 recently certified WIC participants; (2) a Denied Applicant Survey with up to 240 WIC applicants who did not qualify for the program; (3) a Program Experiences Survey with up to 2,500 current WIC program participants; (4) a Former Participant Case Study with 125 inactive WIC program participants who have stopped redeeming WIC benefits; (5) a State Agency Survey with 90 agencies, including 50 States and the District of Columbia, 34 Indian Tribal Organizations (ITOs), and 5 U.S. Territories; and (6) a Local WIC Agency Survey with 1,500 local WIC agency directors.

Purpose

The purpose of each data collection activity is described below.

State Agency Survey

Federal guidelines grant SAs substantial authority to determine the State’s WIC Program operations and procedures, including providing guidance to LAs on implementing procedures to determine an applicant’s eligibility, negotiating and determining food options, establishing application and payment procedures, and establishing program data management systems and procedures. The State Agency Survey is designed to identify policies and practices that each WIC SA has established under these discretionary powers, and to enable comparisons of their potential effects. The State Agency Survey was created by incorporating and modifying questions from NSWP-II. Some questions are new to the NSWP-III survey, some NSWP-II questions have been dropped due to changes in the program since the prior study was conducted, and other questions have been dropped because they were outside the scope of research questions detailed in the Performance Work Statement (PWS).

Local Agency Survey

The Local Agency Survey will focus on the services the LA provides to WIC participants, as well as the infrastructure of the WIC agency itself, including structure of the agency, clinics and sites under the LA, income eligibility procedures, certification procedures, and food instrument or food distribution procedures. LAs will also be asked about the policies and practices at their sites. These questions will characterize heterogeneity in site-level policies and practices across the nation. The Local Agency Survey was created by incorporating and modifying questions from NSWP-II. Some questions are new to the NSWP-III survey, some NSWP-II questions have been dropped due to changes in the program since the prior study was conducted, and other questions have been dropped because they were outside the scope of research questions detailed in the PWS.

Certification Survey

The Certification Survey’s purpose is to meet the objective of calculating improper payment rates due to certification error in the WIC program. Data from the survey, combined with State administrative data on WIC participants and redemptions of food instruments, will allow the research team to estimate rates of case error and improper payments in a nationally representative sample, as well as for each of the five certification categories (pregnant women, breastfeeding women, non-breastfeeding postpartum women, infants, and children), by asking respondents questions related to their eligibility for WIC. The Certification Survey has been adapted from NSWP-II’s Certification Survey to maintain as much comparability as possible, while balancing the need to reflect current certification regulations.

Denied Applicant Survey

The purpose of the Denied Applicant Survey is to determine whether WIC applicants who were deemed ineligible were correctly or erroneously denied WIC benefits. The survey data will inform the estimation of the rate and associated dollar cost of underpayments due to erroneous denials. An additional purpose is to describe the reasons why applicants were correctly or erroneously denied WIC benefits. To answer the study objectives, the Denied Applicant Survey differs from that used in NSWP-II. The prior study’s Denied Applicant Survey did not ask requisite questions needed to determine whether the denial was made correctly or erroneously. In order to know if an applicant was correctly or erroneously denied, the study must confirm the applicant’s residence, certification category, and income at the time of application. To achieve these objectives, the NSWP-III Denied Applicant Survey, in large part, mirrors the Certification Survey, with appropriate modifications to introductory language and question stems—for example, referring to the applicant’s “date of application” rather than “date of certification.”

Program Experiences Survey

The Program Experiences Survey will collect data on WIC participants’ program experiences, participation in other programs, food security, and other characteristics not available from administrative data. The Program Experiences Survey was created by incorporating and modifying questions from NSWP-II. Some questions are new to the NSWP-III survey.

Former WIC Participants Case Study

The Former WIC Participants Case Study consists of qualitative interviews designed to examine the barriers and facilitators to WIC program retention. The instrument also seeks to determine reasons why a participant stopped redeeming benefits within their period of eligibility. The research team will select former participants from the previously identified sampling units of LAs for this qualitative study, using administrative data provided by the SAs.

Methodology/Research Design

State Agency Survey

A paper-and-pencil, self-administered survey will be administered to selected SAs.

FNS will aid the research team in contacting the Regional Offices of three selected regions. Each of the three Regional Offices will be provided a list of four States selected for participation in the pretest of the State Agency Survey. The Regional Offices will be asked to alert the selected SAs to expect an invitation to participate in the pretest and to confirm contact information of the person who will be responsible for completing the survey.

The research team will mail a hard copy of the invitation letter and two hard copies of the questionnaire (one to keep and one to return) to nine of the selected SA directors. The remaining three SAs will be kept as potential back-up participants, if the research team is having difficulty gaining participation from an SA during the data collection period. The mail package will be sent using FedEx priority mail; this will allow for the package to stand out among the other mail items the official may receive, and it allows for the package to be tracked. The package will include a prepaid FedEx envelope to return the completed survey to the research team.

The research team will follow up with a telephone call 2 days after the package should have been received. The purpose of this call will be to answer any questions, identify the point of contact, and emphasize the timing of the pretest.

Reminder emails will be sent each week of the data collection period. No more than four reminder emails will be sent. If the State WIC Agency Director does not return the survey within 2 weeks after the invitation letter and survey are sent, the research team will place a follow-up reminder telephone call.

Once the research team receives each survey response, a trained interviewer will contact the SA director to schedule a debriefing telephone interview. During the debriefing telephone interview, the interviewer will ask pretest participants to estimate how long it took them to complete the survey, and to identify any survey questions that were confusing or difficult to answer. After the pretest data collection period has ended, the research team will send a thank-you email (or letter) to all SA directors who participated in the pretest. Feedback provided during the pretest will be used to revise the State Agency Survey. The survey will be programmed into the web-based survey software program, Qualtrics, for use with the full study sample.

Local Agency Survey

A paper-and-pencil, self-administered survey will be sent to selected LAs with contact information provided by the SAs.

The research team will email an invitation to selected LAs and mail a hard copy of the invitation letter with two hard copies of the questionnaire (one to keep and one to return). The mail package will be sent using FedEx priority mail; this will allow for the package to stand out among the other mail items the official may receive, and it allows for the package to be tracked. The package will include a prepaid FedEx envelope to return the completed survey to the research team.

The research team will follow up with a telephone call 2 days after the package should have been received. The purpose of this call will be to answer any questions, identify the point of contact, and emphasize the timing of the pretest.

Reminder emails will be sent each week of the data collection period. No more than four reminder emails will be sent. If the LA director does not return the survey within 2 weeks after the invitation letter and survey are sent, the research team will make a follow-up reminder telephone call.

Once the research team receives each survey response, a trained interviewer will contact the LA director to schedule a debriefing telephone interview. During the debriefing telephone interview, the interviewer will ask pretest participants to estimate how long it took them to complete the survey, and to identify any survey questions that were confusing or difficult to answer. After the pretest data collection period has ended, the research team will send a thank-you email (or letter) to all LA directors who participated in the pretest. Feedback provided during the pretest will be used to revise the Local Agency Survey. The survey will be programmed into the web-based survey software program, Qualtrics, for use with the full study sample.

Certification Survey

In-person survey interviews will be conducted with nine selected current WIC participants (e.g., with five interviews in English and four in Spanish). Participants will be recruited as described in the Recruiting and Consent section.

Selected participants will be scheduled for an interview which will take place at their home. After obtaining informed consent in person, the Field Interviewer (FI) will administer the survey from a paper questionnaire, reading each question aloud, following interviewer instructions, and writing down the responses indicated by the WIC participant. As specified in the questionnaire, the FI will obtain and review documentation of income and other determinants of eligibility, and then return the documentation to the respondent. The FI will use a stopwatch to record the total duration needed for the survey. The FI will debrief the respondent immediately following the completion of the Certification Survey and identify any questions that were confusing or difficult to answer. After completing the survey and debriefing, the FIs will thank each pretest participant and provide a $25 gift card. After incorporating changes needed to improve item response rates, the flow and timing of questions, and/or clarity of questions asked, the survey will be programmed for computer assisted personal interviewing (CAPI) for use with the full study sample.

Denied Applicant Survey

In-person survey interviews in either English and Spanish will be conducted with a total of nine selected denied WIC applicants (e.g., with five interviews in English and four in Spanish).

Data collection protocols are identical to that for the Certification Survey. Selected participants will be scheduled for an interview which will take place at their home. After obtaining informed consent in person, the FI will administer the survey from a paper questionnaire, reading each question aloud, following interviewer instructions, and writing down the responses indicated by the survey participant. As specified in the questionnaire, the FI will obtain and review documentation of income and other determinants of eligibility, and then return the documentation to the respondent. The FI will use a stopwatch to record the total duration needed for the survey. The FI will debrief the respondent immediately following the completion of the Denied Applicant Survey and identify any questions that were confusing or difficult to answer. After completing the survey and debriefing, the FIs will thank each pretest participant and provide a $25 gift card. After incorporating changes needed to improve item response rates, the flow and timing of questions, and/or clarity of questions asked, the survey will be programmed for CAPI for use with the full study sample.

Program Experiences Survey

Interviews with a total of nine current WIC participants will be conducted by telephone (e.g., with five interviews in English and four in Spanish). Participants will be recruited as described in the Recruiting and Consent section.

Trained telephone interviewers (TI) will administer the survey using paper copies that will have the same questions, response options, and prompts as the questionnaires that will be programmed for use with the full study sample. After verbally obtaining informed consent by telephone, the TI will administer the survey, reading each question aloud, following interviewer instructions, and writing down the responses indicated by the WIC participant. The TI will use a stopwatch to record the total duration needed for the survey.

The TI will debrief the respondent immediately following the completion of the Program Experiences Survey and identify any questions that were confusing or difficult to answer. After completing the survey and debriefing, the TIs will thank each pretest participant and obtain a delivery address for sending a $25 gift card. After incorporating changes needed to improve item response rates, the flow and timing of questions, and/or clarity of questions asked, the survey will be programmed for computer assisted telephone interviewing (CATI) for use with the full study sample.

Former WIC Participant Case Study

Telephone-based interviews will be conducted with nine selected former WIC participants (e.g., with five interviews in English and four in Spanish). Participants will be recruited as described in the Recruiting and Consent section.

The Former WIC Participant Case Study interviews are qualitative in nature and the interview guide will be available on paper for interviewers to conduct by telephone. After verbally obtaining informed consent by telephone, the TI will administer the survey, reading each question aloud, following interviewer instructions, and audio recording the interview, if recording is permitted by the respondent. If permission is not given to audio-record the interview, an additional TI will be asked to take notes. The TI will use a stopwatch to record the total duration needed for the survey.

The TI will debrief the respondent immediately following the completion of the Former WIC Participant Survey and identify any questions that were confusing or difficult to answer. After completing the survey and debriefing, the TIs will thank each pretest participant and obtain a delivery address for sending a $25 gift card. After incorporating changes needed to improve item response rates, the flow and timing of questions, and/or clarity of questions asked, the survey will be programmed for CATI for use with the full study sample.

Design/Sampling Procedures

State Agency Survey

The research team will select three FNS Regional Offices and nine SAs under the jurisdiction of those offices as the primary SA sample. An additional SA will also be identified in each region to serve as a back-up if the research team is having difficulty gaining participation from an SA during the data collection period. The research team will present a list of SAs recommended for the pretest to FNS for review. The research team will seek a mix of urbanized and rural States from each of three regions. The team also will seek to include an ITO or territory among the nine responding agencies, if possible.

Local Agency Survey

SAs that have agreed to participate in the pretest of the State Agency Survey will be asked to help the research team identify one or two LAs that could participate in this pretest. The overriding criterion for selection would be to select LAs that the SA believes would be most cooperative in the short timeframe of the pretest. Again, following a convenience sample study design, the research team will continue to recruit LAs to pretest the instrument until up to nine LA directors agree to participate.

Certification Survey

A subsample of two SAs (and LAs within these two States, if necessary) that participate in the pretest will be asked to provide the names of 50 English-speaking participants and 50 Spanish-speaking participants, who were certified up to 4 weeks prior to the start of pretesting, who may be invited to participate in the pretest. These State or local agencies will be selected such that in-person pretesting can occur in one or two geographic areas where trained interviewers are available, to minimize travel costs and to facilitate efficient data collection during the short pretesting period. The selected participants will be invited to take part in the pretest until nine current participants have responded to the survey (e.g., with five interviews in English and four in Spanish). Both versions of the instrument—that is, the instrument for use with adult WIC participants themselves, and the version for parents or legal guardians of infant or child WIC participants—will be pretested. Therefore, each version of the survey (English Version A for adult WIC participants; Spanish Version A for adult WIC participants; English Version B for infant/child WIC participants; and Spanish Version B for infant/child WIC participants) will be pretested with two to three people, for a total of nine pretest respondents.

Denied Applicant Survey

A subsample of SAs (and LAs within these SAs, if necessary) that participate in the pretest will be asked to provide the names of 50 English-speaking applicants who were denied WIC assistance and benefits no more than 6 months prior to the onset of pretesting, and 50 Spanish-speaking applicants who were denied WIC assistance and benefits no more than 6 months prior to the onset of pretesting, who could be contacted for the pretest. These SAs or LAs will be selected based on geographic proximity to facilitate efficient data collection during the short pretesting period. The selected participants will be contacted until nine denied WIC applicants have responded to the survey. The research team will pretest the instrument in English and Spanish with a total of nine respondents selected from a list of denied WIC applicants. Both the adult and child versions of the instrument will be pretested. Therefore, each version of the survey (English Version A for adult WIC applicants; Spanish Version A for adult WIC applicants; English Version B for infant/child WIC applicants; and Spanish Version B for infant/child WIC applicants) will be pretested with two to three people, for a total of nine pretest respondents.

Program Experiences Survey

A subsample of the SAs (and LAs within these SAs, if necessary) that participate in the pretest will be asked to provide the names of a total of 50 English-speaking participants and 50 Spanish-speaking participants who could be contacted for the pretest. The research team will pretest the instrument with a total of nine respondents selected from a list of WIC participants (e.g., with five interviews in English and four in Spanish). Both the adult and child versions of the instrument will be pretested. Therefore, each version of the survey (English Version A for adult WIC participants; Spanish Version A for adult WIC participants; English Version B for infant/child WIC participants; and Spanish Version B for infant/child WIC participants) will be pretested with two to three people, for a total of nine pretest respondents.

Former WIC Participant Case Study

A subsample of the SAs (and LAs within these SAs, if necessary) that participate in the pretest will be asked to provide the names of 50 English-speaking former participants and 50 Spanish-speaking former participants who could be contacted for the pretest. Depending on the capabilities of the SA’s management information system (MIS), the Study Plan refers to two methods that would be used for selecting samples of former participants. The preferred method for the pretest is to select those States where the MIS is capable of identifying former participants who stopped redeeming benefits before the end of their certification period as target participants for the survey.

The research team will pretest the instrument in English and Spanish with a total of nine respondents selected from a list of former WIC participants. Both the English and Spanish versions will be tested (e.g., with five interviews in English and four in Spanish).

Recruitment and Consent

Using the sampling plan detailed above, recruitment will follow the procedure(s) described below:

Collection of WIC Participant and Denied Applicant Information

The research team and FNS will work collaboratively to obtain necessary information on current and former WIC participants, and denied applicants. The research team will identify up to three potential LAs within two States to provide this information. The Regional Offices will notify the SA directors in both States that the research team is interested in obtaining this data from three preferred LAs in their jurisdiction. The research team will request that the SA identify the most suitable candidate from the list provided. Following an emailed notification of selection from the SA to the LA, the research team will contact the selected LA and request the participant and denied applicant information needed for the pretest of the Certification Survey, Denied Applicant Survey, Program Experiences Survey, and Former Participant Case Study. The research team will only contact the other two LAs if the first is unable to provide the number of participants needed for the pretest. The research team will use the information obtained to recruit respondents for each of the surveys and to complete the pretest of the Certification Survey and Denied Applicant Survey.

State Agency Survey

FNS will aid the research team in contacting the Regional Offices of three selected regions. Each Regional Office will be provided a list of three SAs selected for participation in the State Agency Survey pretest. The research team will identify a fourth SA as a back-up if it is having difficulty gaining participation from an SA during the data collection period.

The research team will use current contact information to send an email invitation letter to the nine State WIC Agency Directors to request their participation in the study and remind them of their obligation to participate in these types of studies. The research team will mail a hard copy of the invitation letter and two hard copies of the questionnaire (one to keep and one to return) to the selected State WIC Agency Directors. The research team will follow up with a phone call 2 days after the package should have been received. The purpose of the call will be to answer any questions, confirm the point of contact, and emphasize the timing of the pretest.

Reminder emails will be sent each week of the pretest period, and up to two reminder telephone calls will be made, until a SA completes its survey.

Local Agency Survey

The research team will email an invitation letter to selected Local WIC Agency Directors in the sample to request their participation in the study and remind them of their obligation to participate in these types of studies.

The research team will mail a hard copy of the invitation letter and two hard copies of the questionnaire (one to keep and one to return) to the selected LA directors. The research team will follow up with a telephone call 2 days after the package should have been received. The purpose of the call will be to answer any questions, confirm the point of contact, and emphasize the timing of the pretest.

Reminder emails will be sent each week of the data collection period. If the LA director does not return the survey within 2 weeks after the invitation letter and survey are sent, the research team will make up to two follow-up reminder telephone calls.

Certification Survey

Prior to conducting the in-person interviews, trained recruiters will contact potential respondents to participate in the pretest of the Certification Survey by telephone. Trained FIs will then schedule in-person interviews by telephone with those who have agreed to participate. FIs will schedule a time that is convenient for the participant and follow up with a confirmation and reminder phone call 1 day prior to the scheduled interview time. If a participant gives the FI permission to send text messages, then text messages will be used to confirm a scheduled interview. Prior to administering the Certification Survey, the FI will thoroughly review the study consent form. Consent will be obtained in writing, and a copy of the signed consent form will be provided to the respondent (one copy kept by the FI). In the event of non-consent, the FI will thank the sampled participant, conclude the visit and leave their home.

Denied Applicant Survey

Recruitment protocols will be identical to that for the Certification Survey. Prior to conducting the in-person interviews, trained FIs will schedule in-person interviews by telephone. FIs will schedule a time that is convenient for the participant and follow up with a confirmation and reminder phone call 1 day prior to the scheduled interview time. If a participant gives the FI permission to send text messages, then text messages will be used to confirm a scheduled interview. Prior to administering the Denied Applicant Survey, the FI will thoroughly review the study consent form. Consent will be obtained in writing and a copy of the signed consent form will be provided to the respondent (one copy kept by the FI). In the event of non-consent, the FI will thank the sampled participant, conclude the visit and leave their home.

Program Experiences Survey

TI will use current contact information to call the WIC participants selected for the telephone Program Experiences Survey. No more than five attempts (telephone calls) will be made to contact the respondent. Prior to administering the telephone Program Experiences Survey, the TI will read the study consent form; consent will be obtained verbally.

Former WIC Participant Case Study

TI will call the sample selected for interview for the Former WIC Participant Case Study. No more than five attempts (telephone calls) will be made to contact the respondent. Prior to administering the Former WIC Participant Case Study, the TI will read the study consent form; consent will be obtained verbally.

Incentives

To reduce burden on respondents, the research team proposed an incentive of, $25 for completing the Certification Survey, $25 for completing the Program Experiences Survey, $25 for completing the Program Denied Applicant Survey, and $25 for completing the Former WIC Participant Case Study interview. Providing an extrinsic incentive increases cooperation rates, especially in populations defined as being in poverty,^1,2 and a monetary incentive even more so.³

Based on the review of the existing incentive literature, the research team firmly believes that the $25 for the participant and denied applicant surveys is an appropriate incentive for the estimated burden, ranging from 40 minutes to 50 minutes, on WIC participants. The proposed incentive of $25 for each survey completed is slightly higher than the $20 incentive used for the NSWP-II in-person surveys, however the research team believes this is an appropriate incentive value given previous experience with this population and the sensitive nature of the questions. NSWP-II obtained complete data for approximately 81 percent of participants in the in-person

interviews (WIC rates by category ranged from 79.3 percent for infants to 84.9 percent for breastfeeding women), however, experienced a low response rate for the telephone survey (an overall 51.3 percent).⁴

Similar incentive plans have been approved in the past. For example, in the OMB-approved Impact of Housing and Services Interventions for Homeless Families study (OMB No: 2528-0259), respondents received up to a maximum of $240 in incentives, including $35 for a baseline survey, and $50 each for two different follow-up surveys. For the WIC Peer Counseling study OMB approved a $20 incentive for WIC participants who completed a Baseline Survey and an additional $20 incentive for those who also completed a Follow-up survey (OMB No: 0584-0548). These surveys, however, were 15 to 20 minutes in duration when administered in 2012 and thus were shorter than the estimated duration of the participant and denied applicant surveys proposed for NSWP-III.

Accurate estimation of certification errors and improper payments in WIC depends on participation in the Certification and Denied Applicant Surveys. The validity of responses to the Program Experiences Surveys, and to a lesser extent, the Former WIC Participant Case Study, depend on a sufficient response rate. While pretesting will take place on a small number of respondents, the quality of pretesting based on participation will allow the survey instruments to be refined to modify the flow, order and timing of questions, clarify questions asked, and revise or omit items that yield high item nonresponse rates.

Incentives will be used for the following instruments during pretesting (no incentives will be provided for SAs or LAs):

Certification Survey

The research team proposes an incentive of $25 in the form of a gift card for completing the Certification Survey. The justification for providing this level of incentive is to help mitigate the potential for nonresponse bias and to enhance the quality of the data collected. There is burden on the respondent to provide sensitive information and documentation about their income and other eligibility factors. The incentive will be used to compensate the respondent for this unusual reporting burden. The incentive will also mitigate some expenses related to child care or travel (in the event that the interview cannot be conducted at the participant’s home).

Denied Applicant Survey

The research team proposes an incentive of $25 in the form of a gift card for completing the Denied Applicant Survey. Similar to the Certification Survey, the justification for providing this level of incentive is to help mitigate the potential for nonresponse bias and to enhance the quality of the data collected. There is burden on the respondent to provide sensitive information and documentation about their income and other eligibility factors. The incentive will be used to compensate the respondent for this unusual reporting burden. The incentive will also mitigate some expenses related to child care or travel (in the event that the interview cannot be conducted at the participant’s home).

Program Experiences Survey

The research team proposes an incentive of $25 in the form of a gift card for completing the Program Experience Survey. The justification for providing this level of incentive is to help mitigate the impact of nonresponse bias and to enhance the quality of the data collected. There is burden on the respondent to respond to this survey. The incentive will be used to compensate the respondent for this reporting burden.

Former WIC Participant Case Study

The research team proposes an incentive of $25 in the form of a gift card for completing the Former WIC Participant Case Study interview. The justification for providing this level of incentive is to help mitigate the impact of nonresponse bias and to enhance the quality of the data collected. There is burden on the respondent to identify sensitive information about their experiences. The incentive will be used to compensate the respondent for this unusual reporting burden.

Data Analysis

Analyses of the pretest data will include calculation of the average, median, and maximum duration of each survey (to provide an empirical estimate of burden); identification of items with high nonresponse and/or frequent indications by respondents during debriefing of questions being unclear; and qualitative review of suggestions received during debriefing for modifying the order, flow, or wording of items. Data collection instruments will be revised as needed based on these sources of data.

7. Confidentiality:

FNS complies with the Privacy Act of 1974. All information gathered from SA and LA directors, current and former WIC participants, and denied applicants who participate in this study, is for research purposes only and will be kept private to the full extent allowed by law. Data from the data collection efforts will be presented in aggregate form, and therefore cannot be linked back to the response of any individual. All respondents will be given consent forms that state their personal information will be kept private, and that their responses will only be used for summary tabulations and statements of best practices. FNS published a system of record notice (SORN) titled FNS-8 USDA/FNS Studies and Reports in the Federal Register on April 25, 1991, volume 56, pages 19078–19080, that discusses the terms of protections that will be provided to respondents. To ensure that personal information remains private, the contract executed between FNS and Capital Consulting Corporation (CCC) requires that CCC create and keep data on secure networks, and utilize data collectors who sign confidentiality agreements binding them to protect private information. The Contractor will assign a unique ID number to each respondent. A data file (called the “master key file”) will contain each respondent’s ID number, name, and contact information. This file will be encrypted and retained by CCC and its subcontractors, Abt Associates Inc., Abt SRBI, and 2M Research Services separate from data files containing respondent responses to data collection. Data files used to analyze pretesting of the instruments will not include the name or other personally identifiable information (PII) of any respondent. Such PII will be replaced with the respondent ID number in analysis data files. Once the contract is over, the Contractor will destroy the master key file with private information.

8. Federal Costs:

The total cost to the Federal Government is $157,998.01, which includes the total cost for Contractor and Federal Staff costs for the abbreviated supporting statement, developing the instruments, and the collection of pretesting information. The Contractor cost is estimated at $157,044.97. This is based on an estimate of 1005 hours, with a salary range of $54.33–$220.75/hour, and includes overhead costs. This information collection also assumes a total of 22 hours of Federal employee time: for GS-12, step 6 at $43.32 per hour, for a total of $953.04 on an annual basis. Federal employee pay rates are based on the General Schedule of the Office of Personnel Management (OPM) for 2016.

9. Research Tools/Instruments:

All instruments being pretested in the field and recruitment materials being pretested in the research lab are listed below. Instruments are included as appendices.

Table 4. NSWP-III Pretest Instruments and Materials

Data Collection Activity	Appendix	Document Name	Pretest Method	Document Type
State Agency Survey	A1	State Agency Survey	Field	instrument
	B1	Notification Email to Regional and State Offices	Lab	recruitment materials
	B2	Letter to State Agencies from Regional Offices	Lab	recruitment materials
	B3	State Agency Survey Invitation Email	Field	recruitment materials
	B4	State Agency Survey Invitation Letter with Instrument	Lab	recruitment materials
	B5	State Agency Survey Reminder Email	Lab	recruitment materials
	B6	State Agency Survey Reminder Phone Script	Lab	recruitment materials
	C3	State Agency Survey Thank You Letter	Lab	communication materials
Local WIC Agency Survey	A2	Local WIC Agency Survey	Field	instrument
	B7	Local WIC Agency Survey Invitation Email	Field	recruitment materials
	B8	Local WIC Agency Survey Invitation Letter with Instrument	Lab	recruitment materials
	B9	Local WIC Agency Survey Reminder Email	Lab	recruitment materials
	B10	Local WIC Agency Survey Reminder Phone Script	Lab	recruitment materials
	C4	Local WIC Agency Survey Thank You Letter	Lab	communication materials
Certification Survey	A3a	Certification Survey: Version A (Adult)-English	Field	instrument
	A3b	Certification Survey: Version B (Infant/Child)-English	Field	instrument
	N/A	Certification Survey: Version A (Adult)-Spanish	Field	instrument
	N/A	Certification Survey: Version B (Infant/Child)-Spanish	Field	instrument
	B11	Certification Survey Recruitment Phone Script	Field	recruitment materials
	B12	Text Message Reminder for Scheduled Certification Survey	Field	recruitment materials
	B13	Scheduled Certification Survey Reminder Phone Script	Field	recruitment materials
	C1	Informed Consent Form Certification Survey	Field	communication materials
Denied Applicant Survey	A4a	Denied Applicant Survey: Version A (Adult)-English	Field	instrument
	A4b	Denied Applicant Survey: Version B (Infant/Child)-English	Field	instrument
	N/A	Denied Applicant Survey: Version A (Adult)-Spanish	Field	instrument
	N/A	Denied Applicant Survey: Version B (Infant/Child)-Spanish	Field	instrument
	B14	Denied WIC Applicant Survey Recruitment Telephone Script	Field	recruitment materials
	B15	Text Message Reminder for Scheduled Denied Applicant Survey	Field	recruitment materials
	B16	Scheduled Denied Applicant Survey Reminder Phone Script	Field	recruitment materials
	C2	Informed Consent Form Denied Applicant Survey	Field	communication materials
Program Experiences Survey	A5a	Program Experiences Survey: Version A (Adult)-English	Field	instrument
	A5b	Program Experiences Survey: Version B (Infant/Child)-English	Field	instrument
	N/A	Program Experiences Survey: Version A (Adult)-Spanish	Field	instrument
	N/A	Program Experiences Survey: Version B (Infant/Child)-Spanish	Field	instrument
	B17	Program Experiences Survey Invitation Phone Script	Field	recruitment materials
Former WIC Participant Case Study	A6a	Former WIC Participant Interview Guide-English	Field	instrument
	N/A	Former WIC Participant Interview Guide-Spanish	Field	instrument
	B18	Former WIC Participant Survey Invitation Phone Script	Field	recruitment materials

1 Singer E. (2002). The use of incentives to reduce non response in households surveys. In Groves R, Dillman D, Eltinge J, Little R (eds.), Survey non response. New York: Wiley, pp 163-177.

2 James T. (1996). Results of wave 1 incentive experiment in the 1996 survey of income and program participation. Proceedings of the Survey Research Section, American Statistical Association, 834–839.

3 Groves, R., Fowler, F., Couper, M., Lepkowski, J., & Singer, E. (2009). Survey methodology. New York: Wiley, pp 205-206.

4 U.S. Department of Agriculture, Food and Nutrition Service, Office of Research and Analysis, National Survey of

WIC Participants II: Improper Payments Report, by Gary Huang, et al. Project Officer: Sheku G. Kamara, Karen-

Castellanos-Brown, Alexandria, VA: 2012.

1