Supporting Statement Part B for NFS Pilot

SUPPORTING STATEMENT PART B FOR

National Food Study Pilot

Food Assistance Branch

Food Economics Division

Economic Research Service

US Department of Agriculture

1400 Independence Avenue, SW, Mail Stop 1800

Washington, DC 20250-1800

Phone: 202-694-5398

Fax: 202-694-5661

E-mail: [email protected]

Table of Contents

Section Page

B Collections of Information Employing Statistical Methods 1

B.1 Respondent Universe and Sampling Methods 1

Respondent Universe 1

Pilot Study Sample Sizes and Expected Response Rates 3

Recall Interview Sample 5

Nonresponse Followup (NRFU) Sample 5

B.2 Procedures for the Collection of Information 6

Statistical Methodology 6

Estimation 8

Degree of Accuracy 10

Specialized Sampling Procedures 11

Any Use of Periodic (Less Frequent Than Annual) Data Collection Cycles to Reduce Burden 11

B.3 Methods to Maximize Response Rates and to Deal With

Issues of Nonresponse 12

Nonresponse Bias Analysis 13

B.4 Test of Procedures or Methods to be Undertaken 14

B.5 Individuals Consulted on Statistical Aspects and

Individuals Collecting and/or Analyzing Data 15

Contents (continued)

Tables Page

B.1.1 Pilot Target Sample Sizes 3

B.1.2 Pilot Study Assumed Attrition Rates and Sample Sizes 4

Part B. Collections of Information Employing Statistical Methods

B.1 Respondent Universe and Sampling Methods

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, state and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

Respondent Universe

The respondent universe is described in this section with a general description of the sampling methods. More details on the sampling methods are provided in Section B.2. As part of the sample selection process for the National Food Study Pilot, the survey contractor, Westat, plans to request from states administrative lists of addresses with SNAP participants to facilitate oversampling of SNAP households. This same procedure was used for USDA’s National Household Food Acquisition and Purchase Survey (FoodAPS-1). The key sampling domains for the National Food Study (NFS) Pilot are:

1. SNAP households (of any income)

2. Non-SNAP WIC households (of any income)

3. Non-SNAP and non-WIC households with income at or below 130% of Federal Poverty Level (FPL)

4. Non-SNAP and non-WIC households with income between 131% and 185% of FPL

5. Non-SNAP and non-WIC households with income above 185% of FPL

A three-stage sample design will be employed in which the primary sampling units (PSUs) are counties or groups of contiguous counties. As described further in Section B.2, a probability-based sample of 12 PSUs has been selected from the 50 PSUs that were selected for FoodAPS-1. Given the probability-based selection, the 12 PSUs are nationally representative. The 12 PSUs fall into 9 states. Therefore, the National Food Study Pilot is limited to nine of the 27 states with selected sample for FoodAPS-1. The purpose is to minimize the effort that state SNAP agencies will need to make to provide the requested data as the Pilot will be asking for essentially the same information. The respondent universe excludes persons living in institutions or group quarters.

In the second stage, segments comprised of census block groups or combinations of block groups are selected. As discussed further in Section B.2, segments with a larger number of SNAP participants will be given a higher chance of selection.

In the third stage, residential addresses will be selected within the selected segments from two strata (addresses on the state’s SNAP address list, and addresses not on the SNAP address list). To do so, a sampling frame of addresses will be obtained from two sources of address listings. To reduce cost, Westat will not be conducting traditional listing in areas where addresses may have low coverage (e.g., due to the use of PO boxes). In such areas, a larger number of addresses will be selected to account for the loss due to housing units that are not on the address listings. The first source of address listings will be a list of all residential addresses from the most recent United States Postal Service (USPS) computerized delivery sequence file (CDSF). The USPS address lists include street addresses along with the carrier route information. Qualified vendors receive updated lists from the USPS on a bimonthly or weekly basis and attach additional information from outside commercial databases, including Census geography (e.g., FIPS county codes and Census block group), telephone numbers, and demographics (e.g., number of children in the household and estimated household income). For the second source for creating the frame of addresses, Westat will seek to obtain lists of addresses with SNAP participants from the states. As described further in Section B.2, a matching operation will be conducted to determine which address-based sampling (ABS) addresses are not on the SNAP administrative list of addresses. The matching operation effectively stratifies the addresses into two strata, 1) SNAP list addresses and 2) addresses not on the SNAP list. A systematic random sample of addresses will be selected within each stratum from these lists. Once addresses are selected, a screening interview will be conducted to identify the sampling domain of both the SNAP and non-SNAP households. Among the screener completes, all who report to be on SNAP will be retained for the initial interview. A random proportion of those who are classified as non-SNAP in the highest income category will be screened out in order to reduce the number of high income participants in the sample. The remaining households will continue on with the main survey.

The NFS Pilot study will be comprised of a probability-based nationally representative sample of 500 households. Table B.1.1 shows national Current Population Survey (CPS) estimates of the percentage of households in each sampling domain, out of the approximately 120 million households in the nation, along with the target number of completes. The sample of 500 total completed households will allow for adequate precision for planned performance measures; however, there will not be enough cases to analyze separately in any sampling domain.

Table B.1.1. Pilot target sample sizes

Sampling domain	Estimated population percentages (CPS 2013)	Target number of households
SNAP households	11.1%	150 (30%)
Non-SNAP WIC households	1.2%	5 (1%)
Non-SNAP and non-WIC households with income at or below 130% of FPL	13.8%	70 (14%)
Non-SNAP and non-WIC households with income between 131-185% of FPL	9.2%	45 (9%)
Non-SNAP and non-WIC households with income above 185% of FPL	64.6%	230 (46%)
Total		500 (100%)

The oversample of SNAP households is accomplished by the use of SNAP lists from state agencies. Westat is aware of the under-reporting of program participation in CPS (e.g., FNS reports about 18.5% SNAP households compared to 11% in CPS). Therefore, we would expect to achieve more SNAP households and possibly more households with WIC individuals than shown in Table B.1.1. Furthermore, we expect that there will be a great deal of overlap between these groups, and this pilot test will further inform the degree of this overlap to inform a more efficient design of future FoodAPS surveys.

Pilot Study Sample Sizes and Expected Response Rates

As shown in Table B.1.2, an overall initial sample size of 2,500 would achieve at least four participating households per segment, accounting for vacancies and nonresponse to the study (assuming the rates shown in Table B.1.2). Among the 2,500 housing units initially selected, 580 will be selected from the SNAP list, and 1,920 from the non-SNAP ABS list. The non-SNAP ABS list includes both occupied and vacant units. The SNAP list includes only addresses with SNAP participants, and therefore only occupied housing units, unless the unit was vacated between the time the list was created and data collection. Therefore, we expect a lower vacancy rate (ten percent) for the SNAP list than the non-SNAP list, which is assumed to be 15 percent.

Table B.1.2. Pilot study assumed attrition rates and sample sizes

	SNAP list		Non-SNAP list		Total
	N	Rates	n	Rates	n
PSUs	12		12		12
Segments	120		120		120
Selected dwelling units for screening attempts (Advance Letters)	580		1,920		2,500
Occupied addresses rate		0.90		0.85
Screener response rate		0.72		0.72
Screener completes prior to subsampling out cases	376		1,175		1,551
Complement of Screen-out rate (due to domain 5)		1		0.79
Web only rate		0.61		0.61
Screener completes after subsampling, and to proceed with web collection	230		566		796
Food acquisition recording, final and feedback form response rate (1 week)		0.75		0.64
Completes	172		362		534

Among the screener completes, a random proportion of about 21 percent of those on the non-SNAP list will be screened out due to high income. It is expected that a proportion of those on the SNAP list will not be classified by the screener responses as a SNAP household. This loss from the SNAP list will be off-set to a certain extent by a proportion of those on the non-SNAP list that will be classified as a SNAP household. All cases classified as SNAP in the screener will be retained for the initial interview. Releases of reserve sample from the 12 PSUs will occur in case of observed shortfalls during the data collection period. Among all screener completes, we conservatively estimate about 61% would be capable of entering survey data via the web or a smartphone. This number is a general approximation from statistics reported in File and Ryan (2014)¹ using data from the American Community Survey (ACS). There is variation in computer ownership, handheld devices and internet use by income. For example, more than 90 percent of households with income of at least $50,000 dollars own a computer compared to only 62 percent of households with an income of less than $25,000. More than 80 percent of households with an income of $100,000 or more own a handheld device compared to approximately 40 percent of households with income of less than $25,000. Further, use of a high-speed Internet connection follows this pattern, with more than 90 percent for those with household incomes of $100,000 or more using high-speed Internet, compared to 47 percent for those with less than $25,000 in household income. The percentage on the web could be as high as 85% based on estimates from PEW.² The distribution of this pilot study’s sample is such that we expect a disproportionately larger sample of low income households.

We expect that about 64 percent to 75 percent (similar to FoodAPS-1) of the 796 screened households who are capable of completing via web will agree to participate in the one week recording of their food acquisitions, depending on being in the non-SNAP or SNAP group, respectively. This will result in approximately 534 cases that would complete the food acquisitions during the week, of which 172 are expected to be SNAP participants, and about 30 expected to be WIC participants (assuming 16 percent on SNAP are also WIC households, and a handful of households come from the non-SNAP WIC sample domain 2).

At the same time that the 2,500 residential units are selected from the two sampling frames, an additional 1,250 units will be selected to serve as a reserve sample. Given the uncertainty in the percentage of households that will be able to complete the survey via the web, Westat plans to initially release a random 70% of the 2,500 selected dwelling units and closely monitor the assumptions. The rates and sample yield will be monitored over the first couple of weeks of data collection, and the additional 30% sample will be released as needed. Beyond the additional 30%, Westat will use the reserve sample as needed.

Recall Interview Sample

An extended final interview (Recall Interview) will be administered to two household members in the approximately 534 responding households to walk them through their day and ensure that all food purchasing was reported. All interviewers will be trained on the recall interview protocol to ensure that the recall procedures are administered consistently across respondents. The extended interview will be used to collect more accurate data on food acquisitions from the primary shopper and one other randomly selected household member for the two most recent days of the household’s data collection week. The purpose is to better understand the overall extent of food acquisition underreporting, how it differs between food-at-home (FAH) events and food-away-from-home (FAFH) events, the characteristics of individuals most likely to underreport their food acquisitions, and the extent to which day of week or other factors affect underreporting.

Nonresponse Followup (NRFU) Sample

To help with estimating the potential for nonresponse bias in the NFS Pilot Test, a short questionnaire will be administered to households that were screened but did not participate in the survey for any of the following reasons:

• Not invited to participate because of likely difficulty using a smartphone- or web-based data collection mode

• Eligible to participate but refused after the screener

• Eligible to participate but refused after the initial interview but before training

• Eligible to participate but refused after training

• Entire household refused during the reporting week

All of the above nonresponding households, about 362 expected, will receive the NRFU questionnaire. The incentive to complete the questionnaire is $5. The short questionnaire will include questions whose responses are expected to be correlated with the Pilot Test’s main findings (e.g., total weekly spending on food by all household members for both food-at-home (FAH) and food-away-from-home (FAFH), and weekly number of FAH and FAFH food acquisition events).

B.2 Procedures for the Collection of Information

Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

• Estimation procedure,

• Degree of accuracy needed for the purpose described in the justification,

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Statistical Methodology

For the initial stage of sampling for the pilot, the design includes a subsample of 12 Primary Sampling Units (PSUs) from the 50 PSUs selected for FoodAPS-1, which were formed as counties or groups of counties. In FoodAPS-1, there was one PSU selected with certainty, and the other 49 PSUs were selected using Chromy’s method of sequential random sampling in SAS Proc SurveySelect. Prior to selection, the PSUs were sorted by metro status and region, which was defined by seven USDA Food and Nutrition Service (FNS) administrative regions. The PSUs were selected using a composite measure of size that was a function of the estimated number of households in key sampling domains and their associated overall sampling rates.

To select PSUs for the NFS Pilot, the one certainty PSU in FoodAPS-1 was assigned a measure of size of 1.5 to reflect the multiple hits, and the other PSUs each received a measure of size of 1. By assigning the measure of size in this manner for the NFS Pilot, the approach retains the key features of the composite measure of size assigned to PSUs in FoodAPS-1. To ensure a wide representation of different characteristics of geographic areas, the 50 PSUs were sorted by FNS region, urbanicity, and the percentage of the population on SNAP. The subsample of 12 PSUs obtains a diverse mix of communities (e.g. having both urban and rural communities) to ensure the web administration is tested under a variety of real world conditions. The subsample was selected using systematic sampling from the sorted list, and probabilities proportionate to the measure of size. In the end, there were seven FNS regions and nine states that had at least one selected PSU. There were two PSUs that contained a mix of counties inside and outside metropolitan/ micropolitan statistical areas.

In the second stage, an average of 10 segments (individual block groups or combination of adjacent block groups) will be selected per PSU.³ While the NFS Pilot test includes fewer PSUs than FoodAPS-1 for cost efficiency, the number of sampled segments within a PSU has been increased to spread the sample across the sampled PSU and cover diverse communities. The segments will be selected using a similar composite measure of size as in FoodAPS-1, where the composite measure of size is calculated as a function of the segment population count and the sampling fraction of the SNAP domain, and other sampling domains.

There will be two mutually exclusive list frames constructed within each segment. First, an address-based sampling (ABS) frame (based on files that originate from the U.S. Postal Service) of addresses will be obtained from a vendor for each of the 10 segments in the selected counties. Westat’s ABS vendor, Marketing Systems Group (MSG), geocodes every address in its ABS frame and identifies the census block associated with each address through its geocoding process. Typically, in rural areas and in highly urban areas, traditional listing of dwelling units would occur. For the pilot test, to reduce cost, Westat will not conduct traditional listing in areas where addresses are problematic (PO boxes) and will only rely on the ABS frame. In such areas, a larger number of addresses can be selected to account for the loss due to ones that are not locatable, or some additional work can be done to identify and remove non-locatable addresses from the frame prior to selection.

Second, Westat will seek address lists of SNAP households from the states for each sampled PSU. These addresses will be geocoded to identify those that fall within the sampled segments. For each sampled segment, the addresses from the SNAP lists that geocode to the block group(s) in the segment will be matched to addresses from the ABS frame that geocode to the block group(s). For each address on the ABS frame, flags will be assigned that indicate successful matches to SNAP addresses. The flags will be used to control the sample size for the SNAP sample domain. Any program list addresses in selected segments but not on the ABS frame will be matched to the complete ABS frame (without the restriction of the block groups), to determine whether the non-match was due to geocoding error or whether the list address was missing from the ABS frame entirely; those that are missing from the ABS frame will be added to the frame prior to the sampling of addresses.

Within each segment, a systematic random sample of addresses will be selected from a list sorted by frame listing type (SNAP list and non-SNAP list), and order of geographic location. The household screener will be used to confirm the program participation in SNAP. This will allow for an evaluation of the accuracy of the lists. SNAP participation of the current residents will be determined through the screener. Households also will be screened to gather income information and computer/internet usage.

In the NFS Pilot test, whole households that are unable to do data entry via a computer, tablet or smartphone because they lack these skills will be screened out (dropped). The pilot test will reveal what percentage of households is not capable of handling the web-based technology. As we will enumerate the socio-demographics of cases that are dropped, we will know if these households are concentrated within specific subgroups. It is important to know the number of households that will need to use the paper version of the survey to plan for the next FoodAPS. However, it is planned to not collect food acquisition data from these households via paper as part of the NFS Pilot test. A boost in the initial sample size will account for these sample losses.

Estimation

Westat will adjust the NFS Pilot test base weights by assuming those households who could only do a paper survey are like households who did the web survey, within specified weighting cells. Certainly bias will exist, and a basic nonresponse bias analysis will be conducted to check on the extent of the bias, and use weighting (auxiliary) variables to reduce the potential for bias. The basic nonresponse bias analysis will include auxiliary variables available for three dispositions: respondents, nonrespondents, and those not able to use the technology. We plan to compare distributions of the auxiliary variables for the three subgroups, and evaluate the relationship of the dispositions to the auxiliary variables through a multinomial regression. The auxiliary variables will include: (1) variables from the sampling frames (such as whether households on the ABS frame are on the SNAP frame and vice versa), (2) American Community Survey tract-level characteristics (such as median income), (3) demographic data from our commercial vendor, assuming the quality (number of missing values) is acceptable, (such as the number of children in the household), and (4) interviewer observations (such as the income-level of the neighborhood). For households that completed the screener, screener data can also be analyzed, including the variables on technology use included in the screener.

Sampling weights will be produced to facilitate the estimation of the target population parameters. Replicate weights will be computed to facilitate variance estimation, and will capture the variation due to the sample design and selection, as well as weighting adjustments. As discussed above, a subsample of 12 PSUs will be selected from the 50 PSUs in FoodAPS-1 for the NFS Pilot test so that a nationally representative sample can be compared statistically with FoodAPS-1. For the analysis, the FoodAPS-1 sample data will be reduced to the same subsample of PSUs to improve the comparison, with re-weighting of the FoodAPS-1 subsample.

Westat will perform an analysis to evaluate the quality of the automated data collection methodology in the NFS Pilot. First, they will select a number of estimates of food acquisitions and compare them to estimates from the FoodAPS-1. For instance, Westat will estimate the average number of food events that occurred during the data collection week and the average number of food items obtained during the data collection week and compare them to estimates from FoodAPS-1. The comparisons will tell if NFS Pilot estimates are close to the FoodAPS-1 estimates. Differences could arise from the differences in the two data collection methods. They could also arise from differences in other aspects of essential survey conditions (e.g., number of PSUs sampled, field length, and the targeted number of completed interviews).

Second, Westat will examine a certain number of indirect quality indicators that have critical implications for cost. Westat will look at, for instance, missing data rates, the amount of outlying values, the amount of inconsistent data (e.g., price is missing whereas the item is not marked as free), and the percent of cases that need manual review or post-survey processing (e.g., barcodes without item descriptions, food items to be added from receipts, food items dropped from scanned data).

For the Recall Interview, the collected information will be processed into an analysis file and used to better understand the overall extent of food acquisition underreporting, how it differs between food-at-home (FAH) events and food-away-from-home (FAFH) events, the characteristics of individuals most likely to underreport their food acquisitions, and the extent to which day or week or other factors affect underreporting. This information will be used to create estimates of underreporting for Days 6 and 7 of the overall number of food acquisitions made during the week and their total dollar value.

For the nonresponse followup sample, a weight adjustment will account for nonresponse to the NRFU questionnaire, and then indications of nonresponse bias in the NFS Pilot Test results will be generated by comparing the respondents to the main survey to the respondents of the NRFU sample using the items asked in the NRFU questionnaire (which are the same as on the main survey).

Degree of Accuracy

The overall sampling fraction for each sampling domain will be designed to balance cost with the effective sample size, which is the equivalent sample size under simple random sampling that will produce results with the same precision as the complex sample design for the pilot test. When oversampling occurs (i.e., sampling with rates that are disproportionate to population distributions), the effective sample size decreases, which can lead to increased costs to offset the reduction in sampling precision.

For the pilot test, with the nominal target number of completes of about 500, the assigned targets from Table B.1.1 and the clustering within 12 PSUs, the design effect is assumed to be about 1.5, which results in an overall effective sample size of about 333. Under this design, for direct estimates of survey characteristics reported by 30 (or 70) percent of the population (for example, the percentage of a certain characteristic such as FAFH expenses, food insecurity), a margin of error (MOE) of five percent can be achieved at the 95% confidence level.

For the Recall Interview, the comparison being made is between food log reports and the recall interview among two selected persons within the 534 households that completed the final interview. About 962 individuals are expected to complete the recall interview, and a design effect of 1.7 is assumed, which is higher than in the analysis above due to clustering within households. This implies an effective sample size of 566 individuals. When the proportion with a characteristic is p_1=0.30, the minimum detectable difference of underreporting is δ = 0.031. The computation assumes a significance level of α=0.05 for a two-tailed test and 80% power, and assumes a unit correlation of 0.80 between the results from the main NFS Pilot and the recall interview on the same sample cases.

For the nonresponse followup sample, with the comparison being made between the 500 dwelling units and 362 nonresponse households (217 responses, or effective sample size of 167), we assume a design effect of 1.3, which is lower than stated above due to the covariance from the two groups of cases being in the same segments. When the proportion with a characteristic is p_1=0.30, the minimum detectable difference of underreporting is δ = 0.15, assuming a significance level of α=0.05 for a two-tailed test and 80% power, and δ = 0.12, assuming a significance level of α=0.15.

Specialized Sampling Procedures

To reduce costs, in lieu of traditional listing, the sampling frame of dwelling units will be formed from purchased addresses based on files that originate from the U.S. Postal Service and a list of addresses for SNAP participants from State SNAP agencies.

The pilot test will also be introducing a new sampling domain (Non-SNAP WIC households) captured through information from the screener questionnaire. With regards to sampling, the domain will allow control of the sampling rates for WIC participants in future cycles of FoodAPS. The screener questionnaire will be used to sample households at different sampling rates to meet the sample sizes required for the NFS Pilot test.

Any Use of Periodic (Less Frequent Than Annual) Data Collection Cycles to Reduce Burden

All data collection activities will occur within a 4-month period. Use of periodic data collection cycles to reduce burden is not applicable for this Pilot Study.

B.3 Methods to Maximize Response Rates and to Deal With Issues of Nonresponse

Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

By explaining the importance and potential usefulness of the study findings in the NFS Pilot introductory letter, providing interviewers will the tools to gain the households cooperation, explaining the incentive structure, and implementing a series of follow-up reminders, we expect to achieve an overall screener response rate of 72%, an overall completion rate of 74% for the Initial Interview conditioning on being screened in as eligible, an overall completion rate of 90% for the Final Interview conditioning on completing the Initial Interview and agreeing to participate in the 7-day Food Log, and an overall completion rate of 95% for the Feedback Form conditioning on completing the Final Interview. In addition, we expect to achieve an overall completion rate of 80% for the Purchase and Free Food Log conditioning on completing the Initial Interview and the training. These procedures will be used to maximize response rates from both SNAP and non-SNAP Participants:

• Mail an introductory letter stating the importance of the study and their participation and the incentives they will receive upon full participation.

• In accordance with the Confidential Information Protection and Statistical Efficiency Act ("CIPSEA") enacted in 2002; respondents will be assured that the information that they provide will be kept confidential.

• Provide in-person interviewing for the Screener, Initial Interview, training, Final Interview, Feedback Form, and Recall Interview.

• Provide multiple reminder email messages and/or text updates throughout the data collection week informing respondents of the incentives they have earned and motivating them to complete their Purchased and Free Food Log.

• Provide incremental incentives based on respondents’ reporting behavior to encourage reporting of food acquisition.

• Make multiple visits to a sampled address without reaching someone before considering whether to treat the case as “unable to contact.”

• Make a refusal conversion attempt to convert households that refuse to participate after the interviewer’s initial visit to complete the Household Screener.

• Provide a toll-free number for respondents to call to verify the study’s legitimacy or to ask other questions about the study.

• Allow access to the Food Log through computers or smartphones.

• Smartphone application of the Food Log has the full functionality of scanning, taking pictures, recording text or audio memos, increasing convenience for respondents.

• Implement standardized training for field data collectors.

Nonresponse Bias Analysis

Although efforts will be made to achieve as high a response rate as practicable with the available resources, nontrivial nonresponse losses are likely to occur. OMB requires that a nonresponse bias analysis (NRBA) be conducted if the overall response rate falls below 80 percent.⁴ In this case, a nonresponse bias analysis will be conducted to assess the impact of nonresponse on the survey estimates and the effectiveness of the weight adjustments to dampen potential nonresponse biases. The types of analyses to be conducted to evaluate nonresponse will include:

• Comparing characteristics of nonrespondents (or the total sample) to those of respondents using information available for both nonrespondents and respondents;

• Modeling response propensity using multivariate analyses;

• Evaluating differences found in comparisons between survey respondents and comparable data from extant outside sources;

• Comparing cases completed at different levels of data collection effort (e.g., cases completed with limited follow-up compared to those requiring considerable follow-up);

• Comparing weighted estimates of characteristics available for both respondents and nonrespondents using unadjusted (base) weights versus nonresponse-adjusted weights; and

• Comparing weighted survey estimates using unadjusted (base) weights versus nonresponse-adjusted weights.

• Performing additional methods, such as selection models, to assess non-ignorable nonresponse bias, as appropriate.

B.4 Test of Procedures or Methods to be Undertaken

Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

Westat conducted two rounds of lab tests in February 2016 and an initial round of field tests in March 2016. Field testing will be completed in August 2016. The primary goal of the two lab tests was to test the usability of the Purchased and Free Log that can be accessed from personal computers, tablets, and smartphones. The two lab tests found several design issues of the Log including allowing some text fields to only accept numbers and other text fields to accept texts as well as numbers, allowing people to scan multiple barcodes, allowing people to create meal combos, and so on. The Purchased and Free Log was redesigned based on the results from the lab tests.

The primary purposes of the March field test were to test the full functionality of the Purchases and Free Food Log in a field setting. Different from the lab tests conducted at Westat, this field test allowed recruited household members to use their own device at a location of their own choice. As a result, this pretest tested the performance of the Food Log on different devices. The field test uncovered some minor issues with the Food Log, and Westat has modified the Food Log accordingly.

A second round of field test is planned for August 2016, focusing on testing the full integration of various CAPI instruments (e.g. CAPI Screener, CAPI Initial Interview, and CAPI Final Interview), and the Food Log. Westat again will use what is learned to further streamline the integration of the systems.

As part of the NFS Pilot, Westat will evaluate the potential use of WIC administration files to oversample WIC participants in future FoodAPS surveys. To do so, Westat will contact WIC agencies in the nine sampled states to obtain a list of address information for WIC participants within the selected PSUs (or for the entire state, whichever is most straightforward for state personnel). The WIC lists will be used as follows with the goal of evaluating the creation of a sampling frame of WIC addresses in the next FoodAPS:

• Determine if WIC lists can be obtained in a timely manner that is conducive for use as a component of a sampling frame.

• Match to ABS and SNAP addresses to evaluate the process and completeness of the files.

• Review the participation dates to determine the currency of the data.

• Compare (match) the screener results to the WIC addresses to determine its’ accuracy.

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The sampling plans were reviewed by Wendy Van de Kerckhove, Tom Krenzke, Aaron Maitland, Ting Yan, Erika Bonilla, Janice Machado, Laurie May, and Roger Tourangeau at Westat. In addition, John Kirlin of ERS and Brady West from the University of Michigan Survey Research Center (U-M SRC) have reviewed this supporting statement. This supporting statement was revised per comments from the Westat team, ERS, and U-M SRC. All data collection and analysis will be conducted by Westat.

Name	Affiliation	Telephone number	e-mail
John Kirlin	Project Officer, ERS	202-694-5398	[email protected]
Brady West	Research Associate Professor, University of Michigan Survey Research Center	734-647-4615	[email protected]
Roger Tourangeau	Senior Vice President, Westat	301-294-2828	[email protected]
Laurie May	Vice President, Westat	301-517-4076	[email protected]
Janice Machado	Project Director, Westat	301-294- 2801	[email protected]
Erika Bonilla	Study Manager, Westat	301-610-4879	[email protected]
Tom Krenzke	Senior Statistician, Westat	301-251-4203	[email protected]
Wendy Van de Kerckhove	Senior Statistician, Westat	240-453-2785	[email protected]
Aaron Maitland	Senior Statistician, Westat	240-314-2595	[email protected]
Ting Yan	Senior Statistician, Westat	301-250-3302	[email protected]

1 File, T., and Ryan, C. (2014). Computer and Internet use in the United States: 2013. American Community Survey Reports. Retrieved from http://www.census.gov/history/pdf/2013computeruse.pdf

2 Pew Research Center. (2015). The smartphone difference. Available at: http://www.pewinternet.org/2015/04/01/us-smartphone-use-in-2015/

3 On average there are about 69 block groups per county in the US with about 604 housing units in each.

4Office of Management and Budget Standards and Guidelines for Statistical Surveys. Retrieved from http://www.whitehouse.gov/sites/default/files/omb/inforeg/statpolicy/standards_stat_surveys.pdf.