U.S. Department of Agriculture
Food and Nutrition Service
Enhancing Completion Rates for SNAP
(Supplemental Nutrition Assistance Program) Quality Control Reviews
Request for Clearance
Supporting Statement and
Data Collection Instruments
Part B: Collection of Information Employing Statistical Methods
Project Officer: Robert Dalrymple
TABLE OF CONTENTS
PART b: Collection of Information Employing Statistical Methods 1
B.1. Respondent Universe and Selection Methods 1
B.2. Procedures for the Collection of Information 3
B.3. Methods to Maximize Response Rates and the Issue of NonResponse 6
B.4. Tests of Procedures 7
B.5. Consultants 7
Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.
The purpose of this section is to document any statistical procedures used for the “Enhancing Completion Rates for SNAP QC Reviews” study. The data collection will involve interviews and surveys about the quality control (QC) review process with QC staff in all Supplemental Nutrition Assistance Program (SNAP) States (including the District of Columbia, Guam, and U.S. Virgin Islands). The universe is a census of 53 SNAP QC directors, 106 SNAP QC supervisors (on average 2 per State), and 265 SNAP QC reviewers (on average 5 per State) across all 53 SNAP States. Some variation among States in the actual number of supervisors and reviewers at the time of the data collection is expected; these numbers represent an upper bound. Interviews in six States will be conducted in person during the course of site visits. Surveys in 45 States will be conducted via Web or telephone. Potential respondents will be emailed links to the survey Web site. Those who do not respond will receive follow-up phone calls and offered the opportunity to complete the survey either on the Web or through computer-assisted telephone interviewing (CATI). Lists of potential respondents will be received from FNS in May 2014, approximately 1 month prior to data collection. Nine surveys and interviews were conducted in two States to pre-test the questionnaires; these respondents will not be resampled during the full implementation of the surveys and interviews.
In addition, similar interviews and surveys will be conducted with Federal regional office staff; as Federal employees, they are not subject to OMB approval and burden estimates for data collection. The universe for Federal regional offices consists of up to seven reviewers per regional office; interviews with regional QC supervisors will also be conducted in two regions. Some variation in staff numbers among regions is expected. FNS will provide a list of potential respondents for interviews in two regions, which will be conducted in person during site visits in August and September 2013. In addition, FNS will provide the complete list of potential respondents in May 2014, and surveys in five regions will be conducted by telephone in June and July 2014.
Finally, in a related study component, an attempt will be made to complete up to 25 QC reviews previously designated as incomplete from each of 3 States. The purpose of these re-reviews is to provide information regarding how incomplete cases are reviewed and processed, assessing whether additional or different steps may result in completing any cases, and, if possible, assessing the impact of incomplete cases on overall payment error rates. The re-review attempts will be from the most recent incomplete cases in each of the three States and are not intended to provide results generalizable to the national QC error rate. Since the QC instrument used for these will be the one prepared by USDA for use by the States and previously approved by OMB—and for which burden has been previously determined—these reviews are not considered a new data collection.1
B.1.1 Target Population. Pending recruitment, FNS has identified six States in which semi-structured interviews will be conducted during site visits. These include two high-performing States based on high completion rates, three low-performing States based on low completion rates, and one improving State. Two States were interviewed in the pre-test of the questionnaires. In the remaining 45 States,2 surveys will be conducted via Web or telephone (CATI).
FNS has identified two Federal regional offices where the staff will be interviewed in person, using semi-structured interview protocols. One regional office has a mix of low-performing States, and the other regional office has a mix of high-performing States. Surveys will be conducted via telephone (CATI) in the five remaining Federal regional offices.
B.1.2 Survey Eligibility and Frame. All individual QC directors, supervisors, and reviewers on lists received from FNS are eligible for the study, so no screening will be needed beyond verification of the position and no selection will be applied. Those on the list who may have recently left the position will not be included.
B.1.3 Response Rates. Estimated survey response rates are 81 percent of State QC directors, 82 percent of State QC supervisors, and 83 percent of State QC reviewers. These high rates reflect the requirements of the position; the certainty of eligibility (although some named persons on the telephone and email lists could have recently left their positions); and the fact that members of each group are expected to be notified by their superiors of the need to comply with the survey request. For the semi-structured interviews during site visits, response rates are estimated to be 100 percent for QC directors, supervisors, and reviewers. (State QC directors are expected to pick days for site visits when their staff will be in the office.)
Describe the procedures for the collection of information including:
Statistical methodology for stratification and sample selection
Estimation procedure
Degree of accuracy needed for the purpose described in the justification
Unusual problems requiring specialized sampling procedures
Any use of periodic (less frequent than annual) data collection cycles to reduce burden
B.2.1 Data Collection. The proposed surveys (with the exception of those in six States, which will be conducted in person during site visits) will obtain information on QC procedures, challenges, and best practices, and will be implemented on the Web and by telephone. Potential respondents will be emailed links to the survey Web site, and nonrespondents will receive follow-up phone calls giving the opportunity to complete the survey either on the Web or through CATI. The Web and CATI survey modes were selected for this target population as an efficient way to reach known respondents where contact information is accurate. Overall, a combined Web and CATI survey should yield a higher response rate for this study than other modes of data collection. The data collection methodology is as follows:
The survey instruments will be developed, tested, and programmed with status codes.
A survey management system will be programmed to track completed cases, partially completed cases, call history, and locating history.
A training program will be developed and interviewers will be thoroughly trained on all aspects of the study.
Response rates will be monitored and analyzed by completed interviews by time of day and days of the week to optimize calling times.
Refusal conversion calls will be made by specialists trained in refusal conversion.
Maximizing response. We propose a multipronged strategy for ensuring strong response rates.
Email contact with telephone follow-up. Initial contacts to the target population will be made via email with links to the Web survey that can be completed at their convenience. Telephone follow-ups will then be conducted for those who do not initially respond to the Web survey, to provide the option to complete the survey over the phone.
Follow-up attempts on different days/at different times of day for a total of up to seven calls. Messages will be left for recipients to call a toll-free number to complete the survey. Research shows that the incremental increase in response rates diminishes beyond seven calls.
Landline and mobile telephone numbers. For QC reviewers, who are frequently on travel away from their offices, both types of phone numbers will be obtained wherever possible. Initial attempts will be made to both telephone numbers. Subsequent attempts will be made to mobile numbers.
Call rotation and flexibility. The CATI system can schedule calls to rotate among various times throughout the day (and evening, as may be appropriate) during callbacks. The system allows respondents to call in to complete a survey or continue a survey over multiple sittings. Interviewers can also schedule appointments so that respondents can participate at a time convenient to them.
Refusal conversion. We plan to implement refusal conversion appropriate to the needs of the project. The types and methods of conversion will be communicated to interviewers as part of the training.
Cross-sectional design. The survey is cross-sectional, so no future contacts are planned after the interview is completed by Web, telephone, or in person.
B.2.2 Statistical Methodology for Stratification and Sample Selection. As a census (based on current information on the number of State QC employees), there will be no stratification or sample selection. FNS will provide a full list of the target population for the interviews and surveys.
B.2.3 Measurement/Estimation Procedures. As a census (based on current information on the number of State QC employees), no weighting or adjustment will be needed. Given the high anticipated response rate, the small number of nonrespondents should have little or no impact on the results. Due diligence will be exercised, however, in examining any known differences between respondents and nonrespondents to understand any potential bias introduced by nonresponse. Results of this analysis will be included in the final report.
All data management and cross-tabulations/frequency distributions will be conducted using SAS v9.2. Variations in output, per type of analysis, will depend on what measures are appropriate for the variable and the measurement level (i.e., nominal, ordinal, or scale) for each defined variable. For example, a nominal measure (e.g., nondirectional categories, related to the respondents’ background and SNAP QC history) will be analyzed using frequencies and percentages. For an ordinal measure (e.g., directional categories, such as strongly agree to strongly disagree), we will produce frequencies and percentages for the variables. Finally, for a scale measure (e.g., a numerical value, such as years of experience as QC reviewer, number of review cases received per month, percentage of time dedicated to QC reviews), we will produce a mean, median, or other descriptive measure.
Describe methods to maximize response rates and to deal with issues of nonresponse. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.
The methods described above have been shown to yield high response rates when the survey is of reasonable length and when potential respondent members consider the topic salient and are urged to comply by superiors. The following strategies will be used to help achieve the estimated response rate, unless otherwise noted:
Personalized pre-notification letters and study information sheets (State SNAP QC directors only)
Bimodal data collection, allowing respondents to complete the survey either online or by phone (Web survey with telephone follow up giving the opportunity to complete the survey by CATI or online)
Strategically scheduled follow-up attempts
Survey sponsorship by a recognized Federal agency
A brief introduction that underscores the salience of the survey topic for sample members
Interviewer training that addresses potential obstacles in reaching or communicating with QC staff and offers strategies for overcoming these obstacles
A toll-free number for respondents with questions
The pre-notification letter will be printed on USDA letterhead and emailed to State SNAP QC directors along with an information sheet about the study including Frequently Asked Questions (see Attachments A.1A and A.1B). These materials will briefly explain the purpose of the study, describe the reasons why those in the frame are expected to comply, and urge QC directors to ask supervisors and reviewers to respond. The letter will also include the estimated completion time of the survey and identity safeguards. Stating the sponsorship of the survey helps to engage sample members by providing immediate assurance that the survey is legitimate and not an attempt to sell them something. The likelihood of acceptance is greatly increased when sample members are told early why the survey is being conducted and why their responses are important.
Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.
A pre-test of instruments used for State QC staff was conducted with a total of eight respondents. The shared telephone instrument developed for State directors and supervisors (with most sections of the instrument shared but with a few questions specific to each of the positions) was pre-tested with one director. The telephone instrument for State QC reviewers was pre-tested with five QC reviewers. The semi-structured interview instruments were pre-tested with one director and one QC reviewer. The pre-tests resulted in clarifications in instructions, changes in question wording and question order, and changes in response options. In addition, the pre-tests clarified the overall flow of the questionnaires and confirmed estimates of questionnaire length and burden calculations. Upon completion of the pre-tests, Insight prepared a memo describing reactions to each instrument and any changes made in response to the pre-test (see Attachment D). Final edits to the instruments were implemented and submitted following FNS review of suggested changes.
Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
A review by the USDA National Agricultural Statistics Service (NASS) was conducted (Attachment C). No other individuals outside the evaluation project were consulted on statistical aspects of the design. FNS has contracted with Insight Policy Research to conduct this study. Insight is subcontracting Web and CATI activity to ICF International. Table B5.1 identifies the individuals at these organizations who will be responsible for collecting and analyzing the data. The Project Officer for the contract providing funding for the evaluation, Bob Dalrymple, will be responsible for receiving and approving all contract deliverables. His contact information is also included in Table B5.1.
Table B5.1
Individuals Responsible for Statistical Aspects and Data Collection and Analysis
Name |
Title (Project Role) |
Organizational Affiliation and Address |
Phone Number |
Stéphane Baldi |
Vice President (Executive Project Director) |
Insight Policy Research 1901 N. Moore Street, Suite 204 Arlington, VA 22209 |
(703) 504-9486 |
Brittany McGill |
Senior Researcher (Deputy Project Director) |
Insight Policy Research 1901 N. Moore Street, Suite 204 Arlington, VA 22209 |
(703) 504-9485 |
Betsy Thorn |
Senior Researcher (Project Manager) |
Insight Policy Research 1901 N. Moore Street, Suite 204 Arlington, VA 22209 |
(703) 504-9488 |
Meg Tucker |
Senior Researcher (Data collection and analysis) |
Insight Policy Research 1901 N. Moore Street, Suite 204 Arlington, VA 22209 |
(703) 504-9496 |
Anne Peterson |
Principal (Quality Assurance) |
Insight Policy Research 1901 N. Moore Street, Suite 204 Arlington, VA 22209 |
(703) 504-9483 |
TBD |
CATI Supervisor |
ICF International 9300 Lee Highway Fairfax, VA 22031 |
(703) 934-3603 |
Bob Dalrymple |
Senior Analyst for the SNAP Research and Analysis Division (Project Officer) |
USDA Food and Nutrition Service, Office of Policy Support 3101 Park Center Dr. Alexandria, VA 22302 |
(703) 305-2122 |
Michael Jacobsen |
NASS Reviewer |
USDA, National Agricultural Statistics Service, Statistical Methods Branch 1400 Independence Ave., S.W. Washington, DC 20250 |
(202) 609-0901 |
1 Previous OMB clearances include #0584-0299 and #0584-0074. See Attachment E for further detail.
2 Including the District of Columbia, Guam, and the U.S. Virgin Islands.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Yaeko Tise |
| File Modified | 0000-00-00 |
| File Created | 2021-01-29 |