U.S. Department of Agriculture
Food and Nutrition Service
Feasibility of Tribal Administration of
Federal Nutrition Assistance Programs
Request for Clearance
Supporting Statement and
Data Collection Instruments
Part B: Collection of Information Employing Statistical Methods
Project Officer: Jenny Genser
Office of Policy Support
SNAP Research and Analysis Division
3101 Park Center Drive
Alexandria, VA 22302
Phone: 703-305-2559
E-Mail: [email protected]
February 2, 2014
TABLE OF CONTENTS
PART b: Collection of Information Employing Statistical Methods 1
B.1. Respondent Universe and Selection Methods 1
B.2. Procedures for the Collection of Information 2
B.3. Methods to Maximize Response Rates and the Issue of NonResponse 4
B.4. Tests of Procedures 6
B.5. Consultants 6
Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.
The purpose of this section is to document any statistical procedures used for the “Feasibility of Tribal Administration of Federal Nutrition Assistance Programs” study. In February 2014, Congress passed the Agriculture Act of 2014 (P.L. 113-79) which calls for a study to determine the feasibility of Tribal administration of Federal food assistance programs, services, functions, and activities in lieu of State agencies or other administrating entities. USDA’s Food and Nutrition Service (FNS) is requesting approval from the Office of Management and Budget (OMB) to conduct the Feasibility of Tribal Administration of Federal Nutrition Assistance Programs study. This study contains two data collection activities: (a) a multi-modal survey of Tribal leaders of all Indian Tribal Organizations (ITOs) and (b) in-person interviews with 2 Tribal leaders and 6 program administrators at each of 16 ITOs.
The surveys will be administered to the universe of Federally-recognized Tribes (566). There will be no sampling. Based on our work to-date with this population, we anticipate approximately 2 leaders per Tribe, for a total of 1,132 individuals1. For the site visit, FNS will purposively select ITOs. Onsite, researchers will interview up to 2 Tribal leaders and 6 program administrators at each ITO.2 Eight individuals at each of the 16 ITOs brings the interview respondent total to 128. The purposive sample of ITOs will be based on early survey data and ongoing Tribal Consultations. This will allow researchers to visit sites with varying levels of experience administering FNS or FNS-like programs. Neither survey nor interview respondent groups are based on a statistical sample. Data from this collection are not intended to provide results generalizable to all ITOs.
Based on previous research with this population (Kim et. al, 2008; Lavelle et. al, 2009)3 FNS estimates a survey response rate of 20 percent. Section B3 describes efforts that will be undertaken to ensure a high response rate. For the semi-structured interviews during site visits, response rates are estimated to be 100 percent.
Describe the procedures for the collection of information including:
Statistical methodology for stratification and sample selection
Estimation procedure
Degree of accuracy needed for the purpose described in the justification
Unusual problems requiring specialized sampling procedures
Any use of periodic (less frequent than annual) data collection cycles to reduce burden
B.2.1 Statistical Methodology for Stratification and Sample Selection. The survey component of this research is a census. There is no sampling. For the selection of site visit ITOs, FNS will work with researchers to purposively select up to 16 sites. The biggest factor in the purposive selection will be the Tribes’ experience administering either FNS or other large-scale programs. In order to best answer this study’s research question, researchers will seek to visit ITOs that are experienced, somewhat experienced, and not experienced with complex program administration. Additional ITO selection factors will be determined with FNS and with Tribal leaders using the consultative process. These factors may include geographic location, size of ITO, ITO resources, and ITO demographics. Survey data may also suggest factors to consider in selecting sites to visit.B.2.2 Measurement/Estimation Procedures. As a census, no weighting or adjustment will be needed for the survey. No measurement or estimation procedures are being used in the purposive sample.
For quantitative results, all data management and cross-tabulations/frequency distributions will be conducted using STATA. Variations in output, per type of analysis, will depend on what measures are appropriate for the variable and the measurement level (i.e., nominal, ordinal, or scale) for each defined variable. For example, a nominal measure (e.g., nondirectional categories, related to the respondents’ responsibilities or background) will be analyzed using frequencies and percentages. For an ordinal measure (e.g., directional categories, such as strongly agree to strongly disagree), we will produce frequencies and percentages for the variables. Finally, for a scale measure (e.g., a numerical value, such as years of experience with FNS programming), we will produce a mean, median, or other descriptive measure.
Results of this survey data are not designed to be nationally representative. Given the quick turnaround mandated by Congress and anticipated response rate, results will compare capacity metrics gathered from a representative set of ITOs to the administrative requirements of select FNS nutrition assistance programs.
Site visit interviews will be conducted by pairs of senior staff members who have expertise in both FNS programs as well as research in the Native American context. All interviews will be recorded, with respondent permission, so that researchers have a means of clarifying any discrepancies in notes. While an initial question-based coding scheme will be ready in advance of the visits (so that data entry and analysis can be ongoing once visits begin) this will be augmented continually as new themes emerge. Senior researchers will review coding throughout the process to ensure consistency. Due to time constraints, we do not anticipate any re-contact.
B.2.3 Degree of accuracy needed for the purpose described in the justification. Because of the quick turnaround mandated by Congress, this study seeks to provide basic capacity metrics gathered from a representative set of ITOs. The survey will be administered to all ITOs –more than a representative set. Given Congressional timeline, this approach will provide the best accuracy possible.
B.2.4 Unusual Problems Requiring Specialized Sampling Procedures. This does not apply because no sampling procedures are being used for the survey data. The site visit locations will be purposively selected according to factors including geographic location, size of ITO, ITO resources, and ITO demographics.
B.2.5 Any Use of Periodic (Less Frequent than Annual) Data Collection Cycles to Reduce Burden. Not applicable: This project timeline is one year, as mandated by Congress.
Describe methods to maximize response rates and to deal with issues of nonresponse. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.
Since a 20 percent response rate is anticipated for the survey, a multipronged strategy is planned to ensure the best possible response rates:
Multiple mode options: the survey will be offered in three modes: paper, web, and telephone. This choice offers flexibility of time and place to respondents, and accommodates preferences for oral communication. Table B.1 explains the multi-mode administration.
Table B.1:
Survey Administration Task |
Attachment |
Mode |
Advance Notice of Survey & Information Sheet |
B.1 |
|
Survey Invitation |
B.2 |
E -mail Hard-copy (mail) |
Survey |
B |
Electronic (web-based) Hard-copy Telephone |
Survey Follow-up |
B.3 |
Electronic (web-based) Telephone |
Individuals who opt to complete the survey over the phone will do so with a trained individual from IMPAQ’s Survey Center, who will complete a web-version of the survey based on verbal responses.
Advance Notification from USDA: An advance-notification letter will be prepared on USDA letterhead and emailed to Tribal leaders along with an information sheet about the study. (See Attachment B.1). These materials will briefly explain the purpose of the study and explain why it is important for respondents to participate. Stating the sponsorship of the survey helps to engage sample members by providing immediate assurance that the survey is legitimate and not an attempt to sell them something.
Respondent May Ask Colleagues for Help: The survey instrument will include instructions that encourage respondents to seek assistance with any items they choose. The unit of analysis is the ITO, not the individual. Since some items contain specialized knowledge, the ability for respondents to “ask the expert” as needed will reduce the overall cognitive burden.
Survey Brevity: The survey has been limited to 30 minutes by including the minimum number of questions that will yield the required data.
Monetary Incentive: There is a monetary incentive to complete the survey. Individuals who submit completed surveys will have their name entered into a lottery to win one of ten $200 VISA gift cards.
Non-Response / Refusal Follow-Up: For non-respondents or refusals, IMPAQ has planned extensive follow-up. Non-respondents will each receive 3-5 follow-up attempts. Each one of these will be made using two modalities: telephone and e-mail. Follow-ups will be done weekly, beginning once the survey has been in the field for one week. A paper version of the e-mail follow-up is presented in Attachment B.3.
The response rate for the site visit interviews is expected to be 100 percent.
Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.
Several Tribal community members volunteered to review questions early in the instrument development phase (see Attachment F). The survey and site visit protocol incorporate the advice received. Specifically, an item that asked about staff education levels was removed, since volunteer reviewers uniformly found it offensive. Additional context was added to question introductions and transitions.
Each data collection instrument (survey and interview protocol) was then pre-tested externally with Tribal leaders or staff to determine whether questions were written appropriately and captured data most relevant to the research questions and objectives. Eight individuals tested the survey instrument, and 5 individuals tested the interview protocol. These individuals are part of the survey universe and are potential site visit interviewees. As such, they provided input to the research team about the survey and interview protocol questions, such as which questions were unclear or difficult to answer. As subject matter experts they could provide input on the content of the questions, including which questions should and should not be included. This feedback was then used to refine and finalize the data collection instruments included in this package for OMB clearance. The Tribal leaders who advised us on the survey and interview protocol content are listed with contact information in Attachment F.
Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
No individuals outside the evaluation project were consulted on statistical aspects of the design. FNS has contracted with IMPAQ International, LLC to conduct this study. IMPAQ LLC has subcontracted with Bowman Performance Consulting and WMRA for expertise in conducting evaluations in the Tribal context. Table B.1 identifies the individuals who will be responsible for collecting and analyzing the data. The Project Officer for the contract providing funding for the evaluation, Jenny Genser, will be responsible for receiving and approving all contract deliverables. Her contact information is also included in Table B.2.
Table B.2:
Individuals Responsible for Statistical Aspects and Data Collection and Analysis
Name |
Title (Project Role) |
Organizational Affiliation and Address |
Phone Number |
Steven Garasky |
Vice President (Project Director) |
IMPAQ International 10420 Little Patuxent Parkway Columbia MD 21209 |
(443) 259-5500 |
Kassim Mbwana |
Research Associate (Co-Principal Investigator) |
IMPAQ International 10420 Little Patuxent Parkway Columbia MD 21209 |
(443) 259-5500 |
Nicole Bowman |
President (Co-Principal Investigator) |
Insight Policy Research 1901 N. Moore Street, Suite 204 Arlington, VA 22209 |
(703) 504-9488 |
Anne Chamberlain |
Senior Research Associate (Data collection and analysis) |
IMPAQ International 10420 Little Patuxent Parkway Columbia MD 21209 |
(443) 259-5500 |
Samuel Ampaabeng |
Research Associate (Data collection and analysis) |
IMPAQ International 10420 Little Patuxent Parkway Columbia MD 21209 |
(443) 259-5500 |
Jenny Laster Genser |
Program Analyst |
Office of Policy Support SNAP Research and Analysis Division 3101
Park Center Drive |
(703) 305-2559 |
Audra Zakzeski |
NASS Reviewer |
National Agriculture Statistics Service 1400 Independence Ave, SW Washington DC 20250 |
703-877-8000 X 121 |
1 Some Tribes are grouped into consortia with shared leadership, or affiliated with other Tribal organizations that have shared leadership. Such clustering into Indian Tribal Organizations (ITOs) may reduce our total number of respondents. Still, throughout this proposal, we maintain use of the total number of Federally-recognized Tribes (566) to generate the anticipated number of census respondents (1132 – 2 per Tribe).
2 We anticipate 2 leaders and 6 staff will constitute the maximum number of relevant respondents to interview during site visits.
3 Kim, S.Y., Tucker, M., Danielson, M,. Christopher, J,. Pelagie, S., & Holly, S. (2008). How can PRAMS Survey Response Rates be Improved Among American Indian Mothers? Data from 10 Sites. The Journal of Maternal and Child Health, 12,S119-S125.
Lavelle, B., Larsen, M. & Gundersen, C. (2009). Research Synthesis: Strategies for Surveys of American Indians. Public Opinion Quarterly, 73 (2), 385-403.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Yaeko Tise |
| File Modified | 0000-00-00 |
| File Created | 2021-01-25 |