FIST OMB Supporting Statement Part B Statistical Methods Final

2. Statistical Methods

1. Respondent Universe

Data for this collection will be collected through a voluntary survey administered to state and local reporting agencies that conduct background checks for firearm purchases and transfers. Currently, there are 30 state agencies and over 2,900 local agencies that conduct background checks on persons who apply to purchase a firearm or for a permit that may be used to make a purchase. As discussed previously, each state government determines the extent of its involvement in the NICS process. States may operate as a full POC that requests a NICS check on all firearm transfers originating in the state, as a partial POC that requests a NICS check on all handgun transfers (FFLs are required to contact the FBI for NICS checks for long gun transfers), or as a non-POC in which case FFLs are required to contact the FBI for NICS checks on all firearm transfers originating in the state. The data collection agent will maintain a list of state and local agencies that conduct background checks and their associated functions in the NICS process.

For this data collection, the only factor that would result in an agency’s ineligibility to participate in the survey is when it is not longer authorized to conduct background checks for firearm purchases. Agencies that are ineligible to participate in the survey due to changes in their background check reporting functions are removed from the sample and are not included in the nonresponse rate. The nonresponse rate is composed only of eligible agencies that have elected to not participate in the survey.

To estimate the application and rejection rates within a given area, state and local checking agencies will be stratified by size of the population served: state agencies that served an entire state population; local agencies that served a population greater than 100,000; local agencies that served a population between 10,000 and 100,000; and local agencies that served a population of less than 10,000. Population size is based the most current information from the Census Bureau, and the population categories were chosen to be consistent with those used by the FBI when conducting similar studies. To determine the agency population, the stratification classification of the county will be based on the size of the largest city within the county. If cities within a county were conducting their own background checks, their populations will be subtracted from the county population. If a municipal agency provided services for other selected municipalities, then populations from those municipalities will be added to the populations of the reporting municipalities. If an agency relied upon other jurisdictions to conduct background checks, they will be replaced by those other jurisdictions.

In 2009, the sample for the FIST survey was selected from the total population of 30 statewide agencies and over 2,900 local checking agencies across the nation that conducted background checks for firearm purchases and transfers. A total of 816 agencies were surveyed, including all 30 statewide agencies and a stratified random sample of local agencies. Overall, 559 agencies provided data for a response rate of 69%.

Background Checks for Firearm Transfers, 2009

Population Served by Agency	Total	Sample	Responses	Response Rate
Total	2,951	816	559	69%
Statewide	30	30	30	100%
Under 10,000	1,656	364	236	65
10,000 to 100,000	1,155	373	264	71
Over 100,000	110	49	29	59

2. Information Collection Procedures

The information collection will be administered through a survey to state agencies that serve an entire state population and a stratified sample of local checking agencies randomly selected from the total population. Recognizing that this is a voluntary survey, all reasonable efforts will be made to make the data collection process as seamless and convenient as possible for the respondents. The data collection agent will maintain a list of reporting agencies and their preferred data submission methods so that data requests and follow up efforts can be tailored to individual agencies. Respondents may choose to submit their data on paper (fax or regular mail), diskette, or electronically using a fillable pdf (email). The data collected will be summary statistics of an administrative nature, and will not allow for the identification of any individual.

Data from the state and local reporting agencies will be collected through various measures based on the individual agency’s characteristics and procedures, as well as its preferred reporting method in past collections. For both state and local checking agencies, regardless of the way the agency has elected to submit data in the past, a letter will be sent annually to explain the purpose of the information collection and uses of the data obtained, and request the agency’s ongoing participation in the program. All efforts will be made to personalize each letter, and the letters will be prepared for signature by the data collection agent and/or the BJS Program Manager. The letter will also clearly denote that participation in the information collection is voluntary and acknowledge that there are no assurances to confidentiality as the data collected are available in the public domain. The data collection agent will maintain a contact list for respondent agencies and verify the addresses each year to ensure the accuracy of contact information.

State agencies:

The data collection process for state reporting agencies will modified based on individual agency characteristics to make the process as convenient as possible for each respondent. Data will be collected from state agencies through one of three measures: reports, state websites, or survey. Some state agencies choose to regularly submit reports to the data collection agent, from which the agent will extract the relevant data. These reports are routine reports prepared by the state for legislative purposes and contain various data related to criminal justice statistics. Submitting reports to the FIST program is a voluntary decision made by the responding state agency because it determined this was the most efficient means to provide the data. Other state agencies publish data on background check activities on their state website. The data collection agent will maintain a list of state agencies that publish this information online and will make all efforts to collect the information directly from their websites in order to reduce the respondent burden. In these two circumstances, the data collection agent will complete the Excel survey spreadsheet (Attachment III) with the relevant figures compiled from the data published in state reports or online, and fax the completed forms and data request asking the respondent agencies to review and verify the numbers (Attachment IV). The respondents will be asked to revise figures or update missing information directly on the Excel spreadsheet, as applicable, and return the spreadsheet and signed form to confirm the accuracy of the information. The names will be cross-checked against the contact list to ensure that the individual verifying the data has the authority to do so. There will be no duplication of effort required for the state agencies that choose to submit information through these methods, and the associated burden will be the time spent emailing the reports to the data collection agent and/or verifying data accuracy at the end of the reporting period.

For state agencies that do not elect to submit monthly reports and do not publish data online, the data collection agent will fax the appropriate letter (Attachment V) and Excel survey spreadsheet to the reporting agency and request that the agency POC complete the totals for each category and return both the spreadsheet and signed review. Respondents will be encouraged to submit monthly totals, but cumulative aggregate totals by category will be requested if monthly data are not available. Faxing the data request has been the preferred mode of transmission, although the materials can also be mailed to the agency. When mailed, a self-addressed stamped return envelope will be included. Respondents will be asked to return the completed survey to the data collection agent via fax, diskette, mail, or electronically. The same practice will be in effect regarding cross-checking the name of the respondent against the contact list maintained by the data collection agent. The associated respondent burden will be the time spent completing and returning the form to the data collection agent.

In some states, one statewide agency conducts background checks for purchase and another agency (or division within an agency) issues ATF-approved permits. Controls will be put in place to ensure that state populations are not counted twice in the estimation process. This situation of dual agencies conducting background checks does not occur among local agencies.

Local agencies:

The collection methodology for the sample of local agencies will be collected through a tally sheet approved by OMB in 2007 and a survey cover letter explaining the purpose of the information collection (Attachment VI). Two tally sheets (the survey instruments) will be sent via fax or mail to local agencies selected for the survey based on the type of background check they are responsible for completing: one form collects background check data on carry and concealed permits (Attachment VII), and the other collects data on purchase permit statistics (Attachment VIII). The data collection agent will maintain a listing of local agencies and the type of background check each agency is responsible for conducting, and will send the appropriate form(s) based on the function of each individual agency. In all cases, the data collected will be summary counts and will not allow for the identification of any individual.

The content of the tally sheet to collect Purchase Permit statistics has not changed since receiving OMB approval in 2007. The presentation of the form will be modified slightly to accommodate the change to the data collection schedule from twice to once annually to enable respondents to report the annual aggregate totals instead of monthly totals. A second tally sheet will be sent to applicable reporting agencies to collect data on Concealed/Carry Permit statistics, as applicable. The presentation and reporting directions are the same for both tally sheets. Mailing the forms has been the preferred delivery method for local agencies in the past collection, but respondents will also be encouraged to submit the data via diskette or email if these methods are more convenient. The associated respondent burden will be the time spent completing and returning the form to the data collection agent.

FBI and ATF data:

Transaction data from the FBI NICS and data from the ATF Brady Operations on reasons for denial and appeal information, firearm retrievals, and referrals for investigation and prosecution will be collected and integrated with the data received from state and local agencies to complete a comprehensive source of background check data from application to denial to post-denial activities. The data collection agent will continue to extract the relevant data from FBI reports submitted to FIST throughout the collection period. The data collection agent will continue the process of contacting ATF directly to request and compile the relevant data on post-denial activities.

Future directions and Outreach Efforts:

As new technologies become more available to easily and safely transmit information electronically, a goal of the project will be to create and implement a web-based reporting form that enables respondents to enter data online to simplify the record tabulating functions of the agency and further reduce the reporting burden. The data collection agent will also follow up via phone or email email with reporting agencies to verify the accuracy of reported figures prior to submitting the information to BJS, and/or address any questionable figures. Outreach efforts will be made to the responding agencies to encourage participation, answer questions about the survey, and provide technical assistance as needed.

Data collection strategy in 2011 (for 2010 data)

April to May – The data collection agent will research and update its comprehensive list of state contacts, and will research and confirm any changes in the background check responsibilities of respondent agencies to determine the population universe. The data collection agent will also collaborate with the FBI and ATF to receive relevant data on firearm background check activity and post-denial activities.

June

Week 1: A prenotice letter will be sent via mail, email, or fax based on the contact information available for the agency and the agency’s preferred mode of communication.

Week 2: A survey will be sent via mail, email, or fax based on the agency’s preferred mode of communication. A detailed cover letter will be included to explain the importance of a response and indicate alternate submission modes. A requested response deadline is provided (2 weeks after the survey is sent).

Weeks 3- 4: A thank you note will be sent via mail, email, or fax to express appreciation for responding to the survey, even if the survey has not yet been returned.

The data collection agent will enter data into the project databases as it is received and will continue to review state websites and FBI reports to extract published data. The data collection agent will update contact information for agencies as needed.

July

Week 2: A replacement survey will be sent via mail, email, or fax to agencies that have not yet responded. This will be sent 2-4 weeks after the initial survey is sent.

Week 3: A follow up phone call will be made to agencies who have not responded to the survey (if a telephone number is provided). Otherwise, a request will be made via mail, email, or fax.

The data collection agent will enter data into the project databases as it is received and will continue to review state websites and FBI reports to extract published data. Data verification efforts continue. The data collection agent will update contact information for agencies as needed.

August

Weeks 1-2: The data collection agent will make one final attempt to reach the reporting agency. The mode of outreach will vary depending on the history of past attempts made.

Week 4: Data entry concludes.

The data collection agent will enter data into the project databases as it is received and will continue to review state websites and FBI reports to extract published data. Data verification efforts continue. The data collection agent will update contact information for agencies as needed. Data entry will conclude at the end of August

September to October

Work to produce the estimates will begin. Data processing and analysis continue. Data verification will continue as needed.

November

Final reports and statistical tables will be completed and submitted to BJS for review. Efforts to maintain the master contact list and research state laws for changes in firearm background check procedures continue.

Throughout the data collection process, the data collection agent will maintain a comprehensive record of all follow up and reporting activity and log details of when data is received, from whom, by what means (fax, email, etc.) and applicable changes in address and other contact information. This will be done to ensure that duplicate requests are not made to agencies and that the agency’s preferred mode of submission is noted for subsequent years. Five attempted contacts will be made to each agency before it is considered to be nonresponsive. Specific dates will vary annually depending on holiday and staff schedules. The data collection agent will vary the modes of outreach so the reporting agency receives at least one phone call, one email (if an email address is available), and one fax or letter request.

December to March

BJS and the data collection agent will continue to evaluate the FIST program design and sampling plan. The sample will be redrawn to prepare for the 2011 data collection.

The data collection schedule may be revised and adjusted to accommodate data collection activities based on the selection of the data collection agent for FY 2011.

3. Development of Estimates

National estimates of applications and denials will be developed using population weighting factors. When an agency does not provide data for all months, a simple linear extrapolation or interpolation will be used to generate a 12-month total. Agencies with rejection rates over four standard deviations above the average standard rejection rate will be classified as outliers and their data will not be used for projection of estimates. In addition, rejection rates that cannot be determined with sufficient accuracy will not be used.

The accuracy of the estimates depends on two types of errors: nonsampling and sampling. For this data collection, nonsampling error may occur from the following: nonresponse; differences in the methods checking agencies use to process, code, store, and retrieve their information; differences in interpretation of the survey questions; and activities that delay personnel from completing the survey. In any sample survey, the full extent of nonsampling error is never known. However, continued steps will be taken to minimize the potential for error, such as periodic telephone follow-up calls and emails to encourage responses, answer questions about the request, and generally assist in reporting the information in a useable format. Upon receipt of the completed survey, the data collection agent will review the submission and follow up directly with the agency POC to discuss any questionable figures. Respondents will also be asked to return a signed review with their completed survey forms to verify the accuracy of the reported totals.

4. Procedures to Maximize Response Rates

The overall response rate in 2009 was 69 percent, which was consistent with the rate in 2008 but lower than rates seen in past collections (85 percent in 2005). While 100 percent of state agencies submitted responses in 2009, the number of local agency respondents dropped off from past years. This decrease was due in part to a reduction in the number of local agencies eligible for the survey as a result of changes in policies related to issuing permits. Additionally, as is a challenge associated with any voluntary surveys, there are no tangible incentives for the respondents to submit the data. The higher response rate in past years may also be attributed to efforts made by a marketing research firm previously supported under the grant to make follow up calls to reporting state and local agencies. Due to funding restrictions, the marketing research firm services are no longer employed and agencies are not being contacted as frequently about submitting responses for the FIST survey. Coupled with less consistent outreach by the marketing firm, the change in schedule to a once annual data collection also decreased contact between reporting agencies and the FIST data collection agent, which may have contributed to the lower response rate.

While the overall response rate has declined in previous years, it is important to note that the greatest volume of applications and rejections are transacted by the state agencies. As noted, the response rate for state agencies in the 2009 collection was 100 percent. The responses received by the state agencies drive the volume of responses received and the current response rate is considered to be well above the level needed to be considered statistically significant.

An ongoing priority of this data collection will be to increase the response rate to 80 percent or greater, and a continued focus will be to remain flexible in the ways that data are accepted by both the state and local reporting agencies in order to encourage participation and limit the burden on respondents. To achieve this goal, concerted efforts will be made by the data collection agent to encourage responses from participants, including identifying ways to engage in ongoing outreach to respondents and identifying ways to further decrease the reporting burden. BJS anticipates that moderate increases in the response rate will be realized over the next several years based upon the availability of funds to allocate to outreach techniques such as the marketing research firm services employed under previous awards. The FY2011 FIST solicitation has established maintaining a 100 percent response rate for state agencies and achieving an overall 80 percent response rate as performance benchmarks for the selected data collection agent. It is anticipated that increases in the response rate will in part be based on additional funding made available for concerted follow-up efforts.

5. Test of Procedures

No such tests are planned.

For information on statistical methodology, conducting the survey, and analyzing the data, contact:

Allina D. Boutilier

Bureau of Justice Statistics

810 Seventh Street, N.W.

Washington, D.C. 20531

(202) 307-0765 or [email protected]

4