Report on Pilot Study of Key Informant Interviews

Rural Community Wealth and Health Care Provision Survey

Pilot Phase Report: Key Informant Semi-Structured Interviews

USDA Economic Research Service and

Iowa State University Survey & Behavioral Research Services

October 2014

The following report is in reference to:

ICR Reference #201310-0536-001

OMB Control #0536-0072

OMB Action Date 01/23/2014

The Rural Community Wealth and Health Care Provision Survey is under the direction of John Pender, Senior Economist with the USDA Economic Research Service (ERS) and principal investigator (PI) for the project. Data collection is conducted through a cooperative agreement with Iowa State University’s Center for Survey Statistics & Methodology (CSSM) and Survey & Behavioral Research Services (SBRS).

The primary purpose of the study is to provide information about how rural small towns can attract and retain health care providers, considering the broad range of assets and amenities that may attract providers. The secondary purpose is to provide information on how improving health care may affect economic development prospects of rural small towns. ERS seeks to address these purposes by obtaining input from community leaders (key informants) and primary health care providers in 150 sampled rural small towns and by conducting secondary analysis of existing health and economic indicators.

A pilot study with 12 of the 150 communities is being conducted and must be reviewed and approved by OMB prior to implementation of the study in the remaining 138 communities, per the Notice of Office of Management and Budget Action. This report describes the results of the semi-structured telephone interviews with Key Informants in the 12 pilot communities.

Pilot Key Informant Semi-Structured Interview Procedure

Potential key informants were identified by a combination of web searches, review of community health care facility information from National Provider Identifier listings, and occasional telephone calls to local facilities or organizations. In keeping with the protocol described in the ICR, SBRS staff mailed an advance letter with project brochure to 2 or 3 potential key informants in each of the 12 pilot towns. Efforts were made to conduct semi-structured interviews with these individuals. Additional letters/brochures were sent as needed to other potential respondents. The goal was to interview a minimum of 2 and a maximum of 4 key informants from each town.

Four SBRS professional staff members with significant interviewing experience were trained to conduct the semi-structured interviews. The interview questions were programmed in an Access form which allowed easy movement from one tabbed page to another. The form was designed to maximize the flexibility needed for a semi-structured interview. The contact information of potential key informants was loaded into the Access database, with additional respondents added as necessary.

Interviews were conducted from a designated FISMA-approved computer lab within the SBRS facility. Questions were read as written to respondents; however, in keeping with standard semi-structured interview procedures, interviewers had the flexibility to probe for more information, to clarify questions, and to move around within the interview questions based on the responses obtained.

Pilot Study Outcomes and Respondent Burden

Overall, the key informant semi-structured interview procedure worked quite well. Web searches proved to be the primary source of potential key informants, and few clarification telephone calls were needed or made. SBRS staff learned that sometimes referrals from other key informants led to the identification of additional knowledgeable respondents not identified by other means.

The process of contacting key informants was more time-consuming than anticipated. Some of the telephone numbers obtained online were not in service and some were never answered. Receptionists could be either helpful conduits or barriers. It was difficult to know when to stop attempting to contact certain cases and when to send additional letters to others. Some respondents felt they were not knowledgeable enough to be helpful but referred SBRS interviewers to other individuals, and this generally proved to be very useful. The processes of sending additional letters/brochures to new cases or referrals elongated the timeline more than originally planned for the pilot. However, it is anticipated that the current timeline for the additional 138 communities and minor adjustments to the scheduling of new potential contacts should accommodate the process more efficiently. Overall, the number of potential key informants attempted for the pilot was less per town than the estimate described in the ICR. Attempts were made to contact an average of 4.5 people per town (54 people from 12 towns), while the ICR estimated that attempts would be made to contact an average of 6 people per town.

As of this report, 21 key informant interviews have been completed (9 towns with 2 interviews, 3 towns with one interview). Ten of the completed interviews were conducted with leaders in local health care and 11 were conducted with community leaders. Two additional cases are scheduled. Throughout the data collection process, up to 13 call attempts were made to individual key informants, some of which were answered by receptionists or assistants.

The average interview length was 33 minutes (ranging from 20 to 60 minutes). Interviews generally went well with few problems. De-identified data for 18 key informant interviews was delivered to the ERS PI, who determined that the information obtained from the key informant interviews was on target with project goals. Three potential minor revisions were identified, which are described in the Proposed Changes section below. There were no indications that any changes are needed in the Health Care Provider mail survey.

Table 1 shows the number of clarification calls made prior to Pilot Study data collection as well as the number of potential key informants attempted, the outcomes, and the burden in minutes. The total burden for the Pilot Study to date is 1146 minutes, including 60 minutes total for clarification calls and 1086 minutes for attempted and completed key informant interviews. Assuming that both of the pending cases are completed and that the average burden for those cases is the same as for the already completed cases (9 minutes for pre-interview and 33 minutes for the interview), the total burden for the key informant interviews in the 12 Pilot Study communities would be 1170 minutes, or 19.5 hours. Assuming that the same average burden would be required in the remaining 138 communities, the total burden for the key informant interviews would be 244 hours (19.5 hours for 12 communities x 150/12). This is one-fourth of the total burden estimated in the ICR for the key informant interviews (975 hours). This is because the burden estimate in the ICR conservatively used the maximum number of respondents for each community and conservative estimates of the average time required per respondent.

To calculate the response rate, we divide the number of completed surveys (21) by the sum of the number of completed surveys, the number of refusals (6), and the number of cases for which the maximum number of calls was reached (10). The estimated response rate is 57%, somewhat below the rate assumed in the ICR for the key informant interviews (67%).

Table 1. Key Informant Interview Attempts, Outcomes, and Burden

Outcomes	Number	Average Minutes per Case	Total Minutes
Clarification calls made	10	6	60
Key Informants Attempted	54	20.1	1,086
Completed Interviews	21	Pre-Interview: 9 Interview: 33	883
Referrals	7	5.3	37
Not Eligible 1	3	2.3	7
Refusals	6	9.0	54
Maximum Calls – no more attempts to contact	10	7.5	75
Cases Tabled 2	5	2.2	11
Cases In Process	2	9.5	19

¹Not Eligible includes people who have moved out of the area and/or no longer fulfill a role applicable to the project

²Cases Tabled includes people who received the project advance letter and one or more call attempts, but who did not need to be pursued because sufficient interviews were already completed in that community. As a result, work on these cases was halted before the typical required number of call attempts was made.

Validation of Community-Level Health Care Provider Sampling Frames

The 21 key informants interviewed in the Pilot communities were asked to verify the accuracy of the sampling frame (list) of Health Care Providers developed for their community by SBRS. This process proved to be effective. Some community leaders did not feel qualified to verify all of their local health care providers, but each community’s sampling frame was validated by at least one respondent. Provider lists for eight of the twelve communities were verified over the telephone. The list was emailed to respondents in three other communities and response was received. One community has no health care providers so there was no list to validate; although we did verify that fact with the key informant.

In three communities, no provider list revisions were needed. Additions and deletions were made to other provider lists based on information provided by key informants. SBRS staff believes that the sample frame development process works well and that provider lists can be effectively verified by appropriate key informants in each community. The only change recommended for this process is to more clearly specify the definition of eligible health care providers as described in Q35 in Table 2.

Proposed Changes

There are no recommended changes to the project protocol described in the ICR. The Pilot Study indicates that the project protocols and operational procedures are sound and will provide the information needed to meet project goals.

Key Informant Question Changes: SBRS proposes revisions to three items in the Key Informant semi-structured interview. They are described in Table 2.

Table 2. Proposed Key Informant Interview Changes

Question #	7
Original Q:	How knowledgeable are you about issues related to health care provision in this community? On a scale from 1 to 5, if 1 means you are not knowledgeable at all and 5 means you are very knowledgeable, which number would you choose?
Rationale for change:	Respondents have been answering open text questions and seemed to have trouble switching to the numbered scale. We propose maintaining 5 response options but labeling them with words rather than a numbered scale.
Proposed Revision:	How knowledgeable are you about issues related to health care provision in this community? Would you say you have No Knowledge, a Little Knowledge, Some Knowledge, Quite a Bit of Knowledge, or a Great Deal of Knowledge?
Question #	32
Original Q:	In your opinion, how important is it for your community to actively try to recruit and/or retain health care providers? (Please explain your answer.)
Rationale for change:	As open text, this question is redundant for many respondents. Providing 5 response options would provide variety and be more useful for comparison purposes.
Proposed Revision:	How important is it for your community to actively try to recruit and/or retain health care providers? Would you say it is Not at All Important, Slightly Important, Moderately Important, Quite Important, or Very Important?
Question #	35
Original Q:	Those are all the specific questions I have for you. But I have one more request. We would like to contact (some) health care providers who work in your community, and we want to make sure that we have accurate information about who is currently working there.
Rationale for change:	This item is a request for assistance in verifying the list of health care providers that SBRS has developed for the community. The request needs to include the definition of primary health care providers used for the project so respondents can respond more appropriately.
Proposed Revision:	Those are all the specific questions I have for you. But I have one more request. We would like to contact (some) health care providers who work in your community, and we want to make sure that we have accurate information about who is currently working there. We are using the Medicare definition of Primary Health Care Providers, so this list includes Physicians with a specialty of General or Family Medicine, Internal Medicine, Pediatrics, or Geriatrics. We also include Dentists, Physician’s Assistants, Nurse Practitioners, and Nurse Midwives.

Summary

The completion of Key Informant interviews in the Pilot Study of 12 communities indicates that the project procedure specified in the ICR is sound. The process used for identifying knowledgeable key informants appears to be effective, and the development of sample frames of primary health care providers for each selected community has been verified as complete by knowledgeable individuals in the 12 Pilot communities. The semi-structured interview questions are providing the data required by the PI in order to effectively address project goals. The operational processes ran smoothly. The burden was significantly less than that estimated in the ICR, and the response rate was comparable to the ICR estimate. A few minor changes to the semi-structured questionnaire instrument are proposed based upon the Pilot Study.

5