Supporting Statement B
Evaluation of the Frontier Community Health Care Coordination Grant
OMB Control No. 0915-XXXX
Identify the strengths and challenges of implementing the FCHCNC Grant
Assess the effectiveness of the grantees implementation of the FCHCNC Grant
Determine client satisfaction
Evaluate whether the intervention is meeting FCHCNC Grant goals
Assess health care utilization and cost savings associated with FCHCNC Grant participation
The evaluation will collect data from key stakeholders, grantee sites and clients using the following methods:
In-person and telephonic interviews will be conducted with key stakeholders (e.g., care transition coordinator, health care providers)
Community Health Worker (CHW) data collection forms
Client satisfaction surveys will be completed by individuals receiving health care through the pilot program.
1. Respondent Universe and Sampling Methods
In-Person Interviews/ Key Informant Interviews
The total size of the potential respondent universe is about 132 people from 11 participating pilot sites. There will be two types of entities covered in this data collection process – grant recipient (state government employees?) and intervention site participants (hospital administrators, health care providers, CHWs, care transition coordinator (CTC) and clients (Medicare and/or Medicaid beneficiaries)).
Key informants will be identified and selected with the help of the Montana CTC. Since the one CTC, the 11 CHWs (one at each site) and the two grantee program managers at the Montana Department of Public Health and Human Services are most closely engaged in the pilot program, we will interview all 14 of these key informants. To reduce the interview burden for the other types of informants, we will identify two key hospital administrative staff, two health care providers and two clients from each pilot site location from all those engaged or participating in the program. The hospital administrators, health care providers and clients to be interviewed will be chosen based on involvement in the program, availability and willingness to be interviewed.
In summary, during the course of the two site visits to be conducted as part of the evaluation, we anticipate interviewing a total of 80 people (see Table 1). We expect to have 100% participation based on the interviews planned for the site visits. Key informants are critical to the success of the program implementation and contribute essential information regarding programmatic successes and challenges.
Table 1: Key informant interviews
Form Name |
Number of Eligible Subjects |
Number of Respondents |
Care Transition Coordinator (CTC)Interview Protocol |
1 (1 staff) |
1 (1 staff) |
Community Health Worker Interview Protocol |
11 (1 CHW per site) |
11 (1 CHW per site) |
Grantee Interview Protocol |
2 (2 staff) |
2 (2 staff) |
Hospital Administrator Interview Protocol |
2-4 per site |
22 (2 per site) |
Primary Care Provider Interview Protocol |
2-4 per site |
22 (2 per site) |
Client Interview/ Focus Group Protocol |
About 10 per pilot site |
22 (2 per site) |
Total |
132 |
80 |
CHW Data Collection
For the CHW data collection, the total universe of potential respondents is 11 CHWs. We will be collecting data from one type of entity (persons) for this data collection. We will be surveying the 11 CHWs on a quarterly basis (see Table 2). The CHWs are the only entity in direct contact with the clients and, in their role, they have intimate knowledge of the client intervention goals, client recruitment methods and other programmatic details needed to conduct a comprehensive evaluation. There is a possibility that one or two sites may not have CHWs currently working at their site. Thus, we may not have all pilot sites represented in the evaluation. However, as mentioned above, we will survey all participating CHWs since each pilot site has had a unique experience implementing the FCHCNC Grant. We will work with the CTC and CHWs to obtain a 100 percent response rate.
Table 2: Community Health Worker Collection Respondents
Form Name |
Number of Eligible Subjects |
Number of Respondents |
Community Health Worker Interview Protocol |
11 (1 CHW per site) |
11 |
Total |
11 |
11 |
Client Satisfaction Survey
The data collection for the client satisfaction is focused on one entity type, clients who have completed the program. Due to delays in program implementation, the total universe of all clients served by the pilot program is estimated to be only 110 clients (or about 10 per site). In addition, to be eligible for the program, the individuals must be frail elderly individuals with multiple chronic diseases residing in frontier counties (counties with fewer than 6 people/mile). Therefore, several of these clients have moved out of their homes, become incapacitated or passed away. These factors have resulted in a lower than anticipated enrollment and completion rates of the clients. Therefore, all clients of all FCHCNC critical access hospitals (CAH) will be mailed the survey (see Table 3). The reasons are two-fold. First, we want to ensure we have a large enough population to evaluate and second, we want to ensure perspectives of all the clients are part of the evaluation. Consequently, the survey will be a census of clients and will not require any sampling design. However, we will conduct a careful analysis to assess non-response bias.
Given reasons stated above (i.e., delays in program implementation and inaccessibility of clients), we anticipate 85 participants to be surveyed. Past data collections using this approach have yielded response rates of 50 percent. Therefore we expect about 43 completed responses to be returned.
Table 3 Client Satisfaction Survey Respondents
Form Name |
Number of Eligible Subjects |
Number of Respondents |
Client Satisfaction Survey |
About 10 per site |
85 |
Total |
110 |
85 |
2. Procedures for the Collection of Information
In-Person Interviews/ Key Informant Interviews
There will be two site visits conducted during the course of the evaluation. The site visits will divide up the 11 participating pilot sites. Five pilot sites will be visited during the first site visit and the remaining six pilot sites will be visited during the second pilot site. Each interviewee will be interviewed once with the exception of the one CTC and the two grantees. These three individuals will be interviewed during both site visits to gather detailed information about program implementation activities and challenges. In addition, we will interview the other key informants as identified by CTC. The number and type of interviewees will be selected based on their role in the program implementation. We will interview all of the CHWs, dependent upon whether the positions are filled, to ascertain their experiences recruiting clients and their knowledge of clients’ intervention goals. To reduce interview burden for the hospital administrators, health care providers and clients, the CTC will identify two individuals in each group to provide feedback out of all the potential interviewees at each site. The hospital administrators, health care providers and clients to be interviewed will be chosen based on involvement in the program, availability and willingness to be interviewed.
In advance of the visit, the interviewees will receive a specific meeting time and be provided with a discussion guide/ interview protocol. The interviews will be conducted by Dr. Alana Knudson who has extensive experience interviewing rural populations about health-related programs. All interviews will be transcribed and coded for accuracy and analyzed to find key themes using NVivo.
As mentioned above, a subset of hospital administrators, health care providers, and clients will be selected for interviewing based on their involvement in the program and availability, rather than through a random sampling process.
The discussion guides /interview protocols are included in the Appendix.
CHW Data Collection
Client data will be collected quarterly from each CHW upon OMB approval. Given the program termination date in September 2014, there are only three more potential quarters to collect data. Data will be collected from each CHW with support from the CTC and the evaluation team. The data collection form will ask for details of each client’s intervention goals, recruitment activities, resources utilized during their program and client intervention start and end dates. Given the small number of participants, we will request that each CHW report on each client served. Information about all clients currently participating in the program is critical to understanding the effectiveness of the program, and ultimately, to evaluate the overall success of the program.
The CHWs will report program data to the CTC, who will provide quality control by ensuring that the data are complete and accurate before submitting to the evaluation team. Furthermore, the evaluation team will then review the submitted data and ask any follow up data-related questions (e.g., regarding missing data or data fields) to the CTC. The evaluation team’s goal is to have the most complete data possible for the quarterly reports.
Prior to the first quarterly data collection, a technical assistance training will be provided to the CTC and the CHWs. Hospital administrators and other key personnel will also be invited to attend to participate. The training will provide details on each data element needed and why it is important to collect the information for each client. The CHWs will be given a template and instructional sheet which explains how to fill in each data field. In addition to sharing the electronic data report template files, we will also make hard copies available in order to reach those individuals who do not have access to a computer or internet.
Given the small number of pilot sites and the small number of clients, all of the pilot site’s (11) CHWs will be selected to report on the activity of all of their clients. The CHWs’ direct knowledge of each client is foundational to evaluating this program’s goals and outcomes.
There will be no sampling method or statistical methods employed to collect this information due to small sample size and unique implementation aspects at each of the 11 sites.
The data template, template definitions and training slides are included in the Appendix.
Client Satisfaction Survey
The client satisfaction survey is a one-time survey that will be mailed to clients who have completed program. All clients who complete the program and for whom we have contact information will be mailed a survey. There are so few clients enrolled in this program that we will survey all participants who have completed the program in order to ensure an adequate number of responses. Our goal is to mail the survey to at least 85 participants and to obtain at least a 50 percent response rate (at least 43 respondents).
To increase the response rate, we will inform the CTC and CHWs when the surveys will be mailed out and ask them to encourage their clients to respond to the surveys and to be candid in their responses. In addition, we will send out reminder cards within 10 business days. A second mailing will be sent 10 business days after the reminder cards. For quality control, the data will be thoroughly cleaned prior to analyses. We will test for differences in characteristics (e.g., age, pilot site) of survey respondents and non-respondents to assess potential non-response bias.
There will be no sampling or statistical methods employed to collect this data.
The client satisfaction survey can be found in the Appendix
3. Methods to Maximize Response Rates and Deal with Nonresponse
In-Person Interviews/Key Informant Interviews
The interviewees will be selected based on their roles in the program. Interviewees who agree to participate will meet with the project team during the scheduled site visits. We anticipate having all invited interviewees participate. However, if an issue arises that prevents the scheduled interviewee from meeting with our team, we will offer to follow-up with a telephone interview following the site visit.
CHW Data Collection
In order to obtain a 100% response rate from the CHWs, the evaluation team will hold an initial technical assistance training and be available to provide technical assistance as need by the CHWs. Furthermore, the CTC has agreed to serve as the primary avenue to aid the CHWs in submitting completed forms and to strongly encourage participation by the CHWs. Each CHW will provide client data on a quarterly basis. The CTC and the evaluation team will review submitted data and follow up with any CHWs where missing data elements or other data details are needed. The goal is to have the most complete and accurate data from each CHW.
Client Satisfaction Survey
In order to increase response rates, we will send a reminder card to each client within 10 business days of the initial mailing. A second mailing will be sent to non-respondents 10 business days following the reminder cards. We will examine a potential non-response bias based on the responses we receive.
In conjunction with the mailing, we will inform the CTC and CHWs when the surveys are mailed. The CHWs can inform the clients that the survey will be coming in the mail to strongly encourage the participation in the survey to reduce the non-response rate.
Given our team’s past experiences with surveys conducted with similar populations coupled with the limited number of current clients, we expect a 50% response rate.
4. Tests of Procedures or Methods to be Undertaken
Given the limited number of pilot sites and clients, we did not pilot any data collection component without OMB approval with the exception of the client satisfaction survey.
Client Satisfaction Survey
In the initial phases of the evaluation, there was some concern regarding the ability of this frail elderly population to complete a mail survey. In order to explore this population’s ability to complete a mail survey, the evaluation team mailed surveys to three participating clients. Two of the three surveys were completed and returned to our project team. Pending OMB approval, we will mail the surveys to other clients as described above. No changes to the survey will be made based on this pilot. The pilot test was to determine feasibility of a mailed process.
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
Dr. Brad Smith, Principle Investigator (Former), Altarum Institute (Contractor)
Dr. Smith provided guidance and input into evaluation design for each component of the evaluation.
Email: [email protected]
Phone: 210.788.1938
Dr. Alana Knudson, Current Principle Investigator, NORC (Contractor)
Dr. Knudson provided guidance and input into evaluation design for each component of the evaluation, expert opinion provide for the site visits. Dr. Knudson will be responsible for conducting the site visits and the analysis of the site visits.
Email: [email protected]
Phone: 301.634.9326
Ms. Shena Popat, Senior Research Analyst, NORC (Contractor)
Ms. Popat will participate in the site visits, take notes, create transcripts and code the data in NVivo. She will also assist in the analysis and development of the site visit report.
Email: [email protected]
Phone: 301-634.9521
Dr. Donald Nichols, Lead Researcher for Quantitative Data, IMPAQ International, (Contractor)
Dr. Nichols provided guidance on all quantitative data collection for the evaluation design. Dr. Nichols will be responsible for collecting and analyzing the CHW data collection forms.
Email: [email protected]
Phone: 510.465.7884 ext. 283
Dora Hunter, Project Manager/ Research Analyst, Altarum Institute (Contractor)
Ms. Hunter provided input to the final evaluation design. She will be responsible for disseminating and collecting the client satisfaction survey. She will also aid in the data collection from the pilot sites.
Email: [email protected]
Phone: 202.776.5184
Dr. Guido Cataife, Research Analyst IMPAQ International (Contractor)
Dr. Cataife will be conducting the analysis of the CHW data collection form and provide feedback on data quality.
Email: [email protected]
Phone: 443.832.2321
Sharanjit Toor, Research Analyst, IMPAQ International (Contractor)
Ms. Toor will be conducting the analysis of the CHW data collection form and provide feedback on data quality.
Email: [email protected]
Phone: 443.283.1659
Appendix A- C enclosed via separate attachments
Appendix A Site Visit Materials
Appendix B1 CHW Data Collection Form- Client Data
Appendix B2 CHW Data Collection Form- Design Implementation Data
Appendix B3 CHW Data Collection Form- Help
Appendix B4 CHW Data Collection Form- Excel
Appendix B5 CHW Data Collection-Technical Assistance Training
Appendix C Client Satisfaction Survey
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Instructions for writing Supporting Statement B |
| Author | Jodi.Duckhorn |
| File Modified | 0000-00-00 |
| File Created | 2021-01-27 |