Supporting Statement B
Evaluation of the Patient Navigator Outreach and Chronic Disease Prevention Demonstration Program
OMB Control No. 0915-NEW
There are two main data collections described in this Supporting Statement. The first involves collection of qualitative information from staff associated with Patient Navigator Demonstration Project (PNDP), including administrators and managers from grantee organizations, health care/social service providers, navigators, and staff from community organizations. These data will be utilized for grantee oversight, to help ensure the success of current and future programs, and to assist with the interpretation of quantitative information.
The second collection involves quantitative data related to the demographics and care of all patients entering HRSA’s Patient Navigator Demonstration Program, as well as from patient navigators providing care under the program. These data will be used to describe the populations served by the program, the impact of the program, and factors related to program success.
All data collection methods and analyses build upon previous experience with FY 2008 Patient Navigator Outreach and Chronic Disease Prevention Demonstration Program (OMB number 200903-0915-003); revisions to procedures implemented during that program are designed to improve data quality and decrease data collection burden.
1. Respondent Universe and Sampling Methods
Qualitative Data Collection
During qualitative data collection, respondents will participate in discussion and focus groups administered by the evaluation contractor. Respondents for qualitative data collection will be drawn from multiple pools that have had significant interactions with the local PNDP project, including:
Local PNDP administrators and managers
Patient navigators
Health care/social service providers
Community partner staff
In the previous demonstration, these groups had provided useful information for ascertaining the sophistication and level of development of navigator programs.
All navigators and project administrators hired under the PNDP will be selected and will participate in discussion groups at all sites. However, because of the different characteristics of local PNDP projects, the composition and emphasis of other groups may differ slightly across sites. Local PNDP administrators will invite potential respondents who have knowledge of and who have interacted in significant ways with the PNDP project. We expect that the participation in the groups will be dependent on respondent availability at the time of scheduled group discussion. We anticipate that a total of 50 administrators from grantee organizations, 46 patient navigators, 40 health care/social service providers, and staff from 50 community program partners will participate in these discussion and focus groups. We expect response rates to be high (well above 80 percent), based on our experience with the previous demonstration project. However, no formal response rate was required or tracked in that project.
Table B-1 Qualitative Data (Focus/Discussion Groups) Collected at 10 Grantee Sites
|
Numerical Estimate |
Expected Response Rate |
PNDP Program Administrators |
50 |
100% |
Patient Navigators |
46 |
100% |
PNDP Grantee Organization Administrators |
50 |
90% |
Health Care/Social Service Providers |
40 |
90% |
Program partners (Community Organizations) |
50 |
95% |
Data from discussion/focus groups are expected to provide descriptive data about programmatic processes that work well across a range of grantee sites. Findings will be used to inform analysis of quantitative data and future program recommendations. Though qualitative findings may suggest avenues for future evaluation and program development, without further investigation these findings are not generalizable to all navigator programs, clinics, or community organizations. The relatively small, heterogeneous groups of individuals involved at each site, and our inability to identify a priori factors influencing discussion group outcomes preclude sampling as a strategy for producing accurate information.
Quantitative Data Collection
As part of ongoing quality improvement activities, quantitative data will be collected from a census (all) of:
(1) Navigators;
(2) Navigated patients;
(3) Navigator encounters and related health care referrals.
Grantees will also report administrative data on a quarterly basis. In this section we will describe how respondents will be selected, and then why a census is necessary.
Patient navigators will be recruited and hired by program administrators at each site. The navigators must meet minimum requirements, as defined by each grantee, and should reflect the population the program is designed to serve. We anticipate a total of 46 patient navigators will be hired for the program. Data will be collected from all navigators who provide services through the program. A census is requested in the committee report accompanying the authorizing legislation. Data will be collected from navigators and entered in an online database by PNDP administrators at the local sites.
Navigated patients will be recruited at each clinical site through means developed to meet the unique needs of the local community and/or clinic population. Each site was required to outline the means of recruitment in their grant application. Common means of recruitment involve health care provider referral, identification of patients meeting program criteria through various health information technology (HIT) systems, outreach activities such as presentation at health fairs, or through chart review. We anticipate an average of 4,827 patients per year to participate in the program. Data regarding patient demographics and health status (quality of life, health coverage status, co-occurring disorders, and clinical information) will be collected for all patients who are enrolled in the program initially, with an update at the end of the program. Data will be collected on all patients at intake, and will be entered in the online database by the navigator or by a data entry specialist.
Navigator encounters, barriers addressed, and the status of related health care referrals will be tracked through an online database holding information on navigator activities as well as the resolution of referrals facilitated by navigators. Data will be collected by navigators and entered by them or data entry specialists at each site.
Administrative data related to the number of navigators working, the type of outreach provided, and ongoing navigator training will be reported by administrators to the online database.
A census of patient’s and navigator’s demographic information is requested by the committee report accompanying the authorizing legislation. For other data elements (involving health status, navigator encounters, and related health care services), a census is preferable to a sample for several reasons. First, navigator programs implemented under PNDP vary widely, targeting a broad range of chronic conditions in different patient populations. A large number of cases would be needed to adequately represent this heterogeneity. Second, because navigation for chronic diseases is relatively new, there are limited data available for identifying significant covariates, estimating effect sizes, and determining appropriate sample size. Third, the implementation of a sampling strategy across sites with very different capabilities would increase the burden of data collection, particularly among those without an adequate electronic health record system. Finally, in order for the data system to provide useful information to local PNDP administrators and navigators for continuous quality improvement purposes, information is needed on all navigated patients.
Previous experience with the FY 2008 Patient Navigator Outreach and Chronic Disease Prevention Demonstration Program (OMB number 200903-0915-003), has indicated that sites were able to provide information on nearly 100% of patients and navigator encounters, but that data on services received by the patient were lacking. It seems that this critical clinical information was inadequately obtained and monitored by navigators at several sites. Data requirements for reporting on services received by patients will ensure that navigators use this information in ongoing clinical practice, a quality improvement imperative. Table B-2 shows expected response rates for quantitative information.
Table B-2 Quantitative Information From 10 PNDP Sites
Data Category |
Numerical Estimate |
Expected Response Rate |
Prior Response Rate |
Patient Navigators |
46 |
100% |
100% |
Patient Demographic Data |
6,327 |
95% |
95% |
Patient Health Status |
12,654 |
90% |
No prior experience |
Navigator Encounter/Targeted Services |
37,962 |
95% |
80% |
Administrative Data |
40 |
100% |
100% |
2. Procedures for the Collection of Information
Qualitative Data Collection
Focus groups and less formal discussion groups will be conducted during site visits in spring, 2012. The evaluation contractor will work with local PNDP administrators to set up meetings with appropriate staff from the PNDP program, the grantee organization, and local partner organizations. PNDP administrators will generate a list of potential participants, and the contractor will provide compelling text for use by the sites to recruit participants. As previously mentioned, PNDP program administrators and patient navigators will be required to attend discussions as part of their grant obligations; however, participation of other grantee organization staff and community organization partners will be voluntary. All discussions will be scheduled at least a month in advance, and participants will be asked to confirm attendance one week before the scheduled appointment.
Group discussions will be conducted with the following:
(1) PNDP administrators, including the Principal Investigator, the Project Administrator or Manager, the Clinical Champion, and/or the Lead Navigator;
(2) PNDP navigators;
(3) Organizational leaders, which may include health care administrators, CEOs, financial managers, department heads (these individuals may also provide direct patient care);
(4) Persons who provide health care or social services to navigated patients; and
(5) Community organization representatives who are partnering with the navigator project to provide care to navigated patients.
Discussions and focus groups will be led by an experienced facilitator from the evaluation contractor, and will be attended by a HRSA representative. In the first four groups, discussions will follow a semi-structured discussion guide, and may be conducted with a few or many persons depending on the characteristics of the site and the availability of participants. A semi-structured guide is necessary to allow for tailoring of questions to the specifics of the sites; as previously mentioned, the characteristics of the sites, the conditions navigated, the grantee organization staff involved, and the populations served vary considerably. However, we anticipate implementing a structured, focus group discussion guide with community organizations because there are many standard questions that are universally relevant for this group. The focus group discussion guide is attached.
With permission of participants, all discussions will be recorded for note-taking purposes. Transcripts will be developed and will be coded by two or more trained analysts from the evaluation contractor in order to ensure analysis quality. Outcomes of each discussion will be augmented by review of any observational notes. Results from the group discussions, and topic areas where there are similarities and differences within a group, and between groups, will be analyzed. Quotes from respondents will be included without attribution.
A draft report containing a detailed description of methods, along with findings, will be delivered to HRSA by the evaluation contractor. The HRSA Project Officer will return the draft to the contractors with comments, and the contractor will deliver a finalized document no later than August 2012.
Quantitative Data Collection
Procedures. Data will be reported to HRSA via an online database. Navigators may enter data directly into the online database during or immediately after an intake or encounter. However, because some patient-navigator encounters may occur in locations without internet access, sites may choose to have navigators enter information initially onto paper forms. Information from the forms can then be entered into the database by the navigator or by a data entry specialist at the grantee site. Similarly, additional information related to patient health status, clinical information, and health care coverage may be collected on paper forms first by navigators or data entry staff, then entered into the online database. No sampling procedures will be used; data are needed on every case for both administrative and evaluation purposes.
Based on previous experience, data quality is maximized when data are entered into the online database continually, on a daily basis, as information becomes available. However, this may not be practical in sites collecting data initially on paper. Therefore, sites will be required to enter data within three business days of the intake/encounter date in order to ensure data on patient and navigator encounter forms are not misplaced. In order to maximize efficiency for correction of missing data and errors, sites will have immediate access to error reports. Sites will also receive reminders about correcting data errors on a monthly basis, with the expectation that data will be corrected and updated before the beginning of the following month.
Administrative data may be entered online as the information becomes available (e.g., after an outreach activity has been completed, or after a meeting has occurred). However, collected data will be compiled and reported to HRSA on a quarterly basis, so sites must complete entry of administrative data by the end of the quarter in which the activities occurred.
Sample size. Power analyses reveal that adequate sample size exists to detect findings as statistically significant with power = .8 and alpha =.05 for almost all analyses. In the following paragraphs we examine power to detect statistically significant findings for analyses with the smallest sample sizes (and potential lowest available power). The smallest sample sizes will occur in analyses of changes in clinical measures related to particular navigated conditions (e.g., cancer, diabetes, cardiovascular disease, asthma, hypertension, obesity, hyperlipidemia).
Little information is published about the effects of navigation on clinical measures. Analyses at the site level in the first demonstration project included many patients who had not been in navigation long enough to access their primary health care navigation targets. This fact and the small sample sizes at the site made it difficult to detect the impact of navigation at a statistically significant level. Thus, clinical information is being collected by the cross-site evaluation so as to improve sample size, and grantees will be able to access data for local evaluations by downloading it from the online database. Since measures of distribution were not reported in the last evaluation, distribution information is not available for the particular patient population targeted for navigation.
In order to ensure that within-condition analyses include only those cases with significant navigation exposure, analyses involving clinical information will include only those cases that achieved a primary navigation target between September 2011 and the end of the program. Rough estimates based on the project focus, the necessary time to reach treatment, and number of cases proposed at each grantee site reveal that the minimum number of cases in a single navigated condition group is 160 cases navigated for overweight/obesity. The maximum number of cases available for analyses within a single navigated condition is 1,300 diabetes patients. In the following paragraphs we provide details on analyses to determine minimal sample size involving pre/post tests of proportions (McNemar’s test), matched t-tests, and hierarchical multivariate regression within navigated condition groups.
McNemar’s Test. One set of analyses will focus on whether the percentage of the patient population meeting clinical criteria improves after navigator intervention. Dependent variables include a number of clinical benchmarks that are specific to a particular navigated condition. For example, BMI is tracked for patients navigated for being overweight or obese. The relevant benchmark is adult BMI less than 30, where 30 and above meets criteria for obesity, and BMI of 25 30 is overweight. In order to detect changes in proportions with power =.8 and alpha =.05, such that 15% of navigated patients move from above to below the benchmark (obese to non-obese status), and assuming 5% of patients move in the opposite direction (from overweight to obese status), 155 cases are needed (Machin et al, 2009). Since within-condition groups are projected to have more than 155 cases, sample size is sufficient for this test.
Matched t-test. This test will be used to determine whether mean improvements in clinical scores are statistically different from zero. Repeated measures (i.e., data collected twice from the same case) increase the ability of a t-test to detect statistically significant differences between means. With d´ set at .2 for a small effect size, and estimating the minimum correlation (r) between pre and post scores as .5, the effective d should be .28. With p set to .05 two-tailed, power at .8, a sample size of 200 cases and above should be adequate to detect an effective size d of .28 (Cohen, 1988). Based on initial projections, we expect to achieve this sample size for patients navigated for diabetes, asthma, cardiovascular disease, and hypertension.
Hierarchical Multivariate Regression. This test will be used to identify the importance and independence of predictors of program success within each navigated condition. Based on our previous experience, we expect that the number of unresolved navigation targets will be associated with the number of targets identified by the navigator, with an R-squared = .25. Only 33 cases are required to detect this finding at power = .8 and alpha =.05
In order to detect a statistically significant R-squared increase of .015 (associated with a small effect size .02) attributable to the addition of a variable into the regression equation, the required sample size is 386 cases. This assumes that power is set at .8 and alpha at .05. Furthermore, a sample size of between 380 and 400 cases should allow for testing of additional variables using hierarchical regressions, assuming detection of effect sizes at or above .02.
We expect within-condition sample sizes to be 400 cases or larger for diabetes, cardiovascular disease, asthma, cancer, cancer risk, and metabolic disorder (i.e., the co-occurring disorders of diabetes, hypertension, and hyperlipidemia). However, it seems unlikely that the number of patients navigated for obesity/diabetic risk, hypertension alone, and hyperlipidemia alone will reach this size.
To summarize, sample size is adequate or more than adequate for all but a few within-condition analyses involving the assessment of program success.
3. Methods to Maximize Response Rates and Deal with Nonresponse
Qualitative Data Collection
Response rates for the discussion/focus groups will be maximized by means of the participants’ investment in continued navigator program services. We expect that navigators and PNDP administrators will participate as part of their jobs. Potential participants from within the grantee organization are likely to attend discussions because of their investment in the success of the program. Similarly, community organization staff seem likely to participate to the degree that ongoing support for the program provides benefit to the organization and the population it serves. A letter from HRSA requesting participation will encourage potential participants to attend, and a reminder call in the week preceding the discussions will further improve the response rate. The contractor will track the number of participants identified for participation, as well as the number that actually attend the discussions in order to identify potential areas of selection bias.
Quantitative Data Collection
Response rates for quantitative data collection will be maximized through minimizing the burden associated with data entry, integration of data collection with regular clinical and administrative procedures, and providing error reports regarding missing information.
To minimize burden, data will be collected via submission to an online data collection tool, which will allow users working away from their home desks to enter and review information about a patient or about the program. Since the database will have a common interface across sites, the evaluation contractor will be able to provide standardized training and technical assistance. The interface will be designed to maximize ease of entry and minimize data entry errors. For example, response categories will be available through radio buttons or pull-down menus, and defined ranges will minimize mis-entry of data. Information will be available for navigator and administrator review through online look-up menus and reports.
The patient navigator will collect information that was shown in the previous demonstration program to be important to maintaining and documenting the quality of the navigator intervention. Information important to the clinical management of the patient and the administration of the program will be accessible to both navigators and administrators. For example, navigators will be able to identify which patients need follow-up after doctor’s appointments scheduled for the previous week, or the patients who have not been contacted for over three weeks. Administrators will be able to track how many patients are assigned to each navigator, and the number of encounters each navigator is logging per week.
In order to minimize nonresponse and maximize data quality, sites will be sent monthly reports documenting data omissions or errors, and these will be followed up with a technical assistance call to confirm strategies for correcting data and possibly improving procedures.
Previous experience with collecting similar data (FY 2008 Patient Navigator Outreach and Chronic Disease Prevention Demonstration Program, OMB number 200903-0915-003) revealed that response rates were well above 80 percent. We expect similar results or better for this project given new, improved online data collection systems.
4. Tests of Procedures or Methods to be Undertaken
The evaluation incorporates several lessons learned from the previous short-term demonstration program (OMB number 200903-0915-003), as follows:
Coding categories describing navigator activities were refined so as to differentiate between key components of navigator actions, thus increasing ease of use, and reducing data collection burden.
A central database was developed because wide differences in IT capabilities between grantee sites in the first demonstration program made it challenging to collect consistent data from local databases.
It appears that there was considerable variability in the time it took for patients to access medical services. It took up to nine months after enrollment in navigation for most patients to access disease prevention activities (such as cancer screening or diagnostic testing after an abnormal finding), up to six months to access primary care services for diagnosed disease, and up to nine months for specialty care. Thus, cases involved in analyses for clinical outcomes should be limited to patients who have accessed appropriate care.
Clinical information, initially reported at a site level, was added to the cross-site evaluation. Summary statistics reported at a site level were difficult to interpret because sample sizes were relatively small, variance was high, and different inclusion criteria were used at each site. It became clear that outlier data from patients with higher severity of illness (in terms of multiple co-occurring disorders, or time since initial diagnosis), or very unhealthy initial lab scores) should be analyzed separately from patients with simple or recently diagnosed disorders. For this reason, as part of the cross-site evaluation, clinical information will be collected for each patient, and patients may be stratified by severity of illness for analysis as needed.
The VR-12 health survey, a patient-reported measure of health status, was added in order to capture crucial information about the patient’s perceptions of health. The information can be used to identify case mix compared to national norms, stratify patients by perceived health status, and as an outcome measure (see below).
The following paragraphs describe procedures tested under the previous demonstration program.
Qualitative Data Collection
Discussion/focus group guides are based on unstructured interviews with grantees conducted in the fiscal year 2008 Patient Navigator Outreach and Chronic Disease Prevention Demonstration Program evaluation tools (OMB number 200903-0915-003). Information gained from that experience has allowed the contractor to provide structure for further data collection.
Quantitative Data Collection
Data collection procedures are based on those implemented during the fiscal year 2008 Patient Navigator Outreach and Chronic Disease Prevention Demonstration Program evaluation tools (OMB number 200903-0915-003), approved by OMB on August 31, 2009 and expired on December 31, 2010. The new data collection is similar to the initial collection. Minor modifications have been made to existing instruments so as to decrease burden, increase ease of use and improve data quality. In addition, we have included two new data elements.
The first, clinical information, was collected by grantees in the first demonstration, and is now a required cross-site evaluation data element. Clinical information includes measures that have been identified by HRSA and NCQA as critical quality benchmarks for each of the chronic conditions navigated (HRSA, 2010; NCQA, 2011).
The second, Health Status /Health-Related Quality of Life, will be measured by the VR-12 Health Survey (VR-12). The VR-12 is a patient-reported survey instrument derived from a longer instrument developed during the Medical Outcomes Study. The VR-12 and is comprised of two components – the Physical Health Summary Measure (PCS or physical component score) and the Mental Health Summary Measure (MCS or mental component score) (Kazis, Selim, Rogers, Ren, Lee & Miller, 2006). The validity of the VR-12 has been established in a number of studies, documenting a significant relationship between VR-12 results and quality of life (for example, in men with prostate cancer (Krupski et al., 2005)) and clinical outcome (for example, survival in a sample of cervical cancer survivors (Ashing-Giwa, Lim & Tang, 2010)). It has been used as a measure of patient-reported outcome to evaluate quality of care in diverse populations by the Department of Veterans Affairs and by the Centers for Medicare and Medicaid Studies (Rogers, Kazis, Miller, Skinner, Clark, Spiro, & Fincke, 2004). It is currently used in the Healthcare Effectiveness Data and Information Set (HEDIS) by the National Committee on Quality Assurance (NCQA) as part of the Medicare Health Outcomes Survey (see http://www.hosonline.org/Content/ProgramOverview.aspx). Current population and disease-specific norms are available (Selim, Iqbal, Rogers, Qiam, Fincke, Rothendler & Kazis, 2009), including norms that track VR-12 scores for populations with specific diseases over time (Kazis, 2011). VR-12 scores will be used to describe the patient population, identify case-mix across sites, stratify cases, and measure outcomes of navigation. The survey is administered at intake and at the end of the navigator program.
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or
Analyzing Data
Individuals involved in data collection design and analysis:
Government Project Officers
HRSA/BHPr
Alexis Bakos, PhD, MPH, RN
Deputy Director Division of Nursing
301-443-5688
HRSA/BHPr:
Kyle Peplinski, MA
Public Health Analyst
301-443-7758
NOVA Research Company, Evaluation Contractor:
Paul A. Young, M.B.A., M.P.H.
Executive Vice President/Senior Program Director
240-483-4190
Caroline McLeod, PhD
Senior Evaluation Researcher
240-483-4191
Carmen-Anita Signes, BS
Data Manager
301-986-1891
Debra Stark, MBA
Research Associate
240-752-7337
Individuals responsible for collecting data:
Clinica Sierra Vista
Bill Phelps
PNDP Project Director
661-635-3050 x2156
Coastal Medical Access Project
Jeris Wright
PNDP Project Director
912-554-3559 x11
New River Health Association
Dave Sotak
PNDP Project Director
304-469-2906
Project Concern International
Maria Reyes
PNDP Project Director
619-791-2610 x305
The Queen’s Medical Center
Debbie Ishihara-Wong
PNDP Project Director
808-537-7574
South County Community Health Center
Belinda Hernandez
PNDP Project Director
650-330-7449
Texas Tech University
Christina Esperat
PNDP Project Director
806-743-2736
University of Utah
Randall Rupper
PNDP Project Director
801-587-3410
Vista Community Clinic
Dorothy Lujan
PNDP Project Director
760-631-5000 x1133
William F. Ryan Community Health Center
Nancy Andino
PNDP Project Director
212-316-8367
REFERENCES
Ashing-Giwa, K. T., Lim, J. W., & Tang, J. (2010). Surviving cervical cancer: does health-related quality of life influence survival? Gynecol Oncol, 118(1), 35-42.
Cohen, J. (1988). Statistical Power Analyses for the Behavioral Sciences, 2nd Edition. Hillside, NJ: Lawrence Earlbaum Associates, Inc.
Health Resources and Services Administration (HRSA) (2011). Uniform Data System (UDS) Reporting Instructions for Section 330 Grantees. Retrieved August 20, 2011 at bphc.hrsa.gov/healthcenterdatastatistics/reporting/2010manual.pdf
Kazis, L. E., Selim, A., Rogers, W., Ren, X. S., Lee, A., & Miller, D. R. (2006). Dissemination of methods and results from the veterans health study: final comments and implications for future monitoring strategies within and outside the veterans healthcare system. J Ambul Care Manage, 29(4), 310-319.
Kazis, L.E. (2011). The Use of Patient-reported Outcome Measures in Health Service Decision making: Challenges and Opportunities. Presentation at the HRSA Patient Navigator Outreach and Chronic Disease Prevention Demonstration Program Peer Review Learning Workshop, June 20, 2011 in Bethesda, MD.
Krupski, T. L., Fink, A., Kwan, L., Maliski, S., Connor, S. E., Clerkin, B., et al. (2005). Health-related quality-of-life in low-income, uninsured men with prostate cancer. J Health Care Poor Underserved, 16(2), 375-390.
Machin, D., Campbell, M. J., Tan, S. B., Tan, S. H. (2009). Sample Size Tables for Clinical Studies. Hoboken, NJ: Wiley-Blackwell
National Committee For Quality Assurance. (2011). Disease Management 2011 Measures-2. Retrieved April 4, 2011, 2011, from http://www.ncqa.org/tabid/1256/Default.aspx
Rogers W.H., Kazis L.E.,
Miller D.R., Skinner, K.M., Clark, J.A., Spiro, A. 3rd, Fincke, R.G.
(2004). Comparing the health status of VA and non-VA ambulatory
patients: The Veterans’ Health and Medical Outcomes Studies. J
Ambulatory Care Manage;
27(3),
249-262.
Selim, A. J., Rogers, W.,
Fleishman, J. A., Qian, S. X., Fincke, B. G., Rothendler, J. A., et
al. (2009). Updated U.S. population standard for the Veterans RAND
12-item Health Survey (VR-12). Qual
Life Res, 18(1),
43-52.
Selim, A.J., Iqbal, S.U., Rogers, W., Qian, S., Fincke, B.G., Rothendler, J., Kazis, L.E. (2007). Implementing the HEDIS Medicare Health Outcomes Survey: An Alternative Case-Mix Methodology. Centers for Medicare and Medicaid Services. Retrieved August 20, 2011. http://www.hosonline.org/Content/Publications.aspx
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Seleda.Perryman |
| File Modified | 0000-00-00 |
| File Created | 2021-02-01 |