OMB Clearance Justification Package
Evaluation of Tribal Health
Profession Opportunity Grants (HPOG)
Request
for Extension of a Previously Approved Collection:
Control No:
0970–0395
Supporting Statement Part A
August 2014
JUSTIFICATION
Background, Need and Circumstances Making the Collection of Information Necessary
The Administration for Children and Families (ACF) is requesting a one year extension from the Office of Management and Budget (OMB) to continue to collect qualitative data as part of the evaluation of the Tribal Health Professions Opportunity Grants (HPOG) to develop an understanding of the program’s effectiveness in meeting the tribal health workforce needs. This extension will allow ACF to collect data on grantees through the completion of their grant period. Since initial approval of this information collection, no changes have been made to the instruments (interview and focus group protocols) or the time per response.
The HPOG program is funded by H.R. 3590, the Patient Protection and Affordable Care Act (Affordable Care Act), which provides opportunities for education and training for low-income individuals for healthcare professions that pay well and are expected to experience labor shortages—including low-income individuals living on tribal reservations.
The HPOG program, administered by ACF, funded 32 five-year demonstration projects to design and implement innovative health workforce development training programs targeting Temporary Assistance for Needy Families (TANF) recipients and other low-income individuals. Grants were funded under two separate Funding Opportunity Announcements (FOA): The Health Profession Opportunity Grants to Serve TANF and Other Low-Income Individuals (HHS-2010-ACF-OFA-FX-0126) and the Health Profession Opportunity Grants for Tribes, Tribal Organizations or Tribal College or University (HHS-2010-ACF-OFA-FY-0124). Five of the 32 demonstration projects were awarded to Tribal Organizations and Tribal Colleges to develop culturally-informed training programs.
Since the award of these 32 HPOG grantees in 2010, ACF entered into several contracts to conduct evaluation activities of the HPOG grants, including one with Abt Associates, in partnership with the Urban Institute, for the Implementation, Systems and Outcome Evaluation of the Health Profession Opportunity Grants to Serve TANF Recipients and Other Low-Income Individuals which includes the design and operation of an HPOG Performance Reporting System (PRS) to collect quantitative program management data from all 32 grantees. Data collection activities undertaken under these contracts have been approved by OMB under OMB Clearance number 0970-0394. The subject of this OMB Supporting Statement is the evaluation of the five Tribal HPOG programs. ACF contracted with NORC at the University of Chicago (NORC) and its partners, Red Star Innovations and the National Indian Health Board (NIHB) (the ‘tribal evaluation team’), to conduct an evaluation that uses a culturally responsive framework to build upon the conceptual framework guiding the Tribal HPOG programs.
Although the Affordable Care Act mandates an evaluation of the HPOG demonstration projects (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(B)), it does not stipulate that a separate evaluation of the Tribal HPOG grantees be conducted. To determine whether and how the Tribal component of the HPOG program meets its goals, ACF (via NORC) is engaging in a comprehensive, multi-faceted and rigorous evaluation approach to assess the implementation and outcomes of the five Tribal grantees.
As noted above, ACF issued a separate FOA for the Tribal Health Professions Opportunity Grants. This Tribal FOA indicates that “In an effort to address healthcare disparities and improve the quality of life of 1.9 million American Indians and American Natives, the Patient Protection and Affordable Care Act reauthorized the Indian Health Care Improvement Act (IHCIA), the cornerstone legal authority for the provision of health care to American Indians and Alaska Natives. The Indian Health Care Improvement Act (Pub. L. No. 94-437) authorizes the creation of elder care programs, programs for hospice, assisted living, long-term care and home and community-based services. These facilities will need qualified workers and will need to expand current staff knowledge. Providing education and training to upgrade skills is vital to attaining these goals. Therefore, the opportunity of this funding can encourage tribal communities to design, develop and implement training programs to address the healthcare workforce needs of their communities.”
ACF determined that a separate evaluation of the tribal grantees would be necessary and useful to expand the knowledge base about effective practices in tribal communities. As further described below, the evaluation will address the needs of ACF to learn about how tribal communities design, develop and implement training programs to enhance the health care workforce needs of the community. The Tribal HPOG evaluation provides an opportunity for evaluating the implementation and outcomes of five distinct workforce development approaches that integrate health professional training programs with culturally-informed models of learning and practice.
Three key research questions are guiding this evaluation:
Have grantees incorporated structures necessary to enhance the health care workforce needs of the community?
Have grantees implemented processes to prepare participants for employment in the tribal health care sector?
Is there evidence that participants in the program achieved successful employment and work force capacity building outcomes?
The five grantees are tribes, tribal organizations, tribal colleges or tribal universities. Tribal grantees are integrating health professionals training programs with culturally-informed models of learning and practice, such as the Family Education Model (FEM), a framework for postsecondary education that creates an extended family structure within the academic experience. The evaluation being conducted by NORC and its partners is currently providing and will continue to provide ACF with a wide range of qualitative data to support program management over time and evaluate the success of tribal grantees’ efforts. Through this tribally focused and culturally competent evaluation, the way in which tribal grantees are implementing, creating and adapting culturally‐relevant and appropriate programs has been and will continue to be documented, outcomes will be quantified further, and lessons learned will continue to be disseminated.
Qualitative Data Collection The Tribal HPOG evaluation primarily relies on qualitative data collected by the tribal evaluation team, i.e., interviews and focus groups with program administrators, program implementation staff, participants, and employers.
Data will continue to be collected annually during site visits with each tribal grantee. For the final year of data collection, five in-person site visits will be conducted across the tribal grantees (1 at Blackfeet Community College; 1 at Cook Inlet Technical College; 1 at College of Menominee Nation; 1 at Turtle Mountain Community College; and 1 at Cankdeska Cikana Community College). Each grantee is different in how the program is structured with primary and secondary implementation sites. In total, the tribal evaluation will conduct data collection activities at a total of 13 sites (5 primary implementation sites; 8 secondary implementation sites). The tribal evaluation team will conduct the following data collection activities:
Site visits to the thirteen Tribal HPOG grantee implementation sites. Site visits will include the following data collection activities:
Interviews with Tribal HPOG Grantee and Partner Administrative Staff and Partners. In-person interviews will be conducted during the remaining site visits to gain insight from grantee and partner administrative staff (including directors and up to 6 grantee and partner administrative staff from each grantee program) on high-level program strategies, program development and lessons learned, for a total of up to 35 interviews.
Interviews with Tribal HPOG Program Implementation Staff. Interviews will be conducted with staff responsible for coordinating and implementing the program at each site. These individuals may include program instructors, recruitment and orientation staff, and providers of supportive services. Phone interviews may be conducted in lieu of in-person interviews with implementation staff at secondary implementation sites should scheduling conflicts arise. Up to 9 program implementation staff at each program implementation site will be interviewed, for a total of up to 117 interviews.
Interviews with Employers. Employers’ perspectives on the utility of the training programs are critical, so it will be essential to capture their perceptions and experiences. Phone interviews may be conducted in lieu of in-person interviews with employers at secondary implementation sites should scheduling conflicts arise. Up to 4 employers at each program implementation site will be interviewed, for a total of up to 52 interviews.
Focus groups or Interviews with Program Participants. Focus groups and interviews will be conducted with program participants, however no participant will respond through both modes. (That is to say, if a current program participant participates in a focus group, he/she will not be engaged for a program participant interview). The tribal evaluation team will conduct focus groups with program participants at each primary program implementation site and at secondary sites, if possible. Phone interviews may be conducted in lieu of focus groups with program participants at secondary implementation should scheduling conflicts arise. . Up to 9 program participants at each program implementation site will participate in focus groups or interviews, for a total of up to 117 program participant respondents.
Telephone Interviews with Program Completers. In order to obtain information on key program outcomes related to educational attainment and employment, interviews will also be conducted with program completers of the Tribal HPOG program. (It is possible for program participants to have participated in a focus group while they are in the program and then to participate in an interview after they have completed the program. If this is the case, we will not repeat any of the questions from the earlier data collection. However, questions such as “What program did you participate in? Why did you choose this program?” will need to be asked of respondents who have not previously provided those answers. As such, the modes of data collection are not redundant, and instead seek to collect comparable information across respondents). These interviews will occur after the final site visit. Up to 15 interviews will be conducted with program participants across implementation sites in each grantee program, for a total of up to 75 interviews. After experience collecting these data for four years, we reduced the number of completer interviews as we averaged less completer interviews than anticipated.1
Telephone Interviews with Program Non-Completers. Phone interviews will also be conducted with participants that have not completed the Tribal HPOG program. These interviews will occur after the final site visit. Up to 4 interviews will be conducted with program non-completers across implementation sites in each grantee program, for a total of up to 20 interviews.
Secondary Analysis of Quantitative Data from the HPOG PRS
To gather quantitative data for the HPOG program, ACF contracted with Abt Associates and the Urban Institute to develop the PRS, a quantitative data collection system, for the entire pool of HPOG grantees, including the tribal grantees. The PRS was approved under OMB Clearance Number: 0970-0394.
The tribal evaluation team will continue to utilize the data collected through the PRS, but will not collect additional quantitative data from grantees in an effort to reduce burden.
NORC will be using the PRS as a data source in completing the Tribal HPOG Evaluation. However, the data captured through the PRS needs to be supplemented in order to conduct a comprehensive evaluation of the Tribal HPOG programs. NORC will continue to use the PRS data to report quantitative information, such as how many participants each program serves, retention rates, completion rates, etc. In addition, NORC needs to continue to collect qualitative information to understand what components of the programs’ design and implementation worked or did not work, and why. While the PRS data helps quantify outcomes, the NORC-collected qualitative data enables an understanding of factors that influenced those outcomes, including facilitators and barriers.
This collection of data is authorized by Section 301 of the U.S. Public Health Service Act (42 U.S.C.241). A copy of this legislation can be found in Attachment A1.
Purpose and Use of Information Collection
The information collected as part of this continued evaluation will be used to determine lessons about developing and implementing the Tribal HPOG program, such as identifying the contextual factors that influenced successes and challenges. This evaluation has the potential to identify promising practices and inform the knowledge base of culturally-based models implemented in tribal settings. Continuing the data collection activities will extend the examination of both the Tribal HPOG initiative as a whole and the individual Tribal HPOG programs through the completion of the period of performance of the program grants.
Qualitative data collected as part of the Tribal HPOG Evaluation will continue to serve as a complement to the PRS data, not a duplication or replacement of it. As noted above, the PRS is an important data source for the evaluation conducted by NORC, and provides necessary data on intake, retention, and completion rates, as well as the rates of services provided. However, additional qualitative information continues to be necessary to conduct a comprehensive evaluation of the design, implementation, and outcomes of the Tribal HPOG programs. The qualitative data collection components, including interviews and focus groups, provide necessary information on promising practices, lessons learned, and implementation facilitators and barriers which can ultimately be used to inform future tribal health professions programs.
Furthermore, the continued dissemination of project findings to stakeholders in tribal health will provide documentation and lessons learned around programmatic approaches to health professions training serving tribal populations. By engaging tribal officials, tribal and local program administrators and partners, tribal service agencies/organizations, public health, policy and education research consortiums and other stakeholders, ACF is positioned to facilitate increased knowledge sharing and action around training programs for special populations. The continued dissemination of findings will provide stakeholders with knowledge of efforts to mitigate the health professions workforce shortage that exists in tribal communities; communication of the purpose and design of the evaluation, highlighting culturally specific components; status updates throughout the evaluation process, including report of final results; qualitative and quantitative findings that will support health professions training program improvement over time; knowledge that will enhance programmatic and culturally specific approaches to health professions training and facilitate increased knowledge sharing and action around training programs for special populations; and notification of opportunities for involvement/ dialogue.
Use of Improved Information Technology and Burden Reduction
The Tribal HPOG evaluation is designed to limit the reporting burden for respondents. Where possible, the tribal evaluation team has been and will continue to gather information electronically. For example, grantee program documents and curricula will continue to be collected via email, and then analyzed by the tribal evaluation team.
The bulk of information gathered by the evaluation team will continue to be done through personal interactions during the site visits. When working with American Indian/Alaska Native populations, it is important to consider cultural appropriateness. Among tribal populations, the development of relationships and person-to-person connections is paramount. In order to gather the most accurate and unbiased data possible, the tribal evaluation team believes that in-person interviews, focus groups and phone calls are the most appropriate means to collect information and will produce the most in-depth, accurate, comprehensive and rich data. Best practices for data collection in tribal communities stress the importance of developing trusting relationships with community members and program participants, either through face-to-face meetings or by working through trusted intermediaries. These practices lend themselves to qualitative research methods, so that the tribal evaluation team’s approach is to conduct “in-person interviews, focus groups and phone calls” as opposed to mail or web-based surveys. For this reason, the tribal evaluation team plans to conduct a final annual site visit to grantee program implementation (primary and secondary) sites.
The tribal evaluation team will minimize burden to respondents by providing discussion topics in advance of the call, reducing the burden of the interview and ensuring that discussions are focused and require as little time as possible for the respondents. In-person and telephone interviews will last between 30 and 90 minutes depending upon respondent group; focus groups will last approximately 90 minutes. As such, respondents will not be responsible for providing information beyond their oral responses to interview and/or focus group questions. The tribal evaluation team will schedule the interviews for a time that is convenient for the respondents, and be accommodating should they need to reschedule.
As noted above, in a further effort to reduce burden, the Tribal HPOG evaluation will utilize the quantitative data already being collected through the PRS rather than collecting additional quantitative data from grantees. Approval for the HPOG PRS was obtained from OMB (OMB Clearance Number: 0970-0394).
Efforts to Identify Duplication and Use of Similar Information
The purpose of this data collection effort is to collect information from key informants to evaluate the five Tribal HPOG programs. Prior to the Tribal HPOG Evaluation conducted by NORC, there have been no previous efforts to collect data from stakeholders of the HPOG program as this is a demonstration grant funded through the Affordable Care Act. This evaluation will fill a need for data demonstrating the effectiveness of health professions diversity programs as there is limited evidence on the intervention efficacy of health professions diversity programs and few outcome evaluations using rigorous study designs have been conducted.
Qualitative data collected as part of the Tribal HPOG Evaluation will continue to serve as a complement to the PRS data, not duplicate or replace it. The qualitative interview data will continue to capture stakeholders’ perceptions of the effectiveness of the structures and processes that were put in place to implement the Tribal HPOG programs and how each program and its various components are tailored to meet the needs of diverse Tribal communities.
The data captured in the PRS and the information collected from the interviews serve different but complementary purposes. The PRS captures data on key indicators for internal program management and semi-annual reporting to ACF (i.e., participant demographics, number enrolled, numbers exited, number and types of public benefits received and supportive services provided, educational and employment history of participants, educational and training activities provided, required courses for training programs, educational and employment outcomes). NORC receives individual-level Tribal HPOG data without identifiers from the HPOG PRS (from the Urban Institute) on a semi-annual basis. These data are used for secondary data analyses conducted by the Tribal HPOG evaluation team for each of the five tribal program and its sites, and aggregated analyses across the tribal grantees.
In contrast, the goal of qualitative interviewing is to obtain rich information and insight to describe or explore phenomena through interpersonal interaction or observation. The interview process is expressly designed to establish rapport with the respondent and to elicit information in a voluntarily manner about one’s motivation for and participation in the program. As sound interviewing techniques require establishing a comfort zone and easing into an interview with some preliminary conversation, as well as tracing the key events of a program participant’s journey through a program, the interviewers will need to reiterate some of the specific information documented in the PRS anyway in order to engage the respondent (e.g., What program did you participate in? What services were provided? What courses did you take? What certificate did you receive? Are you currently employed? etc.). In contrast, to begin each interview by providing information about the respondent derived from the PRS would not be well-received nor would the amount of time gained be of consequence (i.e., burden reduced). Participants would find it off-putting for a stranger to present them with personal information and perceive this as invasive and possibly coercive.
ACF and the tribal evaluation team have examined the overlap in questions in the PRS and interview/focus group protocols. Attachment A2 identifies those questions in the interview and focus group protocols that ask for similar data to that collected in the PRS. In the final column, we articulate how we will streamline or why we think this apparent duplication is necessary. Please note that in the majority of cases, this information is asked only as a lead question to allow the interviewer to frame the question. Having these data from the PRS would not significantly reduce the burden to grantees – as this information would need to be stated by the interviewer to frame the question (and as mentioned above, this may be off-putting to respondents).
Impact on Small Businesses or Other Small Entities
This study will continue to only have minimal impact on small entities, as small business professionals will only be interviewed if they are employers of HPOG program participants (as an internship or practicum), program graduates or grantee administrative partners. In an effort to reduce burden, the duration of each employer interview will be no more than 45 minutes. The perspective of program participant employers is essential to capture employers’ general impressions of program participants and graduates and the extent to which tribal health workforce capacity has been improved by HPOG.
Consequences of Collecting the Information Less Frequently
The design of this evaluation engages respondents as infrequently as possible. Data collection will take place primarily during a final annual site visit with each tribal grantee. Each site visit will be approximately three to five days in length.
Decreasing the frequency of site visits (i.e., scheduling fewer than one visit per year), would result in a lack of essential and timely information. Collecting data less frequently would preclude the tribal evaluation team from collecting essential information about program progression from year to year. Since this grant is administered over five years, a final year of annual data collection at each site will provide a rich set of data that will clearly demonstrate program successes or struggles over time, and allow for the evaluation of Tribal HPOG grantee program. The federal government will benefit from having the final annual information available on the processes and outcomes of these grant programs.
There are no legal obstacles to reduce the burden.
Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2).
Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency
A 60-Day Federal Register Notice when approval was initially sought was published in the Federal Register on 4/18/2011, vol. 76, No. 74, pp. 21749-21750. There were no public comments.
A 60-Day Federal Register Notice for this one-year renewal request was published in the Federal Register on 6/13/2014, vol. 79, No. 114, pp. 33929-33930 (see Attachment A3). There were no public comments.
The tribal evaluation team staff consulted are listed below. This consultation took place in 2011 and was reviewed and updated in 2014. The table below identifies staff consulted in 2011.
Name |
Title/ Organization |
Telephone number |
Email address |
Michael Meit, MA, MPH |
Program Area Director, Public Health, NORC at the University of Chicago |
(301)634-9324 |
|
Alana Knudson, PhD |
Principal Research Scientist, Public Health, NORC at the University of Chicago |
(301)634-9326 |
|
Carol Hafford, PhD |
Senior Research Scientist, Economics, Labor and Population Studies, NORC at the University of Chicago |
(301)634-9491 |
|
Aleena Hernandez, MPH |
Principal and Founder, Red Star Innovations, LLC |
(520) 407-6307
|
|
Evangelyn Dotomain, MBA |
Deputy Director, National Indian Health Board |
(202)374-2034
|
|
Jessica Bushar, MPH |
Senior Research Analyst, Public Health Research, NORC at the University of Chicago |
(301)634-9515 |
|
Hilary Scherer, BA |
Research Analyst, Public Health Research, NORC at the University of Chicago |
(301) 634-9374 |
|
Heather Langerman, BS |
Research Assistant, Public Health Research, NORC at the University of Chicago |
(301)634-9516 |
The table below identifies staff consulted in 2014.
Name |
Title/ Organization |
Telephone number |
Email address |
Michael Meit, MA, MPH |
Program Area Director, Public Health, NORC at the University of Chicago |
(301)634-9324 |
|
Alana Knudson, PhD |
Principal Research Scientist, Public Health, NORC at the University of Chicago |
(301)634-9326 |
|
Carol Hafford, PhD |
Senior Research Scientist, Economics, Labor and Population Studies, NORC at the University of Chicago |
(301)634-9491 |
|
Aleena Hernandez, MPH |
Principal and Founder, Red Star Innovations, LLC |
(520) 407-6307 |
|
Robert Foley |
Acting Director of Public Health Program and Policies, National Indian Health Board |
(202) 355-5494 |
|
Katherine Meyer, MPP, MA |
Principal Research Analyst, Public Health Research, NORC at the University of Chicago |
(301)634-9424 |
|
Jennie Alfaro, BA |
Senior Research Analyst, Public Health Research, NORC at the University of Chicago |
(301)634-9523 |
|
Tess Gilbert, MHS |
Senior Research Analyst, Public Health Research, NORC at the University of Chicago |
(301) 634-9385 |
|
Shannon TenBroeck, MA |
Principal Research Analyst, Public Health Research, NORC at the University of Chicago |
(415) 315-2006 |
|
Noelle Miesfeld |
Research Analyst, Public Health Research, NORC at the University of Chicago |
(301) 634-9560 |
The tribal evaluation team has consulted with outside experts about the proposed data collection, evaluation plan, analyses, and dissemination activities. Experts in the fields of health professions training and tribal health reviewed the Tribal HPOG evaluation plan and all comments, questions and suggestions were resolved during consultation.
Consultants are listed below. This consultation took place in 2011 and will continue in the final year of evaluation in 2014.
Name |
Title/ Organization |
Telephone number |
Matthew L. Boulton, MD, MPH |
Associate Professor of Epidemiology; Associate Professor of Health Management and Policy; Associate Professor of Preventive Medicine; Associate Professor of Internal Medicine, Infectious Disease Division; Director, Preventive Medicine Residency; Director, Center of Excellence for Public Health Workforce Studies, University of Michigan School of Public Health |
(734) 936-1623 |
Mark Doescher, MD, MPH |
Director, Washington-Wyoming-Alaska-Montana-Idaho (WWAMI) Rural Health Research Center; Director, University of Washington Center for Health Workforce Studies; Associate Professor, University of Washington (UW) Department of Family Medicine; Associate Director of the WWAMI Area Health Education Center |
(206)616-9207 |
Kristie Gebbie, DrPH, RN |
Joan Hansen Grabe Dean of the School of Nursing, Hunter College, CUNY (retired) |
Living abroad; contact through email at [email protected] |
Jacque Gray, PhD |
Assistant Professor, Center for Rural Health, University of North Dakota School of Medicine and Health Sciences |
(701) 777-0582 |
Felicia Schanche Hodge, DrPH |
Professor and Director, Center for American Indian Research and Education University of California Los Angeles (UCLA) School of Nursing |
(310)267-2255
|
Hugh Tilson, MD, DrPH |
Adjunct Professor, Public Health Leadership Program; Senior Advisor, Public Health Grand Rounds, North Carolina Institute for Public Health, University of North Carolina School of Public Health |
(919)349-0740 |
In an effort to conduct a culturally responsive evaluation grounded in the principles of reciprocity and collaboration, the tribal evaluation team had also presented the draft evaluation plan to the Tribal HPOG grantees via teleconference on March 21, 2011.
Grantees were in agreement with the tribal evaluation team’s evaluation approach. Grantee representatives from the five Tribal HPOG grantee sites, Blackfeet Community College, Cankdeska Cikana Community College, Cook Inlet Tribal Council, Turtle Mountain Community College and College of the Menominee Nation were present on this call for a total of 9 grantee participants. All questions, comments and suggestions were resolved prior to submission of the final evaluation plan.
Explanation of any Payment or Gift to Respondents
Offering tokens of appreciation to gain cooperation and solicit participation is a well-established practice in social science research and program evaluation for both small-scale studies and sample surveys. Participants are provided tokens of appreciation for their time and as a gesture of appreciation for voluntary participation in data collection activities. Tokens of appreciation are needed because it takes effort for a respondent to participate, by dedicating time on one’s schedule for a 90 minute focus group or a 45-60 minute interview and by incurring personal expenses (e.g., transportation costs, child care).
We have worked closely with the tribal grantees to design and implement a culturally responsive evaluation, and asked for their advice on how best to recruit and provide tokens of appreciation to participants for their time and effort in the focus groups and interviews. We learned that participants in tribal programs very often have substantial family commitments, which pose additional burdens compared to other populations. We took this information into consideration when deciding on the appropriate token of appreciation.
Program participants (via focus groups, program completer interviews, or program non-completer interviews) will receive a token of appreciation (as a non-cash gift card) for their participation in this study. The use of tokens of appreciation has been recommend as an effective strategy for increasing response rates in tribal communities, stating “Many research projects (including the authors’) also compensate individuals for their time.”2 Researchers have found financial incentives to be a motivator for tribal participation in research. Use of tokens of appreciation also increase the likelihood that recruited participants will, in fact, participate in the data collection activities, thus reducing the need for additional time and resources for recruitment.
The Tribal HPOG token of appreciation estimate was based on three factors: (1) the anticipated length of the interview; (2) the additional cost of travel to and from the interview site on reservation or off-reservation lands for low-income, tribal respondents; and (3) tribal member reluctance to participate in research activities (based on past experience and corroborating information from the Tribal grantees).
Detailed explanations for each respondent group are below:
Focus group participants: Tokens of appreciation for participation in the focus group will be a non-cash gift card valued at $50. The evaluation team will consult with each grantee to determine the most appropriate gift card to send to the participant.
Program participants, completers and non-completers who participate in telephone interviews: Program participants and completers who participate in telephone interviews in lieu of focus groups will receive a non-cash gift card valued at $25. Non-completers will be offered a $10 non-cash gift card to participate in a 30-minute telephone interview in an effort to overcome any disinclination to participate. The tribal evaluation team will consult with each grantee to determine the most appropriate gift card to send to the participant.
After conducting three years of data collection, our experience can confirm that these levels (of allotment of time to conduct the interviews and tokens of appreciation), which are consistent with what was previously approved, are appropriate. Additionally, our response rate (as describe in SSB “Methods to Maximize Response Rates and Deal with Nonresponse”) is estimated at 90 percent with very few refusals for participation; therefore, we believe that these levels should remain consistent.
Assurance of Confidentiality Provided to Respondents
For the in-person interviews and telephone interviews with grantee/partner administrative staff, program implementation staff, and employers, the tribal evaluation team will be collecting respondents’ names, titles, organization, and contact information (phone number, email address). For program participant focus groups, the tribal evaluation team will collect respondents’ names. For program participant telephone interviews, the tribal evaluation team will collect respondents’ names and contact information (phone number, email address, mailing address), in order to schedule a time for the interview and to send the token of appreciation. Respondents will be told the purposes for which the information is collected and that any identifiable information about them (e.g. respondent names) will not be used or disclosed for any other purpose. This information will be used for recruitment purposes, to establish rapport with respondents, and to provide a token of appreciation for telephone interview respondents.
Personal identifiers will not be linked with individual responses; study IDs will be assigned for data collection, entry and analysis. NORC does not intend to provide Personally Identifiable Information (PII) to ACF and will take measures to protect PII from inadvertent disclosure. Respondent names will not be included in any information viewed by ACF or any other HHS officials. The tribal evaluation team will keep respondent information private and protect it from unauthorized disclosure, tampering or damage. Any potentially identifying information, including recordings, will be kept in a secure location during the period of the study. Information collected will be used only for the purposes of the study and will be destroyed no later than three years after completion of the project.
All respondents will be informed of their rights as study participants. Respondents will be informed that NORC does not intend to provide PII to ACF and will take measures to protect PII from inadvertent disclosure, and that their participation is voluntary. To ensure that all of this information is conveyed to the respondent, informed consent statements will be read at the start of each interview/focus group. In-person and telephone interview respondents will be read an informed consent statement prior to participating in the interview (Attachment A4). Respondents in the program participant focus groups will be asked to complete a written informed consent prior to beginning the focus group (Attachment A5). Both informed consent statements indicate that there are no foreseeable risks to participation; participation is completely voluntary; and participants have the right to withdraw at any time. Consent forms accurately state the intended use of personally identifiable information and the privacy protections.
Further, results presented in the study’s practice briefs and reports will be predominately reported at the grantee level or as an aggregate across all five Tribal HPOG grants. For grantees with multiple implementation sites, site-specific findings will only be presented if the number of respondents in each respondent group (excluding site and partner administrative staff and program non-completers) is greater than 3 individuals. If these privacy conditions are not met for a specific site, then data will be aggregated to the grantee level. Site level data will be reported using study IDs for each site and will exclude any contextual information that would identify a site to further preserve the privacy of the respondents.
All data collection and privacy procedures for qualitative data collection have been approved by the NORC at the University of Chicago Institutional Review Board, a Federally Approved IRB. The qualitative data collection protocol was submitted to the NORC IRB for continuing review in February 2014. IRB approval was granted on March 30, 2014 (See Attachment A6).
Justification for Sensitive Questions
Neither the interviews nor the focus groups include any questions of a sensitive or personal nature. The questions are not designed to solicit personal information from the respondent other than their perspective on the HPOG training program they are associated with.
Estimates of Annualized Burden Hours and Costs
Exhibit 1 summarizes the reporting hours and cost burden on respondents to the Evaluation of the Tribal Health Professions Opportunities Grants. Response times were estimated based on the past 3 years of data collection for the Tribal HPOG Evaluation, prior experience of the contractors with similar data collection, and review of the instruments by ACF and the Advisory Committee. Time per respondent varies according to one’s role. The total estimated annual burden hours for all primary data collection efforts are 493. The total estimated annual cost burden for all primary data collection efforts is $9,688.67.
Note that the instruments and estimated annual burden hours and cost for this final year of data collection activities remain the same as they were for the previous years of data collection for which we received approval.
Instrument |
Annual Number of Respondents |
Number of Responses Per Respondent |
Average Burden Hours Per Response |
Total Annual Burden Hours |
Average Hourly Wage Ratec |
Total Cost Burden |
Grantee and partner administrative staff interview |
35 |
1 |
1 |
35 |
46.42 |
$1624.70 |
Program implementation staff interview |
117 |
1 |
1.5 |
176 |
25.95 |
$4567.20 |
Employer interview |
52 |
1 |
0.75 |
39 |
43.74 |
$1705.86 |
Program participant focus group or interviewa |
117 |
1 |
1.35 |
158 |
7.37 |
$1164.46 |
Program completers interview |
75 |
1 |
1 |
75 |
7.37 |
$552.75 |
Program non-completers interview |
20 |
1 |
0.5 |
10 |
7.37 |
$73.70 |
Total |
|
|
|
493 |
|
$9688.67 |
Exhibit 1. Estimated Burden Hours and Cost
a Focus groups with program participants will be conducted at each primary and secondary program implementation site during final site visits. Focus groups are an estimated 1.5 burden hours. Interviews may be conducted in lieu of focus groups with program participants at secondary implementation sites should scheduling conflicts arise. Interviews are an estimated 0.75 burden hours. A weighted average was calculated to determine average burden hours per response (1.35) for program participant focus groups or interviews.
c Source: Bureau of Labor Statistics, Occupational Employment Statistics, Occupational Employment and Wages, May 2009. The federal minimum wage rate (MWR) of $7.25 is used to estimate an hourly wage rate for program participants, completers, and non-completers to account for possible low-wage employment during the period of program participation. The MWR in North Dakota and Wisconsin is equal to the federal rate. The MWR in Alaska and Montana is higher than the federal rate ($7.75 and $7.35, respectively). A weighted average was used to determine the average hourly wage rate ($7.37) for all data collection activities with program participants, completers and non-completers across the five Tribal HPOG grantees located in North Dakota (2 grantees), Alaska (1 grantee), Montana (1 grantee) and Wisconsin (1 grantee).
Estimates of other Total Annual Cost Burden to Respondents and Record Keepers
Data collection for this study will not result in any additional capital, start-up, maintenance, or purchase costs to respondents or record keepers. Therefore, there is no burden to respondents other than that discussed in the previous section.
Annualized Cost to Federal Government
The total annual cost to the government for this data collection is $192,796. This includes direct and indirect costs of data collection during the final year.
15. Explanation for Program Changes or Adjustments
This is an extension for a previously approved data collection for the Evaluation of the Tribal Health Opportunity Grants. This request if for one final year of data collection, for which there are no program changes or adjustments proposed.
16. Plans for Tabulation and Publication and Project Time Schedule
The data collected in this evaluation has been analyzed and interrupted to produce several practice briefs, conference briefings, site visit/evaluation reports, and an interim report. The data collected in this additional final year will also be analyzed for use in practice briefs, conference briefings, and site visit/evaluation reports, as well as for a final report. The following section describes the analysis that will be conducted and the vehicles for dissemination that will be utilized.
Data Analysis
The evaluation team will use a structured approach to analyze the findings from site visits and telephone interviews. A key feature of the analysis approach will be the utilization of NVivo to store all data collected as part of the project. The utilization of NVivo software will allow for efficient and versatile coding and analyses of the large volume of data we expect to collect for each grantee and across grantees. The team will conduct analyses of these data throughout the data collection period up until the end of the evaluation team contract (September 26, 2015).
The evaluation team has used and will continue to use constant comparison/grounded theory to analyze data. A coding scheme has been developed for analyzing data. Potential updates to the coding scheme will be assessed prior to the site visits. The coding scheme is organized around evaluation topic areas derived from the evaluation questions. The coding scheme will be applied to all data and emergent key themes relating to evaluation topic areas will be identified. The evaluation team will identify emergent themes on two levels: themes which seem to dominate particular grantees; and themes which relate to our overall evaluation questions across all grantee sites. Initially, the evaluation team will establish the emergent themes of focus based on review of qualitative and quantitative data, as well as hypotheses developed by the team following site visits based on their own qualitative perceptions. Next, the evaluation team will use analytic induction to determine if results from a particular interview or focus group are consistent with the emergent key themes and will add to these key themes if appropriate, noting any exceptions and broadening the set of potential hypotheses. Finally, domain analysis will be used to describe how the social/ organizational/ cultural settings of grantees may have played a role in the key themes that emerged from the evaluation. This method will be applied to both findings from individual grantees as well as across all grantees.
Publications
To disseminate what has been learned about the Tribal HPOG programs, the tribal evaluation team has prepared a variety of reports (see list below). The team will use the data collected in this last year to prepare site visit/evaluation reports, a final report, practice briefs, and presentations (see list below). The reports draw data from multiple information sources: the site visits and interviews, reviews of program curricula and documents, and quantitative data elements from the HPOG PRS. The qualitative data collected from the site visits and telephone interviews provide important contextual background and insights for interpreting and understanding Tribal HPOG program operations and performance. Aggregate analyses of quantitative indicators collected through the HPOG PRS are also featured in the reports. To continue our efforts to be collaborative and culturally respectful, Tribal HPOG grantees will continue to have the opportunity to review all of the reports and any additional publications or presentation to be released publicly that identify their specific tribe. Grantees will be provided a reasonable timeline for their review.
Publications that have been released since this data collection was initially approved include:
An Introduction to the Tribal Health Profession Opportunity Grants (HPOG) and Evaluation
http://www.acf.hhs.gov/sites/default/files/opre/tribal_health.pdf
Overview of Tribal Health Profession Opportunity Grants (HPOG) Supportive Services http://www.acf.hhs.gov/sites/default/files/opre/hpog_practice_brief_supportive_services_june_2013_0.pdf
Tribal Health Profession Opportunity Grants (HPOG) Program Implementation & Evolution
Tribal Health Profession Opportunity Grants (HPOG) Program Evaluation: Interim Report
The data collected will shape the following types of publications:
Site Visit/Evaluation Reports will consist of a fifteen to twenty page grantee program report, including background on the project, and a summary of findings organized by core topic areas. These reports will include only aggregate results from each stakeholder group and will not identify particular individuals. The Site Visit/Evaluation Reports will be prepared by the tribal evaluation team and shared with the grantees and ACF.
The Final Report will build upon the structure of the previously developed Interim Report. It will reflect the aggregated analysis of all qualitative data collected and quantitative data obtained via the PRS. Results will be summarized for the initiative as a whole as well as for each grantee. Grantee-specific differences will be discussed in an effort to identify key drivers of program capacity building and participant outcomes.
The evaluation team will prepare Practice Briefs that describe the Tribal HPOG programs, present program highlights and accomplishments to date and share important lessons learned. They will be disseminated through appropriate stakeholders and targeted to tribes and communities participating in health profession training.
Presentations will serve varying purposes including providing background on the project and its purpose; discussing what the training programs are and how they are being implemented; providing a status update and key findings; and reporting final results and conclusions. Presentations can also be used for staff trainings, community organization or tribal council meetings, or for briefings with other relevant stakeholders. Presentations may be conducted by NORC and/or in collaboration with Tribal HPOG grantees.
Time Schedule
Years 2-4 Activities |
Completion Date |
Conducted Initial Site Visits |
02/12-11/12 |
Conducted Follow-up 1 Site Visits |
02/13-10/13 |
Conducted data Analyses |
03/12-12/12; 03/13-12/13 |
Developed Practice Briefs |
12/11; 07/13; 02/14 |
Developed Interim Report |
03/14 |
Year 5 Activities |
Expected Date |
Conduct Follow-up 2 Site Visits |
1-8 months following OMB approval |
Conduct Data Analysis |
1-12 months following OMB approval |
Develop Practice Briefs |
1-12 months following OMB approval |
Develop Final Report |
6-12 months following OMB approval |
17. Reason(s) Display of OMB Expiration Date is Inappropriate
ACF does not seek this exemption.
18. Exemption to Certification for Paperwork Reduction Act Submissions
There are no exceptions to the certification.
1 As referenced in SSB “Methods to Maximize Response Rates and Deal with Nonresponse,” the reduction in numbers is not due to nonresponse; rather, it is due to contact information (email and phone) that is out of date and we cannot contact the students, resulting in a smaller group of students to contact for interviews.
2 Sobeck, Joanne L. , Chapleski, Elizabeth E. and Fisher, Charles(2003) 'Conducting Research with American Indians', Journal of Ethnic And Cultural Diversity in Social Work, 12: 1, 69 — 84.
| File Type | application/msword |
| Author | Hilary Scherer |
| Last Modified By | Molly |
| File Modified | 2014-08-20 |
| File Created | 2014-08-15 |