OMB Clearance Application
Evaluation of Tribal Health
Profession Opportunity Grants (HPOG)
Supporting Statement Part A
June 2011
JUSTIFICATION
Background, Need and Circumstances Making the Collection of Information Necessary
The Administration for Children and Families (ACF) is requesting Office of Management and Budget (OMB) approval to collect qualitative data through in-person and telephone interviews and focus groups as part of the evaluation of the Tribal Health Profession Opportunity Grants to develop an understanding of the Program’s effectiveness in meeting the tribal health workforce needs. The Health Profession Opportunity Grants (HPOG) program is funded by H.R. 3590, the Patient Protection and Affordable Care Act (Affordable Care Act), which provides opportunities for education and training for low-income individuals for healthcare professions that pay well and are expected to experience labor shortages—including low-income individuals living on tribal reservations.
The HPOG program, administered by the Administration for Children and Families (ACF), funded 32 five-year demonstration projects to design and implement innovative health workforce development training programs targeting Temporary Assistance for Needy Families (TANF) recipients and other low-income individuals. Grants were funded under two separate Funding Opportunity Announcements (FOA): The Health Profession Opportunity Grants to Serve TANF and Other Low-Income Individuals (HHS-2010-ACF-OFA-FX-0126) and the Health Profession Opportunity Grants for Tribes, Tribal Organizations or Tribal College or University (HHS-2010-ACF-OFA-FY-0124). Five of the 32 demonstration projects were awarded to Tribal Organizations and Tribal Colleges to develop culturally-informed training programs.
Since the award of these 32 HPOG grantees in 2010, ACF has entered into two separate contracts; one with Abt Associates, in partnership with the Urban Institute, for the Implementation, Systems and Outcome Evaluation of the Health Profession Opportunity Grants to Serve TANF Recipients and Other Low-Income Individuals which includes the design and operation of an HPOG Performance Reporting System (PRS) to collect quantitative program management data from all 32 grantees. The second evaluation, and the subject of this OMB Supporting Statement, is with NORC at the University of Chicago for an evaluation of the five Tribal HPOG programs. ACF has contracted with NORC and its partners, Red Star Innovations and the National Indian Health Board (the ‘tribal evaluation team’), to conduct an evaluation that uses a culturally responsive framework to build upon the conceptual framework guiding the Tribal HPOG programs.
Although the Affordable Care Act mandates an evaluation of the HPOG demonstration projects (H.R. 3590, Title V, Subtitle F, Sec. 5507, sec. 2008, (a)(3)(B)), it does not stipulate that a separate evaluation of the Tribal HPOG grantees be conducted. To determine whether and how the Tribal component of the HPOG program meets its goals, ACF plans to conduct a comprehensive, multi-faceted and rigorous evaluation approach to assess the implementation and outcomes of the five Tribal grantees.
As noted above, ACF issued a separate FOA for the Tribal Health Professions Opportunity Grants. This Tribal FOA indicates that “In an effort to address healthcare disparities and improve the quality of life of 1.9 million American Indians and American Natives, the Patient Protection and Affordable Care Act reauthorized the Indian Health Care Improvement Act (IHCIA), the cornerstone legal authority for the provision of health care to American Indians and Alaska Natives. The Indian Health Care Improvement Act (Pub. L. No. 94-437) authorizes the creation of elder care programs, programs for hospice, assisted living, long-term care and home and community-based services. These facilities will need qualified workers and will need to expand current staff knowledge. Providing education and training to upgrade skills is vital to attaining these goals. Therefore, the opportunity of this funding can encourage Tribal communities to design, develop and implement training programs to address the healthcare workforce needs of their communities.”
ACF determined that a separate evaluation of the tribal grantees would be necessary and useful to expand the knowledge base about effective practices in Tribal communities. As further described below, the evaluation will address the needs of ACF to learn about how Tribal communities design, develop and implement training programs to enhance the health care workforce needs of the community. The Tribal HPOG evaluation provides an opportunity for evaluating the implementation and outcomes of five distinct workforce development approaches that integrate health professional training programs with culturally-informed models of learning and practice.
Three key research questions will guide this evaluation:
Have grantees incorporated structures necessary to enhance the health care workforce needs of the community?
Have grantees implemented processes to prepare participants for employment in the tribal health care sector?
Is there evidence that participants in the program achieved successful employment and work force capacity building outcomes?
The five grantees are tribes, tribal organizations, tribal colleges or tribal universities. Tribal grantees will integrate health professionals training programs with culturally-informed models of learning and practice, such as the Family Education Model, a framework for postsecondary education that creates an extended family structure within the academic experience. The evaluation being conducted by NORC and its partners will provide ACF with a wide range of qualitative data to support program management over time and evaluate the success of tribal grantees’ efforts. Through an evaluation of these funded grants serving tribal communities, the way in which tribal grantees are implementing, creating and adapting culturally‐relevant and appropriate programs will be documented, outcomes will be quantified, and lessons learned will be disseminated.
Qualitative Data Collection The proposed Tribal HPOG evaluation will primarily rely on qualitative data collected by the tribal evaluation team, i.e.,interviews and/or focus groups with program administrators, staff, participants, and employers.
Data collection will take place primarily during annual site visits with each tribal grantee. Three in-person site visits will be conducted with each grantee over the course of the data collection period. Initial site visits will be conducted in 2011-2012; the first round of follow up site visits (follow-up 1) will be conducted in 2012-2013; and the second round of follow up site visits (follow-up 2) will be conducted in 2013-2014. Data collection activities will take place at each of the 13 sites (5 prime implementation sites; 8 secondary implementation sites) where the Tribal HPOG programs are being implemented. Specifically, the tribal evaluation team will conduct the following activities:
Site visits to the thirteen Tribal HPOG grantee implementation sites. Site visits will include the following data collection activities:
Interviews with Tribal HPOG Grantee and Partner Administrative Staff and Partners. In-person interviews will be conducted on an annual basis to gain insight from grantee and partner administrative staff (including directors and up to 6 grantee and partner administrative staff from each grantee program) on high-level program strategies, program development and lessons learned, for a total of up to 35 interviews annually.
Interviews with Tribal HPOG Program Implementation Staff. Interviews will be conducted with staff responsible for coordinating and implementing the program at each site. These individuals may include program instructors, recruitment and orientation staff, and providers of supportive services. Phone interviews may be conducted in lieu of in-person interviews with implementation staff at secondary implementation sites during the middle data collection year (follow-up 1). Up to 9 program implementation staff at each program implementation site will be interviewed, for a total of up to 117 interviews annually.
Interviews with Employers. Employers’ perspectives on the utility of the training programs are critical, so it will be essential to capture their perceptions and experiences. Phone interviews may be conducted in lieu of in-person interviews with employers at secondary implementation sites in the middle data collection year (follow-up 1). Up to 4 employers at each program implementation site will be interviewed, for a total of up to 52 interviews annually.
Focus groups or Interviews with Program Participants. Focus groups and interviews will be conducted with program participants, however no participant will respond through both modes. (That is to say, if a current program participant participates in a focus group, he/she will not be engaged for a program participant interview). The tribal evaluation team will conduct focus groups with program participants at each primary program implementation site during the annual site visits and at secondary sites during initial and follow-up 2 visits. Phone interviews may be conducted in lieu of focus groups with program participants at secondary implementation sites in the middle data collection year (follow-up 1). Up to 9 program participants at each program implementation site will participate in focus groups or interviews, for a total of up to 117 program participant respondents annually.
Telephone Interviews with Program Completers. In order to obtain information on key program outcomes related to educational attainment and employment, interviews will also be conducted with program completers of the Tribal HPOG program. (It is possible for program participants to have participated in a focus group while they are in the program and then to participate in an interview after they have completed the program. If this is the case, we will not repeat any of the questions from the earlier data collection. However, questions such as “What program did you participate in? Why did you choose this program?” will need to be asked of respondents who have not previously provided those answers. As such, the modes of data collection are not redundant, and instead seek to collect comparable information across respondents). These interviews will occur at two points in time - after initial site visits and after follow-up 1 site visits. Up to 20 interviews will be conducted with program participants across implementation sites in each grantee program, for a total of up to 100 interviews annually (200 total over the duration of the data collection period).
Telephone Interviews with Program Non-Completers. Phone interviews will also be conducted with participants that have not completed the Tribal HPOG program. Interviews will begin after initial site visits and will continue through follow-up 2 site visits. Up to 4 interviews will be conducted with program non-completers across implementation sites in each grantee program, for a total of up to 20 interviews annually.
Secondary Analysis of Quantitative Data from the HPOG Performance Reporting System
To gather quantitative data for the HPOG program, ACF has contracted with Abt Associates and the Urban Institute to develop a quantitative data collection system (‘Performance Reporting System’) for the entire pool of HPOG grantees, including the tribal grantees. ACF has submitted an OMB clearance package for the collection of this quantitative data (OMB Control Number: 0970-0394).
The tribal evaluation team will utilize the data collected through the Performance Reporting System (PRS), but will not collect additional quantitative data from grantees in an effort to reduce burden.
NORC will be using the PRS as a data source in completing the Tribal HPOG Evaluation. However, the data captured through the PRS needs to be supplemented in order to conduct a comprehensive evaluation of the Tribal HPOG programs. NORC will use the PRS data to report quantitative information, such as how many participants each program serves, retention rates, completion rates, etc. In addition, NORC needs to collect qualitative information to understand what components of the programs’ design and implementation worked or did not work, and why. While the PRS data helps quantify outcomes, the NORC-collected qualitative data enables an understanding of factors that influenced those outcomes, including facilitators and barriers. For example, the PRS will provide NORC with the number of enrollees in each program. Interviews with program participants will provide descriptions of why participants enrolled in that program.
This collection of data is authorized by Section 301 of the U.S. Public Health Service Act (42 U.S.C.241). A copy of this legislation can be found in Attachment A1.
Purpose and Use of Information Collection
The information collected as part of this evaluation will be used to determine lessons about developing and implementing the Tribal HPOG program, such as identifying the contextual factors that impacted successes and challenges. This evaluation has the potential to identify promising practices and inform the knowledge base of culturally-based models implemented in tribal settings. The proposed approach allows for the examining of the Tribal HPOG initiative as a whole as well as focusing on individual Tribal HPOG programs.
Qualitative data collected as part of the Tribal HPOG Evaluation will serve to complement the PRS data, not duplicate or replace it. As noted above, the PRS is an important data source for the evaluation conducted by NORC, and will provide necessary data on intake, retention, and completion rates, as well as the rates of services provided. However, additional qualitative information is necessary to conduct a comprehensive evaluation of the design, implementation, and outcomes of the Tribal HPOG programs. The qualitative data collection components, including interviews and site visits, will provide necessary information on promising practices, lessons learned, and implementation facilitators and barriers which can then be used to inform future tribal health professions programs.
Furthermore, the dissemination of project findings to stakeholders in tribal health will provide documentation and lessons learned around programmatic approaches to health professions training serving tribal populations. By engaging tribal officials, tribal and local program administrators and partners, tribal service agencies/organizations, public health, policy and education research consortiums and other stakeholders, ACF will be positioned to facilitate increased knowledge sharing and action around training programs for special populations. The dissemination of findings will provide stakeholders with knowledge of efforts to mitigate the health professions workforce shortage that exists in tribal communities; communication of the purpose and design of the evaluation, highlighting culturally specific components; status updates throughout the evaluation process, including report of interim findings and final results; qualitative and quantitative findings that will support health professions training program improvement over time; knowledge that will enhance programmatic and culturally specific approaches to health professions training and facilitate increased knowledge sharing and action around training programs for special populations; and notification of opportunities for involvement/ dialogue.
Use of Improved Information Technology and Burden Reduction
The proposed evaluation is designed to limit the reporting burden for respondents. Where possible the tribal evaluation team will gather information electronically. For example, grantee program documents and curricula will be collected via email, and then analyzed by the tribal evaluation team.
The bulk of information gathered by the evaluation team will be done through personal interactions during the site visits. When working with American Indian/Alaska Native populations, it is important to consider cultural appropriateness. Among tribal populations, the development of relationships and person-to-person connections is paramount. In order to gather the most accurate and unbiased data possible, the tribal evaluation team believes that in-person interviews, focus groups and phone calls are the most appropriate means to collect information and will produce the most in-depth, accurate, comprehensive and rich data. Best practices for data collection in tribal communities stress the importance of developing trusting relationships with community members and program participants, either through face-to-face meetings or by working through trusted intermediaries. These practices lend themselves to qualitative research methods, so that the tribal evaluation team’s approach is to conduct “in-person interviews, focus groups and phone calls” as opposed to mail or web-based surveys. For this reason, the tribal evaluation team plans to conduct annual site visits to grantee program implementation sites.
The tribal evaluation team will minimize burden to respondents by providing discussion topics in advance of the call, reducing the burden of the interview and ensuring that discussions are focused and require as little time as possible for the respondents. In-person and telephone interviews will last between 30 and 90 minutes depending upon respondent group; focus groups will last approximately 90 minutes. As such, respondents will not be responsible for providing information beyond their oral responses to interview and/or focus group questions. The tribal evaluation team will schedule the interviews for a time that is convenient for the respondents, and be accommodating should they need to reschedule.
As noted above, in a further effort to reduce burden, the Tribal HPOG evaluation will utilize the quantitative data already being collected through the PRS rather than collecting additional quantitative data from grantees. A clearance package for the HPOG Performance Reporting System has been submitted to the Office of Management and Budget (OMB Control Number: 09700394).
Efforts to Identify Duplication and Use of Similar Information
The purpose of this data collection effort is to collect information from key informants to evaluate the Tribal Health Professions Opportunity Grants. There have been no previous efforts to collect data from stakeholders of the HPOG program as this is a new grant recently funded through the Affordable Care Act. This evaluation will fill a need for data demonstrating the effectiveness of health professions diversity programs as there is limited evidence on the intervention efficacy of health professions diversity programs and few outcome evaluations using rigorous study designs have been conducted.
Qualitative data collected as part of the Tribal HPOG Evaluation will serve to complement the PRS data. The qualitative interview data will capture stakeholders’ perceptions of the effectiveness of the processes and structures that were put in place to implement the HPOG program and how each program and its various components are tailored to meet the needs of diverse Tribal communities.
The data captured in the PRS and the information collected from the interviews serve different but complimentary purposes. The PRS will capture data on key indicators for internal program management and semi-annual reporting to ACF (i.e., participant demographics, number enrolled, numbers exited, number and types of public benefits received and supportive services provided, educational and employment history of participants, educational and training activities provided, required courses for training programs, educational and employment outcomes). NORC will receive from the Urban Institute individual-level Tribal HPOG data without identifiers from the HPOG PRS on a semi-annual basis. These data will be used for secondary data analyses conducted by the Tribal HPOG evaluation team for each of the five tribal program and its sites, and aggregated across the grantees.
In contrast, the goal of qualitative interviewing is to obtain rich information and insight to describe or explore phenomena through interpersonal interaction or observation. The interview process is expressly designed to establish rapport with the respondent and to elicit information in a voluntarily manner about one’s motivation for and participation in the program. As sound interviewing techniques require establishing a comfort zone and easing into an interview with some preliminary conversation, as well as tracing the key events of a program participant’s journey through a program, the interviewers will need to reiterate some of the specific information documented in the PRS anyway in order to engage the respondent (e.g., What program did you participate in? What services were provided? What courses did you take? What certificate did you receive? Are you currently employed? etc.). In contrast, to begin each interview by providing information about the respondent derived from the PRS would not be well-received nor would the amount of time gained be of consequence (i.e., burden reduced). Participants would find it off-putting for a stranger to present them with personal information and perceive this as invasive and possibly coercive.
ACF and the tribal evaluation team have examined the overlap in questions in the PRS and interview/focus group protocols. Attachment A2 identifies those questions in the interview and focus group protocols that ask for similar data to that collected in the PRS. In the final column, we articulate how we will streamline or why we think this apparent duplication is necessary. Please note that in the majority of cases, this information is asked only as a lead question to allow the interviewer to frame the question. Having these data from the PRS would not significantly reduce the burden to grantees – as this information would need to be stated by the interviewer to frame the question (and as mentioned above, this may be off-putting to respondents).
Impact on Small Businesses or Other Small Entities
This study will have minimal impact on small entities, as small business professionals will only be interviewed if they are employers of HPOG program graduates or grantee administrative partners. In an effort to reduce burden, the duration of each employer interview will be no more than 45 minutes. The perspective of program participant employers is essential to capture employers’ general impressions of program graduates and the extent to which tribal health workforce capacity has been improved by HPOG.
Consequences of Collecting the Information Less Frequently
The design of this evaluation engages respondents as infrequently as possible. Data collection will take place primarily during annual site visits with each tribal grantee. Each site visit will be approximately three to five days in length.
Decreasing the frequency of site visits (i.e., scheduling fewer than one visit per year), would result in a lack of essential and timely information. Collecting data less frequently would preclude the tribal evaluation team from collecting essential information about program progression from year to year. Since this grant is administered over five years, three years of annual data collection at each site will provide a rich set of data that will clearly demonstrate program successes or struggles over time, and allow for the evaluation of Tribal HPOG grantee program. The federal government will benefit from having annual information available on the processes and outcomes of these grant programs
There are no legal obstacles to reduce the burden.
Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2).
Comments in Response to the Federal Register Notice/ Outside Consultation
A 60-Day Federal Register Notice was published in the Federal Register on 4/18/2011, vol. 76, No. 74, pp. 21749-21750 (see Attachment A3). There were no public comments.
The tribal evaluation team staff consulted are listed below. This consultation took place in 2011.
Name |
Title/ Organization |
Telephone number |
Email address |
Michael Meit, MA, MPH |
Program Area Director, Public Health, NORC at the University of Chicago |
(301)634-9324 |
|
Alana Knudson, PhD |
Principal Research Scientist, Public Health, NORC at the University of Chicago |
(301)634-9326 |
|
Carol Hafford, PhD |
Senior Research Scientist, Economics, Labor and Population Studies, NORC at the University of Chicago |
(301)634-9491 |
|
Aleena Hernandez, MPH |
Principal and Founder, Red Star Innovations, LLC |
(520) 407-6307
|
|
Evangelyn Dotomain, MBA |
Deputy Director, National Indian Health Board |
(202)374-2034
|
|
Jessica Bushar, MPH |
Senior Research Analyst, Public Health Research, NORC at the University of Chicago |
(301)634-9515 |
|
Hilary Scherer, BA |
Research Analyst, Public Health Research, NORC at the University of Chicago |
(301) 634-9374 |
|
Heather Langerman, BS |
Research Assistant, Public Health Research, NORC at the University of Chicago |
(301)634-9516 |
The tribal evaluation team has consulted with outside experts about the proposed data collection and evaluation plan. Experts in the fields of health professions training and tribal health reviewed the Tribal HPOG evaluation plan and all comments, questions and suggestions were resolved during consultation.
Consultants are listed below. This consultation took place in 2011.
Name |
Title/ Organization |
Telephone number |
Matthew L. Boulton, MD, MPH |
Associate Professor of Epidemiology; Associate Professor of Health Management and Policy; Associate Professor of Preventive Medicine; Associate Professor of Internal Medicine, Infectious Disease Division; Director, Preventive Medicine Residency; Director, Center of Excellence for Public Health Workforce Studies, University of Michigan School of Public Health |
(734) 936-1623 |
Mark Doescher, MD, MPH |
Director, Washington-Wyoming-Alaska-Montana-Idaho (WWAMI) Rural Health Research Center; Director, University of Washington Center for Health Workforce Studies; Associate Professor, University of Washington (UW) Department of Family Medicine; Associate Director of the WWAMI Area Health Education Center |
(206)616-9207 |
Kristie Gebbie, DrPH, RN |
Joan Hansen Grabe Dean of the School of Nursing, Hunter College, CUNY (retired) |
Living abroad; contact through email at [email protected] |
Jacque Gray, PhD |
Assistant Professor, Center for Rural Health, University of North Dakota School of Medicine and Health Sciences |
(701) 777-0582 |
Felicia Schanche Hodge, DrPH |
Professor and Director, Center for American Indian Research and Education University of California Los Angeles (UCLA) School of Nursing |
(310)267-2255
|
Hugh Tilson, MD, DrPH |
Adjunct Professor, Public Health Leadership Program; Senior Advisor, Public Health Grand Rounds, North Carolina Institute for Public Health, University of North Carolina School of Public Health |
(919)349-0740 |
In an effort to conduct a culturally responsive evaluation grounded in the principles of reciprocity and collaboration, the tribal evaluation team presented the draft evaluation plan to the Tribal HPOG grantees via teleconference on March 21, 2011. Prior to the teleconference, grantees received a copy of the draft evaluation plan. This draft evaluation plan did not include the actual data collection protocols the tribal evaluation team will use for interviews and focus groups.
Grantees were in agreement with the tribal evaluation team’s evaluation approach. Grantee representatives from the five Tribal HPOG grantee sites, Blackfeet Community College, Cankdeska Cikana Community College, Cook Inlet Tribal Council, Turtle Mountain Community College and College of the Menominee Nation were present on this call for a total of 9 grantee participants. All questions, comments and suggestions were resolved prior to submission of the final evaluation plan.
Explanation of any Payment/Gift to Respondents
Offering incentives to gain cooperation and solicit participation is a well-established practice in social science research and program evaluation for both small-scale studies and sample surveys. Participants are provided incentives to compensate them for their time and as a gesture of appreciation for voluntary participation in data collection activities. Incentives are needed because it takes effort for a respondent to participate, by dedicating time on one’s schedule for a 1 ½ hour focus group or a 45-60 minute interview and by incurring personal expenses (e.g., transportation costs, child care).
Use of incentives also increases the likelihood that recruited participants will, in fact, participate in the data collection activities, thus reducing the need for additional time and resources for recruitment.
We have worked closely with the tribal grantees to design and implement a culturally responsive evaluation, and asked for their advice on how best to recruit and compensate participants for their time and effort in the focus groups and interviews. We learned that participants in tribal programs very often have substantial family commitments, which pose additional burdens compared to other populations. We took this information into consideration when deciding on the appropriate incentive fee.
Program participants; program completers and non-completers; and employers will receive a non-cash gift card or voucher in return for their participation in this study. The use of monetary incentives has been recommend as an effective strategy for increasing response rates in tribal communities, stating “Many research projects (including the authors’) also compensate individuals for their time.”1 Researchers have found financial incentives to be a motivator for tribal participation in research. Use of incentives also increases the likelihood that recruited participants will, in fact, participate in the data collection activities, thus reducing the need for additional time and resources for recruitment.
The Tribal HPOG incentive estimate was based on three factors: (1) the anticipated length of the interview (1 hour); (2) the additional cost of travel to and from the interview site on reservation or off-reservation lands for low-income, tribal respondents; and (3) tribal member reluctance to participate in research activities (based on past experience and corroborating information from the Tribal grantees).
Detailed explanations for each respondent group are below:
Focus group participants: Incentives for participation in the focus group will be a non-cash honorarium valued at $50. The Evaluation Team will consult with each grantee to determine the most appropriate method of payment (e.g., gift certificate to a local grocery store) to send to the participant.
Program participants, completers and non-completers who participate in telephone interviews: Program participants and completers who participate in telephone interviews in lieu of focus groups will receive a non-cash honorarium valued at $25. Non-completers will be offered a $10 incentive to participate in a 30-minute telephone interview in an effort to overcome any disinclination to participate. The tribal evaluation team will consult with each grantee to determine the most appropriate method of payment (e.g., gift certificate to a local grocery store) to send to the participant.
Assurance of Confidentiality
For the in-person interviews and telephone interviews with grantee and partner administrative staff, program implementation staff, and employers, the tribal evaluation team will be collecting respondents’ names, titles, organization, and contact information (phone number, email address, mailing address). For program participant focus groups and telephone interviews, the tribal evaluation team will collect respondents’ names and contact information (phone number, email address, mailing address), in order to schedule a time for the interview or focus group. Respondents will be told the purposes for which the information is collected and that any identifiable information about them (e.g. respondent names) will not be used or disclosed for any other purpose. This information will be used for recruitment purposes only and to establish rapport with respondents.
Personal identifiers will not be linked with individual responses; study IDs will be assigned for data collection, entry and analysis. NORC does not intend to provide Personally Identifiable Information (PII) to ACF and will take measures to protect PII from inadvertent disclosure. Respondent names will not be included in any information viewed by ACF or any other HHS officials. The tribal evaluation team will keep respondent information private and protect it from unauthorized disclosure, tampering or damage. Any potentially identifying information, including recordings, will be kept in a secure location during the period of the study. Information collected will be used only for the purposes of the study and will be destroyed no later than three years after completion of the project.
All respondents will be informed of their rights as study participants. Respondents will be informed that NORC does not intend to provide Personally Identifiable Information (PII) to ACF and will take measures to protect PII from inadvertent disclosure, and that their participation is voluntary. To ensure that all of this information is conveyed to the respondent, informed consent statements will be read at the start of each interview/focus group. In-person and telephone interview respondents will be read an informed consent statement prior to participating in the interview (Attachment A4). Respondents in the program participant focus groups will be asked to complete a written informed consent prior to beginning the focus group (Attachment A5). Both informed consent statements indicate that there are no foreseeable risks to participation; participation is completely voluntary; and participants have the right to withdraw at any time. Consent forms accurately state the intended use of personally identifiable information and the privacy protections.
Further, results presented in the study’s briefs and reports will be predominately reported at the grantee level or as an aggregate across all five Tribal HPOG grants. For grantees with multiple implementation sites, site-specific findings will only be presented if the number of respondents in each respondent group (excluding site and partner administrative staff and program non-completers) is greater than 3 individuals. If these privacy conditions are not met for a specific site then data will be aggregated to the grantee level. Site level data will be reported using study IDs for each site and will exclude any contextual information that would identify a site to further preserve the privacy of the respondents.
All data collection and privacy procedures for qualitative data collection have been approved by the NORC at the University of Chicago Institutional Review Board, a Federally Approved IRB. The qualitative data collection protocol was submitted to the NORC IRB in February 2011. IRB approval was granted on February 22, 2011 (See Attachment A6).
Justification for Sensitive Questions
Neither the interviews nor the focus groups include any questions of a sensitive or personal nature. The questions are not designed to solicit personal information from the respondent other than their perspective on the HPOG training program they are associated with.
Estimates of Annualized Hour and Cost Burden to Respondents
Exhibit 1 summarizes the reporting hours and cost burden on respondents to the Evaluation of the Tribal Health Professions Opportunities Grants. Response times were estimated based on prior experience of the contractors with similar data collection and review of the instruments by ACF and the Advisory Committee. Time per respondent varies according to one’s role. The total estimated annual burden hours for all primary data collection efforts are 485. The total estimated annual cost burden for all primary data collection efforts is $9,629.71.
Instrument |
Annual Number of Respondents |
Number of Responses Per Respondent |
Average Burden Hours Per Response |
Total Annual Burden Hours |
Average Hourly Wage Ratec |
Total Cost Burden |
Grantee and partner administrative staff interview |
35 |
1 |
1 |
35 |
46.42 |
$1624.70 |
Program implementation staff interview |
117 |
1 |
1.5 |
176 |
25.95 |
$4567.20 |
Employer interview |
52 |
1 |
0.75 |
39 |
43.74 |
$1705.86 |
Program participant focus group or interviewa |
117 |
1 |
1.35 |
158 |
7.37 |
$1164.46 |
Program completers interviewb |
67 |
1 |
1 |
67 |
7.37 |
$493.79 |
Program non-completers interview |
20 |
1 |
0.5 |
10 |
7.37 |
$73.70 |
Total |
|
|
|
485 |
|
$9629.71 |
Exhibit 1. Estimated Burden Hours and Cost
a Focus groups with program participants will be conducted at each primary program implementation site during annual site visits and at secondary implementation sites during initial and follow-up 2 visits (N=117 program participant focus group respondents during initial and follow-up 2 visits, respectively; N=45 program participant focus group respondents during follow-up 1 visit). Focus groups are an estimated 1.5 burden hours. Interviews may be conducted in lieu of focus groups with program participants at secondary implementation sites in the middle data collection year (follow-up 1) (N=72 interview program participant respondents). Interviews are an estimated 0.75 burden hours. A weighted average was calculated to determine average burden hours per response (1.35) for program participant focus groups or interviews.
b Up to 200 program completer respondents will be interviewed over the three year data collection period with an average of 67 program completer respondents per year.
c Source: Bureau of Labor Statistics, Occupational Employment Statistics, Occupational Employment and Wages, May 2009. The federal minimum wage rate (MWR) of $7.25 is used to estimate an hourly wage rate for program participants, completers, and non-completers to account for possible low-wage employment during the period of program participation. The MWR in North Dakota and Wisconsin is equal to the federal rate. The MWR in Alaska and Montana is higher than the federal rate ($7.75 and $7.35, respectively). A weighted average was used to determine the average hourly wage rate ($7.37) for all data collection activities with program participants, completers and non-completers across the five Tribal HPOG grantees located in North Dakota (2 grantees), Alaska (1 grantee), Montana (1 grantee) and Wisconsin (1 grantee).
Estimates of other Total Annual Cost Burden to Respondents or Recordkeepers/Capital Costs
Data collection for this study will not result in any additional capital, start-up, maintenance, or purchase costs to respondents or record keepers. Therefore, there is no burden to respondents other than that discussed in the previous section.
Estimates of Annualized Cost to Federal Government
The costs for conducting the Evaluation of the Tribal HPOG program are primarily included in the contract between the ACF and NORC at the University of Chicago under HHS Contract Number HHSP23320095647WC. The estimated annual cost of the data collection to be conducted by NORC is $322,782.33. The annualized cost is based on the prime contractor's budget for the task of conducting the data collection, analysis, and dissemination activities with five tribal sites. These projected costs were based on a detailed budget analysis of the labor time and other costs needed to meet the specifications for this component of the procurement. Incentive payments for the first year of data collection total $6,050.
The estimated annual cost of federal employers involved in the oversight of this data collection is $1,155.20. The total annual cost to the government for this data collection is $317,287.53.
15. Explanation for Program Changes of Adjustments
This is a new data collection.
16. Plans for Tabulation and Publication and Project Time Schedule
The data collected in this evaluation will be analyzed and interrupted to produce briefings, as well as interim and final reports. The following section describes the analysis that will be conducted and the vehicles for dissemination that will be utilized.
Data Analysis
The evaluation team will use a structured approach to analyze the findings from site visits and telephone interviews. A key feature of the analysis approach will be the utilization of NVivo to store all data collected as part of the project. The utilization of NVivo software will allow for efficient and versatile coding and analyses of the large volume of data we expect to collect for each grantee and across grantees. The team will conduct analyses of these data throughout the data collection period up until the end of the evaluation team contract (September 26, 2014).
The evaluation team will use constant comparison/grounded theory to analyze data. NVivo will be used to develop a coding scheme for analyzing these data. The coding scheme will be organized around evaluation topic areas derived from the evaluation questions. The coding scheme will be applied to all data and emergent key themes relating to evaluation topic areas will be identified. The evaluation team will identify emergent themes on two levels: themes which seem to dominate particular grantees; and themes which relate to our overall evaluation questions across all grantee sites. Initially, the evaluation team will establish the emergent themes of focus based on review of qualitative and quantitative data, as well as hypotheses developed by the team following site visits based on their own qualitative perceptions. Next, the evaluation team will use analytic induction to determine if results from a particular interview or focus group are consistent with the emergent key themes and will add to these key themes if appropriate, noting any exceptions and broadening the set of potential hypotheses. Finally, domain analysis will be used to describe how the social/ organizational/ cultural settings of grantees may have played a role in the key themes that emerged from the evaluation. This method will be applied to both findings from individual grantees as well as across all grantees.
Publications
To disseminate what has been learned about the Tribal HPOG programs, the tribal evaluation team will prepare a variety of reports. These reports will draw data from multiple information sources: the annual site visits and interviews, reviews of program curricula and documents, and quantitative data elements from the HPOG Performance Reporting System. The qualitative data collected from the annual site visits and telephone interviews will provide important contextual background and insights for interpreting and understanding Tribal HPOG program operations and performance. Aggregate analyses of quantitative indicators collected through the HPOG Performance Reporting System will also be featured in the reports. To further our efforts to be collaborative and culturally respectful, Tribal HPOG grantees will have the opportunity to review all of the above reports and any additional publications or presentation to be released publicly that identify their specific tribe. Grantees will be provided a reasonable timeline for their review.
The data collected will shape the following types of publications:
Site Visit Reports will consist of a two-to-three page grantee program summary, including background on the project, and a summary of findings organized by core topic areas. These reports will include only aggregate results from each stakeholder group and will not identify particular individuals. The Site Visit Reports will be prepared by the tribal evaluation team annually.
An Interim Report will be prepared that presents key findings to date related to key evaluation questions. This will include descriptions of program types and administrative structures, partnerships, program design and curricula, skills and competencies needed for the local health care workforce, and contextual factors.
The Final Report will build upon the structure of the Interim Report. It will reflect the aggregated analysis of all qualitative data collected and quantitative data obtained via the Performance Reporting System. Results will be summarized for the initiative as a whole as well as for each grantee. Grantee-specific differences will be discussed in an effort to identify key drivers of program capacity building and participant outcomes.
On an annual basis the evaluation team will prepare Practice Briefs that describe the Tribal HPOG programs, present program highlights and accomplishments to date and share important lessons learned. They will be disseminated through appropriate stakeholders and targeted to tribes and communities participating in health profession training.
Presentations will serve varying purposes including providing background on the project and its purpose; discussing what the training programs are and how they are being implemented; providing a status update and preliminary findings; and reporting final results and conclusions. Presentations can also be used for staff trainings, community organization or tribal council meetings, or for briefings with other relevant stakeholders. Presentations may be conducted by NORC and/or in collaboration with Tribal HPOG grantees.
Time Schedule
Activity |
Expected Date |
Conduct Initial Site Visits |
1-8 months following OMB approval |
Conduct Follow-up 1 Site Visits |
15-22 months following OMB approval |
Conduct Follow-up 2 Site Visits |
27-34 months following OMB approval |
Conduct Data Analysis |
1-43 months following OMB approval |
Develop Practice Briefs |
4-5 months, 10-11 months, 20-21 months, 27-28 months, 34-35 months following OMB approval |
Develop Interim Reports |
16-17 months and 28-29 months following OMB approval |
Develop Final Report |
43-44 months following OMB approval |
17. Exemption for Display of OMB Expiration Date
ACF does not seek this exemption.
18. Exemption to Certification for Paperwork Reduction Act Submissions
There are no exceptions to the certification.
1 Sobeck, Joanne L. , Chapleski, Elizabeth E. and Fisher, Charles(2003) 'Conducting Research with American Indians', Journal of Ethnic And Cultural Diversity in Social Work, 12: 1, 69 — 84.
| File Type | application/msword |
| Author | Hilary Scherer |
| Last Modified By | Department of Health and Human Services |
| File Modified | 2011-10-17 |
| File Created | 2011-10-17 |