Supporting Statement
for
Clearance of Data Collection Instruments for
Evaluation of the
Afghanistan Health Initiative Project
Submitted to
Office of Management and Budget
Office of Information and Regulatory Affairs
Submitted by
U.S. Department of Health and Human Services
Office of the Assistant Secretary for Planning and Evaluation
and
U.S. Department of Health and Human Services
Office of Global Health Affairs
June 25, 2008
Supporting Statement
for
Clearance of Data Collection Instruments for
Evaluation of the
Afghanistan Health Initiative Project
Contents
Background 1
A. Justification 1
1. Need and Legal Basis 1
2. Information Users 2
3. Improved Information Technology 2
4. Duplication of Similar Information 2
5. Small Businesses 2
6. Less Frequent Collection 2
7. Special Circumstances 3
8. Federal Register Notice/Outside Consultation 3
External Consultations 3
9. Payment/Gift to Respondents 4
10. Confidentiality 4
11. Sensitive Questions 4
12. Burden Estimate (Total Hours and Wages) 5
Estimated Annualized Hour Burden 5
Estimated Annualized Respondent Cost Burden 5
13. Capital Costs (Maintenance of Capital Costs) 6
14. Cost to Federal Government 6
15. Program or Burden Changes 6
16. Publication and Tabulation Dates 6
Data Sources 6
Tabulation and Statistical Analysis 9
17. Expiration Date 10
18. Certification Statement 10
B. Collection of Information Employing Statistical Methods 10
1. Respondent Universe and Sampling Methods 10
Qualitative Interviews 10
Standards-Based Management Assessments 11
2. Procedures for the Collection of Information 12
Selection Methods 12
3. Methods to Maximize Response Rates and Deal with Nonresponse 13
4. Tests of Procedures or Methods to be Undertaken 13
5. Individuals
Consulted on Statistical Aspects and Individuals Collecting
and/or
Analyzing Data 14
APPENDIXES
Appendix A: Authorizing Legislation A–1
Appendix B: 20 Key Questions the Evaluation Will Address B–1
Appendix C: 60-Day Federal Register Notice C–1
Appendix D: Topic Guide for Qualitative Interviews with RBH Management Staff D–1
Appendix E: Topic Guide for Qualitative Interviews with RBH Clinicians E–1
Appendix F: SBMA Written Exams and Case Studies for First-Year Resident Physicians F–1
Appendix G: SBMA Written Exams and Case Studies for Second-Year Resident Physicians G–1
Appendix H: SBMA Written Exams and Case Studies for Third-Year Resident Physicians H–1
Appendix I: SBMA Written Exams and Case Studies for Fourth-Year Resident Physicians I–1
Appendix J: SBMA Written Exams and Case Studies for Midwives J–1
Supporting Statement
for
Clearance of Data Collection Instruments for
Evaluation of the
Afghanistan Health Initiative Project
The Offices of Global Health Affairs (OGHA) and the Assistant Secretary for Planning and Evaluation (ASPE), within the U.S Department of Health and Human Services (HHS), are requesting Office of Management and Budget (OMB) approval for collection of information to evaluate two components of the Afghanistan Health Initiative (AHI). The AHI’s goal is to improve maternal and child health and to reduce maternal and child mortality in Afghanistan, primarily through strengthening and updating the knowledge and skills of clinical service providers and managers at the Rabia Balkhi Hospital (RBH) in Kabul. Under the AHI, HHS has funded separate cooperative agreements with International Medical Corps (IMC) and CURE International (CURE) to support (1) education and training in obstetrics and gynecology to residents and midwives; (2) instructional and mentorship skills development for attending physicians; and (3) technical assistance and systems strengthening support for RBH leadership, management, and selected administrative staff. ASPE solicited this data collection in task order request 07EASPE000114 and has contracted with RTI International (HHSP233200700004T, Contract No. 233-02-0090) to evaluate the impact of these two components of the AHI. RTI has subcontracted with D3 Systems and their subsidiary the Afghanistan Centre for Socio-Economic and Opinion Research (D3 Systems/ACSOR) to assist in the evaluation.
Given the U.S. government’s commitment to improving the physical and social well-being of women and children in Afghanistan, and the AHI’s prominent role in this effort, HHS is eager to assess whether the training and technical assistance provided to RBH managers and clinicians has improved the clinical management of and health outcomes for women and infants under their care. This evaluation also will fulfill the need identified by the AHI Program Assessment Rating Tool (PART) review for an “independent evaluation of sufficient scope and quality” to determine whether these two components of the AHI have been effective. This collection of data is authorized by Section 301 of the U.S. Public Health Service Act (42 U.S. Code [USC] 241). The initiative for which the data collection is proposed is authorized by the Afghanistan Freedom Support Act of 2002 [Public Law 107–327 §103(a)]. A copy of this legislation can be found in Appendix A.
The purpose of the evaluation is two-fold:
to assess the impact of in-service and refresher training in obstetrics and gynecology for attending physicians, residents, and midwives on the quality of newborn and maternal healthcare services; and assess the strengthening of hospital training management systems for training of trainers; and
assess the effects of management technical assistance and systems strengthening on the capacity of RBH’s leadership, managerial and selected administrative staff to directly affect providers' capacity to deliver quality clinical services.
The 20 evaluation questions to be addressed in this evaluation are presented in Appendix B.
The information from this data collection will provide information on the effectiveness and impact of the components of the AHI implemented by IMC (mainly clinical provider training) and CURE (mainly management technical assistance and mentoring). HHS can use this information for future planning and funding decisions as well as mid-course adjustments to enhance these AHI components. This information can also be used by HHS when designing clinical training and technical assistance interventions in settings similar to RBH and Afghanistan.
The RTI evaluation team will make use of information technology that is appropriate for the type of data being gathered, given the setting and the comfort level of participants. During a visit to RBH in December 2007, members of the RTI team observed that RBH staff appeared more comfortable if interviewers and observers recorded their data and other notes on paper rather than directly into a laptop or handheld computer. Therefore, all data that are collected will be recorded on paper and subsequently entered into computerized systems for tracking, analysis, and reporting. Additionally, the qualitative interviews will be audio recorded to ensure the accuracy of the interview notes.
This information collection does not duplicate similar information. This evaluation will be the first independent and comprehensive assessment of the impact of training and technical assistance activities, implemented by IMC and CURE, on the quality of care provided at RBH. Other AHI-related assessments include several needs assessments carried out at RBH between 2002 and 2006, and an evaluation of the effectiveness of the Afghan Family Health Book in 2005.
RTI has obtained copies of relevant assessment reports and performance data collected by IMC and CURE for purposes of monitoring and reporting on the progress of their training and technical assistance activities carried out under their cooperative agreements, including the results of Standards Based Management (SBM) assessments carried out in 2006 and 2007. These documents will serve as sources of background information for the proposed evaluation.
No small businesses will be involved in this study.
This is a one-time collection of information. If this evaluation were not conducted, it would be difficult to determine the impact of the training and technical assistance provided to RBH under the AHI, and to understand what factors related to the setting either helped or hindered project implementation or the impact of training and technical assistance on maternal and infant health. Finally, failure to collect these data will reduce HHS’s ability to implement the AHI’s program improvement plan under PART. There are no legal obstacles to reduce the burden.
There are no special circumstances that require data collection to be conducted in a manner inconsistent with 5 CRF 1320.5 (d)(2).
A 60-day Federal Register Notice was published in the Federal Register on April 7, 2008, Vol. 73, No. 67; pp. 18800 (see Appendix C).
No public comments were received in response to the 60-day Notice.
There has been extensive consultation with a range of federal and other stakeholders regarding the information collection requirements of the AHI evaluation, including the availability of data, data items to be collected, cultural and methodological considerations of data collection, and confidentiality and other issues related to the protection of human subjects. Exhibit 1 lists the consultations with HHS staff (ASPE, OGHA, and CDC), AHI-implementing agencies (IMC and CURE), RBH staff, and RTI’s in-country subcontractor (D3 Systems/ACSOR) hired to assist with local data collection.
Exhibit 1–Consultations Regarding Information Collection Requirements for the AHI Evaluation
YEAR |
NAME/TITLE/AFFILIATION |
PHONE/EMAIL |
HHS/Centers for Disease Control and Prevention |
||
2007-08 |
Kitty McFarlane, Senior Program Manager of the WHO Collaborative on Reproductive Health |
770-488-6256/ [email protected] |
2007-08 |
Tim Miner, Deputy Director of the WHO Collaborative on Reproductive Health |
770-488-6256/[email protected] |
2007-08 |
Dr. Brian McCarthy, Principal Investigator of the WHO Collaborative on Reproductive Health |
770-488-5229/[email protected] |
CURE International |
||
2007 |
Dennis Randles, Executive Consultant |
93 (0) 798 408 363/[email protected] |
2007 |
Eric Sinclair, Associate Executive Consultant |
93 (0) 797 104 120/[email protected] |
2007 |
Rob Werner, Associate Executive Consultant |
93 (0) 799 760 161/[email protected] |
2007 |
Dr. Fahad Paiman, Assistant Executive |
93 (0) 00 210 603/ [email protected] |
2007 |
Abdullah Sarabi, Logistics Consultant |
93 (0) 799 619 683/[email protected] |
International Medical Corp (IMC) |
||
2007-08 |
Malika Mirkhanova, Desk Officer, Central/Southeast Asia |
202-828-5155/m[email protected] |
2007-08 |
Tracey Shissler, Senior Desk Officer |
202-828-5155/[email protected] |
2007 |
Dr Fatima Noor Zada, Project Officer |
93(0) 70 283 464/[email protected] |
2007-08 |
Ms. Rose Mnzava, Program Manager |
93 (0) 799 460 710/[email protected] |
2007-08 |
Ms. Antonina Odima, Nurse Midwife Trainer |
93 (0) 799 460 710/[email protected] |
2007 |
Dr. Saboor, Training Coordinator |
93 (0) 799 460 710/Contact Rose Mnzava |
2007 |
Dr Sadiqia Deputy Program Manager |
93 (0) 799 460 710/Contact Rose Mnzava |
2007 |
Dr Shaima Sadiqi, QA Officer (Medical Records) |
Phone : 0799-2300-68/[email protected] |
Afghanistan Ministry of Public Health |
||
2007 |
Dr. Mir Lais Mustafa, Director and Head of Public Health and Research Department, Secretary of Ethical Review Board |
93 (0) 799 323 867/[email protected] |
2007 |
Dr. Muhebullah Nejaat, Director of Curative Medicine and Central Hospitals |
93 (0) 799 321 007/[email protected] |
Rabia Balkhi Hospital |
|
|
2007 |
Dr Najia Tareq, Executive Director |
93 (0) 70 26 36 72/[email protected] |
2007 |
Dr. Rosa Taqdeer, QA Officer (Perinatal Health) |
0799-419-666/[email protected] |
2007 |
Dr Shiamsa Anwari, QA Coordinator |
0799-230-068/Contact Dr. Tareq |
2007 |
Dr. Frozan, Nursing Director |
Contact Dr. Tareq |
2007 |
Dr. Sara Anwary, OB/GYN Chief |
Contact Dr. Tareq |
RTI International |
|
|
2007-08 |
Niamh Darcy, Project Director |
919-485-2610/[email protected] |
2007-08 |
Dr. Catherine Elkins, Performance Management Team Leader |
919-541-8898/[email protected] |
2007-08 |
Gordon Cressman, Data Systems Team Leader |
919-541-6363/[email protected] |
2007-08 |
Dr. Christina Fowler, Data Systems Resource Analyst |
919-541-6621/[email protected] |
2007-08 |
Sandra Canfield, Data Systems Resource Analyst |
919-316-3318/[email protected] |
2007-08 |
Dr. James Lea, Consultant |
919-962-6801/[email protected] |
There will be no payments or gifts to respondents.
Data will be treated in a private manner. Each respondent will be assigned an identification number (ID), which will be the only identifier recorded on the data collection instruments (e.g., key informant interview transcripts, SBM modules). All mapping lists linking names to IDs will be de-linked before program evaluation data analysis is started. Although the individual will be asked to report his/her position at RBH, RTI will use this information in conjunction with the RTI-assigned identification number solely to categorize and summarize the data by type of respondent group for analysis. Specific information linking the respondent’s position or job title to a particular response will not be included in any information viewed by OGHA, ASPE, or any other HHS officials. All data collection instruments, informed consent materials, and evaluation protocols will be reviewed by the RTI Institutional Review Board (IRB) and the Afghanistan Public Health Institute’s (APHI) IRB to ensure that the rights of evaluation participants are protected and that respondents are aware of disclosure possibilities. Furthermore, all RTI and D3 Systems/ACSOR project staff will sign an agreement before data collection begins stating that their adherence to all study procedures and protocols for treating data in a confidential manner. This signed agreement is a condition of their employment on this project.
The data collection instruments will not include any questions of a personal or sensitive nature. The respondents will be asked questions that focus on issues related to the training or technical assistance received under the HHS cooperative agreements with CURE and IMC, as well as factors related to the implementation or impact of these activities.
The estimated annualized hour burden of responding to this information collection is 298 hours or an average of 1.60 hours per respondent. The form-specific, average hour burden presented in Exhibit 2 is an estimate. The hour-burden estimates include the time for administering informed consent, answering any questions the respondent may have, and gathering the data. The respondents do not need to be screened prior to data collection. A formal pretest will be conducted for each form with a sample not to exceed nine from any specific group of respondents. The average hour response burden for each form will be updated on the basis of the pilot test results. The average hour response burden for the qualitative instrument has been revised based on initial testing.
Exhibit 2–Estimated Burden Hours
Type of Respondent |
Form Name |
Number of Respondents |
Number of Responses per Respondent |
Average Annualized Burden per Response (Hours) |
Annualized Total Burden (Hours) |
Management Staff |
Management Interview Guide (see Appendix D) |
21 |
1 |
50/60 |
18 |
Attending Physicians |
Clinician Interview Guide (see Appendix E) |
8 |
1 |
50/60 |
7 |
1st–4th Year Residents |
Clinician Interview Guide (see Appendix E) |
11 |
1 |
50/60 |
9 |
Midwives |
Clinician Interview Guide (see Appendix E) |
15 |
1 |
50/60 |
13 |
1st Year Resident Physician (Y1R) |
Y1R–SBMA |
31 |
1 |
1.6 |
50 |
2nd Year Resident Physician (Y2R) |
Y2R–SBMA |
8 |
1 |
1.6 |
13 |
3rd Year Resident Physician (Y3R) |
Y3R–SBMA |
9 |
1 |
1.1 |
10 |
4th Year Resident Physician (Y4R) |
Y4R–SBMA |
8 |
1 |
1.6 |
13 |
Midwife (MW) |
MW–SBMA |
75 |
1 |
2.2 |
165 |
TOTAL |
|
186 |
|
Annualized |
298 |
Notes: SBMA= Standards Based Management Assessment
The total estimated annualized cost to respond to this information collection is $372.70, as presented in Exhibit 3. The hourly wage rate is based on actual average prevailing wage rates at RBH, as of February 2008, for each type of respondent. The rates were converted into U.S. dollars using the exchange rate of 1.00 Afghanistan Afghani per US$0.020329.
Exhibit 3–Estimated Annualized Cost to Respondents for Information Collection
Type of Respondent |
Annualized |
Hourly Wage Rate (US$) |
Annualized Total Respondent Cost (US$) |
Attending Physicians |
7 |
$ 2.06 |
$ 14.40 |
Resident Physicians |
95 |
$ 1.46 |
$ 138.70 |
Midwives |
178 |
$ 1.10 |
$ 195.80 |
Hospital Management |
18 |
$ 1.32 |
$ 23.80 |
TOTAL |
298 |
|
$372.70 |
Data collection for this evaluation will not result in any additional capital, start-up, maintenance, or purchased services costs to either the respondents or record keepers. Therefore, there is no burden to respondents other than the burden described in Section A.12.
All costs for conducting the AHI evaluation are included in the contract between ASPE and RTI under task order HHSP233200700004T, Contract No. 233-02-0090. The total estimated cost is $1,049,820 over 23 months. This represents an annualized cost of $547,732. The evaluation costs include 1% evaluation funds.
This is a new information collection.
The data collected for this evaluation will be analyzed and interpreted to produce a final evaluation report, which will be submitted to HHS. RTI will deliver the final report to ASPE in hardcopy and in an electronic format (PDF). Publication of the findings on the internet or elsewhere is at the discretion of HHS. Information will be gathered over a three-month period following OMB approval. Exhibit 4 provides a schedule of data collection, analysis, and reporting following OMB approval.
Exhibit 4–Timetable for Data Collection, Analysis, and Publication
Activity |
Expected Date of Completion |
Data Collection |
1-3 months following OMB approval |
Data Analysis |
3-5 months following OMB approval |
Outline for summary report |
4 months following OMB approval |
Final Report |
10-11 months following OMB approval |
Final Briefing |
11 months following OMB approval |
To address the 20 evaluation questions presented in Appendix B, the RTI evaluation team will use two approaches – qualitative interviews and modified Standards-Based Management Assessments (SBMAs) – to gather data from RBH staff. The RBH staff include attending physicians, residents, midwives, and management staff who received clinical training or management technical assistance provided by IMC or CURE under the AHI.
Qualitative Interviews. The RTI team will conduct 1-hour, qualitative interviews with all four respondent groups to understand their perspectives on the training or technical assistance they received, including the extent to which it has enhanced their effectiveness and ways that it could be improved. The information gathered from managers and clinicians will be used to help the RTI team understand the context and conditions at RBH prior to the CURE and IMC interventions, thereby helping to create a retrospective baseline for this evaluation. Additionally, these qualitative data will provide valuable insights into the contextual and other factors that may facilitate or hinder the success of the training or technical assistance, and complement the quantitative findings for clinicians from the SBMAs described below. Finally, the interview data will provide information needed to assess the relationship between CURE and IMC interventions and important outcomes of interest, including provider competence, management systems that support delivery of high quality care, and aggregate patient outcomes. The qualitative interview topic guides are presented in Appendix D (RBH management staff) and Appendix E (RBH clinicians [attending physicians, residents, and midwives]). As shown in Exhibit 5, the topic guides share a common set of questions that will be asked of all interview participants, as well as questions specific to each respondent group.
Standards Based Management Assessment. Additionally, the RTI team will administer a modified version (i.e., subset) of the Standards Based Management Assessment (SBMA) to evaluate clinical knowledge, skills, and decision competency among residents and midwives. The SBMA, developed originally by JHPIEGO,1 was adapted by IMC and Tulane University to assess clinician competency based on the OB/GYN residency curriculum approved (October 2006) by the Afghanistan Ministry of Public Health and the midwifery curriculum approved (October 2006) by RBH. IMC has administered the SBMA three times since September 2006 to measure the effects of IMC supported clinical training on clinician competency. The SBMA consists of three components: clinical observation, a self-administered multiple-choice exam, and interviewer-administered case studies. The RTI team will use a subset of the IMC SBMA due to resource constraints and concerns about respondent burden.
Exhibit 5–Topics Addressed in Qualitative Interviews, by Respondent Group
Topics |
Attending |
Resident |
Midwives |
Managers |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Using the SBMA for the evaluation allows a uniform, streamlined, and step-by-step process for measuring the multiple dimensions of clinician performance and quality of care. Exhibit 6 presents the subjects addressed in each SBMA according to clinician group.
Exhibit 6–Topics Addressed in SBMA Instruments, by Respondent Group
Topic |
1st Year Resident |
2nd Year Resident |
3rd Year Resident |
4th Year Resident |
Midwives |
critical thinking |
|
|
|
|
|
general knowledge |
|
|
|
|
|
ambulatory care |
|
|
|
|
|
neonatal care |
|
|
|
|
|
Adult resuscitation |
|
|
|
|
|
Newborn resuscitation |
|
|
|
|
|
reproductive endocrinology |
|
|
|
|
|
gynecology |
|
|
|
|
|
obstetrics |
|
|
|
|
|
oncology |
|
|
|
|
|
midwifery management |
|
|
|
|
|
components of basic care |
|
|
|
|
|
components of neonatal care |
|
|
|
|
|
antepartum care |
|
|
|
|
|
intrapartum care |
|
|
|
|
|
postpartum care |
|
|
|
|
|
newborn resuscitation |
|
|
|
|
|
health promotion |
|
|
|
|
|
This section details the tabulations and statistical analyses that will be conducted for the AHI evaluation. Data analysis will focus on identifying results that address the 20 evaluation questions, listed in Appendix B.
For the qualitative interviews, we plan to use NVivo qualitative analysis software to code the responses to the interviews using the themes or issues addressed in the 20 evaluation questions as the basis for the coding “tree.” We will then use NVivo to generate theme- or topic-specific reports (“node reports”), and to analyze them within and across respondent groups. We will implement accepted procedures to ensure a high level of intercoder reliability.2 First, the two coders will independently code the same transcript, assigning one or more codes to passages of text. They and two other investigators will then met to compare and discuss differences in their coding and problems with code definitions and coding structure. After making revisions, they will each code a second transcript and repeat this process. Upon reaching an acceptable level of intercoder reliability, they will code the remaining transcripts.
For the SBMAs, RTI will calculate overall scores for each SBMA, as well as section and subsection scores. We will assess the differences in overall, section, and subsection scores across clinician groups, and compare overall, section, and subsection scores for each group with scores on previous administrations of the SBMA. Both descriptive and inferential statistics, such as the standard t-test, chi-square test, and multiple comparison procedures will be utilized in the analysis. Standard errors will be provided for these estimates. Non-parametric statistical techniques may also be used to analyze the data, including the chi-square test for cross tabulations. The RTI SBMAs will not replace SBM assessments that may be conducted by IMC or RBH. Because RTI will use a subset of the SBMA, some reanalysis of prior assessment results may be required to make the SBMA results comparable across time.
The 3-year expiration date for OMB approval will be displayed on all data collection forms.
There are no exceptions to the certification statement.
RTI will conduct 55 qualitative interviews with a sample of 34 clinicians and all 21 RBH managers. (includes RBH management team and senior administrative staff). Exhibit 7 presents the sample statistics for each group. With assistance from RBH management, RTI will obtain complete lists of the names of clinicians who received training from IMC, according to type of clinician. These lists will serve as sampling frames for the three clinician groups who will participate in the qualitative interviews. These frames will be used to select a 20% sample from each group. We do not propose to stratify within clinician groups because the variables needed to do this are either unavailable to the evaluation team or unreliable, as determined by a two person team from RTI that traveled to Kabul in December 2007.3 Breaking the clinician interviews into these respondent groups is appropriate because of differences in the level of training each group received from IMC, and the markedly different roles they play in providing patient care and clinical training at RBH. These interviews will occur over a period of four (4) weeks, with prior agreement by the RBH Executive Director to maximize staff availability for scheduling and administering the qualitative interviews. In the non-census sampling frames, if one staff member is unavailable the next clinician on the list will be selected for interview. Both of these approaches will help to maximize response rate.
RTI will conduct qualitative interviews with a census of RBH management staff because we expect this respondent group to have substantial heterogeneity. The source of this heterogeneity is due to the managers coming from diverse occupations and functional areas. Each manager heads a different functional area, such as administration, blood bank, anesthesiology, lab, and pharmacy.
Exhibit 7–Qualitative Interview Sample Statistics
Type of Respondent |
Population |
Sample |
Expected Response Rate |
Total expected Respondents |
Midwives – Frame A |
75 |
15 |
95% |
15 |
Residents– Frame B |
56 |
11 |
95% |
11 |
Attending Physicians – Frame C |
16 |
8 |
95% |
8 |
Management Staff |
21 |
21 |
95% |
21 |
Except as noted above, gathering qualitative information does not require an entire census. Experience of qualitative researchers shows that at some number of qualitative interviews saturation occurs.4 Saturation is the state where little additional information is yielded by additional interviews. The number of interviews required to reach saturation depends on various factors such as the homogeneity of the respondents and the research questions of interest. Saturation occurs faster in more homogeneous groups. On factors of interest to this evaluation, we expect that individuals within a group are more alike than individuals across groups. As noted earlier, however, data are lacking to stratify within groups. We expect that saturation will be reached with the proposed sample for each clinician group, and note that the final sample size for each group could be lower than presented in Exhibit 7 if saturation is achieved prior to reaching the planned sample size.
RTI will administer the modified SBMAs to a census of clinicians who received training from IMC (see Exhibit 8). The decision to take a census versus selecting a sample is based on prior use of SBMAs by IMC and the needs of the evaluation. In prior IMC SBMAs, IMC achieved a response rate of 94%. RTI expects to compare the findings from this round of SBMAs with those from previous rounds. Prior rounds of the SBMA included a census of clinical staff. A census ensures that the findings are fully representative of the respondent groups. Furthermore, the overall number of clinicians in each respondent group (OB/GYN residents and midwives) would be too small to select a sample for each respondent group that would give sufficient power to detect a reasonable amount of change over time. A census of individuals avoids this problem, because results from the different rounds of data collection can be compared directly, without concerns about power or statistical tests.
Exhibit 8–SBMAs Sample
Type of Respondent |
Population |
Expected Response Rate |
Midwives |
75 |
95% |
1st Year Residents |
31 |
95% |
2nd Year Residents |
8 |
95% |
3rd Year Residents |
9 |
95% |
4th Year Residents |
8 |
95% |
Qualitative Interviews. Regarding selection of clinicians to participate in the qualitative interviews, RTI will select systematic samples from sample frames A (midwives), B (residents), and C (attending physicians) (see Exhibit 7). As noted under Section B.1, the sample frames will consist of an alphabetical listing of the names of clinicians who received IMC training that are provided to RTI by RBH management. If a selected participant declines participation for the qualitative interview, we will select a replacement respondent from the complete sample. If the person is unavailable, then the immediate next person on the list will be chosen to replace the selected participant. Each participant will be interviewed only once. The RTI team (RTI and D3 Systems/ACSOR staff) will conduct the qualitative interviews, with RTI staff obtaining informed consent and leading the interviews, and D3 Systems/ACSOR staff providing translation and transcription services. The team will record the interview to assist with note keeping and translation. The interview guide for clinicians consists of a set of common questions that will be asked of all clinicians interviewed, as well as a set of questions that are specific to each group. To minimize data collection and data entry errors, RTI will train members of the evaluation team on the use of these interview instruments, as well as prepare a Manual of Operations, which delineates the procedures for collecting and managing data collected for the evaluation. RTI will not attempt to contact interview participants after the interview is complete. Any clarification needed will be asked by the RTI staff leading the interview during the interview and before the interview concludes. No personal identifying information will be recorded on the interview notes.
All data from the completed qualitative interviews will be entered into Microsoft WORD and coded using NVivo. RTI staff will enter the handwritten notes into Microsoft WORD for each interview. These are reviewed for consistency with the handwritten notes, and once these are determined to be accurate, they are then converted to RTF and coded in NVivo.
Standards-Based Management Assessments. For the SBMA written exams and case studies, RTI, an ex-patriate OB/GYN physician, two local Kabul based OB/GYN physicians, and staff from D3 Systems/ACSOR will administer these components. RTI and the ex-pat OB/GYN will take a lead role in this data collection activity. All information collected, from the written exam responses, the observation notes, and the case study responses will be keyed into a D3 Systems/ACSOR electronic data collection system in Kabul. Prior to data entry, the de-identified data from the exams and case studies will be translated into English, with each translation verified by RTI and D3 Systems/ACSOR staff. Ten percent of the entries will be randomly selected for keying a second time (double entry). The accuracy of the data entry process will be verified by comparing the data from the first entry with the data from the second entry. The double keying verification process will allow researchers to report the rate of accuracy to the Project Director. To minimize data collection and data entry errors, RTI will train members of the evaluation team on the use of the SBM assessment, as well as prepare a Manual of Operations, which delineates the procedures for collecting and managing data collected for the evaluation. The data collected will be entered on a weekly basis, and is anticipated to be completed within two weeks of the close of data collection.
RTI and D3 Systems/ACSOR records retention policy does not conflict with the HHS OS records disposition schedule (Appendix A, Chapter 300).
RTI expects to achieve at least a 95% response rate for interviews and SBMAs for several reasons, including high levels of interest and support for these projects from the RBH Executive Director and other RBH staff, close coordination with RBH management to plan and schedule data collection to maximize response, in-person data collection, and use of instruments that have been piloted. These two projects have provided important financial and technical support to RBH. Since their implementation in 2003, OGHA, CURE and IMC have built a strong working relationship with the RBH Executive Director and staff.
The RTI team will work closely with RBH management and the RBH Executive Director to inform relevant RBH staff about the program evaluation and to stress the importance of their participation. These discussions will occur either at separate meetings, or as a part of the daily clinical staff meeting. The RTI team will also work with RBH management to schedule data collection activities at times that are sanctioned by the RBH Executive Director, are convenient for the participants, do not disrupt patient care, and do not occur on local holidays. Finally, to ensure that their content and structure is user-friendly, all data collection instruments will be pilot tested (see Section B.4), and revised based on feedback from pilot subjects. The self-administered SBMA written exams are tailored to the respondent group and have no skip patterns. Copies of the interview topic guides, self-administered written exams, and interviewer-administered case studies are presented in Appendix D through Appendix J.
During the initial OMB review period and after RTI has received approval from Institutional Review Boards at RTI and the Afghanistan Public Health Institute, RTI will pilot test the management qualitative interview guide with six individuals (in a mixture of English and Dari) and the clinician qualitative interview guide with six individuals. Pilot participants are from the D3 Systems/ACSOR staff and the two local OB/GYN physicians participating in the evaluation. This will be an opportunistic sample based on the timing of the pilot test and availability of these staff. RTI does not plan to pilot the SBMAs because an expanded version has been used at RBH to measure clinicians’ knowledge, skills, and decision competency three times since September 2006.
The following individuals contributed to the questionnaire and study design and will be involved in the interpretation and analysis of findings.
Dr. James Lea
|
Phone: 919-962-6801 Email: [email protected]
|
Dr. Catherine Elkins
|
Phone: 919-541-8898 Email: [email protected]
|
Niamh Darcy
|
Phone: 919-485-2610 Email: [email protected]
|
Dr. Rod Knight
|
Phone: 919-918-7678 Email: [email protected] |
Karl Feld
|
Phone: 919-601-7060 Email: [email protected] |
Matt Warshaw
|
Phone: 0799-328-714 Email: [email protected] |
1 JHPIEGO is an international nonprofit agency affiliated with Johns Hopkins University. See http://www.jhpiego.org/resources/pubs/infosheets/JHPinfo_SBMR.pdf) for a description of the SBM.
2 Carey, J., Morgan, M., and Oxtoby, M. (1996) Intercoder agreement in analysis of responses to open-ended interview questions: Examples from tuberculosis research. Cultural Anthropology Methods, 8,(3):1–5.
3 The RTI team found that (1) information on the performance, placement, and rotation of OB/GYN residents was maintained at the Ministry of Public Health (MoPH) and not at RBH; (2) information on the years of practice, tenure at RBH, and areas of expertise for RBH midwives is incomplete or unavailable due to weak record keeping practices; and (3) criteria and processes for selecting attending physicians to participate in the IMC-sponsored or for residents to rotate through RBH are unclear.
4 See Miles, M.B. and Huberman, A.M. (1994). Qualitative Data Analysis, 2nd Ed., Newbury Park, CA: Sage, pp. 10-12; Sandelowski, M. (1995). Sample size in qualitative research. Research in Nursing and Health. (18):179-183; Guest, G., Bunce, A. and Johnson, L. (2006). How many interviews are enough?, Field Methods 18(1): 59–82.
| File Type | application/msword |
| File Title | Supporting Statement |
| Author | cfowler |
| Last Modified By | Adelle.Simmons |
| File Modified | 2008-07-09 |
| File Created | 2008-07-09 |