Medical Monitoring Project
0920-0740
Supporting Statement
Part B
Project Officer:
Dr. A.D. McNaghten
Team Leader, Clinical Outcomes Team
National Center for HIV, STD, and TB Prevention
Coordinating Center for Infectious Diseases
Centers for Disease Control and Prevention
1600 Clifton Rd, NE, MS E-46
Atlanta, Georgia 30333
Phone: (404) 639-6325
Fax: (404) 639-8640
E-mail: [email protected]
B. Collections of Information Employing Statistical Methods
Respondent Universe and Sampling Method
The respondent universe is HIV-infected adults receiving medical care during the population definition period (January 1 – April 30) at sampled HIV care providers in the 26 participating project areas in the 20 sampled states. The Medical Monitoring Project (MMP) uses a three-stage sampling approach designed in collaboration with statisticians from the RAND Corporation. The first stage of sampling resulted in the selection of 20 of 52 eligible geographic primary sampling units (PSUs, defined as 50 states, Washington, DC, and Puerto Rico) using probability proportional to size (PPS) sampling methods. The six cities separately funded for HIV/AIDS surveillance were included in the 20 selected PSUs and were thus also funded as project sites, resulting in a total of 26 project areas. Sampling methods ensured representation of all regions of the US. In the second stage, providers of HIV care (i.e., providers that prescribe antiretroviral therapy [ART] or order CD4 or HIV viral load tests) are sampled. The sampling frame of providers is developed every two years in each participating state using data from local HIV/AIDS case surveillance, laboratory reporting, AIDS Drug Assistance Programs and other available data sources. Providers will be sampled PPS based on their patient caseload. In the third stage, local HIV/AIDS surveillance staff will work with each selected provider to develop a list of HIV-infected patients who received care from the provider at least once during the previous calendar year. From this list, a sample of patients will be chosen by systematic random sampling.
Through an informed consent process, selected patients are offered participation in an interview with the understanding that their medical records will also be reviewed. Data collected from the interview and medical record abstraction include demographics, access to health care and quality of care received, prescription of ART and other medications, adherence to ART, met and unmet needs, high-risk sexual and drug use behaviors, laboratory indicators (e.g., CD4 counts, viral loads), AIDS opportunistic illnesses (OIs), quality of life and access to prevention services. The questionnaires comply with OMB standards on race and ethnicity. Eligible patients will only be interviewed once during a project year. Health department staff will attempt to collect basic demographic data on patients who refuse to participate in the interview from the patient or provider, or from existing surveillance data using a non-response form (Attachment 4c).
Sampled states will have a minimum sample size of 400 patients. Some states will enroll more patients, because the sample size in each state or city is proportional to the size of the epidemic in that site. This sample size will allow the description of outcomes of interest – for example, the proportion of eligible patients prescribed prophylaxis for Pneumocystis jiroveci pneumonia.
These methods will result in a representative sample of patients receiving HIV care at the national and the project area level. More detail about each of these stages of sampling is provided below.
The first stage of sampling employed a random, stratified sample with probabilities proportional to a measure of size. Because our goal is to obtain a national probability sample of adults in care for HIV infection in the US, all 50 states plus the District of Columbia (DC) and Puerto Rico (PR) were considered eligible to participate. Fifty states, DC, PR, and six cities: Chicago, Houston, Los Angeles, New York City, Philadelphia, and San Francisco were eligible to receive funding. The decision was made to include these separately funded areas (cities) in their respective states for the purposes of sampling. Therefore the first stage sampling frame consisted of 52 PSUs: the 50 states plus DC and Puerto Rico.
Systematic PPS sampling was used with the measure of size being the total number of persons living with AIDS (reported to the national HIV/AIDS Reporting System [HARS]) (collected under OMB Control No. 0920-0573: Adult and Pediatric Confidential HIV/AIDS Case Reports) at the end of 2002. Based on available funding it was decided to select 20 PSUs at the first stage of sampling. Since the first stage of sampling was carried out with probabilities proportional to a measure of the number of persons living with AIDS associated with each PSU, it is estimated that this first stage sample included more than 80% of the prevalent AIDS cases in the United States. The original 26 project areas were retained for the second funding cycle. The prevalence of HIV/AIDS was again examined using HARS data at the end of 2006, and it was determined that the 20 PSUs had the same probability of selection as when the PSUs were originally selected.
At the second stage of sampling, facilities currently providing medical care for HIV-infected adults will be sampled separately within each project area. A facility is defined as any hospital, clinic, health care facility, group or private physician practice that share common medical records or a medical records system.
In each funded area a sampling frame of unique (i.e., unduplicated) facilities currently caring for HIV-infected patients during the project period will be constructed. In addition, because facilities will be sampled PPS, an estimate of the number of patients currently in care for HIV at each facility, or estimated patient load (EPL), is also needed.
A starting point for this sampling frame is facilities that have reported information on patients with HIV or AIDS to HARS. However, because the goal is to have a complete list of facilities currently caring for HIV-infected patients in each project area, the facility list from HARS will need to be supplemented with lists of facilities obtained from other data sources. These supplemental sources may include: state laboratory reporting databases, AIDS Drug Assistance Programs, Medicaid claims, and/or HIV medical association membership lists. For each data source used, an EPL for each unique facility should be determined.
Once the lists from HARS and each of the supplemental sources have been completed, they will be combined into a single facility sampling frame. The next step is to determine which EPL will be used for PPS sampling of the facilities. The determination of which of the EPLs from various sources should be used will be a subjective process. That is, health department staff, based on their knowledge of the facility and of the accuracy of the data sources will determine which data source produced the most accurate EPL, which will be the one they recommend will be used for sampling. Once the matrix of EPLs has been completed, each site should contact their CDC project officer to discuss the data sources used to construct the sampling frame and determine the reliability of the EPL from each of those sources.
Any facility which provided outpatient HIV care during the facility time period is eligible to be included in the facility sampling frame. For the purposes of MMP, HIV care is defined as conducting CD4 or HIV viral load testing or providing prescriptions for antiretroviral medications. Thus, facilities providing HIV care could include outpatient facilities such as hospital-affiliated clinics, free-standing clinics or private physician offices, and Veterans Administration facilities.
Facilities known not to provide medical care such as counseling and testing sites should be excluded from the facility sampling frame. Other facilities that should be excluded from the facility sampling frame are: emergency rooms, inpatient facilities, facilities located outside of the funded area, facilities that have closed or at which access to medical records is known to be impossible, prisons and jails, and health facilities located on military installations. Facilities that have provided HIV care to only patients under the age of 18 should also be excluded from the facility sampling frame.
We do not currently have an estimate of the proportion of state cases represented by these facilities where access is not possible, such as federal penitentiaries and military bases. This estimate would be difficult to determine without the direct cooperation from those facilities because the state the person was diagnosed and reported in may not be the state in which they are institutionalized or serving in (OMB Pass-back Agreement, Attachment 15, page 5).
Facilities will be stratified for sampling based on size (i.e., the EPL, during a four month time period) into either a large, medium, or small stratum. These three size strata will be formed based on the proportion of patients in each facility and the methodology of PPS sampling.
Before the stratification of facilities can occur, the number of facilities to be sampled within a project area (call this nfac_tot) must be decided. Based on theoretical and practical consideration, between 40 and 60 facilities will be sampled in each project area. These considerations include having an adequate number of facilities included in the project area – not too few so the community and providers do not feel it could not be representative, and not too many so the amount of travel to reach all of them proves burdensome to health department staff conducting the project activities.
Several pieces of information are used to determine into which stratum (i.e., large, medium, or small) each facility is placed. These include:
the number of facilities to be sampled (nfac_tot )
the assigned patient sample size for each project area (call this npat_tot)
the total estimated patient load for all the facilities on the facility sampling frame (total EPL)
the overall patient sampling rate (overall sampling rate = assigned patient sample size / total EPL)
We will make use of the following relationships:
the number of facilities to be sampled in each stratum adds to the total number of facilities to be sampled (nfac_tot = nfac_large + nfac_medium + nfac_small)
the number of patients to be sampled in each stratum adds to the total number of patients to be sampled (npat_tot = npat_large + npat_medium + npat_small)
Once these parameters are known they drive the definition of facility size strata and other aspects of the sampling.
We will use an example to describe the process of how facilities are placed into one of the three strata. In our example, we have the following values:
the number of facilities to be sampled (nfac_tot = 50)
the assigned patient sample size for the project area (npat_tot = 750)
the total estimated patient load for all the facilities on the facility sampling frame (total EPL = 7,500 )
the overall patient sampling rate (overall sampling rate = assigned patient sample size / total EPL = 750/7,500 = 1/10 = 0.10)
Under PPS sampling, any facility with at least (100/ nfac_tot)% of the total EPL is defined as a large facility and sampled with certainty. The number of patients to be sampled from large facilities is calculated as the total EPL for the large facilities times the overall patient sampling rate. The identification of facilities to be sampled with certainty is an iterative process.
In our example, any facility with at least 2% of the total EPL (i.e., (100/50)% = 2% ) is defined as a large facility and sampled with certainty. Another way of saying this for our example is that any facility with an EPL of 150 or larger is defined as large (i.e., 2% of 7,500 = 150) and sampled with certainty.
In this example, the overall patient sampling rate is 0.10; consequently, 10% of patients will be sampled overall. In addition, this is the rate at which patients will be sampled from facilities in the large facility stratum. Suppose in our example that there are only 3 large facilities (i.e., nfac_large = 3). Also suppose that the total EPL for the 3 large facilities is 1,500. Then 150 patients would be sampled from the large facilities (i.e., total EPL for the large facilities time the overall patient sampling rate = 1,500 x 0.10 = 150 patients).
The next step is to remove the large facilities from the sampling frame. The facilities remaining on the sampling frame will be partitioned into medium facilities and small facilities. The number of patients to be sampled from the medium and small facilities is the total patients to be sampled minus the number of patients to be sampled from the large facilities. The average cluster size for the remaining facilities is calculated as the total patients to be sampled from the medium and small facilities divided by the number of remaining facilities to be sampled. Those facilities with EPL smaller than the average cluster size are defined as small; all remaining facilities not previously identified as large are classified as medium.
In our example, there are 47 facilities remaining to be sampled (i.e., nfac_medium + nfac_small = nfac_tot - nfac_large = 50 – 3 = 47). The number of patients to be sampled from the small and medium facilities is 600 patients (i.e., npat_medium + npat_small = npat_tot – npat_large = 750 – 150 = 600). The average cluster size is 13 (i.e., (npat_medium + npat_small) / (nfac_medium + nfac_small ) = 600/47 = 12.8 ~ 13). Any facility in our example that had an EPL less than 13 would be defined as a small facility and the remaining ones not previously identified as large would be defined as medium-size facilities.
Once completed, each site will send its facility sampling frame, which must include an EPL for each facility to CDC via the Secure Data Network for sampling. The sampling frame sent to CDC should be stripped of any identifying information; facilities will be identified only by a unique numeric facility ID number that will be assigned at the project area. Facility ID numbers will be made unique across all project areas by the addition of a 4 digit numeric site code in front of the initial 4 digit facility ID number.
For each site the CDC sampling statistician, in conjunction with the CDC project officer and the site, will select a PPS sample of facilities. Each project area will determine, in consultation with CDC, the number of facilities to be sampled; in most project areas, between 40 and 60 facilities will be sampled each year. While CDC and the state or local health department will jointly review the final stratified list of facilities, ultimately the demands of the sampling design will determine the number of facilities that will be selected from each stratum.
The exact number is chosen by the RAND consultant sampling statistician, taking into account the number of large facilities, the total number of facilities, and the distribution of facilities within the different size strata. The facilities will be selected with probability proportional to size, and in most project areas, a total of 400 patients will be selected. Most states use 4 to 5 geographic strata to ensure face validity of the sample of facilities selected. We set a minimum number of 25 facilities sampled per project area, which is sufficient to select a representative sample of 400 patients from in areas with many large facilities (and therefore, large HIV patient loads). For example, Los Angeles and Houston will each have a sample of 400 patients drawn from 25 facilities. Areas with larger geographic areas and more medium and small facilities will need more facilities in their sample from which to draw patients. For example, California (excluding Los Angeles and San Francisco) has 68 facilities and Oregon has 60. These decisions are made on a project area-by-project area basis in consultation with the sampling statistician (OMB Pass-back Agreement, Attachment 15, page 5).
Once the sample of facilities is selected, the local area will contact each sampled facility to inform them that they have been selected to participate in the project, and to determine when and how a list of the HIV infected patients currently in their care will be obtained. Because the patient list is necessary for calculating sampling fractions, they must include all HIV-infected patients in care, whether or not they have been reported to HARS. Details of how medical record abstraction will be conducted and how patients will be recruited for interviews should also be discussed.
The goal is to obtain participation in MMP from all sampled facilities. The generalizability of a probability sample depends upon an adequate overall coverage or response rate. The validity of population estimates from MMP could be questioned if the overall response rate obtained is less than 75%. Therefore, an overall response rate of at least 75% should be obtained for MMP at both the local and the national level. The higher the overall response rate the more credible the population estimates obtained will be. Project areas have been marketing the project to providers and patients in their jurisdictions and support for the project is strong, which should contribute to higher response rates (OMB Pass-back Agreement, Attachment 15, page 6).
The overall response rate is the product of site, facility, and patient response rates. If 100% of project areas, 75% of facilities, and 75% of patients from each participating facility are enrolled, the overall response rate is 1.0*.75*.75=.56 or 56%. Since all 26 project areas selected in the first stage of sampling have agreed to participate, an overall 75% response rate at both the local and national level can be achieved through any of the following scenarios:
Facility response rate = .80 Patient response rate = .94
Facility response rate = .85 Patient response rate = .88
Facility response rate = .90 Patient response rate = .83
Facility response rate = .95 Patient response rate = .79
The lower the facility response rate is the higher the patient response rate will need to be to achieve the same overall response rate.
MMP staff in participating project areas rely on existing relationships with providers and the informing providers of the importance of the project to achieve an adequate sample size. The facility response rate for the 2007 data collection cycle was 88%.
It is expected that a high level of effort will be needed in order to get each sampled facility to participate in the project. Each site should have a strategy for contacting sampled providers based on their experience working with facilities on similar projects. Experience from previous surveillance projects suggests that difficult to enroll facilities might best be contacted by the medical director of the health department or HIV program. Alternatively, a local provider advisory board member might be used to recruit facilities that are reluctant to participate. Because a high facility response rate is critical to the success of MMP, each participating health department should develop a strategy for facility recruitment that will maximize this response rate.
Even if a facility is not willing to participate, the facility will remain in the sample. No substitutions will be made for facilities that cannot be persuaded to participate. A facility that refuses to participate has refused participation for all of its patients. This means that these patients and patients like them would have NO opportunity to be represented by this project. Substitution of sampled facilities or patients would invalidate the sampling design of the project. If substitutions are allowed, inference to the population of HIV infected patients in care in the US cannot be made. Facilities that were not selected and their patients may not have the same attributes as sampled facilities and their patients. Substitutions would bias the sample in a manner that cannot be predicted nor adjusted for (OMB Pass-back Agreement, Attachment 15, page 6).
Within each participating facility, patients will be sampled for inclusion in MMP with equal probability of selection. Patients will be sampled from lists of patients seen during the PDP. The 2009 PDP is the 4 month period January 1-April 30, 2009.
A list of patients who received HIV care during the PDP should be requested from all facilities selected into the sample during the second stage of sampling. Methods for constructing patient lists may vary based on the type of facility. Some suggested strategies for different types of facilities include using lists of patients seen in the specialty clinic or a list of patients with HIV-related ICD-9, ICD-10, procedures or tests (i.e., CPT), or prescription codes during the PDP. Note that HARS is only used as a way to identify facilities during the second stage of sampling. HARS is not used as a source for generating patient lists during this third stage of sampling.
The facility can give the health department a list of patient names without patient consent (facility and patient names are not sent to CDC). These patients should be in the HIV/AIDS reporting system; the health department in every area has explicit legal authority, conferred by state law, to collect information on patients with HIV within the state. In most cases, the health department will already have the names. Although this legal authority exists in every state, providers that do not want to provide a list of patient names can provide the health department with a list of coded identifiers. Methods for constructing patient lists may vary based on the type of facility. Most facilities have automated systems and can easily generate a list of patients. Providers without automated systems are generally those with small HIV caseloads. Starting at the beginning of the population definition period, these facilities can keep a log of all HIV-infected patients that receive care during the population definition period (OMB Pass-back Agreement, Attachment 15, page 7).
At each selected facility, all patients who meet the following conditions are eligible for inclusion: (1) the patient has a diagnosis of HIV infection, with or without AIDS-defining conditions; (2) the patient is at least 18 years old at the beginning of the PDP; and (3) the patient received medical care (defined as any visit to the facility or prescription of medications, including refill authorizations) at the facility during the PDP.
Other subsets of patients in care, such as those who received all their HIV-related care from emergency rooms or medical facilities on military bases, or in prisons or jails, may have been excluded in a project area when the facility sampling frame was constructed based on criteria set forth in the section on second stage sampling. Note that these exclusions are based on eliminating certain types of facilities from the facility sampling frame not from excluding all patients who receive any care at such facilities. Information on patient visits to ERs and inpatient facilities will be obtained during interviews and/or may be documented in medical records.
Once a project area has obtained patient lists, they should be stripped of identifying information and sent to the CDC using the Secure Data Network. It is not necessary to wait until all patient lists within a stratum are obtained before sending de-identified lists to CDC. Individual patients will be identified only by a 12 digit numeric participant ID number that will be assigned at the project area. This should be a unique identifier that will be associated with that patient throughout the project and which should appear on all data collection forms and in all data bases. Participant ID numbers will be formed starting as 4 digit numbers that are assigned consecutively to patients on each facility’s edited patient list. The allocation of patient sample among the facility size strata will be done in a manner that will result in an equal probability of selection method (EPSM) sample at the patient level. In general this means that an equal number of patients will be sampled from each facility within a facility size stratum. Sampling of patients will be done using SAS Proc SurveySelect to draw a simple random sample of patients within each facility. Lists of selected patients’ ID numbers will be returned to the site after sampling is completed for patients. All patients included in the sample should be pursued for enrollment in the study; the total number of sampled patients will be used in the denominator for calculating patient response rates.
Persons selected during third stage sampling may be offered enrollment through two recruitment scenarios; staff-contact enrollment, or provider-referred enrollment. The recruitment strategy utilized by facilities will vary based on clinic needs and patient load. Instead of giving the health department the names of the sampled patients, some providers prefer to contact the patient first and let them know they have been selected to participate (OMB Pass-back Agreement, Attachment 15, page 8). It is anticipated that each project area may utilize a variety of recruitment scenarios.
During staff-contact enrollment, facilities will provide local MMP staff with contact details for patients being sought for recruitment. Local MMP staff will use patient contact lists to initiate phone contact with eligible persons to describe the project and offer enrollment. Standardized contact scripts developed by the project areas with CDC input will be used by sites to ensure a standardized approach is used for recruitment. Model patient recruitment scripts are included as Attachment 11. Project areas can modify these scripts to meet their specific needs. Unless the CDC model scripts are modified, additional OMB approval will not be sought for modifications made by individual project areas. The individual project area modifications will likely be minor (OMB Pass-back Agreement, Attachment 15, page 8). Patients who are eligible for enrollment and express interest in participating will be scheduled to have an interview done in a location meeting the needs for patient privacy.
Patients recruited through provider-referral enrollment will have their initial contact with the project made by staff from the provider’s office from which they were sampled. Staff from the clinic will provide patients with a brief verbal description of the project and ask permission to provide their contact information to MMP staff to complete enrollment or staff will provide the sampled patient with the MMP health department staff contact information. The same verbal description of the project used in the Model Patient Recruitment Script described above can be used on the phone or in the provider’s office. Model scripts for facility use and health department staff use are included in Attachment 11. Consent for participation or providing information to the health department is not obtained at this time.
Based on experience from previous projects, the staff contact enrollment method appears to be able to achieve higher enrollment rates. In all cases, MMP staff will coordinate with the patient’s provider in order to ensure that provider and patient privacy issues are addressed.
At high volume facilities using real-time sampling, MMP staff will approach eligible individuals attending the facility for enrollment into the project, describe the project and offer enrollment. Persons agreeing to participate then can either be administered the interview at that time or schedule an appointment for an interview in the future.
Nine MMP (OMB Pass-back Agreement, Attachment 15, page 12) project areas conducted medical record abstraction and/or interview during 2005. Sample sizes per site ranged from 100 to 500 during 2005. The remaining project areas were conducting start-up activities in 2005. Start-up activities included all project activities with the exception of participating in interviewer and abstractor trainings and data collection. In Years 3-4 (2007-2008) all 26 areas conducted both interview and medical record abstraction on sampled patients.
Because MMP is mainly descriptive, power calculations – which are used in sample size determinations for testing specific hypotheses – were not performed. Instead the level of precision – i.e., the estimated 95% confidence interval half-width – was the criteria used to determine individual project area sample sizes. Ninety-five percent (95%) confidence interval half-widths were calculated for a variety of sample sizes and design effects. It was decided that the minimum sample size that would be necessary for a state to obtain total population estimates with an acceptable level of precision (assuming a moderate design effect) was 400. This sample size was assigned to the states with the lowest AIDS prevalence. Sample sizes for states with higher AIDS prevalence were determined by considering the distribution of cases among the 20 sampled states and 6 separately funded cities contained within them and a target national sample size of approximately 10,000. This sample size will allow national estimates to be obtained with an acceptable level of precision (assuming a moderate design effect) for subpopulations that comprise as little as 5% of the total population of interest. Attachment 13 outlines the target sample size and associated activities for the project areas during 2007 and in subsequent years.
It is expected that this number of paired interviews/chart abstractions will be obtained while maintaining an interview response rate needed to achieve an overall response rate of at least 75% (see Second Stage Sampling).
The targeted national population of inference for MMP Provider Survey is health care providers practicing HIV medicine in the United States during 2007, including physicians, physician assistants and nurse practitioners. Interns, residents and fellows who are in training programs are not included.
As previously described MMP uses a stratified three stage sampling design. In the second stage of MMP sampling, a representative sample of all facilities in a project area is selected using PPS sampling. Using the Population Definition Period Patient Load for each MMP selected facility (i.e., the actual count of HIV-infected patients seen at a selected facility during January 1 through April 30, 2007 derived from a facility’s patient list or lists) a sub sample of HIV care providers from all participating facilities will be selected using PPS sampling.
At each sampled facility, all selected providers who meet the following conditions are eligible for inclusion: (1) have provided care to HIV-infected patients age ≥ 18 years old during 2007; (2) have completed their respective residency and, if applicable, fellowship training programs (that is not an intern, resident, or fellow); (3) are a physician or physician assistant or nurse practitioner.
The MMP Provider Survey will be administered to a national probability sample of HIV health care providers derived from the national probability sample of facilities providing HIV care obtained through the MMP facility sampling activities. Since the MMP Provider Survey will be administered in all 26 project areas in 2009, it is estimated that approximately 2,550 providers (based on 850 facilities, each with an average of 3 providers), will be eligible to be selected to participate in the MMP Provider Survey. Ultimately, because of logistical and funding considerations, a sub sample of approximately 1,920 providers (about 75% of all eligible providers) will be selected to participate in the MMP Provider Survey.
The overall response rate is the product of site, facility, and patient response rates. If 100% of project areas, 75% of facilities, and 75% of providers from each participating facility participate, the overall response rate is 1.0*.75*.75=.56 or 56%.
Because the MMP Provider Survey is mainly descriptive, power calculations – which are used in sample size determinations for testing specific hypotheses – were not performed. Instead, the level of precision that may be expected from the available sample was determined. The following table indicates the expected level of precision – i.e., the estimated 95% confidence interval (CI) half-width – for a survey of providers.
|
|
|
|
|
|
|
CI half-width |
CI half-width |
CI half-width |
CI half-width |
CI half-width |
n |
design effect = 1 |
design effect = 2 |
design effect = 3 |
design effect = 4 |
design effect = 5 |
500 |
4.38% |
6.20% |
7.59% |
8.77% |
9.80% |
750 |
3.58% |
5.06% |
6.20% |
7.16% |
8.00% |
1000 |
3.10% |
4.38% |
5.37% |
6.20% |
6.93% |
1500 |
2.53% |
3.58% |
4.38% |
5.06% |
5.66% |
2000 |
2.19% |
3.10% |
3.80% |
4.38% |
4.90% |
2500 |
1.96% |
2.77% |
3.39% |
3.92% |
4.38% |
3000 |
1.79% |
2.53% |
3.10% |
3.58% |
4.00% |
4000 |
1.55% |
2.19% |
2.68% |
3.10% |
3.46% |
5000 |
1.39% |
1.96% |
2.40% |
2.77% |
3.10% |
The required precision will depend on the purpose for which an analysis is done. CDC, in consultation with the states, have determined that the expected precision (which won’t even be known until after the data collection is complete) will result in estimates and confidence intervals (CIs) that are useful for local planning and policy purposes. For some comparisons, data will need to be combined at the national level to have acceptable precision. In addition, the design effect will be different for different outcomes, and also depends on the within-provider correlation. We will not know a priori what level of precision we will have until the first data are collected and analyzed.
The level of precision of these estimates will depend on the number of patients from whom data is obtained and also on the design effect. Design effect refers to the variance inflation that is introduced by using a multi-stage complex sampling design to obtain our patient samples.
Design effect is the variance obtained using the complex sampling design divided by the variance that would have been obtained from a simple random sample of the same size. A design effect of 2 means that the variance obtained using a complex sampling design was twice as large as the variance that would have been obtained from a simple random sample of the same size.
Because CIs are calculated using the standard error, which is the square-root of the variance, a design effect of 2 means that CIs are 1.41 times as wide as those that would have been obtained using a simple random sample of the same size. Similarly, 95% CI half-widths for a design effect of 4 will be 1.41 times as wide as those for a design effect of 2 given the same sample size and sampling design.
Less precision means that a wider 95% CI is obtained; more precision means that a narrower 95% CI is obtained. Please see the table and examples below.
95% Confidence Interval Half-widths
for various sample sizes and design effects*
Design effect = 2 |
|||||
|
CI half-width |
CI half-width |
CI half-width |
CI half-width |
CI half-width |
n |
total population |
subpopn = 50% |
subpopn = 25% |
subpopn = 15% |
subpopn = 10% |
100 |
13.86% |
19.60% |
27.72% |
35.79% |
43.83% |
200 |
9.80% |
13.86% |
19.60% |
25.31% |
30.99% |
400 |
6.93% |
9.80% |
13.86% |
17.90% |
21.91% |
500 |
6.20% |
8.77% |
12.40% |
16.01% |
19.60% |
800 |
4.90% |
6.93% |
9.80% |
12.65% |
15.50% |
1000 |
4.38% |
6.20% |
8.77% |
11.32% |
13.86% |
1200 |
4.00% |
5.66% |
8.00% |
10.33% |
12.65% |
1300 |
3.84% |
5.44% |
7.69% |
9.93% |
12.16% |
|
|
|
|
|
|
Design effect = 5 |
|||||
|
CI half-width |
CI half-width |
CI half-width |
CI half-width |
CI half-width |
n |
total population |
subpopn = 50% |
subpopn = 25% |
subpopn = 15% |
subpopn = 10% |
100 |
21.91% |
30.99% |
43.83% |
56.69% |
69.30% |
200 |
15.50% |
21.91% |
30.99% |
40.02% |
49.00% |
400 |
10.96% |
15.50% |
21.91% |
28.30% |
34.65% |
500 |
9.80% |
13.86% |
19.60% |
25.31% |
30.99% |
800 |
7.75% |
10.96% |
15.50% |
20.01% |
24.50% |
1000 |
6.93% |
9.80% |
13.86% |
17.90% |
21.91% |
1200 |
6.33% |
8.95% |
12.65% |
16.34% |
20.00% |
1300 |
6.08% |
8.60% |
12.16% |
15.70% |
19.22% |
Consider Project Area A that obtains interview and medical record abstraction data on approximately 400 patients and where African-American patients comprise approximately 15% of the patients in their MMP data. For a design effect of 2 they could expect to obtain 95% CI half-widths of approximately + 17.9% on an estimate for African-American patients. If a design effect of 5 is assumed the expected 95% CI half-width for the same subpopulation estimate would be approximately + 28.3%.
By contrast, Project area B, where African-Americans comprise approximately 50% of the patients in a set of 400 observations, would expect a narrower 95% CI half-width of approximately + 9.8% for the same subpopulation estimate for a design effect of 2. Assuming a design effect of 5 an estimate for African-American patients would have an expected 95% CI half-width of approximately + 15.5%.
Estimates that will have acceptable level of precision at both the national and local level will include the following:
The distribution of patients receiving HIV care by demographic characteristics (sex, race/ethnicity, age group, education).
The proportion of eligible persons prescribed highly active antiretroviral therapy.
The proportion of eligible persons prescribed prophylaxis for Pneumocystis jiroveci pneumonia.
The proportion of persons reporting ever using injection drugs.
The proportion of persons reporting sex without a condom in the past 12 months.
When estimates are stratified by patient characteristics or for rare events, we may not have adequate precision for estimates using data from a single year at the local level. Instead, national or multi-year analyses may have to be performed to provide adequate precision.
(OMB Pass-back Agreement, Attachment 15, pages 10-12).
2. Procedures for the Collection of Information
The MMP design is a three-stage sampling approach. The first stage of sampling resulted in the selection of 20 of 52 eligible geographic primary sampling units (PSUs, defined as 50 states, Washington, DC, and Puerto Rico) using probability proportional to size (PPS) sampling methods. The six cities separately funded for HIV/AIDS surveillance were included in the 20 selected PSUs and were thus also funded as project sites, resulting in a total of 26 project areas. In the second stage, providers of HIV care (i.e., providers that prescribe antiretroviral therapy [ART] or order CD4 or HIV viral load tests) are sampled PPS based on their patient caseload. In the third stage, a sample of patients will be chosen from selected providers using equal probability selection method sampling.
MMP data collection activities will occur during each calendar year from the approval date for 3 years. Each year a sample of facilities will be drawn. From each selected facility, a sample of providers will be selected to participate in the MMP Provider Survey. It is possible that an HIV care provider be selected to participate in the MMP Provider Survey in more than one year, as providers will have some probability of being selected each project year. Providers selected during a calendar year are only eligible to participate once during that year. Each provider sampled will only be surveyed once during the project year. Providers will be assigned a unique provider MMP identification number; therefore, only one MMP Provider Survey can be completed per provider.
Patients will be interviewed first and then their medical record will be abstracted. The time period of interest for the interview (i.e., the surveillance period) will be the 12-month period directly preceding the interview. Information from the patients’ medical records will be abstracted for this same time period.
All patient interviews (Attachment 4a) will be conducted by trained MMP staff in a private location either as part of a routine visit to a medical facility, or by an interview at home, in a hospital or clinic, or other mutually agreed upon location.
The entire interview is expected to last for approximately 45 minutes. Interviews of patients who engage in few risk behaviors or have no risk behaviors (sexual behavior, drug and alcohol use) or who take few HIV-related medications or no medications will take slightly less time. Interviews of patients who engage in many risk behaviors or are taking many HIV-related medications may take slightly longer. The interview will collect behavioral information relevant to medical care and clinical outcomes. The questionnaire (Attachment 4a) will consist of 5 required (core) modules that all sites will administer.
The standardized interview instrument (Attachment 4a) will be provided by CDC in a Handheld-Assisted Personal Interview format so that data will be collected electronically. The interview will be administered face-to-face using electronic handheld devices. The interview instrument was developed using Questionnaire Development System (QDS) software (NOVA Research Company, Bethesda, Maryland).
Health department staff will contact patients to schedule appointments to conduct the interview when possible. Difficult to locate or contact patients may be approached at their next scheduled health care visit and the interview conducted at that time or scheduled for a later date.
Participants will receive prevention materials at the end of the interview, referrals to local prevention and care services, and also prevention information from the MMP staff, as requested.
For quality assurance purposes, a 10% subset of interviews will be observed by the project coordinator to determine accuracy and completeness. Additionally, interviewers will have periodic peer review of interviews to ensure the consistency in administration techniques across interviewers.
In order to avoid data loss, and to ensure data security, at the end of each field visit the interviewers will be responsible for downloading and saving all data records into the local database. Once the downloading has occurred, all patient records should be deleted from the handheld computer’s hard drive before leaving for the next interview.
CDC will regularly train the interviewers and convene lessons learned meetings to understand the problems that can occur with the software and hardware that is used for conducting the interviews. Automated edit checks will be built into the computer software programs as a further quality control measure.
Medical record abstraction (Attachments 6a-6d) will be conducted by local project staff trained in the abstraction of clinical variables from medical charts. Standardized software on a laptop computer will be used for medical record abstraction. The information to be collected will be primarily related to diagnosis of opportunistic illnesses, provision of preventive therapies, prescription of antiretroviral medications, adverse events due to medications, and health services utilization.
The personally identifying information used to select patients will not be collected on the completed abstraction forms; however, each person will be assigned a unique ID as defined in the section Third Stage Sampling. If selected patients do not have medical records due to loss or misfiling, they will not be replaced by another patient. One record will be used for each patient visit; however, all visits that occur during the surveillance period to the selected facility need to be abstracted. A patient will have as many records as the number of visits he/she had during the surveillance period.
In addition to the facility from which the patient was sampled, data will also be abstracted from the medical records at other facilities from which the patient received care during the surveillance period. If records at the sampled facility document care received at another facility, or there is information captured by interview showing additional sources of care during the surveillance period, the project staff should abstract those records. Records are accessible from non-sampled facilities through the project areas’ HIV/AIDS surveillance authority, but will be accessed with the facilities’ permission. The additional facilities from which medical records will be abstracted will include:
Infectious disease specialists or other providers of primary HIV care
Sexually Transmitted Disease (STD) clinics
Tuberculosis (TB) clinics
OB/GYN practices or clinics (for women)
Acute care hospitals (for hospitalizations)
CDC is responsible for developing and distributing the medical record abstraction software program to the participating state and local health departments. CDC will conduct abstractor training, and also provide a manual with detailed instructions for data abstraction to participating state and local health departments.
CDC will regularly train the abstractors and convene lessons learned meetings to understand the problems that can occur with the software and hardware that are used for conducting the abstraction. Automated edit checks will be built into the computer software programs as a further quality control measure.
CDC will conduct training and site visits to provide instructions and technical assistance on how to use the CDC-provided software and hardware, conduct the interviews, archive the collected data, and transfer the data. CDC will also provide a manual with detailed instructions on interview conduct to participating state and local health departments.
Completed MMP electronic abstraction records (Attachments 6a-6d) should be visually scanned to check for completeness. A 10% subset of medical records should be re-abstracted by a second, independent reviewer and compared to the original abstraction form to determine completeness and discrepancies. The medical records selected for re-abstraction should be from a variety of facilities, abstractors, and time periods.
In addition, to enhance the quality of the data collected, standardized definitions, codes, abstraction instructions and standard training procedures for data abstractors will be provided to all participating sites. Periodic site visits by CDC will be made to all project areas and technical assistance will be available through the CDC project officers.
Similar information is being collected from both the interview and the medical record abstraction in this first full data collection year to evaluate which data elements are best collected by which data collection method. We will do analyses to test for concordance among information collected by self report and information documented in medical records for these variables. Once we have evidence that certain data elements are better collected using interview or abstraction, questions will be eliminated from the less suitable instrument.
Inconsistencies will be examined to determine the reasons for discordant findings. We expect that patients will not know the answers to many of the clinical questions (e.g., highest ever HIV viral load), and that time since the event may decrease the patients’ ability to recall (e.g., date and result of first CD4 test). We also expect that patient self report will result in better information on race/ethnicity since this information may be documented in the medical records without consulting the patient. Self-reported drug use, which may be fully disclosed to a provider in a clinical setting, may not be documented in detail in the medical record, and therefore, may be better ascertained through the interview process.
Information will also be used to help determine what data to collect in future data collection cycles. Some patients have been living with HIV for over 20 years and have seen multiple health care providers during that time. Historical data of important events (ever had an AIDS-defining opportunistic illness, ever been prescribed prophylaxis for Pneumocystis jiroveci pneumonia, types of antiretroviral medications prescribed) may not be available in patients’ medical records if they have moved often or parts of their records have been archived. It is important to determine if this information can be obtained by patient self-report, or if efforts to collect such historical information are not worthwhile.
Information that will be collected in both the interview and medical record abstraction for evaluation include the following:
Demographics (date of birth, sex, race/ethnicity, insurance status)
CD4 count (value and date of first, lowest and most recent in past 12 months)
HIV viral load (value and date of first, highest and most recent in past 12 months)
Ever prescribed antiretroviral therapy and classes of drugs ever prescribed
Ever prescribed prophylaxis for Pneumocystis jiroveci pneumonia or Mycobacterium avium complex
Receipt of influenza and hepatitis vaccinations
Diagnosis of sexually transmitted infection (syphilis, gonorrhea, herpes or human papillomavirus) in the past 12 months
Drug use (injection and non-injection) in the past 12 months
Information collected using both instruments will not always be identical. For example, in the interview patients are asked about their drug use; in the medical record physical evidence of drug use or referral to drug treatment may be documented. This may indicate drug use among participants who denied drug use when interviewed. Another example is that respondents are asked during the interview if they had unprotected sex, and documentation of sexually transmitted infections is collected in the medical records (OMB Pass-back Agreement, Attachment 15, pages 12-13).
For the MMP Provider Survey, sampled providers will be able to access the MMP Provider survey at their convenience either via a Web-based application or paper survey. Time required to complete the survey is expected to be approximately 20 minutes.
Both web and paper surveys will be self-administered and will have explicit completion instructions. If the provider has technical difficulties in accessing the web-based survey, the provider can contact the CDC contractor. Contact numbers and web addresses for the CDC contractor staff associated with the MMP Provider Survey will be provided in the recruitment packet. At the end of the MMP Provider Survey, the provider will have the option to print the survey questions and their responses.
The CDC contractor will be responsible for designing and hosting the web-based survey. For providers who complete the paper survey, the CDC contractor will enter their responses into the web-based application. The CDC contractor will forward these paper surveys to CDC. The CDC contractor will help in the recruitment of providers by preparing all materials to be included in the recruitment packets and providing logistical support to project areas as needed. Additionally, the CDC contractor will archive the collected data, clean the data, and transfer the data to CDC where it will be stored in a secure, locked location. CDC will then transfer the data to the individual project areas.
In order to avoid data loss, and to ensure data security, the paper survey will be mailed to providers in non-transparent envelopes. A stamped envelope addressed to the CDC contractor will be included in the recruitment packet. After the CDC contractor has entered the paper survey responses into the web-based application, the CDC contractor will send the paper surveys to CDC.
For the Web-based application, the website will be secure, and data will be automatically saved. The providers will be assigned a unique provider MMP identification number; therefore, only one MMP Provider Survey can be completed per provider.
The CDC contractor will test the draft version of the MMP Provider Survey in both web and paper formats prior to finalizing the survey and survey distribution.
Facilities selected to participate in the 2007 MMP cycle will be contacted by the local MMP staff (or the CDC contractor staff working in collaboration with the local MMP staff) in order to obtain the names of the providers working at those facilities. Contacts will be made as part of regular MMP activities intended to obtain information regarding facility characteristics, which include assessing number of providers working at each sampled facility, or may be a separate contact. The project areas will then document the number of providers per facility and assign a unique MMP Provider Survey identification number to each provider. Project areas will forward a list with the MMP Provider Survey identification numbers to CDC where a sample of HIV care providers will be selected for each project area using PPS sampling. After the sample of providers is selected, CDC will forward a list with the selected MMP Provider Survey identification numbers to the CDC contractor. The CDC contractor will use the identification numbers to create individualized recruitment packets, and send the recruitment packets to the project areas for personalization and distribution to sampled providers. Some project areas may elect to have the CDC contractor personalize and mail the recruitment packets directly to providers.
The recruitment packets will include a CDC recruitment letter that will explain the purpose of the survey, instructions on how to complete the survey (including instructions on how to access the web-based survey via the provider’s unique identification number), and information regarding the gift card.
An additional recruitment letter from the project area may also be included along with a copy of the paper survey, a pre-stamped contractor addressed envelope to be used by the providers who elect to complete the paper survey, and a gift card to reimburse providers for their time and effort in completing the survey. All materials will be mailed to providers in a stamped plain white letter sized envelope.
The provider will complete the web or paper survey using his/her unique provider identification number. These unique provider identification numbers will be used to identify which providers have completed the survey and which providers need to be followed-up.
The Dillman method will be used to follow-up on non-responders. Dillman suggest 3 follow-up contacts in order to assure adequate response rates. One week after the mailing of the provider recruitment packets, a postcard reminder will be sent to everyone. The postcard will have standard language thanking all those who have responded and providing a friendly reminder for those who have yet to complete the survey. After personalizing the postcards, the project areas will mail them to the providers. Some project areas may elect to have the CDC contractor personalize and mail the postcards directly to providers.
Three weeks after the original mailing, the CDC contractor will send the project areas a list of provider identification numbers with a status update of providers who have and have not completed the survey. At this time a nonrespondent letter, the original CDC recruitment letter, and replacement paper survey will be sent only to nonrespondents. CDC will write the text of the nonrespondent letter and the CDC contractor will be responsible for preparing the follow-up packages and will send them to the project areas to be personalized and mailed to the providers. Some project areas may elect to have the CDC contractor personalize and mail the follow-up packages directly to providers.
Finally, seven weeks after the original provider survey mailing, a final mailing will be sent to providers. The procedures are the same as for the three week mailing.
CDC will regularly convene lessons learned conference calls with the project areas and the CDC contractor to address any issues with the software and discuss mechanisms that are being used for administering the survey. For the web-based application, automated edit checks will be built into the computer software program as a further quality control measure.
The CDC contractor will be responsible for all data management activities. If the survey data is in paper format, then the CDC contractor will be responsible for shipping these paper survey forms to CDC for locked storage. The CDC contractor will not be permitted to make copies of these completed paper-surveys. The CDC contractor will also be responsible for data entry of the paper surveys into the electronic application. The CDC contractor will then transfer all electronic survey information to CDC using CDC’s Secure Data Network. The secure transmission encrypts all data transferred from the client machine and the Secure Data Network server. Each record in the MMP Provider Survey database will be identified by the pre-assigned unique provider ID and will not contain any directly or indirectly personally identifying information. CDC will provide project area specific combined weighted data sets back to each project area at the end of the survey period.
Participating state/local health departments will have the primary responsibility for analysis and use of data at the local/state level and for developing reports based on local data. CDC will be responsible for analysis of these data in aggregate at the national level, as well as for developing reports that utilize this multi-site data. Neither CDC nor the project areas will analyze individual provider results. Prior to analysis, an analysis group, comprised of CDC, the local MMP project areas, and MMP Provider Advisory Board members, will discuss the level at which these results will be analyzed.
MMP Provider Surveys will not contain specific identifiers (e.g., name, address, social security number). Paper surveys will be destroyed three months after survey activities are completed.
The web-based software, which will be used as one form of collecting data, supports the ability to encrypt response data and password-protect surveys so that unauthorized users are unable to view, export, or modify collected data.
3. Methods to Maximize Response Rates and Deal with Patient Non response
Because the MMP interview will take approximately 45 minutes to complete, to increase response rates, patients will be offered reimbursement for their participation. Participants will be reimbursed approximately $25 in cash for participation in the interview. If local regulations prohibit cash reimbursement, equivalent reimbursement may be offered in the form of personal gifts, gift certificates, or bus or subway tokens.
Reimbursement was used in the Supplement to HIV/AIDS Surveillance (SHAS) project (OMB 0920-0262, exp. 06/30/2004) (described in A.1.), for persons who agreed to participate in the interview. Participants were offered $25 as reimbursement for their time.
A national provider advisory board, made up of providers of HIV care, provides input on the project to CDC regarding how data are collected and how to increase provider participation. A national community advisory board (CAB) made up of community members from each project area, serves as a link between MMP staff and patients who participate. The national CAB shares information about the project and provides feedback to CDC about patient recruitment, data collection, and how the project is seen by the community. Input from these two groups help to maximize provider and patient response and minimize patient non response.
Minimal data from the HIV/AIDS Reporting System will be collected by each project area on all sampled patients (Attachment 4c). Minimal data on respondents and nonrespondents will be compared to assess non-response bias.
For the MMP Provider Survey, providers will receive a gift card in the amount of $25 for their participation. The survey will only take 20 minutes to complete. However, since providers frequently receive surveys in the mail, the decision was made to include an incentive in an attempt to increase participation rates.
Incentives were used in the ARTAS project. Providers who mailed back the completed survey were sent a check for $25.
4. Tests of Procedures or Methods to be Undertaken
The MMP data collection instruments were developed using questions from previous CDC surveillance projects.
Since these questions comprising the data collection instruments have been previously tested and used, only internal testing by CDC staff was needed. CDC staff tested the skip patterns and responses both electronically and using paper versions of the data collection instruments. CDC staff also conducted mock interviews of CDC staff members using the handheld computers to interview other CDC staff. Mock medical records were developed to serve as training aides to the data abstractors. CDC staff also used the mock medical records to test the data abstraction instrument.
Several project areas are piloted the data collection instruments on patients in care for HIV infection and community members who consented to be interviewed. Pilot testing was determined not to require OMB approval. The purpose of the pilot testing was to allow the pilot project areas to test facility and patient recruitment methods. This was done using elements from a previously OMB approved questionnaire (SHAS, OMB 0920-0262, exp. 06/30/2004). All project areas used these approved MMP interview and abstraction instruments for 2007 data collection.
The MMP Provider Survey data collection instruments were developed using questions from the ARTAS and HCSUS provider surveys.
Since these questions comprising the data collection instruments have been previously tested and used, only internal testing by CDC and contract staff was needed. CDC and the CDC contractor staff tested the skip patterns and responses both electronically and using paper versions of the data collection instruments. The CDC contractor staff also conducted cognitive interviews of 9 HIV care providers to test the survey instrument and to ensure respondents were interpreting the questions in a consistent manner. Cognitive testing was determined not to require OMB approval.
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
Consultants on Statistical Aspects:
RAND Corporation:
Ms. Sandra Berry, MA
Senior Behavioral Scientist
RAND Corporation
1700 Main Street
Santa Monica, CA 90407-2138
(310) 393-0411 X7051
Dr. Sam Bozzette, MD, PhD
Senior Natural Scientist
1776 Main St., m5s
Santa Monica, CA 90407
(310) 393-0411
Dr. Marty Frankel, PhD
Statistician
14 Patricia Lane
Cos Cob, CT 06807
(203) 869-1324
Dr. Martin Shapiro, MD, PhD
Researcher
911 Broxton Ave
LA, CA 90024
(310) 393-0411
Grantees:
California (excluding LA, SF) |
Chicago, IL |
Delaware |
Florida |
Georgia |
Houston, TX |
Illinois (excluding Chicago) |
Indiana |
Los Angeles, CA |
Maryland |
Massachusetts |
Michigan |
Mississippi |
New Jersey |
New York (excluding NYC) |
New York City, NY |
North Carolina |
Oregon |
Pennsylvania (excluding Philadelphia) |
Philadelphia, PA |
Puerto Rico |
San Francisco, CA |
South Carolina |
Texas (excluding Houston) |
Virginia |
Washington |
CDC Project Staff:
A.D. McNaghten, PhD, MHSA Team Leader, Clinical Outcomes Team Centers for Disease Control and Prevention 1600 Clifton Rd, NE MS E-46 Atlanta, GA 30333 Phone: (404) 639-6325 Email: [email protected] |
Linda Beer, PhD Epidemiologist Centers for Disease Control and Prevention 1600 Clifton Rd, NE MS E-46 Atlanta, GA 30333 Phone: (404) 639-5268 Email: [email protected]
|
Janet Blair, PhD, MPH Epidemiologist Centers for Disease Control and Prevention 1600 Clifton Rd, NE MS E-46 Atlanta, GA 30333 Phone: (404) 639-6298 Email: [email protected]
|
Catherine Carroll, MA Project Coordinator Centers for Disease Control and Prevention 1600 Clifton Rd, NE MS E-46 Atlanta, GA 30333 Phone: (404) 639-0959 Email: [email protected] |
Ann Do, MD, MPH Epidemiologist Centers for Disease Control and Prevention 1600 Clifton Rd, NE MS E-46 Atlanta, GA 30333 Phone: (404) 639-6347 Email: [email protected] |
Jennifer Fagan, MA Behavioral Scientist Centers for Disease Control and Prevention 1600 Clifton Rd, NE MS E-46 Atlanta, GA 30333 Phone: (404) 639-8396 Email: [email protected] |
Elaine Flagg, PhD, MS Epidemiologist Centers for Disease Control and Prevention 1600 Clifton Rd, NE MS E-46 Atlanta, GA 30333 Phone: (404) 639-8413 Email: [email protected] |
Emma Frazier, PhD, MPH Senior Data Manager Centers for Disease Control and Prevention 1600 Clifton Rd, NE MS E-46 Atlanta, GA 30333 Phone: (404) 639-6148 Email:[email protected]
|
Corliss Heath, MPH, MDiv Data Analyst Centers for Disease Control and Prevention 1600 Clifton Rd, NE MS E-46 Atlanta, GA 30333 Phone: (404) 639-2989 Email:[email protected]
|
Christopher Johnson, MS Statistician Centers for Disease Control and Prevention 1600 Clifton Rd, NE MS E-46 Atlanta, GA 30333 Phone: (404) 639-2989 Email: [email protected]
|
Rita Morgan, MPH Public Health Advisor Centers for Disease Control and Prevention 1600 Clifton Rd, NE MS E-46 Atlanta, GA 30333 Phone: (404) 639-1930 Email: [email protected]
|
Glenn Nakamura, PhD, MS Data Manager Centers for Disease Control and Prevention 1600 Clifton Rd, NE MS E-46 Atlanta, GA 30333 Phone: (404) 639-2981 Email: [email protected] |
Srijana Sharma Data Manager Centers for Disease Control and Prevention 1600 Clifton Rd, NE MS E-46 Atlanta, GA 30333 Phone: (404) 639-5268 Email: [email protected]
|
Terri Sherman Administrative Assistant Centers for Disease Control and Prevention 1600 Clifton Rd, NE MS E-46 Atlanta, GA 30333 Phone: (404) 639-5268 Email: [email protected]
|
Minn Minn Soe, MBBS, MPH Data Analyst Centers for Disease Control and Prevention 1600 Clifton Rd, NE MS E-46 Atlanta, GA 30333 Phone: (404) 639-5268 Email: [email protected] |
Eduardo Valverde, MPH Epidemiologist Centers for Disease Control and Prevention 1600 Clifton Rd, NE MS E-46 Atlanta, GA 30333 Phone: (404) 639-2048 Email: [email protected] |
| File Type | application/msword |
| Author | ziy6 |
| Last Modified By | ziy6 |
| File Modified | 2009-03-18 |
| File Created | 2009-02-26 |