EVALUATION OF NIAID’S HIV VACCINE RESEARCH EDUCATION INITIATIVE
REQUEST FOR OMB REVIEW
AND SUPPORTING STATEMENT
January 14, 2010
Project Officer:
Katharine Kripke, Ph.D.
Assistant Director, Vaccine Research Program
Division of AIDS, NIAID, NIH, DHHS
6700 B Rockledge Drive, Rm 5144
Bethesda, MD 20892
Telephone: 301-594-2512
Fax: 301-402-3684
Supporting Statement Section A
Table of Contents
A. Justification
A.1. Circumstances Making the Collection of Information Necessary 3
A.2. Purpose and Use of the Information 5
A.3. Use of Information Technology and Burden Reduction 6
A.4. Efforts to Identify Duplication and Use of Similar Information 7
A.5. Impact on Small Business or Other Small Entities 7
A.6. Consequence of Collecting the Information Less Frequently 8
A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 8
A.8. Comments in Response to the Federal Register Notice and Efforts to
Consult Outside Agency 8
A.9. Explanation of Any Incentive or Gift to Respondents 10
A.10. Assurance of Confidentiality Provided to Respondents 11
A.11. Justification for Sensitive Questions 13
A.12. Estimates of Hour Burden Including Annualized Hourly Costs 13
A.13. Estimate of Other Total Annual Cost Burden to Respondents or Recordkeepers 15
A.14. Annualized Cost to the Federal Government 15
A.15. Explanation for Program Changes or Adjustments 15
A.16. Plans for Tabulation and Publication and Project Time Schedule 15
A.17. Reason(s) Display of OMB Expiration Date is Inappropriate 23
A.18. Exceptions to Certification for Paperwork Reduction Act Submissions 23
SUPPORTING STATEMENT
A. JUSTIFICATION
A.1. Circumstances Making the Collection of Information Necessary
The National Institute of Allergy and Infectious Diseases (NIAID) supports basic and applied research to prevent, diagnose, and treat infectious and immune-mediated illnesses, including illness from human immunodeficiency virus/acquired immunodeficiency syndrome (HIV/AIDS). NIAID’s activities are authorized under 42 USC 285f, wherein it is stated,
“The general purpose of the National Institute of Allergy and Infectious Diseases is the conduct and support of research, training, health information dissemination, and other programs with respect to allergic and immunologic diseases and disorders and infectious diseases, including tropical diseases.”
Developing a vaccine that protects against HIV infection is one of NIAID’s highest priorities. NIAID has undertaken a range of activities to support and facilitate HIV vaccine research. Given the daunting complexity of the HIV virus, developing a safe and effective vaccine will ultimately require tens of thousands of volunteers to participate in HIV vaccine clinical trials. Minority participation in HIV vaccine clinical trials is essential; nearly two-thirds of people diagnosed with HIV in the United States are African American or Hispanic/Latino. Historically, recruitment of racial/ethnic minorities has been a critical challenge for medical researchers, and initiatives to increase recruitment of these groups into cancer and chronic disease trials have only been partially successful.
To address the need for volunteers in HIV vaccine clinical trials, and enable NIAID to fulfill its Congressional mandate to prevent infectious diseases like HIV/AIDS, NIAID created the NIAID HIV Vaccine Research Education Initiative (NHVREI), which is planned and implemented under a contract with the Academy for Educational Development (AED). The goal of NHVREI is to increase knowledge about and support for HIV vaccine research among U.S. populations most heavily affected by HIV/AIDS—in particular, African Americans, Hispanics/Latinos, men who have sex with men (MSM), women and youth, recognizing the intersection of these groups.
A critical component of NHVREI is outreach to and involvement of individuals who are considered to be key influencers within communities highly impacted by HIV/AIDS. With the assistance of funded community-based and national organizations through the Local and National Partnership Programs (Partners) and a network of other influential organizations (Network), NHVREI is designing, developing, and disseminating HIV vaccine research-related messages to NHVREI target audiences. These messages are delivered through print (e.g., brochures, posters, fact sheets, information kits), radio, TV, and Internet resources. Print materials are distributed through various NHVREI program activities (e.g., trainings, conferences, symposia) and other NIAID-funded partners, governmental and non-governmental organizations. For this purpose of this document, “NHVREI program staff” refers to NIAID and AED staff working directly on the initiative, “Partners” refers to local and national organizations funded under NHVREI, and the “Network” refers to those organizations affiliated with the NHVREI Network component of NHVREI.
NIAID is conducting an evaluation of the NHVREI program in order to assess its impact and generate key learnings applicable toward the design of future educational initiatives. The evaluation will include a process evaluation to extract details of how NHVREI is implemented, as well as an outcomes evaluation to investigate whether contact with the program is associated with awareness of, knowledge about, and support for HIV vaccine research among key influencers working with populations highly impacted by HIV/AIDS.
In accordance with the Paperwork Reduction Act of 1995 (44 U.S.C. 3501 et seq.) NIAID is requesting clearance from the Office of Management and Budget (OMB) to conduct two data collection procedures as part of the NHVREI evaluation. Specifically, clearance is requested for:
(1) A survey of key influencers; and
(2) A series of focus groups conducted with the Partners and the Network.
The survey and the focus groups will each be conducted twice, once in 2009, and once in 2010. The total number of respondent burden hours will not exceed 400 annually, or 800 overall.
A.2. Purpose and Use of the Information
The evaluation will assess whether NHVREI activities are associated with increased support for vaccine research among key influencers of target audiences over time, and will describe factors that impede or facilitate program effectiveness.
The major questions to be answered by the survey include:
What proportion of key influencers that have no known direct contact with NHVREI programs are aware of HIV vaccine research?
What proportion of key influencers agree with positive statements regarding HIV vaccine research, and does this proportion vary with the level of NHVREI contact?
What proportion of key influencers engage in behaviors that support HIV vaccine research, and does this proportion vary with the level of NHVREI contact?
How confident are key influencers about their ability to improve support for HIV vaccine research, and does their level of confidence vary with the level of NHVREI contact?
Do individuals who have been involved with NHVREI show increased knowledge, positive attitudes, and supportive behaviors over time when compared to individuals with no known NHVREI contact?
The major questions to be answered by means of focus groups with the Partners and Network are:
How did Partner and Network organizations implement NHVREI activities, and were the activities successful in meeting NHVREI goals (i.e., increasing support for HIV vaccine research)?
What were the challenges involved in increasing support for HIV vaccine research? How were these challenges addressed?
What lessons were learned and what should be done differently in the future?
How did NHVREI program staff assist Partner and Network organizations to meet NHVREI goals? What services were particularly helpful, and what additional services are requested?
Evaluation data will be used to refine program strategies and messages, so that the effectiveness of the program can be improved. Data will be used internally by NIAID, and may be reported to interested professional audiences through presentations and/or published papers as the opportunity arises.
A.3. Use of Information Technology and Burden Reduction
Technological strategies will be used to minimize the burden of data collection for both the focus groups and the survey. Telephone focus groups will be convened to maximize geographic diversity without requiring travel time for respondents. Arrangements with respondents will be made using e-mail.
The primary means of data collection for the survey is online. Since most of the key influencers have access to and use the Internet in their work, the online survey will be easy and convenient for most respondents to use. Where e-mail addresses are available, respondents will be able to click on a URL to find the survey Web site, and the incentive will be conferred via an online code immediately after survey completion. The survey will also be transmitted via hard copy mail where necessary.
Online administration will limit the presentation of questions to those that are relevant to the respondent. In contrast, in paper surveys, respondents are often asked to skip items that are rendered irrelevant by answers to previous questions. For example, on paper, respondents reporting no awareness of HIV vaccine research must skip past questions related to the source of HIV vaccine research information to answer the next item. Online surveys can move respondents directly from a screen showing an awareness question to a screening showing the next relevant item.
Online administration of the survey is efficient, because data are entered by the respondent directly into the database, avoiding the separate step of key entry of paper survey data into a database. The cleaning of the data is also facilitated by online administration, because entering out-of-range answers will not be permitted by the survey program software. If a respondent has reservations or comments about the forced choices presented to him/her, the respondent can enter text in a comment box associated with the item at any time.
The online survey system will be used as the primary data entry mechanism for all survey modes, thus improving efficiency. For example, if respondents refuse to participate online, the survey will be administered via a hard copy paper survey or by telephone. In either case, information will be entered into the database by evaluation staff using the online system; no additional data entry system will be required. A Privacy Impact Assessment (PIA) was completed and has been reviewed by the NIAID ISSO and CTO.
A.4. Efforts to Identify Duplication and Use of Similar Information
The general area in which information needs to be gathered (as described in A.2.) to evaluate NHVREI has never been collected before. The NHVREI program began in September 2006, and this will be the first evaluation of the program.
NHVREI program staff have collected information from key influencers for the purpose of developing effective messages regarding HIV vaccine research. OMB previously reviewed and cleared this data collection process (OMB No. 0925-0585; Exp: 02/28/2011). In contrast, the data collections for which NIAID requests clearance in this submission will focus on gathering information about how NHVREI programs affect influential organizations and individuals working in communities highly impacted by HIV/AIDS.
Importantly, the same NIAID Contracting Officer’s Technical Representative (COTR) oversees both the NHVREI and NHVREI Evaluation contracts, thus ensuring coordination of efforts and minimization of overlapping activities. Furthermore, NIAID actively works with other government agencies including CDC regarding HIV/AIDS prevention, and NHVREI program staff attend national meetings and are in frequent contact with HIV/AIDS experts across the globe. There are no known data sources other than the proposed primary data collection activities that will meet the needs of the NHVREI evaluation.
A.5. Impact on Small Businesses or Other Small Entities
Key influencers that are the target for this data collection activity are often employed in small community-based organizations. Work is unlikely to be disrupted because participation in the survey can occur at the respondent’s convenience, in off hours; an incentive will be conferred because completing the survey is not part of respondents’ work.
Staff from Partner organizations who are working on NHVREI-related activities will participate in the telephone focus groups in place of scheduled monthly calls that are a requirement of their NHVREI contract, so no additional burden is expected, and no additional funding is conferred. Network members will participate in the focus groups as part of their agreement with NIAID.
A.6. Consequence of Collecting the Information Less Frequently
It is necessary that evaluation data be collected twice, since assessing the effects of the program requires measurement of change over time. It is hypothesized that continued exposure to NHVREI materials and activities will be associated with increased positive attitudes and behaviors supporting HIV vaccine research over time. In other words, improvements on these outcomes from time 1 to time 2 will be significantly greater among respondents with NHVREI contact than any change reported by respondents with no known NHVREI contact. (See page 4 in the Supporting Statement). Data collection will occur in late 2009 and late 2010. The data collections will assess changes in parameters thought to be critical to the development of HIV vaccine research knowledge and support in populations highly impacted by HIV/AIDS. It is hypothesized that increases in attitudes and behaviors supportive of HIV vaccine research will be more likely to occur in individuals with exposure to NHVREI programs. If the data collection is not conducted according to this timetable, NIAID will not be able to track any changes that are expected to result over time as the Initiative is progressing. The timing of the data collection is also essential to inform NIAID’s future educational initiatives.
A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
This information collection fully complies with 5 CFR 1320.5(d) (2).
A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency
The required 60-day notice appeared in the Federal Register on 7/16/2009 (Volume 74, Number 135, pp. 3480-81), soliciting comments on the requested new data collection project. No comments were received.
NIH, along with other Public Health Service agencies, has been a leader in the development of methods for developing, testing, and disseminating health information. A number of outside health communications experts were consulted to review the plans contained herein for program development research and evaluation of NIAID educational initiatives and their comments and suggestions have been incorporated into these data collection plans.
Those outside NIAID who have been consulted about this study include:
Dan Eckstein, NOVA Research Company, 301-986-1891, [email protected]
Paul Young, NOVA Research Company, 301-986-1891, [email protected]
Lisbeth Jarama, NOVA Research Company, 301-986-1891, [email protected]
Caroline McLeod, NOVA Research Company, 301-986-1891, [email protected]
Allison Zambon, NOVA Research Company, 301-986-1891, [email protected]
Gail Broder, HIV Vaccine Trials Network, 206-667-7348, [email protected]
Steven Wakefield, HIV Vaccine Trials Network, 206-667-6705, [email protected]
Kaijson Nolimar, HIV Vaccine Trials Network, 206-667-7481, [email protected]
Sarah Alexander, HIV Vaccine Trials Network, 206-667-5296, [email protected]
Enid Moore, HIV Vaccine Trials Network, [email protected], 206-667-4721
Cornelius Baker, Academy for Educational Development, 202-884-8612, [email protected]
Russell Brewer, Academy for Educational Development, 202-884-8797, [email protected]
Stacey Little, Academy for Educational Development, 202-884-8727, [email protected]
Catharine Laube, Henry M. Jackson Foundation, 301-451-2795, [email protected]
Sandra Sitar, Kelly Services, 301-594-8569, [email protected]
Diane Johnson, Kelly Services, 301-451-8715, [email protected]
NIAID staff who have been consulted about this study include:
Margaret Johnston, 301-402-0846, [email protected]
Katharine Kripke, 301-594-2512, [email protected]
Kathy Stover, 301-451-2278, [email protected]
Kevin Wright, 301-402-3574, [email protected]
Brandie Taylor, 301-451-3068, [email protected]
Tarsha McCrae, 301-443-8972, [email protected]
Fulvia Veronese, 301-402-4148, [email protected]
NIH is the U.S. Government lead for HIV vaccine research, discovery and development, and coordinates with other U.S. Government agencies on all HIV vaccine research efforts through the Partnership for AIDS Vaccine Evaluation (PAVE). Through this mechanism, NIAID consults regularly with other agencies to help ensure accuracy and consistency and to avoid duplication of effort. NIAID also regularly consults and coordinates with non-US Government HIV vaccine research organizations through formal and informal channels.
A.9. Explanation of Any Incentive or Gift to Respondents
Incentives are commonly used in order to obtain assistance from those who may not otherwise participate in data collection efforts. There is extensive literature to support the use of incentives, primarily monetary incentives, as a supplement or complement to other efforts of persuasion to ensure recruitment of a representative sample. Incentives are particularly important for hard-to-reach and minority populations, or with survey topics that may seem insignificant to the potential respondent. In studies for both commercial market research and social sciences, findings indicate that respondents who receive these tokens of appreciation provide valid input, and their inclusion makes for a more representative sample.1,2
Focus Groups. All focus group participants have worked with NHVREI, so the topic is relevant and important to their professional lives. Participants will not receive an incentive to participate in the focus group, since the focus group call will take the place of a usual monthly call funded by NHVREI and the focus group topic is directly related to the participants’ work on NHVREI.
Survey. Use of a $25 incentive is recommended for the survey. Most potential respondents are likely to have minimal awareness of the survey topic, so the incentive is necessary to arouse interest and attention. Furthermore, because key influencers have multiple demands on their time, completing the survey is likely to fall to the bottom of their priority list unless there is some incentive involved.
Both Partner and Network representatives will be offered an incentive, even though they have high awareness of the topic. The incentive will be offered to them, because (1) completion of the survey is likely to occur in off-hours, during their free time, for which there are multiple competing priorities, and (2) many survey items are related to their personal attitudes and actions, rather than their professional opinions or their jobs. Offering an incentive helps to clarify that participants’ responses on the survey are unrelated to any job assessment, thereby helping to avoid potential biases.
An incentive level of $25 has been shown to be effective in increasing response rates;3 increasing incentive rates above $25 have been shown to result in diminishing returns 4 and may not be necessary. Sampled individuals are memebers of the Highly Impacted Populations whose participation it has been so difficult to obtain in NIAID vaccine development studies. These populations have a history of distrust or disenfranchisement related to HIV research, and it is this history that NHVREI is attempting to change. Potential respondents are working at non-profit organizations working to address the HIV/AIDS epidemic. We expect most potential respondents to be underfunded, overburdened with their outreach efforts, and unlikely to prioritize completing a survey. Of particular concern is that without an incentive, respondents with more negative views toward NIAID and vaccine research will not take the time to participate. Yet, the opinions of these individuals are ones that will prove critical to the guidance of future NHVREI efforts. While prepaid incentives have been shown to work better than incentives provided by mail after survey data has been received,5 prepaid incentives create extraneous costs when some individuals accept the incentive without providing data. The online survey described will utilize a technology that allows for immediate release of an incentive upon completion of data collection.
A.10. Assurance of Confidentiality Provided to Respondents
Information provided by respondents will be kept private to the extent permitted by law. This will be communicated to respondents by means of introductory letters, explanatory texts on the cover pages of questionnaires, scripts read prior to focus groups, and consent forms. NIAID and its contractor, NOVA Research Company, will follow best practices to maximize privacy and security of all data. For the focus groups, this means that though the identities of participants will be known to NIAID, the contractor will report information obtained from group discussions in such a manner that the specific ideas cannot be linked to an individual.
For the survey, each respondent will be assigned a unique identification (ID) number. This number will be used as a unique record identifier for survey answers. The data file containing names and ID numbers will be maintained separately from the file containing survey answers. Both files will be maintained in a secure environment. Contact information will be used by the contractor only for mailing a letter requesting subject participation and for subsequent follow up in the case of non-response and for the second survey in 2010.
Instructions on the survey and for focus groups and a document providing background information on the study will apprise the respondent of the following:
The survey and focus groups are sponsored by the National Institute on Allergy and Infectious Diseases, part of the National Institutes of Health.
Survey and focus group data will be used to improve public health experts’ understanding of public knowledge, attitudes, and behaviors related to HIV vaccine research.
Information provided by individuals will be kept private. Reports based on survey and focus group data will involve information combined across all respondents. Personal identifying information will be used only by researchers following up with respondents for a second data collection, and identifying information will not be disclosed except as required by law;
For survey respondents, providing the information is voluntary, and there are no penalties for not responding to the information collection as a whole or to any particular questions. Though some focus groups respondents are required to participate as part of their funded responsibility, there are no penalties to any individual for not responding to a particular line of questions.
NIAID will retain ownership of all data collected, but when the contractor submits data to NIAID, no personal identifiers will be included.
In order to protect respondents’ privacy, all presentation of data in reports will be in aggregate form, with no links to individuals preserved. Reports will be used to guide NHVREI and public health policy as well as to improve knowledge in the field of health education.
The data collection is covered by NIH Privacy Act Systems of Record 09-25-0156, “Records of Participants in Programs and Respondents in Surveys Used to Evaluate Programs of the Public Health Service, HHS/PHS/NIH/OD,” which is available at http://oma.od.nih.gov/ms/privacy/pa-files/0156.htm. A statement from the Privacy Act Officer at the NIH is found in Attachment A.
The instruments utilized in this data collection include a survey and focus group guide. NOVA’s Institutional Review Board (IRB) reviewed the key influencer survey and focus group moderator’s guide and gave them expedited approval since the questions focus on non-sensitive issues, and there is low risk of breach of confidentiality. (See Attachment B for IRB approval documentation).
A copy of the survey for key influencers can be found in Attachment C. The data collection instrument for the focus groups is provided in Attachment D. The key influencer survey invitation and reminder e-mails are available as Attachments E and F. The key influencer survey reminder phone call scripts are available as Attachment G. The key influencer focus group invitation can be found in Attachment H and the informed consent form for the focus group is available as Attachment I.
A.11. Justification for Sensitive Questions
No questions of a sensitive nature are being asked. Questions are of a general nature and information is reported in the aggregate rather than attributed to specific individuals. Personal Identifying Information (PII) will be collected; however PII is being retained by the contractor and will not be shared with the government. PII is held separately from survey answers and focus group transcripts.
A.12. Estimates of Hour Burden Including Annualized Hourly Costs
Response burden estimates are shown in Table A.12 - 1. The survey questionnaires will require 20 minutes to complete. The number of expected survey respondents is 656 individuals at Time 1, assuming an expected 80% response rate, and 590 at Time 2, given an expected 90% followup rate (the same individual responding to the second survey) for the followup survey. Table A12-1 shows the average burden hour per survey respondent is 0.33, with the estimated total annual burden hours estimated at 216 for year 1 and 195 for year 2. This estimate is based on experience with the cognitive interviews described in Section B.4.
The focus groups are estimated to take approximately one hour. The number of respondents is targeted at 78 individuals. Table A12-1 shows the average burden hour per respondent is 1, with the estimated total annual burden hours estimated at 78 for year 1 and 78 for year 2.
Table A.12-1 Estimates of Hour Burden by Anticipated Data Collection Methods and Year
|
Total # of Respondents |
Hours Per Response |
Total Hours |
Time 1 |
|
|
|
Focus Groups |
78 |
1 |
78 |
Key Influencer Surveys |
656 |
0.33 |
216 |
Total Time 1 |
734 |
|
294 |
Time 2 |
|
|
|
Focus Groups |
78 |
1 |
78 |
Key Influencer Surveys |
590 |
0.33 |
195 |
Total Time 2 |
668 |
|
273 |
Total Time 1 & Time 2 |
1,402 |
|
567 |
The total annualized cost to respondents for both data collections is estimated at $7,303 for year 1 and $6,763 for year 2 as shown in Table A.12‑2. Annualized costs use the mean hourly wage for social and community service managers provided by the U.S. Department of Labor, Bureau of Labor Statistics (the latest data available)6.
Table A.12-2 Cost to Respondents
|
Total # of Respondents |
Time Per Response |
Hourly Wage Rate |
Respondent Cost |
Time 1 |
|
|
|
|
Focus Groups |
78 |
1 |
$24.80 |
$1,934 |
Key Influencer Surveys |
656 |
0.33 |
$24.80 |
$5,369 |
Total Time 1 |
734 |
|
|
$7,303 |
Time 2 |
|
|
|
|
Focus Groups |
78 |
1 |
$24.80 |
$1,934 |
Key Influencer Surveys |
590 |
0.33 |
$24.80 |
$4,829 |
Total Time 2 |
668 |
|
|
$6,763 |
Total Time 1 & Time 2 |
1,402 |
|
|
$14,066 |
A.13. Estimate of Other Total Annual Cost Burden to Respondents or Recordkeepers
There are no capital or start-up costs to the data collection efforts requested; nor are there any costs associated with operation, maintenance or purchase of services.
A.14. Annualized Cost to the Federal Government
The annualized cost to the government to conduct and analyze the focus groups and survey is $203,000 per year over the 3 years of the project. The budget includes the costs of survey design and development, focus group design, all data collection and follow up, incentive payments, data file preparation and documentation, initial analyses, and other miscellaneous costs such as supplies, expenses, and postage. Professional service time is included for study management and overhead costs.
Table A.14 - 1 Estimates of Annualized Cost to the Government
Year |
Estimated Costs |
2009 |
111,000 |
2010 |
236,000 |
2011 |
261,000 |
Total Over Three Years |
608,000 |
A.15. Explanation for Program Changes or Adjustments
This is a new collection of information.
A.16. Plans for Tabulation and Publication and Project Time Schedule
The current plan is to begin both the survey and focus groups in November 2009, if OMB clearance has been received by that date. Otherwise, the survey will be fielded as soon as possible after clearance has been received.
Table A.16-1a Project Time Schedule- Key Influencer Survey
Activity |
Estimated Time Schedule |
Time 1 Survey |
|
Send initial e-mail/postal invitations |
November 1, 2009 |
First followup |
November 30, 2009 |
Final followup |
December 15, 2009 |
Data collection halted |
January 1, 2010 |
Analysis of Data |
January/February 2010 |
Preliminary Report |
March 2010 |
Final Report |
May 2010 |
Time 2 Survey |
|
Send initial e-mail/postal invitations |
November 1, 2010 |
First followup |
November 30, 2010 |
Final followup |
December 15, 2010 |
Data collection halted |
January 1, 2011 |
Analysis of Data |
January/February 2011 |
Preliminary Report |
March 2011 |
Final Report |
May 2011 |
A.16-1b Project Time Schedule- Focus Groups
Activity |
Estimated Time Schedule |
Time 1 Focus Group |
|
Contact participants and schedule groups |
November 2009 |
Conduct Focus Groups |
November/December 2009 |
Analysis of Data |
January/February 2010 |
Preliminary Report |
March 2010 |
Final Report |
May 2010 |
Time 2 Focus Group |
|
Contact participants and schedule groups |
November 2009 |
Conduct Focus Groups |
November/December 2009 |
Analysis of Data |
January/February 2010 |
Preliminary Report |
March 2010 |
Final Report |
May 2010 |
Publication Plan
NIAID anticipates making evaluation results available to a variety of health program planners at government agencies, community-based organizations, health professional organizations, and medical institutions. An abstract for a presentation at the American Evaluation Association has been submitted. A paper based on the methodology for collecting the sample population was presented and well received at the November 2009 AEA meeting.
This survey will obtain data on key influencer knowledge, attitudes, and behavior at two times. For the purpose of analysis, the key influencers are divided into three groups, corresponding to their level of exposure to NHVREI (i.e., High Contact, Other Contact, and No Known Contact). A description of how the key influencers are identified and assigned to these groups is found in Section B.2.
Key analyses include the following:
Descriptive statistics at Time 1;
Comparisons at Time 1 of key influencer groups;
Comparisons within and between groups on variables expected to change over time;
Identification of factors related to increases in support of HIV vaccine research; and
Nonresponse evaluation.
In the following sections we provide greater detail on these analyses, including examples of table shells.
Descriptive Statistics at Time 1. Table A.16 - 2 indicates examples of descriptive data that could provide important guidance to NIAID and the contractor implementing NHVREI as they continue with outreach to the impacted communities. For example, the survey will provide information about whether public service messages and presentations are being noticed by key influencers outside of the High Contact group.
The High Contact group will not be asked questions that are known to be related to their work as Partners, and thus they are excluded from responses to certain questions. For example, all persons in the High Contact group have worked with HIV vaccine education materials within the last year, so questions about awareness of HIV vaccine research and distribution of materials are irrelevant and redundant.
Table A.16-2 Percent of Key Influencers Reporting Awareness and Positive Attitudes
|
Percent of Sample |
Heard or read about HIV vaccine research in last year* |
|
I am confident that an effective vaccine for HIV will be developed. |
|
* Excludes High Contact Group
Finally, responses from individuals in the High Contact group will provide information about the extent to which underlying concerns have been addressed by NHVREI, whether attitudes indicative of personal and professional commitment have been established, and whether these attitudes give rise to behaviors that can be expected to increase support among key influencers’ social networks. Table A.16-3 indicates examples of how such data might be displayed.
Table A.16-3 Percent of Key Influencers with High Contact with NHVREI Reporting Knowledge, Positive Attitudes, and Behaviors Supportive of HIV Vaccine Research
|
Percent of High Contact Group Reporting |
I am confident that HIV vaccine clinical trials staff take good care of volunteer participants. |
|
Spreading the word about the importance of HIV vaccine research conflicts with outreach on HIV/AIDS prevention. |
|
If I support HIV vaccine research, it will hurt me socially and/or professionally |
|
Have you personally provided information to others about HIV vaccine research? |
|
Comparisons of Key Influencer Groups at Time 1. The key comparisons at Time 1 involve comparing the awareness, attitudes, and behaviors of key influencers with respect to two factors: level of NHVREI contact (High Contact, Other Contact, and No Known Contact with NHVREI) and Organizational Scope (Local or National). Multivariate analyses using measures of continuous and dichotomous outcomes will be conducted as appropriate, including logistic and linear regression. These multivariate models will take into account that there are no cases from National organizations in the “No Known Contact” category. In secondary analyses, awareness of vaccine research is expected to be significantly higher in the Other Contact when compared to the No Known Contact group. However, the Other Contact group’s relatively brief interactions with NHVREI at Time 1 seem unlikely to be associated with higher levels of positive attitudes or supportive behaviors in comparison to the No Known Contact group.
With regard to comparisons with the High Contact group that has 46 cases, statistically significant findings can be detected only if effect sizes are expected to be large. This is likely to be true of comparisons between High Contact and No Known Contact group, where higher levels of knowledge/awareness, positive attitudes, and supportive behaviors are expected in the High Contact group.
Table A.16-3 is an example of how findings will be displayed. The High Contact group will not be asked to respond to items related to their work as NHVREI partners, so the relevant table cells are marked as “NA” (Not Applicable). For example, as a NHVREI partner, all organizations have used “Be the Generation” materials and they distribute materials regarding HIV vaccine research.
Table A.16-3 Comparisons of Positive Attitudes and Supportive Behaviors Among Key Influencers with Different Exposure to NHVREI
6g. Disagree that an HIV vaccine is being unfairly withheld as part of discrimination against certain groups. |
High Contact |
Other Contact |
No Known Contact |
National |
|
|
NA |
Local |
|
|
|
|
|
|
|
4b. Used “Be The Generation” materials in their work. |
|
|
|
National |
NA |
|
NA |
Local |
NA |
|
|
|
|
|
|
16a. Spoken to friends about HIV vaccine research at least once or twice over the last 3 months. |
|
|
|
National |
|
|
NA |
Local |
|
|
|
Comparisons Within and Between Groups on Variables Expected to Change Over Time. Multivariate analyses using repeated measures of continuous and dichotomous outcomes will assess differences within and between groups. A key comparison involves responses from key influencers with different exposure to NHVREI, with a greater number of key influencers in the Other Contact group showing an increase in support for HIV vaccine research compared to individuals in the No Known Contact group after accounting for differences in Organizational Scope.
Moreover, the Other Contact group is expected to show a statistically significant increase in the attitudes and behaviors supportive of HIV vaccine research compared with Time 1. Table A16.-4 is an example of how these comparisons will be displayed.
Table A.16-4 Comparisons of Positive Attitudes and Supportive Behaviors Among Key Influencers with Different Exposure to NHVREI
|
High Contact
|
Other Contact
|
No Known Contact
|
|||
|
Time1 |
Time 2 |
Time 1 |
Time 2 |
Time 1 |
Time 2 |
6g. Disagree that an HIV vaccine is being unfairly withheld as part of discrimination against certain groups. |
|
|
|
|
|
|
National |
|
|
|
|
NA |
NA |
Local |
|
|
|
|
|
|
9e. Willingness to encourage clients to volunteer for an HIV vaccine trial. |
|
|
|
|
|
|
National |
NA |
NA |
|
|
NA |
NA |
Local |
NA |
NA |
|
|
|
|
16a. Spoken to friends about HIV vaccine research at least once or twice over the last 3 months. |
|
|
|
|
|
|
National |
|
|
|
|
NA |
NA |
Local |
|
|
|
|
|
|
Factors Associated with Increased Support of HIV Vaccine Research. For these exploratory analyses, composite scores indicating high versus low levels of support will be constructed based on observed distributions. Key influencers reporting high levels of support on the composite score will be compared with those reporting low levels on a number of different variables, including level of contact, education level, role in the organization, time with the organization, impacted population served, and other variables. Logistic and linear regression modeling will help identify predictors of high levels of support. Chi-square analyses will be used to explore relationships.
Non-Response. As described in Section B.3, considerable effort will be expended to minimize rates of nonresponse. Analyses will be conducted on the characteristics of individuals who do not respond to the survey based on available information, including city, impacted population served, and role in the organization. In the final report, any patterns of nonresponse will be discussed relative to the potential for contributing to any bias in results.
Analysis Plan-Focus Groups
This information collection does not require statistical analyses. Qualitative data from the focus groups will be transcribed verbatim. The analysis of focus group data will emphasize exploration of common themes and issues rather than prescription or prediction. These data will be analyzed and interpreted using content analysis in which main ideas (i.e., themes), comments, and words are grouped based on variables of interest.
Qualitative software, specifically ATLAS.ti, will be used for these analyses. ATLAS.ti is a qualitative analysis research tool that allows for analysis across various data configurations and across all levels of the system. It also allows for individual segments of data to be coded for multiple themes, providing a means for relevant data to be utilized in addressing multiple research questions and emergent analytic themes. ATLAS.ti also creates a data trail to connect data from the transcripts to specific findings.
To maximize reliability, coding (i.e., categorizing) of data and thematic analysis of text will be conducted by experienced evaluators.
A.17. Reason(s) Display of OMB Expiration Date is Inappropriate
NIAID will display the OMB control number and expiration date in the upper right-hand corner of all data collection instruments. We are not seeking a waiver to display the expiration date for OMB approval.
A.18. Exceptions to Certification for Paperwork Reduction Act Submissions
NIAID is in full compliance with the provisions contained within the Certification for Paperwork Reduction Act Submissions.
Supporting Statement Section B
Table of Contents
October 16, 2009
B. Collection of Information Employing Statistical Methods 25
B. 1. Respondent Universe and Sampling Methods 25
B. 2. Procedures for the Collection of Information 27
B. 3. Methods to Maximize Response Rates and Deal with Non-Response 32
B. 4. Test of Procedures or Methods to Be Undertaken 34
B. 5. Individuals Consulted on Statistical Aspects and Individuals Collecting
and/or Analyzing Data 35
B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
This supporting statement addresses two data collection procedures related to the evaluation of the NIAID HIV Vaccine Research Education Initiative (NHVREI): the focus groups and the survey. Only the survey involves the use of statistical methods. Therefore, this section will describe in detail the various aspects of key influencer survey data collection methods.
B.1. Survey Respondent Universe and Sampling Methods
This survey will collect information for an evaluation using a nonequivalent groups design, allowing comparisons between key influencers with different levels of NHVREI contact. Potential key influencer respondents are identified by:
(1) Their association with a NHVREI Partner or Network organization;
(2) Their collaboration with NHVREI Partners; or
(3) Their role in organizations that have had no NHVREI contact but that are matched to Partner, Network, or collaborator organizations.
Potential respondents are key influencers within the Highly Impacted Populations targeted by NHVREI, and by definition are members of that population.
(1) Partner and Network Staff.
NHVREI staff will provide a list of the 23 Partner and 25 Network organizations. Up to 5 individuals (including the core staff responsible for implementing NVHREI activities) will be sampled from each of the 48 organizations, for a total of 240 individuals. Individuals will fill the following positions (or their equivalents, as applicable):
Two individuals from board-level positions, including the board of directors, advisory board or the advisory council. Potential respondents will be selected in the order in which they are displayed on the website.
Two individuals at the administrative/executive level, including executive directors, CEOs, and/or program directors. If the organization is so small that only 1 administrator is involved, an additional board-level member will be selected.
One individual holding a position as Program Outreach/Health Education/Communications Specialist, or other position that involves direct contact with clients from the community.
(2) Collaborators
Each of the 23 Partners will provide a list of 10 individuals who have received information, training, or have formed partnerships with the Partners, for a total of 230 individuals working at the local level. The level of collaboration between organizations or between a partner organization and an individual may vary. Partners that have focused their efforts on developing partnerships with just a few organizations may select more than one individual from an organization; other Partners focusing on broad outreach to key influencers may provide ten individuals associated with ten different organizations.
(3) No Contact Matches
Each of the individuals working in organizations with a local community focus and known NHVREI contact (18 of 23 Partners, 6 of 25 Network, and all collaborators) will be matched with individuals working in a similar role in an organization with similar characteristics, but with no known or minimal NHVREI contact. The matched organizations are expected to be involved at a local level in HIV/AIDS prevention, advocacy or health education/outreach, serving communities highly impacted by HIV/AIDS in cities without a Local Partner. About 90 individuals from 18 Local Partners will be matched with 90 potential no-contact respondents. Similarly, 30 individuals from 6 Local Network organizations will be matched with 30 persons with no known contact, and 240 collaborators will be matched, for a total of 350 individuals with no known NHVREI contact. Notably, staff from 6 National Partner and 15 National Network organizations will not be matched, since most or all national organizations related to HIV and highly impacted communities have already had contact with NHVREI in some way. See Table B1-1 for a summary of this information.
Table B1-1 Source of Individuals in the Sample
NHVREI Organization Type |
Individuals With Some NHVREI Contact |
Individuals From Matched Organizations (No Known Contact) |
Total Number Individuals |
5 National Partners (5 people per org) |
25 |
0 |
25 |
18 Local Partners (5 people per org) |
90 |
90 |
180 |
Local Collaborator (10 people per Partner) |
230 |
230 |
460 |
19 National Network (5 people per org) |
95 |
0 |
95 |
6 Local Network (5 people per org) |
30 |
30 |
60 |
TOTAL |
470 |
350 |
820 |
.
B.2. Procedures for the Collection of Information
NHVREI program staff will request that each Partner and Network organization provide (1) contact information for core staff (i.e., staff funded to perform NHVREI activities), as well as (2) contact information for ten people with whom the Partner plans to collaborate on NHVREI-related activities within the study period. Evaluation staff will identify other potential respondents through a review of the organizations’ Web sites and through the responses of program directors, who will be asked to identify outreach staff within their programs on the survey itself.
Sampled individuals will be sent an e-mail and/or letter requesting their assistance in an online survey related to HIV vaccine research. The recruitment e-mail/letter will indicate that the study is sponsored by the National Institute of Allergy and Infectious Diseases of the National Institutes of Health. The e-mail/letter will succinctly inform the reader of the importance of the survey, as well as procedures for maintaining the privacy of respondents (i.e., the identities of individuals will not be released, identifying information will be stored separately from the survey responses, and all collected information will be analyzed in the aggregate). The e-mail/letter will state that completion of the online survey will result in the release of an incentive. The incentive will be provided immediately on the computer screen as a unique coupon code to be used at Amazon.com. The message will also mention that there will be a follow up survey in about a year. A draft of the e-mail/letter may be found as Attachment E.
The letter/email from NIAID will be accompanied by a letter of support from a national-level organization known to command respect among community-based organizations working with populations highly impacted by HIV/AIDS. The letter will encourage readers to complete the survey, explaining why participation in the survey is important to organizations serving communities highly impacted by AIDS. An example of the letter may be found as Attachment J.
The letter/e-mail will provide the URL for the online survey. The URL will lead to an online survey that has been customized to provide the questions appropriate for the level of NHVREI contact and the organization involved. On the survey instrument attached in Attachment C, items are marked for use with High Contact, Other Contact, and No Known Contact. The instrument has been designed so that for each individual, the survey takes no more than 20 minutes to complete.
Approximately three weeks after the first e-mail or letter has been sent, a second message will be sent to nonresponders reiterating the request for participation. Further follow up via e-mail, postal service, and telephone will be attempted until the 80 percent response rate is achieved.
Approximately 12 months after initial data collection is completed, an e-mail or letter will be sent to individuals who had previously responded to the survey. In instances where an individual has moved from the organization, a survey will be sent to the person in a comparable position at Time 2 if the person has been in that position at least 3 months; otherwise the original respondent will be sent the survey. As with the Time 1 letter, the Time 2 letter will provide an URL for an online survey, along with information about the study, circumstances of privacy, and how to access an incentive.
Power Analysis. There are two key analytic objectives that drive the determination of sample size. The first involves comparisons between groups describing level of contact at Time 1, while the second involves comparisons between changes in scores between Time 1 and Time 2.
Group comparisons. Individuals will be categorized into one of three groups associated with three levels of NHVREI contact:
High Contact, involving core staff in NHVREI Partner organizations who have responsibility for implementing NHVREI programs and have direct contact with NHVREI program staff;
Other Contact, involving:
Partner organizations’ peripheral staff and board members,
Collaborators;
Network organization staff; and
No Known/Minimal Contact, involving staff from local organizations that have no known or minimal direct involvement with NHVREI, but are matched with local organizations known to have had direct NHVREI contact.
Table B.2-1 shows the number of individuals to be recruited for the survey, with the expected yield at Time 1 (with 80 percent response rate) and Time 2 (with a 90 percent followup rate from people who responded at Time 2).
Table B.2 – 1 Survey Sample Numbers and Yield Response Rates
Organization Type |
Number Receiving Request to Participate |
Number Responding at Time 1 |
Number With Complete Data at Time 1 and Time 2 |
(80% Response Rate) |
(90% FU Rate) |
||
High Contact |
|
|
|
Core staff from 23 Partner Organizations (2 staff per org) |
46 |
37 |
33 |
Total |
46 |
37 |
33 |
Other Contact |
|
|
|
Peripheral Staff/Board from 23 Partner Organizations (3 staff per org) |
69 |
55 |
50 |
Individuals from 25 Network Organizations (5 staff per org) |
125 |
100 |
90 |
Collaborators |
240 |
184 |
166 |
Total |
424 |
339 |
305 |
No Known/Minimal Contact |
|
|
|
Individuals from Matched Local Organizations |
350 |
280 |
252 |
Total |
350 |
280 |
252 |
TOTAL |
820 |
656 |
590 |
* Core staff and peripheral staff come from the same Partner Organizations
A key variable for the survey is the proportion of key influencers reporting that they have spoken at least once to friends or family about supporting vaccine research, and there is adequate power to detect differences between the Other Contact and No Known Contact groups at Time 1. Table B.2-2 shows that with the harmonic mean at 306 Time 1 cases in the Other Contact and No Known Contact groups, power to detect a 10 percent difference between proportions at the midpoint is over 79 percent with alpha = .05 one-tailed. Comparisons involving a larger difference, proportions well above or below the midpoint, or those involving a greater number of cases will have greater power.
Table B.2-2 Expected Power to detect the specified difference between No Known Contact and Other Contact (harmonic mean=300) at alpha = .05 one-tailed |
|||
Proportion No Known Contact |
Proportion Other Contact |
H (effect size) |
Power to detect difference between P1 and P2 alpha = .05 one-tailed |
45% |
55% |
.2 |
.79 |
45% |
60% |
.3 |
.98 |
10% |
21% |
.3 |
.98 |
Comparisons involving the High Contact group will have power to detect only large differences between the groups because of the relatively low number of cases in that group (n=37). Fortunately, a large difference in proportions is expected to exist between the High Contact group and the No Known Contact group at Time 1. As shown in Table B.2-3, if 10 percent or less in the No Known Contact group report a given opinion, and 30 percent or more in the High Contact group report it, the power to detect that difference as statistically significant will be at least 86 percent, well above the standard of 80 percent. If the proportion of No Known Contact cases in the analysis is much higher, power to detect differences is lower. At the midpoint, differences of 24 percent exceed the 80 percent power target by several points, with differences of 20 percent detectable 73 percent of the time.
Table B.2-3 Expected Power to detect the specified difference in proportions between No Known Contact and High Contact (harmonic mean = 65, alpha=.05 one-tailed) |
|||
Proportion No Known Contact |
Proportion High Contact |
h (effect size) |
Power to detect difference between proportion alpha = .05 one-tailed |
10% |
29% |
.5 |
.88 |
45% |
69% |
.5 |
.88 |
45% |
65% |
.4 |
.73 |
Changes over time. Respondents will be asked to complete a second survey about one year after the first one has been completed. Comparisons will be made between the Other Contact and No Known Contact groups on the number of respondents reporting an increase in positive attitudes or supportive behaviors. If there are at least 305 and 252 cases in each group, respectively, the evaluation will be able to detect at least a 15 percent difference between the two groups at a power of 96 percent or greater at all proportions, as shown in Table B.2-4. Note that if the proportion of cases showing increases in desired knowledge, attitudes, and behaviors in the No Known Contact group is low (about 5 percent), the study will be able to detect a much smaller difference between the groups (between 5 and 13 percent). The 45-55 percent comparison in the last row of the table displays the power (72 percent) associated with the detection of a 10 percent difference between the proportions at n’=250, alpha=.05 one-tailed. This is the minimum power to detect at 10 percent difference for the sample.
Table B.2-4 Expected Power to detect the specified difference over time for Other Contact and No Known Contact (harmonic mean =276, independent sample, alpha = .05 one-tailed) |
|||
P1 (No Known Contact) |
P2 (Other Contact) |
H (effect size) |
Power to detect difference between P1 and P2 alpha = .05 one-tailed |
5% |
13% |
.3 |
.96 |
10% |
21% |
.3 |
.96 |
20% |
33% |
.3 |
.96 |
30% |
44% |
.3 |
.96 |
40% |
55% |
.3 |
.96 |
45% |
60% |
.3 |
.96 |
45% |
55% |
.2 |
.72 |
B.3. Methods to Maximize Response Rates and Deal with Non-Response
High response rates minimize selection bias in survey findings. Several procedures will be implemented to maximize the response rate. Survey response rates are more robust when the research topic is salient to the respondent’s work, when the questionnaire has been designed for maximum ease of administration, and when the data collection protocol is tailored through a variety of incentives and accommodations to acknowledge respondents’ cooperation and contribution. The presentation of the survey is also important, so that respondents can differentiate it from other mail and research requests.
The introductory letter or e-mail with the link to the survey will indicate that it is sponsored by NIAID, a prestigious NIH institute known by these HIV/AIDS prevention educators to be at the forefront of HIV/AIDS research. The e-mail/letter will succinctly inform the reader of the importance of the survey, as well as procedures for maintaining the privacy of respondents (i.e., identities of individuals will not be released, identifying information will be stored separately from the survey responses, and all information collected will be analyzed in the aggregate). The e-mail will be sent from an NIH address, and letters will be sent on NIH letterhead.
The letter will also discuss the conferment of a monetary incentive or honorarium that sufficiently acknowledges the respondent’s time and cooperation. The incentive is designed to be conferred immediately upon completion of the survey; which is expected to improve completion rates. An incentive of $25 will be conferred. A full discussion of how the incentive amount was determined may be found in Section A.9.
The recruitment letter or email will also enclose or attach a letter from an organization known to command respect among organizations that provide HIV/AIDS services among communities highly impacted by that disease. The letter will encourage participation in the survey by explaining how the information will be used to benefit the community. A copy of the letter may be found in Attachment J.
Online administration of the survey is expected to greatly increase the ease of data collection for these sampled persons, who are expected to be computer literate. Importantly, the incentive for completion of the survey can be released electronically as soon as the survey is completed. The incentive code can be redeemed online immediately at Amazon.com. We expect that gaining immediate access to the incentive rather than waiting for a mailed check will prove attractive to many participants.
The strategy for telephone follow up has been carefully designed and will be staffed with callers trained in refusal conversion. By following up consistently and persistently, NIAID will demonstrate that it is committing time and energy to obtain the most valid data possible by obtaining the opinions of as many persons as possible. The contractor will use refusal avoidance methods during all communications to lessen the need for refusal conversion. For those who do refuse, an experienced refusal conversion interviewer will attempt to collect responses on the questions deemed most critical.
Survey staff will work with the respondent to obtain data in whatever manner is convenient to the participant. Hard copies of the survey will be sent if so required, and the respondent will have the option of calling the contractor to respond to the survey by telephone if necessary.
Consistent with the response rate calculations approved by the American Association for Public Opinion Research (AAPOR), response rates for this study will be calculated as follows:
Number of Completed Surveys
Number of Completed Surveys + Number of Nonrespondents
B.4 Test of Procedures or Methods to Be Undertaken
When constructing the survey instrument, items used previously in other surveys by other NIH Institutes and Centers or organizations were carefully evaluated for inclusion. The survey instrument was tested with cognitive interviews with nine respondents from organizations similar to the ones that will provide respondents for the survey. In response to their comments, questions were revised, dropped or combined, response categories were added to several items, and several small wording changes were made.
A pre-test of sampling procedures was conducted with ten organizations through searches of Web sites and publicly available materials. (No individuals were actually contacted). For the five local and national organizations, we were able to identify board members and program directors, but the names of outreach workers could not reliably be obtained online. Therefore, the survey was revised to enable collection of outreach workers’ names from program directors. Of identified potential respondents, we were able to obtain e-mails for about half of them; addresses for all organizations were available online. Thus, a high proportion of potential respondents will need to be contacted by mail.
A trial of the matching process for five National Partners found that most HIV/AIDS organizations working at the national level have had some contact with NHVREI through a variety of different channels. For this reason, organizations in the matched “No/Minimal Contact” group will limited to organizations focused on local communities.
B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
The contractor collecting and analyzing information for the NHVREI will be NOVA Research Company (NOVA). Responsibility for collecting and analyzing information obtained through the methodologies described above will rest with NOVA. All data collection and analysis will be performed in compliance with OMB, Privacy Act, and Protection of Human Subjects requirements.
1 Singer E and Kulka RA. Paying respondents for survey participation. In Ver Ploeg M. Moffitt RA, Citro CF (eds). Studies of Welfare Populations: Data collection and Research Issues. National Academy Press: Washington, DC 2001. Available at http://aspe.hhs.gov/hsp/welf-res-data-issues02/04/04.htm. Accessed on April 15, 2009.
2 Singer E. The use of incentives to reduce nonresponse in household surveys. The University of Michigan Institute for Social Research Survey Research Center. Available at http://www.isr.umich.edu/src/smp/Electronic Copies/51-Draft106.pdf. Accessed On April 15, 2009.
3 Goritz, AS. Incentives in web surveys: Methodological issues and a review. International Journal of Internet Science, 2006, 1(1), 58-70.
4 Kovac MD, Markesich J. Tiered incentive payments: Getting the most bang for your buck. Presentation at the Annual Conference of the American Association for Public Research, 2002.
5 Church, A.H. (1993). Estimating the effect of incentives on mail survey response rates: A meta-analysis. Public Opinion Quarterly, 57, 62-79.
6 U.S. Dept. of Labor, Bureau of Labor Statistics. May 2007 National Occupational Employment and Wage Estimates, United States. Accessed on May 27, 2008 at http://www.bls.gov/oes/current/oes_nat.htm#b11-0000.
| File Type | application/msword |
| File Title | NHVREI clearance package |
| File Modified | 2010-01-20 |
| File Created | 2010-01-14 |