WISEWOMAN National Program Evaluation
Supporting Statement
Part B: Statistical Methods
April 9, 2019
Contact: Joanna Elmi
Division of Heart Disease and Stroke Prevention
Centers for Disease Control and Prevention
Atlanta, Georgia
Email address: [email protected]
This page left blank for double-sided copying.
CONTENTS
B. Statistical Methods 1
1. Respondent Universe and Sampling Methods 1
2. Procedures for the Collection of Information 2
3. Methods to Maximize Response Rates and Deal with Nonresponse 4
4. Test of Procedures or Methods to be Undertaken 5
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 5
This page left blank for double-sided copying.
LIST OF ATTACHMENTS
Attachment A: BREAST AND CERVICAL CANCER MORTALITY PREVENTION ACT OF 1990
Attachment B: PUBLIC HEALTH SERVICE ACT
Attachment C: PROGRAM Survey INSTRUMENT AND SUPPLEMENTARY DOCUMENTS
C1: PROGRAM SURVEY
C2: PROGRAM SURVEY INVITATION EMAIL
C3: PROGRAM SURVEY REMINDER EMAIL
Attachment D: Site Visit data collection instruments AND SUPPLEMENTARY DOCUMENTS
D1: DISCUSSION GUIDE- INTERVIEW WITH KEY ADMINISTRATIVE STAFF
D2: DISCUSSION GUIDE- INTERVIEW WITH HEALTHY BEHAVIOR SUPPORT STAFF
D3: DISCUSSION GUIDE- INTERVIEW WITH STAFF AT PARTNER CLINICAL PROVIDERS
D4: DISCUSSION GUIDE- COMMUNITY PARTNERS
D5: DISCUSSION GUIDE – INNOVATION SITE VISIT
ATTACHMENT E: CONSENT FORM
Attachment E: Federal Register Notice
Attachment G: summary of public comments
This page left blank for double-sided copying.
TABLES
B.1 Summary of the WISEWOMAN data collection activities 1
B.2 Individuals consulted on methods 7
This page left blank for double-sided copying.
Table B.1 summarizes the potential respondent universe, targeted respondents, and methods for selecting respondents for the three proposed new data collection activities. Below, we discuss the targeted respondents and methods in more detail for each data collection activity.
Table B.1. Summary of the WISEWOMAN data collection activities
Data collection method |
Respondent universe |
Targeted respondents |
Methods for Selection |
Program survey |
All 24 WISEWOMAN recipients |
All 24 WISEWOMAN recipients (program directors, program managers, and data managers may collaborate to complete sections of the survey) |
Not applicable – All recipients will be asked to complete the survey. |
Site visits |
All 24 WISEWOMAN recipients and their providers and partners |
Program staff and partners across all 24 WISEWOMAN recipient sites (7 respondents per site) |
All recipients will receive one site visit. Program directors at each site will be asked to help identify key program staff and partners that are best suited to provide the information sought through the site visits. Seven recipients that are receiving additional CDC funding to implement an innovative approach to delivering WISEWOMAN services will receive a set of supplemental interviews during the site visit to collect additional data about the use and effectiveness of the innovation strategy. |
Attachment B includes the program survey instrument. Attachment C includes the site visit data collection instruments.
All WISEWOMAN recipients will be asked to complete the program survey to ensure that consistent information is collected about implementation across the recipients and to enable analysis of variation in implementation to contribute to the summative evaluation. There are 24 WISEWOMAN recipients (21 states and 3 tribal organizations) with five-year cooperative agreements from 2018 to 2023. Program directors, program managers/coordinators, and data managers may collaborate to complete sections of the survey. The program survey will be self-administered through an editable PDF. Data collection is anticipated for Program Years 2 and 4 to capture early and mature implementation. A pre-test of the program survey instrument will be conducted with a small subset of recipients in Program Year 1 to assess respondents’ understanding of new and revised survey items.
We propose to conduct 24 site visits total, with five to seven site visits conducted per program year in each of Program Years 2 (six visits), 3 (six visits), 4 (seven visits), and 5 (five visits). All WISEWOMAN recipients will be visited at least once during the three-year data collection period to allow for collection of in-depth information about each recipient’s experience with program implementation, as recipients’ experiences may vary.
The order of site visits in Program Years 2, 3, and 5 will be selected based on emerging and best practices in prioritized areas for evaluation as identified through the program survey, recipient reports, and CDC project officer interactions, as well as an effort to visit a mix of new and experienced recipients during each program year. The seven recipients that are receiving additional funding to implement an innovative strategy to improve cardiovascular health will receive a site visit in Program Year 4 to allow sufficient time for recipients to implement these strategies and assess their effectiveness.
During the site visits, in-person interviews will be conducted with key program staff (for example, program directors, program managers, data managers, and lifestyle programs (LSP) and health coaching (HC) coordinators) and their partners (for example, clinical and LSP/HC providers, American Heart Association, quitline, and community-based organizations). The program directors at each site will be asked to help identify key program staff and partners that are best suited to provide the information sought through the site visits.
We anticipate conducting seven interviews per site: one administrator; two additional key program staff; two providers; and two partners. This target would result in 189 total interviews in three years across the 24 sites between Program Years 2 and 5. The number of key informant interviews that can ultimately be scheduled per site within the allotted time will depend on the availability of respondents, project resources, and the amount of travel time required between interviews.
Each component of the evaluation is unique, and as such, the procedures will be described separately below.
No statistical methods will be used to draw a sample for this survey, as it will include the full universe of all 24 WISEWOMAN recipients. The survey will be administered twice, in Program Years 2 and 4, using the same questionnaire and instrument in each round of administration. The second round of administration will be used to measure change across the variables of interest. The survey will be conducted by CDC’s evaluation contractor.
This survey will be conducted using an editable PDF, in a self-administered format sent by email. The survey will take approximately one hour to complete. Recipient program managers are the target respondents, but they may choose to delegate sections of the survey to other knowledgeable staff at their discretion. Respondents will be provided with detailed instructions on how to complete the questionnaire, both in the beginning of the instrument and through visual cues throughout. CDC’s evaluation contractor will provide a toll-free telephone number, as well as a dedicated email address for technical assistance for the project, including associated data collection. This information will be printed on the questionnaire. Trained project staff will respond to any questions that potential respondents have about specific items in the questionnaire or about participation overall.
The 8-week field period will launch with an invitation email including the editable pdf (Attachment B.2). All non-responders will receive up to 4 reminders via email (Attachment B.3) (sent weeks 2-8) and reminder telephone calls placed by trained project staff (weeks 3 - 8). These staff will confirm the contact information on record, ensure the survey was received by email, and encourage participation. These staff will address any concerns raised about participation in the survey, both through non-response follow-up, as well as through responding to inquiries from the study’s toll-free telephone number and email address. Based on response to past rounds of the WISEWOMAN program survey (under OMB#0920-1068), a response rate of 100 percent is anticipated for this effort.
As completed surveys are returned, they will be reviewed for completeness. If a completed questionnaire is missing critical items, or if other questions arise in the review, the contractor will reach out to respondents for clarification. After roughly half of the recipients have returned completed questionnaires, CDC’s contractor will review the existing data and conduct an initial data cleaning. This step will help identify potential issues for cleaning and analysis early so that strategies may be employed to correct these issues. Data will be cleaned based on specifications developed by CDC and its evaluation contractor, and the final data file will be used for analysis.
Site visits will be conducted with each of the 24 WISEWOMAN recipients. These visits will take place across three calendar years in Program Years 2, 3, 4, and 5, with six visits occurring per year in Program Years 2 and 3, seven visits occurring in Program Year 4, and five visits occurring in Program Year 5. No statistical methods will be used to identify the recipients selected for the site visits as all recipients will receive one site visit during the course of the evaluation.
Lead administrators at each site will work with CDC’s contractor to identify optimal dates and times for the visits. Furthermore, they will notify applicable staff that the contractor will be reaching out to set up appointments. Prior to each visit, all staff identified as target site visit respondents will be contacted by email or telephone to set up the appointment. All interviews will be in person.
We will conduct a standard set of interviews with the following seven respondents at each WISEWOMAN site: WISEWOMAN administrative staff (90 minutes, with one interview conducted per site); two partner organization administrators (60 minutes); two of the recipient organization’s “healthy behavior support” staff (60 minutes); and two WISEWOMAN health care providers (60 minutes). In addition, interviews with some respondents at the seven site visits conducted during Program Year 4 with innovation funding recipients will include a supplemental set of questions about recipients’ innovative strategies to deliver WISEWOMAN services. The supplemental questions will take up to 45 minutes per respondent and will be administered to: the program administrator (for a total of 135 minutes of data collection, including 90 minutes for the standard protocol questions); one additional healthy behavior support staff member; one additional medical provider; and one additional partner organization staff member. We anticipate an average of 10 interviews per site for the seven innovation recipients. In total, we expect to conduct 189 interviews across the 24 WISEWOMAN sites.
At each site, we will attempt to schedule interviews to take place over a day and a half and within regular work hours. Site visits at each of the seven recipients visited in Program Year 4 may last up to two days to allow time for administration of the supplemental innovation questions with the program administrator and additional healthy behavior support staff, medical providers, and partner organization staff. A member of CDC’s evaluation contractor team will lead the interviews. Interviews will be audio recorded, if key informants agree. After the visit is completed, audio recordings will be transcribed and then coded for analysis.
The data collection procedures discussed below are designed to maximize response rates and to promote the accuracy and completeness of information collected across each component of the evaluation.
The program survey is anticipated to have minimal non-response because of the value recipients place on the evaluation and its role in helping them serve the populations of interest. Because participation in the survey is seen as a value-added endeavor, non-response is less likely to stem from unwillingness or not recognizing the value of participation, but rather finding the time to complete the questionnaire amidst a heavy workload carried by program administrators. Because the questionnaires are designed in a self-administered format, they can be completed across multiple sessions, as time permits. As noted in section B2, CDC’s evaluation contractor will use highly trained staff to reach out to non-responders by telephone to address any barriers to participation. In addition, all correspondence will include the email and telephone number for a help desk provided to answer any questions that respondents may have throughout the field period.
To collect high quality data in each of the surveys, the following components are included as part of the planned data collection:
Review and follow-up (as needed) for submitted program survey questionnaires to ensure high quality data: Reviewing each completed questionnaire to ensure: 1) the returned questionnaire meets criteria established to be designated as completed; 2) responses are provided for all critical variables; 3) skip patterns are followed; 4) responses fall within allowable ranges.
Reviewing data frequencies. Frequency reviews are an important tool in ensuring data quality. To determine whether the instrument is performing as specified, frequencies will be reviewed early in the field period. If missing data need to be retrieved from respondents, staff will be instructed to follow up and obtain the information.
Minimizing non-response bias. We anticipate minimal non-response to the survey, as participation is integral to their work in implementing WISEWOMAN program activities. When a similar program survey was administered to WISEWOMAN recipients in spring 2018, a 100% response rate was achieved.
Response to the site visits is expected to be high because of the interest of WISEWOMAN recipients in the evaluation. For the site visits, interviews will be scheduled at the convenience of the key informant. We anticipate at least 95 percent of the key informants will complete interviews during the site visits, based on experience with similar activities and a typically high level of motivation from the WISEWOMAN staff and their partners. The notes are then enhanced through the use of transcriptions from the recordings, which will allow the interviewer to engage actively with the respondent without having to take notes and the transcriptions can then be coded and analyzed.
The program survey was adapted from an earlier version of the WISEWOMAN program survey administered to recipient administrators in 2015 and 2018 (OMB #0920-1068). Fifteen questions were added, 13 questions were revised, and 21 questions were deleted. New and revised items will be pretested with a convenience sample of administrators from three recipients. Because the universe for the survey is small, the pre-test sample will also be small to minimize burden for the pre-test respondents (who will also complete the questionnaire during the evaluation itself). Pre-test respondents will complete the revised questions on paper, with a debriefing conducted by a professionally trained member of the team at CDC’s evaluation contractor for the data collection effort. Pre-test respondents will be asked to provide feedback on any questions they felt were confusing, overly burdensome, or difficult to provide accurate responses to without excessive burden. We will revise survey items for clarity based on pre-test respondents’ suggestions.
The site visit data collection instruments will not undergo any testing prior to data collection as they are more qualitative in nature and the interviewer can adjust the data collection instrument to meet the time limits of the interview. The site visit instruments were also adapted from those used in 2015, 2016, and 2018 to conduct semi-structured interviews with a similar set of respondents at 18 recipient programs (OMB #0920-1068).
CDC staff from the WISEWOMAN program and the Applied Research and Evaluation Branch (AREB) and staff from Mathematica Policy Research were consulted about the substantive, methodological, and statistical aspects of the study. Their recommendations were incorporated into the study design and instruments on an ongoing basis. Table B.2 lists the individuals consulted.
Staff members from Mathematica designed the data collection tools with input from CDC. Mathematica staff will also be responsible for overseeing and executing the collection of data with input from CDC staff. The data analysis will be led by Mathematica in close consultation with CDC staff throughout the process. GDIT will provide quality assurance on all evaluation products. Tiffany Dearman, Julia Jordan, and John Whitehill are responsible for receiving and approving contract deliverables.
Table B.2. Individuals consulted on methods
Julia Jordan, Health Scientist IHRC, Inc. Division for Heart Disease and Stroke Prevention 770-488-1053 |
John Whitehill, Health Scientist Division for Heart Disease and Stroke Prevention 770-488-8195 |
Isam
Vaid, Health Scientist |
Cagney Stigger Deloitte Consulting, LLP Division for Heart Disease and Stroke Prevention 770-488-7743 |
So
O’Neil, Senior Researcher |
David
Jones, Senior Researcher |
Michaela
Vine, Senior Researcher |
Rachel
Kogan, Survey Researcher |
Margaret Barton, Senior Policy Analyst General Dynamics Information Technology 703-980-9891 |
Katie Morrison Lee, Researcher Mathematica Policy Research 908-872-9322 |
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Emily Wharton |
| File Modified | 0000-00-00 |
| File Created | 2021-01-15 |