August 2016
This page left blank for double-sided copying.
OMB Part B: Collection of Information Employing Statistical Methods 1
1. Respondent universe and sampling 3
2. Procedures for the collection of information 8
3. Methods to maximize response rates and deal with nonresponse 16
4. Test of procedures or methods to be undertaken 18
5. Individuals consulted on statistical aspects and/or analyzing data 19
APPENDIX A: TELEPHONE SURVEY
APPENDIX B: KEY INFORMANT INTERVIEWS
APPENDIX C: FOCUS GROUPS
APPENDIX D: VOXIVA TEXT4BABY ADMINISTRATIVE DATA
APPENDIX E: STATE MEDICAID DATA
APPENDIX F: STATE VITAL STATISTICS DATA
This page left blank for double-sided copying.
The Center for Medicaid & CHIP Services (CMCS), a division of the Centers for Medicare & Medicaid Services (CMS), is supporting a pilot mobile health (mHealth) program in California, Louisiana, Ohio, and Oklahoma. The three-year mHealth project is being conducted to complement existing CMCS measurement, data collection, and reporting activities to monitor, track, and assess state’s maternal and infant health efforts in Medicaid and CHIP populations. This project focuses on developing key measures of maternal and infant child health (MCH) in the Medicaid population. This information will be used in monitoring performance of the participating states on the Maternity Core Set measures and in updating the Adult Core Set annually as required by Section 2701 of the Patient Protection and Affordable Care Act (ACA). The data collection and synthesis methods and measurement will be tested with state Medicaid agencies within the four participating states.
The project also involves the implementation of a customized version of Text4baby within the four states to assess the role of health education, knowledge, and behavior play in observed perinatal health outcomes. Text4baby is provided free to those who enroll in the service via text message or through the Text4baby website. The service is overseen by Voxiva, an mHealth and wellness company. Zero to Three developed more than 250 messages for the Text4baby service that contain critical information for pregnant women and new mothers with infants younger than one year. The messages contain brief, interactive information encouraging healthy behaviors and share ways to improve maternal and child health.
The unique feature of the pilot mHealth program is its education component delivered through Text4baby. Despite the interest in and potential benefits of mHealth programs, their effect on health knowledge, behavior, and outcomes has been inconclusive. Several studies have demonstrated mHealth’s potential to prompt behavior changes related to use of oral contraceptives, physical activity, and smoking cessation, while others have shown no effect or inconclusive results.1,2,3 Studies specific to Text4baby have also been limited. One study showed small effects on knowledge and beliefs related to preparedness for motherhood, although the effects on behavior change were not conclusive.4 Although several studies have focused on socioeconomically disadvantaged populations, none have concentrated specifically on the effect of Text4baby in the Medicaid population.5,6 The limited evidence in these studies highlights the need to further assess key implementation factors related to outreach, participation, and engagement, which might explain variation in outcomes, especially among disadvantaged populations that might benefit the most from mHealth interventions. Given the limited information available on the effect of Text4baby and the underlying factors that could drive these effects, especially in the Medicaid population, CMCS has funded an evaluation of the pilot mHealth program to further assess the potential of mHealth programs to improve maternal and infant health among Medicaid enrollees. Most importantly, the data collection and analysis methods from this study will be used to track performance of participating states on key prioritized measures for CMCS’ initiatives related to maternal and infant health and the methods developed under the evaluation can be used by other states to conduct similar monitoring and evaluation.
Data for the evaluation will come from several primary and secondary data sources.7 All primary and secondary data for this evaluation will be collected and analyzed by CMCS’s contractor. Data will remain with the contractor until the end of the study, at which time the contractor will destroy data as specified in data use agreements and institutional review board clearances. The three primary data sources for which CMCS seeks Office of Management and Budget (OMB) clearance include: a postpartum telephone survey of Medicaid-enrolled pilot program participants and nonparticipants in the four pilot states (see Appendix A for all materials related to the telephone survey); key informant interviews with CMCS, Voxiva, Zero to Three, state Medicaid agencies, health care providers, and outreach partners (see Appendix B for all materials related to key informant interviews); and focus groups with Medicaid-enrolled pilot participants (see Appendix C for all materials related to focus groups).
The three primary data collection components are summarized next:
Medicaid postpartum telephone survey. Approximately 2,400 postpartum Medicaid-enrolled pilot program participants and nonparticipants, or about 600 women in each of the four states participating in the pilot, will be selected to participate in the telephone survey. To be eligible for the survey, women must have a delivery paid for by Medicaid/CHIP or an infant with Medicaid coverage at birth. Ideally, women will be less than 12 months postpartum during the survey field period. This will be a one-time data collection.
Key informant interviews. CMCS’s contractor will conduct up to 15 telephone interviews with key informants from CMCS (one interview), Voxiva and Zero to Three (one interview from each organization), state Medicaid agencies (one interview per state), and state and local outreach partner organizations (up to two interviews per state).
Site visits to each participating state. The contractor will conduct one round of 1.5-day site visits to each of the four states participating in the pilot. For each site visit, the contractor will select a single geographic area in each state that has a high concentration of Medicaid providers. As part of each site visit, the contractor will conduct two interviews with different providers that deliver care to pregnant and postpartum women enrolled in Medicaid and that have knowledge of the pilot program, and two focus groups with postpartum Medicaid pilot program participants.
Key informant interviews with health care providers. During the site visits to each state, the contractor will conduct up to 8 total in-person interviews (2 health care provider interviews * 4 states) with different health care providers that deliver care to pregnant and postpartum women enrolled in Medicaid and that have knowledge of the pilot program.
Focus groups. The contractor will work with outreach partners serving low-income pregnant women and new mothers to identify a subset of postpartum Medicaid-enrolled pilot program participants to participate in focus groups during the site visits to the four states participating in the pilot. The contractor will conduct two focus groups with pilot program participants during each site visit for a total of 8 focus groups (2 focus groups * 4 states). To be eligible to participate in a focus group, women must have a delivery paid for by Medicaid/CHIP or an infant with Medicaid coverage at birth, less than 12 months postpartum during the scheduled period for the site visit, and must not have participated in the telephone survey.
Medicaid postpartum telephone survey. The potential respondent universe will include women with a birth financed by Medicaid who will be less than 12 months postpartum during the survey field period in each of the four participating states. The sample will be stratified by pilot program participation status. Table 1 shows the estimated number of Medicaid Text4baby subscribers and nonsubscribers by state. The estimate of the potential respondent universe was calculated using Voxiva administrative data and estimates of Medicaid-covered births by state for a six-month postpartum period.8,9 The estimates are based on a six-month postpartum period because it is unlikely women less than 6 months postpartum could be sampled for the survey due to the lag in the availability of Medicaid claims data. CMCS’s contractor expects that the number of pilot program participants in each state will actually be higher than the estimates presented in Table 1 by the time the survey will be fielded in 2017 for two reasons: (1) the response rate on the Text4baby insurance surveys which ask the subscriber the type of insurance she currently has are relatively low and the contractor might be able to identify additional Text4baby subscribers by telephone number when the contractor matches Voxiva and Medicaid data; and (2) these estimates are based on 2014 Voxiva data and the number of Text4baby subscribers is growing. By the time the survey is ready to be fielded in 2017, the contractor expects the number of pilot program participants in each state to be larger.
Table 1. Potential respondent universe
Universe |
Number |
Estimate of Text4baby subscribers enrolled in Medicaid (6-month postpartum period) |
|
California |
900 - 1,100 |
Louisiana |
550 - 600 |
Ohio |
300 - 350 |
Oklahoma |
2,000 - 2,500 |
Estimate of Text4baby nonsubscribers enrolled in Medicaid (6-month postpartum period) |
|
California |
120,300 - 125,100 |
Louisiana |
21,000 - 22,150 |
Ohio |
26,250 - 30,300 |
Oklahoma |
14,100 - 14,500 |
The contractor will select approximately 2,400 postpartum Medicaid–enrolled pilot participants and nonparticipants, or about 600 women in each of the four states participating in the pilot, to participate in the Medicaid postpartum survey. To be eligible for the survey, women must have a delivery paid for by Medicaid or an infant with Medicaid coverage at birth. Ideally, the contractor will draw the sample such that women will be less than 12 months postpartum during the survey field period.
Given the estimated level of penetration of the pilot program, the contractor plans to oversample pilot program participants to arrive at a sample of 300 women who enrolled in the pilot program during their most recent pregnancy, and 300 nonparticipants in each state (see Table 2 for sample sizes by strata). Based on experience with previous surveys of similar populations, the evaluation estimates a 65 percent response rate in this population, for a total of 1,560 completed surveys across the four states.10
Table 2. Sample sizes by strata
Sampling Strata |
Sample Size |
Target
Completed |
Pilot program participants |
1,200 |
780 |
California |
300 |
195 |
Louisiana |
300 |
195 |
Ohio |
300 |
195 |
Oklahoma |
300 |
195 |
Pilot program nonparticipants |
1,200 |
780 |
California |
300 |
195 |
Louisiana |
300 |
195 |
Ohio |
300 |
195 |
Oklahoma |
300 |
195 |
CMCS’s contractor plans to identify pilot program participants by merging Voxiva and Medicaid data and matching on telephone number. Nonparticipants will be identified in Medicaid data and will not have a matching record in the Voxiva data.11 The contractor will use matching techniques to select nonparticipants who are similar to subscribers in terms of characteristics available in the Medicaid data. See Section 2.1 for a detailed description of the statistical methodology for stratification and sample selection, including potential characteristics for use in the matching techniques.
Key informant interviews. The contractor will conduct key informant interviews across a number of different types of agencies (see Table 3). The contractor plans to conduct one interview with CMCS, Voxiva, Zero to Three, and each participating state Medicaid agency. The CMCS, Voxiva, Zero to Three, and state Medicaid agency interviews will cover the entire potential respondent universe. In addition, the contractor plans to conduct up to two interviews per state with different state and local pilot program outreach partners for a total 8 interviews. The contractor also plans to conduct up to two interviews per state with different health care providers who deliver care to pregnant and postpartum women enrolled in Medicaid and who have knowledge of the pilot program for a total of 8 interviews.
Table 3. Potential respondent universe for key informant interviews
Respondent type |
Potential
|
Number
of |
Interviews
per |
CMCS |
1 |
1 |
1 |
Voxiva |
1 |
1 |
1 |
Zero to Three |
1 |
1 |
1 |
State Medicaid agencies |
4 |
4 |
1 |
Outreach partners – Californiaa |
44 |
2 |
1 |
Outreach partners – Louisianaa |
13 |
2 |
1 |
Outreach partners – Ohioa |
18 |
2 |
1 |
Outreach partners – Oklahomaa |
43 |
2 |
1 |
Health care providersb |
4,258 |
8 |
1 |
a Information obtained from Voxiva. To be counted, outreach partners must have corresponded with the Text4baby outreach team since 2014 (when the pilot began).
b The contractor estimated numbers using only OB/GYN providers and that half of all active physicians accept Medicaid patients. This represents a lower bound as some women may see other types of providers and more providers may accept Medicaid than estimated. Sources used for the estimate: Kaiser Family Foundation, “Primary Care Physicians by Field,” http://kff.org/other/state-indicator/primary-care-physicians-by-field/. Baugh, D. and Verghese, S. “Physician Service Use and Participation in Medicaid, 2009.” Mathematica Policy Research: October 2012. http://www.cms.gov/Research-Statistics-Data-and-Systems/Computer-Data-and-Systems/MedicaidDataSourcesGenInfo/Downloads/MAX_IB11_PhysicianParticipation.pdf.
To identify outreach partners for the key informant interviews, the contractor will work with Voxiva, Zero to Three, and each state Medicaid agency to identify a key informant from two outreach partners that have been active at some point since 2014, which is when the pilot began. To identify health care providers for the key informant interviews, the contractor will work closely with state Medicaid agencies to identify health care provider organizations that largely serve Medicaid populations and are aware of the pilot program. The contractor will attempt to recruit the providers that are most knowledgeable and active in regard to the pilot mHealth program in their state in each selected geographic area. The contractor will also conduct online searches to see if the provider has an initiative related to the pilot mHealth program. The contractor will ask Medicaid agencies to provide entrée to health care organizations and will call health organization medical directors to determine individual providers’ eligibility.
The contractor will work with each agency identified for a key informant interview to identify the individual or individuals best suited to participate in the interview. In all stakeholder groups, the contractor will seek to interview people who are actively engaged in the implementation of the pilot mHealth program and have in-depth knowledge of the pilot within their organization.
The contractor will call identified key informants to recruit them to participate and to arrange for a time to complete the interview via telephone. The contractor will obtain the respondents’ verbal consent to participate in the key informant interview, stressing that they will not be identified by name in any reports. A response rate of 95 percent is expected for the key informant interviews based on previous experience conducting interviews with Text4baby outreach partners (OMB Control No: 0915-0347; Expiration Date: 02/28/2015) and a typically high level of motivation from the organizations involved.
Focus groups. To be eligible to participate in a focus group, women must have a delivery paid for by Medicaid or an infant with Medicaid coverage at birth, ideally be less than 12 months postpartum during the data collection period for the focus group, and must not have participated in the Medicaid postpartum telephone survey.
To recruit pilot program participants, CMCS’s contractor will ask Voxiva and Zero to Three to send a series of text messages to subscribers in specified zip codes. The text message will ask participants if they are interested in participating in a focus group and give them a number to call.12 When women call in after receiving a text message from Voxiva about the focus group, project staff will screen them for eligibility13 and availability for the focus groups (see Appendix C for the screening protocol). In anticipation of potential no-shows, the contractor will recruit approximately 20 women for each focus group to ensure each focus group will have 10 to 12 postpartum pilot program participants. Thus, the contractor expects a 50 to 60 percent response rate among women recruited to participate in the focus groups. In total across the four states, the contractor will conduct 8 focus groups with an estimated number of participants of 80 to 96.
Contacting sample members. After merging the Voxiva and Medicaid data files using the contact information in the Medicaid files to contact sample members, the contractor will send sample members a letter that explains that they have been selected to participate in this evaluation, and informs them that an interviewer call them soon to complete the survey (see Appendix A).14 The letter will also include a toll-free number that sample members can use to call to complete the survey or to ask questions about the evaluation, and it will mention the $25 gift card they will receive for completing the survey. About one week after the letters have been sent, trained interviewers will begin calling respondents to obtain verbal consent and conduct the 25-minute interview.
Programming the survey and managing the sample. The survey instrument (see Appendix A) will be programmed into Blaise software and will be administered using computer assisted telephone interviewing (CATI) technology. The survey will be translated into Spanish. The instrument will undergo extensive testing by programmers, survey managers, and CMCS. Training materials for telephone interviewers will be finalized after survey testing in Blaise has been completed. In addition, proprietary sample management software (SMS) will be used to manage the sample. The SMS will track the status of cases as they are called, scheduled, and interviewed, and will be the source reports used to track survey completion rates.
Training telephone interviewers. The basis of interviewer training will be a training manual. The core sections of the manual will include background information on the evaluation, question-by-question specifications, basic respondent interaction techniques, and strategies for averting refusals. Interviewers will receive six hours of training. The training manual will serve as a day-to-day reference for the interviewers’ use during data collection.
Collecting the information. To align with the languages in which the pilot mHealth program is available, the survey will be conducted in English and Spanish. The contractor will mail a $25 gift card to all sample members who complete the survey. The survey field period will last approximately three months, and the full sample will be released for calling at the beginning of the field period.15
When a sample member is reached by telephone or calls to complete the survey, the interviewer will briefly explain that the contractor is conducting a survey for new mothers and ask her to answer a few questions to determine her eligibility and that the interviewer has reached the correct person. The screening questions will determine whether the mother meets eligibility criteria related to being postpartum and having a Medicaid-covered birth. If any of her answers to the screening questions indicate that she is ineligible to participate in the study, the interviewer will inform her that she is not eligible, thank her for her time, and end the call. If she is eligible to participate, the interviewer will describe the purpose of the study and obtain her verbal consent to participate. The consent process will emphasize that participation is voluntary and that her name and responses to survey questions will be kept private. For unemancipated minors, the interviewer will also obtain verbal consent from her parent or guardian.
Trained telephone interviewers will administer the survey to respondents using CATI. Interviewers will attempt to reach respondents to complete the survey at different times of the day and in the evening, and on varying days of the week. After 10 unsuccessful attempts to reach the respondent, the case will be sent to our in-house locating department to search for updated contact information. Cases with incorrect telephone numbers will be sent to locating immediately. Locators will use a variety of tools to search for updated contact information for sample members. Locators may use database searches such as Accurint, Internet searches, telephone 411 reach searches, and social media searches to try to find updated contact information for sample members.
Monitoring telephone interviewers. In order to ensure the collection of high quality data, an important part of data collection involves monitoring the work of the telephone interviewers. During the first week of interviewing, the first interview completed by each interviewer will be monitored. After that, 10 percent of all interviews will be monitored. Monitors provide feedback to interviewers about their performance immediately after the interview. If the monitors observe any areas of weakness in terms of performance, they provide specific suggestions to interviewers about how to improve their performance moving forward.
Daily monitoring of survey response. During the field period, survey staff will generate and examine daily reports on survey response rates. Reports will be broken down by state and pilot program participation status (participant or nonparticipant). These reports provide the information necessary to take corrective action when needed—for example, by identifying patterns of nonresponse among different types of sample members.
Cleaning the data. At the end of the survey field period, collected data will be cleaned based on specifications developed by contractor programming and survey staff. The final clean data file will be used by the contractor for analysis.
Two-person interview teams will conduct all key informant interviews. The team will include a senior person to lead the interviews and a junior person to help schedule and facilitate interviews and take detailed notes. CMCS’s contractor will schedule key informant interviews at a time that is most convenient for the respondent. The contractor will audio-record the interviews if respondents agree. Recorded interviews will be transcribed to facilitate analysis.
For convenience, the contractor will conduct interviews with health care providers in person at the provider’s location. To facilitate the conversation, project staff will follow the semi-structured interview guide developed for these interviews (see Appendix B). The interviews will last about 30 minutes each.
All other key informant interviews (that is, CMCS, Voxiva, Zero to Three, state Medicaid agencies, and active and inactive outreach partners) will be conducted by telephone. To facilitate the conversation, project staff will follow the semi-structured interview guide developed for each type of key informant (see Appendix B). Each interview will last about 60 minutes.
To the extent possible, the contractor will conduct the focus groups at a location that is central and convenient to most of the focus group participants. The contractor will ensure that the space is private (such as an enclosed conference room) to maintain confidentiality and minimize distractions. A contractor staff member will make a reminder call and/or send emails to recruited focus group participants close to the date of the focus groups to remind them of the date, time, and location of the focus group (see Appendix C). Focus group participants will receive a $25 gift card at the end of the session to thank them and to defray transportation and child care costs.
Obtaining consent from focus group participants. At the beginning of the focus group session, women will be asked to sign a consent form. The consent form will describe the purpose of the focus group, focus group procedures, benefits and risks, and the confidentiality of the information that they provide, and it will stress the voluntary nature of participation. The consent form will also include contact information for the project director and an institutional review board (IRB) contact should the participant have additional questions in the future. Participants will sign two copies of the consent form: the project team will retain one and give the other copy to the participant.
Conducting the focus group sessions. Each focus group will be staffed with a moderator and facilitator. The facilitator will be responsible for intake, processing gift cards, welcoming late arrivals, recording the discussion, and taking notes. The moderator will lead the group discussion, ensuring that all participants have an opportunity to speak, drawing out those who are reticent, and cueing participants to share the diversity and similarity of their experiences.
Each focus group session will last about 100 minutes. For the first 20 minutes of the session, project staff will conduct intake and ask participants to fill out a Participant Information Form (PIF), which will ask for basic demographic information, pregnancy history, and pregnancy and postpartum health behaviors (see Appendix C). The contractor will use the PIF to create a demographic profile of focus group participants in order to add context to the analysis. During the rest of the focus group session, the moderator will use the appropriate discussion guide to lead the discussion (see Appendix C). All focus group discussions will be recorded and transcribed for analysis.
Medicaid postpartum telephone survey. The survey will be conducted with a representative sample of women who participated in the pilot mHealth program and a sample of women who did not participate. The contractor will create the sampling frame for the survey from Medicaid administrative data. The frame will include women with prenatal care and/or deliveries that were covered by Medicaid, including women who participated in the pilot program and those who did not. The contractor will construct the sampling frame so that 50 percent of survey respondents will be a stratified random sample of women who participated in the pilot, and the other 50 percent of respondents will be a matched comparison group of women who did not participate in the pilot but have demographic and geographic characteristics similar to the respondents who did.16 The contractor plans to identify pilot program participants by merging Voxiva and Medicaid data and matching on telephone number. Nonparticipants will be identified in Medicaid data and will not have a matching record in the Voxiva data.
The pilot program participants will be stratified on demographic characteristics and their level of participation to ensure they are representative of the sampling frame. The contractor could oversample high-risk groups or minorities (the exact plan for stratification will be developed after the data becomes available). Limiting the comparison group to a matched subsample of women—closely matching observed characteristics of the treatment group—addresses selection bias by reducing differences between subscribers and nonsubscribers. It might also reduce differences in unobserved characteristics that affect outcomes if those characteristics are correlated with the matching variables. As mentioned previously, the contractor has identified an initial target sample size of 2,400 postpartum women, or 600 women in each state (300 pilot program participants and 300 nonparticipants).
To conduct the match, the contractor plans to use exact-matching methods, which are preferred to propensity-score based matching methods.17,18 Because these four states have a large number of Medicaid-covered births per year and, likely, a high ratio of potential comparison group women to treatment group women, the contractor expects to obtain perfect balance (or a high degree of balance) with the pilot program participants who were randomly selected for the survey—even with a large number of matching variables.
The contractor will match on individual-level characteristics derived from Medicaid enrollment data, Medicaid claims files, and the birth certificates (assuming Medicaid and vital records data are linked). The primary criterion for selecting matching variables is to use those associated with participation in the pilot program and the date of the pregnancy (to account for time trends). The face validity of the method will be strengthened by obtaining balance on variables that correlate with birth outcomes or indicate elevated risk of adverse birth outcomes, several of which are available in administrative data (for example, a woman’s age, family income, race/ethnicity, marital status, Medicaid eligibility category, singleton or multiple pregnancy, having a previous preterm birth or poor pregnancy outcomes, low pregnancy weight, and certain medical conditions such as substance abuse or renal failure). Table 4 shows a list of potential demographic matching variables and Table 5 shows a list of potential health matching variables.
Table 4. Demographic characteristics of interest from the vital records and Medicaid files
Characteristic |
Variables |
Data source |
Mother’s characteristics |
||
Agea |
≤ 18 years |
VR/MF |
Agea |
19–34 years |
VR/MF |
Agea |
≥ 35 years |
VR/MF |
Race |
White |
VR/MF |
Race |
Black |
VR/MF |
Race |
American Indian or Alaska Native |
VR/MF |
Race |
Asian or Pacific Islander |
VR/MF |
Race |
Other race |
VR/MF |
Race |
Multiple races |
VR/MF |
Ethnicity |
Hispanic |
VR/MF |
Foreign-born status |
Foreign-born |
VR |
Marital status |
Married |
VR |
Education |
Less than high school |
VR/MF |
Education |
High school or GED |
VR/MF |
Education |
Some college credit but no degree |
VR/MF |
Education |
College degreeb |
VR/MF |
Geographic location |
Geographic regionc |
VR/MF |
Geographic location |
Rural residence |
VR/MF |
Family incomea |
Income $0 |
MF |
Family incomea |
Income 1–100 percent of the FPL |
MF |
Family incomea |
Income > 100 percent of the FPL |
MF |
Aged, blind, and/or disabled |
Mother’s enrollment in Medicaid on the basis of being aged, blind, and/or disabled eligibility |
MF |
Medicaid managed care beneficiaryd |
Enrollment
in a Medicaid managed care plan |
MF |
Notes: The column on the right indicates the data source (VR and/or MF). Variables from the MF files will not be available for analysis of unlinked VR files, and vice versa. For some variables, a dummy variable for missing data might be included (not shown). Some variables might be unavailable on a state-by-state basis.
a Mother’s age and family income will be treated as categorical and as continuous variables. For example, family income will include four categories: $0, 1 to 100 percent of FPL, more than 100 percent of FPL, or missing. In addition, income will be included as a continuous measure (percentage FPL) in univariate analyses. In multivariable analyses, the continuous measures will be interacted with the categorical variables.
b This category includes women with at least a four-year college degree (bachelor’s, master’s, doctorate, and professional [M.D., D.D.S., D.V.M., J.D., and LL.B] degrees). Women with an associate’s degree are included in the “some college but no degree” category.
c See Exhibit V.C.1.
d The contractor will consider analyzing each managed care plan separately.
FPL = federal poverty level; GED = general educational development (diploma); MF = Medicaid files; VR = vital records files.
Table 5. Health risks and conditions from the vital records and Medicaid files
Characteristic |
Variables |
Data source |
Multiple birth |
Multiple birth (twins, triplets, and so on) |
VR/MF |
Newborn’s gender |
Male |
VR/MF |
Pregnancy history |
Number of previous live birthsa |
VR |
Pregnancy history |
Number of previous terminationsa |
VR |
Prepregnancy body mass index |
<18.5 (underweight) |
VR |
Prepregnancy body mass index |
18.5–24.9 (normal) |
VR |
Prepregnancy body mass index |
25.0–29.9 (overweight) |
VR |
Prepregnancy body mass index |
30.0–39.9 (obese) |
VR |
Prepregnancy body mass index |
≥40 (extremely obese) |
VR |
Smoking before pregnancy |
Smoked three months before pregnancy |
VR |
Smoking before pregnancy |
Number of cigarettes/day |
VR |
Previous adverse birth outcomes |
Cesarean delivery |
VR |
Previous adverse birth outcomes |
Preterm birth |
VR |
Previous adverse birth outcomes |
Other poor birth outcomesb |
VR |
Medical conditions |
Prepregnancy diabetes |
VR |
Medical conditions |
Prepregnancy hypertension |
VR |
Interpregnancy interval |
<6 months (very short)c |
VR |
Interpregnancy interval |
6–17 months (short) c |
VR |
Notes: The column on the right indicates the data source (VR and/or MF). Variables from the MF files will not be available for analysis of unlinked VR files, and vice versa. For some variables, a dummy variable for missing data might be included (not shown). Some variables might be unavailable on a state-by-state basis.
a Binary variables for 0, 1, or 2 or more births/terminations.
b As coded in the U.S. Standard Birth Certificate (defined as perinatal death or small for gestational age/intrauterine growth restricted birth).
c Interpregnancy interval calculated from the infant’s date of birth, the date of the last pregnancy (live birth or other outcome), and gestation length. Women were then categorized as follows: (1) first birth, (2) very short interpregnancy interval (fewer than 6 months), short interpregnancy interval (6 to 17 months), other interpregnancy interval (more than 17 months), or unknown (missing data or mother’s age at last pregnancy was less than 12 years or more than 55 years).
MF = Medicaid files; VR = vital records files.
Key informant interviews and focus groups. CMCS’s contractor will not use statistical methodology for stratification or sample selection for the key informant interviews or focus groups. The procedures for selecting respondents for the key informant interviews and focus groups were described in Section 1.
Medicaid postpartum telephone survey. Analysis weights are necessary to minimize the bias of estimates based only on sampled respondents. The contractor will construct analysis weights that account for both the probability of selection for each sample member and for differential response patterns among those sampled. After constructing a sampling weight that is the inverse of each sample member’s selection probability, the contractor will further adjust for eligibility determination and for response among those determined to be eligible. This will be done using information that is available for both respondents and nonrespondents and that is associated with both key outcomes and the propensity to respond. If there are a fair number of such characteristics available, the contractor will use a response-propensity model to obtain a score that can then be used to adjust the weights. If not, a simpler weighting-class approach can be used to weight up the respondents to represent the nonrespondents. After this adjustment, the contractor will trim outlier weights if indicated. Finally, the weights will be normalized so that each state contributes equally to analyses that pool the data across all four states.
Key informant interviews and focus groups. Due to the qualitative nature of the key informant interviews and focus groups, the contractor will not be calculating probability of selection for estimation purposes.
Table 6 shows the minimum detectable differences (MDDs) for comparisons between pilot program participants and nonparticipants for 1,560 survey respondents. This sample size is sufficient to detect significant differences of 7 percentage points or higher for outcomes prevalent in the population at 50 percent or lower across all four states or a difference of 14 percentage points or higher in any given state. Given the sample size and design, the contractor expects the survey will be able to detect differences between pilot program participants and nonparticipants for knowledge and behavior outcomes of interest, which cannot be obtained from administrative data. For outcomes on which effects may not be observable in a short period (such as low birth weight), the survey sample is unlikely to have sufficient power for estimating effects.
Table 6. Minimum detectable differences for 1,560 survey respondents
|
Medicaid responding sample |
Minimum detectable difference for various proportions |
|||||
|
Text4baby |
Comparison |
0.025
|
0.05 |
0.10 |
0.25 |
0.50 |
State-specific analysis |
195 |
195 |
0.044 |
0.062 |
0.085 |
0.123 |
0.142 |
Pooled
analysis |
780 |
780 |
0.022 |
0.031 |
0.042 |
0.061 |
0.071 |
Note: The calculations are based on these assumptions: 80 percent level of power; a two-sided Type I error rate of .05; that 65 percent of 300 pilot program participants per state and 300 women in a matched comparison group of women not participating in the pilot program respond to the survey per state, for a total of 1,560 respondents; a design effect of 1.1 due to nonresponse adjustments; baseline covariates control for 10 percent of the variance of the outcome; 10 percent of the population participates in the pilot program; and states are weighted equally in the pooled analysis.
Key informant interviews and focus groups. Due to the qualitative nature of the key informant interviews and focus groups, the contractor will not be calculating the precision of estimates or MDDs.
Medicaid postpartum telephone survey. CMCS’s contractor will obtain estimates of the effect of participation in the pilot program for outcomes collected in the telephone survey of Medicaid women by comparing outcomes between pilot program participants and the matched comparison group of nonparticipants. The procedures for creating the matched comparison group were described in Section 2.1.
Key informant interviews and focus groups. The contractor will not use statistical methodology for stratification or sample selection for the key informant interviews or focus groups. The procedures for selecting respondents for the key informant interviews and focus groups were described in Section 1.
The information collection schedule for this evaluation is outlined in Table 7.
Table 7. Planned information collection schedule
Activity |
Anticipated schedule |
Select sample for Medicaid postpartum telephone survey |
March 2017 |
Conduct Medicaid postpartum telephone survey |
April – June 2017* |
Recruit key informants from CMCS, Voxiva/Zero to Three, state Medicaid agencies, and active outreach partners |
November – December 2017 |
Recruit health care providers for key informant interviews and recruit focus group participants |
November – December 2017 |
Conduct key informant interviews |
January – March 2018* |
Conduct provider interviews and focus groups |
January – March 2018* |
* Survey, key informant interview, and focus group protocols were pretested and conducted with nine or fewer respondents per protocol per year.
Medicaid postpartum telephone survey. The contractor will conduct one round of the survey from April to June 2017. Therefore, there is no cyclic burden for survey respondents.
Key informant interviews. The contractor pretested protocols for each implementing stakeholder type (e.g., CMCS, Voxiva/Zero to Three, and outreach partners) in 201519; another pre-test round with implementing stakeholders as well as providers was conducted in 2016. One round of key informant interviews with all stakeholder types will be conducted in 2018.
The contractor anticipates that the key informants from CMCS, state Medicaid agencies, and Voxiva and Zero to Three will be the same respondents who participated in the two rounds of pre-test interviews. The providers selected for the pre-test will be different than the providers who will be interviewed in 2018. Therefore, there is no cyclic burden for these types of key informants.
Focus groups. CMCS’s contractor will conduct two focus groups with participants in each of the four pilot states. The contractor conducted a pre-test of the focus group questions through individual interviews with 9 women across the pilot states to test the comprehensibility of the questions. These individuals will not meet eligibility criteria for future focus groups. In addition, individuals can only participate in one focus group, so there is no cyclic burden for focus group participants.
Medicaid postpartum telephone survey. The contractor will use numerous different tactics to maximize survey response rates and to deal with nonresponse.
Locating respondents. Locating sample members is one of the main challenges the contractor foresees to obtaining a high response rate on this survey. Although the contractor expects to obtain contact information for all sample members through Medicaid records, past experience has shown that this will not always lead directly to the sample member. Telephone numbers can be particularly problematic because there is no administrative reason to keep them updated in Medicaid records. Addresses are more reliable because they are sometimes used for mailing correspondence. These might, however, be a post office box, address of a parent or guardian, financial institutions, or other types of addresses that make it difficult to locate the sample member. Before mailing an advance letter to each sample member before the start of the survey, the contractor will send the sample file through Accurint20 to obtain updated contact information, if available. If letters are returned as undelivered, the contractor will conduct additional locating on an individual basis through directory searches to attempt to find the sample member’s current mailing address.
If a telephone number for the sample member is available or obtained, an interviewer will attempt to call her to conduct the interview. Interviewers will use a protocol that calls for repeated contact attempts, including attempts on different days of the week and weekend and different times of the day. After 10 unsuccessful call attempts, cases will be given to highly trained locators who will conduct additional searchers for updated contact information. Locating staff will utilize internet searches, telephone 411 reach searches, and Google and white pages searches to attempt to find contact information for the sample member.
Conducting the interview using Blaise survey software. Implementing the survey in Blaise software will provide a controlled way for the contractor to collect data that ensures high quality and consistency by the enforcement of rules to avoid various kinds of error. The Blaise program will (1) control the routing through the questionnaire, thus avoiding path errors; (2) control response ranges so that out-of-range values are checked and updated in real time by the interviewer; and (3) make consistency checks to ensure that the respondent’s answers are consistent throughout the questionnaire. In addition, it will fill responses from previously asked questions, thus helping interviewers smoothly administer the survey. Finally, Blaise software allows for the review of paradata for management purposes, so that staffing and resources can be adjusted to better meet goals for target completes if necessary.
Conducting interviewer training. Well-trained interviewers are able to improve response rates by successfully convincing sample members to participate in the survey, avoiding refusals by reluctant or hostile sample members or gatekeepers, and by conducting the survey in a professional but friendly manner that keeps the sample member actively engaged in the interview. Interviewer training will be based on a detailed training manual and supplemented by practice exercises in gaining cooperation and becoming comfortable with the various major paths of the survey.
Monitoring interviews to ensure high quality data. Monitoring how interviewers conduct surveys is critical to ensuring the collection of high quality data. Highly trained monitors will review interviews being conducted in real time. During the first week of interviewing, the first case completed by each interviewer will be monitored. After that, 10 percent of all interviews will be monitored. The monitor listens on the telephone while sitting in front of two computer screens: one screen shows the keystrokes the interviewer uses to record answers, the other provides an assessment form on which the monitor records the quality of the interview. If the interviewer makes a serious error that leads to the collection of missing or erroneous information in the survey, the monitor will immediately approach the interviewer and provide corrective feedback to ensure accurate and valid data are collected.
Reviewing data frequencies. Frequency reviews are an important tool in ensuring data quality. To determine whether the instrument performs as specified, survey frequencies will be reviewed after the first 50 cases are completed. If any programming errors are detected (for example, erroneous skip logic or inadequate range specifications) interviewing will stop until the error has been corrected and the correction has been thoroughly tested. If any missing data have to be retrieved from sample members, specially trained quality control staff will call sample members and obtain the missing information. Frequencies will be run a second time to verify that the error has been corrected.
Minimizing nonresponse. CMCS’s contractor understands that many sample members might be balancing many different obligations such as work and taking care of young children. To facilitate responses to the survey, the contractor will instruct interviewers to remain patient, repeat answers for clarification if the sample member becomes distracted, and set appointments if the sample member has to complete the interview at a different time or in more than one session. A Spanish-language version of the instrument will also be developed and administered by Spanish-speaking interviewers to Spanish-speaking subjects. Additionally, interviewers will offer the use of several assistive devices (such as amplifier and text telephone [TTY] phones, Telecommunications Relay Service, and instant messaging) for sample members with physical impairments. Additionally, sample members who complete the interview will receive a $25 gift card in the mail as thanks their time. The contractor will consider additional reminders mailings or postcards to respondents to increase response rates, if needed. All of these steps should encourage sample members to participate in the survey after contact is made, thereby increasing the overall response rate.
Dealing with issues of nonresponse bias. Nonresponse bias is a function of both response rate and how different respondents and nonrespondents are with respect to a particular measure. The contractor will do everything possible to maximize the response rate, which will help mitigate the risk of nonresponse bias. A description of how analysis weights that account for both the probability of selection for each sample member and for differential response patterns among those sampled will be constructed was described in Section 2.2.
Key informant interviews. To maximize participation among key informants the contractor will (1) schedule appointments at times convenient for key informants, and at convenient locations for health care providers; (2) make reminder calls two business days before the scheduled interview; and (3) reschedule appointments when asked to do so.
A response rate of 95 percent is expected for the key informant interviews based on previous experience and a typically high level of motivation from the stakeholders involved.
Focus groups. To maximize response rates for the focus groups, the contractor will (1) offer a $25 gift card as thanks for participation and to defray any costs participants incur for transportation or child care; (2) schedule groups at a time that is convenient for most participants, (3) make reminder calls two business days before the scheduled groups; and (4) offer groups in convenient locations that have easy access to public transportation. The contractor will recruit about twice as many people as are expected to appear for the focus groups to ensure an adequate number of participants.
Medicaid postpartum telephone survey. CMCS’s contractor completed a preliminary test of the survey for timing purposes in December 2014. The survey took 30 minutes to administer in its current form. A formal pretest with nine Text4baby subscribers was conducted in January 2015. The nine women had given birth within the past eighteen months and received Medicaid benefits during their recent pregnancy. The pre-test allowed us to validate the length of the instrument and confirm the anticipated public burden associated with participation in the survey. The pre-test also allowed us to debrief with participants and collect information that helped to inform refinements and clarifications to the wording and order of items. Based on pretest results, we deleted 11 items/sub-items, added two items, and made revisions to 20 items/ sub-items. Specifically, the changes included: the addition of two questions about early childhood development to account for Text4baby’s inclusion of messages related to this topic; deletion of items that are not essential for analysis; simplification of response scales; and minor wording changes and clarifications to improve data quality. Responses collected during the pre-test were not and will not be analyzed.
Key informant interviews and focus groups. To the extent possible, the contractor drew upon protocols that were used in previous studies of Text4baby. The contractor pretested the interview protocols with implementers (CMCS, Voxiva/Zero to Three, state Medicaid agencies, and outreach partners), providers, and pilot program participants (for the focus group protocol). The pretests were conducted with less than nine respondents per protocol annually. Protocols have been updated to reflect feedback on the burden and content of the protocols. During the pre-test with implementers, the contractor made minor changes to questions related to sustainability after the first round. Provider and focus group protocols required no substantial changes.
CMCS, Mathematica Policy Research, and Insight Policy Research were consulted about the substantive, methodological, and statistical aspects of the study. Their recommendations were incorporated into the study design and instruments on an ongoing basis. The person responsible for receiving and approving the instruments and information collection is Lekisha Daniel-Robinson of CMCS. Other people consulted on technical and statistical issues related to data collection are listed in Table 8.
Table 8. Individuals Consulted on Technical and Statistical Issues
Name |
Affiliation/address |
Telephone number/ email address |
Lekisha Daniel-Robinson |
CMCS Baltimore, MD |
(410) 786-8618 |
Marsha Lillie-Blanton |
CMCS Baltimore, MD |
|
Amy Pirretti |
Voxiva, Inc. Washington, DC |
|
Pamela Johnson |
Voxiva, Inc. Washington, DC |
(202) 419-0146 |
Jessica Bushar |
Zero to Three Alexandria, VA |
(703) 838-7509 |
Meg Tucker |
Insight Policy Research, Inc. |
(703) 504-9496 |
So O’Neil |
Mathematica Policy Research Cambridge, MA |
(617) 301-8975 |
Keith Kranker |
Mathematica Policy Research Cambridge, MA |
(609) 945-6551 |
Erin Panzarella |
Mathematica Policy Research Princeton, NJ |
(609) 936-2774 |
1 Cole-Lewis, H., and T. Kershaw. “Text Messaging as a Tool for Behavior Change in Disease Prevention and Management.” Epidemiologic Reviews, vol. 32, no. 1, March 2010, pp. 56–69.
2 Gazmararian, J.A., B. Yang, L. Elon, M. Graham, and R. Parker. “Successful Enrollment in Text4Baby More Likely with Higher Health Literacy.” Journal of Health Communication, vol. 17, no. 3, 2012, pp. 303–311.
3 Hou, M.Y., S. Hurwitz, E. Kavanagh, J. Fortin, and A.B. Goldberg. “Using Daily Text-Message Reminders to Improve Adherence with Oral Contraceptives: A Randomized Controlled Trial.” Obstetrics and Gynecology, vol. 116, no. 3, November 2010, pp. 633–640.
4 Evans, W.D., L.C. Abroms, R. Poropatich, P.E. Nielsen, and J.L. Wallace. “Mobile Health Evaluation Methods: The Text4Baby Case Study.” Journal of Health Communication, vol. 17, suppl. 1, May 2012b, pp. 22–29.
5 Evans, W.D., L.C. Abroms, R. Poropatich, P.E. Nielsen, and J.L. Wallace. “Mobile Health Evaluation Methods: The Text4Baby Case Study.” Journal of Health Communication, vol. 17, suppl. 1, May 2012b, pp. 22–29.
6 Evans, William Douglas, Jasmine L. Wallace, and Jeremy Snider. “Pilot Evaluation of the Text4baby Mobile Health Program.” BMC Public Health, vol. 12, no. 1031, November 2012, pp. 1–10.
7 Secondary data sources are Voxiva Text4baby administrative data, state Medicaid administrative data, state vital records data; and state-reported maternity Core Set measures (from CHIP Annual Report Template System [CARTS]).Through its contractor for this evaluation, CMCS has received clearance for the collection of data from these secondary data sources from the New England Institutional Review Board (IRB) (14-432).
8 Kranker, Keith, Sarah Forrestal, and Vanessa Oddo. “FNS WIC-Medicaid Study II: Feasibility of Expanding the Study to Include Additional States.” Princeton, NJ: Mathematica Policy Research, March 2013.
9 Markus, A.R., E. Andres, K.D. West, N. Garro, C. Pellegrini. “Medicaid Covered Births, 2008 Through 2010, in the Context of the Implementation of Health Reform.” Women’s Health Issues vol. 23, no. 5, September-October 2013, pp. e273 - e280; and “Erratum,” vol. 23, no. 6, November 2013, p. e411.
10 The estimated numbers of completed interviews per state are 195 subscribers and 195 nonsubscribers.
11 If the contractor cannot identify the sample through this method, the contractor will randomly select 300 postpartum Text4baby subscribers in each state during the survey field period, based on the baby’s due date. To be eligible for the survey, subscribers must have replied to a Text4baby message to indicate that they were Medicaid recipients during their most recent pregnancy. For the nonsubscriber sample, the contractor will randomly select 300 postpartum Medicaid recipients from Medicaid data based on the baby’s due date. Because the level of penetration of Text4baby is expected to be low, the contractor does not expect that randomly sampling 300 women from Medicaid data will identify many (if any) Text4baby subscribers. Our survey instrument is designed to identify Text4baby subscribers and nonsubscribers, so if any Text4baby subscribers are selected in the nonsubscriber sample from Medicaid data, the contractor will be able to classify them as subscribers in our analyses. If the level of penetration of Text4baby turns out to be higher than expected, the contractor will work with CMCS and the state to develop a better strategy to identify nonsubscribers using the Medicaid data. The main limitation with this method is that it greatly limits the ability to construct a matched comparison sample.
12 First text message: “You’re invited to talk about text4baby with other moms from your area. You’ll receive a $25 XX gift card. If interested, call X-XXX-XXXX.” Follow-up message one week before the focus group: (1/2) “You can still sign up to talk about text4baby with other moms from your area. You’ll receive a $25 XX gift card. Call X-XXX-XXXX.” (2/2) “If you’ve already called to sign up, thank you! If you’re not interested, that’s okay, too! This will be the last message about this opportunity.”
13 Project staff will follow a short screening protocol to screen women who call in.
14 This may vary by state.
15 The contractor may extend the field period to four months should the response rate be lower than anticipated.
16 This sampling approach effectively oversamples women in the pilot program, as long as enrollment rates are below 50 percent. Analysis will be weighted appropriately to adjust for the complex sample design.
17 Iacus, S.M., G. King, and G. Porro. “Multivariate Matching Methods That Are Monotonic Imbalance Bounding.” Journal of the American Statistical Association, vol. 106, no. 493, 2011, pp. 345–361.
18 Stuart, E.A. “Matching Methods for Causal Inference: A Review and a Look Forward.” Statistical Science, vol. 25, no. 1, 2010, pp. 1–21
19 The two initial rounds of key informant interviews will be considered a pilot test and will be conducted with less than 9 respondents per protocol annually.
20 Accurint is a commercial vendor that uses propriety databases to search for up-to-date contact information.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Office of Management and Budget Evaluating a Pilot Mobile Health Program Support Statement Part B |
| Subject | OMB |
| Author | Mathematica Policy Research |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |