CMS-10634, OMB 0938-TBD (New)
August 2016
This page left blank for double-sided copying.
OMB SUPPORTING STATEMENT PART A FOR EVALUATING A PILOT MOBILE HEALTH PROGRAM 1
A. Background 1
1. Overview of a pilot mobile health program 1
2. Need for evaluation of a pilot mHealth program 1
3. Data collection activities under the evaluation of a pilot mHealth program 2
B. Justification 3
1. Need and legal basis 3
2. Purpose of use of information collection 4
3. Use of improved information technology and burden reduction 5
4. Efforts to identify duplication and use of similar information 6
5. Small businesses 7
6. Less frequent collection 8
7. Special circumstances 8
8. Federal Register/outside consultation 8
9. Payments/gifts to respondents 8
10. Assurance of confidentiality 9
11. Justification for Sensitive Questions 10
12. Estimates of annualized hour and cost burden 11
13. Estimates of other total cost burden to respondents or record keepers 12
14. Annualized cost to federal government 12
15. Changes to burden 12
16. Plans for tabulation and publication and project time schedule 13
17. Reason(s) display of OMB expiration date is inappropriate 15
18. Exceptions to certification for paperwork reduction act submissions 15
APPENDIX A: TELEPHONE SURVEY
APPENDIX B: KEY INFORMANT INTERVIEWS
APPENDIX C: FOCUS GROUPS
APPENDIX D: VOXIVA TEXT4BABY ADMINISTRATIVE DATA
APPENDIX E: STATE MEDICAID DATA
APPENDIX F: STATE VITAL STATISTICS DATA
This page left blank for double-sided copying.
The Center for Medicaid & CHIP Services (CMCS), a division of the Centers for Medicare & Medicaid Services (CMS), is supporting a pilot mobile health (mHealth) program in California, Louisiana, Ohio, and Oklahoma. The three-year mHealth project is being conducted to complement existing CMCS measurement, data collection, and reporting activities to monitor, track, and assess state’s maternal and infant health efforts in Medicaid and CHIP populations. This project focuses on developing key measures of maternal and infant child health in the Medicaid population. This information will be used in monitoring performance of the participating states on the Maternity Core Set measures and in updating the Adult Core Set annually as required by Section 2701of the Patient Protection and Affordable Care Act (ACA).
The project also involves the implementation of a customized version of Text4baby within the four states to assess the role of health education, knowledge, and behavior play in observed perinatal health outcomes. Text4baby is provided free to those who enroll in the service via text message or through the Text4baby website. The service is overseen by Voxiva, a mHealth and wellness company. Zero to Three developed more than 250 messages for the Text4baby service that contain critical information for pregnant women and new mothers with infants younger than one year. The messages contain brief, interactive information encouraging healthy behaviors and share ways to improve maternal and child health.
The unique feature of the pilot mHealth program is its education component delivered through Text4baby. Despite the interest in and potential benefits of mHealth programs, their effect on health knowledge, behavior, and outcomes has been inconclusive. Several studies have demonstrated mHealth’s potential to prompt behavior changes related to use of oral contraceptives, physical activity, and smoking cessation, while others have shown no effect or inconclusive results.1,2,3 Studies specific to Text4baby have also been limited. One study showed small effects on knowledge and beliefs related to preparedness for motherhood, although the effects on behavior change were not conclusive.4 Although several studies have focused on socioeconomically disadvantaged populations, none have concentrated specifically on the effect of Text4baby in the Medicaid population.5,6 The limited evidence in these studies highlights the need to further assess key implementation factors related to outreach, participation, and engagement, which might explain variation in outcomes, especially among disadvantaged populations that might benefit the most from mHealth interventions. Given the limited information available on the effect of Text4baby and the underlying factors that could drive these effects, especially in the Medicaid population, CMCS has funded an evaluation of the pilot mHealth program to further assess the potential of mHealth programs to improve maternal and infant health among Medicaid enrollees. Most importantly, the data collection and analysis methods from this study will be used to track performance of participating states on key prioritized measures for CMCS’ initiatives related to maternal and infant health and the methods developed under the evaluation can be used by other states to conduct similar monitoring and evaluation. A contractor will conduct the evaluation of a pilot mHealth program for CMCS.
Data for the evaluation will come from several primary and secondary data sources.7 All primary and secondary data for this evaluation will be collected and analyzed by CMCS’s contractor. Data will remain with the contractor until the end of the study, at which time the contractor will destroy data as specified in data use agreements and institutional review board clearances. The three primary data sources for which CMCS seeks Office of Management and Budget (OMB) clearance include: a postpartum telephone survey of Medicaid-enrolled pilot program participants and nonparticipants in the four pilot states (see Appendix A for all materials related to the telephone survey); key informant interviews with CMCS, Voxiva, Zero to Three, state Medicaid agencies, health care providers, and outreach partners (see Appendix B for all materials related to key informant interviews); and focus groups with Medicaid-enrolled pilot participants and nonparticipants (see Appendix C for all materials related to focus groups).
The three primary data collection components are summarized next:
Medicaid postpartum telephone survey. Approximately 2,400 postpartum Medicaid–enrolled pilot program participants and nonparticipants, or about 600 women in each of the four states participating in the pilot, will be selected to participate in the telephone survey. To be eligible for the survey, women must have a delivery paid for by Medicaid/CHIP or an infant with Medicaid coverage at birth. Ideally, women will be less than 12 months postpartum during the survey field period. This will be a one-time data collection.
Key informant interviews. CMCS’s contractor will conduct up to 15 telephone interviews with key informants from CMCS (one interview), Voxiva and Zero to Three (one interview with each organization), state Medicaid agencies (one interview per state), and state and local outreach partner organizations (up to two interviews per state).
Site visits to each participating state. CMCS’s contractor will conduct one round of 1.5-day site visits to each of the four states participating in the pilot. For each site visit, the contractor will select a single geographic area in each state that has a high concentration of Medicaid providers. As part of each site visit, the contractor will conduct two interviews with different providers that deliver care to pregnant and postpartum women enrolled in Medicaid and that have knowledge of the pilot program, and two focus groups with postpartum Medicaid pilot program participants.
Key informant interviews with health care providers. During the site visits to each state, CMCS’s contractor will conduct up to 8 total in-person interviews (2 health care provider interviews * 4 states) with different health care providers that deliver care to pregnant and postpartum women enrolled in Medicaid and that have knowledge of the pilot program.
Focus groups. CMCS’s contractor will work with outreach partners serving low-income pregnant women and new mothers to identify a subset of postpartum Medicaid pilot program participants to participate in focus groups during the site visits to the four states participating in the pilot. The contractor will conduct two focus groups with pilot program participants during each site visit for a total of 8 focus groups (2 focus groups * 4 states). To be eligible to participate in a focus group, women must have a delivery paid for by Medicaid/CHIP or an infant with Medicaid coverage at birth, less than 12 months postpartum during the scheduled period for the site visit, and must not have participated in the telephone survey.
This statement requests OMB approval for data collection to support the evaluation of this pilot mHealth program. This data collection effort will be used to assist CMS in tracking maternal and infant health outcomes in the Medicaid population.8 In addition, the methods used for collection and analysis of the data may be useful to states and serve to increase reporting of perinatal core set measures and monitoring and interpretation of state-level maternal and infant health efforts. Furthermore, results from the evaluation will help CMS understand the usefulness of mobile technology for conveying health information to pregnant women and new mothers enrolled in Medicaid/CHIP, as well as the influence this information has on health behaviors and outcomes.
Through its data collection and synthesis, this project focuses on developing key measures of maternal and infant health in the Medicaid population. In particular, it will assess and track the prevalence of maternal and infant outcomes and service utilization. Key project measures include (1) nine Adult and Child Medicaid and CHIP Quality Core Set Measures related to perinatal health, (2) additional indicators of maternal and infant health outcomes that research has shown to be essential to perinatal health (e.g., breastfeeding, family planning, and smoking during pregnancy), (3) Medicaid enrollment before, during, and after pregnancy to understand the role of insurance in outcomes, and (4) use of Text4baby to assess the association of health education with outcomes in the Medicaid population. Generating these measures will require the contractor to collect several types of primary and secondary data, including Medicaid, Vital Statistics, Survey, and Text4baby administrative data. Table 1 below shows the prioritized outcome measures by study data sources.
Table 1. Prioritized outcome measures by study data sources
|
|
Proposed data source(s)a |
|||
Outcome |
Core Set Measure |
Medicaid |
Vital Records |
Survey |
Text4baby |
Behavioral health risk assessment |
X |
|
|
X |
|
Prenatal care: frequency |
X |
X |
X |
X |
|
Prenatal care: timeliness |
X |
|
|
|
|
Antenatal steroids (e.g., 17P) |
X |
|
X |
X |
|
Early elective delivery |
X |
|
X |
|
|
Cesarean rate for nulliparous singleton vertex |
X |
|
X |
|
|
Live births weighing less than 2,500 grams |
X |
|
X |
|
|
Postpartum care |
X |
X |
|
|
|
Well-child visits |
X |
X |
|
|
|
Breastfeeding (initiation; duration; support) |
|
|
X |
X |
|
Family planning (birth spacing; contraception) |
|
|
X |
X |
|
Smoking during pregnancy |
|
|
X |
X |
|
Enrollment in Medicaid |
|
X |
|
|
|
Text4baby enrollment |
|
|
|
|
X |
Use of Text4baby text messages |
|
|
|
|
X |
a When practical, we will use Vital Records and Voxiva data linked to Medicaid data in order to correctly identify births paid by Medicaid and for other purposes.
b Adult and child core set specifications indicate that electronic health records (EHRs) are needed to calculate these measures. Because states often have barriers to collect EHRs, we will develop proxy measures using other sources.
Medicaid postpartum telephone survey. The Medicaid postpartum survey aims to capture information about pilot program participants and nonparticipants that is not available in administrative data sources, including information about their awareness of the pilot program, motivation for signing up, satisfaction with the service, pregnancy experiences and perspectives, and health behaviors. The survey has nine sections: (A) Text4baby enrollment and use of services (participants only), (B) Disenrollment (participants only), (C) Satisfaction with Text4baby (participants only), (D) Pregnancy History, (E) Most Recent Pregnancy, (F) Health Care Access and Utilization, (G) Pregnancy and Postpartum Behaviors, (H) Health Knowledge, and (I) Participant Characteristics (see Appendix A for the telephone survey instrument).
Key informant interviews. The national and state-level stakeholder interviews (CMCS, Voxiva, Zero to Three, state Medicaid agencies, outreach partners, and health care providers) are designed to collect qualitative information on the successes and challenges of implementing the pilot mHealth program, and to capture partners’ views on the lessons learned from the pilot. In addition, to understand the role of providers in promoting health education tools as well as providers’ insights into how tools, such as Text4baby, has affected patient knowledge and behaviors, CMCS’s contractor will also interview providers that are most knowledgeable and active in promoting the pilot mHealth program in their states within a selected geographic area. For each of the different stakeholder types, the contractor has developed a semi-structured discussion guide with questions addressing a range of topics relevant to each stakeholder (see Appendix B for the complete stakeholder interview discussion guides).
Focus groups. The purpose of the focus groups is to collect qualitative information on the implementation and effectiveness of the mHealth pilot from the consumers’ perspective. CMCS’s contractor has developed a discussion guide to guide the focus group sessions (see Appendix C for focus group discussion guide).
Medicaid postpartum telephone survey. The survey will comply fully with the Government Paperwork Elimination Act (GPEA), Public Law 105-277, Title XVII by employing technology efficiently in an effort to reduce burden on respondents. CMCS’s contractor will use computer assisted telephone interviewing (CATI) technology to administer the survey to all respondents. The contractor expects to complete 780 interviews with Medicaid pilot mHealth program participants and 780 interviews with Medicaid enrollees who did not participate in the pilot mHealth program. CATI surveys optimize resources and guarantee high quality data because the technology incorporates automated range checks and skip patterns and enforces consistency among critical questions. CATI programming will enable interviewers to collect only information that specifically applies to each respondent, thereby eliminating undue time burden on respondents. The interview solicits only information from respondents that corresponds to the specific topic areas discussed in question B2. No superfluous or unnecessary information is being requested. In addition, by abstracting information from Voxiva Text4baby administrative data9, state Medicaid administrative data,10 state vital records data,11 and state-reported maternity Core Set measures (from CHIP Annual Report Template System [CARTS],12 the burden on individual respondents will be further reduced by relying on existing secondary data sources for certain pieces of information rather than adding additional questions to the survey. Finally, in order to minimize respondents’ language burden, interviewers can toggle between English and Spanish versions of the instrument.
Focus groups and key informant interviews. Information technology will not be used to collect information during the 15 key informant interviews, eight consumer focus groups with 80-96 participants, and eight provider interviews. These collections are qualitative in nature and so few in number that it is neither practical nor affordable to build separate electronic instruments to collect the information. CMCS’s contractor will collect information using paper instruments. All focus groups and key informant interviews will also be audio recorded and transcribed to facilitate analysis. Focus group transcripts and key informant interview notes will be analyzed using Atlas.ti, a software system used for the qualitative analysis of large bodies of textual data.
Medicaid postpartum telephone survey. CMCS’s contractor sought to avoid duplication of effort in the development of the survey by identifying existing instruments with relevant questions. Survey questions were heavily drawn from the survey of Text4baby subscribers and nonsubscribers used in a previous evaluation by the Health Resources & Services Administration (HRSA), as many of the questions are relevant, the pathways and skip patterns have been developed and tested, and the survey received OMB clearance (OMB Control No: 0915-0347, Expiration Date: 02/28/2015) and IRB clearance. Many of the questions in this survey were drawn from existing validated surveys including the 1989 National Maternal and Infant Health Survey, the Pregnancy Risk Assessment Monitoring System, the Early Childhood Longitudinal Survey 9-month parent interview, and the National Survey of Children’s Health. Furthermore, as described in Section A3, CMCS’s contractor sought to reduce respondent burden and avoid duplication by relying on existing secondary data sources for information rather than adding additional questions to the survey.
As described previously, no other studies have focused specifically on the effect of a mHealth program in the Medicaid population. Thus, this information collection does not duplicate any other effort and the information cannot be obtained from any other source.
Focus groups and key informant interviews. CMCS’s contractor sought to avoid duplication of effort in the design of the focus group and key informant interview protocols by identifying existing instruments with relevant questions. Questions for these protocols were drawn from the protocols used in a previous evaluation by HRSA (OMB Control No: 0915-0347, Expiration Date: 02/28/2015) and were tailored for this specific information collection.
As described previously, no other studies have focused specifically on the effect of Text4baby on the Medicaid population. Thus, this information collection does not duplicate any other effort and the information cannot be obtained from any other source.
Medicaid postpartum telephone survey. This part of the information collection will not affect small businesses or other small entities.
Key informant interviews. Stakeholder interviews with CMCS, Voxiva and Zero to Three, state Medicaid agencies, and outreach partners, will be conducted by telephone. These stakeholders are composed of a wide range of public and private organizations; however, some outreach partners might be small nonprofit organizations. To minimize burden and disruption in other business activities, interviews will be restricted to 60 minutes and will be conducted at a time that is convenient for the respondent.
A small amount of burden could be placed on health care providers when their personnel are invited to participate in the health care provider key informant interviews. These provider interviews will be conducted in person. Burden will be minimized by restricting the interviews to 30 minutes and conducting them at a time and location that is convenient for the respondent.
Medicaid postpartum telephone survey. The purpose of the telephone survey is to collect information about pilot program participants and nonparticipants that is not available in administrative data sources. Examples of this information include use of the pilot mHealth program and information received from the program, satisfaction with the program, pregnancy history, pregnancy and postpartum behaviors, health knowledge, and certain demographic characteristics. If this information is not collected, CMCS’s contractor will not have sufficient information to fully evaluate the effects of the pilot mHealth program.
Key informant interviews. CMCS’s contractor will conduct interviews by telephone with individuals or groups involved in implementing the pilot mHealth program including CMCS (two interviews), staff from Voxiva and Zero to Three (two interviews); and state Medicaid agency staff (eight interviews). If this information is not collected, CMCS’s contractor will not have sufficient information to fully evaluate the effects of the pilot mHealth program over the course of its implementation.
There will be one round of interviews with health care providers that are most knowledgeable of and active in regard to the pilot mHealth program in their states (eight interviews) and with state or local outreach partners (eight interviews). The contractor will select different health care providers and outreach partners to participate in data collection. Therefore, each of these key informants will respond one time only for this information collection. If this information is not collected, the contractor will not have sufficient information to fully evaluate the effects of the pilot mHealth program.
Focus groups. The purpose of the focus groups is to collect qualitative information on the implementation and effectiveness of the mHealth pilot from the consumers’ perspective. CMCS’s contractor will conduct a total of 8 focus groups during this evaluation with 80 to 96 participants in total. If this information is not collected, the contractor will not have sufficient information to fully evaluate the effects of the pilot mHealth program.
Social Security numbers (SSNs) will not be collected in any primary data collection instrument. SSNs will be collected in secondary data sources (Medicaid and vital records) in order to link data between sources. However, if states provide us with linked data for the secondary data, we will not collect SSNs for these sources.
Otherwise, there are no special circumstances with this information collection.
The 60-day notice published in the Federal Register on October 14, 2016 (81 FR 71100). No comments were received.
Medicaid postpartum telephone survey. CMCS recognizes the time burden placed on respondents to participate in the telephone survey. Incentive payments to respondents have been shown to encourage participation and thereby increase response rates, which in turn improve the validity and reliability of the data collected. The role of incentives in increasing survey response rates has been widely documented.13,14,15 Some research suggests that the level of incentives matters; in other words, higher incentives engender higher response rates.16,17 Survey respondents will receive a $25 gift card in the mail about three weeks after completing the telephone survey. The $25 incentive will be in the form of a gift card because gift cards are easier and more convenient to redeem than checks, especially for participants who do not have bank accounts.
Key informant interviews. Key informants will not receive incentive payments because most are participating as part of their professional positions.
Focus groups. Focus group participants will receive a $25 gift card at the end of the focus group session to thank them and to defray transportation and child care costs.
Medicaid postpartum telephone survey. CMCS has embedded protections for privacy in the study design. The information collection will fully comply with all aspects of the Privacy Act. Individuals will be assured of the privacy of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). All respondents will be informed at the start of the interview that the information they provide will be kept private, unless otherwise compelled by law. They will also be informed that any information shared as a result of the study will not enable them or their family members to be identified. They also will be informed that participation is voluntary, that they may refuse to answer any question, and that they can stop the interview at any time. The interviewer will verbally ask for the respondent’s consent to participate before administering the survey. The interviewer will offer to provide contact information for the project director and an IRB contact should the respondent have additional questions in the future. Additionally, the CMCS’s contractor will obtain verbal consent from the parent or guardian of unemancipated minors before administering the survey to a minor.
Key informant interviews. Verbal consent to participate in the key informant interviews will be obtained from respondents before the start of the interview. The consent process will stress that participants will not be identified by name in any reports. The interviewer will offer to provide contact information for the project director and an IRB contact should the key information have additional questions in the future.
Focus groups. At the beginning of the focus groups, participants will be asked to sign a consent form. The consent form will describe the purpose of the focus group, focus group procedures, benefits and risks, the confidentiality of the information that participants provide, and it will stress the voluntary nature of participation. The moderator will also inform participants that information shared during the group is to remain confidential and should not be shared with others outside of the group. The consent form will include contact information for the project director and an IRB contact should the participant have additional questions in the future. Participants will sign two copies of the consent form: the contractor will retain one copy and give the other to the participant.
In addition to the specific assurances of confidentiality for each of the data collection activities described here, two approaches cut across the entire evaluation. First, all contractor employees are required to follow strict data security policies and procedures when accessing confidential and proprietary information in performance of assigned duties. All contractor employees sign a pledge to protect the confidentiality of data and respondent identity; breaking that pledge is grounds for immediate dismissal and possible legal action. Furthermore, confidential data are protected by a firewall and stored on a fully encrypted secure network drive with limited access and special backup procedures. Only authorized contractor staff with a need to know will be able to access study data that that might contain personally identifiable information and protected health information. Second, as mentioned previously, CMCS, through its contractor, has received IRB approval for this information collection through the NEIRB (14-432).
Medicaid postpartum telephone survey and focus groups. Some questions in the telephone survey and on the Participant Information Form (PIF) collected from focus group participants ask about personal behaviors that might be of a sensitive nature for some respondents. Examples of potentially sensitive health behavior questions include those related to smoking and alcohol use during pregnancy, depression, breastfeeding, and family planning methods.
CMCS’s contractor has minimized the number of sensitive questions to those necessary to answer the key research questions for this evaluation. The survey and PIF include questions on topics that are covered in the pilot mHealth program messages and that are directly relevant to assess outcomes and progress towards goals of the pilot mHealth program. Interviewer training for administering the survey will stress the importance of asking all questions in the survey in a professional and non-judgmental manner. Survey and focus group participants can skip any questions they feel uncomfortable answering and can discontinue their involvement in the study at any point. Participants are informed of this during the consent process and are reminded of this at the start of the survey and the focus group.
Key informant interviews. The key informant interview protocols do not include sensitive questions.
Table 2 provides information about the time burden incurred by respondents. As can be seen in the table, the time burden for postpartum Medicaid-enrolled women is 780 hours for the telephone survey. The time burden for Medicaid-enrolled pilot program participants and nonparticipants is 163 hours. The time burden for key informants participating in semi-structured interviews is 38 hours.
Table 2. Time burden estimates
Instrument* |
Person incurring burden |
Number of respondents |
Responses per respondent |
Average burden per response (hours) |
Total burden (hours) |
Telephone survey |
Postpartum Medicaid-enrolled women |
1,560 |
1 |
0.5 |
780 |
Focus groups |
Medicaid-enrolled pilot participants and nonparticipants |
96 |
1 |
1.7 |
163 |
Semi-structured interviews |
Representatives from CMCS, Voxiva and Zero to Three, and state Medicaid agencies |
7 |
1 |
1.0 |
7 |
Semi-structured interviews |
State and local outreach partners |
8 |
1 |
1 |
8 |
Semi-structured interviews |
Health care providers |
8 |
1 |
0.5 |
4 |
Total |
|
1,679 |
|
|
962 |
*The three primary data sources for this information collection request include: a postpartum telephone survey of Medicaid-enrolled pilot program participants and nonparticipants in the four pilot states (see Appendix A for all materials related to the telephone survey); key informant interviews with CMCS, Voxiva, Zero to Three, state Medicaid agencies, health care providers, and outreach partners (see Appendix B for all materials related to key informant interviews); and focus groups with Medicaid-enrolled pilot participants and nonparticipants (see Appendix C for all materials related to focus groups).
Table 3 provides information about the cost burden incurred by respondents.
Medicaid postpartum telephone survey. The average (median) wage, based on the U.S. Department of Labor, Bureau of Labor Statistics, Occupational Employment Statistics, All Occupations, is estimated to be $16.87 per hour for these respondents. The total cost burden for these respondents is $13,158.60.
Key informant interviews. The average (median) wage, based on the U.S. Department of Labor, Bureau of Labor Statistics, Occupational Employment Statistics, All Occupations, is estimated to be $78.61 per hour for national and state-level key informants and health care providers who will participate in semi-structured interviews. The total cost burden for these respondents is $1,886.64.
Focus groups. The average (median) wage, based on the U.S. Department of Labor, Bureau of Labor Statistics, Occupational Employment Statistics, All Occupations, is estimated to be $16.87 per hour for these respondents. The total cost burden for these respondents is $5,499.62.
Table 3. Estimated cost to respondents
Instrument |
Respondent type |
Total burden hours |
Hourly wage rate |
Total respondent costs |
Telephone survey |
Medicaid-enrolled women |
780 |
$16.87 |
$13,158.60 |
Focus groups |
Medicaid-enrolled pilot participants and nonparticipants |
163 |
$16.87 |
$2,749.81 |
Semi-structured interviews |
Representatives from CMCS, Voxiva and Zero to Three, and state Medicaid agencies |
7 |
$78.61 |
$550.27 |
Semi-structured interviews |
State and local outreach partners |
8 |
$78.61 |
$628.88 |
Semi-structured interviews |
Health care providers |
4 |
$78.61 |
$314.44 |
Total |
|
962 |
|
$17,402.00 |
Not applicable. Respondents will not incur any or capital costs.
The evaluation will take place over a four-year period. The total cost of the evaluation to the government is $2,571,493, which includes the amount awarded via contract to the contractor ($2,528,046) and CMCS staff time/resources ($43,447). The total evaluation cost was based on the budget developed by the contractor which includes calculated wages and hours for all contractor staff, all mailing costs, telephone charges, overhead costs per contract year, and the Government staff costs. The annualized contract cost has been determined to be $642,873 per year by dividing the total funded amount by four years.
This is a new data collection.
Medicaid postpartum telephone survey. Information from the telephone survey of pilot program participants and nonparticipants is the primary data source for a number of key outcome variables related to knowledge, behavior, and health care utilization. Key outcomes of interest include smoking cessation and smoking during pregnancy; breastfeeding (initiation, duration, and support); family planning (birth spacing and contraception); behavioral health risk assessment; prenatal care (frequency, timeliness, and adequacy index); use of antenatal steroids; and oral health. For these outcomes, estimates of the effect of participation in the pilot program will be obtained for outcomes collected in the telephone survey by comparing outcomes between pilot program participants and the matched comparison group of nonparticipants. To reduce the threat of bias arising from differences between the two groups, impacts will be estimated in a regression framework and will use the rich data from the survey to account for differences between the two groups that might otherwise bias measured impacts. This information includes factors that might jointly motivate women to participate in the pilot mHealth program and effect their outcomes (such as their concern or focus on healthy birth outcomes) and that left unaccounted for could substantially bias any program effects based on an external control group. For the survey-based outcome variables, to calculate effects of the mHealth pilot program on the target population as a whole, the results from the matched comparison group analysis will be scaled using estimates of the effect of the pilot program on participation.
After survey data are collected, CMCS’s contractor will conduct standard cleaning and construct key outcomes and control variables. The contractor will also assess nonresponse and construct final sampling/nonresponse weights. Next the contractor will create analytic data files with SAS programs used for cleaning and analysis. Using the analytic data files, the contractor will use multivariate regression techniques to develop estimates of program effect on outcomes. Estimates of the effect of enrolling in the pilot mHealth program for outcomes collected in the telephone survey of Medicaid women will be obtained by comparing outcomes of pilot program participants and the matched comparison group of nonparticipants. These results can then be scaled with estimates from analysis with enrollment rates18 to calculate effects of the pilot program on the target population.
Focus groups and key informant interviews. The contractor will use qualitative data collected through key informant and focus groups to identify key themes within and across states. This information will help inform the program’s implementation and shape findings on the replicability and sustainability of the mHealth program in other states and across other topics of high priority for Medicaid populations. After qualitative data are collected, they will be organized and analyzed following five key steps:
Team debriefings. Contractor staff will review transcripts from qualitative data collection efforts, and teams will discuss initial thoughts on key emerging themes by evaluation domain.
Thematic framing. Contractor staff will develop a hierarchy of conceptual categories and classifications (attributes and codes) linked to evaluation domains, research questions, and areas for measurement. A codebook defining each element of the coding scheme to facilitate the reliability and validity of coding will be developed. Contractor staff will then train staff on the use of the coding scheme, validate 100 percent of the coding of the first set of notes and transcripts, and periodically verify the accuracy of the process.
Coding. Using the code book, contractor staff will categorize statements in the transcripts. As data collection proceeds, the contractor will refine the coding scheme to align with themes and topics that emerge from the data. This approach will enable the contractor to access data on a specific topic quickly and to organize information in different ways to identify primary themes, trends, and patterns in the data, and to compile the evidence supporting them.
Triangulation. After coding, CMCS’s contractor will identify themes across data sources. This step will enable the contractor to characterize implementation experiences in each state, as well as variations across and within states, and examine themes and topics from multiple perspectives to highlight similarities and differences. The contractor will also explore relationships across themes (for example, relationships between the types of implementation challenges and pilot program participant satisfaction and perspectives on sustainability), and will develop a descriptive summary of the themes across respondents and data types.
Cross-state comparison. To the extent that the information is available, the contractor will conduct a cross-state comparison of key themes. The contractor will identify additional crosscutting hypotheses to explore in the data and examine implications for the interpretation of quantitative results for inclusion in the interim and final reports. In addition, the contractor will engage CMCS and states in conversations about preliminary results to uncover new insights.
Interim reports will be generated during each of the first three project years; a final report will be generated at the end of the fourth project year.
Interim reports. CMCS’s contractor will incorporate findings from all data collection and analysis activities, summarizing and integrating the themes that emerged across the evaluation components and data sources in the interim reports. Each interim report will provide a comprehensive representation of the year’s activities and will also build upon and consider findings from previous years’ reports.
Final report. The final report will reflect findings from across all project years. The broad outline will be similar to the interim reports, but will incorporate and integrate all data collection activities, analyses, and findings. The report will focus on answering all the evaluation questions. It will include additional discussion related to implications for the future of similar mHealth programs within the Medicaid, pregnant and postpartum, and broader populations.
The project began in June 2014 and is expected to end in June 2018. The schedule for key data collection activities relevant to this request for OMB approval is presented in Table 4.
Table 4. Planned information collection schedule
Activity |
Anticipated schedule |
Select sample for Medicaid postpartum telephone survey |
March 2017 |
Conduct Medicaid postpartum telephone survey |
April – June 2017* |
Recruit key informants from CMCS, Voxiva/Zero to Three, state Medicaid agencies, and active outreach partners |
November – December 2017 |
Recruit health care providers for key informant interviews and recruit focus group participants |
November – December 2017 |
Conduct key informant interviews |
January – March 2018* |
Conduct provider interviews and focus groups |
January – March 2018* |
* Survey, key informant interview, and focus group protocols were pretested. The pretest was conducted with nine or fewer respondents per protocol per year.
Approval not to display the expiration date for OMB approval is not requested.
No exceptions to the certification statement are being sought.
1 Cole-Lewis, H., and T. Kershaw. “Text Messaging as a Tool for Behavior Change in Disease Prevention and Management.” Epidemiologic Reviews, vol. 32, no. 1, March 2010, pp. 56–69.
2 Gazmararian, J.A., B. Yang, L. Elon, M. Graham, and R. Parker. “Successful Enrollment in Text4Baby More Likely with Higher Health Literacy.” Journal of Health Communication, vol. 17, no. 3, 2012, pp. 303–311.
3 Hou, M.Y., S. Hurwitz, E. Kavanagh, J. Fortin, and A.B. Goldberg. “Using Daily Text-Message Reminders to Improve Adherence with Oral Contraceptives: A Randomized Controlled Trial.” Obstetrics and Gynecology, vol. 116, no. 3, November 2010, pp. 633–640.
4 Evans, W.D., L.C. Abroms, R. Poropatich, P.E. Nielsen, and J.L. Wallace. “Mobile Health Evaluation Methods: The Text4Baby Case Study.” Journal of Health Communication, vol. 17, suppl. 1, May 2012b, pp. 22–29.
5 Evans, W.D., L.C. Abroms, R. Poropatich, P.E. Nielsen, and J.L. Wallace. “Mobile Health Evaluation Methods: The Text4Baby Case Study.” Journal of Health Communication, vol. 17, suppl. 1, May 2012b, pp. 22–29.
6 Evans, William Douglas, Jasmine L. Wallace, and Jeremy Snider. “Pilot Evaluation of the Text4baby Mobile Health Program.” BMC Public Health, vol. 12, no. 1031, November 2012, pp. 1–10.
7 Secondary data sources are Voxiva Text4baby administrative data, state Medicaid administrative data, state vital records data; and state-reported maternity Core Set measures (from CHIP Annual Report Template System [CARTS]).Through its contractor for this evaluation, CMCS has received clearance for the collection of data from these secondary data sources from the New England Institutional Review Board (IRB) (14-432).
8 All primary and secondary data for this evaluation will be collected and analyzed by CMCS’s contractor.
9 The purpose of collecting Voxiva Text4baby administrative data is to examine enrollment trends and key characteristics of Text4Baby subscribers who are or are not enrolled in Medicaid. See Appendix D for data specifications.
10 The purpose of collecting state Medicaid data is to identify members of the study population and their infants, to assess health care utilization among Text4baby subscribers and nonsubscribers, and to develop outcome measures related to the Maternity Core Set. The contractor plans to construct measures of frequency of ongoing prenatal care, timeliness of prenatal care, postpartum care rate, Medicaid covered infant dental visits, and well-child visits in the first 15 months of life. See Appendix E for data specifications.
11 The purpose of collecting vital records data is to assess birth outcomes among Text4baby subscribers and nonsubscribers in the four participating states. To this end, vital records data will be used to calculate several clinical outcomes of interest and to collect key demographic, geographic, and health risk information on the study sample. See Appendix F for data specifications.
12 The purpose of collecting Maternity Core Set measures is to evaluate trends over time in aggregate state-level outcomes important to this evaluation in the four states participating in the pilot compared with other states not participating in the pilot. Maternity Core Set measures can be found by visiting the following web address: http://www.medicaid.gov/medicaid-chip-program-information/by-topics/quality-of-care/downloads/maternity-core-set.pdf
13 Holbrook, A. L., J.A. Krosnick, and A. Pfent. “The Causes and Consequences of Response Rates in Surveys by the News Media and Government Contractor Survey Research Firms,” in Advances in Telephone Survey Methodology (edited by J. M. Lepkowski, C. Tucker, J. M. Brick, E. D. d. Leeuw, L. Japec, P. J. Lavrakas, M. W. Link and R. L. Sangster). Hoboken, NJ: John Wiley & Sons, Inc., 2007.
14 Singer, E., N. Gebler, T. Raghunathan, J. Van Hoewyk, and K. McGonagle. “The Effect of Incentives on Interviewer-Mediated Surveys.” Journal of Official Statistics, vol. 15, no. 2, 1999, pp. 217–230.
15 Singer, E., and C. Ye. “The Use and Effects of Incentives in Surveys.” The ANNALS of the American Academy of Political and Social Science, vol. 645, no. 1, 2013, pp. 112–141.
16 Rodgers W.L. “Effects of increasing the incentive size in a longitudinal study.” Journal of Official Statistics, vol. 27, no. 2, 2011, pp. 279–299.
17 Colicchia, M., M. Czaplewski, and A. Jaszczak. “Refusal Conversion Incentives and Participation in a Longitudinal Study of Older Adults.” Survey Practice, vol. 5, no. 3, 2012. Available at http://surveypractice.org/index.php/SurveyPractice/article/view/22/pdf_1. Accessed January 15, 2015.
18 Enrollment rates will be calculated using Voxiva-Medicaid linked data.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Office of Management and Budget Evaluating a Pilot Mobile Health Program, Supporting Statement Part A |
| Subject | OMB |
| Author | Mathematica Policy Research |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |