Pretest Report and Recommendations

ATTAChment B

pretest report and recommendations

This page has been left blank for double-sided copying.

Text4baby Healthy Pregnancy and Parenting Survey:

Pretest Report and Recommendations

Overview of text4baby Evaluation Pretest

Text4baby is a text-message program for pregnant women and new mothers to enhance their health knowledge, behaviors, and outcomes in an effort to reduce disparities in maternal and newborn health. Health education messages are intended to be brief, easy-to-understand, and timed to a woman’s stage of pregnancy or the baby’s age. This evaluation seeks to understand how well text4baby is working, including awareness of the program, enrollment and disenrollment patterns, consumer satisfaction, and role in educating pregnant women and new mothers about having a healthy pregnancy and parenting experience. The ultimate goal of the study is to shed light on the potential of mobile health technology to improve health outcomes.

Differences between Pretest and Survey Respondents. The Healthy Pregnancy and Parenting (HPP) Survey will consist of two longitudinal data collections: Round 1 for pregnant women, and Round 2 for postpartum women. The survey instruments for the two rounds are substantially different because the questions are tailored to the pregnancy and postpartum periods. The same women will be interviewed in both rounds, with the Round 2 survey being conducted approximately nine months after the Round 1 survey. Survey respondents may be of any age and may speak English or Spanish. In the interests of time and cost efficiency, the pretest excluded women less than 18 years of age and non-English speakers. We also conducted the pretest of the Round 1 and Round 2 instruments with different women, due to the lack of a longitudinal component in the pretest.

Pretest Purposes. From the respondent perspective, we wanted to determine whether the question language was clear and understandable, the instrument flow was smooth and “made sense,” and the response categories offered enough options (but not so many to be confusing). From the interviewer perspective, we wanted to learn whether the skip and range instructions were accurate and clear and whether response categories were codeable. Interviewers are excellent judges of language level and question presentation and we wanted to hear their advice about these issues as well. Finally, we wanted to test the average length of the surveys to determine whether they were within the budgeted interview length.

Pretest Sample. To identify an adequate sample for the HPP survey pretest, we contacted a Healthy Start program grantee that had been a text4baby partner long enough to have sufficient text4baby subscribers to help meet the pretest goals. The two survey instruments were pretested on separate samples because of the substantial differences in the questionnaires and the applicability to different populations. Our pretest sample outcomes are shown in Table 1.

Table 1. HPP Survey Pretest Sample, by Instrument and text4baby (t4b) Subscriber Status

Pretest Sample	Round 1 Instrument (Pregnant Women)		Round 2 Instrument (Postpartum Women)
	t4b Subscriber	t4b Non-subscriber	t4b Subscriber	t4b Non-subscriber
Completed Surveys	3	4	2	6

Conducting the Pretest. We conducted 7 interviews with pregnant women (3 text4baby subscribers and 4 non-subscribers) and 8 interviews with postpartum women (2 text4baby subscribers and 6 non-subscribers). The interviews took place during a one-week period between July 18th and 25th, 2011. All interviews were conducted by telephone using paper versions of the two instruments. Completed hardcopy instruments were faxed to the team and electronic recordings were posted on an internal shared drive. All materials were de-identified. The entire research team had the advantage of concurrently seeing faxed copies of the completed instruments and listening to the recorded responses. The availability of recorded interviews proved instructive as we could hear respondent hesitation and confusion, and identify inconsistencies in responses across questionnaire sections that might not appear on the paper version of the completed survey. After reviewing the first 5 faxed/recorded interviews, we held an interim debriefing with the interviewers and their supervisors to discuss issues with the text4baby enrollment questions that had been identified in listening to the recorded interviews. Based on this debriefing, we implemented several mid-course corrections to the questionnaires and then resumed the pretesting later that afternoon. The second phase of pretesting went very smoothly for both the respondents and interviewers. After completion of the pretest, we held a final debriefing and three additional areas for improvement were identified and changes were recommended: (1) the question on signing up for text4baby, (2) the health insurance question sequence, and (3) the usual source of prenatal care question.

Implications of Pretest Timings. An important goal of the pretest was to assess the length of the questionnaire, taking into account differences that are a function of the pretest environment. In particular, it takes more time to conduct an interview using a paper instrument because the interviewer must manage the logistics of skips and range checks, a process normally handled by the CATI program. We estimate that this process added 5 to 10 minutes to the pretest interview length (on average, 7 minutes). Table 2 shows the average length of the pretest interviews by instrument and major pathways. The average length of the Round 1 instrument was estimated to be 17 minutes in CATI, and the Round 2 instrument was estimated at 19 minutes. As expected, the average length of the interview for those who have never heard of text4baby was shorter than for those who have heard of text4baby. The average length of the pretest interview was also affected by respondents’ initial confusion with the text4baby questions, as discussed below.

Table 2. Average Timing of the Pretest Interviews, by Instrument and Major text4baby Pathway

Instrument and Major text4baby Pathways	Number of Pretest Interviews Completed	Average Minutes to Complete Pretest via Hardcopy Instrument	Estimated Minutes to Complete in CATI (Minus 7 Minutes)
Round 1 Instrument (Pregnant Women)	6	24	17
Never heard of t4b	2	19	12
Heard of t4b but did not subscribe	1	30	23
Heard of t4b and subscribed	3	25	18
Round 2 Instrument (Postpartum Women)	8	26	19
Never heard of t4b	2	24	17
Heard of t4b but did not subscribe	4	25	18
Heard of t4b and subscribed	2	29	22

During the pretest, we did not test the introductory portion of the questionnaire, where interviewers telephone the respondent to gain her cooperation and screen to determine eligibility and consent. The sample received from the Healthy Start grantee included a list of phone numbers and names of women who had already consented to participate in the pretest and had been determined to be eligible. To remedy this, we tested the introductory portion with Mathematica staff using five scenarios and found that on average it took somewhat less than 1.5 minutes to complete. This brings the Round 1 instrument timing to an average of 19.5 minutes and the Round 2 instrument timing to an average of 20.5 minutes (which is within the budgeted amount of 20 minutes per respondent).

Recommended Changes to the text4baby Survey Instruments

We first discuss the most prominent change that we recommend to both survey instruments to ease respondents into the instrument content and improve the reliability of information on text4baby awareness and enrollment. We then discuss recommended changes to other questions.

1. Question on text4baby Awareness

The pretest began with the interviewers reading the respondents their rights as a participant. The first substantive section of the pretest assessed whether the respondent had ever heard about text4baby, as follows:

1

HS B2 modified

.1 Now, I’d like to ask you about a program called Text4baby. Text4baby is a program in which pregnant women and women with new babies can sign up to receive free text messages on their cell phones. Messages include health tips, reminders, and information about other resources.

Did you ever hear of the text4baby program before today?

PROBE: Text4baby is a program that sends free text messages to pregnant women giving them health tips and reminders during pregnancy and information about services they might want to use. You need to be signed up to get the messages. Have you heard about this program?

YES 1

N

RECORD ANSWER AT Q1.1 ON COVER

O 0 GO TO Q4.1

DON’T KNOW d GO TO Q4.1

REFUSED r GO TO Q4.1

Discussion

The research team reviewed the first 5 completed questionnaires and listened to the recordings and concluded that several respondents did not respond accurately to this question. The team felt that respondents needed a transition between “your rights as a participant” and “have you ever heard of text4baby.” In addition, the interviewers felt that the language in this question was wordy and lacked clarity. In three of the first five interviews, the respondents initially indicated either they had not heard of the text4baby program or were not signed up for text4baby, but in later questions, indicated they had heard of it or were signed up. As a result, in three interviews, interviewers re-asked the initial questions and then followed the appropriate skip patterns based on the corrected information. (We estimate that this added about 1 minute, on average, to the length of the interview.) For the second half of the pretest, we made two changes to the entry questions: (1) starting the survey with basic questions about cell phone and social media use; and (2) simplifying the wording of the text4baby questions. These changes are described in greater detail below.

After the first 5 completed interviews, we moved forward Section 4: Cell Phone and Social Media Use to serve as a transition between “your rights as a participant” and “have you ever heard of text4baby.” Note that Start 1 was formerly Question 4.1 and Start 2 was formerly Question 4.2. If HRSA agrees to this recommended change, we will renumber the sections and questions to accommodate this change.

S

New

question

tart 1. The first questions ask about your cell phone use.

• Do you have your own cell phone?

YES 1

NO 0 GO TO Q1.1

DON’T KNOW d GO TO Q1.1

REFUSED r GO TO Q1.1

S

Pew Cell Phones American Adults Q17 modified

tart 2. Do you ever use your cell phone to…

	CODE ONE RESPONSE FOR EACH ROW
	yes	no	DK	REF
a send or receive text messages?	1	0	d	r
b access a social networking site like MySpace, Facebook or LinkedIn.com?	1	0	d	r
c access Twitter?	1	0	d	r

Next, we simplified the language in Question 1.1 of both surveys. Following this change, we did not detect any errors in responses to questions about text4baby enrollment status. Table 3 shows the original question 1.1 text side-by-side with the revised text.

Table 3. Proposed Revisions to the text4baby Awareness Question

Original Question Text

Revised Question Text

1.1 Now, I’d like to ask you about a program called Text4baby. Text4baby is a program in which pregnant women and women with new babies can sign up to receive free text messages on their cell phones. Messages include health tips, reminders, and information about other resources. Did you ever hear of the text4baby program before today? PROBE: Text4baby is a program that sends free text messages to pregnant women giving them health tips and reminders during pregnancy and information about services they might want to use. You need to be signed up to get the messages. Have you heard about this program?

1.1 Before today, did you ever hear of a program called text4baby that sends pregnant women and new mothers free text messages on their cell phones? PROBE: IF R SAYS NOT SURE OR SEEMS UNCERTAIN: Women can sign up to receive free text messages about having a healthy pregnancy and baby. The messages include health tips, reminders, and information about other resources. Did you ever hear of the text4baby program before today?

2. Questions on Signing Up for text4baby

We found that one respondent experienced a problem signing up for text4baby that was not addressed in our original questionnaire.

Discussion

One respondent was confused by the original questions 1.2 and 1.2a because she had tried to sign up but received error messages from the text4baby program. We recommend adding the following unread option (TRIED BUT WAS UNABLE TO SIGN UP) to question 1.2 and a new question 1.2a (why were you unable to sign up) to cover the situation where someone had tried to sign up but was not able to. The original question 1.2a becomes question 1.2b after the revisions proposed below. Table 4 shows the changes we recommend.

Table 4. Proposed Revisions to the text4baby Sign-up Questions

Original Question Text

Revised Question Text

1.2 Have you ever signed up for text4baby? PROBE: Text4baby is the text messaging program that offers tips about having a healthy pregnancy. YES N)  go to Q1.4 DK  go to Q1.4 REF  go to Q1.4 1.2a How did you sign up for text4baby? Did you send a text message on your cell phone, sign up on the Internet, or sign up some other way? Cell phone Internet Someone else signed me up Other (Specify)

1.2 Have you ever signed up to receive free text4baby messages on your cell phone? PROBE: Text4baby gives women tips about having a healthy pregnancy and baby. YES  Q1.2b TRIED BUT WAS UNABLE TO SIGN UP  go to Q1.2a NO  go to Q1.4 DK  go to Q1.4 REF  go to Q1.4 1.2a Why were you unable to sign up for text4baby when you tried? Specify__________ Go to Q1.4 1.2b How did you sign up for text4baby? Did you send a text message on your cell phone, sign up on the Internet, or sign up some other way? Cell phone Internet Someone else signed me up Other (Specify)

3. Questions on Health Insurance Coverage

Questions on the health insurance status of parents and children are complex and lengthy, in order to elicit accurate information on whether individuals have coverage and if so, what kind of coverage. The health insurance questions included in the pretest instruments were taken from the 2006 Healthy Start Participant Survey. In the Round 1 questionnaire, insurance questions apply to the pregnant woman and are numbered 7.10a – 7.10e. If the woman does not report that she is covered by health insurance, she is asked a verification question (7.10f) and if she indicates she does have coverage, she is asked what kind of coverage she has (7.10g). In the Round 2 questionnaire, the same sequence of insurance questions is asked twice, once for the woman (7.2a – 7.2g) and once for her infant (7.3a – 7.3g).

Based on the pretest, we recommend revising the wording and sequencing of the health insurance questions to be consistent with those that will be asked in the survey for the ASPE-funded 10-state Children’s Health Insurance Program (CHIP) evaluation (currently undergoing the OMB clearance process). The CHIP survey includes a similar population (low-income families).

Discussion

While the majority of insurance questions in the Round 1 (Pregnant Women) and Round 2 (Postpartum Women) surveys did not cause the respondents or interviewers any difficulties, there was one question that caused some confusion; the question involved the unexplained phrase “purchased directly” (Original Q7.2c). The question is more clearly stated in the CHIP question text, which separates private insurance from a current or past employer or union versus insurance purchased directly. Thus, we opted to use a more recent series of insurance questions that addresses this confusion.

Table 5 shows the sequencing of insurance questions in the original text4baby questionnaire and the recommended changes from the CHIP survey. Rather than repeat all 3 sets of questions, we show the original insurance questions from the Round 1 (Pregnant Women) survey and note that we plan to retain the two verification questions (7.2f-g) and ask the same questions about health insurance coverage for the infant in the Round 2 survey.

Table 5. Proposed Revisions to the Health Insurance Questions

Original Question Text

Revised Question Text

7.2a The next questions are about your health insurance. Are you covered by Medicaid? PROBE: This is a health insurance program for low-income families. 7.2b Are you covered by [State CHIP program]? 7.2c Are you covered by private insurance, that is, health insurance obtained through employment or unions or purchased directly? 7.2d Are you covered by military health care, called CHAMPUS, CHAMP-VA, or TRICARE? 7.2e Are you covered by any other health insurance or health care plan that pays for services obtained from hospitals, doctors, or other health care providers? 7.2f It appears that you do not currently have health insurance coverage to help pay for services from hospitals, doctors, and other health care providers. Is that correct? [IF NO, GO TO Q7.2g] 7.2g What kind of health coverage do you have?

7.2a The next questions are about your health insurance. Are you covered by insurance from a current or past employer or union? 7.2b Are you covered by a private insurance plan purchased directly from an insurance company? Do not include plans that only provide extra cash while in the hospital or those that cover only one type of service, such as dental care, vision care, nursing home care, or accidents. 7.2c Are you covered by [State CHIP program]? 7.2d Are you covered by Medicaid or a Medicaid HMO, the government assistance program for people in need? 7.2e Are you covered by some other type of coverage, I have not yet mentioned? [IF YES, SPECIFY] NOTE: The CHIP survey does not include the verification questions (7.2f-g), but we plan to retain them to increase the accuracy of uninsured rates for women and infants in the text4baby evaluation. NOTE: 7.3a – g will ask the same questions about infant coverage.

4. Questions on Usual Source of Prenatal Care

Interviewers felt that the question on the usual source of prenatal care (Q5.3) was unclear because it embedded a probe to determine whether there was one usual place or more than one place.

Discussion

Table 6 shows the proposed revisions to the usual source of prenatal care questions. The revised question text asks whether the respondent has one place she usually goes to for prenatal care, eliminating the ambiguity of the original question which asked about one or more places. As a result, question 5.4 follows more logically when it asks where the respondent usually goes for prenatal care.

Table 6. Proposed Revisions to the Usual Source of Prenatal Care Questions

Original Question Text

Revised Question Text

5.3 Do you have one or more places you usually go to for your prenatal care? PROBE: IF YES, is that only one place or more than one place? Yes, one place Yes, more than one place No DK REF If Q5.3 = yes, one place or more than one place: 5.4 Where do you usually go for prenatal care? Is it a… Hospital clinic Health department clinic Community health center Private doctor’s office or HMO Clinic, or Some other place (Specify)

5.3 Do you have one place you usually go to for your prenatal care? Yes No DK REF 5.4 Where do you usually go for prenatal care? Hospital clinic Health department clinic Community health center Private doctor’s office or HMO Clinic, or Some other place (Specify)

Conclusion

In summary, the pretest provided important feedback about the clarity of the text4baby questions, which are essential to accurately measuring text4baby awareness and enrollment. The pretest also revealed that the interview length was within the budgeted range. Based on the pretest, we recommend several changes to the instrument that will improve the questionnaire flow, clarity, and brevity, and eliminate potential sources of ambiguity in questionnaire wording. All told, the pretest helped improve the survey substantially. We look forward to HRSA’s feedback on the changes recommended as a result of the pretest.