Pretest Results

Attachment H

Pretest Results

Preconception, Pregnancy, and Parenting (3P’s) Information Form

National Healthy Start Program Survey

Healthy Start Community Action Network (CAN) Survey

I. Preconception, Pregnancy, and Parenting (3P’s) Information Form:
Pretest Report and Recommendations

A. Overview of 3P’s Information Form

The 3P’s Information Form represents a uniform data collection instrument that all Healthy Start grantees will use to collect data for monitoring. Data will be collected through the form for all Healthy Start participants. We intend to collect data on Healthy Start participants at enrollment and once annually thereafter. Data collected through the form will allow grantees to perform real-time internal analysis and allow the Maternal and Child Health Bureau (MCHB), Division of Healthy Start Program Services (DHSPS) to pool data and get a snapshot of implementation and outcomes at both the national and project levels. The information gathered at Healthy Start sites will also feed into the program’s evaluation.

Comparison sites will also use the form to obtain information about women and their children and families for purposes of evaluation. Women for inclusion in the evaluation include those that are four to seven months postpartum at comparison sites at the time of data collection. Comparison sites will be instructed to collect information for women served during the months of approximately March through May of 2015, 2017, and 2019. The information gathered will be essential for the national evaluation of the Healthy Start.

Pretest purposes. By pretesting the participant form, we hoped to gain information on the following:

• The average time it takes to administer the form

• Program participants’ understanding of the questions on the form

• Any questions that could be deleted or revised to improve clarity

Pretest sample. To identify participants for the pretest, we contacted three Healthy Start grantees and asked them to help us recruit two English-speaking individuals per site. Grantees identified participants, gained consent from participants to release their names and contact information, and provided this information to Mathematica Policy Research. All pretest participants had at least one live birth in the past. One of the six participants was pregnant at the time of administering the form. Tables A.1a and A.1b provide more information about pretest participants.

Table A.1.a. Life Course Stage of Pretest Participants at Time of Interview

Life course stage of participant	Number of completes
Pregnant	1
Less than 3 months postpartum	2
4–12 months postpartum	2
13 months postpartum	1

Table A.1.b. Number of Live Births Reported by Pretest Participants at Time of Interview

Number of live births reported by participant	Number of completes
One live birth	3
Two live births	1
Three or more live births	2

Adapting the form for the pretest. A few modifications were made to the survey to fit the constraints of the pretest.

• Administration. The participant form is designed to be administered in-person by Healthy Start staff to program participants using a web-based application. When implemented, program staff will be trained on the instrument to ensure consistent administration. Due to lack of resources and time to train Healthy Start staff across the three grantee sites, a Mathematica staff member administered the form with participants by phone.

• Mode. The form will be programmed as a web-based instrument, such that the individual administering the form can enter the participants’ responses into the fields and skip patterns and fill text will be programmed accordingly. For the pretest, we used a paper copy of the form.

• Changing text from CAN to consortium. At item I5, respondents are asked whether they participate in the Healthy Start Community Action Network. Given that the term Community Action Network is new and not used to describe collaborative efforts in previous grant years, we changed the question wording to ask if they participate in the consortium to align better with current terminology.

Conducting the pretest. Six individuals completed the pretest by phone at times convenient for participants between January 28 and February 14, 2014. Individuals from California Border Healthy Start participated during their home visits while their navigators were present. Participants at Low Country Healthy Start completed the pretest at their local Healthy Start site. Baltimore Healthy Start participants completed the pretest over their cell phones at times that they identified as being convenient. A Mathematica staff member initiated the calls at the specified times, read participants their rights and gained consent, and documented responses and notes on a copy of the paper survey. When participants agreed, a recording devise was used to record the calls for note-taking purposes.

We administered the form starting with item A1 after the Admin Only section and completed the form for timing. After documenting the timing, we then went back to ask follow-up questions about certain items that seemed to cause confusion for respondents.

We asked each of the three Healthy Start grantees whether their participants would prefer Walmart or Target gift cards—all three grantees requested Walmart cards. We gave participants $25 gift cards upon their completion of the pretest.

Pretest timing. One of the goals of the pretest was to assess the length of the form. For timing purposes, we started recording the time at A1 (after reading pretest participants their rights) and stopped the clock at the end of the form. Among the six participants, the range of completion times was 31 minutes to 62 minutes and the mean completion time was 38 minutes. Although we had hoped to pretest the instrument with fluent English speakers, one participant did not speak fluent English and needed assistance from her navigator to translate certain words into Spanish. Administering the form to this individual took 62 minutes. Excluding her interview, the average completion time was 33.5 minutes.

Completion time among pretest participants appeared to be tied to number of live births because this impacted the number of times we asked items E5–E13, which is asked for first, second, and most recent live birth. Had any of the participants not had previous live births, completion time would have been significantly lower because we would not have asked items in Sections D, E, and F, along with three items in Section H. In addition, the form is designed to be administered annually for Healthy Start participants, and, in subsequent administration of the form, women will be skipped out of questions in Sections D, E, and F if she had not had a new live birth in the past year.

Below we outline items to delete and ways to streamline question text to decrease the total administration time to 30 minutes.

B. Recommended Changes to the Healthy Start Participant Form

1. Deletions

We suggest deleting the following items:

• A1: What is your name? Please tell me your first name, middle initial, and last name. Based on the decision that name will not be collected and only participants’ unique ID will be collected, we suggest deleting the first item on the form, which asks participants to provide their full names. Grantees will keep separate files with names and unique ID, but participant name will not be collected by MCHB in the monitoring data system or at any other time.

• E1B: How many months or weeks had you been pregnant when your last child was delivered vaginally? At item E1b, we collect information on participants’ gestational age for her last vaginal delivery. At E7, we gather information about participants’ gestational age at their child’s birth for their first, second, and most recent births. During the pretest, two participants noted at item E7 that we had already asked them this question. Although deleting this item will not allow us to directly link gestational age with delivery method, it will reduce overall duplication and prevent respondent confusion from being asked questions that seem repetitive questions.

• E2B: How many months or weeks had you been pregnant when your last child was delivered by C-section? We suggest deleting E2B because it duplicates information collected at E7 (see above about E1B).

• E11: What was the Apgar score of your [first/ second/ last] child? There should be two numbers. Out of the six pretest participants, none was able to provide Apgar scores for their children. Only one participant knew what an Apgar score is but could not remember the numbers. Based on feedback from pretest participants, we do not think this item will yield useful data and suggest deleting it.

• E13: How many emergency room visits did your [first/ second/ last] child have during his or her first year? Pretest participants found this item difficult to answer. Women with many children could not remember how many visits they had for each child in question. Women who had given birth within the past year found the question wording to be confusing. Therefore, we suggest deleting this question as it would likely not yield accurate information.

• G9: Did you ever get information about breastfeeding? If yes, was it from Healthy Start or some other source? During the pretest, we identified this item as a duplicate to item F3 (Where did you get information about breastfeeding? WIC nutrition program; Hospital staff gave client the information about breastfeeding; Healthy Start staff gave client information about breastfeeding; No information received about breastfeeding). The question at F3 enables us to gather more information about source of information for breastfeeding so we suggest retaining F3 and deleting G9.

2. Text Revisions

During the pretest, a few items caused confusion across the pretest participants. In order to improve the clarity of the items on the form and ensure that all possible response options are included, we recommend implementing the following revisions to question text:

• E9: Nursery Question. During the pretest, all six participants were confused by the wording of item E9. The response categories did not include an option for women whose newborn stayed in the room and did not spend time in a nursery. Additionally, respondents said that the question phrasing led them to believe that the question only applied to unhealthy babies. We suggest the following revisions to create more comprehensive response options and clarify the wording.

Proposed Revisions to E9

Original Question Text

Revised Question Text

E9. Did your [first/ second/ third] child spend any time in a full term nursery, a special care nursery, or a neonatal intensive care unit (also known as a NICU)? Select one only. 1  Full term nursery 2  Special care nursery 3  Neonatal intensive care unit d  DON’T KNOW r  REFUSED

E9. Which of the following types of nursery did your [first/ second/ last] child spend time in after birth: a full term nursery, a special care nursery, a neonatal intensive care unit (also known as a NICU), or did your infant stay in the room with you? Select one only. 1  Full term nursery 2  Special care nursery 3  Neonatal intensive care unit 4  Stayed in room 5  Other:_________________ d  DON’T KNOW r  REFUSED

• E14: Individuals present for delivery. When asked who was in the room during their last delivery (E14), multiple pretest participants responded “my doctor.” When probed whether a nurse, midwife, doula, spouse or partner, or family members were also present, most participants shared that additional individuals were also present for delivery. To obtain complete data, we suggest modifying the text such that we ask about each type of person individually.

Proposed Revisions to E14

Original Question Text

Revised Question Text

E14. Who was in the room with you at your delivery?

PROBE IF CLIENT HAD MORE THAN ONE LIVE BIRTH: Please tell me for your last child.

Select all that apply.

1 □ OB/ GYN

2 □ Nurse

3 □ Midwife

4 □ Doula

5 □ Spouse or partner

6 □ Family member or friend

7 □ Other

d □ DON’T KNOW

r □ REFUSED

E14. Which of the following types of people were in the room with you at your last delivery?

PROBE IF CLIENT HAD MORE THAN ONE LIVE BIRTH: Please tell me for your last child.

	SELECT ONE RESPONSE PER ROW
Was there a(n)…	YES	NO	DON’T KNOW	REFUSED
a. OB/GYN?	1 	2 	d 	r 
b. Nurse?	1 	2 	d 	r 
c. Midwife?	1 	2 	d 	r 
d. Doula?	1 	2 	d 	r 
e. Partner or spouse?	1 	2 	d 	r 
f. Family member or friend?	1 	2 	d 	r 
e. Another person: _______?	1 	2 	d 	r 

• E15: One week checkup question. During the pretest, item E15 caused confusion. Pretest participants thought the question was asking for the type of health professional that saw their child for a one-week checkup rather than whether or not their infant had a one-week checkup. We recommend revising the question text to better align with the response options (yes or no).

Proposed Revisions to E15

Original Question Text	Revised Question Text
E15. Was your child seen by a doctor, nurse, or other health care worker for a one-week checkup after he or she was born? PROBE IF CLIENT HAD MORE THAN ONE LIVE BIRTH: Please tell me for your last child. Select one only. 1  Yes 0  No d  DON’T KNOW r  REFUSED	E15. Was your child seen by a health care worker, like a doctor or nurse, for a one-week checkup after he or she was born? PROBE IF CLIENT HAD MORE THAN ONE LIVE BIRTH: Please tell me for your last child. Select one only. 1  Yes 0  No d  DON’T KNOW r  REFUSED

• E16: Postpartum visit question. One pretest participant had given birth less than two weeks prior to completing the pretest. She had not yet received a post-partum checkup because it was still too early for one. We suggest rewording the question to not specify the time period for the postpartum check-up.

Proposed Revisions to E16

Original Question Text	Revised Question Text
E16. Since your child was born, have you had a postpartum checkup for yourself? A postpartum checkup is the regular checkup a woman has about six weeks after she gives birth. Select one only. 1  Yes 0  No d  DON’T KNOW r  REFUSED	E16. Since your child was born, have you had a postpartum checkup for yourself? A postpartum checkup is the regular checkup a woman has after she gives birth. Select one only. 1  Yes 0  No d  DON’T KNOW r  REFUSED

• Section G: Health Education question series. When presented with the health education question series in Section G, five of the six pretest participants tried to say that they received information from both Healthy Start and some other source; however, this was not a response option during the pretest. We recommend adding a response option and slightly modifying question text to account for the possibility of receiving information from Healthy Start and another source.

Proposed Revisions to Section G (Using G1 as an Example)

Original Question Text	Revised Question Text
G1. Did you ever get information about taking folic acid or a multivitamin during pregnancy? IF YES: Was it from Healthy Start or another source, or both? Select one only. 1  Yes, from Healthy Start 2  Yes, from another source 0  No d  DON’T KNOW r  REFUSED	G1. Did you ever get information about taking folic acid or a multivitamin during pregnancy? IF YES: Was it from Healthy Start, another source, or both? Select one only. 1  Yes, from Healthy Start only 2  Yes, from another source only 3  Yes, from Healthy Start and some other source 0  No d  DON’T KNOW r  REFUSED

• Usual Source of Care Series (H1, H1a, H2, H2a, H5, H5a, H6, H6a). Pretest participants were confused by the questions at the beginning of Section H regarding usual source of care for sick visits and well visits (H1, H1a, H2, H2a for adults and H5, H5a, H6, H6a for children). Both the well-visit and sick-visit questions contained the phrase, “Where do you go for health advice?” We suggest revising the questions as described below to remove the duplicated text and make the questions clear to respondents.

Proposed Revisions to H2, H2a, H3, H3a and H5, H5a, H6, H6a

Original Question Text	Revised Question Text
H1. Is there a place that you usually go when you are sick or you need advice about your health? Select one only. 1  Yes 0  No - SKIP TO H3 d  DON’T KNOW r  REFUSED H1a. What type of place do you usually go to when you are sick or need advice about your health? Select one only. 1  Clinic or health center 2  Doctor’s office or an HMO 3  Hospital emergency room 4  Outpatient department or urgent care d  DON’T KNOW r  REFUSED H2. Is there a place that you usually go for health advice or a checkup? Select one only. 1  Yes 0  No – SKIP TO H4 d  DON’T KNOW r  REFUSED H2a. What type of place do you usually go for health advice or a checkup? Select one only. 1  Clinic or health center 2  Doctor’s office or an HMO 3  Hospital emergency room 4  Outpatient department or urgent care d  DON’T KNOW r  REFUSED H5. Is there a place that you usually go when your child is sick or you need advice about your child’s health? Select one only. 1  Yes 0  No - SKIP TO H6 d  DON’T KNOW r  REFUSED	H1. Is there a place that you usually go when you are sick? Select one only. 1  Yes 0  No - SKIP TO H3 d  DON’T KNOW r  REFUSED H1a. What type of place do you usually go to when you are sick? Select one only. 1  Clinic or health center 2  Doctor’s office or an HMO 3  Hospital emergency room 4  Outpatient department or urgent care d  DON’T KNOW r  REFUSED H2. Is there a place that you usually go for a checkup? Select one only. 1  Yes 0  No – SKIP TO H4 d  DON’T KNOW r  REFUSED H2a. What type of place do you usually go to for a checkup? Select one only. 1  Clinic or health center 2  Doctor’s office or an HMO 3  Hospital emergency room 4  Outpatient department or urgent care d  DON’T KNOW r  REFUSED H5. Is there a place that you usually go when your child is sick? Select one only. 1  Yes 0  No - SKIP TO H6 d  DON’T KNOW r  REFUSED
H5a. What type of place do you usually go to when your child is sick or you need advice about your child’s health? Select one only. 1  Clinic or health center 2  Doctor’s office or an HMO 3  Hospital emergency room 4  Outpatient department or urgent care d  DON’T KNOW r  REFUSED H6. Is there a place that you usually go for health advice or routine preventative care for your child? Select one only. 1  Yes 0  No - SKIP TO H7 d  DON’T KNOW r  REFUSED H6a. What type of place do you usually go for health advice or routine preventative care for your child? Select one only. 1  Clinic or health center 2  Doctor’s office or an HMO 3  Hospital emergency room 4  Outpatient department or urgent care d  DON’T KNOW r  REFUSED	H5a. What type of place do you usually go to when your child is sick? Select one only. 1  Clinic or health center 2  Doctor’s office or an HMO 3  Hospital emergency room 4  Outpatient department or urgent care d  DON’T KNOW r  REFUSED H6. Unchanged H6a. Unchanged

• H8 Vaccination Question. Only one respondent was able to answer the question about which vaccinations her child had and that was because she had documentation on hand. Based on pretest participants’ inability to recall which vaccinations their child received, we suggest revising the question text to reflect whether or not participants’ last child received vaccines and when they last received them, rather than asking participants to recall all the vaccines their child received.

Proposed Revisions to H8

Original Question Text

Revised Question Text

H8. Which vaccinations has your child gotten? Vaccinations are usually a shot the doctor gives to keep your child healthy and prevent your child from getting sick later. Select all that apply. 1 □ HepA 2 □ HepB 3 □ RV 4 □ Tetanus, diphtheria, pertussis (Td/Tdap, DTaP) 5 □ Hib 6 □ PCV 7 □ IPV 8 □ Influenza (flu) 9 □ Measles, mumps, rubella (MMR) 10 □ Varicella (chickenpox) d □ DON’T KNOW r □ REFUSED

H8. Has your child been given any vaccines or baby shots yet? Please do not include the shots given when your baby was born. Select one only. 1  Yes 0  No - SKIP TO H9 d  DON’T KNOW r  REFUSED H8a. How old was your child the last time he or she got vaccines or shots? | | | WEEKS OR | | | MONTHS d  DON’T KNOW r  REFUSED

3. Revisions to Probes

The instrument contains some text in the form of probes. Unlike regular question text that should be read to all participants, probes should only be used if the respondent seems to require more information to answer the question. During the pretest, we identified questions on the form that could benefit from the addition of probes. We also identified questions where regular question text could be converted to probes because it is not relevant to most respondents.

• Adding probes for all questions about the last pregnancy/ child (D2, F2, F4, F5, F6, F7, F8, F9, F10, F12a, F12b, F12c, F13a, F13b, F13c, F14). The form collects data on participants’ first, second and last child in Section E. For the remainder of the survey, however, the questions pertain to the respondent’s most recent pregnancy or most recent birth. Although we include instructions about this at the beginning of the relevant sections, respondents with multiple children occasionally became confused during the pretest and asked for clarification. We suggest adding probes to these items so if participants ask which child they should provide an answer for, staff administering the form will have a consistent, prescribed response on the screen.

Examples of Proposed Addition

Revised Question Text

D2. Now, I would like to ask you a few questions about what you did when you were pregnant before. Please answer these questions for the last time you were pregnant.

How many weeks or months pregnant were you when you had your first visit for prenatal care during your last pregnancy? Do not count a visit that was only for a pregnancy test or only for WIC (the Special Supplemental Nutrition Program for Women, Infants, and Children).

PROBE: Please tell me for the last time you were pregnant.

| | | WEEKS OR

| | | MONTHS

d  DON’T KNOW

r  REFUSED

F2. Is your child of Hispanic, Latino, or Spanish origin?

PROBE: Please tell me for your last child.

Select one only.

1  Yes, of Hispanic/Latino origin

0  No, not of Hispanic/Latino origin

d  DON’T KNOW

r  REFUSED

F4. Did you ever breastfeed or pump breast milk to feed your new baby after delivery, even for a short period of time?

PROBE: Please tell me for your last child.

Select one only.

1  Yes

0  No SKIP TO F5

d  DON’T KNOW SKIP TO F5

r  REFUSED SKIP TO F5

• Transforming regular question text into probes (D10, D15, D16, F13a). Some of the items on the form include definitions and clarifying text. For example, items about health conditions include descriptions of the conditions to ensure that participants understand the question and are not confused by the medical terms. Other items include definitions of the activities described in the question. For example, we provide a definition of what constitutes one drink in the items about alcohol consumption (C11, D10) and describe what constitutes “reading a book to your child” in the questions about parenting (F12a, F13a). Although this text is necessary, it is also wordy and extends the length of the form and frustrates participants who already understand the meaning of the questions, particularly when we use the same definitions and/or descriptive text twice during the course of the form. We suggest keeping the expanded text the first time a given definition appears on the form and converting the definition/clarifying text to a probe for subsequent items. If participants ask questions about the meaning of one of these words, staff administering the survey will have the definition on the screen as a probe; however, they will not burden participants who remember the meaning from previous questions.

Proposed Revisions to Probes

Revised Question Text

D10. During the last three months of your last pregnancy, how many alcoholic drinks did you have in an average week?

PROBE: A drink is one glass of wine, wine cooler, can or bottle of beer, shot of liquor, or mixed drink.

(NOTE: We will keep the definition of one drink as part of the regular question text in C11.)

D15. When were you tested for Chlamydia during your last pregnancy? Was it during the first three months, the second three months, the last three months, or never?

PROBE: Chlamydia is a common sexually transmitted infection (STI) caused by a bacterium. It can infect both men and women.

(NOTE: We will keep the definition of Chlamydia as part of the regular question text in C16.)

D16. When were you tested for other STIs (other than HIV and Chlamydia) during your last pregnancy? Other STDs may include gonorrhea, herpes, or syphilis. Was it during the first three months, the second three months, the last three months, or never?

PROBE: Other STIs may include gonorrhea, herpes, or syphilis.

(NOTE: We will keep the definition of “other STIs” as part of regular question text in C17.)

F13a. During the past week, how many days did other family member(s) read to your child?

PROBE: Please tell me for your last child.

PROBE: Reading stories includes books with words or pictures but not books read by an audio tape, record, CD, or computer.

(NOTE: We will keep the definition of “reading stories” as part of regular text in F12a.)

• Adding probes to personal doctor or nurse questions (H4, H7). The questions on the form about whether the participant has one or more people she thinks of as a personal doctor or nurse (H4) and as a personal doctor or nurse for her child (H7) are directly from SLAITS. During the pretest, these items prompted a “yes” or “no” response from the majority of pretest participants, rather than a response of “yes–one person”, “yes–more than one person” or “no.” We suggest adding the probe, “Is there one person or more than one person?” so that we maintain the integrity of the item but also encourage respondents to provide a complete answer to the question.

Proposed Addition of Probe

Revised Question Text

H4. A personal doctor or nurse is a health professional who knows you well and is familiar with your health history. This can be a general doctor, a specialist doctor, a nurse practitioner, or a physician’s assistant. Do you have one or more persons you think of as your personal doctor or nurse?

IF RESPONDENT SAYS YES, PROBE: Is there one person or more than one person?

Select one only.

1  Yes, one person

2  Yes, more than one person

0  No

d  DON’T KNOW

r  REFUSED

H4. A personal doctor or nurse for your child is a health professional who knows your child well and is familiar with your child’s health history. This can be a general doctor, a pediatrician, a specialist doctor, a nurse practitioner, or a physician’s assistant. Do you have one or more persons you think of as your child’s personal doctor or nurse?

IF RESPONDENT SAYS YES, PROBE: Is there one person or more than one person?

Select one only.

1  Yes, one person

2  Yes, more than one person

0  No

d  DON’T KNOW

r  REFUSED

4. Reordering, Reformatting and Other Minor Revisions

• Add Fill Text for Questions About a Specified Period of Time After Birth (E17, E18, E19, F8 and F9). Pretest participants who had recently given birth expressed confusion with the question wording for items related a specific period of time after delivery. For example, one respondent was confused when we asked about her feelings for the three months after her last delivery when she had just given birth two weeks ago. For items about behaviors or feelings during a specified period of time after the respondent’s last delivery, we suggest adding fill text based on the child’s date of birth (E5) to ensure that the question is phrased in a manner that is relevant to the respondent.

Proposed Fill Text Additions to E17, E18, E19, F8 and F9

Revised Question Text

I will read a list of feelings and experiences that women sometimes have after childbirth.

[IF LAST CHILD WAS DELIVERED LESS THAN 3 MONTHS AGO: How often have you felt or experienced these things in this way after giving birth to your last baby?]

[IF LAST CHILD WAS DELIVERED MORE THAN 3 MONTHS AGO: How often did you feel or experience these things this way during the three months after your baby was born?] Please think about the time after your last child was delivered.

	SELECT ONE RESPONSE PER ROW
How often have you felt… How often did you feel…	Never	Rarely	Sometimes	Often	Always	DK	REF
E17. Down, depressed, or sad? Would you say never, rarely, sometimes, often or always?	1	2	3	4	5	d	r
E18. Hopeless? Would you say never, rarely, sometimes, often or always?	1	2	3	4	5	d	r
E19. Slowed down? Would you say never, rarely, sometimes, often or always?	1	2	3	4	5	d	r

F8. In which position [IF LAST CHILD >= 1 YEAR OLD: did you most often lay your child down to sleep during the first year?][IF LAST CHILD < 1 YEAR OLD: do you most often lay your child down to sleep?] [Was/Is] it on the child’s side, back, or stomach?

PROBE: Please tell me for your last child.

Select one only.

1  On the child’s side

2  On the child’s back

3  On the child’s stomach

d  DON’T KNOW

r  REFUSED

F9. How often [IF LAST CHILD >= 1 YEAR OLD: did your child sleep in the same bed with you or anyone else during the first year?][IF LAST CHILD < 1 YEAR OLD: does your child sleep in the same bed with you or anyone else?] Would you say always, often, sometimes, rarely, or never?

PROBE: Please tell me for your last child.

Select one only.

1  Always

2  Often

3  Sometimes

4  Rarely

5  Never

d  DON’T KNOW

r  REFUSED

• Reformat Question About Health Care Providers That Participant Has Spoken to in the Past 12 Months (H1). All six pretest respondents provided yes or no answers to H1 even though we are interested in collecting information on the types of health care providers participants talk to. We suggest reformatting the question so that we can gather complete data on the health care providers of interest.

Proposed Revisions to H1

Original Question Text

Revised Question Text

H1. Tell us all the types of health care providers you talked to in the last 12 months. Did you talk to a doctor or physician, nurse practitioner, physician assistant, midwife, or some other kind of provider?

Select all that apply.

1 □ A doctor or physician

2 □ A nurse practitioner, physician assistant or midwife

3 □ Other type of provider: ____________________

d □ DON’T KNOW

r □ REFUSED

H1. Tell us all the types of health care providers you talked to in the last 12 months. Did you talk to…

	SELECT ONE RESPONSE PER ROW
Did you talk to…	YES	NO	DON’T KNOW	REFUSED
a. A doctor or physician	1	2	d	r
b. A nurse practitioner, physician assistant, or midwife	1	2	d	r
c. Some other type of health care provider? [SPECIFY]	1	2	d	r

• Reformat Health Insurance Question (J9). Given our knowledge of the Healthy Start population, we suggest rearranging the order in which we ask respondents about the types of health insurance they have. During the pretest, we asked participants first about whether they had private insurance of any kind and then moved on to ask about Medicaid, CHIP, TRICARE, Indian Health Service, and indigent care programs. All six pretest respondents indicated that they had Medicaid. We recommend reordering the question and asking about the types of insurance that are most familiar to respondents first. This will reduce confusion and help retain respondents’ attention during this item.

Proposed Revision to J9

Revised Question Text

J9. Now, I’m going to ask about health insurance. Please tell me all the types of health insurance that you have.

	SELECT ONE RESPONSE PER ROW
Do you have…	YES	NO	DON’T KNOW	REF
a. Medicaid [LOCAL PROGRAM NAME]?	1	2	d	r
b. CHIP [LOCAL PROGRAM NAME]?	1	2	d	r
c. Health insurance from your job or the job of your husband, partner, or parents?	1	2	d	r
d. Health insurance that you or someone else paid for (not from a job)?	1	2	d	r
e. TRICARE or other military health care?	1	2	d	r
f. Indian Health Service?	1	2	d	r
g. Indigent Care Program [LOCAL PROGRAM NAME]?	1	2	d	r

• Reformat Children in Household Questions (J18, J19, J20). Pretest participants were initially confused by the progression of questions about number of children in the household (J17–J20). Specifically, they noted that the questions did not flow well and it was confusing that we asked J17 (total household members under age 18) and then asked for the breakdown by age group in J18–J20 (under 12 months, 1–5 years old, and 6–17 years old). We recommend revising the flow of questions J18–J20.

Proposed Revisions to Household Roster Series (J18–J20)

Original Question Text

Revised Question Text

J17. How many babies, children, and teenagers (under 18 years of age) live in the same house, apartment, or trailer as you?

| | | TOTAL NUMBER BABIES, CHILDREN AND TEENAGERS UNDER AGE 18 (IF NONE, ENTER 0)

d  DON’T KNOW

r  REFUSED

J18. How many children less than 12 months old live in the same house, apartment, or trailer as you?

| | | TOTAL NUMBER CHILDREN AGE 1 TO 5 YEARS OLD

d  DON’T KNOW

r  REFUSED

J19. How many children age 1 year to 5 years live in the same house, apartment, or trailer as you?

| | | TOTAL NUMBER CHILDREN AGE 1 TO 5 YEARS OLD

d  DON’T KNOW

r  REFUSED

J20. How many children age 6 years to 17 years live in the same house, apartment, or trailer as you?

| | | TOTAL NUMBER CHILDREN AGE 1 TO 5 YEARS OLD

d  DON’T KNOW

r  REFUSED

J17. [UNCHANGED]How many babies, children, and teenagers (under 18 years of age) live in the same house, apartment, or trailer as you?

| | | TOTAL NUMBER BABIES, CHILDREN AND TEENAGERS UNDER AGE 18 (IF NONE, ENTER 0)

d  DON’T KNOW

r  REFUSED

J18. Thinking about all the people under 18 years of age who live in the same house, apartment, or trailer as you…

How many are…	NUMBER	DON’T KNOW	REFUSED
a. Less than 12 months old?	| | |	d	r
b. Age 1 year to 5 years?	| | |	d	r
c. Age 6 years to 17 years?	| | |	d	r

• Reorder Section J Questions. After reviewing the form for flow, we recommend moving all demographic questions in Section J to Section A. This order will better mirror the types of questions that grantees will need to ask women to enroll them.

C. Conclusion

In summary, the pretest provided important feedback about the clarity, flow and timing of the questions on the Healthy Start form. The suggestions outlined above would improve respondent comprehension and ease of staff administration of the form.

II. National Healthy Start Program Survey:
Pretest Report and Recommendations

A. Overview of National Healthy Start Program Survey

The National Healthy Start Program Survey (NHSPS) will collect data from Healthy Start grantees to be used for the multilevel implementation and network components of the Healthy Start evaluation. The survey is designed to promote consistent collection of information about implementation, systems collaboration and coordination, and activities related to quality improvement. Project directors and other Healthy Start staff will complete the survey. Healthy Start grantees will complete the survey three times—once as a baseline at the start of the grant, once in the middle of the grant, and finally at the end of the grant.

Pretest purposes. By pretesting the NHSPS, we hoped to gain information on the average time it takes to complete the survey; grantees’ understanding of the survey questions and ability to provide empirical responses; and any questions that could be deleted or revised to improve clarity.

Pretest sample. MCHB provided contact information for project directors at four Healthy Start grantee sites. Two grantees completed the survey, one grantee partially completed the survey, and one grantee was unable to participate in the pretest due to competing priorities during the pretest period.

Adapting the survey for the pretest. We modified the survey in the following ways to fit the constraints of the pretest:

• Mode. The NHSPS is designed as a web survey with skip patterns and fill text programmed into the instrument. Due to the small scale of the pretest and the high costs associated with programming a survey, we administered the survey to grantees as a paper-and-pencil survey.

• Changing text from CAN to consortium. For the network questions in Section 5 regarding grantees’ collaborative efforts, we changed the wording from Community Action Network to consortium for the purpose of the pretest so that the wording reflected the current grantee terminology.

• Administering Question 5.9 in advance. Given that we were administering the survey as a hard copy instrument, we could not rely on a computer to fill the items for us. As a result, we asked question 5.9 (equivalent to C1 in the Healthy Start Community Action Network Survey) in advance of other survey items and used the responses to 5.9 to fill 5.10, 5.11, 5.12, 5.14 and 5.15. This enabled us to get a better sense of respondents’ comprehension of the subsequent items.

5.9. Collaboration can be defined as any joint planning, service coordination, cost-sharing initiatives, or other activities in which your organizations worked together toward a common goal.

From the list of CAN members below, please select up to 10 organizations with which your Healthy Start project collaborated during the past 12 months.

If your Healthy Start project collaborated with more than 10 organizations, select the 10 with which [ORG NAME] collaborated most closely.

PROGRAMMER: INSERT DROP DOWN FIELDS LISTING ALL CAN MEMBERS NEXT TO CAN MEMBER 1–10

▼

CAN Member 1

▼

CAN Member 2

▼

CAN Member 3

▼

CAN Member 4

▼

CAN Member 5

▼

CAN Member 6

▼

CAN Member 7

▼

CAN Member 8

▼

CAN Member 9

▼

CAN Member 10

Conducting the pretest. Grantees received electronic copies of the instrument via email in early January 2014. They printed the survey and responded to the items using pen. We asked pretest participants to record their start and finish time for each section on a time sheet, including any start and stop times for breaks within sessions. Grantees were instructed to mark confusing and/or problematic items with a star while they completed the survey and then to refer back to these items to add a description of the issue after they completed the section and recorded the total time for the section. Each of the three grantees returned the survey through different means, including scanning/ emailing, faxing, and regular mail. One grantee only submitted Sections 1 through 6 and was unable to provide insight into the length of time spent completing these sections.

Implications of pretest timing. One of the goals of the pretest was to assess the length of the survey. We asked programs to self-report their time by survey section using a timing sheet that we had compiled for the purpose of the pretest. Of the three programs that returned the survey, one took 144 minutes to complete the survey, one took 234 minutes, and the program that partially completed the survey did not provide time estimates. A number of factors likely influenced the amount of time it took program staff to complete the survey, including accessibility of statistics on outcomes and participation, knowledge of their program (which could vary based on the type of staff/ individual completing a given section), and number of distractions while completing the survey (for example, whether the respondent was multi-tasking or was fully focused on the survey). Additionally, we need to account for the pretest survey mode being paper and pencil instead of web based. This adds time to the survey because respondents need to pay attention to notes about skip patterns rather than automatically being routed to the correct questions as they would be for the web-based survey. Regardless, pretest results suggest that we need to decrease survey length by roughly 25 percent to get the total length down to an average of 120 minutes.

B. Recommended Changes to the National Healthy Start Program Survey

Based on feedback from the three pretest participants, we suggest making the modifications listed below to decrease the amount of time it takes grantees to complete the survey and to improve question clarity.

1. Revise Response Options for Questions About Outcomes and Participation to Be Multiple Choice Instead of Open Ended

The two Healthy Start grantees that participated in the pretest and reported timing information noted that Sections 2 and 4 took longer to complete than other sections. These sections contain a series of questions that ask about the total number of participants receiving a particular type of service, screening positive for a subset of conditions, or engaging in certain behaviors. For items about health conditions and health behaviors, one pretest participant noted that it was arduous to look up these statistics and suggested we request a percentage instead of a total. For questions about total number of participants receiving a given service (for example, socio-emotional screenings for children or parenting support groups), grantees tended to write in the total number of participants, suggesting 100 percent participation rate. Although it is possible that both pretest participants have 100 percent participation rate for all of the services of interest, it is more likely that this was simply an estimate rather than a number they retrieved from program data.

In attempt to reduce the burden on grantees and ensure that we collect useful data, we suggest revising these questions such that we ask grantees to select a percentage range (0 percent, 1–24 percent, 25–49 percent, 50–74 percent, 75–99 percent, 100 percent) rather than provide a raw number of participants. Below are a few examples:

Examples of Proposed Revisions to Response Options

Original Question Text

Revised Question Text

2.2 In [GRANT YEAR], how many participating women received a comprehensive needs/risk assessment?

Your best estimate is fine.

	Number of participating women receiving comprehensive needs/risk assessment
FILL 1.6a	Total number of participating women
FILL %	Percent (RANGE 0–100)

NO RESPONSE M

2.2 In [GRANT YEAR], what percentage of participating women received a comprehensive needs/risk assessment?

Select one only

 0%

 1-24%

 25-49%

 50-74%

 75-99%

 100%

 None

NO RESPONSE M

We suggest implementing this change for the following questions:

• 2.2: Participants receiving comprehensive assessment

• 2.23a-d: Participants with Medicaid, free care, private insurance and other insurance

• 2.37: Participants with reproductive life plans

• 2.39: Participants with primary care provider

• 2.41a-d: Participants use of and attempts to quit using tobacco

• 2.42a-c: Participants abuse/ use of alcohol and other substances and attempts to quit

• 2.44a-c: Participants breastfeeding at discharge, at six months (partially or exclusively), and at six months (exclusively)

• 2.45a-c: Participants screening positive for HIV, Chlamydia, and STDs

• 4.3: Participants (children) receiving socio-emotional screening

• 4.19: Male participants who are partners/fathers

• 4.32: Participants who attended support groups

• 4.33: Participants who received one-on-one parenting services

For 2.43a-c: Participants who are overweight, obese, and underweight, we suggest changing to asking for percentages of overweight, obese, underweight, and normal weight. This is recommended in this case as overweight, obese, underweight, and normal weight are mutually exclusive and should total to 100 percent.

C. Deletions

In addition to minimizing burden on grantees by modifying the type of data we collect, we also recommend deleting items from the survey. We used the pretest to identify potential items for deletion. Specifically, we paid particular attention to items that respondents seemed to just circle “yes” for all of the options in the series; items that respondents noted were very confusing; and items that respondents perceived as duplicates or questions that we had already asked earlier in the survey. Additionally, we identified a few items that did not create confusion for respondents or yield questionable data, but they were not essential, so they seem like good candidates for deletion given the time constraints of the survey. Below is a list of deleted items:

• 1.2, 1.2a: Local name of Healthy Start program

• 1.17: Retention strategy by life-course stage. We deleted 1.17 and changed the text of 1.16 so that we ask more generally about retention strategies for all participants.

• 2.4: Whether or not project assigns risk categories. We will get this from the site visits.

• 2.5: Description of risk categories. This was an open-ended question that yielded a lot of text from respondents (and likely took more time to respond). Given that we will get this information from the site visits and need to make cuts, we suggest deleting it from the survey.

• 2.6: Risk factors that program screens for by life-course stage. We do not suggest deleting this item in full but rather, deleting the “Other Specify” text where respondents can enter additional risk factors for which they screen participants. Text entered at 2.6 carries over to other items in the screening series so it will save time at this item and in future items.

• 2.7: Percent of participants screened by risk factor for each life-course stage. Pretest respondents wrote in “all participants” for each risk factor and indicated that this was their best estimate. Given that the question was difficult for pretest respondents to answer and does not seem to yield useful data, we suggest deleting it.

• 2.10: Whether or not program offers different case management models by risk level. We will get this from the site visits and focus groups.

• 2.11: Description of levels of case management. Like 2.5, this is an open-ended response that potentially takes a lot of time to answer. We will get this information from the site visits to reduce burden during the survey.

• 2.14: Average case load and range of cases. We suggest deleting range of cases per case manager and ask only for average case load because the question about range seemed to confuse pretest respondents.

• 2.30, 2.34: Activities covered under formal partnerships (2.30) and informal partnerships (2.34) with certified application counselor (CAC) organizations. Pretest respondents indicated confusion and wrote “yes” for all activities. We will get similar information from the site visits.

• 2.6 b and c: Number of formal partnerships with primary care providers broken down by memorandum of understanding (MOU) and contract. Rather than ask about number of MOUs and contracts, we suggest simply asking about the total number of formal partnerships.

• 3.24: Age of participants served through home visiting. We collect more information about home visiting and who it serves later in the survey and during the site visits.

• 3.36 (Parts a, c, d, I, o, p, r, s, t, u, v, w, y, z, aa, bb, cc, dd): Health education topics addressed. Pretest participants noted that the list of health education topics is very lengthy and they address most of them in some way. We suggest deleting the health education topics for which we collect data through the Healthy Start Participant Form and retain the topics we do not inquire about on the form. These deletions cut the question by more than 50 percent.

• 3.38: Collaboration with community organizations that offer “quit lines” for smoking cessation. We collect more information about the nature of Healthy Start and partner smoking cessation services in the following questions.

• 3.43: Topics covered in group tobacco-cessation counseling. We suggest rolling this in to item 3.42 (topics covered in one-on-one tobacco cessation counseling) and simply asking about topics covered in tobacco cessation counseling.

• 3.46: Tobacco-use cessation opportunities for other (non-partner) family members. We suggest rolling this into 3.45 (tobacco-use cessation opportunities for partners) and asking about opportunities for partners and other family members in one question.

• 3.50: Drug/alcohol counseling opportunities for other (non-partner) family members. We suggest rolling this into 3.49 (opportunities for partners), as we did for the tobacco cessation question 3.46.

• 3.52: Healthy weight services provided through referral. We suggest deleting this item and, in 3.51, ask about healthy weight services provided on-site and through referral.

• 3.54: Healthy Start refers to nutritionists at partner agencies. We suggest combining this with 3.53 (nutritionist on-site) and asking in general whether Healthy Start or partner agencies employ a nutritionist.

• 4.25: Health education topics covered with male participants. Both pretest participants selected “yes” for all items in this list. We suggest dropping the item because it is difficult and time consuming and seems to encourage respondents to simply check off “yes.”

• 4.31: Duration of parenting education. Parenting education services are most likely part of other services offered that we already ask about in the survey. We suggest deleting this item.

• 5.3: Does the CAN have a smaller leading body? We will be able to get information about the composition and structure of the CAN from grantee reports and the site visit. We suggest dropping this item.

• 5.4: How many individuals make up the smaller leading body? We will be able to get information about the composition and structure of the CAN from grantee reports and the site visit. We suggest dropping this item.

• 5.7: Main areas of CAN activity. We ask a similar question at 5.16a. We suggest deleting this item.

• 5.22, 5.23, 5.24: Change in community since beginning of funding cycle. This is a very subjective question. We will use secondary data on the community if available and qualitative data collection activities to capture this in the evaluation.

• 5.30: Overall impact of Healthy Start on community. This is another subjective question that will be better answered through qualitative data collection efforts.

• 6.26: University-affiliated evaluator (if external). This is not an essential question and we can capture this through grantee reports.

• Section 7: Healthy Start Achievements (with the exception of 7.4, 7.5 and 7.7). This section contains subjective questions about the respondent’s perception of Healthy Start’s impact, including multi-part items about each specific Healthy Start goal that is achieved. Given that we will capture this through administrative data analysis during the evaluation, we suggest dropping these burdensome questions from the survey.

D. Minor Revisions Based on Pretest Feedback

Minor revisions were also identified during the pretest based on feedback from respondents.

• Adding “During [GRANT YEAR]” to certain items that do not specify time period. Pretest respondents pointed out that certain questionnaire items do not specify whether we are asking about the present (current grant year) or during the previous grant year. We suggest adding clarification text to these ambiguous items to make it clear we are talking about the grant year of interest (for example, in 3.21).

• Adding Intimate Partner Violence and Immigration to types of referrals (1.11) and topics addressed through class and counseling (3.36). Pretest respondents added a few items to the “other-specify” categories, including immigration and intimate partner violence. We imagine that numerous grantees might write in these topics and suggest adding them to the survey.

• Revise percentage ranges to match other items with percentage ranges (3.9, 3.13, 3.16). Three of the items in the survey already use percentage ranges as response options, however the percentage ranges do not align with the new response options. We suggest revising them to match the following:

Select one only

 100%

 75 – 99%

 50 – 74%

 25 – 49%

 1 – 24%

 0%

NO RESPONSE M

• Add clarifying text about the Certified Application Counselor (CAC) organizations (2.28). During the pretest, organizations that are CAC organizations thought the CAC questions were not applicable to their project. We added in text to clarify the question.

Proposed Revisions to 2.28

Original Question Text

Revised Question Text

2.28. A formal partnership can be defined as a written agreement (usually involving a subcontract or memorandum of understanding [MOU]) with providers to provide care to Healthy Start participants. A subcontract is a legally binding document with an organization that states that the organization will provide services for Healthy Start. An MOU is a written agreement between entities that formalizes a relationship, but it is not legally binding as a contract. Does your project have a formal partnership with any certified application counselor (CAC) organizations in the community?  Yes 1  No 0 NO RESPONSE M

2.28. A formal partnership can be defined as a written agreement (usually involving a subcontract or memorandum of understanding [MOU]) with providers to offer care to Healthy Start participants. A subcontract is a legally binding document with an organization stating that the organization will provide services for Healthy Start. An MOU is a written agreement between entities that formalizes a relationship, but it is not legally binding like a contract. IF 2.27 = 1(ORG IS CAC ORG): Even if your organization is a certified application counselor (CAC) organization, does your project have a formal partnership with any certified application counselor (CAC) organizations in the community? IF 2.26 = 0 or 2.27 = 0 (ORG IS NOT CAC ORG): Does your project have a formal partnership with any certified application counselor (CAC) organizations in the community?  Yes 1  No 0 NO RESPONSE M

• Add response option “Healthy Start staff make appointments for participants” (3.17—assistance offered to participants to complete primary care referrals). Pretest participants noted that one of the main ways they assist participants with their referrals to primary care participants is to make the appointment for the participant. Given that both pretest participants noted this, we suggest adding it as a response option.

• Add field for pregnant women in perinatal depression screening question (4.1). Pretest participants asked why we do not inquire about the number of pregnant women receiving perinatal depression screenings. We suggest revising the question to include both pregnant and post-partum participants.

Proposed Revisions to 4.1

Original Question Text

Revised Question Text

4.1. During [GRANT YEAR], how many participating women with a live birth received a perinatal depression screening on site or at a partner site?

Your best estimate is fine.

________ NUMBER OF PARTICIPATING WOMEN RECEIVING PERINATAL DEPRESSION SCREENING

________ TOTAL NUMBER OF PARTICIPANTS WITH A LIVE BIRTH

4.1. During [GRANT YEAR], what percentage of participating pregnant and postpartum/interconceptional women received a perinatal depression screening on site or at a partner site?

Your best estimate is fine. (NHSPS 2.24 modified)

	Select one only
	0%	1 – 24%	25 – 49%	50 – 74%	75 – 99%	100%	No Response
a. pregnant women receiving a depression screening	1	2	3	4	5	6	M
b. postpartum or interconceptional women receiving a depression screening	1	2	3	4	5	6	M

• Other revisions and clarifications. We made a number of non-substantive revisions to grammar and to make questions more consistent with other items in the same section. Examples include modifying the multiple-choice response options in 3.27 (On average, what is the duration of a scheduled home visit?) to account for different lengths of time. Previously, the response options did not include a choice for 30–45 minutes. Another example is item 3.22 (Are home visits conducted by Healthy Start staff, contract staff, or both?). During the pretest, the question read “Are these conducted by Healthy Start staff, contract staff, or both”, as a follow-up to a question about whether they provide home visits in general.

E. Conclusion

In summary, the pretest provided important feedback about the clarity, flow, and timing of the questions on the NHSPS. We suggest the following:

• Deleting 35 questions out of 238 that were included in the pretest version of the instrument, including some of the lengthiest items with multiple sub-sections (for example. 2.6, 2.7, 3.36, and 4.5) and open-ended questions asking respondents to provide short response answers (for example 2.5, 2.7)

• Asking respondents to select percentage ranges instead of filling in total number of participants in open-ended fields for 13 of the most time-consuming questions

• Improving question clarity by revising question text

Implementing these suggestions would substantially reduce the amount of time grantees take to complete the survey. It would also make the survey questions clearer to respondents and the response options to multiple-choice items more robust.

III. Healthy Start Community Action Network (CAN) Survey: Pretest Report and Recommendations

A. Overview of Healthy Start Community Action Network (CAN Survey) Pretest

Funded by the Health Resources and Services Administration’s (HRSA), Maternal and Child Health Bureau (MCHB), Healthy Start aims to reduce disparities in infant mortality and improve perinatal outcomes in the United States. Under its next funding cycle, beginning in September 2014, the program will transform its framework from nine service and systems core components to the following approaches:

• Improving women’s health

• Promoting quality services

• Strengthening family resilience

• Achieving collective impact

• Increasing accountability through quality improvement, performance monitoring, and evaluation

MCHB requires an accompanying evaluation to describe the program’s implementation; provide insights into its characteristics, system features, and activities that affect outcomes; and assess the program’s overall success. One of the new components of the evaluation is a network study that will measure the level of collaboration among the Healthy Start grantees and other local health and social sector programs. The Healthy Start Community Action Network (CAN) Survey is designed to collect data needed for the network study and provide important information on the extent to which Healthy Start and other community organizations are working together to achieve common goals.

Pretest purposes. From the respondent perspective, we wanted to determine whether the question language was clear and understandable, the instrument flow was smooth and made sense, and the questions were not too difficult to answer. For multiple-choice items, we wanted to confirm that the answer choices reflected the main ideas that respondents wanted to convey. Finally, we wanted to test the average length of the survey.

Pretest sample. To identify respondents for the CAN survey pretest, we contacted four Healthy Start program grantees and asked them to provide a list of all organizations that participate in their Healthy Start consortium, and the contact information for two or three consortium members (excluding Healthy Start clients) who would be willing to assist with the pretest effort. Specifically, we requested that each grantee provide contact information for at least one health care provider. Two of the four grantees provided contact information for respondents that completed the survey. By the end of the pretest, we had tested the instrument with a total of five individuals: two health care providers, two individuals working at community organizations and one individual community member. The community member was not a Healthy Start client and did not work for or represent a community organization. This posed some problems for this respondent regarding the applicability of the questions, which we discuss in the next section under “Defining the Healthy Start CAN Survey Sample.” Our pretest sample outcomes are shown in Table A.3.a.

Table A.3.a. Healthy Start CAN Survey Participants

Pretest Sample	Completed Survey by Participant Type
Health care provider	2
Other community service provider	2
Community member involved in the consortium (not a Healthy Start client)	1

Adapting the survey for the pretest. A few modifications were made to the survey to fit the constraints of the pretest.

Mode. The survey is designed to be a web survey but we administered it as a paper instrument. We removed programming text to make it more visually appealing as a paper survey.

Administering C1 separately. Given that we were administering the survey as a hard copy instrument, we could not rely on a computer to fill the items for us. As a result, we asked C1 in advance of other survey items and used the responses to C1 to fill C2, C3, C4, D3 and D4. This enabled us to get a better sense of respondents’ comprehension of the subsequent items.

C1. Collaboration can be defined as any joint planning, service coordination, cost-sharing initiatives, or other activities in which your organizations worked together toward a common goal.

From the list of CAN members below, please select up to 10 organizations with which [ORG NAME] collaborated during the past 12 months. If [ORG NAME] collaborated with more than 10 organizations, select the 10 with which [ORG NAME] collaborated most closely. (New)

Select 10 organizations.

• [FILL CAN MEMBER #1] 1

• [FILL CAN MEMBER #2] 2

• [FILL CAN MEMBER #3] 3

• [FILL CAN MEMBER #4] 4

• [FILL CAN MEMBER #5] 5

• [FILL CAN MEMBER #6] 6

• [FILL CAN MEMBER #7] 7

• [FILL CAN MEMBER #8] 8

• [FILL CAN MEMBER #9] 9

• FILL CAN MEMBER #10] 10

Changing text from CAN to consortium. The 2014 Funding Opportunity Announcement for Healthy Start requires that grant applicants describe plans for developing a cross-sector CAN. According to the FOA, the CAN is intended to “increase trust among community partners/ members, assess and “map” the community using data, encourage effective and equitable allocation of limited resources, ensure that the contributions of community partners/members are valued and respected, and use varied communication modalities and technologies to provide community partners/members with full and timely access to information.” The CAN is similar to the Healthy Start consortium developed under previous rounds of funding, which requires grantees to develop a group of community members, clients, medical providers, social service agencies, and members of the faith and business communities. Given the similarities between the CAN and the consortium, we decided to modify the wording in the pretest version of the Healthy Start CAN Survey to refer to “the consortium” instead of “the community action network (CAN)” to ensure that respondents understood the terminology in the questions.

Conducting the pretest. Five individuals completed the survey. The surveys were distributed to participants between December 19, 2013 and January 6, 2014 and were returned between December 31, 2014 and January 15, 2014. All surveys were formatted as a paper-and-pencil survey and were distributed to respondents via email as a PDF. All questions requiring pre-filled text (for example, [ORG NAME] or [Healthy Start Grantee]) were populated prior to being sent to the respondents. Respondents could choose to return the survey through one of three options: 1) scanning and emailing the completed paper survey, 2) faxing the completed paper survey, or 3) returning the completed paper survey by mail. We reviewed the surveys as they were returned to us. After completion of the pretest, we held a final debriefing and identified the minor changes outlined below.

Implications of pretest timing. One of the goals of the pretest was to assess the length of the questionnaire. We instructed respondents to self-report how long it took them to complete the survey by noting the time they started and stopped working on the questionnaire, including start and stop times for any breaks. The self-reported survey lengths ranged from 23 minutes to 85 minutes, with an average of 55.8 minutes per complete. Because the times were self-reported, we have no way of ascertaining the accuracy of the reported times with precision. We have reason to believe that on average, the reported durations overestimated the length of the survey—for example, all but one of the respondents indicated that they completed the survey while at work, and potentially multi-tasking. Differences resulting from the pretest environment also impacted the length of the survey. It takes more time to complete the survey as a paper instrument because the respondent had to navigate the logistics of skips, a process normally handled by the web instrument.

Based on these factors, we estimate that the pretest conditions added between 5 and 30 minutes to the reported pretest survey length for any given respondent. In the next section, we recommend making a few deletions that would likely reduce the total survey time by 5 minutes. Taking all of this into account, we predict that the survey takes between 30–45 minutes to complete with the recommended deletions.

Interestingly, completion time did not appear to be associated with the number of organizations selected at C1—the question that asks respondents to choose up to 10 organizations part of the CAN with whom they collaborate with the most. We would expect that respondents selecting more organizations would take longer because the subsequent network questions ask about the respondents’ collaboration with each organization selected in C1. However, the number of organizations did not appear to have an impact on length.

B. Recommended Changes to the Healthy Start Community Action Network (CAN) Survey

1. Defining the Healthy Start CAN Survey Sample

The Healthy Start CAN Survey asks respondents how their organization interacts with Healthy Start, the CAN and the community as the purpose of the network study to assess organizational ties and networks in the community. During the pretest, most of the respondents did not have problems responding on behalf of their organization, with the exception of the one respondent who was an individual community member who did not represent an organization on the consortium. Many of the questions were irrelevant to this respondent as questions were framed to be from an organizational perspective (that is, “When did [FILL ORG NAME] start collaborating with Healthy Start?” or “During the past 12 months, in which of the following ways did [FILL ORG NAME] collaborate with [Healthy Start Grantee] and the other CAN members regarding services for women, children, and their families?”).

As a result, we recommend defining the sample such that it only includes CAN members who represent an organization that is formally or informally partnered with Healthy Start. Given that the purpose of the survey is to assess organizational networks in the community, this modification will help align the sample with the goals of the survey.

2. Addition of “Don’t Know” Response in Section A and B (A3a, A3b, A3c, A4, B1, B2 and B3)

Sections A and B include open-ended questions about organizational background and history working with Healthy Start. While most respondents did not have trouble answering these questions, there were a few instances (one at A3b, one at A3c, one at B2) where respondents wrote in “I don’t know.” We decided to add the text, “Your best estimate is fine,” to encourage respondents to provide input, but added a “don’t know” response option for cases where they truly do not know the answer. This will help us distinguish between “don’t know” responses from missing data.

Proposed Revisions to the Open-Ended Organization Background (Section A) and Collaborative History (Section B) Questions

Revised Question Text

A3a. How many clients did [ORG NAME] serve in the past year?

Your best estimate is fine.

Total: ______________________

 Don’t know

A3b. How many of the total served in the past year are women of reproductive age (15–44 years old)?

Your best estimate is fine. If your organization does not serve women, enter 0.

Women (reproductive age): ______________________

 Don’t know

A3c. How many of the total served in the past year are children under the age of 2?

Your best estimate is fine. If your organization does not serve children under the age of 2, enter 0.

Children:____________________

 Don’t know

A4. How many staff members are employed by [ORG NAME]? Include staff members who interact directly with clients and administrative staff.

Your best estimate is fine.

	Number of Staff Employed	Don’t Know
a. Total staff members	______________________	d
b. Full-time staff members (35 or more hours a week)	______________________	d 
c. Part-time staff members (less than 35 hours a week)	______________________	d 

B1. When did you first learn about [Healthy Start Grantee]?

Your best estimate is fine.

____/____

(MM/YYYY)

 Don’t know

B2. Collaborating can be defined as any joint planning, service coordination, cost-sharing initiatives, or other activities in which [ORG NAME] and [Healthy Start Grantee] worked toward a common goal.

When did [ORG NAME] begin collaborating with [Healthy Start Grantee]?

Your best estimate is fine. (New)

____/____

(MM/YYYY)

 Don’t know

B3. When did [ORG NAME] become part of the CAN with [Healthy Start Grantee] and other organizations in the community? Your best estimate is fine. Please include a date after June 2014 to reflect the newest cycle of the Healthy Start project. If your organizations participated in the CAN prior to June 2014, please enter 06/2014. (New)

____/____

(MM/YYYY)

 Don’t know

3. Deletion of A6 (Organization’s Budget)

Respondents are asked about their organization’s annual budget in Item A6.

A6. What is [ORG NAME]’s annual budget?

Your best estimate is fine.

$______________________

We found that only one out of five respondents answered the question with confidence—three respondents were unable to answer this item and one respondent answered the question but indicated lack of confidence in the response provided. Given our doubts that the question will yield useful data, we recommend deleting the question to reduce overall length of the survey and burden on the respondent.

4. B3 Clarification

In Item B3, respondents are asked about the length of participation in the CAN. For the pretest, we revised the wording to ask how long the respondents’ organizations have participated in the consortium instead of asking about the CAN. All but one respondent were able to identify the month and year that their organization began participating. Although respondents seem to understand the text during the pretest, we are concerned that the question might be interpreted differently once we change the text back to “CAN.. Specifically, for Healthy Start grantees that are funded again under the new funding opportunity and simply convert the consortium to a CAN, we are unsure whether CAN survey respondents would be able to distinguish activities between the two funding cycles (and thus provide the date that they began participating in the consortium). In order to eliminate potential confusion, we recommend adding the instructions below. This additional text would be accompanied with a range check that would generate an error message if the respondent tried to enter a date prior to 09/2014.

Proposed Revisions to B3

Original Question Text	Revised Question Text
B3. When did [ORG NAME] become part of the CAN with [Healthy Start Grantee] and other organizations in the community? Your best estimate is fine. ____/____ (MM/YYYY)	B3. When did [ORG NAME] become part of the CAN with [Healthy Start Grantee] and other organizations in the community? Your best estimate is fine. Please include a date on or after June 2014 to reflect the your participation in the CAN under the newest cycle of the Healthy Start project. ____/____ (MM/YYYY)  Don’t know

5. C1 Clarification

Respondents found the wording of item C1 ambiguous. In this question, respondents are asked to select up to 10 organizations with which their organization collaborates. Respondents who work for very large organizations with multiple departments (like a County Health Department) said they were not sure which organizations on the list worked most closely with their organization. In attempt to reduce ambiguity, we added a note advising respondents to only take into consideration their knowledge of their organization.

Proposed Revisions to C1

Original Question Text

Revised Question Text

C1. Collaboration can be defined as any joint planning, service coordination, cost-sharing initiatives, or other activities in which your organizations worked together toward a common goal. From the list of CAN members below, please select up to 10 organizations with which [ORG NAME] collaborated during the past 12 months. If [ORG NAME] collaborated with more than 10 organizations, select the 10 with which [ORG NAME] collaborated most closely. (New) Select 10 organizations. [FILL FROM LIST PROVIDED BY HEALTH START] [FILL CAN MEMBER #1] [FILL CAN MEMBER #2] [FILL CAN MEMBER #3] [FILL CAN MEMBER #4] [FILL CAN MEMBER #5] [FILL CAN MEMBER #6] [FILL CAN MEMBER #7] [FILL CAN MEMBER #8] [FILL CAN MEMBER #9] [FILL CAN MEMBER #10] [FILL CAN MEMBER #n]

C1. Collaboration can be defined as any joint planning, service coordination, cost-sharing initiatives, or other activities in which your organizations worked together toward a common goal. Based on your knowledge of your organization, please select up to 10 organizations from the list of CAN members below with which [ORG NAME] collaborated during the past 12 months. If [ORG NAME] collaborated with more than 10 organizations, select the 10 with which [ORG NAME] collaborated most closely. (New) Select 10 organizations. [FILL FROM LIST PROVIDED BY HEALTH START] [FILL CAN MEMBER #1] [FILL CAN MEMBER #2] [FILL CAN MEMBER #3] [FILL CAN MEMBER #4] [FILL CAN MEMBER #5] [FILL CAN MEMBER #6] [FILL CAN MEMBER #7] [FILL CAN MEMBER #8] [FILL CAN MEMBER #9] [FILL CAN MEMBER #10] [FILL CAN MEMBER #n]

6. Deletion of “Add-In” Responses in Section C (C2, C3, C4)

Items C2, C3 and C4 currently allow respondents to list names of organizations not on CAN with whom they partner. Below is the unrevised wording of C2:

C2. During the past 12 months, in which of the following ways did [ORG NAME] formally and/or informally partner with [Healthy Start Grantee] and the other CAN members?

If [ORG NAME] was involved in these activities with additional agencies in the community who are not CAN members, please list the names of those organizations in the spaces provided at the end and indicate which ways your organization partnered with them.

		SELECT ALL THAT APPLY
Community Action Network Members		Signed formal memorandum of understanding with organization			Met with organization for joint planning outside of CAN meetings			Participated in collaborative group or working group with organization in addition to the CAN			Submitted joint grant proposal
	[LIST OF CAN MEMBERS FROM C1]
	[Healthy Start Grantee]	1			2			3			4
		1			2			3			4
		1			2			3			4
		1			2			3			4
		1			2			3			4
		1			2			3			4
		1			2			3			4
		1			2			3			4
		1			2			3			4
		1			2			3			4
		1			2			3			4
	Other community agencies (Please specify)
		1			2			3			4
		1			2			3			4
		1			2			3			4
		1			2			3			4
		1			2			3			4

Of the five respondents, only one of the respondents used the additional lines to enter other organization names and the respondent did this only for item C2 because she found it to be burdensome to fill in the other organizations for other questions. Another respondent was confused by the additional lines. A third respondent mentioned that the organizations in C1 were fairly comprehensive and did not feel the need to use the additional space. Given respondents’ feedback, we recommend dropping the open-ended spaces where respondents can list additional organizations, as very few respondents would likely use these lines. In addition, deleting the item will decrease the length of the survey and the burden on the respondent.

Revised Question Text

C2. During the past 12 months, in which of the following ways did [ORG NAME] formally and/or informally partner with [Healthy Start Grantee] and the other CAN members

		SELECT ALL THAT APPLY
Community Action Network Members		[ORG NAME] signed formal memorandum of understanding with organization			[ORG NAME] met with organization for joint planning outside of CAN meetings			[ORG NAME] participated in collaborative group or working group with organization in addition to the CAN			[ORG NAME] submitted joint grant proposal
	[LIST OF CAN MEMBERS FROM C1]
	[Healthy Start Grantee]	1			2			3			4
		1			2			3			4
		1			2			3			4
		1			2			3			4
		1			2			3			4
		1			2			3			4
		1			2			3			4
		1			2			3			4
		1			2			3			4
		1			2			3			4
		1			2			3			4

C3. During the past 12 months, in which of the following ways did [ORG NAME] collaborate with [Healthy Start Grantee] and the other CAN members regarding services for women, children, and their families?

		SELECT ALL THAT APPLY
Community Action Network Members		[ORG NAME] made referrals to organization			[ORG NAME] received referrals from organization			[ORG NAME] shared/used the same data system
	[LIST OF CAN MEMBERS FROM C1]
	[Healthy Start Grantee]	1			2			3
		1			2			3
		1			2			3
		1			2			3
		1			2			3
		1			2			3
		1			2			3
		1			2			3
		1			2			3
		1			2			3
		1			2			3

C4. During the past 12 months, which of the following activities did [ORG NAME] engage in with [Healthy Start Grantee] and the other CAN members?

		SELECT ALL THAT APPLY
Community Action Network Members		[ORG NAME] organized/ implemented grassroots activities (for example, health fair or other community events)			[ORG NAME] participated in joint training with organization			[ORG NAME] developed joint program materials			[ORG NAME] met with policymaker or attended public meeting or hearing with the organization			[ORG NAME] developed media messages/ organized media events			[ORG NAME] assessed or “mapped” community needs using data
	[LIST OF CAN MEMBERS FROM C1]
	[Healthy Start Grantee]	1			2			3			4			5			6
		1			2			3			4			5			6
		1			2			3			4			5			6
		1			2			3			4			5			6
		1			2			3			4			5			6
		1			2			3			4			5			6
		1			2			3			4			5			6
		1			2			3			4			5			6
		1			2			3			4			5			6
		1			2			3			4			5			6

C. Conclusion

In summary, the pretest provided important feedback about the clarity of the Healthy Start CAN Survey questions, which is essential to accurately measuring the size and strength of the organizational networks in the Healthy Start community. The pretest also suggests that the interview length was within a reasonable range for this type of survey. Based on the pretest, we recommend several changes to the instrument that will improve the questionnaire clarity and brevity and eliminate potential sources of ambiguity in questionnaire wording.