Pretest Memo

M EMORANDUM

111 East Wacker Drive, Suite 920

Chicago, IL 60601-4303

Telephone (312) 994-1002

Fax (312) 994-1003

www.mathematica-mpr.com

TO: Amy Farb

FROM: Laura Kalb, Jennifer Walzer DATE: 6/30/2015

PAF - 009

SUBJECT: Follow up Pretest Findings for the PAF Study

This memo describes the PAF follow up survey pretest that Mathematica Policy Research conducted during April and May, 2015, for the purposes of improving the instrument for 12-month follow up data collection. The memo describes 1) the youths selected for the pretest, 2) the pretest and debriefing process, and 3) overall findings about the clarity and relevance of the survey content. At the end of this memo, we include the questions from the follow up survey and comments on issues discovered during the debriefing, along with possible solutions to these issues (Appendix A). We also include a version of the survey instrument with the suggested revisions in track changes (Appendix B). We recognize that the possible solutions will require further discussion before making modifications to the instrument.

A. RECRUITING PROCESS AND FINAL PRETEST SAMPLE

Mathematica worked with program staff in in California and Texas to recruit potential pretest respondents. In California, we recruited from AFLP pilot sites that are not participating in the study. We worked with these sites on our initial pretest of the baseline survey approximately one year ago. In Texas, we worked with the Healthy Family, Healthy Futures program staff. Each group provided several names of young women who have been enrolled in the program for approximately one year and who are not currently participating in the PAF study. At the time of enrollment in their respective programs, participants were either parenting or pregnant. These pretest respondents are representative of the enrolled study population who will be asked to take the PAF 12-month follow up survey on length of time since program enrollment and parenting status.

We pretested the California version of the instrument with six respondents, and the Texas version with five. One pretest respondent in the California group previously assisted us with the baseline pretest in April, 2014. The pretest and debriefing was administered over the phone. Upon completion of the debriefing, participants were given $50 gift cards for their participation and time.

B. PRETEST AND DEBRIEFING PROCESS

The administration of the pretest occurred over the phone by trained Mathematica survey research staff. Before beginning the interview, Mathematica staff provided a brief background on the study, the pretest and debrief procedures, reminded participants that their participation was completely voluntary and obtained verbal assent. Respondents were informed that after we completed the survey there would be a 15-20 minute debrief, asking their thoughts and opinions on specific survey items. We asked them to take note of any questions or words that were confusing or unclear. We reminded participants that their actual answers to the survey questions were less important than the process of formulating and providing the responses and that individual responses would never be shared outside the study team.

C. SURVEY ADMINISTRATION OVERALL

We were interested in learning more about several different survey administration issues, the length of time needed to complete the survey, comprehension of survey items, and what mode would be preferable for completion. Overall, the pretest respondents had little trouble completing the instruments. The biggest issue discovered through the pretest was that the length was slightly over our goal of 30 minutes.

• Length of the survey instrument. The average time for completion on the pretest was 33 minutes (range 28–35 minutes). See Table 1 for breakdown by the specific version of the instrument. The pretest version, which averaged 33 minutes, did not include the 2-3 minutes for gathering contact information. Therefore, we have proposed a few cuts to the survey instrument, but suggest increasing the burden estimate to 35 minutes.

Table 1: Average time to complete

Version of instrument	Average Time to Complete	Range
California	33.2	28-41
Texas	33	30-35

• Comprehension of survey items. Overall, respondents reported that the instrument was easy to understand and follow. The pretest did provide insights about several potential issues on specific questions. The suggested revisions based on this feedback are listed in detail in Appendix A, question by question.

• Comments on mode. Eight out of eleven reported that they would most likely complete this survey over web if it was an available mode. Three respondents indicated that they may have trouble accessing a computer and would prefer phone. This confirms our plans to use a mixed-mode administration, starting with web, but following up with non-responders or those who prefer phone with a CATI interview. Of those who suggested they would prefer to complete by web, six out eight said they would do so using their phones. We plan to have a mobile friendly version of the instrument.

cc: Matthew Stagner, Susan Zief

APPENDIX A
Recommended Revisions for Baseline Survey Items

This appendix includes detailed feedback from the pretest on specific items and our recommended revisions based on these comments. In the first column, we list the original questionnaire item. In the second column, we include comments from the pretest and our rationale for revisions, followed by the suggested revision to each question. Bolded text in the suggested revisions indicates a change from the original item. All item numbers correspond to the original version of the instrument, before any suggested revisions.

Original Questionnaire Item	Comments and Suggested Revisions
1.1. In what month and year were you born? 1.2 Are you Hispanic/Latina? 1.3 What is your race?	These questions were included in the baseline and we have very few missing values. So far we only have two cases that did not complete the baseline survey and nine respondents that have indicated “don’t know” or “refused” at the race item. We have DOB for all respondents. To help reduce the overall time, we suggest dropping these items. Suggested revision: Drop these items from the survey..
2.2, 2.4, 2.6: And did the classes or sessions take place with your case manager, with someone or somewhere referred by your case manager or program, or with someone or somewhere else?	There was confusion by both sets of respondents about who we were referring to when we said case manager. One respondent in Texas assumed this was her Medicaid case manager. Another respondent indicated she had two case managers, so she was not sure who we meant and answered about both. Two respondents were thinking about their welfare or social worker. We suggest specifying the program name for California cases and Texas treatment cases. We also recommend dropping these items for Texas control cases, since they are not receiving any program services from Healthy Family, Healthy Futures. We believe that it will be sufficient to learn if they received the services at 2.1, 2.3, and 2.5. The follow up items were included to better understand what program services are being offered/recommended by the programs begin evaluated and this is not applicable for Texas controls. We think Texas respondents may think of Healthy Family, Healthy Futures program staff as home visitors, more than case managers. Two out of eleven respondents did not know what referral meant. We suggest changing this to “recommend” to make it easier for our study population to understand. Suggested revision: For TX: And did the classes or sessions take place with your home visitor from Healthy Family, Healthy Futures at somewhere recommended by your home visitor, or somewhere else? An example of somewhere else could be from WIC clinic staff. For CA: And did the classes or sessions take place with your case manager from [PROGRAM SITE], at somewhere recommended by your case manager at [PROGRAM SITE], or somewhere else?
5.8 During the past 12 months, did [CHILD] see a doctor, nurse, or other health care professional for any kind of medical care, including sick child care, well child checkups, physical exams, and hospitalizations? (NSCH-2011 S4Q01)	If a respondent indicates “No” here, we suggest they skip item 5.9 (see line below).
5.9 During the past 12 months, how many times did [CHILD] see a doctor, nurse, or other health care provider for preventive medical care such as a physical exam or well-child checkup? Your best estimate is fine. (NSCH-2011 K4Q20	The concept of preventative child care was not understood by seven respondents. Four reported that they would include hospitalizations in their response. One respondent reported that preventative medical care was when you give your child Pedialite, and two reported they were unsure what preventative meant. Suggested revision: During the past 12 months, how many times did [CHILD] see a doctor, nurse, or other health care provider for a regular checkup, not a sick child care visit or hospitalization? Your best estimate is fine.
5.11: Since [CHILD] was born, have you had a postpartum checkup for yourself? (PRAMS – Standard – L8)	There was confusion on what the term postpartum meant. Examples: checking you for cervical/uterine cancer, checking to see if you have depression (3 respondents), going to the doctor for birth control. Suggested revision: In the 3 months after you had [CHILD], did you have a checkup with a doctor, nurse or other health care worker for yourself?
5.14 Are you currently covered by private insurance, Medicaid or any other government program that pays for medical care? Private insurance can be insurance through your or someone else’s employer or insurance purchased directly from an insurance company.	The example after the question was confusing and caused one respondent to assume we were only asking about private insurance. Additionally, we want to change the question format so that it is consistent with the format of 5.13, insurance for the child. We also probed respondents to the type of insurance they have. Several in CA reported MediCal, the specific name of CA Medicaid. In the CA version, it might be helpful to include this specific name. Suggested revision: The next questions are about health insurance. This can include private insurance, Medicaid, [MediCal (for CA)] or any other government program that pays for medical care. 5.13. Do you have health insurance for [CHILD]? 5.14. Do you have health insurance for yourself?
5.18. Since [CHILD] was born, how many months have you lived with [CHILD] in the same household? (HFSA 5.12-revised)	Counting the number of months was problematic for almost all respondents. They would have preferred years or an option “Since the baby was born”. Several respondents had children that were 2-3 years old and have lived with the child since they were born. We suggest breaking this into two items and only asking number of months to respondents who indicate they have been living with the child “some of the time”. Suggested revision: 5.18. Since [CHILD] was born, have you lived with [CHILD] in the same household All of the time,……………………………….1 GO TO 5.19 Some of the time……………………..2 None of the time?.................................3 GO TO 5.19 INTERVIEWER: IF 5.18=2 ASK 5.18a 5.18a Since [CHILD] was born, how many months have you lived with [CHILD] in the same household? INTERVIEWER, IF STRUGGLING: If your child has lived with you on and off since [his/her] birth, please count the total number of months. | | | number of months
5.19 The next questions are about things YOU may have done with [CHILD] in the past month. In the past month, how often have you done each of the following? Was it more than once a day, every day or almost every day, a few times a week, a few times in the past month, once or twice in the past month, or never? (HFSA 5.13) Played games like peek-a-boo or gotcha with child Sung songs with child Read or looked at books with child Told stories to child Played with game or toys with child	We asked about other things mom’s do with their children that might be missing from this list. We were interested to see if this list still works well for children that are older, 2-4 year olds. One respondent suggested adding “Going for walks or the playground with child”. Additionally, we believe it is difficult to distinguish the difference between options c & d, reading and looking at books and telling stories to the child. In order to not add to the overall length, we suggest revising item d. The response categories were also lengthy and will be collapsed in the analysis. We suggest combining “More than once a day” with “Every day or almost every day” and combining “A few times in the past month” with “Once or twice in the past month”. Suggested revision: 5.19. The next questions are about things YOU may have done with [CHILD] in the past month. In the past month, how often have you done each of the following? Was it everyday, a few times a week, a few times in the past month, or never in the past month? EVERY DAY OR ALMOST EVERYDAY A FEW TIMES A WEEK A FEW TIMES IN THE PAST MONTH NEVER a. Played games like “peek-a-boo” or “gotcha” with [CHILD] 3 2 1 0 b. Sung songs with [CHILD] 3 2 1 0 c. Read or looked at books with [CHILD] 3 2 1 0 d. Played outside or at the playground with [CHILD] 3 2 1 0 e. Played with games or toys with [CHILD] 3 2 1 0
5.21 The next questions are about the father of [CHILD’S NAME/THE BABY YOU LOST]. So I can refer to him, what is his first name? (HFSA 5.15)	We encountered one case where the dad and child had the same name. We are worried this is problematic on the web where we fill subsequent questions with the father’s name Suggest revision: Drop this question and leave these fills as ([Child’s Name]’s Father).
5.26. INTERVIEWER: SKIP IF 5.18 = 3 OR IF BABY DIED 5.1 = 2, 4, OR 5 You mentioned that since [CHILD] was born, you have lived in the same household with [CHILD] for [NUMBER OF MONTHS] months. Since [CHILD] was born, how many months have you lived in the same household with both [CHILD] and [CHILD’S FATHER’S NAME]? (HFSA 5.21)	Similar to 5.18, the number of months proved to be problematic for respondents that either lived with the father all of the time or none of the time. We suggest making the same revisions here, splitting it into two questions and only asking number of months if they told us they lived together at least some of the time. Suggested revision: 5.26. You mentioned that since [CHILD] was born, you have lived in the same household with [CHILD] (if 5.18a has months fill “for [NUMBER OF MONTHS] months”) (if 5.18=1 fill “all of the time”). Since [CHILD] was born, have you lived with [CHILD] and [CHILD]’s father in the same household Always,………..…………………………1 GO TO 5.27 Sometimes, or…………………………2 Never?......................................3 GO TO 5.27 5.26a Since [CHILD] was born, how many months have you lived in the same household with both [CHILD] and [CHILD]’s father? | | | NUMBER OF MONTHS DON’T KNOW……………d REFUSED……………………r
5.30-TX The next questions are about things [CHILD’S FATHER’S NAME] may have done with [CHILD] in the past month. In the past month, how often has [CHILD’S FATHER’S NAME] [STATEMENT a to h]? Was it more than once a day, every day or almost every day, a few times a week, a few times in the past month, once or twice in the past month, or never? (HFSA 5.24)	One respondent reported that due to a court order, she cannot be with the father when he has visitation with the child, so she could not answer any of these questions. Respondents that report that they are not in contact with the father skip this item, but in her case she sees him to do drop offs. We suggest adding a “don’t know” option to this question since many respondents may have complicated arrangements. We also propose making the same revision to option d as we did in 5.19 and the same change in the response option categories. Suggested revision: 5.30. INTERVIEWER: SKIP IF (5.16 = NO) OR (5.28 = NO) OR (5.25=6) The next questions are about things [CHILD]’s father may have done with [CHILD] in the past month. In the past month, how often has [CHILD]’s father [STATEMENT a to h]? Was it every day or almost every day, a few times a week, a few times in the past month, or never in the past month? CODE ONE PER ROW EVERY DAY A FEW TIMES A WEEK A FEW TIMES IN THE PAST MONTH NEVER IN THE PAST MONTH DON’T KNOW a. Played games like “peek-a-boo” or “gotcha” with [CHILD] 3 2 1 0 D b. Sung songs with [CHILD] 3 2 1 0 D c. Read or looked at books with [CHILD] 3 2 1 0 D d. Played outside or at the playground with [CHILD] 3 2 1 0 D e. Played with games or toys with [CHILD] 3 2 1 0 D f. Helped [CHILD] to get dressed 3 2 1 0 D g. Changed [CHILD]’s diapers or helped [him/her] use the toilet 3 2 1 0 D h. Given [CHILD] a bottle or something to eat 3 2 1 0 D
6.13. Do you want to have any more children? (HFSA 5.35)	One respondent was currently pregnant. It was not clear if we were asking about children including the child with whom she was currently pregnant or beyond this child. Another respondent thought we were asking if she wanted to have another child now and said “no”, therefore skipping the follow up item 6.14. Suggested revision: 6.13. [IF 6.9=1 FILL “After your current pregnancy”] Do you ever want to have any more children?
6.14. How soon would you like to have your next child? Would you like to have it… (HFSA 5.36) Within the next year 1-2 years 2-3 years > 3years	One respondent answered “two years” and was not sure which category her response fell into. The current answer categories will not work well on the web, since they are not mutually exclusive. Suggested revision: 6.14.How soon would you like to have your next child? Would you like to have it… Within the next year, One year from now, Two years from now, Three years from now, or Four or more years from now?

An Affirmative Action/Equal Opportunity Employer