PACT Appendix D - HM followup QxQ

PACT HM Follow-up survey

question-by-question justification

Table1. Follow-up HM Survey: Question-by-Question Justification

Question	Source	Rationale
Introduction and Contact Information
Introduction (i1-i7)	PACT	Questions are asked to ascertain whether the respondent is the sample member and to inform the respondent about: the expected length of the interview, the Certificate of Confidentiality, and the possibility that the interview may be recorded.
Contact information (A1a-A2)	PACT	Obtaining the exact spelling of the respondent’s name is important so that we can send them the respondent payment.
Family Structure
Marital status of HM couple (B1)	BSF	A central goal of the HM programs is to improve the relationship of the couple. These set of questions ask about relationship of the respondent with the person he or she applied to the program with when they enrolled in the study. Outcomes will include marital status, whether in a romantic relationship, whether they live together and, if not, how often they have contact. If the romantic relationship ended, the survey asks about why it ended. This is important because an evaluation of Building Strong Families found that the program in one site increased the likelihood that the couple would break up (Wood et al. 2012). The three questions on plans to get married explore the concreteness of those plans by asking when the wedding will take place.
Relationship status of HM couple (B2, B5, B6)	BSF
When and why relationship ended (B2a-B2c)	BSF
Plans for marriage with HM partner (B3-B4)	Provided by Scott Stanley
Baby(ies) of respondent and HM partner that the couple was expecting at random assignment (B7-B7d)	BSF	Some couples in the HM program will have been expecting a child at baseline and have no other children. We want to include this child as the focal child. Hence, we ask about the birth of the child that the couple was expecting at baseline.
Parenting
Quality of co-parenting relationship (C1)	PAM	The PACT HM programs aim to improve couple’s co-parenting relationships by improving the couple’s communication skills, co-parenting skills, and emphasizing that parenting is a shared task. These questions are from the Parenting Alliance Measure (Abidin and Brunner 1995).
Living arrangements of focal child (C2 and C2a)	BSF	The PACT program may affect the living arrangements of the children directly or via its effect on the parents’ relationship. These questions ask whether the child is living with the respondent or the PACT partner, both, or neither.
Activities conducted with focal child* (C3)	SHM, EHS, PSID, and PACT	The HM program intends to improve the quality of parenting. Hence, C2 questions ask about the types of activities that the parent may do with the child, focusing on active engagement. The activities asked about depend on the age of the child.
Positive parenting of focal child* (C4)	SHM, PPQ, PACT	The HM program intends to foster positive parenting behaviors. Hence, C3 questions ask about four positive parenting behaviors.
Approaches to discipline (C5) of focal child*	Modified from CTSPC	The HM program intends to improve the way that the couple disciplines the child. This question asks about appropriate and inappropriate discipline approaches.
Relationship Quality with PACT Partner**
Happiness with the relationship (D1)	BSF	Overall happiness with the relationship is one of the most frequently used measures of relationship quality because it is highly (negatively) correlated with the likelihood of later breakup (Karney and Bradbury 1995). This question is asked of everyone (unless the PACT partner has died).
Relationship in trouble (D5)	SHM	This question asks whether the respondent has felt that their relationship was in trouble in the past three months. The SHM study found that the program could reduce the likelihood that the couple reported feeling that their relationship was in trouble (Hsueh et al. 2012).
Commitment and trust (D1a, D1b, D2d, D3b)	Suggested by Professor Scott Stanley	Higher levels of commitment and trust have been shown to predict relationship stability even during bad times (Amato 2003). Lack of trust has shown to be a significant barrier to marriage for low-income couples (Edin and Kefalas 2005). The items we include in the instrument were recommended by Professor Scott Stanley. All but D1a and D1b were used in BSF.
Support and affection (D2a, D2b, D2c, D2e, D2f and D3a, D3b-g)	BSF	A factor analysis using BSF data found these three distinct domains of relationship quality (Wood et al. 2010) reflected in these three sets of questions. They include three questions that address conflict resolution skills taught by the HM programs (D4o,p, and q) but that were not asked in the BSF survey.
Constructive conflict behaviors (D4a, c, e, g, j, n, and q)	BSF
Destructive conflict behaviors (D4b, d, f, h, i, k, l, m, o, and p)	BSF
Fidelity (D6 and D7)	BSF	Infidelity has been found to be a major obstacle to marriage for unwed couples as well as a major predictor of divorce among married couples. Two questions are included to determine whether the respondent has been unfaithful and whether the respondent believes that his or her partner has been unfaithful.
Intimate partner violence (D8 and D9)	BSF and SHM	The HM programs aim to improve conflict management and increase the chance of a healthy relationship. An absence of intimate partner violence and psychological control are important elements of a healthy relationship. The questions on physical violence (D8) are taken from the revised Conflict Tactics Scale (CTS2, Strauss et al. 1996). The questions of psychological abuse (D9) were taken from the SHM survey.
Perceptions of improvements in relationship over the past year (D10)	PACT	These questions get at whether the respondent perceives that his or her relationship with the PACT partner has improved over the past year. This may capture specific improvements in the relationship that are not captured with the other relationship quality measures.
Economic Stability
Information on all jobs respondent has had in the past three months (E1-E9)	WIA	The PACT HM programs all provide some services to assist the participants get and keep a job or advance in their careers. These questions ask about all the jobs that the respondent has currently and in the past three months. For each job, we ask about when the job began, ended (if applicable), type of job (e.g. full time, part time, self-employed, day laborer), wage rate, hours worked, and fringe benefits. Earnings over the past three months can be calculated from the wage rate and the number of work hours reported.
Other earnings from work (E10 and E11)	WIA	Many people, especially those with low income, do other work for pay that may not be considered formal employment. For example, they may babysit, or help out with a family business occasionally. These questions ask about whether they have done work that they do not consider “a job” and the amount they made from that work.
Steps made toward improving economic stability (E12-E14)	PACT	Within one year after enrollment in the program, the respondent may not have had time to get a job or improve his or her earnings. This series of questions asks if the respondent has taken steps to getting a job or better jobs: whether he wants a job or a better job, whether he has planned to take steps, and whether he has actually taken steps.
Readiness for job search (E15)	PACT	The HM program aim to improve job readiness, so this question asks if the respondent has an updated resume.
Perceptions of economic stability (E17)	PACT	The HM program may assist the couple become financially better off—not only by helping them find, keep, or advance in their jobs, but also by helping the couple develop a budget and connecting the couples with other resources. This global question asks whether the respondent is better off financially compared to a year ago to capture the many ways the program can help the participant.
Ability to manage money (E18)	PACT	Some of the PACT HM programs offer classes in developing a budget and managing finances. This question asks about whether the respondent feels that he or she has improved his or her ability to manage his or her finances.
Mental Health
Depressive symptoms (F1)	PHQ-8	By improving relationships, providing opportunities for social interactions, and linking respondents to mental health services, , the PACT HM programs may reduce depressive symptoms. Eight items from the Parental Health Questionnaire are included in this survey. The PHQ-8 has been shown to be a valid measure of depression in population-based studies (Kroenke et al. 2009)
Service Receipt
Whether participated in a relationship skills education group or received one-on-one relationship skills education (G1, G4)	BSF	Asking both members of the program and control group about the receipt of services will provide information about the “counterfactual”—the services that would have been received in the absence of participation in the program. This series of questions ask about programs and services (such as relationship skills education, employment services, and mental health services) that the program and control group members may have received since random assignment. Understanding the services received by program versus control group members will help us interpret the statistical impacts.
How many hours spent in relationships skills education group or one-on-one session (G2, G2a, G5, G6, G6a)	BSF
Whether PACT partner** usually attended relationship skills group or one-on-one session with respondent (G3 and G7)	BSF
Whether participated in a parenting class and number of hours spent in class in total (G8a)	BSF
Whether participated in an employment service group activity and number of hours spend in group (G8d)	BSF
Whether participated in a training program (G8b)	PACT
Whether received job leads (G8c)	PACT
Whether taken education classes (G8e)	BSF
Whether received anger management services (G8f)	BSF
Whether received mental health or substance use services (G8g)	BSF
Satisfaction with program (G9-G11)	PACT	Programs would like to know whether the participants feel they benefited from participating in the study. This series of questions, asked only of respondents who are in the program group, ask how the respondent to rate his or her satisfaction with the program, whether they would recommend the program, and to say how much each of the three main types of services—relationship skills education, parenting, and employment—helped him or her.
Contact Information
Contact information	PACT	Additional contact information is required to send the $25 appreciation payment to the respondent and in case an additional wave of interviews will be conducted. A Facebook name is collected in case we have difficulty contacting the respondent using the other information provided.

HM: Healthy marriage

*Focal child: one child chosen from among the children who made the couple eligible for the study

**PACT partner: this is the other person in the couple that enrolled in the HM program

Sources: Building Strong Families Study (BSF), American Recovery and Reinvestment Act COBRA Subsidy Study (CBRA), Fragile Families and Child Well-Being Study (FFCWS), Work First New Jersey (WFNJ), Parenting Alliance Measure (PAM), Rural Welfare-to-Work Demonstration Evaluation (RWTW), Patient Health Questionnaire (PHQ-9), Serious Violent Offender Reentry Initiative Evaluation (SVORI), Supporting Healthy Marriage (SHM), Early Head Start survey (EHS), Mother and Infant Home Visiting Program Evaluation (MIHOPE), Panel Study of Income Dynamics (PSID), Parenting Practices Questionnaire (PPQ), Conflict Tactics Scale-Parent Child (CTSPC), WIA Gold Standard Follow-up Survey

.

SURVEYS REFERENCED

The list below contains brief descriptions of the eight surveys referenced in the PACT HM follow-up survey, as well as locations of the surveys referenced. Descriptions were compiled from websites about the surveys and descriptions of Mathematica studies were gathered from project summaries. When necessary, we modified questions drawn from these surveys to make them easier to understand or to have the questions align more closely with the baseline survey’s goals.

1. Building Strong Families Study (BSF)

The United States Department of Health and Human Services/Administration for Children and Families (ACF) initiated the Building Strong Families (BSF) project to help interested and romantically involved low-income, unwed parents build stronger relationships and thus enhance their child’s well being and their own future. The BSF evaluation being conducted by Mathematica is designed to test the effectiveness of these programs for couples and children. BSF data collection included a baseline information form to collect demographic and socioeconomic data along with two follow-up surveys. The follow-up surveys included questions related to mother-father relationships, family structure, fathers’ involvement in child rearing, parent-child relationships and the home environment, family functioning, child well-being and development, and parental well-being.

Surveys are available from Mathematica upon request.

2. American Recovery and Reinvestment Act COBRA Subsidy Study (CBRA)

Sponsored by the U.S. Department of Labor, Mathematica’s American Recovery and Reinvestment Act (ARRA) COBRA Subsidy study examines the effect of the availability of an ARRA COBRA premium subsidy on the take-up of COBRA coverage and other health and employment outcomes. As part of the study, Mathematica will conduct a survey of COBRA-eligible individuals drawn from state Unemployment Insurance recipients. The CBRA survey asks questions related to respondents’ demographic characteristics, employment history, receipt of social services, and health insurance.

Surveys are available from Mathematica upon request.

3. Fragile Families and Child Well-Being Study (FFCWS)

The Fragile Families and Child Well-Being Study is a longitudinal study of a cohort of nearly 5,000 children born between 1998 and 2000 from birth through age five. Approximately one-third of the children were born to unmarried parents. Interviews were conducted with both mothers and fathers covering a range of topics including attitudes, relationships, and parenting behavior.

Study protocols and codebooks can be found here: http://www.fragilefamilies.princeton.edu/documentation.asp

4. Work First New Jersey (WFNJ)

Mathematica evaluated the effects of New Jersey’s initiative to help welfare recipients transition from welfare to work. WFNJ interviewed sample members annually for five years documenting changes in household composition, income, employment, and other indicators of well-being.

Surveys are available from Mathematica upon request.

5. Rural Welfare-to-Work Demonstration Evaluation (RWTW)

Mathematica’s Rural Welfare-to-Work Strategies Demonstration Evaluation used random assignment to assess innovative approaches to helping welfare-dependent and other low-income families in rural areas to enter, maintain, and advance in employment and to secure family well-being. Data collection included a baseline information form to collect demographic and socioeconomic data on sample members and two follow-up surveys to collect detailed employment history data as well as information on various outcomes related to individual and family well-being.

Surveys are available from Mathematica upon request.

6. Evaluation of the Serious Violent Offender Reentry Initiative (SVORI)

The Evaluation of the Serious Violent Offender Reentry Initiative (SVORI) was a multi-year, multi-site evaluation funded by National Institute of Justice. The impact evaluation was designed to measure the impact of enhanced reentry programming on post-release outcomes. As part of the evaluation, interviews were conducted at four points in time.

Surveys are available from the National Archive of Criminal Justice Data.

7. Supporting Healthy Marriage (SHM)

ACF sponsors the Supporting Healthy Marriage project (SHM). SHM is a multi-year, multi-site evaluation of marriage education programs for low-income married couples. Data collection includes baseline information forms to determine eligibility and collect demographic information, a 12-month follow-up survey to measure short-term impacts of the marriage education program, and a 30-month follow-up survey to measure longer-term impacts.

Surveys are available from MDRC.

8. The Mother and Infant Home Visiting Program Evaluation (MIHOPE)

ACF and the Health Resources and Services Administration jointly administer the Maternal, Infant, and Early Childhood Home Visiting Evaluation. MIHOPE is a multi-year, multi-site evaluation of the Home Visiting program designed to prevent child maltreatment, improve maternal and child health outcomes, and increase school readiness. Data collection includes a baseline survey and a 15-month follow up survey.

Surveys are available from MDRC.

9. Early Head Start (EHS)

The U.S. Department of health and Human Services, Administration for Children and Families funded an evaluation of the Early Head Start program—a program for pregnant women and families with infants and toddlers that is based on the Head Start program model. The Administration for Children and Families sponsored an experimental evaluation of the program based in 17 sites. This study conducted five follow-up surveys after random assignment.

Surveys are available from Mathematica Policy Research

10. Workforce Investment Act Adult and Dislocated Worker Programs Gold Standard Evaluation (WIA)

The U.S. Department of Labor sponsored an experimental evaluation of the Adult and Dislocated Worker programs funded under the Workforce Investment Act of 1998 (WIA). The study is taking place in 28 randomly-selected local workforce investment areas across the USA. Two follow-up surveys are being conducted with over 6,000 study participants at 15 and 30 months after random assignment.

Surveys are available from Mathematica Policy Research.

REFERENCES

Abidin, R.R., and J.F. Brunner. “Development of a Parenting Alliance Inventory.” Journal of Clinical Child Psychology, vol. 24, no. 1, 1995.

Amato, P.R. Reconciling Divergent Perspectives: Judith Wallerstein, Quantitative Family Research, and Children of Divorce. Family Relations, vol 52, 2003, pp. 332-339.

Edin, Kathy, and Maria Kefalas. Promises I Can Keep: Why Poor Women Put Motherhood Before Marriage. Berkeley, CA: University of California Press, 2005.

Hsueh, J., D.P. Alderson, E. Lundquist, C. Michalopoulos, D. Gubits, D. Fein, and V. Knox. “The Supporting Healthy Marriage Evaluation: Early Impacts on Low-Income Families.” Washington DC: OPRE Report 2012-11, 2012.

Karney, B.R., and T.N. Bradbury. “The Longitudinal Course of Marital Quality and Stability: A Review of Theory, Method, and Research.” Psychological Bulletin, vol. 118, 1995.

Kroenke, Kurt, Tara W. Strine, Robert L. Spitzer, Janet B.W. Williams, Joyce T. Berry, and Ali H. Mokdad. “The PHQ-8 as a Measure of Current Depression in the General Population.” Journal of Affective Disorders, vol. 144, no. 1, 2009, pp. 163–173.

Stanley, Scott. “Assessing Couple and Marital Relationships: Beyond Form and Toward a Deeper Knowledge of Function.” Paper presented at Healthy Marriage Interventions and Evaluation Symposium of the Measurement Issues in Family Demographic Conference. Washington, DC, November 13-14, 2003.

Strauss M., S. Hamby, S. Boney-McCoy, D. Sugarman. The Revised Conflict Tactics Scales (CTS2): Deveopment and Preliminary Psychometric Data. Journal of Family Issues, vol. 17, no. 3, 1996, pp. 283-316.

Wood et al 2010 Wood, Robert G., Sheena McConnell, Quinn Moore, Andrew Clarkwest, JoAnn Hsueh. Strengthening Unmarried Parents’ Relationships: The Early Impacts of Building Strong Families. Mathematica Policy Research, Inc., May 2010.

Wood, Robert G., Sheena McConnell, Quinn Moore, Andrew Clarkwest, JoAnn Hsueh. “The Effects of Building Strong Families: A Healthy Marriage and Relationships Skills Education Program for Unmarried Parents.” Journal of Policy Analysis and Management, vol. 31, no. 2, 2012, pp. 228-252.