APPENDIX F
|
Instructions for Instrument 3: |
Descriptive Evaluation Analysis Plan Template for HMRF Grantees |
The Administration for Children and Families (ACF), Office of Family Assistance (OFA) is requiring all Healthy Marriage and Responsible Fatherhood (HMRF) grantees conducting descriptive local evaluations funded by OFA to provide an analysis plan for their evaluations. Descriptive evaluations can include either an outcomes study or a process/implementation study, or both types of studies. Generally, an outcomes study seeks to answer questions about whether and how the outcomes for program participants change during the intervention. These changes in outcomes cannot be described as intervention impacts, because the study does not include a control group whose outcomes can be compared with those of the intervention group. A process/implementation study aims to answer questions about how the intervention was delivered in this particular application. Developing a structured analysis plan before examining outcomes or implementation data will foster an efficient and effective approach for analyzing the data and reporting the findings.
Below are definitions and instructions for completing the analysis plan for a descriptive evaluation. Grantees must provide information on all required sections. Please do not use the format of the instructions presented below to complete your descriptive evaluation analysis plan. Instead, please use the provided template (FaMLE_Descriptive_Analysis_Plan_Template) for this analysis plan. OFA strongly encourages grantees to require their evaluators to share this analysis plan internally among their team, and perhaps even with grantee staff, so that everyone understands the descriptive analysis plan and has an opportunity to discuss key decisions. The analysis plan can be considered an agreement between grantees and their evaluators regarding what research questions the evaluation will examine and the approaches to use to address those questions. The instructions presented below are organized as follows.
Section 1 provides guidelines for describing the intended intervention
Sections 2 and 3 provide a blueprint for describing the plans for outcomes and process evaluations, respectively, including the proposed research questions, the data that will be collected, and the methods that will be used to analyze the data and describe the findings.
These instructions have been created so that all grantees can fill out each section of the analysis plan regardless of the specifics of their evaluation, but some grantees may need to adapt some sub-sections to fit their approach. In addition to documenting your approach to the descriptive analysis, this document is designed to assist you in detailing information that you can use in final evaluation reports or other dissemination products.
Under the direction of ACF, your Evaluation Team Technical Assistance (ETTA) liaison will review your analysis plan to provide input and support as you draft it. Please email your analysis plan to your Federal Program Specialist (FPS) and copy your ETTA liaison by [date]. For consistency, please use this common naming convention when submitting your analysis plan:
[Grantee Name] Evaluation Analysis Plan.
Your FPS and ETTA liaison will review the analysis plan and provide comments and suggested edits and return it to you for revisions. Your analysis plan must be revised and approved by your FPS by [date].
ACF expects that evaluators will complete the analysis plans, with input from intervention or program staff as appropriate. For that reason, these instructions are mainly directed to the evaluators and include a few technical terms.
Describe the intended experiences of those in the intervention condition(s) (that is, what the intervention(s) aims to offer them). In particular, describe the following:
Intended components: Describe all of the key structural elements of the intervention (group classes, workshops, one-on-one services, etc.). If this is an intervention that includes multiple components, please describe all of them. For example, “This is a multi-component intervention in which parenting couples receive classes in relationship skills, workshops on economic stability topics, case management, and booster sessions.” If the intervention consists of adding services to a particular program, describe the program and all the additional services that are provided as part of the intervention. If the intervention consists of providing a number of services not related to a curriculum or program (for example, case management, counseling, home visits), describe each of the services.
Intended content: Describe the topics that the intervention covers and the resources/materials provided. Also, indicate whether specific content is based on a particular healthy marriage or responsible fatherhood curriculum.
Target population: Describe the target population, that is, provide information on the characteristics of the population that the intervention intends to serve, such as age, gender, marital status, and socio-economic status. For example, “The intervention is intended to be delivered to low-income unmarried fathers and their partners.”
Planned dosage and implementation schedule: Describe the number of sessions and the duration of the intervention. Include the length of each session and how frequently they occur. Describe variation in the frequency or length of sessions across sites, if applicable. For example, “This is an eight-month intervention, with sessions occurring once a week for two hours per session.”
Intended delivery: Describe where the intervention takes place and who delivers the intervention. Include information on the required education, gender, cultural background, and hiring requirements of the intervention providers/facilitator, and on the training and technical assistance that are offered to those providers as part of this study.
Please use a table to clearly and succinctly summarize intervention components. See Table 1 for an example (sample text included in italics). If there are different sets of services offered to different populations included in your descriptive evaluation study, consider creating separate tables for each.
Table 1. Intended intervention components
Component |
Curriculum and content |
Dosage and schedule |
Delivery |
Relationship skills workshops |
Healthy relationships curriculum: Understanding partner’s perspectives; avoiding destructive conflict; and communicating effectively |
20 hours, with 2-hour sessions occurring twice a week, or 4-hour sessions occurring every Saturday |
Group lessons provided at the intervention’s facilities by two trained facilitators in every session |
Economic stability workshops |
Resume preparation; interview and communication skills; appropriate work attire; financial literacy |
Monthly 2-hour workshops |
Workshops are provided by one facilitator |
Note: The intervention intends to serve low-income married couples. Male and female facilitators at the X location will deliver the intervention. Facilitators hold varying educational degrees, including bachelor’s and master’s degrees, and receive ongoing training in the intervention’s curriculum from study staff.
Describe the research questions, sample formation process, data collection procedures, outcome measures, and analytic approach for your outcomes study (if you are conducting one).
Research questions for an outcomes study typically focus on how participation in the intervention is associated with healthy relationship/marriage or responsible fatherhood outcomes such as the status and quality of couples’ relationships, the quality of co-parenting or parenting, and economic stability and well-being. Outcomes can be measured in a variety of ways, such as surveys, direct assessments, and observations. A best practice is to clearly connect the outcome(s) and the time point when they are measured with the intervention’s logic model or theory of change. Examples of research questions that specify an outcome and time point include:
“What is the likelihood that couples are married at the end of the intervention?”
“What is the level of support and affection that couples feel toward each other at the end of the intervention?”
“What is the level of fathers’ nurturing behavior and engagement in age-appropriate activities with children at the end of the intervention?”
“What is the employment status of fathers at the end of the intervention, relative to their employment status at the baseline?”
Provide the name of the Institutional Review Board that approved the study and data collection plans, and the date of approval (and the dates of any supplemental review approvals).
Describe the ways in which the members of the target population become part of the sample you will use to answer these research questions. Provide the information for all members of the sample, including agencies and schools from which they were recruited, all service locations/sites, and eligibility criteria to be part of the study. Include information on the following:
Sample eligibility criteria: Describe any required characteristics for sample inclusion (for example, age, marital status, involvement with the child support system, attending a particular school, geographical area, employment status, ability to speak and understand particular languages used for intervention delivery).
Purposeful sampling: Describe any additional criteria for selecting the sample beyond the eligibility criteria. These might include criteria to limit the number of clients who need to be tracked (for example, randomly selecting eligible participants, or only including specific classrooms in participating schools).
Describe, in detail, the consent process for enrollment in the study. Describe the timing, process, and materials used (such as the consent forms, incentives, etc.). Provide the start date of sample enrollment and the estimated date (month and year) for completing sample enrollment.
Describe the data sources for the analyses, for example, surveys of intervention participants or administrative data. Describe the timing of each data collection point (for example, at enrollment, at the first workshop X weeks after enrollment, at the final workshop Y weeks after the first workshop), and any other follow-up data collection time period(s) after the intervention ends. Describe the modes and methods of collecting data at each data collection point (in-person paper surveys, online surveys, phone interviews, etc.). Provide estimated dates (month and year) for starting and completing each planned data collection effort. Finally, please provide a copy of your data collection instruments in an appendix to your analysis plan (at a minimum, provide the instruments that you used to collect data used for your outcomes study).
A table may help summarize information on each feature of data collection (see Table 2; sample text included in italics).
Table 2. Sources of data to address the research questions
Data source |
Timing of data collection |
Mode of data collection |
Start and end date of data collection |
Intervention participants |
At the first workshop (1 month after enrollment) |
In-person online survey |
September 2016 through March 2019 |
Describe the specific outcome measures you will use to answer the research questions. If you will construct measures from multiple questions, describe the survey items you will use to create each construct and how you will code them to create the measure.
Complete Table 3 (sample text included in italics), describing all measures that you will use to answer the research questions of the outcomes study. Describe how you are operationalizing the outcome measure and the time periods when you will measure outcomes.
Table 3. Outcomes used to answer the research questions
Outcome name |
Description of the outcome measure |
Source of the measure |
Timing of measure |
Marital status |
The outcome measure is a yes/no response taken directly from the question in the survey, “Are you currently married?” |
nFORM exit survey |
A post-test (immediately after the intervention ends) |
Level of affection |
The outcome measure is a scale (value range 1 to 5) calculated from both partners’ responses as the average of five survey items measuring support, intimacy, commitment, trust, and friendship. |
Local follow-up survey |
3 months after the intervention ends |
The analysis plan should lay out the approach you will use to examine outcomes and answer the research questions. Define the analytic sample and describe your approaches to cleaning data and examining outcomes to answer the research questions, including your plans to deal with attrition.
Analytic sample. The analytic sample is the sample you will use to examine outcomes and answer the research questions. Describe how you will define the analytic sample (for each research question, if applicable). Clearly articulate what data you require for a person to be part of the analytic sample. For example, perhaps the analytic sample for the study will be people with complete baseline and outcome data for all variables of interest (that is, a complete-case sample). Alternatively, the analytic sample might be people who have complete outcome data but some missing baseline data, which will be imputed (as described below). Imputing outcome data is generally not an acceptable practice, so the primary analysis approach cannot include people with missing outcome data in the analytic sample. This approach can be used for secondary or sensitivity analyses, at the grantee’s option.
Data preparation. Describe how you will clean the baseline and follow-up data and prepare them for analysis. This includes whether and how you will merge or combine data from different sources. Describe in detail how you will handle missing data (such as the process to impute missing baseline data), and your plans to identify and handle responses that are inconsistent with each other or are seemingly inaccurate data, across both baseline and outcome (at post-test and follow-up) surveys. For example, if you are administering surveys to both members of a couple separately, describe the strategies you will use to verify that the answers are consistent, such as checking that both people report the same marital status.
Analytic approach. Describe how you will analyze outcome measures to answer the research questions. For example, explain whether you will describe summary statistics (for example, means, percentages) of outcome measures at the follow-up(s); compare means and other indicators at baseline and the follow-up(s) (that is, a pre-post analysis); conduct correlational analyses of the associations between implementation variables (such as dosage/participation) and outcome measures; and conduct growth-curve analyses. Provide the following information:
Model specification: Describe the type of model you will use to examine outcomes (linear regression, logistic regression, etc.). Define the criteria you will use to assess the statistical significance of the findings (for example, “findings are considered statistically significant based on p < .05, two-tailed test”). Specify the statistical software package you will use.
Covariates: List all potential covariates you plan to include in the models you will use to examine the outcomes of interest. If you have not determined the covariates yet, describe a plan for determining the covariates you will include in your analyses.
Attrition is the number of people in the baseline sample for whom follow-up was not completed or who are missing outcome data. Provide the following information:
Plans to minimize sample attrition. Describe the approaches you used to maximize participation in the follow-up data collection(s), that is, describe the strategies you used to follow up with study participants and increase response rates for the outcomes data collection efforts. For example, describe the number of attempts, time period, and mode of communication (phone calls, text messages, etc.) for reaching out to study participants during the follow-up(s) to collect outcomes data. In addition, describe the incentives you provided.
Approach to report attrition. Describe the approach you will use to report attrition from the baseline sample (the sample of study participants who completed the baseline data collection). This could include your approach to reporting (1) the proportion of the sample that is missing each outcome measure and (2) the difference in means of baseline variables/measures between samples with and without outcomes data. These steps will help you explore whether systematic differences exist between those two samples.
The May 23, 2019 webinar on non-response bias in descriptive evaluations presents the recommended approach to assessing non-response and considering this information when describing findings and implications.
If you are conducting a process/implementation study, describe the research questions you aim to answer, the data you will use to answer the research questions, and the methods you will use to analyze the data and describe the findings. This information will be useful in contextualizing the analysis of outcomes, generating hypotheses about changes in outcomes, and informing intervention improvement.
The research questions articulate the main hypotheses of your process/implementation study. Research questions can refer to implementation elements such as fidelity, dosage, quality of implementation, engagement, and context, as the following examples show:
Fidelity: Was the intervention (and each of its components, if applicable) delivered as intended?
Dosage: How much of the programming (or components, if applicable) did participants receive?
Quality: How well was the intervention implemented or delivered to participants?
Engagement: Did couples and/or fathers engage in intervention services, and if so, how engaged were they?
Context: What other types of services are available to intervention participants? What external events or unplanned adaptations may have affected implementation of the intervention as intended?
Table 4 provides examples (in italics) of research questions for each implementation element.
Table 4. Examples of research questions for each implementation element
Implementation element |
Research question |
Fidelity |
|
Dosage |
|
Quality |
|
Engagement |
|
Context |
|
Provide the name of the Institutional Review Board that approved the study and data collection plans, and the date of approval (and the dates of any supplemental review approvals).
Describe the sample you will use to answer these research questions. If you are conducting an outcomes study, describe whether and how the sample in the process/implementation study differs from the sample in the outcomes study. If not described anywhere else in the analysis plan, provide the following information for all clients in the sample, the agencies and schools from which they were recruited, and all service locations/sites:
Sample eligibility criteria: Describe any required characteristics for sample inclusion (for example, age, marital status, involvement with the child support system, attending a particular school, geographical area, employment status, ability to speak and understand particular languages used for intervention delivery).
Describe, in detail, the consent process for study enrollment. Describe the timing, process, and materials used (consent forms, incentives, etc.). Provide the start date of sample enrollment and the estimated date (month and year) for completing sample enrollment.
Describe the data sources you will use to answer the research questions (for example, nFORM surveys, fidelity protocols, attendance logs). Describe the timing and frequency of each data collection effort (for example, during all sessions, once a week, annually), and the party responsible for collecting the data. Use a table to link the information on data collection to the research questions. See Table 5 below for an example of such table (sample text included in italics). Finally, please provide a copy of your data collection instruments in an appendix to your analysis plan (at a minimum, provide the instruments that you used to collect the data you will use for your process or implementation study).
Table 5. Examples of data for addressing the research questions
Implementation element |
Research question |
Data source |
Timing/frequency of data collection |
Party responsible for data collection |
Fidelity |
Were all intended intervention components offered and for the expected duration? |
Workshop sessions in nFORM |
All sessions delivered |
Intervention staff |
Fidelity |
What content did the clients receive? |
Fidelity tracking log or protocol; attendance logs; session observations |
Every session for fidelity tracking and attendance logs; two times a year for session observations |
Intervention staff for fidelity tracking and attendance logs; study staff for session observations |
Fidelity |
Who delivered services to clients? |
Staff applications; hiring records; training logs |
One time X months after start of implementation; annually |
Intervention staff |
Fidelity |
What were the unplanned adaptations to key intervention components? |
Adaptation request; work plan; 6-month progress report; annual progress report |
Annually; ad hoc |
Intervention staff; study staff |
Dosage |
How often did clients participate in the intervention on average? |
Workshop sessions and individual service contacts in nFORM; attendance logs |
All sessions delivered |
Intervention staff |
Quality |
What was the quality of staff–participant interactions? |
Observations of interaction quality, using protocol developed by study staff |
X percent of sessions selected at random for observation |
Study staff |
Engagement |
How engaged were clients in the intervention? |
Observations of engagement, possibly using an engagement assessment tool; ratings from facilitator fidelity logs; engagement ratings from participant satisfaction surveys |
Y percent of sessions selected at random for observation |
Study staff |
Context |
What other HM/RF programming was available to study participants? |
Interviews with staff from partnering agencies in the community; survey items on baseline and follow-up assessments; websites of other agencies in the community providing HM/RF programming |
Once a year; ad hoc |
Study staff |
Context |
What external events affected implementation? |
Interviews with community/county representatives; list of site/school closures |
Once a year; ad hoc |
Study staff |
Describe the specific measures you will construct and your approaches to using those measures to answer the research questions. For example, describe whether you will calculate averages, percentages, and frequencies using the data you will collect for the study. In addition, include information on your approaches to examine and summarize qualitative data from interviews, focus groups, and observations. Use a table to link the description of the measures to the research questions. Table 6 presents an example of such table (sample text included in italics).
Table 6. Examples of measures for addressing the research questions
Implementation element |
Research question |
Measures |
Fidelity |
Were all intended intervention components offered and for the expected duration? |
|
Fidelity |
What content did the clients receive? |
|
Fidelity |
Who delivered services to clients? |
|
Fidelity |
What were the unplanned adaptations to key intervention components? |
|
Dosage |
How often did clients participate in the intervention on average? |
|
Quality |
What was the quality of staff–participant interactions? |
|
Engagement |
How engaged were clients in the intervention? |
|
Context |
What other HM/RF programming was available to study participants? |
|
Context |
What external events affected implementation? |
|
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Instructions for Instrument 3: |
| Author | Julieta Lugo-Gil |
| File Modified | 0000-00-00 |
| File Created | 2021-01-13 |