Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
Using Concept Mapping to Develop a Theory to Describe the Work of the National Domestic Violence Hotline with Family and Friends of Victims/Survivors – Formative Data Collection
Formative Data Collections for Program Support
0970 - 0531
Supporting Statement
Part B
July 2020
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officers: Tia Brown and Kriti Jain
Part B
The overall goal of Phase 3 of the SAF-T study is to develop a framework that accurately reflects the work of The Hotline as it relates to family and friends of victims/survivors. This study will use group concept mapping (GCM) data collection activities to revise the original survivor-centered framework developed in Phase 1 (OMB # 0970-0356) or create a new theoretical framework that applies to family/friends who contact The Hotline.
The objectives of the GCM activities are to:
Obtain stakeholder perspectives regarding ways The Hotline can support family and friends of victims/survivors,
Rank these ideas or constructs and explore the relative importance and usefulness of each of them, and
Identify key indicators for a theoretically-based framework for ongoing monitoring and assessment.
This study is intended to yield results to inform the development of a new or expanded theoretical framework on The Hotline’s work with family members and friends of victim/survivors. The GCM results are not intended to be generalizable to other agencies or populations.
GCM is, “a structured process, focused on a topic or construct of interest, involving input from one or more participants that produces an interpretable pictorial view (concept map) of their ideas and concepts and how these are inter-related.”1 It is a mixed-method research methodology that helps explain how a group views a topic or aspect of a topic through a visual display. GCM includes sequential steps ranging from recruiting stakeholders to brainstorming concepts, sorting and rating concepts, analysis, and interpretation.
GCM is the appropriate method for this study because it is designed for contexts in which groups of stakeholders with different views on a subject present, arrange, and organize ideas that represent theoretical concepts. The core of GCM consists of eliciting ideas or concepts, sorting and ranking them, and applying multivariate statistical methods to analyze input from all participants. The product is an aggregate representation of the stakeholders’ input. The method uses specialized computer software to collect responses from participants and then to analyze and map findings.
GCM results are intended to inform the development of a new or expanded theoretical framework for the work of The Hotline and are not intended to be generalizable to other agencies or populations. As noted in Supporting Statement A, Section A2, Study Design, the GCM method contributes to the production of a theoretical framework that is based in key stakeholders’ perspectives, but is not meant to produce findings generalizable beyond the stakeholder groups represented in the data. Additionally, given that the proposed sampling methods will produce a convenience sample, some bias may be introduced. It is possible that the data may reflect individuals who are more open to sharing their experiences and opinions.
As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.
The target population will be comprised of stakeholders representing a broad range of organizations and constituent groups relevant to the Hotline’s work with family members and friends of victims/survivors of relationship abuse. There will be five categories of stakeholders including (1) practitioners (e.g., staff of shelters, local or state hotlines, victim services, individual therapy/family counseling services, Family Justice Centers, and domestic violence service organizations), (2) staff/advocates of The Hotline, (3) victims/survivors of relationship abuse, (4) family members of victim survivors, and (5) friends of victims/survivors. Approximately 20 individuals age 18 or older will be recruited from each of the five stakeholder groups for a total of 100 participants.
The SAF-T project team will employ a purposive sampling approach, leading to a snowball or chain sampling approach to recruit respondents; the specifics of the approach will vary by stakeholder type.2 In light of the diversity among stakeholder groups as well as the difficulty in accessing some of these populations (e.g., family/friends of victims/survivors), the SAF-T project team has designed recruitment strategies tailored to accommodate these differences and to yield the desired number of respondents. Additionally, these methods will allow for potential participants to opt out before ever being contacted by the research team, maintaining full anonymity. See Appendix B: Recruitment Strategies for GCM Activities for an overview of stakeholder recruitment strategies, which includes a “confidential route” and an “anonymous route.” These two routes are summarized in the sections that follow.
The SAF-T project team will employ non-probabilistic methods including purposive sampling to recruit stakeholders within targeted organizations with domestic violence practitioners (e.g., shelters, local or state hotlines, victim services, individual therapy/family counseling services, Family Justice Centers, domestic violence service organizations) and snowball sampling to recruit stakeholders within their respective personal and professional networks. An introductory email will invite the stakeholders to participate in online GCM activities and ask the stakeholder to forward the email to other people in their networks who meet the criteria for stakeholder participation (e.g., other domestic violence practitioners, family and friends of victims/survivors who are not in crisis). In addition, The Hotline management staff will recommend staff members, called “Advocates,” who respond to text, chat, and calls to The Hotline. The SAF-T project staff will send each Advocate the introductory email inviting them to participate. The introductory email will ask stakeholders to provide their email address to participate and project staff will email them a username, password, and link to the GCM site to begin the brainstorming activity (see Appendix A.1: Introductory Recruitment Email for GCM Activities). Next, the project team will email an invitation to each stakeholder to participate in subsequent sorting and rating GCM activities (see Appendix A.2: Recruitment Email for GCM Brainstorming Activity; Appendix A.3: Recruitment Email for GCM Sorting and Rating Activities; and Appendix A.4: Recruitment Email for GCM Rating Only Activity), as well as one of three interpretation webinar discussions (See Appendix A.5: Recruitment Email for GCM Interpretation Webinar).
The anonymous route is specifically for recruitment of respondents from the last three stakeholder groups (i.e., victims/survivors, family of victims/survivors, and friends of victim survivors) who are users or “contactors” of The Hotline. Contactors who represent any of these three groups who are not in crisis will be asked toward the end of their text/chat/phone session if they would like to participate anonymously in a GCM activity underway at the time of the contact (see Appendix A.6: Recruitment Script/Instructions for The Hotline Contactors via Digital-based Services and Appendix A.7: Recruitment Script/Instructions for The Hotline Contactors via Phone-based Services). Advocates will use their judgement to determine whether a contactor is in crisis and if it is appropriate to introduce a contactor to the project. Contactors who are not in crisis and agree to participate will be provided with a link to connect to the GCM project website. These participants will remain anonymous and therefore will not receive email reminders regarding future participation. However, at the end of each activity, anonymous participants will receive a thank you message, which will include an invitation and instructions to participate in subsequent GCM activities (see Instrument 1: Online GCM Directions – Annotated for OMB Review).
Following the recruitment activities, respondents who agree to participate and access the GCM website will be given additional information via the informed consent process (see Appendix C.1 – GCM Online Consent Form – Confidential Route and Appendix C.2 – GCM Online Consent Form – Anonymous Route). The first page on the GCM website includes an informed consent that will explain participation is voluntary and how participants’ input will contribute to developing the theoretical framework.
Development of Data Collection Instrument(s)
The SAF-T project team and ACF staff provided input on the GCM design. In March 2020, the SAF-T project team developed two focus prompts and pilot tested these for the initial “brainstorming” data gathering activity and the GCM platform (i.e., groupwisdom™) with five stakeholder volunteers representing GCM target populations. In April 2020, the SAF-T project team shared the results from the pilot test with an expert panel (i.e., seven expert consultants who advise this project on major methodological and conceptual issues). The expert panel also tested the GCM platform, including each of the three online GCM activities (brainstorming, sorting, and rating), and provided feedback on the data collection materials, which included recommendations for content to ensure only useful information will be collected. The SAF-T project team revised the data collection materials based on feedback provided from the expert panel. The expert panel reviewed the revised data collection materials and provided no further recommendations for revisions.
The final GCM platform is designed to collect only the data necessary to perform GCM data analysis and yield results to address the project objectives and answer the specified research questions.
Stakeholder recruitment methods are outlined in preceding Section B2 and described fully / in detail in Appendix B: Dual Recruitment Strategies for GCM Activities. The GCM project includes three online data collection activities as well as interpretation webinar discussions with stakeholders who participate in one or more of the on-line GCM activities. Respondents will participate in the on-line GCM activities using groupwisdom™ software produced by Concept Systems, Inc. The four phases of the GCM project with brief descriptions are listed below.
Brainstorming Session. Stakeholders will contribute information and insight on factors that enable The Hotline to empower and support family and friends of victim/survivors. They will complete the following focus prompt or statement as many times as they would like: “I think one way The Hotline can support family or friends who contact The Hotline is …”
Sorting Session. Stakeholders then sort a de-duplicated list of responses to the brainstorming session into categories that make sense to them.
Rating Session. Stakeholders will rate the importance and usefulness of each response on a 5-point Likert scale (e.g., 1=not at all important/useful to 5=extremely important/useful).
Interpretation Webinar Discussions. Stakeholders will be asked to participate in an interpretation webinar discussion during which the SAF-T project team will share preliminary results of the GCM data analysis (e.g., concept maps or visual clustering of ideas) and get feedback from all participants through an informal discussion. There will be three one-hour interpretation webinar discussions – each targeting a different stakeholder group (note: only those who participate in at least one of the on-line GCM activities will be invited to join an interpretation webinar discussion). One interpretation webinar discussion will be held with each of the following stakeholder groups: (1) family/friends of victims/survivors, (2) victims/survivors, and (3) The Hotline staff and other practitioners. With permission from participants, discussion sessions will be audio recorded.
The initial brainstorming session will be open for approximately one month. Sorting and rating activities will follow brainstorming and also remain open for one month. Subsequently, the SAF-T project team will convene the three one-hour interpretation webinar discussions within a one-month period. Note that participants who participate in sorting and/or rating will also be asked to answer five participant questions related to: the participant group(s) with which they identify, their age group, their race/ethnicity, and their preferred method of contacting The Hotline (see Instrument 1: GCM Online Directions – Annotated for OMB Review). If participants do not want to share this information, all questions include an “I prefer not to answer” option.
Quality control activities for GCM include multiple data validation checks after data has been collected. First, rating responses are reviewed for completeness. If a respondent rated less than 80% of the statements, that respondent’s rating data will be excluded from the final dataset. For respondents who have completed 80% or more of the rating items for each rating scale, the data will be reviewed for variation in responses. If a respondent rated all items with the same rating (sometimes referred to as “straight-lining”), the data will be examined in the context of any additional data from that respondent in an attempt to identify whether the respondent thoughtfully rated statements or selected the same answer for every rating in order to complete the activity quickly. This process ensures that only respondents who participated intentionally have their data included in the final maps.
Next, sorting data is examined for completeness. If a respondent sorted less than 80 percent of the statements, that respondent’s sorting data will be excluded from the final dataset. Sorting data will be examined for conceptual integrity based on the number of sorted groups and the descriptive names created for each group by the respondent. If a respondent sorts all statements into one group, this data will be excluded. Most participants sort items into anywhere from two to 15 groups, with corresponding descriptive names assigned to each group. The number of groups, the statements within each group, and the descriptive names for each group are examined for each respondent to identify instances in which respondents may have randomly assigned statements to groups without considering the meaning of each item (which is rare). In this manner, the research team will examine the statement content for each group for conceptual cohesiveness within the context of how each respondent has sorted the data. Sorted data without group names will be considered carefully for conceptual integrity, and included if items did not appear to be grouped randomly. The research team will collaboratively examine the data in this manner and discuss disagreements until the team reaches consensus about whether to include or exclude a respondent’s responses.
GCM activities are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported.
As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Respondent demographics will be documented and reported in written materials associated with the data collection.
The data will not be used to generate population estimates, either for internal use or dissemination.
GCM data is collected via the online groupwisdom™ (Concept Systems, Inc.) platform. Collected data is viewable only by the two GCM specialists (Drs. Petrucci and Hartigan). Upon completion of the brainstorming data collection activity, Drs. Petrucci and Hartigan download and review the full statement set generated by respondents. Statements are reviewed for: (1) uniqueness of ideas and (2) singularity of statements. Drs. Petrucci and Hartigan review statements separately and then collaborate to compare suggestions for data cleaning. Both researchers must come to an agreement on the final cleaning decisions described below.
Uniqueness of ideas. Statements that are deemed substantively the same are combined into a single statement so that ideas are not duplicated within the final statement set. Note that two statements must be identical in substance and only vary in semantics in order to be considered duplicate. If they have nuance that makes them distinct, they are not combined. Statements are not re-written for syntax or grammar unless they are deemed to be too difficult to understand.
Singularity of statements. In the final statement set, each statement should describe only one idea. While respondents are prompted to include only one idea in each statement they generate, many still brainstorm statements that consist of multiple main ideas. These statements are split apart so that each statement reflects only one main idea.
Additional quality control data checks are done for the sorting and rating activities as well. As described in Section B4, once the sorting and rating activities are complete, participants’ data will be examined for completeness. See Section B4 for details. Minimal variation in responding is common in GCM. Therefore, each respondent’s data will be handled on a case-by-case basis.
The first steps in the data analysis process (multidimensional scaling and hierarchical cluster analysis) are carried out within the groupwisdom™ GCM platform. Using multi-dimensional scaling, the software first creates a point map, which visually depicts the relationships among statements based on participants’ sorting. On this point map, statements that are further apart are conceptually distinct and statements closer together are conceptually similar or related.
In the next step, using hierarchical cluster analysis, the software draws “boundaries” around groups of similar statements, forming clusters. The research team will indicate how many clusters to use as the starting point and then review each subsequent cluster map generated by the system. This process continues until the best cluster solution is agreed upon unanimously by the research team. Each cluster should represent a unique theme.
In the next step, the higher-level themes, referred to as the two-dimensional structure, will be ascertained based on the final cluster solution. This iterative process will include examining overlapping themes in each of four quadrants of the final cluster map and considering how the represented ideas change from one side of the map to the other. Additional maps will be produced to further examine the data; these maps include layered cluster rating maps, pattern matches, and go-zone maps.
The final GCM results will be shared during three interpretation webinar discussions with respondents who contributed to brainstorming, sorting, and/or rating to incorporate their perspective on the final results. Drs. Petrucci and Hartigan will walk participants through the maps used to determine the themes, starting from the point map and continuing through to the final map. The purpose of these interpretation webinar discussions is to obtain feedback from GCM participants on the final interpretation and where differences emerge and integrate participant feedback into the final interpretation of the concept maps.
The final results of the GCM, including the two-dimensional structure, interpretation of the four quadrants, and the final cluster themes, will inform development of the theoretical framework for friends and families of domestic violence victims who contact The Hotline. This theoretical framework will be included in all final reports, publications, and presentations generated for this project. In all presentations of findings, research design methodology and limitations with regard to generalizability will be described.
Lacey Hartigan, Ph.D. Senior Research Associate EMT Associates, Inc. (615) 678-1037
|
Carrie Petrucci, MSW, Ph.D. Senior Research Associate |
Instrument 1: Online GCM Directions – Annotated for OMB Review
Appendix A.1: Introductory Recruitment Email for GCM Activities
Appendix A.2: Recruitment Email for GCM Brainstorming Activity
Appendix A.3: Recruitment Email for GCM Sorting and Rating Activities
Appendix A.4: Recruitment Email for GCM Rating Only Activity
Appendix A.5: Recruitment Email for GCM Interpretation Webinar
Appendix A.6: Recruitment Script/Instructions for The Hotline Contactors via Digital-based Services
Appendix A.7: Recruitment Script/Instructions for The Hotline Contactors via Phone-based Services
Appendix B: Dual Recruitment Strategies for GCM Activities
Appendix C.1: GCM Online Consent Form – Confidential Route
Appendix C.2: GCM Online Consent Form – Anonymous Route
1 Trochim, W. M. (1989). An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning, 12(1), 1-16.
2 Patton, M. Q. (1990). Qualitative evaluation and research methods. New Park, California: Sage Publications
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Carol Hagen |
| File Modified | 0000-00-00 |
| File Created | 2022-05-04 |