revisedsupportingstatement_full study

TDV OMB Clearance

Supporting Statement

Information Collection Request

Supporting Statement for the

Teen Dating Relationships: Opportunities for Youth to Define what’s Healthy and Unhealthy

also known as the

Understanding Abuse in Teen Dating Relationships Through Concept Mapping Project

Project Officer:

Carrie F. Mulford, PhD

Crime, Violence and Victimization Research Division

Office of Research and Evaluation

National Institute of Justice

Office of Justice Programs

Department of Justice

810 7th Street NW
Washington, DC 20531
Phone: (202) 307-2959
Fax: (202) 616-0275
E-mail: [email protected]

Section A: Justification

A.1. Circumstances Making the Collection of Information Necessary…………………………………….3

A.2. Purpose and Use of the Information…………………………………………………………………..9

A.3. Use of Information Technology and Burden Reduction……………………………………………..12

A.4. Efforts to Identify Duplication and Use of Similar Information……………………………………..13

A.5. Impact on Small Businesses or Other Small Entities………………………………………………...13

A.6. Consequences of Collecting the Information Less Frequently………………………………………14

A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5………………………………...15

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency……15

A.9. Explanation of Any Payment or Gift to Respondents ……………………………………………….16

A.10. Assurance of Confidentiality Provided to Respondents……………………………………………16

A.11. Justification for Sensitive Questions………………………………………………………………..18

A.12. Estimates of Hour Burden Including Annualized Hourly Costs……………………………………20

A.13. Estimate of Other Total Annual Cost Burden to Respondents or Recordkeepers………………….22

A.14. Annualized Cost to the Federal Government……………………………………………………….22

A.15. Explanation for Program Changes or Adjustments…………………………………………………23

A.16. Plans for Tabulation and Publication and Project Time Schedule………………………………….23

A.17. Reason(s) Display of OMB Expiration Date is Inappropriate……………………………………...27

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions…………………………...27

Section B: Collections of Information Employing Statistical Methods

B.1. Respondent Universe and Sampling Methods……………………………………………………….27

B.2. Procedures for the Collection of Information………………………………………………………...28

B.3. Methods to Maximize Response Rates and Deal with Nonresponse………………………………...29

B.4. Test of Procedures or Methods to be Undertaken……………………………………………………30

B.5. Individuals Consulted on Statistical Aspects and Individuals Collection and/or Analyzing Data…..30

References………………………………………………………………………………………………...31

A.1. Circumstances Making the Collection of Information Necessary

The prevalence of adolescent dating violence has surfaced as a significant issue with implications for the criminal justice, public health and education fields. In a recent national survey, 9.8 percent of high school students reported being hit, slapped or physically hurt on purpose by their partner within the year prior to the survey (Centers for Disease Control and Prevention, 2009 Youth Risk Behavior Survey). Additionally, about 1 in 5 women and 1 in 7 men who ever experience rape, physical violence, and/or stalking by an intimate partner first experienced some form of partner violence between 11 and 17 years of age (Centers for Disease Control and Prevention, 2010 National Intimate Partner and Sexual Violence Survey). Unfortunately, these estimates describe only victimization, revealing only part of the story.  Information on the prevalence of perpetration is lacking, in part because the gaps in understanding of the dynamics of teen romantic relationships, the contexts in which these relationships exist, and the attitudinal perceptions that frame dating behaviors.  The need to more fully identify and explicate these issues is clear, as acceptance of dating abuse among friends is one of the strongest links to future involvement in dating abuse (Arriaga and Foshee, 2004, Bergman 1992). 

Existing research provides some evidence of conceptual disparities between adult and teen populations on relationship related topics. Previous studies indicate that teens may perceive certain emotions and behaviors differently than do adults and researchers. Teens, for example, tend to perceive controlling and jealous behaviors as signs of love (Levy, 1990), and “…do not perceive of dating aggression as deleterious to the relationship, nor do they view violence as a cause for ending the relationship” (O’Keefe, 2005). In other recent studies, researchers found that adolescent couples tended to work harder than adult couples to limit the intensity of negative exchanges (Galliher, Enno, & Wright, 2008), invested more in circumventing, minimizing and disowning the differences between them and their partner (Tuval-Mashiach & Shulman, 2006), and described their romantic relationships in mainly positive terms (Shulman & Kipnis, 2001).

These findings indicate that disparities between how adults and adolescents perceive certain behaviors and emotions in romantic relationships do exist; however, little research directly compares how teen and adult populations conceptualize overall healthy and unhealthy relationship behaviors and characteristics.

Furthermore, while existing research has examined distinctions between adult and adolescent dating behaviors, few studies have examined these dynamics from a standpoint of preventing teen dating violence. Recent research argues that “more research, particularly qualitative studies, are needed to enhance our understanding of adolescent dating violence, including the nature of relationship conflicts, as well as the meaning, content, intent and consequences of the violence” (O’Keefe, 2005). Before we are able to address teen dating violence prevention, however, it is “critical to understand how [emotions and behaviors] at the core of adult relationships…may be played out in adolescent relationships” (O’Keefe, 2005). The present study intends to strengthen our understanding of these characteristics in teen dating relationships.

Few studies to date have considered teens’ perceptions or definitions of dating behavior as described in their own words; rather, the majority of research on adolescent dating has been analyzed through measures and concepts developed by adult researchers and practitioners (Shulman & Kipnis, 2001, O’Keefe, 2005, Tuval-Mashiach & Shulman, 2006, Galliher, Enno, & Wright, 2008, Murphy & Smith, 2010). This field of research, therefore, lacks a definition of dating and dating behaviors that considers the perspective of adolescents, and signals the need to further identify and consider the developmental, social, and environmental components of their interactions, and mechanisms of influence that are specific to teen dating violence. While much of the prior research on teen dating violence apply adult frameworks and assumptions to characterizing relationships, “further research is needed to test these assumptions” and to determine their validity in being applied to adolescents (Murphy and & Smith, 2010).

To date, the Office of Justice Programs within the Department of Justice has conducted a pilot information collection (OMB Control #1121-0333) with a small group of teens ages 14-18 to assess the use of concept mapping as an appropriate tool for creating an adolescent-based conceptual framework that can be used to better understand how the youth population considers relationship behaviors. Additionally, the purpose of the pilot was to test the data collection processes (concept mapping and facilitated discussions) in order to refine both the recruitment procedures and the project research questions before conducting a study with a larger group of youth ages 11-22 and adults. The insight gained from this pilot has proved instrumental in enhancing the participation approach and refining the necessity for the more comprehensive information collection that is detailed in this supporting statement.

In accordance with the concept mapping process, approximately 50 teens in the DC and New York area were recruited through youth-serving organizations to submit ideas through a dedicated project website that completed the sentence: “A specific thought, feeling, action or behavior that teens in dating relationships might have or do is…” A final set of 86 unique ideas were elicited through this brainstorming activity. The teens were then asked to sort these ideas into piles in ways that made sense to them (sorting activity), and to rate these ideas on their relative frequency in teen dating relationships and their relative desirability as part of teen dating relationships (rating activity). The analysis and aggregation of the teens’ sort data formed the basis of the conceptual framework (Figure 1). The emergent concept map reveals nine higher-order themes by which the ideas can be considered as a meaningful structure to understand how teens conceptualize thoughts, actions, feelings and behaviors related to teen dating relationships.

Figure 1. Pilot Concept Map

The agency and research team facilitated discussions with the project Planning Group and teens in the DC and New York areas to interpret the meaning of the results. The output of these discussions further contributed to refining the goals of the subsequent information collection. Participants in all interpretation sessions recognized the emergence of two latent dimensions by which the content was structured on the map (Figure 2). Teen and Planning Group participants suggested that the map content described increasingly healthy, or positive, aspects of dating relationships the farther “south” one looked along the “north-south” dimension. Likewise, ideas described relatively increasingly unhealthy or negative aspects of teen dating the farther “north” one looked along this same dimension. A review of the content as it appears along the “east-west” dimension suggests that the farther “east” one reads, the more internal to the individual the ideas are in content. This gradation coincides with the location of most feelings-related statements on the eastern side of the map. On the contrary, the farther “west” one reads on the map, the more external, or public, the ideas are in content.

Unhealthy

Healthy

Internal/Private

External/Public

Figure 2. Concept Map: A Dimensional Perspective

Subsequent information collection among a broader age range of youth and adults is necessary to determine whether these dimensions emerge when young adults (19-22) and adults conceptualize characteristics of teen dating relationships. The broader participant group will also include a more demographically and geographically varied respondent universe, thereby allowing for an overall more diverse sample with which the pilot map results can be compared. In this sense, the dimensions that emerged in the pilot concept map provide a basis by which the agency can compare subsequent collections to better determine how, if at all, teens of various age ranges and adults understand the relationships among thoughts, actions, feelings and behaviors in fundamentally different ways. The Healthy/Unhealthy and Internal/External or Private/Public gradients provide benchmarks for further examining convergence and divergence in youth and adult conceptualizations.

Facilitated discussions with teen participants also illuminated specific statements and clusters, particularly those located toward the center of the map, that could be perceived as healthy or unhealthy depending on the specific context or circumstances in which they occur. Discussion with project Advisory Group members suggested that this ambiguity or continuum along which teens might consider a certain idea as healthy/unhealthy or desirable/undesirable may be reflective of their stage of development or level of experience with dating relationships. Subsequent information collection that includes a wider age range of respondents is necessary to better understand whether such ambiguity or context dependency exists among teens and young adults, or whether it is more likely a factor of teens’ developmental stage.

Analysis and aggregation of all pilot concept mapping participants’ ratings data also provided a basis for further inquiry. As the Pattern Match (Figure 3) demonstrates, the correlation between the average Frequency and Desirability ratings by cluster is 0.89, indicating that perceptions of frequency are highly predictably aligned with their perceptions of desirability.

Figure 3. Frequency and Desirability Pattern Match: All Participants

Additional analyses that examined the correlations between the Frequency and Desirability ratings by participant subgroups (age, sex, dating experience) also revealed near perfect correlations (.96, .99 and .95, respectively). Despite these considerably high correlations, consultation with project Planning and Advisory Group members supported the inclusion of these rating scales in the larger study, as the broader participant group may yield further insight into variation in participant ratings. One of the critical steps toward discerning and better understanding commonalities and variation in the opinions of participant groups is to broaden the respondent universe. Although this study does not intend to be nationally representative or generalizable, the information collection outlined in this supporting statement includes a more geographically diverse sample of respondents than the pilot, in part to allow the agency to examine differences among teens, young adults and adults to determine potential variation across a wider age range of participants. A broader sample will also include a more racially, ethnically and socioeconomically diverse respondent universe, from which the agency may derive insight about demographic variables associated with different rating responses that could be more deliberately researched in future inquiries. Also, because the adult respondent universe will intentionally include a subset of teachers, practitioners, researchers and advocates, the information collection may reveal any variation in perceived frequency and desirability across professional affiliation, which may have important implications for how the field addresses teen dating violence. As such, the present information collection is necessary to determine whether variation in these rating scales emerges among a more diverse respondent universe that did not emerge in the more limited pilot participant group, and what implications this variation may have for future research and the field at large. Appendix G includes a comprehensive summary of the pilot collection, including planning, recruitment, methodology, results and interpretation.

A.2. Purpose and Use of the Information

The results of the pilot information collection provide a rich foundation for better understanding how youth conceptualize teen dating relationships through the use of concept mapping. The present information collection intends to enhance the information gathered in the pilot to better understand how both adolescents and adults think about dating relationships, and to provide the field with guidance for future research inquiries and methodological tools for undertaking such research.

The pilot information collection has confirmed the feasibility of concept mapping as a means for arriving at a conceptual framework that will allow for an initial understanding of how youth conceptualize relationships. The pilot also confirmed that the content derived from the concept mapping process comports with broader research questions regarding how youth understand the range of healthy and unhealthy characteristics associated with teen dating. Given the appropriateness of the method to illustrate the perspective of teens on this topic, a primary purpose of the present information collection is to include adults and a broader sample of youth in the concept mapping process as a means to understand how the conceptualizations among these respondent groups comport with one another.

Each stage of the process will allow for such comparison. As a result of brainstorming activity, the research team will be able to compare the statements generated by youth and adults, and develop a single set of ideas for sorting and rating that represent a synthesis of both respondent groups’ ideas. As a result of the sorting activity, the research team will be able to compare the emergent frameworks and determine a) to what extent, if at all, the dimensions that emerged in the pilot framework are reproduced and/or b) specific areas of convergence and divergence in how youth and adults conceptualize teen dating relationships. The process will allow the research team to compare cluster solutions among both groups to reveal any tendencies in how youth and adults categorize features of dating relationships. For example, do adults and teens differ in the number of piles into which they sort ideas, thereby suggesting that one group may conceptualize the topic more specifically or distinctively than the other? As a result of the ratings activity, the research team will be able to compare the extent to which youth and adults (and subgroups of both populations) perceive the relative frequency and desirability of the ideas. This comparison will be instrumental in illuminating any areas of disagreement between how teens and adults working in the field perceive what occurs in dating relationships. Finally, the facilitated discussions will enrich the conceptual framework even further by allowing groups of youth and adults in various geographic locations to comment on their interpretation of the map and its potential utility.

In addition to providing a set of constructs that capture the conceptualizations of youth and adults, the results of the concept mapping and discussions will be used to generate further research questions on the topic of teen dating relationships. The respondent universe for the present information collection will be more geographically diverse than the pilot, but will still include a convenience sample based on youth-serving organizations and a select sample of adult researchers, practitioners, teachers and advocates that are nominated by project Planning, Advisory Group and Federal Interagency Workgroup on Teen Dating Violence members. As such, the insight derived from this information collection will provide the basis for future research with samples that are intentionally representative of demographics such as race, ethnicity, sexual orientation and socioeconomic status, and in further determining more generalizable distinctions in how demographic subgroups conceptualize teen dating relationships. Such distinctions may provide important insight in the development of effective teen dating violence prevention efforts that are specifically designed to be implemented among certain populations.

Additionally, the concept mapping process will produce methodological tools that can be used as a basis in future studies. The ideas generated through the brainstorming activity may be used as an “idea bank” that researchers can ask specific respondent groups( i.e. gay/lesbian teens, teens in rural communities, etc.) to sort and/or rate in order to generate a population-specific conceptual framework on the topic, and which can be compared with other frameworks using the same ideas. In an interpretation session with the project Advisory Group, several members noted aspects of the pilot results, particularly around the context dependency of whether certain ideas are perceived as healthy or unhealthy, that comport with some of their ongoing research initiatives. To this end, the results of this information collection may be used to confirm, supplement and/or enhance other studies on the topic.

In this regard, the information collection will create an initial conceptual framework and definitional model that can serve as inroads for more specific research efforts in the future. The broader purpose of this information collection and subsequent research is to ultimately advance our understanding of ways in which conceptualizations of dating relationship characteristics vary across and within groups. The results of this and later research can better inform the language and content of interventions and responses to teen dating violence, and may help to resolve existing definitional ambiguity and methodological variations across studies that have led to a lack of consensus and compounded challenges in accurately understanding the prevalence of and responses to dating violence (Barnett, Miller-Perrin, & Perrin, 2005). The emergence of the conceptualization patterns among the adolescent population, which concept mapping is able to achieve, is critical for the development of measurement approaches.

Finally, the work proposed here is critical, as an emergent framework that emphasizes the complex, interrelated aspects of teen dating violence from multiple perspectives (particularly those from youth) will help to advance a unified practice, research, and policy agenda. The Federal Interagency Workgroup on Teen Dating Violence comprised of representatives from 18 agencies, representing the Departments of Health and Human Services, Justice, Education, and Defense will review and use the collected information to shape future efforts to better understand the issues and organize effective responses to violence prevention and healthy relationship development. Without such information, planning and development may be limited in terms of appropriately matching language and communications, intervention and responses, and measurement and data collection with targeted beneficiaries.

A.3. Use of Information Technology and Burden Reduction

The majority of the information collected for the concept mapping portion of this study will be conducted via a dedicated project website. The website will be administered by Concept Systems, Inc. (CSI), the contractor for this project. This web based collection technique will reduce the participation time burden for respondents, as it will allow them to respond virtually and remotely, during time that is convenient for them. The project website will also allow for respondents to complete the concept mapping activities over multiple visits to the website (i.e. respondents will not need to complete the activities in one sitting). The use of information technology for the concept mapping will also reduce the financial burden for respondents, as they will not incur any travel expenses in completing the tasks.

The facilitated discussions will not use information technology, as the in-person, face-to-face context of these discussions will be critical to engaging participants in a productive conversation that will elicit the necessary feedback on the conceptual framework and will confirm its validity for use in teen dating violence prevention efforts.

A.4. Efforts to Identify Duplication and Use of Similar Information

This collection of information involves no effort to identify duplication. Although similar information was derived through the pilot information collection, this information is insufficient for fulfilling the aforementioned purposes of the present information collection. The present collection seeks to obtain information from a much broader range of participants (both youth and adults) in order to address those research questions that have been further refined since the pilot collection.

The Federal Interagency Workgroup on Teen Dating Violence has been meeting every 6-8 weeks since September 2006. During this time, participating agencies have collaborated on several joint efforts, including co-hosting an HHS-DOJ scientific 2 day workshop in December 2007 examining the research, both basic and applied, on teen dating violence, and determining research, evaluation and practice gaps for further study. Through consultation withorganizations and research partners and a review of existing literature on the subject, it has been determined that a project focusing on the research questions outlined in this data collection does not currently exist.

A.5. Impact on Small Businesses or Other Small Entities

The collection of information may impact not-for-profit youth-serving organizations or agencies with which researchers request assistance in youth respondent recruitment. These organizations will receive recruitment flyers and parental permission slips to distribute to potential youth respondents. In order to minimize the burden of these requests, the research team will mail a set of instructions and recruitment materials to organizational representatives who will be prepared in advance with distribution instructions. As part of the pilot study, the research team was able to confirm that the recruitment instructions and process materials are clear and efficient for organizational representatives to use appropriately. The research team will host a conference call with organizational representatives to answer any questions about the recruitment tasks, and will also provide ongoing support on an individual basis as needed. The research team anticipates that the recruitment requests will take organizational representatives no more than two hours total over the course of the information collection period.

The pilot information collection allowed the agency and research team to test the youth recruitment process in order to be as efficient as possible with organization time and resources. The recruitment instruments themselves (recruitment instructions, consent forms, assent forms, activity instructions) all proved to be clear, concise and easy to follow, such that no changes are needed. The pilot recruitment process revealed several important factors for the research team to consider in subsequent collections, including: 1) ensuring to the extent possible that brainstorming, sorting and rating recruitment does not coincide with school vacations and holidays, 2) seeking organizations with regular in-person contact with youth to facilitate reminders to turn in signed permission forms and/or complete the activities, 3) seeking organizations that regularly contact youth virtually, i.e. email, Facebook, etc., to facilitate website visits to complete the online activities, and 4) preparing organizational representatives as far in advance as possible of the necessary recruitment tasks.

A.6. Consequences of Collecting the Information Less Frequently

This information will be collected only once, although certain respondents may be asked to participate up to three to four times during the course of the study, depending on the tasks for which they are recruited. Responding to multiple participation requests is voluntary. Once participants provide their ideas on teen dating relationships (“brainstorming”), a selected subsample of participants will be asked 4 to 8 weeks later to sort the ideas into groups based on how they understand the ideas to be related (“sorting”). All individuals who provided ideas during the brainstorming activity will also be asked approximately 4 to 8 weeks after the brainstorming to rate the ideas on perceived frequency and desirability (“ratings”). The respondent re-contact interval is necessary to allow the research team to review the originally brainstormed statement set for clarity, relevance and redundancy, and to synthesize and finalize a set of ideas (100 or fewer) that is manageable in number for respondents to engage in the next stage of the research (sort and rate), thereby reducing burden.

A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

Concept mapping participants will be asked to respond to the brainstorming activity within approximately six weeks. Participants that are asked to respond to the sorting and rating activities will be asked to do so within approximately six weeks after the conclusion of the brainstorming activity. Responses to all portions of the concept mapping activities are voluntary, thus any individual who is not able to provide a response within the requested time period is not obligated to do so. The information collection outlined fully complies with all guidelines of 5 CFR 1320.5.

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency

The agency contracted with Concept Systems, Inc. to provide project management and to facilitate the concept mapping and facilitated discussion processes. In order to ensure the clarity of instructions for all collection materials, the agency also consulted with project Planning and Advisory Groups. These Groups are comprised of researchers, practitioners and advocates in fields related to teen dating violence, as well as representatives from youth-serving organizations. These Groups advised the agency on project design, and reviewed respondent materials for their readability to make sure that all youth participants would be able to understand and take part in the different project activities. The agency and research team has met with the Planning Group on a bi-weekly basis and the Advisory Group on an as needed basis since October 2010.

A.9. Explanation of Any Payment or Gift to Respondents

Teen and young adult respondents who complete the rating activities will receive a $10 Visa gift card. Teen and young adult respondents who complete sorting and rating activities will receive a $15 Visa gift card for completion of both activities. The difference in compensation amount reflects the additional 30-60 minutes of time it will take respondents to complete the online sorting task in addition to the online rating task. The agency decided to provide youth sorters and raters with a gift because the sorting and rating are the most time intensive tasks of the concept mapping activities, for which youth participation is critical in developing a useful, accurate conceptual framework. It is believed that by offering a token of appreciation, teens and young adults will be more likely to complete the sorting and rating tasks than they would if there was no incentive offered. It is critical that completion of the entire task is observed, as the validity and reliability of the final product is dependent upon a full response. An expression for appreciation for data collection and research activities for youth is common, and the amount offered in this effort is consistent with similar studies. Parental permission forms indicate that youth who complete each activity will be receiving a gift card, so that parents are made fully aware that their child may be receiving such expression of appreciation. Adult respondents will not be compensated for their participation in the concept mapping activities. The adult respondents will be mainly comprised of teachers, advocates, practitioners and researchers who will likely complete the tasks out of interest in the project and in contributing to the field.

A.10. Assurance of Confidentiality Provided to Respondents

Phase I: Concept Mapping

Youth participation in all components of the concept mapping process will be kept confidential. Each recruitment flyer that partner organizations distribute to participants will include a unique, randomly generated username and password, which will become the username and password for the individual participant who receives the flyer. (Organizational representatives that are assisting in recruitment will be instructed to only distribute flyers to individuals 17 years old or younger if they return a signed parental permission slip.) The username and password will allow each participant to log onto the project website to complete the concept mapping activities confidentially. Participants will only be identifiable to project management by their username.

The research team will include with the mailings of recruitment flyers and parental permission slips a Master Tracking List for each organizational representative to keep a record of the individual name associated with each username. No one affiliated with this project or project management will ever receive the names of the individual participants. The database serves only as a tool for the organization representatives to keep track of which participants are affiliated with which usernames, so that the researchers can communicate to the representatives which participants have completed the concept mapping activities and are therefore eligible to receive a token of appreciation. Gift cards will be distributed to the participants by organizational representatives. The pilot information collection confirmed the effectiveness and efficiency of this process for ensuring confidentiality to respondents.

Adult participation in the brainstorming component of concept mapping will be completely anonymous. However, adult participation in the sorting and rating activities of the concept mapping process will require that each adult participant access and complete the activities with a unique username and password. Due to project resourcing, the research team, rather than a third-party organization or agency, will be managing the distribution of these username and passwords to adult participants. The agency will maintain complete confidentiality of all input from adult participants, as the data elicited from participants will not be associated with any individual at any point in the project analysis or reporting. Adult participants will be made fully aware of these parameters and confidentiality assurance during the online consent process (See Appendices A8-A10.)

Phase II: Facilitated Discussions

Given the in-person, face-to-face context of the facilitated discussions, participant identity will not be kept anonymous. During the discussions, participants will be identifiable to one another by name tags that will display first names only. The research team will have a database of all participants’ contact information, as this information is necessary for communication, recordkeeping and travel reimbursement purposes. Youth participants ages 14-17 will only be allowed to participate if both assent and parental consent forms are received by the researchers in advance of the discussions (specific date to be determined). (Please refer to Appendix A for the assent and consent documents that participants and/or parents will receive.)

Any content from these discussions that is incorporated into the final project report will not be specific to any participant(s). Although the research team and the Planning Group will be using the content to enhance and support the final report recommendations, readers will be unable to trace statements or ideas to individual participants. The researchers will maintain a confidential database of participant names and contact information in the security of their Ithaca, New York office on a password protected server for three years after the dates of the facilitated discussions. This data will be kept in a password protected folder on the researcher organization’s server to ensure that only members of the research team will have access to this information. Participants will not be contacted after the conclusion of the project unless they express an interest in being contacted in the future. All participants will be provided with contact information for the research team members.

A.11. Justification for Sensitive Questions

All concept mapping participants will be asked to respond to the focus prompt, “A thought, feeling, action or behavior that teens in dating relationships might have or do is…” After extensive consultation with project Planning and Advisory Groups, the research team determined that this focus prompt was the best question to ask respondents in order to elicit the most useful content for the purposes of the study. The pilot information collection further confirmed that this focus prompt was appropriate for obtaining the breadth and depth of ideas necessary for the agency to sufficiently understand the range of characteristics that teens associate with dating relationships.

The statements derived from this focus prompt will be reviewed and synthesized by the research team and project Planning Group, to yield a set of 100 or fewer ideas that respondents will sort and rate in the subsequent concept mapping activities. These statements will represent the range of ideas that all respondents provided on the topic of teen dating relationships, and will become the basis of the resulting conceptual framework. The pilot facilitated discussions yielded two important insights to be considered in the idea synthesis process of the subsequent information collection. First, the project Planning Group should ensure that ideas included in the final statement set are clear in meaning. Statements that are ambiguous or unclear to the Planning Group members will likely also be difficult for the youth and adult respondents to understand and in turn difficult for respondents to sort and rate. Second, the final statement set should include more ideas related to the role of technology and social media in teen dating relationships (specifically text messaging, Facebook, etc.), as teens felt that these ideas were underrepresented in the pilot framework.

Teens may feel some sense of unease around the topic of dating, however we do not anticipate this discomfort to be more than what teens would likely encounter on a day-to-day basis. All participants will also have the option to abstain from any activity of the study at any time without penalty. In order to limit the discomfort that reading or listening to others’ answers and ideas may cause, the research team will feature a website (www.breakthecycle.org) at the beginning and end of each concept mapping activity. Upon visiting the website, participants will have access to a list of resources, including websites and phone numbers that participants can use to learn more about healthy teen dating behaviors and what to do if they feel that they are in an unhealthy relationship. A hard copy of these resources will also be distributed to facilitated discussion participants at the meetings. No issues regarding sensitivity arose among participants of the pilot information collection.

The information and instructions that will be provided to concept mapping and facilitated discussion participants, as well as consent forms for each group of respondents are included as appendices to this document. This informed consent process has been approved by the Institutional Review Board at SUNY-Cortland under Protocol # 101139.

A.12. Estimates of Hour Burden Including Annualized Hourly Costs

Four hundred participants will be invited to participate in the brainstorming task, which will take respondents no more than five to ten minutes to complete. One hundred and fifty participants will be invited to participate in the sorting task, which will take respondents approximately sixty to ninety minutes to complete. Four hundred participants will be invited to participate in the rating task, which will take respondents approximately forty to sixty minutes to complete. The total annual hour burden for a concept mapping participant will vary based on the specific combination of concept mapping activities he or she will be asked to participate in, but this hour burden will be no more than two hours and forty minutes at most.

Each facilitated discussion will be approximately three hours in duration and will require no more than approximately one hour of total travel time for each participant. These discussions require three hours of participants’ time in order to serve their intended purpose of confirming and enhancing the results of the concept map. The three hour agenda includes time for participants to introduce themselves, for facilitators to provide an overview of the discussion, review the concept mapping process and results map, and allow for ample discussion time. The three hours also includes multiple breaks for participants to shift focus, break into smaller group discussions, ask questions, and record on paper any additional thoughts or comments. We anticipate that the topic – teen dating – will be of particular interest to youth participants, and will therefore seem less burdensome that a discussion of a topic unrelated to their personal lives. There will be twelve facilitated discussions (four with adults comprised of twenty participants each, four with youth ages 14-18 comprised of ten participants each, and four with youth ages 19-22 comprised of ten participants each.

There are an estimated 540 annual total public burden hours associated with this collection. The agency has estimated the annual hour burden for respondents of the concept mapping and facilitated discussion phases based on the contractor’s extensive past experience administering concept mapping activities and facilitated discussions. Recent benchmarking data indicates participant completion rates of approximately 52% for web-based sorting and 59% for web-based rating (Rosas and Kane, 2012) ¹. The hour burden estimation for this collection was calculated using the higher end of the expected ranges of time each activity takes to complete, and assumes the average completion rates suggested by the concept mapping benchmarking data. It is highly likely that most participants will take less time than is reflected in the higher end time estimate.. The charts below reflect burden hours for both the higher and lower estimated time frames to complete each activity.

Higher end of time estimate range:

Task	Estimated time (minutes)	Total Participants	Total minutes per task
Brainstorming	10	400	4000
Sorting	90	78	7,020
Rating	60	236	14,160
Facilitated Discussions	180	40	7200
Total			32,380 minutes (~540 hours)

Lower end of time estimate range:

Task	Estimated time (minutes)	Total Participants	Total minutes per task
Brainstorming	5	400	2000
Sorting	60	78	4,680
Rating	40	236	9,440
Facilitated Discussions	180	40	7200
Total			23,320 minutes (~389 hours)

A.13. Estimate of Other Total Annual Cost Burden to Respondents or Recordkeepers

The DOJ anticipates no additional cost burden to respondents or recordkeepers beyond that which results from their customary or usually business or private practices.

A.14. Annualized Cost to the Federal Government

The contract to conduct the concept mapping project was competitively awarded to Concept Systems, Inc. The total time and materials contract estimate is based on a 30-month contract amount of $272,511.40. The following are cost estimates to the Federal government based upon activities anticipated over the next three years:

1. August-December 2012: $46,438.46

2. 2013: $187,616.79

3. 2014: $38,456.15

4. Total Federal Government Cost: $272,511.40

A.15. Explanation for Program Changes or Adjustments

The annual hourly burden has been adjusted in A12 from the burden indicated in the pilot in order to account for the additional number of respondents who will be asked to participate in this information collection. There is no change in the estimated time that each task will require of respondents. There is no change in the total annual cost burden to respondents or recordkeepers (A13). The annual cost to the Federal government has been adjusted in A14 to account for the remaining funds allocated to the contractor since the completion of the pilot information collection.

A.16. Plans for Tabulation and Publication and Project Time Schedule

Concept mapping will be the primary method for analyzing the data. The Concept System® software will be used to combine the individual participant’s sort data and, using several multivariate statistical algorithms, will organize the information and displays it in a series of easily readable concept maps. This process will begin with construction from the sort information of an NxN binary, symmetric matrix of similarities, Xij. For any two items i and j, a 1 is placed in Xij if the two items were placed in the same pile by the participant, otherwise a 0 is entered (Weller and Romney, 1988, p. 22). The total NxN similarity matrix, Tij will be obtained by summing across the individual Xij matrices. Thus, any cell in this matrix could take integer values between 0 and the number of people who sorted the statements. The value will indicates the number of people who placed the i,j pair in the same pile.

The total similarity matrix Tij will be analyzed using non-metric multidimensional scaling (MDS) analysis with a two-dimensional solution. The solution will be limited to two dimensions because it is generally easier to work with two-dimensional configurations than with those involving more dimensions, (Kruskal & Wish, 1978). Ease of use considerations are important for decisions about dimensionality. For example, when an MDS configuration is desired primarily as the foundation on which to display clustering results, then a two-dimensional configuration is far more useful than one involving three or more dimensions (p. 58).

The analysis will yield a two-dimensional (x,y) configuration of the set of statements based on the criterion that statements piled together most often are located more proximately in two-dimensional space while those piled together less frequently are further apart. The x,y configuration will serve as the input for the hierarchical cluster analysis utilizing Ward's algorithm (Everitt, 1980) as the basis for defining a cluster. Using the MDS configuration as input to the cluster analysis in effect will force the cluster analysis to partition the MDS configuration into non-overlapping clusters in two-dimensional space. There is no simple mathematical criterion by which a final number of clusters can be selected. The procedure that is typically followed is to examine the initial cluster solution that was the maximum desirable for interpretation in this context. Then, successively lower cluster solutions will be examined, with a judgment made at each level about whether the merger seems substantively reasonable. The pattern of judgments of the suitability of different cluster solutions will be examined and the final number of clusters selected to preserve the most detail and still yield substantively interpretable clusters of statements.

The MDS configuration of the statement points will be graphed in two dimensions automatically by the Concept System program. This "point map" will display the location of all the brainstormed statements with statements closer to each other generally expected to be more similar in meaning. A "cluster map" will be also generated that displays the original statement points enclosed by polygon-shaped boundaries for the clusters. Additional reports will demonstrate similarities and differences in perception and opinion among the participants. Additional graphics, including pattern matches and bivariate scatter plots, will display stakeholders’ ratings of the ideas. The 1-to-5 frequency and desirability rating data will be averaged across persons for each item and each cluster. This rating information will be depicted graphically in a "point rating map" that will show the original point map with the average rating per item displayed as vertical columns in the third dimension, and in a "cluster rating map" that will show the cluster average rating using the third dimension. Two additional graphic and statistical analyses will be computed based upon the map results. A “pattern match” is defined as the bivariate relationship between the cluster average ratings for two groups, variables or occasions. This will be visually displayed as a ladder graph or pair-link diagram with two vertical axes that represent the two variables and horizontal lines connecting them to represent the ratings for each cluster. A standard Pearson Product-Moment Correlation (r) will be also computed to indicate the overall pattern match. In addition, standard descriptive statistics will be produced (mean, SD, N) that will enable significance tests of differences between ratings on clusters. Pattern match graphs will be used to assess consensus or differences of different participant groups on the relative frequency and desirability. A “go zone” is defined as a within-cluster bivariate plot of average statement ratings for two groups, variables or occasions. Like a pattern match, it also displays the Pearson Product-Moment Correlation (r) between the two variables. The plot is a restricted form of a standard bivariate plot in that it: (a) sets the minimum and maximum values for all plots to the same range (based on minimum and maximum statement average for that variable); and (b) the bivariate space is divided into quadrants based on the cluster average of the x and y variables. This effectively means that every go zone plot will have a quadrant that shows which statements in the cluster were rated above average on both variables, one that shows which statements were below average on both and two that show the statements that were above average on one and below on the other. This plot, like a pattern match, will be used to explore consensus, in this case, within-cluster.

The content derived from the facilitated discussions will be integrated into the final conceptual framework through an iterative process of data reduction and synthesis. It is anticipated that the facilitated discussions will yield a substantial volume of information, from which information can be extracted to further elucidate emergent concepts. Using the concept mapping framework as the analytic structure for organizing, reducing and analyzing the content captured during the facilitated discussions, we expect to identify insights, themes, patterns and constructs that will aid in the interpretation of the domain. Traditional approaches to qualitative data reduction, coding, and analysis will be used within the concept mapping structure to illuminate the connections individuals make between and among the concepts.

The following table indicated the project time schedule:

Task	Month after receiving OMB Approval
Obtain IRB Approval and OMB Clearance	0
Concept Mapping	1-8
Brainstorming	1-2
Idea Synthesis	3
Sorting and Rating	4-5
Analysis	6
Preliminary Interpretation	7
Draft Concept Map Report	7-8
Facilitated Discussions	9-17
Planning and Participant Recruitment	9-10
Conduct Facilitated Discussions	11-13
Conduct Content Analysis of Facilitated Discussions to Link to Framework	14-15
Create Report of Facilitated Discussion Activity	16-17
Final Reporting and Data Aggregation	18-20

A summary report will be developed that will include the aggregated concept mapping and facilitated discussion results. The summary will also include a statistical report of the participant response rate for the sorting and rating activities. At the conclusion of the study, the data will be archived at the National Archive of Criminal Justice Data (NACJD).

A.17. Reason(s) Display of OMB Expiration Date is Inappropriate

Not applicable. The agency is not seeking approval to not display the expiration date for OMB approval of the information collection.

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions

Not applicable. There are no exceptions the Certification for Paperwork Reduction Act Submissions for this information collection.

B. Collections of Information Employing Statistical Methods

B.1. Respondent Universe and Sampling Methods

The following table displays the number of individuals that will be invited to participate in the different information collection tasks:

Concept Mapping Participation Targets
Task	Preteens (11-13)	Teens (14-18)	Young Adults (19-22)	Adults	Total task target
Brainstorming	50 100		100	150	400
Sorting	0 50		50	50	150
Rating	0 125		125	150	400
Total group target					400
Facilitated Discussion Participation Targets
Suggested location	Preteens (11-13)	Teens (14-18)	Young Adults (19-22)	Adults	Total regional target
Washington, DC	0 10		10	20	40
Atlanta	0 10		10	20	40
Chicago or Kansas City	0 10		10	20	40
San Francisco	0 10		10	20	40
Total group target	0 40		40	80	160

This information collection will not require purposefully sampling on the basis of any specific social demographics other than age, although every effort will be made to include youth representing various demographic groups (e.g., racial, ethnic, SES, religious, sexual minority).

B.2. Procedures for the Collection of Information

The information collection approach for this project will be to purposefully sample on the basis of heterogeneity, which is to non-randomly select a broad range of persons who are likely to reflect the full range of ideas as possible that are relevant to the topic under investigation. However, the adequacy of ideas that are captured is dependent upon whether there is a match between the focus and the participants selected to participate. We are not expecting that the results will be generalizable to the youth and adult populations at large; however we will be looking for saturation of the topic, as bounded by the participant groups. Thus, we anticipate that in the brainstorming task, we will reach a point where there will be a fair amount of redundancy and some homeostasis in the topic. The goal is to achieve a broad sampling of ideas rather than a representative sample of individuals. Youth respondents will be recruited through youth serving organizations with collective access to a diverse group of youth ages 11-22 recommended by members of the project Planning and Advisory Groups. These groups will be selected in part through a nomination process that solicited recommendations for organizations from the project’s Planning and Advisory Groups. Within this group of youth serving organizations, we are also looking to purposefully stratify our sample by age across three age groups: 11-13 (preteens), 14-18 (teens) and19-22 (young adult), in order to capture the variation of perspectives as they may be influenced by age. The organizations that will be used for recruitment purposes will be instructed by the research team to recruit a specific number of youth within the specific age ranges for participation in each activity (See Appendices B1-B3). Inclusion of youth participants will be based on a first-come-first-serve criterion; those youth who are the first to respond to the organizational representatives’ advertisements will have the opportunity to participate. In addition, adult participants will be purposefully selected from Planning and Advisory Group nominations, based on the relevance of their professional roles to the youth population and their relationships (i.e. practitioners, teachers, advocates, researchers, etc.).

In terms of quality control, the sorting, rating and demographic/background data will be gathered directly over the web, thus eliminating any concerns about mis-entering or mis-reading handwritten data. The software has some constraints built in that prevent errors in the data entry. For example, since this is an unstructured forced-choice sort method (Weller & Romney, 1988), the software does not allow a statement to be placed in more than one group simultaneously. For the ratings, the software only allows legitimate entries (e.g., the integers 1-to-5). Before any participant data can be used in data analysis, the software requires that the concept map analyst visually inspect the data and indicate by checking a setting that the data are complete and useable. Without this check, the participant’s data will not appear in any subsequent screen that calls for data analysis. Because the Concept System® software was expressly designed to accomplish the concept mapping process and analysis, there is also no danger that the statistical analysis procedure might be mis-specified by the analyst.

B.3. Methods to Maximize Response Rates and Deal with Non-Response

The brainstorming task will be conducted completely anonymously for all respondents. The number of statements elicited in response to the focus prompt will be measureable, but there will no way to determine the actual response rate, as respondents will be able to provide as many statements as they choose without any way to identify which respondents provided which statements.

The response rates for sorting and rating will be calculated over the course of the information collection, as the project website administrators will be able to monitor the progress of each participant according to their username. In order to maximize response rates for brainstorming, sorting and rating, a reminder notice will be emails to all invited adult respondents at multiple points during the period for each task. Response rates for youth invited to participate in the sorting and rating will also be maximized by offering the incentives described in section A9 of this supporting statement, and by asking youth-serving organization representatives to remind youth participants to complete the activities. In addition, because the use of incentives to maximize completion of the sorting and rating tasks, we anticipate that our response rates will be higher than the estimates outlined in section B1 above.

The facilitated discussion response rate will be calculated based on how many invitees (10-20 per discussion) actually attend. Attendance rates will be maximized by sending invitees reminders of the discussions in advance of the sessions.

B.4. Test of Procedures or Methods to be Undertaken

Not applicable. No tests of procedures or methods will be undertaken.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collection and/or Analyzing Data

Concept Systems, Inc. (CSI) is the contractor that will be consulted on statistical aspects of the design, and that will actually collect and analyze the information for the agency. Individuals from CSI who will be contributing to the statistical design and analysis are as follows:

1. Mary Kane, Principal Consultant

2. Scott Rosas, Senior Consultant

3. Alyssa Goldman, Project Manager

4. Jennifer Royer, Junior Consultant

These individuals can be reached by telephone at (607) 272-1206.

References

Arriaga XB, Foshee VA. Adolescent dating violence. Do adolescents follow in their friends’ or their parents’ footsteps? Journal of Interpersonal Violence 2004;19(2):162–84.

Anderberg, M. R. (1973). Cluster analysis for applications. New York, NY: Academic Press, Inc.

Barnett, O., Miller-Perrin, C. L., & Perrin, R. D. (2005). Family violence across the lifespan 2^nd Ed. Thousand Oaks, CS: Sage Publications.

Bergman, L. Dating violence among high school students. Social Work 1992;37:21–7.

Brown, J. (2006). Rethinking concept mapping for youth participatory evaluation in the context of youth development programs. Journal of Youth Development,) 1(2).

Cook, C., Heath, F., & Thompson, R. L. (2000). A meta-analysis of response rates in web- or internet-based surveys. Educational and Psychological Measurement, 60(6), 821-836.

Coxon, A. P. M. (1999). Sorting data: Collection and analysis. Thousand Oaks, CA: Sage.

Everitt, B. (1980). Cluster analysis. 2nd edition. New York, NY: Halsted Press, A Division of John Wiley and Sons.

Galliher, R. V., Enno, A. M., & Wright, R. (2008). Convergence and divergence across multiple methods for assessing adolescent romantic relationships. Journal of Adolescence, 31, 747-769. .

Hays, S., & Scholla, K. (2003). Engaging Youth In Tobacco Policy Change: Results from a Concept Mapping Project in Madison County, Illinois. Paper presented at the 2003 Annual Meeting of Midwest Political Science Association, Chicago, Illinois, April 4, 2003.

Kane, M., & Trochim, W. (2007). Concept mapping for planning and evaluation. Thousand Oaks, CA: Sage Publications.

Kaplowitz, M. D., Hadlock, T. D., & Levine, R. (2004). A comparison of web and mail survey response rates.  Public Opinion Quarterly, 68(1), 94-101.

Krippendorf, K. (2004). Content analysis: An introduction to its methodology (2nd ed.). Newbury Park, CA: Sage Publications.

Kruskal, J. B., & Wish, M. (1978). Multidimensional scaling. Beverly Hills, CA: Sage Publications.

Levy, B. (1990). Abusive teen dating relationships: An emerging issue for the 90s. Response to the Victimization of Women and Children.) 13(1).

Levy B. Dating violence: Women in danger. Seal, Seattle, WA: 1991.

Mulford, C., & Giordano, P. C. (2008) Teen dating violence: A closer look at adolescent romantic relationships. NIJ Journal, 261, 34-40.

Murphy, K. & Smith, D. (2010). Adolescent girls’ responses to warning signs of abuse in romantic relationships. Journal of Interpersonal Violence, 25(4),626-647.

O’Keefe, M. (2005). Teen Dating Violence: A Review of Risk Factors and Prevention Efforts. VAWnet: The National Online Resource Center on Violence Against Women (www.vawnet.org: Applied Research Forum). .

Osborn, A. F. (1957). Applied imagination. New York: Scribner.

Patton, M. Q. (1990). Qualitative evaluation and research methods 2^nd Ed. Thousand Oaks, CA: Sage Publications.

Ries, A., Voorhees, C., Gittelsohn, J., Roche, K. & Astone, N. (2008). Adolescents’ perceptions of environmental influences on physical activity. American Journal of Health Behavior, 32(1), 26-39. American Academy of Health Behavior.

Rosas, S. R., & Kane, M. (2012). Quality and rigor of the concept mapping methodology:  A pooled study analysis. Evaluation and Program Planning, 35(2), 236-245.

Rosenberg, S., & Kim, M. P. (1975). The method of sorting as a data gathering procedure in multivariate research. Multivariate Behavioral Research, 10, 489-502.

Schwartz, R. M. (1994). The skilled facilitator: Practical wisdom for developing effective groups. San Francisco: Jossey-Bass.

Shulman, S. & Kipnis, O. (2001). Adolescent romantic relationships: A look from the future. Journal of Adolescence, 24,337-351.

Silverman J, Raj A, Mucci L, Hathaway J. Dating violence against adolescent girls and associated substance use, unhealthy weight control, sexual risk behavior, pregnancy, and suicidality. Journal of the American Medical Association 2001; 286(5):572-9.

Snider, C., Kirst, M. Abubakar, S. Ahmad, F. & Nathens, A. (2010). Community-based articipatory research: Development of an emergency department-based youth violence intervention using concept mapping. Academy of Emergency Medicine, 17,877-885.

Tuval-Mashiach, R. & Shulman, S. (2006). Resolution of disagreements between romantic partners, among adolescents, and young adults: Qualitative analysis of interaction discourse. Journal of Research on Adolescence, 16(4)), 561-588.

Weller, S. C., & Romney, A. K. (1988). Systematic data collection. Newbury Park, CA: Sage Publications.

1 Online brainstorming allows for participants to anonymously contribute ideas via a dedicated project website. Participants may visit the site as often as they choose and may contribute as many ideas as they like within the allotted time frame. As such, it is not possible to accurately measure average participation in the brainstorming activity; therefore, for the purpose of this hour burden calculation, the total participant target is used as a conservative estimate.

33