Appendix I Memorandum on Item and Construct Discussions with the Bureau of Justice Statistics

APPENDIX I

Memorandum on Item and Construct Discussions

with the Bureau of Justice Statistics (BJS)

Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011)

Spring Third-Grade National Data Collection, Fourth-Grade Recruitment, and Fifth-Grade Tracking

OMB Clearance Package

# 1850-0750 v.15

_MEMORANDUM UNITED STATES DEPARTMENT OF EDUCATION

Institute of Education Sciences

National Center for Education Statistics

To: Shelley Martinez, OMB

From: Gail Mulligan and Jill McCarroll, NCES

Through: Kashka Kubzdela, NCES

Date: July 19, 2013

Re: Documentation regarding discussions with the Bureau of Justice Statistics (BJS) about the peer victimization items proposed for inclusion in the third-grade child questionnaire

I. Background. NCES’s request for approval to conduct the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:2011) Pilot Tests of the Third-, Fourth-, and Fifth-Grade Direct Child Assessment, Child Questionnaire, and Online School Administrator Questionnaire (OMB# 1850-0750 v.13) was approved on 1/22/2013 with the following as part of the terms of clearance:

"While NCES is to be applauded for its proposal to develop and test using ACASI for younger audiences and for collecting comprehensive information on "peer victimization," NCES should recognize the methodological and subject matter expertise and interests of other statistical agencies and should take greater efforts to collaborate with them on these topics.  Specifically in this case, NCES should include colleagues from the Bureau of Justice Statistics in its discussions of pilot results and planning the full scale administration, both on the use of ACASI and on responsible uses of the victimization data given the different methods and questions being used to develop them in this study from those in production use by BJS for only a slightly older population.  NCES should demonstrate in its next clearance package how it has consulted in a meaningful way with BJS."

II. Initial Contact with BJS. In accordance with these terms, NCES consulted with Michael Planty and Jenna Truman, who work on the National Crime Victimization Survey, School Crime Survey, at BJS. This memo documents those consultations.

NCES staff sent BJS a copy of the Child Questionnaire (CQ) being tested in the 2013 pilot test, highlighting the proposed peer victimization items in this questionnaire. The parent interview and teacher questionnaire were also provided, since they include comparable questions on children’s peer victimization asked of those respondents. BJS staff were asked for any feedback they may have on the wording of the items or the ECLS-K:2011 staff’s rationale for including these items in the ECLS-K:2011 child questionnaire.

The following background information was also provided to staff at BJS:

-There was a desire to include a child questionnaire beginning in the third-grade round of the ECLS-K:2011, just like there was in the ECLS-K.  The CQ includes all the items from the self-description questionnaire, which was used in the ECLS-K, as well as items tapping newer constructs that were recommended by our content review panel (CRP). One of the constructs we have included items for is peer victimization.

- In the ECLS-K, the CQ was a hard-copy instrument that children completed after the cognitive assessment. Assessors entered children’s responses into their laptops at the end of the day. We wanted to move to ACASI to give children a greater sense of privacy as well as reduce burden on assessors and reduce the potential for data entry error. We’ve only been testing for about 2 weeks now, and the children seem to be doing OK with the ACASI. They have headphones and use a stylus pen to enter their responses.

- The purpose of our pilot really is to test the ACASI and the data collection methods, not the items themselves. Our schedule is very compressed, so we purposely picked items from established scales for which the psychometric properties are known. The one exception is the items on peer victimization.  

More specific background on the peer victimization items:

We are currently in the field (for the second-grade national data collection) asking parents and teachers four items tapping peer victimization, which we pilot tested (for parents only) about a year ago. When we initially submitted a request for approval to ask these items, we were calling them bullying items. At that time, OMB made us aware of the federal effort to develop items tapping bullying that could be asked in surveys across the government. At that time, we talked with Kathy Chandler who is working on this issue for NCES due to her involvement with the School Crime and Safety Survey. We realized that our items were not tapping bullying because they missed a key aspect of bullying (the real or perceived threat of repeated occurrences) and decided that we could not effectively tap bullying in items asked of parents and teachers. Another key issue was the bullying work group had not yet finalized their items, and we did not think it was a good idea to use interim items. Thus, we decided to use the four items on peer victimization that are adapted from existing sources

Now that we are asking questions of children, we could ask about bullying, but our CRP strongly recommended asking the same questions we ask parents and teachers (reworded slightly to fit the context/respondent) so that researchers could compare data across respondents. Before submitting our clearance request, we talked to Kathy again to see whether the bullying items had been finalized, and she said no. In light of that and the CRP’s recommendation, we decided to stay with the four items we have.

III. Feedback from BJS and NCES Responses. BJS reviewed the items and provided thoughtful feedback on them. This section describes that feedback, as well as NCES’s responses to the comments received.

1. BJS agreed with the general assessment that these items do not comport with the official, or rather working, definition of bullying and are better conceptualized as items assessing peer victimization.

2. BJS indicated it is important to consider the purpose of these peer victimization items and how they will be presented. They asked whether the items would be used to state victimization in the last year (e.g., X% of students were victimized in the previous year) and provided the following supporting information:

• We know from the NCVS and other victimization surveys that respondents telescope events into the reference period and that this over-reporting can be as great as 30% or more. The proposed items do not attempt to quantify the specific numbers of crimes/incidents just relative amounts which BJS indicated probably reduces the potential error. However, how do you assess the validity and reliability of the response categories? Have there been studies that demonstrate substantive differences between these categories (e.g., rarely versus sometimes; often versus very often)? Maybe there could be some quantitative guidance given (Never, 0 times; rarely, 1 or 2 times; Sometimes, 3 or 4 times a year; often, once or twice a week or every month; very often, on a daily basis or multiple times a week). BJS noted that they are not sure about the science behind this type of guidance/cueing.

NCES responded that these items are not intended to be used to obtain an estimate such as the one BJS asked about. ECLS staff recognize that producing statistics such as victimization in the last year is the purview of the NCVS School Crime Supplement. NCES noted that these items are included to obtain one more piece of information about the educational experiences of children in the ECLS-K:2011 that can affect their academic outcomes.

NCES also noted that we use these response categories for many items in the parent interview. The purpose of the peer victimization items really is to get a general sense of children’s experiences with peer victimization, not a specific number of instances. Very often it is the case that the items with response options like this end up being dichotomized in analyses because of the response distributions we obtain. We expect that we will not see much of a difference in outcomes between the children whose parents or teachers say “often” compared to those who say “very often.” The differences are likely to be seen between the extremes (e.g., rarely and very often). NCES noted that the recommendations for quantitative guidance associated with the categories are good, but there is some reluctance to provide guidance like that for some items in the instrument and not all of the items that have the same response categories. What would be considered “rarely” or “often” for one behavior or experience might not translate to the equivalent frequency for some other behavior or event.  If quantitative guidance was provided for these items, it might lead parents to use the same frequencies for other items.

3. BJS noted that the questions state “school year” and use “students” and asked if NCES is interested in peer victimization that occurs anywhere or only on school grounds and/or during school hours. BJS noted they have spent a lot of time with their definition of school victimization in the NCVS and it includes both location and activity. There are victimization incidents that occur on school property during summer days when school is closed, events that happen on the bus on the way to and from school, after-school activities, during sporting events on the weekends or even at another school. All may introduce confusion or error when answering victimization questions. There may be simple ways to modify the items or instructions in the ECLS-K:2011 items to clarify to the child as to what should be included.  

NCES responded that we are interested in peer victimization that occurs anywhere, not just on school grounds or during school hours. This is why the questions are asked of both parents and teachers. Similar questions are asked of both parents and teachers for several of our constructs, because children’s experiences in different contexts can be different. For example, we ask both parents and teachers about children’s social skills and behaviors since children can behave very differently at home than they do at school. Also, a parent may know little about what happens to the child at school, and vice versa for teachers and the home environment.  We use the term “students” to convey that we are talking about victimization by the child’s peers or school-age students. We are not trying to assess abuse by adults. We also do not want parents to report on interaction with siblings. Some help text was added to the third-grade parent interview to clarify this.

NCES noted that we do understand that teachers are likely to report only what they know about the child’s experiences at school. This is a consideration people using the data will have to keep in mind and is a consideration that applies to many of our ECLS items.    

4. BJS asked why the ECLS would ask the child, teacher, and parent the same questions about victimization and wondered how a researcher would decide which estimate to use. They indicated that we know that the further one is away from the event/incident the less he or she knows about it and the less likely he/she is to report it accurately. BJS suspected that parents would have very little information and/or possibly selective information if they only know about the severe instances that involved their child because it was reported to them by the principal. BJS also provided the following observations: Parents are likely to generalize across many school grades and talk in general about their child’s experiences, going beyond the current school year. They may also expand the scope to issues outside of school when an incident involves another neighborhood child who also attends the same school.  Teachers on the other hand are only likely to know the student in that given year. BJS concluded this comment by asking how a researcher would negotiate between a high victimization rate given by the child, the teacher’s estimate, and probably the lowest estimate by the parent, and wondered whether there is value in making comparisons between these sources of information. While they believed there is value, given the various differences and potential sources of error, BJS thought it might be hard to sort out why the estimates are different.

NCES addressed this issue to some degree with the response to the previous comment.  NCES noted that another issue is that we believe the ECLS-K:2011 can make a unique contribution to the study of peer victimization from a methodological standpoint because the study obtains this information from different reporters. We expect parents and teachers to be reporting with different knowledge about the child’s experiences and in different contexts. The question then becomes which reports are more associated with children’s outcomes? Is it what happens to children only in school (from teachers), or is a more general report about children’s experiences from parents more strongly related to outcomes? Or are both reports uniquely and strongly predictive? Regarding the data collected from both parents and teachers about children’s social skills, NCES had found that in later grades teacher reports are better predictors of children’s assessment scores, which led us to drop the social skills items from the parent interview in later rounds. This would not have been known if the study had not asked both parents and teachers the same (or very similar) questions in earlier rounds.

5. BJS asked whether these items will be vertically aligned across grades to allow for comparisons. They noted that the research with the NCVS demonstrated that they could not interview children under 12 with the same instrument as adults due to cognitive differences.  They wondered whether there is evidence that children in kindergarten and 3^rd grade differ substantially in terms of how they view victimization. Along the same lines, they asked what NCES had planned to allow for a crosswalk for the estimates from grade to grade by mode differences, given that the CQ at third grade would be using ACASI. BJS noted that the literature does show evidence for a substantial increase in the reporting of victimization when privacy issues are addressed. Their concern is that the ECLS-K:2011 could show an increase in victimization across grades without knowing if and/or how much of the increase was due to mode.

NCES responded with an apology for not making it clear that third grade would be the first time these questions would be asked of children. The questions could potentially be asked up to three times (third, fourth, and fifth) in the ECLS-K:2011 but would always be asked using the same mode. NCES noted that the victimization questions were currently being asked of parents and teachers in the spring second-grade collection.

6. BJS had the following specific questions about the items in the parent questionnaire:

• Why are the questions split (yes/no and how often) in the parent questionnaire, but not on the child or teacher questionnaire?

• Why is the ‘other students told lies or untrue stories’ question missing from the parent questionnaire?

NCES responded that the difference in format is simply an issue of time. The ECLS staff was looking for ways to cut the length of the parent interview, which is conducted by telephone. Asking these questions as yes/no and then only reading the frequencies if a parent says “yes” to the initial question saves time. NCES did not include the item about lies and stories in the second-grade collection because the ECLS staff thought it might be difficult for parents to answer and then would be subject to error.  (Note that due to strong encouragement from the CRP who wanted the same four victimization questions asked of parents, teachers, and children, this item is being included in the parent questionnaire for third grade.)

7. BJS had the following specific comments about the items in the teacher questionnaire:

• Questions are not split, similar to student questions.

• Interesting that these items ask teachers about the actions students take as well as the victimization against them. Are you interested in asking these “offending” (of sorts) questions to the children?

NCES responded that the Department of Education is constrained in what it can ask students. NCES cannot ask children about these topics without explicit parent permission, as they are seen as reports of self-incriminating behavior.

III. Final summary feedback. BJS felt that NCES’s responses to their questions and comments were reasonable. They indicated that they did not see anything that is particularly sensitive or of concern in asking the peer victimization items of children, given the context for the questions that NCES described in their initial background about the items and in their responses to comments.

It should also be noted that while the NCVS School Crime Survey is not administered to students younger than age 12, the survey asks about more severe and sensitive topics than are covered by the ECLS peer victimization items, such as drug use and possession of weapons. Also, the questions asked about similar topics as those included in the ECLS are much more extensive.

As a final note, NCES indicated that they would share the results of the pilot test of the child questionnaire to obtain BJS input on anything of concern. However, the pilot test was highly successful so there was nothing to consult with BJS about.