HIGH SCHOOL LONGITUDINAL STUDY OF 2009-TECHNICAL REVIEW PANEL MEETING
November 28-29, 2007
Appendix I
High School Longitudinal Study of 2009 (HSLS:09)
Technical Review Panel Meeting
January 30-31, 2008
Meeting Summary
Submitted by:
RTI International
PO Box 12194
Research Triangle Park, NC 27709
Prepared February 22, 2008
High School Longitudinal
Study of 2009 (HSLS: 09)
Technical Review Panel Meeting Summary
January 30–31, 2008
Clifford Adelman
Sharon Anderson
Eric Banilower
Kathy Borman
Robert Bozick
Jack Buckley
Laura Burns
Daryl Chubin
Jeremy Finn
Kristin Flanagan
Mary Frase
Tate Gould
James Griffith
Debbie Herget
Rebecca Herman Thomas Hoffer
Lisa Hudson
Tracy Hunt-White
Steven Ingels
Vinetta Jones
Steve Leinwand
Laura LoGerfo
Patricia Martin
Rochelle Martinez Edith McArthur
Jeffrey Owings
Gary Phillips
Daniel Pratt
John Riccobono
Donald Rock
James Rosenbaum
(via
teleconference)
Michael Ross
Russ Rumberger
Leslie Scott
Marilyn Seastrom
Sharon Senk
Timothy Urdan
Andrew White
John Wirt
January 30, 2008
Welcome From NCES
Laura LoGerfo
The purpose of the meeting is to review the revised HSLS:09 instruments discussed during the last meeting.
Items for discussion include the following:
identify HSLS:09 priorities;
how much do we want to ask the students, principals and parents;
which items provide the richest data;
what is missing from the existing instruments;
do the questions address what we want ; and
which questions should be removed from the instruments.
RTI will send out the revised instruments to the panel.
Gary Phillips
The operational test features and design elements:
include two stages at each grade level and will vary in difficulty from low to high;
each student will have 40 minutes to complete 40 items;
include linking items between grades 9 and 11; and
include 84 items in grade 9 and 76 items in grade 11 (138 unique items with 22 of them linking items).
The field test elements:
A pool of 266 items has been assembled (2 more than needed for field-testing).
fall 12th graders will be surrogates for the spring 11th graders of the main study;
four forms will be field tested at grades 9 and 12 (total = 8 forms);
1100 students are needed at grades 9 and 12 to take the test (total = 2200 students);
race and sex for students will be collected;
each student will have 40 minutes for the 40 items on the given form;
items will be ordered according to difficulty; and
timing information will be saved for each item.
Steve Leinwand
Since the first TRP meeting (11/28/07), the following general specifications have been revised:
moved from 30 to 40 items per student;
moved from field test pool of 172 items to 266 items;
moved from operational test pool of 94 items to 138 items; and
moved from 30 items in 30 minutes to 40 items in 40 minutes.
264 items will be field tested to get 138 items for the operational test - 9th and 12th graders will be field tested in the fall of 2008; for the main study, testing for 9th graders will take place in fall 2009 and 11th graders in the spring of 2012.
A math advisory panel meeting was held on 12/14/07; the panel reviewed 240 items and rejected 12, made corrections and established consistency of wording, checked answer keys and distractors, and checked distractor rationales.
After the panel meeting, all math items were reviewed by an outside expert (John Dossey), 6 additional items were written and 32 additional NAEP items were selected.
The item pool includes 266 items (two more than needed – but these are still in item pool):
32 NAEP released items;
234 “new items”;
106 developed by John Dossey; and
128 developed by AIR.
4 distractors will be used rather than 5; this will save time on the test.
Instructions for the test have been drafted but they are still being reviewed.
Students will be allowed to skip questions. They will also be able to go back to questions.
A cognitive lab will be conducted in NC and DC to test the computerized delivery of the items and the instructions and to test the set-up on the computer
If accommodations cannot be made, students will be documented as test ineligible.
Test exclusions will be described in detail for schools and schools will determine which students need accommodations or must be excused from testing.
The math test will focus on algebraic content and algebraic processes. There will be a balance in focus between skills and problem solving.
The approximate distribution of item complexity is as follows: 37% are low, 54% are moderate and 9% are high.
The actual difficulty (as contrasted to complexity) of the items will be determined in the field test
The existing items show:
at the 9th grade level, the calculator is not helpful for 77 items and helpful but not essential for 10 items;
at the 9th-11th grade level, the calculator is not helpful for 102 items and helpful but not essential for 6 items;
at the 11th grade level the calculator is not helpful for 67 items and helpful but not essential for 4 items.
AIR recommends that all students, at both grades, have access to a scientific (but not a graphing) calculator during the test - either one that they bring or one that is provided on-line as part of the test.
Don Rock recommended starting off with the easiest items.
Instructions about when to leave a question blank and when to guess need to be carefully written in order to maximize the number of items completed.
The following concerns were expressed about the use of a graphing calculator:
will the calculator impact the amount of time allotted for the test?;
will the calculator create an expectation on how/if students will use the calculator?;
the test should include a direction along the lines of “a calculator may help you to answer these questions, feel free to use one if you wish”.
Language in the instructions needs to reflect that students are not required to use the calculator.
Steve Leinwand
During the last TRP, the panel recommended that a science assessment not be included at the 9th grade level.
The 11th grade field test will take place in the spring of 2011 and the operational test will be in 2012.
The science assessment, if pursued for the 11th grade follow-up, may draw items from NAEP, NELS, and PISA.
Panelists are encouraged to offer suggestions on how to shape the content (i.e., scientific literacy, the nature of science etc.) for the science assessment.
One question is whether science literacy questions should be posed in lieu of the 9th grade science assessment.
Additional conversation on the science assessment will resume at a later TRP meeting.
TRP comments on a possible science assessment at 11th grade
9th grade – not clear what high school science is or applied science is at this juncture
How best to shape science content? Inquiry? Learning progressions? Utilize PISA items or strive for something new to capture scientific literacy?
In 11th grade, you’d need to keep the math assessment at 40 minutes, because that’s the timing for math in 9th grade, and won’t want to skimp on the student questionnaire. Then where does the time for a science assessment get carved out?
If plan to assess in science (and certainly NCLB will soon require testing in science), measuring scientific literacy may be the right approach – could consider following up with postsecondary assessment
Surveys give window into scientific literacy from different perspective… a back-door approach to science
NSF has documented scientific literacy for decades… adult scientific literacy not considered predictor/precursor to much besides perspective
Science courses and grades, as will be captured in the HSLS high school transcript component after the first follow-up, are far more richly informative than a science test that amounts to a literacy test
Math test foundational but science test more doubtful for utility and predictive abilities
Daniel Pratt and Debbie Herget
RTI developed a school based solution for administering the test at schools. They will plan to use school computers (or will bring laptops if the school computers are unsuitable or unavailable)
Using school computers cuts down on the financial burden of paying for all of the equipment and reduces the burden of having the test administrators carry computer equipment to each school.
The session administrator will bring 5 backup laptops.
RTI recommends using a bootable CD to administer the test. The CD will load the operating system and internet browser into memory on school computers so that students can take the tests directly from the internet. The survey site will be hosted by NCES and data will be entered and stored on the secure site.
In order to use the bootable CD the computer must be a PC or Mac, have a high speed internet connection, a dynamic IP address, and a bootable CD-Rom drive.
RTI cited the following benefits for using a bootable CD:
eliminates concerns about viruses;
ensures consistency of operating systems;
data loss will be minimized;
privacy will be protected;
school equipment will not be compromised; and
students will be able to access the test and the survey on the computer.
RTI will work with the schools in advance to gain access to school computers.
NAEP computerization experiments are being carefully reviewed by NCES and RTI to anticipate and be prepared for potential problems associated with computerized assessment.
According to an NCES report (Internet Access in U.S. Public Schools and Classrooms: 1994-2005), nearly 100 percent of U.S. public schools had access to the Internet in fall 2005 and 94 percent of public school instructional rooms had Internet access. The same report indicated that in 2005, 97 percent of public schools with Internet access used broadband connections to access the Internet. The most recent comparable NCES study in private schools and classrooms was in 1998 so no presumptions can be made with regard to private schools.
There was concern about RTI relying on school computers and their availability. The following comments were made:
to reduce the burden on the test administrators, get a computer cart that can be used to carry all computer equipment to the schools;
compare costs between supplying all computers and adding personnel to carry the computers to the site and administer the sessions.
some schools may not have bootable CD drives but they may have non-bootable CD drives. Might we be able to load the operating system and browser on a floppy disk to boot up the computer and then use the non-bootable CD after that?
screen display differences may be an issue. It is important for the screen display not to compromise the assessment results. All test takers should have the same basic stimulus in presentation of the assessment.
similarly, if there are issues with bandwidth or connectivity, the assessments could be compromised. Such issues must be thoroughly investigated ahead of time -- each school will have different issues.
while high speed Internet connectivity is available at most schools, will working computers with high speed Internet access be made available for assessments?
It may be worthwhile to explore alternatives to a web-based design such as the use of software installed temporarily on school computers with the data saved temporarily on an external device (e.g., a memory stick) which could then be transmitted at the end of the session.
Steven Ingels
Updates since the last TRP meeting:
The new time allocation is 35 minutes: 30 minutes of substantive questions and 5 minutes for future locating questions.
The student portion will take 90 minutes (15 minutes for set-up and closure; 40 minutes for math assessments; and 35 minutes for the student questionnaire).
The parent questionnaire will be available in English and Spanish. A self-administered paper and pencil questionnaire option is proposed.
Under consideration:
a census of 9th grade math and science teachers.;
surveying department chairs/coordinators;
a student-driven linked-to-teacher design ; and
a census of all math-science teachers in the high school.
Consider surveying all math/science teachers and not just 9th grade math-/science teachers.
The following areas of concern were mentioned:
Teacher turnover may impact survey results.
Both school climate and culture, and math and science departmental climate and culture, are of interest to measure
Add a question for the parent about the child’s IEP.
Support was expressed for surveying math-science departmental chairs as sources of information about rules and practices for student placement and progression in the two subject areas, as informants on the school’s subject-specific culture and ethos, and as sources of information on standards and requirements shaping the delivery of math and science instruction in the school. It was also thought that department chairs could relieve burden from teachers by providing information about the math and science textbooks in use at 9th grade.
Steven Ingels
Ingels reviewed the purpose and research questions for the student questionnaire. The research questions include:
How do students decide what courses to take in high school and what to pursue after high school? What factors affect their decision-making, particularly factors that are malleable to school or parent influence?
What factors lead students towards or away from STEM?
How do students‘ attitudes and learning approaches (i.e. confidence, self efficacy, motivation, engagement, and belonging) evolve during high school?
How do students prioritize and balance various commitments, i.e. family, friends, school, job while in high school?
Jeremy Finn, Cliff Adelman, Russell Rumberger, Vinetta Jones and Daryl Chubin were asked to review the student questionnaire beforehand and provide feedback to the TRP. They were asked to pay specific attention to items that should be removed from the instrument and items that should be added to the instrument. The following insights and concerns were shared by these panelists:
Jeremy Finn’s feedback
Not enough emphasis on marginal students and students at risk
Models of decision-making processes from the November draft have been lost
Do students know what they have to do in the 9th and 10th grade to become science majors?
Figure out how connected the student feels to their schools and classmates (i.e., I feel welcomed by my school’s personnel; my friends are at school; school is the most important thing I do).
Figure out if the student values the practical things (utility) that schools provide (i.e., I get something useful out of my classes; I plan to finish school; school is a waste of time).
Figure out how active a student is in the school (i.e., extracurricular activities, participating in class learning activities; participation in school events).
Cliff Adelman’s feedback
What do students know about science/math?
It is helpful to know at the beginning what the students can do on computers..
Define what college level science means for this study.
Reorganize the items chronologically for a better flow.
Ask a question to see what else students might be interested in outside of math and science (i.e., art, history).
Find out where/how students begin to form their images concerning occupations (i.e., their parents, teachers, older siblings, television).
Find out which occupations present the most negative and positive images.
Find out who the students admire.
Ask in 11th grade in what subject they expect to be their major in college.
Russell Rumberger’s feedback
Want to know if coursetaking is related to a long-term plan, a means to an end.
Link between educational plan and occupational goal; aligned ambition
Determine the student’s perceived level of confidence in math.
Engagement should be linked to motivation, planning, and coursetaking, so that courses may be seen as a critical pathway
There is no coherence in the order of the items.
Find out if the students know what they want to be and if they know the course path they need to follow to achieve their goals.
Early adolescents generally have only vague notions about science and math careers or even subject matter
Find out if they want to go to college.
Vinetta Jones’ feedback
Need to add items that capture the experiences of those students that are underrepresented to go into STEM.
Students don’t know about pipelines. They are put into a track by the system based on race unless parents are proactive.
Add questions that ask about role models.
Add questions about involvement in after school math and science programs. Who encouraged them to participate?
Ask if students know what it takes to excel.
Ask the students to indicate how much time they think students who do well in school spend on homework.
Find out who encouraged them to go to college.
Ask student what they think their teachers and counselor expect them to be doing in 20 years.
Ask the students how they see themselves (i.e., leader, good student, bad or smart).
Ask students for reasons why they are not going to take advanced math/science.
Daryl Chubin’s feedback
Engagement is key and is a filter for other influences.
How much influence have parents, teachers, and others had?
Measure awareness of possibilities and interest.
Students do not think in terms of pipelines. Need to think about how to put these questions in their frame of reference.
Ask about their interests in high school.
Ask the students to define science.
Ask if their interests have been reinforced.
Measure of intensity of interest; have they revisited a museum or applied learning from museum to something else
Student background domain
Consider not asking about Asian subgroups if there will not be enough students in each subgroup to be analytically useful.
Remove the academic environment data element question: it is not relevant.
Previous experience domain
Consider removing previous school year grades data element, check the student transcript instead.
The bilingualism data element question should read, “how often do you speak (preloaded language) with your parents? Your friends at school? Your friends in your neighborhood?”
Laura Burns will provide a clearer item on student bilingualism from NHES:2003.
The middle school activities question should read, “have you participated in the following activities in grades 8 and 9” or “between the start of G8 and now”?
The activities question should include out-of-school activities as they relate to engagement.
Reword the science activities question, the current wording may yield inaccurate results.
Update the question stem so that it reads, “watched science movies and…”
In the self-reported 8th grade math course data element, include an option for honors courses.
Add a computer technology item to the instrument.
Social context domain
Remove the “school climate” data element.
Interpersonal influences domain
Remove versions 2 and 3 of the discuss school and work with significant others data element. Keep only version 1.
Values domain
In the occupational values data element, remove the question stem. Instead ask “what do you want to be at age 30?” and “what do you have to do to get there?”.
Jeremy Finn will send a guide that helps to identify occupational values.
Motivation domain
Intrinsic motivation items focus more on experience. The current item is listed as a value. This item should not focus on its importance but should identify if the student likes or dislikes math/science.
Remove the extrinsic motivation data element question.
Identity domain
These items are not focused. Identity questions ask “am I capable”, “do you see yourself as a math person”.
Additional questions should be added to identify ways the student believes their peers view them.
Consider asking “which do you value most”.
Remove the “future identity adult role model” data element.
Utility value domain
For the value in learning class material data element, add the option “The information is important for my career and everyday life”.
For the value in school data element, lump the multiple items into one.
Perceived opportunities and barriers domain
For the future barriers to math/science data element, add “check the two most important reasons” to the stem.
Remove the following sections: abstract attitudes toward educational opportunity data element and the concrete attitudes toward educational opportunity data element.
Costs domain
Remove the current time use data element section.
Expectancy domain
Improve the wording in the item concerning plans to take PSAT/SAT/ACT/AP/IB.
Ask “if there were no barriers, what is the highest level of education you expect to attain”.
Remove the question about plans right after high school. Instead ask what students are most likely to do after high school. Remove the option “go to college” and reorder all of the options.
After the intensity item ask “How confident are you?” – need full line to create context and thus clarity
Remove the attributions and self concept domains.
Deterrents and negative experiences domain
The question should read, “was there any class that you especially wanted to take this school year but it was not offered in your curriculum or you were discouraged from taking it?” This question should be asked to 11th graders.
Remove the question about negative experiences.
Decisions domain
Remove the future courses and influence on future courses questions.
Remove the following decision engagement questions: when I am working on a math/science assessment; when I finish a math/science assessment; do you feel bored because you do not understand what’s going on; do you feel bored because you know the answers).
Remove the time use intensity checklist. It is found elsewhere.
The panelists were asked to recommend elements that could be added to the special academic program participation question.
Math & science classroom environment domain
Remove the questions about liking the teacher and teacher approach to students
Shorten the list of options for the teacher competency and effectiveness section.
All of the items in this domain can be condensed and combined. Identify which questions stems should be paired and which question stems are repetitive.
Use a 5-point Likert instead of 7-point
Too little on peer effects
Need more on attendance patterns
Need more on how, where and why they use computers
January 31, 2008
Steven Ingels
Ingels reviewed the purpose of the parent questionnaire and the research questions for the parent questionnaire. The research questions include:
What social capital resources are available in the home environment to support children’s academic development and decision making?
What human capital resources are available in the home environment to support children’s academic development and decision making?
What financial capital resources are available in the home environment to support children’s academic development and decision making?
Three modes of administration will be available to parents: self-administration using a web interview, self-administration of a paper and pencil questionnaire, and Computer-Assisted Telephone Interview (CATI) using the web instrument.
Some concern was expressed that someone other than a parent will complete the web survey. It was noted that a password will be required to access the web survey and that no monetary incentive would provided to parents.
Cognitive pretesting will be conducted on selected new items.
Given that the material presented could not all be covered in a 30-minute interview, the TRP was asked to recommend items that could be removed from the instrument.
Kathy Borman was asked to review the questionnaire in advance of the meeting and provide feedback to the TRP.
Kathy Borman’s feedback
She thought the instrument did a good job of addressing human, social and financial capital.
She recommended asking specifically about math and science academic classes outside of school not just academic classes in general.
She recommends asking about informal math and science activities such as after school programs and summer camps.
She is concerned that some terms and language used in the instrument would not be familiar to parents.
She thinks some questions related to postsecondary plans are premature and redundant. Students may not know about specific jobs they will apply for after high school. They may not even know what they are likely to be doing as their main activity after high school.
She did not understand what was meant by a number of the subitems in the “perceived obstacles to future career plans” question. She suggested condensing the list of subitems.
She noticed some overlap between the student and parent questionnaire.
Insert a question for the student and parent questionnaire that asks if parents use math/science on the job.
Family structure domain
Panelists did not understand what was meant by “change in family situation.”
Panelists want information on divorce, whether there is a parent outside the home involved in the student’s life, and death of a parent.
It was suggested that change in family structure between the 9th grade and the 11th grade surveys can be measured by comparing household rosters and asking for reasons a parent is no longer in the household. This approach has been used in ECLS.
It was recommended that a question about the number of people in a household be added.
Demographic characteristics domain
Concern was expressed that undocumented immigrants may not want to answer questions about immigrant status.
Socioeconomic Status domain
Some panelists wanted to ask parents (and students) how much they use math and science in their job. Others were concerned that parents who use very basic math (e.g., cashiers) would say they use it a lot. These items are candidates for cognitive testing.
A panelist suggested asking college graduates from which college they received their degree.
The value of the question about assets greater than $10,000 was questioned. It is used on postsecondary studies because it is one variable used to calculate expected family contribution for financial aid. Panelists agreed that $10,000 was too low.
Previous educational experiences domain
The questions about behavior problems need a time period as a frame of reference. The past year was suggested. Also, it would be more helpful to know how many times the school contacted the parent about a behavior problem rather than whether they did or did not.
There was some debate about the merit of the question about stopping out of high school. Some panelists thought it was more important to have an estimate of the number of days absent although other panelists indicated that parents may not know if their teenager is skipping school. Others thought the stopout question was more appropriate for the 11th grade questionnaire.
The question about academic classes outside of school should refer to science and math. A distinction should also be made between remedial and enrichment.
The question about tutoring should be expanded to include Saturday academies, learning centers, and after school programs. Need information on the subjects studied in these programs and whether they were remedial or for enrichment.
Current education/activities domain
Must ask about whether the student has an IEP even if in a Gifted and Talented program
There was some debate about whether parents should be asked whether their 9th grader has a disability. Some thought the question was too subjective, but others thought the parent’s perception was important. Also, some said that students with disabilities may not have an IEP.
Some wanted to know what disability the parent believed the teenager had, while others just wanted to know whether it was a learning disability.
The wording of the question about exchanging knowledge with other parents needs to be simplified. It was suggested that HSLS ask parents how often they talk with other parents about classes, schools, and teachers.
There was some discussion about whether the question should be limited to discussions with parents of the student’s friends in keeping with Coleman’s concept of social closure. But since the friends may attend other schools than the 9th grader, a more general question was suggested.
The question about conversations with other parents should not be limited to advice. Many parents may be willing to exchange helpful information but not comfortable advising other parents. Or acknowledging receipt of advice
Throughout the domain, refer to the past year
School choice
“Career academies” should be “Career and technical programs”
Parent-school relationship domain
Parents recommended cutting the question about frequency of contact with school teachers and counselors because it will be too early in the school year to be meaningful. Also, the question does not capture why the parent is talking to the teacher.
Panelists recommended splitting the question about requesting a particular teacher or course into two questions.
Panelists thought it was more important to know if the parent know what math and science course the student is taking in 9th grade than in the next school year. If ask this question, ask them for the course name (verbatim), not just yes or no.
The panelists suggested eliminating the question about satisfaction with teachers because it will be too early in the school year for them to assess this.
Home environment domain
Consider adding a question about what subjects the student prefers
Some panelists suggested adding question about whether the parent encourages the student in some subjects more than others, but there was also concern about social desirability biases with such a question
Some panelists suggested considering adding the question from NELS about decision-making to characterize parenting style. Others did not think this was a priority given the limited length of the interview.
Panelists suggested that the question about family rules have a balance of items related to school and socializing; others thought the focus should be on school.
One panelist suggested referring back to NELS for the questions about curfew.
One panelist recommended adding the NELS question about whether there is a place set aside for the student to do homework.
Panelists considered the question about STEM-related activities important. One panelist suggested broadening the scope of this question to activities with extended family members, but others thought that “family” would be interpreted as extended family so “family” suffices.
Educational environment at home domain
Remove option ‘g’ and only use for 11th grade. Explain option g
Parent child relationship domain
Panelists recommended making the question about parent influence specific to school and career choices.
Education expectations domain
There was debate about whether educational aspirations should be measured as well as educational expectations. If a question about aspirations was posed to parents it was recommended that the same question be asked of students. Also, it was recommended that the phrase “We know that things don’t always turn out the way we would like” be replaced with “If there were no barriers.”
Panelists recommended replacing the questions about how many years of math and science they expected the 9th grader to take with the questions from the student questionnaire about expectations for taking advanced math and science courses in high school.
Occupational expectations Domain
This question only applies to students who do not anticipate continuing their education after high school. Add a “none of these” option for question about reasons for not continuing education after high school.
Panelists critiqued the items in the perceived obstacles to career question. They did not know what “lack of ability” meant; lack of academic ability or lack of opportunity? Military should not be listed as an interference with career plans because for many students it is a chosen career path. Others thought that some of the items were useful.
Steven Ingels
Ingels reviewed the purpose of the parent questionnaire and the research questions for the parent questionnaire. The research questions include:
What do mathematics and science teachers do in the classroom that engages and encourages students to pursue STEM pathways, or alternatively, disengages and discourages students from choosing STEM pathways?
How do mathematics and science teachers view the quality and supply of the school’s resources and support available?
Thomas Hoffer and Sharon Senk were asked to review the questionnaire and provide feedback to the TRP. The TRP was asked to pay specific attention to items that should be removed from the instrument.
Thomas Hoffer’s feedback
Consider a focus on professional background.
The information on textbooks should be removed.
There is too much detail on college coursework; look at major/minor specialty.
Sharon Senk feedback
Clearly identify which teachers are being asked and for what purpose.
There are 3 ways of referring to teachers: “this class”, “your classroom”, “your school”. It should be consistent throughout.
There is an inconsistency between attitudes, beliefs, and expectations; it needs to be well thought out in relation to mapping.
Delete questions 22-30 (professional development), 44 (textbook book usage), and 37, 45, and 48.
Ask if teachers feel prepared to teach math.
The section on certificates can be complicated.
Include more questions about teacher expectations.
Include more questions about math quality.
More content questions (i.e., how much emphasis do you place on skills vs. problem solving).
Include the item: “all students should take algebra” agree/disagree
Teacher education domain
Insert a question that asks “do you have a degree from a college of education?”
Insert a question that asks “do you have a degree in arts and sciences?”
Remove items k-bb on the match construct.
Teacher certification domain
The first question should ask if the teacher is certified in math or science.
Remove items 9-13.
Item number 14 should really be item number 3.
Teacher preparedness domain
Consider asking the department chair the extent to which they use each of the options listed in item 15.
Consider asking teachers how prepared they feel to teach the course content.
Professional development domain
Put these questions in the context of math and science.
Consider asking these questions to the administrator.
Remove the STEM encouragement as a student construct
Teacher attitudes/beliefs domain
Add an option ‘f’, “if a student has never done well in math they never will”.
Don’t ask for percentages. Figure out a better way to ask the question.
Ask the teachers to guess how many students will graduate from a 4 year university and a 2 year community college. Guess seems like the wrong word that might start us down the wrong path
Ask the teachers to guess how many students will major in STEM related majors.
Instructional practices domain
Remove the remediation construct. Consider adding it to the department chair questionnaire.
Spell out all acronyms.
Find out how teachers encourage those students who have displayed talent in STEM areas.
The limit on instruction construct overlaps with items 45 and 48.
Use of textbooks domain
Consider asking what percentage of the textbook the teacher covers.
Textbook questions meaningful only in context of teachers’ uniquely different classes – burdensome to ask of teachers, but as a point-in-time measure not readily connectable to achievement gain.
Instead of teachers, ask department chair to identify the textbooks used for math and science courses, if textbook information is to be obtained at all.
Remove item 40.
In item 44, change text to “in your classes”.
School climate domain
Remove item 45.
Expand the school/students construct into the beliefs and attitudes construct.
Remove item 49.
Option d in item 52 should be removed.
Remove item 52 and 53 and move the item to the department chair questionnaire.
Include a question about collegiality.
Include a question about common planning time.
Review the literature on trust and schools.
Consider referring to “school leadership” instead of “principal”
Steven Ingels
Ingels reviewed the purpose of the parent questionnaire and the research questions for the parent questionnaire. The research questions include:
What are the school-level correlates of high achieving schools, particularly in math and science?
What is the math and science focus of schools?
Is the math and science focus of schools associated with a student's subsequent decisions to pursue careers in math and science?
What programs and policies do schools offer to assist student at risk of school failure, transitioning from middle school to high school, and struggling in math and science?
The existing instrument is 95 minutes. It needs to be condensed to 30 minutes.
Remove the school size and grade span construct; that information may be found in the CCD, which in future will be more timely than in the past.
Ask the department chair (if surveyed) about student/teacher ratios.
Include a question that asks about the length of the school day and class period.
Include an item on teacher absenteeism.
Teacher staff characteristics domain
In the staffing construct, remove the “do you find that it is easier to hire qualified math teachers and science teachers if they WHO? enter alternative certification programs?”
Insert the word “district” at the top of page 4.
In the qualifications construct, remove a “successfully completed postsecondary period” from the “what are the requirements for employment as a full-time math teacher in your school” item.
In the qualifications construct, remove a “successfully completed postsecondary period” from the “what are the requirements for employment as a full-time science teacher in your school” item.
Remove the first two items under the retention/turnover construct. In remaining items, change “this year” to “last year”. This survey had no page # so refresh us – what’s this about?
In the last question in the retention construct, insert the option ‘left teaching’ or ‘retired’.
School, policies, practices and programs domain
Remove the flexibility of course assignment practices construct; the counselor is asked that question.
For the last question in the accountability construct, remove the phrase “when a student fails a competency test”.
Remove the first item in the extracurricular activities offered construct. Add the following options to the second item: “career exploration and internship programs” and “tutoring opportunities”.
For the dropout prevention program, include a question that asks how many schools are transferred out into alternative programs.
Include a question that asks how schools support struggling students.
Include a question that asks how schools support students who excel.
In the next to last item under transition construct, remove the “full or part time” from the item.
Remove the last item in the transition construct.
Remove the parent and community outreach construct.
Technology domain
Remove the technology resources/availability construct
School governance domain
Remove the mission statement construct.
Remove the autonomy construct.
Remove the evaluation of performance construct.
For the crime and safety construct, the question should read, “How would you describe the crime level in the neighborhood in which the school resides”.
The principal perceptions/beliefs construct can be used if additional time is available at the end of the test; otherwise remove it.
Steven Ingels
Ingels reviewed the purpose of the parent questionnaire and the research questions for the parent questionnaire. The research questions include:
How do students get placed into and out of classes?
What counseling resources are available to the students within school?
What are the tracking procedures and policies and graduation requirements?
What college preparation programs are in place at the school?
Patricia Martin and James Rosenbaum were asked to review the questionnaire beforehand and provide feedback to the TRP. The TRP was asked to pay specific attention to items that should be removed from the instrument.
Pat Martin feedback
Include more questions about beliefs and behaviors, in particular about math and science.
Ask how long they have been a counselor.
Ask if they have any teaching experience.
Ask if they have a math or science background.
Include questions about academic plans.
Ask if the academic plan is used in preparing course schedules.
Check for placement and tracking procedures.
Tracking begins before a student gets to high school. Counselors take information from former teachers such as eighth grade instructors.
Every school is different; find out about the formal and informal process.
Keep in mind counselors will know very little about the 9th graders at the time of the survey. Head counselors will know even less than the regular counselors.
Find out how students are assigned to the counselor.
Ask parents, students, and teachers about the perception of students being “counseled out”.
Ask counselors to describe how students are placed in classes.
James Rosenbaum feedback
Some questions can be answered beforehand without asking the counselors.
Define the purpose for the survey. Is the survey’s purpose to support information received from the other questionnaires or identify barriers or support in having success in the STEM pipeline.
There are not enough questions about beliefs and behaviors.
What is the allotted time for each question?
Include questions about decisions to take technical education courses.
| File Type | application/msword |
| File Title | Meeting Attendees: |
| Author | AlisonM |
| Last Modified By | kathy.axt |
| File Modified | 2008-05-13 |
| File Created | 2008-05-13 |