ED Response to OMB Questions 8-7-09

Response to Questions from OMB

Fast Response Survey System (FRSS) 100-103 –

Arts Education in Public Elementary and Secondary Schools

1850-New

August 7, 2009

Question 1 from OMB

“Combined Schools”

• Do counts or proportions exist for combined schools?

• How will probabilities of selections be determined for these schools, as including combined schools in both elementary and secondary schools would double chance of selection; are combined schools considered 2 schools for purposes of making estimates (elementary and secondary)?

Response from NCES

• As indicated in Table 2 of Section B.1, there are approximately 2,200 combined schools (or about 2.5% of the 86,472 regular schools in the public elementary/secondary school frame). The combined schools account for 4% of the 55,064 schools that provide instruction in elementary grades and 6.5 % of the 33,597 schools that provide instruction in secondary grades.

• For reasons discussed in Section A.1, it is necessary to administer separate surveys to elementary and secondary schools. Since combined schools are capable of responding to either survey, they will be included in both sampling frames. However, under the proposed sampling design, a combined school will have a chance of selection from either the elementary stratum or the secondary stratum but not both. In effect, a combined school is treated as two separate schools (an elementary part and a secondary part) for both sampling and analysis purposes. Although a particular combined school will have a multiple chance of selection for the study, only the part (instructional level) of the school for which it was sampled will be surveyed and included in the analysis. In other words, the elementary part of the school will have only a single chance of selection for the elementary school surveys and, similarly, the secondary part of the school will have only a single chance of selection for the secondary school surveys.

Question 2 from OMB

Given the list of teachers that the principal is instructed to send back at the first stage of the survey, how will researchers determine which schools have non-coverage (missing teachers on the list) and which schools do not have these teachers at all? Given the stipulations for creating the list, coverage could become an issue.

Response from NCES

We expect that some schools with music and visual arts specialists may not identify these teachers on their lists. For example, some schools may send us a general list of teachers without identifying information for music and visual arts specialists. Some other schools may not even send us a list of teachers. In both cases, we will conduct followup calls to determine whether the school has music and visual arts specialists.

In addition, we will add the following check boxes on the list collection return form for principals to indicate whether or not they had a music or visual arts specialist.

If your school does NOT have a music specialist, please check this box

If your school does NOT have a visual arts specialist, please check this box

Question 3 from OMB

(a) Will the survey itself be administered during the non-response follow up? Has a survey been designed for this purpose? Given that the survey itself is more a form than a questionnaire, it seems it would need to be modified to be interviewer-administered. (b) Additionally, has a web version been designed?

Response from NCES

(a) We do not plan to administer the survey over the phone. Telephone followup will focus on prompting survey nonrespondents to mail or fax the completed survey or complete the web version of the survey.

(b) A web survey has been developed and it is currently being finalized pending survey clearance from OMB. FRSS surveys have a standardized template designed to maintain comparability between mail and web versions of the survey. In addition, during survey development, we discussed complex items with our in-house web survey designers and senior computer programming staff to be sure that all items on the paper survey could be administered on the web version of the survey.

Question 4 from OMB

Table A12

• How has the 60% availability for VA specialists in elementary schools, 79% availability of music specialists in elementary schools, etc been determined?

• For row 4 (Lists of arts specialists only collected from elementary and secondary schools…) how was the sample size calculated to be 2200? Given the samples listed on page 2 (A1), there are 1800 elementary schools and 1600 secondary schools in the sample; yielding a total of 3400 schools. Do combined schools account for the discrepancy?

• How are combined schools accounted for in this table?

Response from NCES

• As discussed on pages 2 and 9 of Section A of the supporting statement, the availability of arts specialists is estimated from a combination of the following pieces of information:

  • Information received during the pretests of the 2009-10 FRSS arts surveys. During survey development, some school principals and arts specialists indicated that schools/districts may be required to cut back on staffing for arts instruction because of the current economic climate. For example, some principals indicated that their schools will no longer have a music or visual arts specialist in fall 2009. In addition, some arts specialists indicated that they had already received notices of termination of services. This information led to the expectation that the availability of arts specialists may be somewhat lower in 2009-10 than previously due to potential staff reductions.

  • The availability of specialists in public elementary schools in the 1999-2000 FRSS arts surveys. In 1999-2000, visual arts specialists were available in 67 percent of elementary schools. Based on preliminary feedback from schools (discussed above), we took potential staff reductions into consideration and lowered our availability assumptions by about 10 percent down to 60 percent for 2009-10. Music specialists were available in 88 percent of elementary schools in 1999-2000. For the same reason mentioned above, we lowered our availability assumption by about 10 percent down to 79 percent for 2009-10.

  • The rates of visual arts and music instruction in public secondary schools in the 1999-2000 FRSS arts surveys (there were no secondary teacher specialist surveys in 1999-2000). In 1999-2000, visual arts instruction was offered in 93 percent of secondary schools, and music instruction was offered in 90 percent of secondary schools; our assumptions regarding the availability of visual arts and music specialists are also lowered by roughly 10 percent each down to 81 percent of secondary schools in 2009-10.

NCES believes that estimating the rates conservatively for the 2009-10 surveys is the best approach for ensuring that the number of completed questionnaires received from arts specialists will support the proposed analyses. In the 1999-2000 studies, the sample sizes were too small to allow for analyses of the data by type of arts specialists and school characteristics. For example, in the 1999-2000 elementary teacher surveys, completed surveys were received from 453 out of a total of 528 eligible music specialists, and 331 out of total of 386 eligible visual arts specialists. These numbers did not allow for breakouts of the data by school characteristics such as enrollment size, locale, geographic region, and poverty concentration. The proposed increase in sample sizes for each group of specialists is expected to yield adequate cell sizes to allow for this detailed breakout of the data. Table 4 in Section B1 of the supporting statement provides information about the approximate sample sizes and standard errors to be expected under the proposed design for the teacher surveys.

• The 2,200 schools for which lists of arts specialists only will be collected (row 4) plus the 1,200 schools for which lists of classroom teachers and arts specialists will be collected (row 3) yield the total of 3,400 schools. The list collection is explained in more detail on page 9 of Section A of the supporting statement. Of the 1,800 elementary schools, 1,200 will be asked to provide a list of their classroom teachers and their arts specialists. These 1,200 schools are shown in row 3. The remaining 600 elementary schools will be asked only for their arts specialists. These 600 schools are in row 4, along with all of the 1,600 secondary schools, which will be asked only for their arts specialists. Thus, row 4 includes 600 elementary schools and 1,600 secondary schools, for a total of 2,200 schools providing only lists of arts specialists.

• In this table, the combined schools are included either as an elementary school or as a secondary school depending on the stratum from which it is sampled (see Section B.1). In other words, the set of combined schools associated with elementary schools in this table is assumed to be different from the set of combined schools associated with secondary schools for the purposes of estimating response burden.

Question 5 from OMB

Given the letter to principals, where is it clear as to which grades a combined school should report for (lower or upper grades)?

Response from NCES

The letter will be attached to a specific questionnaire (i.e., elementary or secondary survey) depending on how the combined school was sampled (see response to question 1 above). Thus, a combined school that is sampled as an elementary school will receive an elementary survey, and a combined school that is sampled as a secondary school will receive a secondary survey.

We will add the following four cover letters for combined schools only (letters are attached):

• Combined schools that are sampled as elementary schools and required to complete the survey and provide a full list of eligible teachers (i.e., music, visual arts, and classroom teachers);

• Combined schools that are sampled as secondary schools and required to complete the survey and provide a list of music and visual arts teachers;

• Combined schools that are sampled as elementary schools and required to provide only a list of music and visual arts teachers; and

• Combined schools that are sampled as secondary schools and required to provide only a list of music and visual arts teachers.

For example, the cover letter to combined schools that are sampled as elementary schools and required to complete the survey and provide a full list of eligible teachers will include the following statement:

As principal of a school that has both elementary and secondary grades, we ask that you provide information only for grades that are considered elementary at your school. Please complete the enclosed school questionnaire about arts education at your school and have a list of eligible teachers prepared according to the enclosed instructions.

We will also affix a label to the list collection form for combined schools to instruct principals to provide lists for teachers in either elementary or secondary grades, depending on how the school is sampled.

Question 6 from OMB

Questionnaire - general

The questionnaire is deceivingly long. Although the ICR stresses it is only 3 pages, each question asks at least 12 or so questions. For example, question 5 on the first page is 24 questions (a-f * 4 columns). While it is “short” given the page amount, there are a substantial number of questions compressed onto each page, which can cause measurement error (intentional or non-intentional), boredom, or non-response. Could consider extending the design onto at least 1 more page, ideally 2 or 3, to provide for better respondent directions (between skips, etc) and space between questions to visually separate sections.

Response from NCES

FRSS surveys are designed to be 3 pages in length and the survey is required to take no more than 45 minutes to complete. Based on information collected during the pretests of the surveys, respondents took an average of 30 minutes to complete the survey.

While many of the questions are compressed into a grid format, most of the questions require a YES/NO response or a response that is based on the principal’s perception or overall assessment. Regarding questions that ask for data from school records, we took care during survey development to ask for information that is readily accessible to school principals. For example, we eliminated questions that require substantial checking of school records (e.g., questions about unduplicated counts of students enrolled in arts classes) and kept those questions that school principals would be able to answer with little effort (e.g., questions about staffing and duplicated enrollment counts in arts classes).

In addition, some of the questions will not be applicable to all schools. For example, most elementary schools do not offer dance or drama/theatre as separate subjects. Thus, these schools will skip the last two columns for questions 1 to 6. Several other questions also have skip patterns.

Question 7 from OMB

Questionnaire – specific – Principal Survey

• Definitions page: what is the difference between an Arts Specialist and a Certified (credentialed) Specialist? Will this resonate with respondents?

Response from NCES

The definitions below came from the 1999-2000 surveys. The definition for certified (credentialed) arts specialist came from the elementary school survey and the definition for arts specialist came from the music and visual arts teacher surveys.

Arts specialist is defined as an education professional with a teaching certificate in an arts discipline, such as visual arts, music, dance, or drama/theatre, who provides separate instruction in that discipline. Certified (credentialed) specialist is defined as an education professional with an advanced, regular, or alternative state certificate in one of the arts disciplines, or with a degree from an accredited arts education program.

We agree that the two definitions are similar because both include the condition that the teacher should be certified with a teaching certificate in an arts discipline. The second definition clarifies that the teaching certificate could be advanced, regular, or alternative.

We will drop the second definition from the elementary and secondary school surveys. Thus, the first definition will be used consistently across the school and teacher surveys. Dropping the definition from the school surveys will require that we modify question 5 in the elementary school survey and questions 5 and 8 in the secondary school survey to refer to read “specialists” instead of “certified (credentialed) specialists.” We think these changes will adequately address the problem of having two definitions that clearly overlap.

• Consider thanking respondents initially and after completing the form.

Response from NCES

We will add a note on the cover page to thank respondents initially for completing the surveys. There is no available space to add a similar note at the end of the survey. Please note that the cover letter also thanks the school principal for participating in the survey.

• Q1: Instruction on last line “skip to the next column” (?)

Response from NCES

We will drop this instruction even though it did not appear to cause any confusion for respondents during the pretests. In addition, we will simplify the instructions at the top of the grid so that we do not need any further instructions in Q1.

In the elementary survey, the simplified instructions will read: “If the answer to question 1 is “Not at all,” then skip questions 2 through 6 for that subject.” For the secondary survey, the instructions will read: “If the answer to question 1 is “No,” then skip questions 2 through 8 for that subject.” This simplified instruction reflects how respondents generally answered the question during the pretests. It also reflects the shorter instructions that were provided for questions 9 and 10 in the secondary survey

• Q2 (and others): consider defining “typical”, class period may vary by grade, etc

Response from NCES

It is difficult to provide a definition for “typical” and other major NCES surveys do not provide such a definition. For example Q25 of the 2007-08 public school questionnaire for the Schools and Staffing survey reads:

25. How long is the TYPICAL FULL WEEK of school for THIRD GRADE students?

Report BOTH hours and minutes, e.g., 30 hours and 0 minutes, 32 hours and 54 minutes, etc.

• Q3: “Other” only gives space for single response, respondent may wish to provide an open-ended response for each type of arts

Response from NCES

We expect a small percentage of respondents (2 percent or less in the 1999-2000 surveys) to report an “other” response for this question. As in the last study, we will use respondents’ written responses to upcode the question, if necessary, but we will not report the text data. We will simply report the percentage of schools that indicated “other types of class duration” for each arts subject area.

• Q1-Q6: if this format is kept, heading should be repeated.

Response from NCES

Repeating column headers will make the question look very busy and it will also use up valuable space. In relatively short grids that take up less than a page, it is not typical to repeat column headers. Column headers are generally repeated only if the question continues to a second page.

• Q7: Q1-5 uses numbers 1-5; Q6 uses Y/N; Q7 uses Yes/No (1, 2). Could make this more consistent.

Response from NCES

Questions 1-4 require only one answer, with the numbers 1 to 5 reflecting the various response options for the question. Questions 5 and 7 are structured differently. Question 5 is a mulit-part question with each part, a through f, requiring a response yes or no. Question 7 is also multi-part but it is not part of a grid series of questions. To make questions 5 and 6 similar in structure to question 7, we could change the response options to read “Yes……No” as shown below. In the secondary survey, we could also change the response option for Q1 to read “Yes……No,” and we could change the response options for Q9 to read “Yes…No….NA.”

We would like to avoid including a “YES/NO” header for Q5 because this would mean adding a header under the header for the various arts areas.

Please note that respondents did not report any problems with answering questions 1 through 7 in the way these questions were structured.

5. What is the position of the person(s) who teach(es) the subject? (Count itinerant teachers as part time.) (a) Full-time, certified (credentialed) specialist (b) Part-time, certified (credentialed) specialist (c) Classroom teacher (d) Other faculty member (Specify) (e) Artist-in-Residence (f) Volunteer	Yes No Yes No Yes No Yes No Yes No Yes No	Yes No Yes No Yes No Yes No Yes No Yes No	Yes No Yes No Yes No Yes No Yes No Yes No	Yes No Yes No Yes No Yes No Yes No Yes No
6. Does your district have a written curriculum guide in the subject that your teachers are expected to follow?	Yes No Yes No Yes No Yes No Yes No Yes No	Yes No Yes No Yes No Yes No Yes No Yes No	Yes No Yes No Yes No Yes No Yes No Yes No	Yes No Yes No Yes No Yes No Yes No Yes No

• Q7: Why does this question not have the same NOTE as Q8 “if there are no art specialists at your school … check the box & skip…” for dance and theater?

Response from NCES

We intentionally did not add a skip pattern to this question because we would like all respondents to read through question 7 so that they could be prompted to consider the various ways in which dance and drama/theatre could be taught at the school. Adding a skip box at the top of the question could lead some schools (e.g., those that do not teach dance and drama/theatre as separate subjects) to mistakenly check the box and skip question 7.

• Q9: Double-barreled; principal may observe art teachers but not evaluate or vice versa

Response from NCES

We will drop “observe and” from the question. We did not change the question from the way it was asked in 1999-2000 because we assumed that principals would generally observe teachers for the purpose of evaluation.

Q10, Q11, Q19: do these questions refer to this school year or calendar year?

Response from NCES

The questions refer to the school year. Throughout the surveys, we will change “this year” to “this school year.” We will also change “in 2008-09” to “in the 2008-09 school year.”

• Q13: Consider defining adequate

Response from NCES

The question is asking for respondents’ perceptions or assessments of what is considered adequate, based on the needs of the school and the support they receive to meet those needs. It would be difficult to provide a definition for this commonly understood term.

• Q15: Seems out of context, consider moving to end.

Response from NCES

The question is in its current location because of space considerations. It is typically placed at the end of FRSS surveys but the last page on this questionnaire does not have space.

7