1875-0258 Carol_White_PEP_Supporting_Statement_Part_A_12112012

American Institutes for Research®

Evaluation of the Carol M. White Physical Education Program -
Supporting Statement for Paperwork Reduction Act -
Part A: Justification

August 29, 2012

Revised December 11, 2012

Prepared By:

American Institutes for Research^®

1000 Thomas Jefferson St., NW

Washington, DC 20007

Prepared for:

U.S. Department of Education

Policy and Program Studies Service

Office of Planning, Evaluation, and Policy Development

Supporting Statement for Paperwork Reduction Act

A. Justification

1. Circumstances making collection of information necessary

The Carol M. White Physical Education Program (PEP) is authorized by the Elementary and Secondary Education Act of 1965, as amended, Title V, Part D, Subpart 10. In establishing PEP, Congress acknowledged the critical need to improve physical education programs for K-12 students, in order to help them make progress toward meeting state standards for physical education. Since its inception in 2001, PEP has supported a variety of projects that encourage fitness and healthy lifestyle choices among K-12 students. Prior to the 2010 application and funding year, PEP underwent substantive changes to strengthen and enhance its principal objectives so that they align more closely with current best practices and research related to improving children’s health and fitness. The primary focus of PEP continues to be the development of high-quality physical education programs and an environment supportive of physical activity. However for Fiscal Year (FY) 2010, PEP expanded in a variety of ways, including an additional focus on promoting healthy eating habits and good nutrition, requiring nutrition- and physical-activity-related policy development, establishing community partnerships for a collaborative approach to youth fitness and nutrition, and implementing new outcome measures.

Given the significant changes to the program, it is important to determine how the new PEP is implemented by individual grantees. The U.S. Department of Education (ED) is interested in gaining a thorough understanding of basic program implementation, partnerships, and data use; the evaluation is intended to address these and offer insight into how the program may be enhanced to further promote healthy living among America’s youth.

2. Purposes and uses of the data

The U.S. Department of Education (ED) is conducting an evaluation of the Carol M. White Physical Education Program (PEP). The American Institutes for Research (AIR) is under contract with ED to conduct the evaluation. The multimethod evaluation integrates both quantitative and qualitative data by including two components: an OMB-approved set of project design and implementation (PDI) surveys of project directors at year 1 and year 3 of the grant (OMB control number 1875-0258) and an illustrative case study¹ component.

The PDI surveys will systematically gather rich information from project directors about the design and implementation of the new PEP. The case study data will supplement the quantitative data collected through the surveys and serve as an exploratory tool. Researchers will conduct case study telephone interviews to gain an in-depth understanding about grantees’ program experiences not captured by the quantitative data. The interviews will be conducted with project directors and an additional 14 partner and grant-related personnel from five selected PEP project sites. Partner and grant-related personnel include school administrators (e.g., principals, vice principals), teachers (e.g., physical education [PE], health, nutrition), PE or health and wellness curricula coordinators, athletic directors, nutritionists, and food service staff. After discussion with PPSS; the Department’s Office of Safe and Healthy Students (OSHS); the Department’s Office of Planning, Evaluation and Policy Development (OPEPD); and the Office of Management and Budget (OMB); the research team agreed to focus the interviews on those elements that changed most significantly with FY 2010 grantees: the program’s two new competitive preference priorities. The two new competitive preference priorities for FY 2010 grantees were the establishment of official partnerships and the collection of body mass index (BMI) measurements.

As case studies are descriptive in nature, the interviews will not yield generalizable data but will provide information regarding the implementation of these new elements into PEP projects. The case studies will also provide valuable feedback to ED on partnerships and BMI measurement to inform future improvement of the program.

3. Use of technology to reduce burden

The Year 1 and Year 3 PDI surveys will be administered online via the World Wide Web. The use of multiple skip patterns integrated to the online survey will greatly reduce burden on respondents by only presenting them with the questions that are relevant to them. All respondents will have the option to request the survey in hard copy format. The survey website for the web-based surveys will be password protected.

AIR will collect the case study interview data by phone using detailed electronic notes. All materials and data will be securely stored electronically and clearly labeled with the date and time of the interview as well as with identifiers to indicate the specific grantee and interviewee and maintain confidentiality. With the permission of the participants, the interviews will be audio recorded to ensure accuracy.

4. Efforts to identify duplication

The evaluation of PEP represents ED’s first and only study that is aimed at compiling data on each of the new PEP projects to promote healthy living among children and adolescents. ED has not conducted any other program evaluations of PEP since the start of the program. The case study component in particular provides the opportunity to capture in greater detail a subset of grantees’ experiences with partnerships (e.g., how grantees work with partners and what roles partners play) and BMI data collection (e.g., how grantees inform parents about BMI and how BMI data are being used). The case study questions incorporate and expand on a subset of the statement of work (SOW) evaluation questions to garner illustrative data related to both partnerships and BMI data collection and use. Thus, no duplication of the surveys will result from this component of the study. Data from the surveys and the interviews will be triangulated to capture more detailed information about PEP projects’ partnerships and BMI data.

5. Methods to minimize burden on small entities

Twelve grantees are community-based organizations and some of these may be small entities. Related to the surveys, these grantees will experience the same level of burden as all other grantees and will be able to use technology to reduce burden. Of the five PEP grantees that will be selected for the case study, some are community-based organizations and some may be small entities. Every effort will be made to minimize that burden by establishing and maintaining clear communications with participants and remaining flexible to the changing demands of schedules and logistics.

6. Consequences of not collecting the data

PEP is a government-funded program and is one of the few federal funding sources for physical education. Given the fact that it has not been evaluated previously, ED does not know how grantees implement their PEP projects. The consequences of not collecting the data will result in the lack of information on the program’s design and implementation. Without the data collected as part of this study, ED would be unable to judge how well the program is implemented and how the program may be improved in the future. In addition, the data collection speaks directly to the Administration’s call for action to prevent and decrease childhood overweight and obesity to improve children’s health and fitness.

7. Special circumstances

No special circumstances apply to this study.

8. Adherence to 5CFR 1320.8 guidelines and consultation outside the agency

A 30-day and 60-day notice about the case study component of the evaluation of PEP were published in the Federal Register to provide the opportunity for public comment. To assist with the development of the case study component of the evaluation of PEP, the study team has drawn on the input of several outside experts in the fields of physical education, physical fitness, health, and nutrition.

The members for the Technical Work Group (TWG) are:

• Barbara Ainsworth, Exercise and Wellness Professor, Arizona State University

• Randy Kuhnau, High School Principal, Wisconsin Dells High School

• Sarah Lee, Health Scientist, Centers for Disease Control and Prevention

• Russell R. Pate, Associate Vice President for Health Sciences and Exercise Physiology Professor, University of South Carolina

• Patty Pursell, Physical Education Teacher, Highland Elementary School

AIR organized three meetings to consult with TWG members on the study design, survey instruments, and case study interview protocols. Project staff will also use the workgroup members for consultation on an as-needed basis throughout the study.

9. Payment or gifts

No payments or gifts will be used over the course of this study.

10. Assurances of confidentiality

AIR research staff will protect the privacy and confidentiality of the PDI survey respondents and case study participants. The names and any other identifying information about the participants will be kept completely confidential, and omitted when recording information. The names and any other identifying information about the interview participants will be kept completely confidential and omitted when recording information. Caution will be exercised in limiting data access to authorized project staff and those who have been instructed in the confidentiality requirements of the study. The PDI survey and case study data will contain no information that could be used to identify participants other than that which is publicly available. No individual identifying information will appear in any of our reports.

The electronically stored PDI survey and case study data will be password protected. Paper data (if any) will be stored under lock and key. In addition, all materials will be stripped of all individually identifiable information to further protect respondent confidentiality.

11. Justification of sensitive questions

No questions of a sensitive nature will be included in this study. Respondents are reporting on program-level activities only.

12. Estimates of hour burden

The total estimated hour burden for the case study is 77 hours (or 4,600 minutes), and for the entire evaluation (i.e., case study and surveys) is 231 hours (or 13,840 minutes). The hourly burden breakdown follows.

Case Study Component:

• PEP Project Director Interviews: 5 interviewees (3 district and 2 community-based organization [CBO] grantees) * (20 minutes for scheduling and participant recruitment + 60 minutes per interview) * 100% response rate = 400 minutes.

• PEP Project and Partner Personnel Interviews: 5 grantees (3 district and 2 CBO) * 14 interviewees * (20 minutes for scheduling and participant recruitment + 40 minutes per interview) * 100% response rate = 4,200 minutes;

  • 42 project and partner personnel from district grantees

  • 28 project and partner personnel from CBO grantees.

• Total burden for the case study component = 400 + 4,200 = 4,600 minutes = 77 hours.

OMB Approved Survey Component (OMB Control Number: 1875-0258)

• PEP Year 1 PDI survey (district): 65 respondents * 60 minutes per survey * 100% response rate = 3,900 minutes.

• PEP Year 3 PDI survey (district): 65 respondents * 60 minutes per survey * 100% response rate = 3,900 minutes.

• PEP Year 1 PDI survey (CBO): 12 respondents * 60 minutes per survey * 100% response rate = 720 minutes.

• PEP Year 3 PDI survey (CBO): 12 respondents * 60 minutes per survey * 100% response rate = 720 minutes.

• Total burden for the survey component = 3,900 + 3,900 + 720 + 720 = 9,240 minutes = 154 hours.

13. Estimate of cost burden to respondents

There are no additional respondent costs associated with this data collection other than the hour burden accounted for in Item 12. Interviewee professions vary, including district administrators, school professional personnel, and teachers to name a few. Based on an average of hourly rates for K—12 personnel,² a conservative (i.e., high) estimated monetary cost amounts to $9,498.91. Exhibit 1 summarizes the estimates of respondent burden for study activities.

Exhibit 1. Cost Burden to Respondents

	Task	Total Sample Size	Estimated Response Rate	Number of Respondents	Time Estimate (hours)	Total Hours	Hourly Rate	Estimated Monetary Cost of Burden
Project Directors	PEP Year 1 PDI survey	77	100%	77	1.0	77	$41.21	$3,173.17
	PEP Year 3 PDI survey	77	100%	77	1.0	77	$41.21	$3,173.17
	Case Study Interview	5	100%	5	1.3	6.5	$41.21	$267.87
	Totals					160.5		$6,614.21
Project and Partner Personnel	Case Study Interview	70	100%	70	1.0	70	$41.21	$2,884.70
	Totals					70		$2,884.70
Total						230.5		$9,498.91

14. Estimate of annual cost to the federal government

The estimated cost for the case study, including the development of the case study design and protocols, justification package, data collection, data analysis, and preparation of the Final Report (which includes the integration of the survey results), is $329,469. This work covers 21 months; therefore the annual cost to the government is $188,268.

15. Program changes or adjustments

The revision to this ICR included an additional case study protocols. Even though this increases the burden by 23 annual hours, there was a correction of hours from the previous collection which evens the burden out to no change. The decrease noted was the correction -- reporting in annual terms versus a 3 year amount.

16. Plans for tabulation and publication of results

AIR reported Year 1 survey findings to ED in a Year 1 Brief (released in August 2012). AIR will report on survey and case study findings in a Final Report. In preparing the Year 1 Brief, AIR analyzed the Year 1 program design and implementation data through descriptive analyses to understand how grantees with different design features operate during the early stage of the program. For the Final Report, AIR will perform descriptive analyses of the two waves of PDI survey data to describe features of program design and implementation and to document changes in program implementation over time. AIR will rely on content analysis methods for analyzing the case study data to describe features of partnerships and BMI data collection and use. As noted in A2, the case studies are descriptive in nature and the data from the interviews will not yield generalizable data nor be presented as such, but will provide illustrative examples related to the PEP projects’ experiences with the new competitive preference priorities. Qualitative analytic procedures for this study will establish and adhere to a set of standards to limit bias and ensure reliable findings. These include standards of evidence, triangulation of data, and procedures for measuring inter-rater agreement. The study team will develop coding guidelines based on the overarching research questions related to partnerships and BMI data, the materials received from grantees (e.g., BMI-related protocols), and key findings and themes extracted from the interviews. A qualitative software program (e.g., Atlas or Nvivo) may be used to identify trends and conduct cross-examinations of the different data sources based on the coding guidelines.

In drafting the report, AIR will address the questions presented in the Overview of Study Design section and any subsequent agreed-upon modifications to the study plan. The report will be crafted in a manner that is suitable for distribution to a broad audience, following the guidance developed by PPSS and the ED report, Guide to Publishing at the U.S. Department of Education. AIR will first create an outline of the contents. The contents will include, but are not limited to, the following:

1. Executive summary

2. Purpose of project

3. Research methodology

   1. Survey development

   2. Data collection procedures

   3. Analysis methods

4. Results from the survey of PEP Project administrators at Year 1 and Year 3

5. Results from the case study

6. Implications of study findings and future directions

The final report will include results for the full evaluation. It will also include a description of the methodology employed, findings, and implications. The report will include an executive summary in non-technical language, which will be appropriate for a wide range of audiences. Particularly for the case study component, there will be no statistical tables or significance levels; rather data will be summarized by site (solely for analytic purposes and not identified as such in any reporting), by role, and across sites. In addition to reporting general findings, quotes and descriptive stories of relevant experiences will be presented.

The final report and brief will be provided in three formats: camera-ready copy, Microsoft Word, and a copy compatible with PPSS’s Website formatting.

17. Approval to not display OMB expiration date

All data collection instruments will include the OMB expiration date.

18. Explanation of exceptions

No exceptions are requested.

1 See the following for uses of case studies in research and evaluation: U.S. Government Accountability Office, Case Study Evaluations (1990),

2 Based on average salaries reported on http://www.edreform.com/2012/04/03/k-12-facts/#salaries (retrieved June 22, 2012).

Notice of Trademark: “AMERICAN INSTITUTES FOR RESEARCH” and “AIR” are registered trademarks. All other brand, product, or company names are trademarks or registered trademarks of their respective owners.

1000 Thomas Jefferson Street, NW | Washington, DC 20007‑3835