Supporting statement B Yes Initiative

Supporting Justification for the OMB New Request for Clearance of the

“Youth Engagement in Sports (YES) Performance Measures”

Supporting Statement B: Statistical Methods for the Collection

August, 2019

Submitted by:

US Department of Health and Human Services (HHS)

Office of the Assistant Secretary (OASH)

Office of Minority Health (OMH)

1101 Wootton Parkway, Suite 600

Rockville, MD 20852

Project Officer: Dianne Rucinski, PhD

CONTENTS

CONTENTS ii

TABLES ii

ATTACHMENTS iii

PART B INTRODUCTION 4

B1. Respondent Universe and Sampling Methods 5

B2. Procedures for Collection of Information 6

B3. Methods to Maximize Response Rates and Deal with Non-Response 8

B4. Test of Procedures or Methods to be Undertaken 9

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 9

TABLES

B1.1. Minimum Detectible Impacts for The Federal Evaluation of YES Initiative 6

ATTACHMENTS

ATTACHMENT A: QUESTION BY QUESTION SOURCE LIST FOR THE PHYSICAL ACTIVITY AND NUTRITION SURVEY

ATTACHMENT B: PERSONS CONSULTED ON INSTRUMENT DEVELOPMENT AND/OR ANALYSIS OF THE YES INITIATIVE INFORMATION COLLECTION

ATTACHMENT C: NONDISCLOSURE AGREEMENT

ATTACHMENT D: ANALYSIS PLAN

ATTACHMENT E 60 day federal register notice

INSTRUMENTS

Instrument 1: Program Performance and Implementation Measures

Instrument 2: Physical Activity and Nutrition Survey

Instrument 3: PLAYBasic

Instrument 4: PLAYInventory

PART B INTRODUCTION

Under the authority of 42 U.S.C. § 300 u-6, (Public Health Service Act § section 1707) and in support of Executive Order 13824 which intends to increase youth participation in sports and reduce barriers to play, the HHS Office of Minority Health (OMH) and the Office of Women’s Health (OWH) intend to awards grants in FY19 for the Youth Engagement in Sports: Collaboration to Improve Adolescent Physical Activity and Nutrition (YES Initiative). The YES Initiative is intended to increase opportunities for and participation in sports and physical activity among racial/ethnic minority and/or socio-economically disadvantaged male and female 6th, 7th and/or 8th grade students. Primary outcomes are to diversify youth populations participating in sports and the range of sports activities (e.g., increase the proportion of girls and ethnic/racial minorities participating in sports and the number of sports in which youth participate) and to increase amount of daily physical activity. Additional outcomes are to increase physical literacy; increase the number of days consuming fruits/vegetables, and; decrease the number of days consuming sugar-sweetened beverages.

The OMH and OWH plan to use performance and outcome measures to manage the YES initiative and to assess the relative effectiveness of the YES Initiative programs implemented by grantee organizations compared to a comparison group. The collection of performance measures from its grantees is critical to OMH and OWH because it provides the agencies with data to both effectively manage the YES Initiative, and to comply with accountability and federal performance requirements for the 1993 Government Performance and Results Act (P.L. 103-62). Collecting and reporting on data for performance measures are funding requirements for the grants, as stated in the funding opportunity announcement. Data will help grantees to make continuous quality improvements in programming. Finally, in accordance with the President’s Management Agenda CAP Goal 8, data will allow for the assessment of the effectiveness of the YES Initiative in increasing participation in sports and physical activity among racial/ethnic minority and/or socio-economically disadvantaged male and female 6th, 7th and/or 8th grade students to guide policy recommendations.

With this submission, OMH and OWH are requesting OMB approval for the reporting on a set of program participation measures and of survey and observational instruments which will be used to collect data from study participants (see Instrument 1: Program Performance and Implementation Measures). Grantee staff members will record and report several aggregate program participation measures: number of unique (de-duplicated) youth actively participating in the program (quarterly and annual); number of youth exiting the program (quarterly and annual); proportion of unique youth completing at least 85% of the intended dosage of the program; number of sessions attended by youth (median, mean and mode), and; program participation rate for population of interest.

Each youth participant for whom consent and assent are secured will be administered three instruments. The (1) Physical Activity and Nutrition Survey and (2) PLAYBasic will be administered at baseline and approximately 3 and 6 months post-baseline. The (3) PLAYInventory, a record of sporting activities, will be administered at baseline and 6 months post-baseline.

The Physical Activity and Nutrition Survey instrument was derived by OMH and OWH from the Youth Risk Behavior Surveys (YRBS). The proposed measure of physical activity is adapted from the 2019 Youth Risk Behavior Survey middle school instrument. The same question also appears in the 2019 National Youth Risk Behavior Survey high school instrument. Proposed measures of whole fresh fruit, green leafy vegetables, orange vegetables, other vegetables, sugar-sweetened beverage consumption are adapted from the 2019 National Youth Risk Behavior Survey high school instrument. No comparable nutrition questions appear on the 2019 Youth Risk Behavior Survey middle school instrument, nor have dietary behavior items appeared since 1997. Questions from the 1997 Youth Risk Behavior Survey middle school instrument ask only about the previous day (“Yesterday, how many times did you eat fruit?”). Attachment A provides a question-by-question review of items on the Physical Activity and Nutrition Survey and their source.

The PLAYBasic and PLAYInventory were developed by Dr. Dean Kriellaars, University of Manitoba, for the Sports for Life Society, and have been used in multi-cultural contexts, including with First Nation youth, to measure physical literacy and sports participation in youth. The PLAYBasic is a shorter version of PLAYFun, has demonstrated moderate internal consistency (α = 0.61-.62) and is moderately correlated (r = 0.40-0.61) with the CAMSA obstacle course, an objective assessment of of movement competency for children 8-12 years of age.¹ The PLAYBasic is a concise tool for providing a snapshot of a child’s level of physical literacy.

B1. Respondent Universe and Sampling Methods

The evaluation will be conducted in 14 communities, with approximately 350 youth per community (175 intervention, 175 control) for a total of 4,900 (2,450 intervention and 2,450 control). Community settings may include schools, park districts, YMCA/YWCA, Boys and Girls Clubs, and other NGOs. These figures and the figures below are based on conducting the evaluation at the end of the second year of the program. If OMH and OWH determine to fund an additional year of the YES Initiative, an ICR revision will be submitted for review and an updated analysis plan will be presented.

It is expected that all youth in the study will be 6^th, 7^th and 8^th grade students. Youth will be randomized to one of two conditions: (1) an intervention group participating in the YES Initiative programming or (2) a control group that is not participating in the YES Initiative yet has access to community sports and recreation resources (i.e., a business-as-usual control condition).

Eligible evaluation youth will be those 6^th, 7^th, and 8^th grade youth participating in youth programming in Yes Initiative organizations (schools, park districts, YMCA/YWCA, Boys and Girls Clubs, and other NGOs) in the 14 Yes Initiative communities. We anticipate that a total of 3,066 youth will be enrolled in the Yes Initiative intervention and 3,066 youth will be in the control condition. No youth will be denied the opportunity to participate in YES program activities nor control condition activities. Based on previous OMH experience, we expect more youth to participate in the programs that we expect to consent to participating in the evaluation. We expect to consent approximately 80 percent (79.91%) of the eligible youth into the evaluation, for a total evaluation sample size of 4900. We anticipate collecting data from 2800 youth in year 1, and 2800 youth in year 2 of whom 2100 are expected to be new to the YES Initiative in Year 2. Of the 4,900 youth consenting, 88 percent are expected to complete the baseline survey (n= 4,312). We expect 8 percent to move within the six month period at a uniform rate. We anticipate 67 percent will complete the 3-month follow-up survey (inclusive of moving, n=3,152) and 61 percent will complete the second follow-up survey (inclusive of moving, n=2,753). To promote high response rates, a small gift or monetary incentive of $5-10 is commensurate with the level of effort are allowable and encouraged. These expected response rates are in line with other federally funded physical activity projects focusing on youth, such as The WRTC program, U58-DP524470 (Steps to a Healthier Cleveland), U58DP00107 (Racial and Ethnic Approaches to Community Health), and U48DP005030 (Prevention Research Centers Program), all funded by the Centers for Disease Control and Prevention. The WRTC program was a youth fitness program implemented across a large metropolitan school district where most (85%) students are from racial/ethnic minority groups and from low-income households. The twelve to fourteen-week response rate is 88%². While not all YES Initiative grantees will be school-based, all will be site-based (e.g., YMCA/YWCA, Boys and Girls clubs). Similarly, on The Eat Well and Keep Moving Program in Baltimore school-based sites (Department of Education-PEP Grant; Walton Foundation)) the follow-up response rate was 66 percent and 61 percent³.

Although individual site variation may exist, the evaluation sample of all participating sites is expected to be equally male and female, primarily African American or Hispanic (at least 75 percent of the sample), and low-income (with more than 75 percent of the sample qualifying for free and reduced price lunch). These expectations are consistent with past OMH grant programs. For example, in FY15 data across all OMH grant programs, grantees reported over 80% African American or Hispanic and 44% male/56% female participants.

Statistical Power. Based on similar programs described above, we anticipate response rates for the YES Initiative evaluation to be 88% at the baseline, 66% at 3-month follow-up and 61% 6-month follow-up. We expect to retain all 14 communities participating in the YES Initiative evaluation, and calculations are based on retaining all 14 for the final analytic sample.

The primary impact analysis will focus on a difference-in-difference analysis between baseline and the 6-month follow-up survey data, regardless of their level of participation in the program. This will enable the team to conduct a rigorous impact analysis that meets the standards of the HHS Evidence Review. As noted above, we expect some non-response to the surveys. We expect that 88 percent of consented youth will complete the baseline survey (n=4,312), 8 percent to move within a six month period at a uniform rate, 67 percent will complete the 3 month follow-up (inclusive of moving, n=3,152), and 61 percent will complete the 6-month follow-up (inclusive of moving, n=2,753). At the 6-month follow-up when compared to the baseline, for a prevalence rate of 90 percent (such as eating a fruit or vegetable on a weekly basis) we can detect a 3 percentage point difference between the two groups; for a prevalence rate of 50 percent (such as participation of girls in sports programming), we can detect an 5percentage point difference between the two groups. Not all outcomes are binary, the PlayBASIC offers a continuous visual analogue scale (0–100) with a reported standard deviation amongst one hundred 8 to 14 year olds of 9.5 and a mean of 65. At the 6-month follow-up when compared to the baseline, we can detect a 1.04 point difference between the two groups.

Table B1.1. Minimum Detectible Impacts for Federal Evaluation of YES Initiative

	Baseline to First follow-up (67% response rate)	Baseline to Second follow-up (61% response rate)
MDES, binary	0.10	0.11
MDI (90% prevalence rate in control group)	3 percentage points	3 percentage points
MDI (50% prevalence rate in control group)	5 percentage points	5 percentage points
MDES, continuous	0.10	0.11
MDI (9.5 SD)	0.99 point difference	1.04 point difference

Note: These calculations assume a level of significance of 0.05, with a two-sided test, 80 percent power, and a 0 percent of individual-level variance in the outcome explained by covariates; Calculations include enrollment, (unit level) non-response, and moving rates. Item level non-response is not included in the calculations.

MDES = Minimum Detectible Effect Size

MDI = Minimum Detectable Impact

We also plan to conduct analyses on subgroups defined by baseline measures. These analyses will be considered exploratory, and will not be used as a primary test of the effectiveness of the intervention. Instead, they are intended to help program providers and practitioners understand whether the pattern of the findings for the full sample is similar to or different from trends observed for particular subgroups. We will observe trends for subgroups defined by (1) sex, and (2) sport participation at baseline.

We acknowledge that statistical power for these exploratory analyses may be insufficient due to smaller sample sizes within the subgroups. For that reason, the analyses are intended not as a primary test of the intervention’s effectiveness, but instead as a means to understand whether the overall pattern of findings are similar to trends observed within and across particular subgroups.

B2. Procedures for Collection of Information

In each of the schools, all youth with parental consent will be considered for follow-up data collection. OMH, OWH and their evaluation contractor, NORC, staff will work with grantee sites prior to 3 and 6 month follow-up data collection to ensure that they identify which YES Initiative participants are still enrolled and which have moved or transferred to another school.

The data collection plan for the follow-up survey is the same across all participating sites and reflects sensitivity to issues of efficiency, accuracy, and respondent burden. The Physical Activity and Nutrition survey and PLAYBasic will be administered to consented youth at baseline and approximately 3 months after completing the baseline survey and again approximately 6 months after completing the baseline survey. The PLAYInventory will be administered at baseline and at 6-month post-baseline.

As with the baseline Physical Activity and Nutrition survey, the 3 and 6-month follow-up Physical Activity and Nutrition surveys will be paper-based and administered in a group setting at each site. The Physical Activity and Nutrition survey will be completed for ten-days by youth and will be self-administered. A summary measure for each physical activity and nutrition outcome will be derived (e.g., proportion of days youth consumed fresh fruit; proportion of days a youth engaged in 60 minutes of moderate/vigorous physical activity) for each youth. Summary measures for each youth that encompass multiple days (up to 10) of data collection is proposed to increase the reliability of the baseline, 3- and 6-month measures and to give ample opportunity for youth to complete the surveys, militating against non-response bias. The Physical Activity and Nutrition survey will be administered over a ten-day period for each youth to provide a more representative indicator of a youth’s physical activity and nutritional behavior. It is acknowledged and is expected that not all youth will complete all ten days. We expect most youth to complete four or more Physical Activity and Nutrition surveys during each data collection period.

Coaches and program staff will administer PLAYBasic, which involves observing youth engaged in physical activity and recording a youth’s skill in performing four activities: locomotion, throwing, kicking, and balance. The PLAYBasic scoring rubric will be used to determine an individual score (0 to 100) for each youth at baseline, 3- and 6 month.

The baseline PLAYInventory and 6-month follow-up will be paper-based and self-administered administered in a group setting at each site.

NORC will train grantee staff on answering questions about the study, collecting student assent, and administering the follow-up Physical Activity and Nutrition survey, PLAYInventory and PLAYBasic to youth. The evaluation team will work with sites to determine the best day, time, and location for the group survey administration. Grantees will begin the administration by reviewing the details of the study and obtaining youth assent. ⁴ Any student who chooses to opt out of the survey will be led to another room with students who do not have permission to participate in the study. Youth who agree to take the survey will be provided a paper Physical Activity and Nutrition survey, PLAYInventory and prompted to begin. The survey will be self-administered. Students will be instructed to begin the survey and work through at their own pace.

The Physical Activity and Nutrition survey asks all youth to report physical activity, fruit and vegetable consumption and background information. The PLAYInventory asks youth to report the number of distinct sporting activities in which the youth has engaged in the past 12 months. In addition to 57 distinct sporting activities, youth may also record additional sport activity not listed on the PLAYInventory. The PLAYBasic is used to assess physical literacy and is completed by a coach or trained grantee staff member based on observing youth perform a set of physical activities.

A unique identifier will be assigned to youth completing the baseline and matched to youth for 3- and 6-month Physical Activity and Nutrition survey, PLAYInventory and PLAYBasic data collection. The same unique identifier with a suffix will be used for each data collection form for each child (e.g., 09056-A (Physical Activity and Nutrition survey), 09056-B (PLAYInventory) and 09056-C (PLAYBasic). No personally identifying information will appear with the survey. A question-by-question list of sources for the follow-up survey is in Attachment A, and a description of the sources referenced is in Attachment B. Once they have completed the survey, youth will place the completed Physical Activity and Nutrition survey form in a secure box attended to by program staff. When the survey administration is complete, NORC staff will work with site staff to arrange make-up administrations for any students who were absent during the administration.

At the end of the administration, youth will place the entire survey in the return envelope, seal it, and return it to a member of the evaluation team. Completed surveys will be shipped immediately via FedEx to NORC’s Survey Operations Center, where they will be logged and then checked for completeness. Any forms with identifying information, such as assent forms, will be shipped separately.

Students who have moved out of the area will be considered lost-to-follow-up. While the numbers may vary by community, we expect less than 8 percent to have moved out of the area. This anticipated loss is incorporated in the anticipated response rates at 3 and 6-months.

B3. Methods to Maximize Response Rates and Deal with Non-Response

OMH and OWH expects to achieve a response rate of 66 percent for the 3-month follow-up and 61 percent for the 6-month follow-up data collections. These expectations are based on response rates achieved in prior follow-up surveys with youth populations. We can expect to achieve these completion rates for the Federal Evaluation of the YES Initiative at the follow-up periods for several other reasons. First, the follow-up surveys will occur approximately 3 and 6 months after the baseline administration. This timing will ensure that contact data are quite current, which should minimize location problems. In many cases, youth will be enrolled in the same programs at follow-up that they were enrolled in at baseline, which will simplify locating efforts and improve response rates.

NORC staff will work with grantees to schedule the date and time of any follow-up data collections. As in the baseline process, to help attain high response rates, NORC and grantee staff will collaborate with each site to arrange additional make-up sessions for youth who are absent on the data collection administration days.

When youth complete the baseline, 3-month and 6-month Physical Activity and Nutrition survey and PLAYBasic and PLAYInventory at baseline and at 6 month, they will receive a small gift or monetary incentive ($5-10: baseline, 3-month follow-up and 6 month follow-up data collections). Grantees will encouraged to employ incentives and will have discretion to tailor the exact amount per administration for youth their communities. Research suggests that providing an incentive for earlier surveys may contribute to higher response rates for subsequent surveys.⁵ Therefore, providing a small gift during the 3-month follow-up may help boost response rates during subsequent rounds of data collection. The proposed incentive structure is similar to those used on other federally funded studies with similar populations, including The WRTC program, U58-DP524470 (Steps to a Healthier Cleveland), U58DP00107 (Racial and Ethnic Approaches to Community Health), and U48DP005030 (Prevention Research Centers Program), all funded by the Centers for Disease Control and Prevention, where the twelve week response rate was 67.

In addition, we expect that the site’s continued willing assistance will be very important to maximizing the response rates; we will therefore invest significant effort in maintaining positive relationships to minimize burden on the sites and assure privacy to the youth participants. By applying identical methods for maximizing the response rates of the treatment and control groups, the evaluation team does not anticipate differences in response rates across research groups.

The evaluation team anticipates high response rates to follow-up surveys. Even so, the team will take steps to understand the nature of any non-response and to account for the threat it may pose to the validity of the study’s impact estimates. Using data from the baseline survey, evaluation team members will first test for statistically significant differences across demographic and baseline outcome variables between the treatment and control group members who are follow-up respondents and control for these differences using covariates when estimating program impacts (see Attachment E).

B4. Test of Procedures or Methods to be Undertaken

OMH and OWH have made it a priority to align measures in the YES Initiative up survey across evaluations of similar programs and populations. Many of the items on the Physical Activity and Nutrition Survey are identical to the items in the already-approved Youth Behavioral Risk Survey (OMB Control Number 0920-0493). The concept of “physical literacy” is relatively new in the literature, but a recent study found inter-rater reliability of the PLAYbasic was good-to-excellent for average measures and moderate-to-good for single measures.⁶

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Follow-up survey data for the impact study will be collected and analyzed by OMH and OWH’s prime contracting organization, NORC. See Attachment B for a list of individuals OMH and OWH consulted on follow-up instrument development, and on the data collection and analysis plan.

1Stearns, J. A., Wohlers, B., McHugh, T. L. F., Kuzik, N., & Spence, J. C. (2019). Reliability and Validity of the PLAY fun Tool with Children and Youth in Northern Canada. Measurement in Physical Education and Exercise Science, 23(1), 47-57. DOI: 10.1080/1091367X.2018.1500368

2 Borawski EA, Jones SD, Yoder LD, Taylor T, Clint BA, Goodwin M, et al. We Run This City: Impact of a Community–School Fitness Program on Obesity, Health, and Fitness. Prev Chronic Dis 2018;15:160471. DOI: https://doi.org/10.5888/pcd15.160471.

3 Gortmaker SL, Cheung LWY, Peterson KE, et al. Impact of a School-Based Interdisciplinary Intervention on Diet and Physical Activity Among Urban Primary School Children: Eat Well and Keep Moving. Arch Pediatr Adolesc Med. 1999;153(9):975–983. doi:10.1001/archpedi.153.9.975

4 Youth assent is obtained prior to each round of data collection (baseline, 3-month follow-up and 6-month follow-up).

5 Singer, E., & Ye, C. (2013). The use and effects of incentives in surveys. The ANNALS of the American Academy of Political and Social Science, 645(1), 112-141.

6 Stearns, et al, 2019.

iv