Supporting-Statement PartA_FINAL

SUPPORTING STATEMENT – PART A

The Impact of ChalleNGe on Participants’ Noncognitive Skills

A. JUSTIFICATION

1. Need for Information Collection

The National Guard Youth Challenge Program (ChalleNGe) is a unique residential program for “at-risk” youth ages 16 to 18 who have dropped out of high school. Enrollment in the program is voluntary. Participants must be high school dropouts or expellees, unemployed, and drug free. Those on probation or parole, as well as those awaiting sentencing or indictment, are not eligible for enrollment. The program is open to both men and women, but roughly 80 percent of the participants are men. There are no income-based requirements for eligibility for the program. However, participants must be residents of the state in which the program they attend is located.

The ChalleNGe program is residential and 22 weeks in length. The environment is perhaps best described as “quasi-military;” participants (referred to as “cadets”) form platoons, drill and march, and engage in intensive physical training. The program also includes classroom instruction on both academic topics and noncognitive “life skills,” such as financial management, drug avoidance, and health and sexual education. Cadets also work with counselors and other staff members to form a long-range life plan; programs emphasize the importance of planning, perseverance, and investing in the future through their focus on this plan. The academic focus of the program is designed to help cadets attain GED (General Educational Development) credentials. Participants also perform volunteer work in the communities where the programs are located. The ChalleNGe programs include an important mentoring aspect that is designed to last beyond the end of the residential phase of the program. Cadets are matched with volunteer mentors who assist them with meeting their post-ChalleNGe goals.

The ChalleNGe program was first authorized by Congress in FY93 as a pilot program. It was authorized as a permanent program in the FY98 Defense Authorization bill. The program is operated jointly by the states and the state National Guard units, with federal funding to cover a portion of the program’s costs. In each state where the program is operating, the Governor and the National Guard Bureau enter into a cooperative agreement charging the State Adjutant General with administering the program in that state.

The ChalleNGe program has grown over time. In 1993, 10 states established ChalleNGe programs; today, there are 34 programs in 29 states (plus Puerto Rico and the District of Columbia). While most programs assist students with preparing to successfully obtain a GED, several of the programs award high school diplomas or alternate credentials to some or all graduates, either through agreements with a local high school or through designation as a high school of some sort. Overall, the ChalleNGe programs have graduated over 100,000 youth. For the 2010 program year, over 50 percent of graduates received either a GED or high school diploma.

The purpose of this study is to look at the impact of the ChalleNGe program on cadets’ noncognitive skills (those skills that are not academic in nature). There is no control group. Rather, the study involves administering a single paper-and-pencil survey at two different points in time to ChalleNGe cadets at seven sites: (1) at the beginning of their time at ChalleNGe and (2) at the time of graduation when the same survey is re-administered to cadets who remained in the program.

The powerful role of cognitive function in determining educational and economic success is well established and widely recognized, but recognition of the effects of other noncognitive factors, such as motivation and perseverance, is on the rise.¹ Just as academic abilities can be changed over time, noncognitive skills have also been shown to be malleable.² Because of this, intervention programs, including the ChalleNGe program, generally place a great deal of emphasis on developing noncognitive skills. In ChalleNGe, academic excellence is one of the eight core components of the program, but most of the others focus partly or completely on outcomes requiring noncognitive skills (other components include citizenship, life-coping skills, service to community, and leadership). In addition to the core components, the program’s focus on discipline and on mentorship provides examples of the importance placed on noncognitive skills. Previous research on the ChalleNGe program has highlighted the need for research on the development of noncognitive skills among ChalleNGe participants.³

While program staff clearly track cadet outcomes related to noncognitive skills (such as leadership and attitude), there is no single expression of cadets’ noncognitive skills. Indeed, no standard measure of noncognitive skills exists. This is an issue for other youth programs as well; for example, Job Corps places a substantial emphasis on developing noncognitive skills but measures only job- and placement-related outcomes. While this survey will not provide a single all-encompassing measure of noncognitive skills, it is likely to provide insight into how the ChalleNGe program actually works and how cadets’ attitudes could be expected to change over time. In addition, cadets’ initial attitudes may be predictive of performance in the program. Finally, academic achievement is tied to teens’ changes in locus of control over time;⁴ therefore, it is possible that survey responses will differ between cadets who earn different types of credentials.

Beginning in FY11, the Office of Management and Budget (OMB) launched an initiative to encourage rigorous evaluation of government programs with the specific goals of strengthening programs and thus increasing the effectiveness of taxpayer dollars spent on such programs. This survey was proposed as part of this voluntary initiative to evaluate the ChalleNGe Program. OMB approved funding in the amount of $100,000 for this initiative. The funding was awarded to the DASD, Reserve Affairs (Resources) and was used to fund part of the study examining several aspects of the ChalleNGe program.

2. Use of the Information

The data collected will be used by the Office of the Assistant Secretary of Defense (Reserve Affairs) to evaluate the impact of the ChalleNGe program on participants’ noncognitive skills. The study will also focus on whether there are program-specific differences in how much of an impact is seen. In addition to comparisons of initial noncognitive skills based on gender and program, research questions include the following: (1) Do credit recovery programs versus GED programs see different changes in participants’ noncognitive skills? (2) Is the reason for a cadet’s departure from high school associated with the change in noncognitive attitudes during ChalleNGe? (3) How is the cadet’s gender related to changes in noncognitive skills? The data collected will only be used for the purposes of this study. The outcomes of the study will be used to advise the ChalleNGe sponsors and program directors on ways to improve the program. In particular, based on the results of this study, ChalleNGe program sponsors and directors may decide to place additional emphasis on tracking and measuring cadets’ noncognitive skills over the course of the program.

3. Use of Information Technology

In collecting the information associated with this Information Collection Requirement (ICR), DoD will be utilizing a paper-and-pencil survey. To ensure uniformity in the format of the data that are received, and thus increase the ease of database entry, a standardized reporting form (i.e., survey) has been developed. This method is being used, vice an electronic data collection, because the ChalleNGe sites are not equipped to have groups of cadets take an online survey. The programs do not have enough computers or, in some cases, reliable internet access. In addition, taking a paper-and-pencil survey will allow for multiple cadets to take the survey at the same time, thereby reducing the burden on the program. Responses on the paper surveys will be entered into a database using personal computers and applicable database software; accuracy and completeness of the data entry will be ensured by double-checking.

4. Non-duplication

The ChalleNGe program is run jointly by the states and the state National Guard units. The program is federally funded, and oversight is provided by the Office of the Assistant Secretary of Defense for Reserve Affairs, which does not have any forms or information collections that duplicate this information collection. After consultation with the various ChalleNGe program locations, it was determined that the information sought is not currently available at the program locations either. There is no other way to collect this information.

5. Burden on Small Business

Collection of this information does not have a significant impact on small businesses.

6. Less Frequent Collection

This information collection has been developed in support of a one-time research effort. The study methodology calls for the survey to be given twice to each respondent—once at the start of his or her participation in the ChalleNGe program and once at the completion. It is necessary to have each respondent complete the survey twice to determine how participation in the program affects the respondents’ survey responses. This will enable researchers to study changes in the participants’ noncognitive skills over time.

The proposed survey is made up, in part, of existing instruments: the locus-of-control scale and the Grit scale. Existing research has established that the locus-of-control scale changes during the teen years.⁵ While there is no research looking at how the Grit scale changes over time, the scale is strongly associated with outcomes that have noncognitive aspects.⁶ Because the ChalleNGe program places substantial emphasis on noncognitive skills, especially on developing and refining a detailed future plan for each cadet, we expect that the characteristics captured by our survey will predict cadets’ success, and that many cadets will exhibit a more internal locus of control (suggesting a stronger belief in the relationship between one’s actions and eventual outcomes) by the final survey. We have designed our data collection effort to be large enough to reveal measurable, statistically significant differences on survey responses.

7. Paperwork Reduction Act Guidelines

There are no special circumstances that require this collection to be conducted in a manner inconsistent with the guidelines in 5 CFR 1320.5(d)(2).

8. Consultation and Public Comments

An initial 60-day Federal Reserve Notice (FRN) was published in the Federal Register on June 5, 2012, soliciting comments on the information collection prior to submission to OMB (77 FR 33201). No comments were received.

Due to the time needed to prepare the submission to OMB, a second 60-day FRN was published in the Federal Register on January 30, 2014, soliciting comments on the information collection (79 FR 4885-4886). No comments were received.

A 30-day FRN was published in the Federal Register on April 29, 2014, soliciting comments on the information collection prior to submission to OMB (79 FR 23598).

9. Gifts or Payment

No payment or gift will be provided to the respondents.

10. Confidentiality

Participation in the data collection will be voluntary for respondents. No personally identifying information (PII) will be collected. Rather, to link the pre- and post-surveys for each cadet, each cadet will be given a “ticket” before completing the survey that contains a unique random survey identifier code and a blank for the cadet to fill in their name. The ChalleNGe program staff will use these tickets to create a crosswalk of survey identifier codes and cadet names. Research staff will have no access to the crosswalk of survey identifier codes and cadet names and ChalleNGe program staff will have no access to completed surveys. All of the paper surveys will be sealed in an envelope and hand-carried until they can be secured in a safe. After the data have been entered into a database and checked, all paper surveys will be destroyed. In addition, ChalleNGe program staff will be asked to destroy the tickets and crosswalks after the surveys have been completed.

To protect the data collected, it will reside on CNA’s Scientific Computing Operations (SCO) system. SCO is a separate computing environment that stores and processes all of CNA’s sensitive data. SCO allows CNA staff members secure access to statistical analysis and mathematical applications in addition to other job-related information resources for studies requiring the use of sensitive data. SCO users can access SCO systems only from thin clients in their offices, in commuter offices, and remotely using a SCO-supplied thin client laptop. Access to all SCO computing resources is permitted on an as-needed, need-to-know basis only. 

11. Sensitive Questions

The proposed survey does not include any sensitive questions.

12. Respondent Burden and its Labor Costs

0. a. Estimation of Respondent Burden

The survey was tested twice. First, it was tested on five cadets at the Fort Gordon ChalleNGe site. It took the cadets an average of 8 minutes and 38 seconds to complete the survey. Second, it was tested on four cadets at the Washington State ChalleNGe site. The cadets took an average of 7 minutes to complete the survey. Based on this, we expect each survey to take approximately 10 minutes to complete. We expect 1,200 cadets to take the initial survey, which equates to 200 hours spent, in total, on the initial survey. Of those cadets completing the initial survey, we expect 800 to complete the survey again at the end of their participation in ChalleNGe. This equates to 133.3 hours spent, for a total of 334 hours for the ICR. We are including a relatively large number of cadets so that our sample will be of sufficient size to compare outcomes in programs that award different credentials, and to compare outcomes between male and female cadets.

b. Labor Cost of Respondent Burden

Respondents (the cadets) are all high school dropouts. They are all unemployed, but if they were employed it is likely they would earn the federal minimum wage ($7.25/hr). At this rate, we estimate the labor cost of respondent burden to be $2,422 (334 hours x 7.25/hr).

13. Respondent Costs Other Than Burden Hour Costs

Respondents will incur no other costs.

14. Cost to the Federal Government

This survey is part of a larger study, funded by the Office of the Secretary of Defense for Reserve Affairs (Resources). As discussed above, some of the funds to develop the survey and collect the data were provided by OMB as part of an initiative to encourage more rigorous program evaluation. Other than the funding for the study ($100,000 from OMB and an additional $50,000 from DASD, Reserve Affairs, totaling to $150,000), there will be no additional cost to the federal government associated with this survey.

15. Reason for Change in Burden

This is a new collection.

16. Publication of Results

We plan to publish the results of the survey as a CNA report. We will publish only averages; we will publish no individual results and will publish results only when the cell size is too large to be linked to an individual. Based on our sampling framework and expected response rates, we expect that even our smallest cells will have more than 25 observations. We plan to collect data throughout 2014/2015 (program start dates are staggered). We anticipate publishing a CNA report detailing our results in late 2015 or early 2016.

17. Nondisplay of OMB Expiration Date

Approval to omit display of the expiration date is not being sought.

18. Exceptions to “Certification for Paperwork Reduction Submissions”

No exceptions to the certification statement are being sought.

1 James Heckman, Jora Stixrud, and Sergio Urzua, "The Effects of Cognitive and Noncognitive Abilities on the Labor Market," Journal of Labor Economics 24, no.3, 2006:411-482 (discusses both cognitive and noncognitive factors with an emphasis on labor market outcomes); Margo Coleman and Thomas DeLeire, "An Economic Model of Locus of Control and the Human Capital Investment Decision," Journal of Human Resources 38, no. 3, Summer 2003:701-721 (discusses the relationship between noncognitive factors and classroom/educational success).

2 See, for example, Blackwell, L., Trzesniewski, K., & Dweck, C.S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and an intervention. Child Development, 78, 246–263.

3 J. Wenger, C. McHugh, S. Sayala, and R. Shuford, Variation in Participants and Policies Across ChalleNGe Programs, CNA Research Memorandum D0017743.A2, Apr. 2008.

4For example, see Thomas Dee and Martin West, “The Noncognitive Returns to Class Size,” National Bureau of Economic Research Working Paper 13994, Apr. 2008.

5 Margo Coleman and Thomas DeLeire, "An Economic Model of Locus of Control and the Human Capital Investment Decision," The Journal of Human Resources 38, no. 3, Summer 2003: 701-721.

6 For example, see Angela L. Duckworth, Christopher Peterson, Michael D. Matthews, and Dennis R. Kelly, "Grit: Perseverance and Passion for Long-Term Goals," Journal of Personality and Social Psychology 92, no. 6, 2007:1087-1101.

5