Research Protocol - ChalleNGe

The Impact of ChalleNGe on

Participants’ Noncognitive Skills

Dr. Lauren Malone

Principal Investigator

CNA

4825 Mark Center Drive

Alexandria, VA 22311-1850

(703) 824-2741

Sponsor:

Office of the Assistant Secretary of Defense (Reserve Affairs)

Contact: Mr. Ernie Gonzales

1500 Defense Pentagon

Room 2E593

Washington, DC 20301-1500

(703) 693-8630

Protocol No. A30500

Date: April 22, 2014

Version: 1

Purpose of Study and Background

Purpose

The purpose of this research is to assess the degree to which participation in the National Guard Youth ChalleNGe program improves participants’ noncognitive skills (those skills that are not academic in nature).

Background

The National Guard Youth Challenge Program (ChalleNGe) is a unique residential program for “at risk” youth age 16 to 18 who have dropped out of high school. Enrollment in the program is voluntary. Participants must be high school dropouts or expellees, unemployed, and drug free. Those on probation or parole, as well as those awaiting sentencing or indictment, are not eligible for enrollment. The program is open to both men and women, but roughly 80 percent of the participants are male. There are no income-based requirements for eligibility for the program. However, participants must be residents of the state in which the program they attend is located.

The ChalleNGe program is residential and 22 weeks in length. The environment is perhaps best described as “quasi-military;” participants (referred to as “cadets”) form platoons, drill and march, and engage in intensive physical training. The program also includes classroom instruction on both academic topics and noncognitive “life skills” such as financial management, drug avoidance, and health and sexual education. Academic topics include math, language arts, science, and social studies. Cadets also work with counselors and other staff members to form a long-range life plan; programs emphasize the importance of planning, perseverance, and investing in the future through their focus on this plan. The ChalleNGe programs include an important mentoring aspect which is designed to last beyond the end of the residential phase of the program; cadets are matched with volunteer mentors who assist them with meeting their post-ChalleNGe goals.

The ChalleNGe program was first authorized by Congress in FY93 as a pilot program. It was authorized as a permanent program in the FY 98 Defense Authorization bill. The program is operated jointly by the states and the state National Guard units, with federal funding to cover a portion of the program’s costs. For each state where the program is operating, the Governor and the National Guard Bureau enter into a cooperative agreement charging the State Adjutant General with administering the program in that state.

The ChalleNGe program has grown over time. In 1993, 10 states established ChalleNGe programs; today, there are 34 programs in 28 states (plus Puerto Rico). While most programs assist students with preparing to successfully obtain a GED (General Educational Development) credential, several of the programs award high school diplomas or alternate credentials to some or all graduates, either through agreements with a local high school or through designation as a high school of some sort. Overall, the ChalleNGe programs have graduated over 100,000 youth. For the 2009 program year, over 60 percent of graduates received either a GED or high school diploma.

Programs publicize and recruit through advertising, through building relationships with a variety of people who come in contact with young people, and through word of mouth. Each program typically receives more applications than it can accommodate during each class. In addition to being turned away due to ineligibility, applicants may be denied admission due to space or funding limitations. However, programs do not deny admission based on test score requirements.

The powerful role of cognitive function in determining educational and economic success is well established and widely recognized, but the effects of other noncognitive factors, such as motivation and perseverance, are increasingly recognized as well.¹ Intervention programs, including the ChalleNGe program, generally place a great deal of emphasis on developing noncognitive skills. In ChalleNGe, academic excellence is one of the 8 core components of the program, but most of the others focus partly or completely on outcomes requiring noncognitive skills (other components include citizenship, life-coping skills, service to community, and leadership). In addition to the core components, the program's focus on discipline and on mentorship provides additional examples of the importance placed on noncognitive skills. Previous research on the ChalleNGe program has highlighted the need for research on the development of noncognitive skills among ChalleNGe participants.² While program staff clearly track cadet outcomes related to noncognitive skills (such as leadership and attitude), there is no single expression of cadets' noncognitive skills and no measure of how those skills change during the course of the program. This is an issue for other youth programs as well; for example, Job Corps places a substantial emphasis on developing noncognitive skills but measures only job- and placement-related outcomes.

The purpose of this study is to look at the impact of the ChalleNGe program on cadets' noncognitive skills. The study involves administering a single paper and pencil survey to cadets at seven of the ChalleNGe sites; each cadet will complete the survey: (1) at the beginning of their time at ChalleNGe; and, (2) at the end of the residential phase. (The final survey will include only those who remain in the program until graduation). The survey (attached) includes two established scales, the locus of control scale and the Grit scale, as well as questions to determine the reason(s) the cadet dropped out of school.³ We have obtained permission to use both established scales. Dr. Angela Duckworth, the developer of the Grit scale, has approved the use of that scale. Permission to use the locus of control scale was granted by the office of Dr. Julian Rotter, with the caveats that we cannot publish the entire scale and that our survey (which contains the scale) must be collected from the cadets after it has been completed.

While this study will not provide a single all-encompassing measure of noncognitive skills, it is likely to provide insight into how the ChalleNGe program actually works, and how cadets' noncognitive skills and attitudes could be expected to change over time. Additionally, cadets' initial survey responses may be predictive of performance in the program. Finally, academic achievement is tied to teens' changes in locus of control over time;⁴ therefore, it is possible that survey responses will differ between cadets who earn different types of credentials.

To support this objective and to obtain an independent assessment, the Office of the Assistant Secretary of Defense (Reserve Affairs) has contracted with CNA to assist them with developing a survey instrument, conducting the surveys, and performing the data analysis.

Criteria for Subject Selection

Number of Subjects

The potential respondent universe includes all “cadets” (participants) at ChalleNGe programs. There are 34 ChalleNGe programs operating across the United States; this includes programs in 27 different states (some states have multiple programs) and 1 program in Puerto Rico. The number of cadets in each program varies slightly over time. In most classes, however, over 17,000 cadets register and enroll, while nearly 15,000 graduate (these numbers include all cadets at all sites in a single “class;” there are 2 classes per year).

Due to budget and time constraints, we plan to survey cadets at seven programs. We chose these programs in order to obtain a cross-section of cadets from different geographic areas. In addition, previous research has shown that ChalleNGe programs are managed differently. For example, some are more “militaristic” than others. As a result, we also chose sites in order to obtain data across program management styles. Programs were also selected based on whether the cadets work towards a GED or whether the program emphasizes credit recovery.

All cadets enrolled in the seven programs will be invited to participate in the study. These programs vary in size, but over recent cycles they have registered and enrolled some 1,500 cadets, and have graduated over 1,000 cadets. (See Table 1)

Table 1. Potential Subjects

Program(s):	Cadets registered	Likely responders	Cadets graduating	Likely responders	Female grads	Likely responders
All	~17,000		~15,000		~3,000
Sample (7 programs)	~1,500	~1,200	~1,000	~800	~200	~160

In terms of the sample size, we would like to be able to compare cadets who receive different credentials. The majority of cadets are in programs that award primarily GEDs; we expect about two-thirds of our sample to have received a GED. We would also like to be able to compare responses from male and female cadets. Because about 80 percent of cadets are male, the male-female comparison is binding in terms of sample size; in other words, if our sample is of sufficient size to compare responses of men and women, it will also be of sufficient size to compare responses of cadets with GEDs versus those with high school credits.

Based on an expected response rate of 80 percent, we estimate that our sample will include about 1,200 cadets who take the initial survey and roughly 800 cadets who take the final survey some 20 weeks later. (See Table 2 for a breakdown by program). Therefore, our sample is likely to include at least 240 cadets with each type of credential and 160 female cadets. These sample sizes will provide us with sufficient power to discern differences in responses to our survey measures.

Based on conversations with Program Directors, we believe our estimated response rate of 80 percent is conservative and we are likely to achieve a higher rate, but we have no previous data collection and so no prior response rate for comparison.

Table 2 shows the approximate number of cadets we expect to be enrolled in each program. It also indicates the number of cadets we expect to complete the first survey and the second survey. Due to the fact that enrollment at these sites has not been finalized yet, these numbers are projected enrollment based on prior class size and bed capacity at each site. We expect the sample size to decrease over the course of the study because nearly one-third of cadets typically leave the program before completing it.

Table 2. Cadet Sample Size

Site	Total	N First Survey*	N Second Survey*
California	250	200	133
Georgia	250	200	133
Louisiana	250	200	133
Illinois	344	275	184
Maryland	125	100	67
Washington	156	125	83
Wisconsin	125	100	67
Total	1,500	1,200	800

*Assumes a response rate of 80 percent and a completion rate of 67 percent.

Gender of Subjects

The gender distribution of the subjects can be found in Table 3, below. There are no gender-based enrollment restrictions. No pregnant cadets will be included in the research as they are not allowed to be enrolled in the program while pregnant. Again, the cadet figures are based on projections as enrollment has not yet been finalized. Historically, 80 percent of cadets are male and 20 percent are female.

Table 3. Gender Distribution of Cadets

Gender	Total	N First Survey	N Second Survey
Male	1,200	960	640
Female	300	240	160
Total:	1,500	1,200	800

Age of Subjects

ChalleNGe cadets range in age from 16-18. Cadets of any age at the selected sites are eligible to participate in the research. Below we provide the age of majority for the seven states where data will be gathered. We also discuss any special considerations that apply to our research and our handling of those requirements.

California

Age of Majority: 18

California follows the Federal Common Rule, but also has specific informed consent requirements, which can be found at: http://oag.ca.gov/research/consent. The consent forms we developed comply with these requirements. California law also requires that subjects be provided a copy of the “California Experimental Subject’s Bill of Rights.” A copy of this document will be provided to all subjects and their parent or guardian (where applicable).

Georgia

Age of Majority: 18

Georgia follows the Federal Common Rule and does not have any other requirements that apply to our research.

Illinois

Age of Majority: 18

Illinois follows the Federal Common Rule. Illinois law also requires not just a parent or guardian’s consent for a minor to participate in research, but also the minor’s assent to participate. Our cadet consent form fulfills this requirement.

Louisiana

Age of Majority: 18

Louisiana follows the Federal Common Rule and does not have any other requirements that apply to our research.

Maryland

Age of Majority: 18

Maryland follows the Federal Common Rule and does not have any other requirements that apply to our research.

Washington

Age of Majority: 18

Washington follows the Federal Common Rule and does not have any other requirements that apply to our research.

Wisconsin

Age of Majority: 18

Wisconsin follows the Federal Common Rule and does not have any other requirements that apply to our research.

Racial and Ethnic Origin

The racial and ethnic distribution of the population of ChalleNGe cadets is as follows:

Ethnicity:
Hispanic: 16%
Non-Hispanic: 84%

Race:
American Indian/Alaska Native: 4%
Asian/Pacific Islander: 3%
Black: 41%
White: 49%
"Other": 3%

There are no enrollment restrictions based on race or ethnic origin. Since not all cadets will be surveyed and the population of enrolled cadets varies with each class, we do not know what the racial and ethnic distribution of the subjects will be. However, since we do not plan to use the data to make comparisons by racial or ethnic group, under or overrepresentation should not have an impact on our results.

Inclusion and Exclusion Criteria

All cadets who choose to participate in the research will be eligible to participate. There are no inclusion and exclusion criteria.

Vulnerable Subjects

Children are included in this research because the program under study is specifically designed to help at-risk youth. It is not possible to exclude this vulnerable population and accomplish the research objectives. We could have chosen to ask adults that teach or otherwise supervise the cadets to provide us with their impression of the cadets’ noncognitive skills. However, these adults will have no knowledge of the cadets’ noncognitive skills at the beginning of the program since they will have just met the cadets. Asking for adults to give an evaluation of the cadets’ noncognitive skills at the end of the program could be done, but it would give a much less reliable measure than surveying the cadets directly.

Methods and Procedures

Methods and Procedures

The study involves the administration of two identical surveys:

1. A paper and pencil survey of cadets to be conducted at the beginning of their time at ChalleNGe

2. Re-administering the same paper and pencil survey to cadets who: (a) took the survey at the beginning of their time at ChalleNGe; and, (b) remain in the program until graduation.

The process we will use for obtaining consent/assent and conducting each survey is described below.

Survey #1

Consent/Assent Process: We will be obtaining parent/guardian consent for those cadets who are minors, in addition to the assent of the minor. Cadets who are of legal age will be asked for their consent to participate. To give potential subjects, and their parent or guardian (where applicable), ample time to review information on the study and the consent/assent documents, each ChalleNGe program in our study will be including these documents in each cadet’s acceptance letter package. The documents (attached) include: (1) a letter describing our study, inviting the cadet to participate, and requesting consent from interested parties; (2) the consent document asking for parent/guardian consent and subject assent for minors and consent for those of legal age; (3) a Research Subject Information Sheet for Adolescents for minors to read before they provide their assent to participate; and (4) for cadets residing in the state of California, potential subjects will be provided with a copy of the “California Experimental Subject’s Bill of Rights.” While all of the cadets speak and read English, some of their parents/guardians may only speak Spanish. Accordingly, the package will also include Spanish translations of all four items. If they are willing to participate in the study, cadets are asked to bring the consent/assent documents with them on registration day. We will have additional copies available for those who do not bring the documents (see below).

Survey Process: On registration day, we will set up a three-step process that will be incorporated into the ChalleNGe program’s registration process. A graphic display can be found in Figure 1. In Step 1, a CNA employee will ask each cadet if they would like to participate in the study and if they have their consent paperwork. If the cadet does not want to participate in the study, they will move on to another part of the program’s registration process. For cadets with paperwork in hand, the CNA employee will check the documents to ensure they are complete. If they are not complete, the cadet will move on to Step 2. If they are complete, the cadet will be given a “ticket” that allows them to move on to Step 3. The ticket will contain a random survey identifier code and a blank for the cadet to fill in their name. Cadets who would like to participate, but do not have signed documents with them, will move on to Step 2. In Step 2, a CNA employee will explain the study and consent process to the cadet and his or her parent or guardian (for minors). Any questions will be answered and, if they choose to participate, the documents will be signed and the cadet will be given a ticket and move on to Step 3. Cadets choosing not to participate and minors who do not have a parent/guardian with them will continue with the program’s registration process.

In Step 3, cadets with a ticket in hand will be taken into a classroom to complete the survey. First, the CNA researcher will read a script explaining how the survey administration will work, reminding the cadets that their participation is voluntary, and explaining what to do with the survey once it is complete. A copy of the script is attached. Based on tests of the survey, we estimate that it will take approximately 10 minutes to complete. After completing the survey, the cadets would continue with the program’s registration process.

All of the paper surveys will be sealed in an envelope and hand-carried until they can be secured in a safe. After the data have been entered into a database and checked, all paper surveys will be destroyed. The entrance “tickets” will also be collected and used by the ChalleNGe program staff to create a crosswalk of survey identifier codes and cadet names. Research staff will have no access to the crosswalk of survey identifier codes and cadet names and ChalleNGe program staff will have no access to completed surveys. ChalleNGe program staff will be asked to destroy the tickets and crosswalks after the surveys have been completed.

Figure 1. Process for Survey #1

Survey #2

The second survey will be administered during the week prior to the last day of the residential phase at a time that is convenient for the program. Prior to that time, CNA staff will work with the seven programs to identify those cadets that took the first survey and are still enrolled in the program. Cadets who have reached the age of majority since they completed the first survey will also be identified. Of the total number of cadets enrolled in any class, we estimate that 12 percent reach the age of majority during their participation in ChalleNGe.

CNA staff will first meet one by one with any cadet who has reached the age of majority and review the consent document with them. Cadets will then be asked if they would like to continue their participation in the study and if so, they will be asked to consent to participate.

Cadets who have consented to participate will be administered the survey either in one group, or by platoon depending on which option will be least disruptive to the program. As with the first survey, it will take approximately 10 minutes to complete. The appropriate unique survey identifiers will be pre-filled on each survey. The crosswalk will be used to ensure each cadet receives the correct survey to complete.

Data Analysis and Data Monitoring

Our research design is simple; we will collect survey data from cadets at the beginning and the end of the ChalleNGe program. Our survey measures cadets’ noncognitive skills. Our primary research goal is to evaluate the impact of the ChalleNGe program on cadets’ noncognitive skill growth. Doing this requires comparing cadets’ initial and final survey responses. However, we are also interested in comparing initial and final survey results for several groups within the data. First, we are interested in comparing the noncognitive skills and skill growth of cadets at programs that offer GEDs versus high school diplomas. Second, we are interested in comparing the noncognitive skills and skill growth of male versus female cadets. Third, we will look at how the probabilities of completing ChalleNGe vary with noncognitive skills. Finally, we will test the hypothesis that cadets who report leaving traditional high school for different reasons will demonstrate different skills or skill growth.

Answering these questions requires matching the surveys to additional information. We will do this using academic skills data provided by the ChalleNGe programs. This data will be coded by the programs using the crosswalk they created with cadet names and survey identifier codes. This will allow us to carry out our analysis without collecting names, gender, or other additional information on the surveys themselves.

Our analysis will include detailed descriptive statistics of noncognitive skills and skills growth of all cadets surveyed, and of each sub-group defined above. We will also utilize regression analysis to determine the relationships between personal characteristics, such as age and standardized test scores, and initial noncognitive skills/skills growth. When appropriate, we will produce estimates of the total skills growth on all ChalleNGe cadets; to do this, we will weight our data using information from the ChalleNGe programs.

Data Storage and Confidentiality

No personally identifying information (PII) will be collected. Rather, to link the pre- and post-surveys for each cadet, each cadet will be given a “ticket” before completing the survey that contains a unique random survey identifier code and a blank for the cadet to fill in their name. The ChalleNGe program staff will use these tickets to create a crosswalk of survey identifier codes and cadet names. Research staff will have no access to the crosswalk of survey identifier codes and cadet names and ChalleNGe program staff will have no access to completed surveys. All of the paper surveys will be sealed in an envelope and hand-carried until they can be secured in a safe. After the data have been entered into a database and checked, all paper surveys will be destroyed. In addition, ChalleNGe program staff will be asked to destroy the tickets and crosswalks after the surveys have been completed.

CNA is committed to safeguarding the rights and welfare of human participants in all research and adheres to the Privacy Act of 1974. CNA has taken several steps to ensure that subject information is kept confidential and to protect the privacy of these individuals. CNA data centers follow safe handling procedures to insure the safety of project data. These include provisions to monitor and prevent adverse events that involve maintaining data security and acting to prevent unauthorized release of data.

Only members of the study team will have access to the data. All data for this study will be stored on a securer server. The CNA LAN exists behind a Cisco Systems, Inc., firewall. All Windows-based Scientific Computing Operations (SCO) servers reside in a Windows domain separate from the corporate Windows domain, and on a separate virtual LAN (VLAN) from corporate servers and desktops. Access to the SCO domain requires a separate username/password and is restricted to only those with a need to know. Access to SCO servers is via Remote Desktop Connection or from a thin client server. Users do not have access to removable media on these servers. Users are not permitted to store sensitive data on desktops, laptops, or home computers. Integrity controls are used to maintain the security of the SCO servers. These controls include appropriate anti-virus software, updating of software via Microsoft Security Update Service, and monitoring of all log flags. The server is maintained in a locked, environmentally-controlled computer room.

Survey results for the cadets will be aggregated by ChalleNGe site, age, gender, and type of credential awarded. CNA follows the data rules of the Bureau of the Census in reporting summary statistics on small populations, including the rule that no summary statistics will be reported for groups of fewer than five individuals. Risk of identification is minimal. CNA requires that all project deliverables, reports, briefings, etc., be reviewed internally before release to the sponsor. That review will include a search for occasions of inadvertent release of information that could lead to subject identification. The individual selected to review reports will not be part of the same team as the principal investigator or the project research staff. Data will be stored for three years after the conclusion of the study and then destroyed.

Risk/Benefit Assessment

This research poses minimal risk to the cadets. It is possible that the cadets may not feel comfortable answering some of the questions on the survey. When we field-tested the survey, respondents reported no discomfort associated with any of the questions. To minimize the potential for discomfort, subjects will be told that they may skip any question(s) they do not feel comfortable answering and that they also have the option of stopping the survey and withdrawing from the study at any point if they so choose. There are no anticipated benefits to the subjects. Cadets who choose not to participate in the study will proceed with registration (for the first survey) and will continue with normal cadet activities (for the second survey).

Subject Identification, Recruitment and Consent/Assent

Method of Subject Identification and Recruitment

Potential subjects include all cadets enrolled at seven ChalleNGe sites. All cadets that are admitted will be mailed information on the study and will be invited to participate. The programs already mail all accepted cadets a package of information. Information on the study will also be contained in this mailing. Cadets will have the option of either participating in the study or not and it will be made clear that this choice will have no impact on their acceptance to ChalleNGe or their participation in the program.

Process of Consent

In order to promote rational and thoughtful decision-making by potential subjects (and their parent/guardian, where applicable) they will be able to review the study and consent information at home. This will allow them sufficient time to thoroughly review the documents, consider whether or not to participate, and contact CNA with any questions. The processes for obtaining consent for the two surveys are described above. All members of the study team are authorized to obtain consent. They are:

Dr. Lauren Malone

Ms. Jennifer Atkin

Mr. Christopher Sun

Ms. Jane Alexander

Subject Capacity and Subject/Representative Comprehension

We expect all potential subjects and their parent/guardian (where applicable) to have sufficient capacity to understand the study and to give informed consent. We will have CNA personnel available to discuss the survey and answer any questions as we obtain consent.

Documentation of Consent

All consent documents will be thoroughly reviewed prior to the subject being surveyed. The review will ensure that the consent form has the necessary signatures and that assent has been obtained where necessary. All signed consent forms will be retained by CNA until the completion of the study and publication of final results. The documents will be stored in a safe at CNA; they will not be available to CNA personnel who are not on the study team.

Costs to the Subject/Payment for Participation

Subjects will incur no costs to participate in the study and they will receive no compensation for their participation.

1 James Heckman, Jora Stixrud, and Sergio Urzua, "The Effects of Cognitive and Noncognitive Abilities on the Labor Market" 2006. Journal of Labor Economics 24(3):411-82 discusses both cognitive and noncognitive factors with an emphasis on labor market outcomes; Margo Coleman and Thomas DeLiere "An Economic Model of Locus of Control and the Human Capital Investment Decision" Journal of Human Resources, Summer 2003 38(3):701-721 discuss the relationship between non cognitive factors and classroom/educational success.

2 Wenger, J; McHugh, C; Sayala, S; and Shuford, R. “Variation in Participants and Policies Across ChalleNGe Programs,” April 2008 (CNA Research Memorandum D0017743.A2)

3 For information on the locus of control, see Julian B. Rotter. “Generalized expectancies for internal versus external control of reinforcement.” 1966. Psychological Monographs 80 (1, Whole No. 609). For information on the Grit scale, see Angela L. Duckwort et al. “Grit: Perseverance and Passion for Long-Term Goals,” 2007. Journal of Personality and Social Psychology 92(6): 1087-1101.

4See, e.g., Thomas Dee and Martin West “The Noncognitive Returns to Class Size” National Bureau of Economic Research Working Paper 13994, April 2008.