Supporting Statement part B_FINAL-07-21-14

SUPPORTING STATEMENT – PART B

B.  COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1.  Description of the Activity

The potential respondent universe includes all “cadets” (participants) at National Guard Youth Challenge (ChalleNGe) programs. There are 34 ChalleNGe programs operating across the United States; this includes programs in 29 different states (some states have multiple programs) plus the District of Columbia and Puerto Rico.

Each year there are two “classes” of cadets, which includes cadets across all programs. Typically, one class of cadets at each program starts in the winter and the other in the late summer/early fall. Although the number of cadets enrolled in each class varies slightly, most classes have registered and enrolled over 17,000 cadets with nearly 15,000 of them graduating from the program. We will collect data at 7 different sites; these sites vary in size, but over recent cycles they have registered and enrolled approximately 1,500 cadets per class, and have graduated over 1,000 cadets. (Table 1 provides these figures.)

Table 1

Program(s):	Cadets registered	Likely responders	Cadets graduating	Likely responders	Female graduates	Likely responders
All	~17,000		~15,000		~3,000
Sample (7 programs)	~1,500	~1,200	~1,000	~800	~200	~160

Due to budget and time constraints, we plan to survey cadets at seven programs. The sites chosen as the focus of this study are: San Luis Obispo, CA; Fort Gordon, GA; Gillis Long Youth ChalleNGe Center, LA; Wisconsin ChalleNGe Academy, WI; Lincoln’s ChalleNGe Academy, IL; Freestate ChalleNGe Academy, MD; and Washington Youth Academy, WA. These were selected in an effort to have a balance of programs that provide GED preparation and those that focus on credit recovery, while at the same time having a wide geographic disbursement. We also took into consideration the size of the programs as we wanted to ensure we would have a sufficient sample size. Lastly, we considered the intake process used at each site since part of our recruitment and consent processes have to be incorporated into the program’s intake day.

All cadets enrolled in the seven programs will be invited to participate in the study. Based on an expected response rate of 80 percent, we estimate that our sample will include about 1,200 cadets who take the initial survey and roughly 800 cadets who take the final survey some 20 weeks later (again, see table 1). Therefore, our sample is likely to include at least 160 female cadets. These sample sizes will provide us with sufficient power to discern differences in responses to our survey measures (see below).

Based on conversations with Program Directors and the steps we plan to take to maximize our response rate (outlined below), we believe our estimated response rate of 80 percent is conservative and we are likely to achieve a higher rate. However, we have no previous data collection and thus no prior response rate for comparison.

2. Procedures for the Collection of Information

The study involves the administration of two identical surveys:

1. A paper and pencil survey of cadets to be conducted at the beginning of their time at ChalleNGe

2. Re-administering the same paper and pencil survey to cadets who: (a) took the survey at the beginning of their time at ChalleNGe; and, (b) remain in the program until graduation.

The process we will use for obtaining consent/assent and conducting each survey is described below.

Survey #1

Consent/Assent Process: We will be obtaining parent/guardian consent for those cadets who are minors, in addition to the assent of the minor. Cadets who are of legal age will be asked for their consent to participate. To give potential subjects, and their parent or guardian (where applicable), ample time to review information on the study and the consent/assent documents, each ChalleNGe program in our study will be including these documents in each cadet’s acceptance letter package. The documents include: (1) a letter describing our study, inviting the cadet to participate, and requesting consent from interested parties; (2) the consent document asking for parent/guardian consent and subject assent for minors and consent for those of legal age; (3) a Research Subject Information Sheet for Adolescents for minors to read before they provide their assent to participate; and (4) for cadets residing in the state of California, potential subjects will be provided with a copy of the “California Experimental Subject’s Bill of Rights.” While all of the cadets speak and read English, some of their parents/guardians may only speak Spanish. Accordingly, the package will also include Spanish translations of all four items (Spanish language versions are in development). If they are willing to participate in the study, cadets are asked to bring the consent/assent documents with them on registration day. As a reminder about the study, we will have each of the seven programs send an email to each cadet one week prior to their arrival at ChalleNGe that reminds them about being invited to participate in the study and includes all of the same paperwork that was mailed to them. We will also have additional copies available at intake for those who do not bring the documents (see below).

Survey Process: On registration day, we will set up a three-step process that will be incorporated into the ChalleNGe program’s registration process. Every cadet will be contacted regarding the study. A graphic display can be found in Figure 1. In Step 1, a CNA employee will ask each cadet if they would like to participate in the study and if they have their consent paperwork. If the cadet does not want to participate in the study, they will move on to another part of the program’s registration process. For cadets with paperwork in hand, the CNA employee will check the documents to ensure they are complete. If they are not complete, the cadet will move on to Step 2. If they are complete, the cadet will be given a “ticket” that allows them to move on to Step 3. The ticket will contain a random survey identifier code and a blank for the cadet to fill in their name. Cadets who would like to participate, but do not have signed documents with them, will move on to Step 2. In Step 2, a CNA employee will explain the study and consent process to the cadet and his or her parent or guardian (for minors). Any questions will be answered and, if they choose to participate, the documents will be signed and the cadet will be given a ticket and move on to Step 3. Cadets choosing not to participate and minors who do not have a parent/guardian with them will continue with the program’s registration process.

In Step 3, cadets with a ticket in hand will be taken into a classroom to complete the survey. First, the CNA researcher will read a script explaining how the survey administration will work, reminding the cadets that their participation is voluntary, and explaining what to do with the survey once it is complete. A copy of the script is included with this submission. Based on tests of the survey, we estimate that it will take approximately 10 minutes to complete. After completing the survey, the cadets would continue with the program’s registration process. Because the tickets will have the cadet’s ChalleNGe ID number on it, these pieces of paper will be shredded immediately after the cadet completes his/her survey.

All of the paper surveys will be sealed in an envelope and hand-carried until they can be secured in a safe. After the data have been entered into a database and checked, all paper surveys will be destroyed. The entrance “tickets” will also be collected and used by the ChalleNGe program staff to create a crosswalk of survey identifier codes and cadet names. Research staff will have no access to the crosswalk of survey identifier codes and cadet names and ChalleNGe program staff will have no access to completed surveys. ChalleNGe program staff will be asked to destroy the tickets and crosswalks after the surveys have been completed.

Figure 1. Process for Survey #1

Survey #2

The second survey will be administered during the week prior to the last day of the residential phase at a time that is convenient for the program. Prior to that time, CNA staff will work with the seven programs to identify those cadets that took the first survey and are still enrolled in the program. Cadets who have reached the age of majority since they completed the first survey will also be identified. Of the total number of cadets enrolled in any class, we estimate that 12 percent reach the age of majority during their participation in ChalleNGe.

CNA staff will first meet one by one with any cadet who has reached the age of majority and review the consent document with them. Cadets will then be asked if they would like to continue their participation in the study and if so, they will be asked to consent to participate.

Cadets who have consented to participate will be administered the survey either in one group, or by platoon depending on which option will be least disruptive to the program. As with the first survey, it will take approximately 10 minutes to complete. The appropriate unique survey identifiers will be pre-filled on each survey. The crosswalk will be used to ensure each cadet receives the correct survey to complete.

We chose our sample sites to achieve variation in terms of credentials awarded and geographic location, and to produce sufficient expected sample sizes to be able to discern differences on our surveyed measures between male and female cadets. We are using no sampling procedures at the selected sites; instead, we are surveying all cadets at each of the selected sites. The sampling plan outlined in table 1 will help ensure several levels of accuracy. The results of a power analysis are presented in table 2. Assuming an 80-percent response rate and a 20-percent proportion of female cadets, as well as that the mean and variance of our measurements will be similar to those in past literature, we will have at least 80-percent power to detect a small difference (about a quarter of a standard deviation) with 95-percent confidence. The calculations are based on power formulae for a 2-sample t-test for a difference of means when the two group sizes are not equal. We do not foresee any unusual problems and do not plan to implement special sampling procedures.

Table 2. Power analysis results

Respondents	Female respondents	Effect size	Power (2-sided test)
800	160	0.25	0.81
800	160	0.33	0.96
800	160	0.50	0.99

3. Maximization of Response Rates, Nonresponse, and Reliability

We will take several steps to maximize our response rate. First, we will have research staff on site at each data collection to answer questions. We will collect our initial survey data as part of the ChalleNGe intake process; therefore, we will have the opportunity to speak with all cadets and their families in person. This is likely to result in a much higher level of response than, for example, a mailed survey.

We also will have research staff in the room while cadets are filling out our survey, to answer any questions and address concerns that could otherwise result in inaccuracy. Finally, as discussed above and below, we field-tested our survey on small numbers of cadets at two different sites; we modified the survey based on these cadets’ comments with a focus on increasing clarity. We expect that nearly all cadets who take the first survey and remain in the program at the conclusion will agree to complete the second survey.

We will ensure accuracy and reliability of responses in the following ways:

• Because there are no benefits to those who complete the survey and no costs to those who do not, and because the survey does not ask polarizing questions, we expect no selection bias in those who choose to participate. Thus, we expect that the study conclusions will be generalizable to the population of cadets who chose not to take the survey.

• Most of the survey questions are multiple-choice, so there is little chance of the respondent misunderstanding a question.

• Respondents are allowed to skip any question(s), and thus have no motivation to give inaccurate answers.

4. Tests of Procedures

To ensure the survey will yield useful and valid results we developed it based in large part on two established scales. We also tested the survey on ChalleNGe cadets to increase the chances that our population of interest would find it easy to understand and complete.

The majority of the survey questions are identical to those currently used in the locus of control scale and the Grit scale.¹ Both scales have been extensively tested, including on children. The Grit scale, for example, was used for a study on children competing in the National Spelling Bee.² The locus of control scale has been included in several large surveys of young people, such as the National Education Longitudinal Survey of 8^th graders begun in 1988 (NELS88).³ The cadets read at approximately a 7^th grade level.⁴ Since the scales have been used successfully on children, we are confident that the questions will be easily understood.

We have obtained permission to use both established scales. Dr. Angela Duckworth, the developer of the Grit scale, has approved the use of that scale. Permission to use the locus of control scale was granted by the office of Dr. Julian Rotter, with the caveats that we cannot publish the entire scale in our report and that our survey (which contains the scale) must be collected from the cadets after it has been completed.

Our survey was tested twice. The purpose of the tests was to ensure the survey questions were written at a reading level that made them clear and easy to understand. The survey was tested on five cadets at the Fort Gordon ChalleNGe site. In selecting cadets to be surveyed, we specified that we did not want to survey only those cadets that were performing well academically. Rather, we were interested in a population that would likely have the most difficulty understanding a survey. It took the cadets an average of 8 minutes and 38 seconds to complete the survey. We received comments on several of the proposed questions that the cadets found to be confusing. The cadets made suggestions to make the questions easier to understand. We used these suggestions to amend the survey. After amending the survey in response to the first test, we conducted an additional test on four cadets at the Washington State ChalleNGe site. Again, we surveyed cadets that had more difficulties academically. These cadets took an average of 7 minutes to complete the survey. No additional substantive comments or concerns were raised.

5. Statistical Consultation and Information Analysis

Dr. Yevgeniya (Jane) Pinelis at CNA was consulted on statistical aspects of the study design. Dr. Pinelis can be reached at (703) 824-2052. Dr. Lauren Malone at CNA will be responsible for the collection and analysis of the data. She can be reached at (703) 824-2741.

1 For information on the locus of control, see Julian B. Rotter. “Generalized expectancies for internal versus external control of reinforcement.” 1966. Psychological Monographs 80 (1, Whole No. 609). For information on the Grit scale, see Angela L. Duckworth et al. “Grit: Perseverance and Passion for Long-Term Goals,” 2007. Journal of Personality and Social Psychology 92(6): 1087-1101.

2 Angela Duckworth et al., “Deliberate Practice Spells Success: Why Grittier Competitors Triumph at the National Spelling Bee,” October 4, 2010. Social Psychological and Personality Science 2: 174-181.

3 See, e.g., Margo Coleman and Thomas DeLeire, "An Economic Model of Locus of Control and the Human Capital Investment Decision," Journal of Human Resources 38, no. 3, Summer 2003:701-721.

4 Jennie W. Wenger and Cathleen M. McHugh with Seema Sayala and Robert W. Shuford, April 2008,“Variation in Participants and Policies Across ChalleNGe Programs,” CNA Research Memorandum D0017743.A2.

5