Response to Public Comments

National Reporting System for Adult Education

Information Collection Request

OMB Control Number 1830-0027

Responses to Public Comments

Comment

One commenter stated that Table 1 conflates educational functioning level (EFL) with pretest score. There are three ways to measure EFL, only one of which is by pretesting, and the table should make clear that Table 1 is determined by pretest level only.

Discussion

Table 1 is intended to capture the EFL at which the participant entered the program, rather than capturing whether a participant has made an EFL gain. Table 4 is intended to capture EFL gains.

The EFL at which a participant enters the program can only be determined by a pretest score, high school credits, or Carnegie Units at entry.

Change

No change.

Comment

Two commenters noted that clarification is needed for columns C and D of Table 4A to indicate whether speaking and listening tests may be used to measure and report EFL gains in these columns.

Discussion

The heading for Column C indicates the reporting of the number of participants with an EFL Gain for English Language Arts (ELA)/Literacy or English Language Proficiency (ELP) by pre- and post-testing. Column D indicates the reporting of the percentage of participants achieving ELA/Literacy or ELP EFL gains by pre- and post-testing. We agree that the instructions for Table 4A should be made more explicit.

Change

Instructions were added at the bottom of the table to clarify that EFL gains reported in Columns C and D may be measured by reading, writing, literacy skills, speaking or listening tests approved for use in National Reporting System for Adult Education (NRS).

Comment

Two commenters stated that instructions on the use of Carnegie Units to measure EFL gain are needed for reporting on Tables 4 and 4A.

Discussion

The Office of Career Technical and Adult Education (OCTAE) will provide technical assistance to clarify how to measure EFL gain using Carnegie Units.

Change

No change

Comment

One commenter stated that clarification was needed on Table 4A about whether entry into developmental education could be counted as entry into postsecondary education.

Discussion

OCTAE agrees that further clarification on what constitutes entry into postsecondary education is needed and will provide technical assistance to clarify that entry into developmental education could be counted as entry into postsecondary education.

Change

No change.

Comment

One commenter requested information about whether OCTAE would use Table 4A for performance indicators and whether targets would be required for each test.

Discussion

Table 4A is intended to provide additional aggregate information about State performance. The Workforce Innovation and Opportunity Act (WIOA) Statewide performance report, which was issued in the WIOA joint performance information collection request (ICR) under OMB control number 1205-0526, is used to determine State performance against State adjusted levels of performance. Guidance for negotiating levels of performance will be issued by the Departments of Education and Labor (the Departments).

Change

No change.

Comment

A few commenters disagreed with the types of gain designated for Title II use under the measurable skill gains indicator. Some commenters recommended that all five types of gain be used to measure success under the indicator because other core programs are allowing all five types to be used. Another commenter stated that the title of Table 4, Measurable Skill Gains, is inaccurate and could be confusing because measurable skill gain is defined five ways under WIOA, only two of which are used in adult education. The two methods do not comprise the full definition of measurable skill gain. A few commenters suggested adding rows to report all five types of measurable skill gains in Table 11 and an additional commenter suggested adding progress toward milestones to the table.

Discussion

The Departments described an approach to implementing the measurable skill gains indicator of performance in the preamble of the joint WIOA Notice of Proposed Rulemaking (April 16, 2015, 80 FR 20574) as including a variety of types of gain intended “to capture important progressions through pathways that offer different services based on program purposes and participant needs and can help fulfill the Departments’ vision of creating a workforce system that serves a diverse set of individuals with a range of services tailored to individual needs and goals…[f]or low-skilled adults, this proposed indicator provides an opportunity to track progress in reading, writing, mathematics, and English proficiency while they are participating in an adult education program prior to completing the high school credential and entering post-secondary education or training or employment.” Public comment supported this approach.

The Departments received comments in the proposed joint performance ICR related to types of gain that would apply to the adult education program. The Departments responded that they would issue program guidelines to identify the types of gain appropriate for each core program. On December 19, 2016, OCTAE issued Program Memorandum 17-2 OCTAE Performance Accountability Guidance for Workforce Innovation and Opportunity Act Title I, Title II, Title III and Title IV Core Programs. The guidance states that the decision to use types of gain only for services which the adult education program is authorized to provide is intended to keep accountability focused on the educational measures and activities authorized in statute, acknowledging basic skills are the foundation to success in other workforce services.

OCTAE identified achievement of at least one educational functioning level and documented attainment of a secondary school diploma or its recognized equivalent as appropriate types of gain for use under the measurable skill gains indicator based on the adult education and literacy services authorized in AEFLA. We recognize that successful passage of an exam that is required for a particular occupation or progress in attaining technical or occupational skills, listed as the fifth type of gain in the joint WIOA final rule at 34 CFR 463.155(a)(1)(v)(E), may be appropriate for adult education under limited circumstances. States may report on this type of gain in the NRS program-specific ICR for participants enrolled in integrated education and training (IET) programs. We intend to use the data reported by States on Table 11 of this ICR to determine whether this type of gain should be identified for performance accountability in the future.

OCTAE believes that the remaining two types of gain are not appropriate types of gain to use for the measurable skill gains indicator of performance in adult education programs:

The third type of gain described in 34 CFR 463.155(a)(1)(v)(C), a secondary or postsecondary transcript or report card for a sufficient number of credit hours that shows a participant is meeting the State unit’s academic standards, is not a type of gain appropriate to use for measuring accountability of adult education services for two reasons.  First, with respect to a secondary transcript, an individual enrolled in secondary school under State law is not an eligible individual under AEFLA; thus, individuals participating in AEFLA are not enrolled in secondary school and not positioned to receive secondary transcripts.  The Departments expanded the definition of educational functioning level gain to include the awarding of credits or Carnegie units in order to provide an appropriate measure for adults enrolled in credit-based adult education.  Second, with respect to a postsecondary transcript, adult education is defined in WIOA as “academic instruction and education services below the postsecondary level” (see section 203(1) of WIOA), which means that AEFLA participants are not positioned to earn postsecondary credit hours. A postsecondary transcript is therefore an inappropriate accountability measure of adult education service success under AEFLA.  A State, however, may register a gain under educational functioning level if a participant exits the AEFLA program and enters postsecondary education. We believe the expansion of the educational functioning level definition to add credit completion and entrance into postsecondary education provides appropriate ways to measure the success of the AEFLA program.

The fourth type of gain described in 34 CFR 463.155(a)(1)(v)(D), a satisfactory or better progress report towards established milestones from an employer or training provider who is providing training, is not an appropriate type of gain to use to determine measurable skill gain for adult education because AEFLA programs and services authorized in statute do not include “placing” a participant with an employer or training provider, as do other core programs.  For example, the vocational rehabilitation (VR) program may pay for the placement of a participant in a training program and appropriately measure success of the VR program on this type of gain.  AEFLA delivers educational services below the postsecondary level through direct service rather than via “placement” with a training provider or an employer.

Change

No change.

Comment

Several commenters stated that OCTAE’s method for populating the Statewide performance report (ETA-9169) does not comply with the joint reporting specifications and should not be permitted. The commenters stated that calculations in NRS tables do not fully comply with the Participant Individual Record Layout (PIRL). The first concern relates to language which says that reporting for postsecondary credentials would only apply for participants in IET programs. The second concern relates to Title II only reporting two types of measurable skill gains (EFL gain and secondary credential completion). The commenters suggested completely disconnecting the Statewide performance report (ETA-9169) from the NRS or redesigning the NRS to follow the joint reporting specifications.

Discussion

OCTAE agrees that the postsecondary education portion of the credential attainment indicator applies to all participants who exit postsecondary education, not just participants who are enrolled in IET programs. We have made the correction to the footnote. In our response above, we address the second concern related to the appropriate types of gain for purposes of the measurable skill gains indicator of performance for AEFLA programs.

Change

OCTAE revised language in the NRS tables to note that credential attainment applies to participants who are co-enrolled in any program leading to a recognized postsecondary credential. We will address this issue further through technical assistance.

Comment

A few commenters stated that a participant who remains a participant across program years should not need 12 hours of service in both years and should be reported only in the year of exit as one Period of Participation. For example, a participant in June who does not exit before the end of the program year on June 30 remains a participant in the next program year (which begins July 1), regardless of whether the participant has 12 hours of service.

Discussion

We agree with the commenters and note that a participant does not need 12 hours in the subsequent program year to be counted as a participant if he or she was already a participant in the previous program year. This policy is reflected in Program Memorandum OCTAE 17-2, Performance Accountability Guidance for Workforce Innovation and Opportunity Act Title I, Title II, Title III and Title IV Core Programs in Attachment 6.

Change

No change.

Comment

A few commenters stated that OCTAE should continue from previous reporting a participant designation known as work-based project learners. This designation allowed participants to be evaluated through an exam that assessed progress on occupational skills.

Discussion

The work-based project learner optional table in the NRS program-specific ICR was eliminated, because very few States submitted the optional table and the data elements contained in the table are not required by WIOA. Elimination reduces reporting burden on States. States may continue to collect this information if it is valuable within the State, but there is no requirement to report it at the Federal level.

Change

No change.

Comment

Several commenters noted that the burden required to collect employment and earnings indicators is extensive if the State is unable to perform data matching. Such States must use surveys or other supplemental sources to collect these data and the number of participants for whom these data must be collected will increase significantly under WIOA. Several other commenters stated that employment data to be collected from participants through a supplemental survey should be limited only to whether they were employed in the second quarter after exit, the fourth quarter after exit, and an estimate of their earnings, because any additional data requested would be burdensome.

Discussion

The Departments received comments about the obstacles associated with wage record matching during the WIOA rule making process. Some commenters recommended that participant records without Social Security Numbers (SSN) be excluded from the employment and earnings indicators. In response to these comments, the final rule was changed to allow States to use supplemental data to report on participants for whom a data match could not be completed because of missing data elements, such as an SSN. The Departments do not have the authority to exclude these participants from performance indicators. The Departments will provide guidance on the use of acceptable supplemental information to collect data for the performance indicators. We are unable to make changes to the program-specific ICR that are not consistent with the joint WIOA final rule.

Change

No change.

Comment

A commenter stated that OCTAE should extend the time allowed for reporting of entry into postsecondary education and attainment of a secondary credential for measurable skill gain from the end of the program year (June 30) to October. This additional time is needed to allow for follow up of these data for participants who attend and exit at the end of the program year.

Discussion

The WIOA reporting deadline for all core programs is October following the end of the program year. Extending the time for collecting data for the measurable skill gain indicator as suggested would not allow States to meet the October reporting deadline, triggering failure to report provisions under section 116 of WIOA. Therefore OCTAE cannot extend the time for data collection of this measure.

Change

No change.

Comment

A commenter stated that there were no options for high school completion or passing the General Education Development (GED) on the reporting tables.

Discussion

Table 4 allows the reporting of obtaining a secondary school diploma or its recognized equivalent (which would include high school completion or the GED) as a means of determining measurable skill gain.

Change

No change.

Comment

A few commenters suggested expanding the IELCE financial reporting from the “basic” levels (c. Programs of Instruction 1-8) as was reported under WIA, to include the shaded “d. Programs of Instruction 9-12”.

Discussion

We agree that the headings for columns C and D in Box 10 of the federal financial report could be made more explicit and have added clarifying text for better alignment with the levels of instruction provided in the adult education program. We are instructing that expenditures at all ESL levels are to be reported in column C. This instruction is not intended to restrict program eligibility. Any student who meets the program eligibility requirements may be served. We believe that expanding IELCE reporting would unnecessarily increase burden.

Change

The words “ABE levels 1-4 and ESL levels 1-6” were added to the heading for column C. The words “ABE levels 5 and 6” were added to the heading for column D.

Comment

Several commenters stated that there should be separate rows on Table 4 for reporting each period of participation, rather than columns for each period of participation because this information is needed for the Statewide performance report (ETA-9169). Another commenter noted there is no information on how to define period of participation and whether more than one measurable skill gain per period of participation can be reported for each participant per year.

Discussion

OCTAE considered several different ways for States to record multiple periods of participation and outcomes per period of participation on Table 4, including adding additional rows. The complexity and level of burden for reporting period of participation on a table with additional rows increased substantially and made the table difficult to complete and interpret. The current design of adding period of participation and corresponding outcomes per period of participation in columns creates minimal extra burden, simplifies interpretation, and allows for accurate reporting of indicators in the joint ICR.

The definition of period of participation is described in a separate joint guidance document. A footnote to Table 4 provides an explanation for how to report measurable skill gains for each period of participation.

Change

No change.

Comment

We received several comments on the instructions in a footnote for completing Table 3 concerning reporting of participants in Integrated English Literacy and Civics Education programs (IELCE), provided under section 243 of WIOA. The footnote states “This number includes those enrolled in IET programs. It does not include those enrolled in ELA programs”. The commenters state that this footnote needs to be revised to clarify that participants, served under section 231 of WIOA, are not included here and to recognize that IELCE activities are not limited to instruction for English language acquisition.

Discussion

Table 3 allows non-duplicative reporting of participants served in programs under section 231 of WIOA (Adult Basic Education, Adult Secondary Education and English Language Acquisition) and in IELCE programs under section 243 of WIOA. In addition, IET participants are disaggregated separately for each program area. The footnote in question also states the reporting row includes the “Number of participants enrolled in IELCE programs (Sec. 243 of WIOA)” to clarify that this row is to include only participants served in programs under section 243 of WIOA and not participants served in English language acquisition programs under section 231 of WIOA. Although the footnote makes no reference to the services in IELCE programs, we agree that a clarification of the footnote would be helpful.

Change

The footnote is revised to clarify that participants, reported on this table as enrolled in IELCE programs under section 243 of WIOA, should not be reported elsewhere in this table.

Comment

A number of commenters noted that the footnote in Table 5, for reporting participants who attain a postsecondary credential, limits this measure to participants enrolled in IET programs. The commenters state that this instruction should be expanded to include all participants dually enrolled in adult education and any postsecondary education program, not only IET programs.

Discussion

We agree that the footnote should be clarified to indicate that all participants jointly co-enrolled in adult education and postsecondary education should be included in the postsecondary credential portion of the indicator.

Change

The footnote has been revised to indicate that reporting must include all participants co-enrolled in adult education and postsecondary education programs.

Comment

Two commenters stated that the instructions on Table 5 should make it clearer that if a participant exits more than once during the program year the participant is included in the denominator once per exit.

Discussion

Columns E and F in Table 5 are for reporting the total number of periods of participation for participants and number achieving outcomes, respectively. A footnote to the table for Column E indicates that all periods of participation should be reported and a footnote to the table for Column F indicates that all outcomes for periods of participation should be reported. The instruction requested by the commenters is already present.

Change

No change.

Comment

A few commenters suggested enhancing the instructions in Table 5 for the secondary credential component of the credential indicator to recognize achievement of a diploma/equivalent in a language other English. They noted that the footnote to the table states that the secondary credential is for students at a ninth grade equivalent or higher upon entry and there is no corollary for English language learners who may achieve a secondary credential in another language. The commenters state this makes it impossible to identify these students for the credential rate and recommended that language allowing the recognition of achievement of a diploma/equivalent in a language other than English be included in the footnote.

Discussion

Program Memorandum 17-2 OCTAE Performance Accountability Guidance for Workforce Innovation and Opportunity Act Title I, Title II, Title III and Title IV Core Programs provides a definition of a secondary school diploma or equivalent. A secondary diploma is one that is recognized by the State and that is included for accountability purposes under the Elementary and Secondary Education Act of 1965 as amended by the Every Student Succeeds Act. A secondary school equivalency certification signifies that a student has completed the requirements for a high school education. Types of equivalents are those recognized by the State. If a participant receives a diploma in a language other than English, it may be counted as a secondary school credential if it is a diploma/equivalency recognized by the State. The policies related to diplomas and equivalencies vary among States. We believe the Program Memorandum is clear and no change is needed to the footnote on Table 5.

Change

No change.

Comment

A few commenters recommended that the credential rate indicator be broken out in an additional table by participants at the secondary or postsecondary level, in order to identify those who achieve a diploma/credential within a year of exit and whether they were both employed and enrolled in postsecondary education.

Discussion

Table 5 allows the reporting of all required elements of the credential indicator for each period of participation for all participants. There is no WIOA reporting requirement to identify participants who achieve a diploma/credential within a year of exit and to identify whether they were both employed and enrolled in postsecondary education. We believe the addition of a new reporting table as recommended creates an unnecessary burden on States.

Change

No change.

Other Comments not relevant to the ICR

Several comments were received that were on topics not related to the ICR. These included comments related to the statistical adjustment model required by WIOA for adjusting State performance levels and how pretests and posttests are used to measure educational functioning level gain. Additional information on these topics may be addressed in other guidance and technical assistance documents.

Technical and Conforming Changes

In addition to the changes described above, OCTAE has made technical and conforming changes for purposes of clarity and for consistency with WIOA and the implementing NRS regulations at 34 CFR part 462.

Page 10 of 10