NRS State Data Quality Checklist

A Project of the U.S. Department of Education

NRS STATE DATA QUALITY CHECKLIST
OMB Number 1830-0027

State:

Date:

Completed by (name and title):

A. Data Foundation and Structure
Acceptable Quality
1. State has written assessment policies that specify:







Yes

No

Standardized assessments to use for accountability that are valid and appropriate for
adult students.
Time periods (in hours or weeks) for when to pre- and posttest.
Score ranges tied to educational functioning levels (EFL) for placement and for
reporting gains for accountability.
Appropriate guidance on tests and placement for special populations (e.g., students
who are unable to be tested due to language or disability).
Unacceptable methods of assessment for EFL placement.
Appropriate guidance on requirements and conditions for testing distance education
students reported in the NRS (if applicable).

1a. List up to three of the most of commonly used assessments permitted for ABE and
ESL.
ABE Assessments:

ESL Assessments:

2. State has written policies for follow-up that explain:




Yes

No

Which students are to be followed.
How to determine tracking cohorts.
Follow-up methodology (survey or data match) for each measure that meets NRS
requirements.

2a. Indicate your follow-up methods for each measure.
Employment 2nd Quarter:

Survey

Data match

Both (explain)

Employment 4th Quarter:

Survey

Data match

Both (explain)

Median Earnings 2nd Quarter:

Survey

Data match

Both (explain)

Secondary Diploma:
Survey
Data match
Both (explain)
(Enrolled in postsecondary education or training within one year of exit)

Secondary Diploma:
Survey
(Employed within one year of exit)

Data match

Both (explain)

Postsecondary Credential:

Data match

Both (explain)

Survey

2b. If you indicated that you use both survey and data match for follow-up methods for one
or more measures in 2a, please explain.
3. If state uses survey follow-up method for any measure (check not applicable (NA) and
skip to the next item if survey is not used):
NA




Local programs or state can produce a list of students to survey, according to NRS
requirements.
Yes
No
Survey is conducted with a state provided, standard survey instrument.
Yes
No
State has a regular schedule (e.g., quarterly) for submission of survey data or student
names from local programs.
Yes
No

4. If state uses data matching for any measure (check not applicable (NA) and skip to the
next question if data matching is not used):
NA



Local or state data system can produce files for matching that include exit dates and
employment status for each student.
Yes
No
State has established a procedure for collecting Social Security numbers or other
unique identifiers, including how to deal with missing numbers.



Yes
No
State has set a regular schedule for data submission from local programs and for data
matching with external agencies.
Yes
No

5. States has provided to all local programs a copy of the assessment policy and an
explanation of the policy.
Yes
No
6. State has provided to all local programs the written state policies, procedures and
requirements for student follow-up and an explanation of the procedures.
Yes
No
7. The state has written definitions for all measures (including demographic measures and
actual or proxy contact hours, if applicable), defined according to NRS requirements and
has provided them to all programs.
Yes
No
8. The State has written policies on the use of proxy hour models to assign the proxy hours
for distance education learners participation in adult education distance education
programs. (Check NA and skip to the next item if proxy hours are not used)
NA
Yes
No
8a. If yes, please identify which model or models were used to assign proxy hours
___
Clock Time Model
___
Teacher Verification Model
___
Learner Mastery Model
Superior Quality
1. The state has a comprehensive data dictionary, which defines all measures on state
student data forms and in the state data system, and has provided it with an explanation to
all local programs.
Yes
No
2. State has standards or requirements for the percentage of students to be pre- and
posttested.
Yes
No
2a. If yes, indicate the standards or requirements.

3. State has made available to local programs on a continuous basis additional technical
assistance and resources on assessment, data collection and follow-up procedures (e.g.,
site visits, contact persons, manuals, online resources).
Yes
No
3a. If yes, briefly describe the assistance and how it is provided.

4. If state uses survey follow-up method for any measure, the state has taken steps (e.g.,
through data review, discussion with staff or observation) to verify that the survey is
being conducted according to NRS guidelines (check NA and skip to the next item if
survey is not used).
NA
Yes
No

4a. If yes, briefly describe your verification procedures.

5. If state uses survey follow-up method, the state has provided written guidance or
assistance on how to improve response rates to survey staff (check NA and skip to the
next item, if survey is not used).
NA
Yes
No
6. If state uses data matching, the state has written procedures on how to conduct data
matching that comply with NRS guidelines (check NA and skip to the next item if data
matching is not used).
NA
Yes
No

7. State has procedures in place that verify whether proxy hours are calculated and assigned
appropriately (check NA and skip to the next item if proxy hours are not used).
NA
Yes
No
Exemplary Quality
1. State has a system for verifying that local programs are following state data policies and
procedures through program reviews, auditing or a certification process.
Yes
No
1a. If yes, briefly describe your verification procedures.

2. State has reviewed technical reports or research studies of the comparative reliability and
validity of the NRS assessments it uses to accurately evaluate EFL completion
performance of its students.
Yes
No
2a. If yes, briefly describe the technical reports or research studies you reviewed.

Data Collection and Verification
Acceptable Quality
1. The state has an electronic management information system (MIS), used by all programs,
that has individual student records within a relational data base structure. The MIS
incorporates NRS measures using common definitions and categories.
.
Yes
No
2. Database has error checking functions used by state and/or local programs (e.g., that
identify out-of-range values and missing data). .
Yes
No
3. State has standardized forms (electronic or paper) for collecting student information (e.g.,
intake, attendance) that include all NRS measures and have correct NRS definitions and
categories. .
Yes
No
4. All programs are required to use state student data forms.

Yes

No

5. State has provided to local programs guidelines or procedures for recording actual and,
if applicable, proxy contact hours that conform to NRS requirements.
Yes
No
6. All or most local programs have staff with clear responsibility for data collection and data
entry.
Yes
No
7. State staff checks data for errors after submission by local programs.
Yes
No
7a. If yes, explain error checking process, including what data are checked and how
often.
Superior Quality
1. Programs and/or state enter data into MIS at least quarterly.

Yes

No

2. State staff reviews local data at least quarterly for errors, missing data, out-of-range
values and anomalous data, and to identify program improvements and accomplishments
and has a system to resolve them.
Yes
No
3. State has timely (e.g., quarterly) follow-up back to local programs to have them correct
missing and erroneous data.
Yes
No
4. State has documented procedures for correcting errors and resolving missing data that
programs use.
Yes
No
4a. If yes, briefly explain your data review and error correction system.

5. State provides additional technical assistance to local programs with poor data, as needed.
Yes
No
Exemplary Quality
1. State has a regular system for verifying (through software, onsite auditing, contact with
local staff) that local programs are following state data collection procedures.
Yes
No
1a. If yes, briefly describe the methods used for verification, including use of the correct
assessments and assessment forms, reporting of accurate score ranges for placement and
for reporting advancement for accountability.

2. State verifies data have been corrected in state or local database after errors have been
found.
Yes
No

3. State has procedures for regular contact with local staff on data issues to identify
problems and provide assistance.
Yes
No
3a. If yes, specify procedures and type of contact.

4. If state uses survey follow-up method, state tracks survey response rates on at least a
quarterly basis and takes corrective action if problems are identified (check NA and skip
if survey is not used).
NA
Yes
No

Data Analysis and Reporting
Acceptable Quality
1. The state MIS can produce NRS required reports for state, including federal NRS tables
Yes
No
2. NRS tables are calculated accurately to include error checks and prevent double counting.
Data is reported using minimally acceptable levels of response as specified in the
reporting tables.
Yes
No
3. State staff (or designee) checks NRS reports for errors and missing data and obtains
corrected data from local program reports.
Yes
No
4. The MIS is capable of reporting disaggregated data by subpopulation (e.g., student age,
race, sex) and program (e.g., ABE, ESL, ASE, correctional education, distance
education).
Yes
No
Superior Quality
1. State staff person familiar with the data, but not directly involved with collection and data
entry, reviews NRS data reports for errors and accuracy.
Yes
No
2. State staff uses data for program management and improvement.
Yes

No

2a. If yes, provide at least one example of use of data for this purpose in the last year.

3. Local programs can access data reports that are useful for program management and
improvement.
Yes
No
3a. If yes, briefly describe the usefulness of two reports produced by your system.

4. Local staff uses data for program management and improvement.
Yes

No

Exemplary Quality
1. State has a system of regular contact with local programs on data analysis issues and
reporting needs to identify technical assistance needs.
Yes
No
1a. If yes, specify method and frequency of contact.
2. State has documented procedures for dealing with analysis problems and deviations.
Yes
No
3. State compares data among programs and with prior years’ data for discrepancies,
reasonableness and to identify trends in good and bad performance.
Yes
No
4. State has procedures to verify that local reports accurately reflect data collected (e.g.,
through review of local program documentation, onsite auditing).
Yes
No
4a. If yes, describe the report verification process.

Staff Development
Acceptable Quality
1. Local programs and state staff have been provided training on general NRS requirements,
including assessment policy and procedures, and follow-up policies.
Yes
No
1a. If yes, briefly describe when the most recent training occurred, its duration and about
what percent of local providers attended.
2. Local staff has received training on data collection procedures.
Yes

No

3. State and local staff have been trained on data entry into the state or local MIS
Yes
No
4. Local staff has had training on how to produce and/or interpret reports produced by the
MIS.
Yes
No
5. Training has been provided on conducting follow-up survey or data matching procedures,
to state or local staff involved in survey or matching.
Yes
No

6. State has trained staff on distance education policy and use of proxy hours, if estimated.
Yes
No
7. State provides at least one additional training annually to local programs on NRS issues,
MIS data entry or data analysis issues.
Yes
No
7a. If yes, briefly describe when the most recent additional training occurred, its duration
and about what percent of local providers attended. This training should not be the same
as the one described above in item number 1.

Superior Quality
1. There is planned, continuous training (at least one training annually) on data collection
and NRS issues.
Yes
No
1a. If yes, briefly describe frequency, duration and content of trainings.

2. NRS training is planned and delivered based on needs of local staff and evaluations of
previous trainings.
Yes
No
2a. If yes, briefly describe your needs assessment process.

3. State has ongoing technical support to local programs to improve data matching and/or
survey follow-up procedures, such as collecting the data.
Yes
No
3a. If yes, describe support and how it is provided.

Exemplary Quality
1. State has developed and is implementing a plan for ongoing staff development on NRS
and data use issues to promote continuous improvement.
Yes
No
1a. If yes, briefly describe the plan.
2. State has a system for continuous training of local staff on NRS issues, data collection
and data reporting through regularly scheduled training sessions or other resources.
Yes
No
3. State has timely intervention strategies to identify data problems as they occur and to
provide training to programs to correct the problems.
Yes
No

3a. If yes, briefly describe the process.

A Project of the U.S. Department of Education

DATA QUALITY IMPROVEMENT PLAN
The state must submit a quality improvement plan for each content area that does not meet all
of the standards within the superior level. A separate plan must be completed for each content area.
The plans should not exceed one page and include the following information.

1. Content area (e.g., Data Foundation and Structure, Staff Development) and specific
standard(s) not met.

2. For each standard not met, describe your planned approach to implementing changes that
will allow you to meet the standard.

3. Describe the barriers or problems you anticipate, if any, to implement these plans.

4. Describe any technical assistance you might need to implement these planned changes.

5. If you believe you will be unable to meet any standard please explain why.

A Project of the U.S. Department of Education

NRS DATA QUALITY CHECKLIST CERTIFICATION
Note: The state director of adult education or head of the state administrative agency in which
the federal adult education program resides must sign this certification.

CERTIFICATION
I certify that to the best of my knowledge, the information contained in this document is true and
correct and accurately reflects the state’s data collection policies and procedures for collecting and
reporting data for the U.S. Department of Education’s National Reporting System for adult
education.

Signature

Name and Title

Date

Seal