|
Expiration Date: 05-31-2008
DEPARTMENT OF HEALTH AND HUMAN SERVICES ADMINISTRATION FOR CHILDREN AND FAMILIES
Statewide Automated Child Welfare Information Systems SACWIS ASSESSMENT REVIEW GUIDE
2008 - Version March-31-2008
Prepared by: Division of State Systems Children’s Bureau Administration on Children, Youth and Families Administration for Children and Families
|
TABLE OF CONTENTS
1 INTRODUCTION 1-1
1.1 PURPOSE 1-1
1.2 BACKGROUND 1-1
1.3 REVIEWS 1-2
1.4 AUTHORITY AND REFERENCES 1-3
1.5 ASSESSMENT REVIEW PROCESS SUMMARY 1-4
2 SAR PROCESS 2-1
2.1 OVERVIEW OF THE SAR PROCESS 2-1
2.2 REVIEW INITIATION PHASE 2-1
2.2.1 Request Review and Submit Documentation 2-1
2.2.2 Federal Response 2-3
2.3 ASSESSMENT REVIEW PLANNING PHASE 2-4
2.3.1 Review Documents 2-4
2.3.2 Plan Review and Draft Agenda 2-4
2.3.3 Finalize Arrangements with State 2-4
2.4 ON-SITE REVIEW PHASE 2-5
2.4.1 Entrance Conference and Detailed System Demonstration 2-5
2.4.2 Local and State Office Visits and Interviews 2-5
2.4.3 Document Review Findings 2-6
2.4.4 Conduct Exit Conference 2-8
2.5 POST-REVIEW PHASE 2-8
2.5.1 Finalize Findings and Conclusions 2-8
2.5.2 Prepare Final Report 2-9
2.5.3 Post-Report Activities 2-10
3 INSTRUCTIONS 3-1
3.1 OVERVIEW 3-1
3.2 STATE COMPLETION OF THE GUIDE 3-1
3.3 EXAMPLE FOR COMPLETING THE PROCESS SECTION OF THE GUIDE 3-6
3.4 ACF COMMENTS 3-10
3.5 STATE RESPONSE TO ACF COMMENTS 3-11
3.6 LESSONS LEARNED FROM PREVIOUS REVIEWS 3-11
4 APPENDIX A: GUIDE - OVERVIEW SECTION A-1
5 APPENDIX B: GUIDE - PROCESS SECTION B-1
6 APPENDIX C: GUIDE – NEW INITIATIVES SECTION C-1
7 APPENDIX D: Sample Interview Questions D-1
8 APPENDIX E: Sample SAR Agenda E-1
LIST OF EXHIBITS
Exhibit 2-1 SACWIS Assessment Review Methodology……………………….………. 1
Exhibit 2-2 Finding Summary Worksheet……………………………………………….. 9
NOTICES
THE PAPERWORK REDUCTION ACT OF 1995
Public reporting burden for this collection of information is estimated to average 160 hours per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information.
An agency may not conduct or sponsor and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.
ALTERNATIVE SUBMISSIONS
States MAY provide the information requested in the SACWIS Assessment Review Guide in an alternative form to the extent that the State already has it available.
PLEASE NOTE – Before committing resources to write or develop supporting documents requested as part of this effort, contact ACF to discuss. It is not the intent of ACF to have States develop the supporting documents submitted along with the SACWIS Assessment Review Guide. ACF believes that most of the requested supporting documents will have been developed as part of the overall system development project. ACF has attempted to reduce the overall burden on the States by relying on existing documentation.
The purpose of this document is to provide guidance to Federal and State personnel on conducting Statewide Automated Child Welfare Information System (SACWIS) Assessment Reviews.
The Department of Health and Human Services' (DHHS) Administration for Children and Families (ACF) provides national leadership and direction in planning, managing, and coordinating the administration and financing of a broad range of comprehensive and supportive programs for vulnerable children and families. These programs are, in large part, administered by public and private State and local agencies and are designed to promote stability, economic security, responsibility, and self-sufficiency.
For many years, concerns have been raised about the lack of information available on children in foster care and their families. To address some of these concerns, Congress amended title IV‑E of the Social Security Act in 1986 by adding section 479 which requires the Federal government to institute a foster care and adoption data collection system. Federal regulations at 45 CFR 1355.40 (Code of Federal Regulations) set forth the requirements for the Adoption and Foster Care Analysis and Reporting System (AFCARS).
The Omnibus Budget Reconciliation Act of 1993 (OBRA 93) provided enhanced Federal financial participation (FFP) at the 75 percent rate (beginning October 1, 1993) for development of SACWIS systems to carry out the States' programs under title IV-E of the Social Security Act. In response, requirements for SACWIS were implemented under regulations at 45 CFR 1355.50. The Personal Responsibility and Work Opportunity Reconciliation Act of 1996 extended the period of eligibility for the enhanced rate to September 30, 1997. Under these laws, enhanced funding was made available for the planning, design, development, and installation of statewide systems that meet the following standards.
The systems must meet the requirements imposed by regulations promulgated pursuant to section 479(b)(2) (AFCARS) of the Social Security Act.
The systems must, to the extent practicable, interface with the State data collection system that collects information relating to child abuse and neglect.
The systems must, to the extent practicable, interface with, and retrieve information from, the State data collection system that collects information relating to the eligibility of individuals under the title IV-A program.
The systems must be determined by the Secretary to improve program management and be likely to provide more efficient, economical, and effective administration of the programs carried out under a State's plans approved under title IV-B or IV-E of the Act.
States are encouraged to develop SACWIS systems. States electing to develop such systems with Federal financial participation (FFP) come under the existing Federal review and approval processes, initiated and updated by Advance Planning Documents (APDs) submitted to ACF. ACF has established the following types of reviews.
MONITORING REVIEWS: 45 CFR 95.621 requires that ACF continually review, assess, and inspect the planning, design, and operation of SACWIS systems to determine how such systems meet and continue to meet the requirements imposed in the law, regulations, and guidelines.
These reviews, conducted on an as-needed basis, assess the State's progress in developing the comprehensive statewide system described in the approved APD. During planning, development, and installation, these reviews are generally concerned with the overall progress, work performance, expenditure reports, system deliverables, security, and supporting documentation. ACF assesses the State's overall conformance with the approved APD and provides technical assistance and information sharing from other State projects.
ASSESSMENT REVIEWS: There are no specific certification requirements for SACWIS. Once a system is operational, ACF will conduct and report the results of a SACWIS Assessment Review (SAR). The purpose of these reviews is to ensure that all aspects of the project, as described in the approved APD, have been adequately completed, and conform to applicable regulations and policies. Either the State or ACF may initiate these reviews. However, ACF reserves the right to initiate a SACWIS Assessment Review at anytime in the life cycle of a system.
A SACWIS Assessment Review (SAR) is based on the requirements of law, implementing regulations, the SACWIS Action Transmittals (AT) and Program Instructions (PI), the State's approved APD, State contract documents, and any additional policy guidance or conditions provided to the State. Every effort will be made to reach mutually acceptable solutions to any identified issue. However, if this is not possible, the State may be subject to the remedies described at CFR 45 1355.56.
The timing and manner in which SACWIS reviews are conducted is based on available staff and resources; not all SARs will be conducted on-site nor will they necessarily be conducted at the time the system becomes operational. Therefore, States that use contractor assistance in the development of their system should not link final acceptance or payment to an ACF review. Instead, States are encouraged to base contractor payments on task-specific deliverables and system acceptance on demonstrations and system acceptance tests.
In preparation for a SAR, every effort will be made by ACF (considering resource availability) to conduct a technical assistance consultation with the State. This discussion should occur during or immediately after pilot implementation. The State and ACF will use this technical assistance consultation as an opportunity to estimate system conformance with SACWIS requirements, specify a time frame for resolution of obvious (i.e., highly visible) issues (if needed) and discuss when the SAR should be conducted.
Generally, ACF will not conduct the Assessment Review until, at a minimum, the following conditions are met:
30 percent of the total foster care and adoption caseloads (State and Federal) have been converted to, and are maintained under the fully functional SACWIS system; and
One (1) or more county/district offices are fully operational.
States should note that they MAY provide the information requested in the SACWIS Assessment Review Guide in an alternative form to the extent that the State already has it available.
AFCARS REVIEWS: The purpose of the AFCARS review is to assess States’ information system’s capability to accurately collect, extract and transmit the AFCARS data to ACF. The AFCARS review also assesses the State child welfare staff’s ability to accurately collect and document the AFCARS information related to the foster care and/or adoption case of a child. The review process goes beyond the edit checks that must be met in order to pass the AFCARS compliance error standards. The review focuses on ascertaining a State's degree of compliance with all of the AFCARS requirements and the quality of its data. Therefore, AFCARS reviews have a separate and distinct purpose from SACWIS reviews and may be conducted before, during, or after a SAR.
Title IV-E SARs are conducted under the authority of 45 CFR 1355.55, which requires that DHHS evaluate SACWIS system projects.
The following regulations and guidelines apply to the SAR process or are of related interest:
Conditions for approval of funding in 45 CFR 1355.53;
Action Transmittal (AT), ACF-OISM-001, Issued February –24, 1995 (referred to as the "SACWIS AT 001");
Action Transmittal (AT), ACF-OSS-05, Issued August 21, 1998;
AFCARS Regulations (45 CFR 1355.40), Policy Information Questions, Technical Bulletins and other policy guidance;
Office of Management and Budget (OMB) Circular A-87, Cost Principles for State and Local Governments;
Automated data processing services and acquisition procedures and requirements of 45 CFR Part 95, Subpart F;
ACF's Information Systems Review Guide, Cost/Benefit Guide, and Companion Guide 2: Cost/Benefit Analysis Illustrated for Child Welfare Systems;
SACWIS Conceptional Design, SACWIS Data Element Dictionary, SACWIS Integrated Data Set, and SACWIS Functional Requirements;
NCANDS Detailed Case Data Component Guidelines and Procedures;
ACF's review guides for programs that interface with SACWIS systems (e.g., the Automated Systems for Child Support Enforcement: A Guide for States); and
Federal statues and regulations applicable to the programs administered through the State’s SACWIS.
To obtain relevant policy documents, interested parties may access the information at the Children's Bureau’s web page at
http://www.acf.dhhs.gov/programs/cb
Information about SACWIS, AFCARS, and NCANDS is available at
http://www.acf.dhhs.gov/programs/cb/dis
To obtain other relevant SACWIS related Action Transmittals, interested parties may contact the Division of State Systems in ACF's Children’s Bureau or visit the Division of State Systems' Homepage at
http://www.acf.dhhs.gov/programs/oss/
This Guide describes the SAR process and provides guidance and examples on completing the questions in appendices A, B, and C. A major component of the SACWIS review process is the self-assessment conducted by the State. As a result of its self-assessment, the State will be in a position to describe fully and accurately its system through its responses in the Guide. The Guide can be broken out by section or even by question, so that the responsibility for completing the Guide may be separated into smaller components and sent to the appropriate staff for completion.
The Guide sets forth the specific areas of the review and is divided into three parts, which can be found in Appendices A, B, & C. The three parts of the Guide are the Overview, Process, and New Initiatives Sections.
The Overview Section describes the general characteristics of the SACWIS system, including the objectives, applications, and architecture. It provides general identifying information that will help other States and ACF understand the overall size, cost, and programmatic functionality of the system.
The Process Section describes the SACWIS system's conformance with mandatory program requirements derived from statute and regulation. The questions in this section are derived from the SACWIS Action Transmittal (AT) No. ACF-OISM-001, dated February 24, 1995. The Section numbering conforms to the AT. Sequential numbering has been added to the Guide for ease of use. ACF relies on the sequential numbering to reference areas of the document in discussions with the State.
The New Initiatives Section describes how the system supports new Federal and State initiatives. SACWIS systems are expected to be comprehensive child welfare case management tools and need to be enhanced as program requirements change.
The review is normally initiated when a State submits an electronic copy of the Guide and supporting documentation to the ACF Central and Regional Offices. An electronic version of the Guide serves as the means of documenting the review, providing the State with the Federal review team's findings and producing a final report. ACF is able to provide an electronic copy of the original document in the following formats: Microsoft (MS) Word, or as an electronic rich text file (*.rtf). ACF prefers to receive the completed document as an MS Word file. If this format is not available, the document may be returned to ACF as an electronic rich text file.
After the documentation is evaluated, ACF will conduct the review either on-site (at the State offices) or off-site (by telephone conference calls). The State completed Guide is used by ACF to record comments, issues, resolution dates and negotiated action plans. Based on these comments, the State may update the Guide before finalization of the report.
The SAR Process is described in Chapter 2, while Chapter 3 provides instructions for completing the different parts of the Guide. Additional guidance may be requested from ACF's Central and Regional Offices, ACF systems staff will be available to work with the State through the entire review process.
The process of conducting an on-site SACWIS Assessment Review (SAR) involves four phases, illustrated in Exhibit 2-1 below. These reviews will consist of:
a Review Initiation Phase;
an Assessment Review Planning Phase;
an On-Site Review Phase; and
a Post-Review Phase.
SAR METHODOLOGY
REVIEW INITIATION |
ASSESSMENT REVIEW PLANNING PHASE |
ON-SITE REVIEW PHASE |
POST-REVIEW PHASE |
|
|
|
|
Depending on resource availability, reviews may be conducted off-site instead of on-site. With the obvious exception of the site visit, the process is essentially the same. For an off-site review, the depth of the Document Review Phase will be expanded and supplemented with telephone interviews with appropriate State staff.
The Review Initiation Phase involves two steps:
SARs should take place after the SACWIS system becomes operational (see section 1.3 for guidance on requesting a SAR before completing statewide implementation). These reviews are to be initiated approximately six months after the approved project completion date. If the State is initiating the review process, it should submit a written request to the:
Associate Commissioner
Children’s Bureau
Attention - Director, Division of State
Systems
Administration for Children and Families
Department
of Health and Human Services
330 "C" Street,
S.W.
Washington, D.C. 20201
The written request must provide assurances that all operational components of the SACWIS system are used (i.e., that staff are not using another information system or paper processes to perform the work of a SACWIS component), operates uniformly statewide (or in the converted jurisdictions if the Assessment is being conducted prior to statewide implementation) and meets all mandatory requirements.
The request should be accompanied by the following documentation in hard and, if available, soft copy (please contact ACF if these documents are not already available – States should not create these documents for the sole purpose of this review):
System diagrams (both the technical architecture and elementary processes performed by the system include functional decomposition, data flow, and hardware configuration diagrams if available);
An Organizational Chart (inclusive of all SACWIS personnel);
Current user manual (for both technical and end users);
Training manual and materials;
Data element dictionary (with table definitions, data element data types, mandatory vs. optional elements, and table and key constraints)
Numbered list of system screen prints;
Numbered list of system alerts/ticklers (with a brief description of the alert/tickler including how the tickler is invoked, resolved and a general description of the tickler process);
Numbered list of system notices(with a brief description of the notice including how the notice is invoked, resolved, and a general description of the notification process);
Numbered list of reports (including management and financial reports and reference to whether reports are on-line, available in hardcopy, and generated in real-time or by batch);
Planned schedule for submitting NCANDS Detailed Case Data Component (DCDC) data (If State's Child Abuse & Neglect system is included within the SACWIS);
Copies of SACWIS reports used to complete the Federal IV-E 1 report(formerly IV-E 12); and
Other cross-reference material the State may wish to provide.
The Guide must be submitted to ACF in an electronic format. If possible, ACF prefers to receive the files in MS Word. If that format is not available, the files should be submitted as an electronic rich text format file (*.rtf).
In addition, if the State has received approval to link a "SACWIS type system1 existing in a large urban area" to the new SACWIS system, the "existing SACWIS type" system must be identified, the linkages described, and the following assurances made.
The staff using either SACWIS type systems (existing or new) must be able to easily check all SACWIS related systems for prior incidents and other available information.
Either system is able to generate management alerts if an individual is active in more than one of the SACWIS type systems.
A copy of all documents should be sent to the ACF Regional Administrator.
SARs may or may not be scheduled for the same time as AFCARS Reviews. Beyond the condition that the system must be operational statewide, scheduling depends on ACF and State staff time and resources. For additional information on AFCARS reviews, see the "State Guide to an AFCARS Review. This can be accessed from the Children's Bureau's web page.
If ACF decides to initiate a formal SAR, the State will be notified in writing of the need to conduct the review and the reasons for this decision. The failure to cooperate with an ACF-initiated SAR could result in suspension of the project's approval and the disallowance of applicable title IV-E funds.
The Federal review team and leader will be selected. The SAR team is composed of Federal (Regional and Central Office) staff that have technical and program expertise, as well as a general knowledge of the State under review.
The Federal and State teams will work together to schedule the review.
The Assessment Review Planning Phase involves the following three steps:
The Federal team members will analyze the documents submitted by the State. Given schedule and resource availability, States should plan to submit the review documents well in advance of the review to ensure that all parties have sufficient time to either prepare or review the SAR documentation.
The preliminary evaluation of the State documentation is intended to prepare the Federal team, expedite the review and evaluation process, determine if there are any impediments to the review, identify those areas of the system that require additional evaluation, and document early findings. This will maximize the effectiveness of the on-site review.
The Federal staff will examine the State’s APD and contract documents to establish the scope of the project.
The Federal team, in consultation with the State, will plan review activities. The State team will be responsible for developing an agenda based upon the agreed upon activities. Activities will include all proposed county and local office visits, computer center visits, and interviews/demonstrations to review all functional areas. These Federal/State discussions will focus on: who the team must interview; where the review will take place (e.g., on- or off-site); and reach agreement on the tasks, responsibilities, milestones, deliverables, and schedules. The purpose of this phase is to clarify review expectations, answer State questions, and advise the State of preliminary findings resulting from the document review.
Appendix E contains a sample on-site SAR agenda. The schedule lists days, time estimates, activities, and recommended State participants. The State will identify the participants for each activity and schedule interviews with key State, county, and local officials before the visit. Interviews with local office staff (e.g., social worker and supervisors) should not be scheduled in advance. These interviews will be conducted at workers’ desks, as they are available.
After the agenda is defined, the Federal/State teams will conduct at least one conference call prior to the on-site review to finalize schedules, clarify expectations, organize the interviews, confirm sites for the local office visits, and the availability of key State staff. At the discretion of the Federal team leader, the State may be asked to provide meeting space, if available, for team members during the on-site review.
Once the details are finalized, the Federal team leader will conduct a conference call that will:
provide information on how the review will be conducted, addressing entrance and exit conferences, names of Federal team members, scheduling, etc.;
discuss the on-site agenda, including the local sites to be visited; and
identify the review techniques that will be employed, such as interviews, examination of outputs, case sampling, review of general production outputs, viewing screens and operations, collateral verifications, etc.
Appendix A of ACF's Information Systems Review Guide (see resource list in Chapter 1 of this document) contains a generic "Sample Entrance Letter to State" on page A-5.
The State should disseminate information on the nature and scope of the review to prepare program and data-processing staff that will either participate in, or be interviewed during the on-site review.
The on-site review phase is usually conducted over a four to five-day period and includes the following:
The first day of the on-site visit begins with the entrance conference which serves to:
introduce the Federal team to the State management and project staff;
establish contacts for consultation later in the on-site review;
advise the State of the nature and scope of the on-site review; and
respond to any questions the State staff might have related to the review process.
Immediately following the entrance conference, the State presents a detailed functional walk-through of the system and an overview of actual and projected program improvements resulting from the system implementation. The demonstration focuses on how the system meets all of the SACWIS functional requirements and references the State's responses in the completed Guide. The State should ensure that appropriate State technical and programmatic staff participate in the demonstration. We would expect that State staff would lead the actual walk-through of the system, although contractor staff can be available during the walk through.
The purpose of the local and State office visits and interviews is to confirm that the system functions in the field as it was described in the State's APD, Guide responses, and functional walk-through. In that light, it is the Federal review team's intent to interview at least one local or State office staff member in each of the different functional positions represented in those offices. Site reviews should include visits to at least two counties or local offices of varying size and complexity as well as State offices. In order to complete all the tasks identified on the agenda, the Federal team will divide into smaller groups.
The Federal team will use a variety of techniques to assess the functional conformance of the operational system. The team will:
interview office managers and staff (including staff involved in referral/intake, investigation, assessment, resource development, out-of-home placement, family preservation, adoption, claims, contracts, etc.) about how they use the system, what works well and what needs to be improved;
ask the office staff to share their “lessons learned”;
observe actual case information being entered into the system; and
ask follow-up questions based on the information gathered during the review of the State documentation and on-site functional walk-through.
While State project staff should accompany the Federal team to the office site visits, they should not participate in the actual interviews. The State project staff person that accompanies the Federal team to the local and State offices should be available to answer questions that may arise during the interviews. The importance of allowing field office personnel to share their experiences will be re-confirmed at the entrance conference.
The Federal team will work with the State to identify the specific offices to be visited (no fewer than two and no more than ten). Each local office visit should include at least one hour of observing workers entering case data into the system. The Federal team will identify the types of system users that they would like to interview.
The local office visit will allow the Federal team to talk to the SACWIS system users about how the system supports the flow of work. These discussions cover the SACWIS functions supported by the system, but less formally than was done in the Guide and the supporting documentation.
Daily Meetings.
As necessary, the full Federal team will reconvene at the end of each day to review the day's findings, summarize results, complete documentation, and plan the next day's schedule. The purpose of these daily meetings is to:
Prepare a one to two page overview of the initial findings to present to the State;
Discuss issues; and
Refine or refocus the on-site process because of the day's findings.
If part of the Federal team cannot return to the central site for the daily team conference, the meeting should take place by teleconference. The full team should be present the night before an exit conference to finalize findings.
The State's completed Guide is used during the on-site inspection to direct the review process. ACF will not change a State response, which the State can update later. Federal team members may record notes on their own copy of the Guide.
The Federal team leader ensures that appropriate documentation is collected and recorded in the Guide. The Guide provides the primary documentation of the review and its findings. All supporting documentation collected and prepared during the preliminary evaluation, on-site inspection, and final evaluation is included in the permanent State SACWIS review file, which is retained by ACF.
The Federal team records the results of the functional review on the ACF portion of the Guide. For each mandatory functional requirement and each funded optional requirement, the team notes whether the system conforms, conditionally conforms, or does not conform. If the system has the required functionality, no further supporting documentation is required. However, if the system does not have full functionality (conditionally conforms) or lacks functionality (does not conform), the team will note the basis for the finding and record in the “comment” section of the report any actions the State must take or information the State must provide. A draft report will be provided to the State after the appropriate ACF managers have reviewed it.
The completed Guide will be initially issued to the State as a draft report. While it is anticipated that the State's responses and ACF's comments in the Guide will evolve after the draft report is issued, the final version will reflect the identified issues, planned or completed actions, and the corresponding resolution dates.
c. Finding Summary Worksheet
The Finding Summary Worksheet (see Exhibit 2-1 on page1) may be completed if there are significant deficiencies that must be corrected in order for the system to meet the requirements established in the SACWIS AT or the State's approved APD. Use of the Finding Summary Worksheet will only be necessary when State and Federal staff cannot reach agreement on a specific finding. The Finding Summary Worksheet will be prepared after the on-site review is completed and the draft report has been given to the State.
The Finding Summary Worksheet is founded on the performance-based review methodology commonly used by evaluators for the examination of facts. The methodology requires that the following information be obtained and evaluated to determine whether the finding is valid and corrective action is needed:
Criteria: What should be;
Condition: What is;
Gap: Difference between "what should be" and "what is;"
Effect: Significance;
Cause: Why; and
Recommendations: How to make the condition equal the criteria.
Before the Exit Conference, the Federal team will summarizes its findings, consolidate the documentation, notify Central and Regional Office management of the preliminary review results, and prepare for the last day of meetings. The purpose of the exit conference is to:
present specific initial, but not all, findings resulting from the review process;
report the preliminary recommendations and/or proposed resolutions to outstanding issues; and
answer any questions that the State might have as a result of the review.
The Post-Review Phase involves three steps:
Although the major portion of the evaluation of the SACWIS system will be conducted on-site, a number of steps in the assessment and documentation process may be completed off site. These steps include:
consolidation of findings from all team members;
final consultation among Federal team members;
organization of the review record;
recording ACF's findings, requirements, recommendations, and/or observations in its portion of the Guide; and
completing any collateral verifications from other sources such as State plans, NCANDS and AFCARS reports/reviews, State Cost Allocation plans, etc.
All documentation or verifications from collateral sources are retained by ACF as a part of the archive file on the State's SACWIS project.
an overview of the SACWIS system;
a summary of the functional components of this system that work particularly well;
a summary of the deficiencies and recommendations for improvements; and
the completed Guide.
Once the Guide (and any Finding Summary Worksheets) is complete, the Federal team will draft a brief overview of the SACWIS system. The overview will describe the findings and identify any actions the State needs to complete. The overview will be included in the cover letter sent to the State with the completed Guide.
The completion of the draft SAR report is based on available resources, however, ACF will attempt to have the report drafted and approved within 120 days after the on-site review ends. The reports are unlike the "typical" audit or management review report which includes separate sections on background, purpose and scope of the review, review methodology and so forth. Instead, this Guide establishes background, purpose, scope, and methodology for all SACWIS reviews.
The purpose of the report is to support a decision by ACF that the SACWIS system has met the tests of efficient, effective, and economical operation and qualifies for the SACWIS enhanced funding and cost allocation methodology.
All Federal team members will contribute to the final draft report. The draft report requires review and approval prior to issuance by the:
Associate Commissioner, Children’s Bureau
Director, Division of State Systems.
Once approved, the draft report will be sent to the State. The State has an additional 160 days to respond to the draft report. The process will continue until all issues have been resolved and/or corrective action plans approved. The final report will be issued within 120 days of receiving the State comments and ACF approval of the response.
Although the final report is prepared primarily for ACF, the State under review, and other States for transfer evaluation, it is public information and will be prepared in such a manner that it may be shared freely.
As noted at 45 CFR Part 95.605, cost-benefit and other program-performance improvements must be reported through the Annual APD Update until the Department determines that the projected benefits have been achieved. More rigorous follow-up is required for systems that do not meet SACWIS functional requirements. States will be required to report the results of their approved action plan(s) as part of their Annual APD.
ACF may also take the following steps:
For functionally deficient systems, ACF may request an APD Update that presents a workplan for any additional development necessary to meet the required functionality. In addition, periodic monitoring reports may be required if development is extended.
To assist the State in eliminating reported deficiencies, ACF staff may regularly contact the State about its progress towards achieving compliance in each of the deficient functional components. If corrective actions are not undertaken in a timely way, ACF may limit funding or recoup previously approved funds.
ACF may require revisions to the State's past claims for FFP or the SACWIS cost allocation methodology to reflect benefits other programs derived from the system's use, or to adjust funding for functionality that exceeds the SACWIS requirements, but was never approved by ACF in an APD.
Before initiating a negative financial action against a State, ACF will work diligently with the State to come to a mutual agreement on the findings. If necessary, ACF will work with the State on action plan(s) to resolve any outstanding issues. It is ACF's desire to support innovative State solutions and program successes that ensure that program needs are met.
FINDING SUMMARY WORKSHEET
FINDING: |
[Condition: What is.] |
BASIS FOR FINDING: |
[Criteria: What should be.] [Gap: Difference between what should be and what is.] |
CORRECTIVE ACTION: |
[Description of any corrective action underway or planned.] |
CRITICALITY: |
[Effect: Significance.] |
CONCLUSION: |
[Cause: Why?] [Assessment as to whether further action is required. Reflects conclusion based on "why," "so what?" and current or planned corrective action. Note that conclusion may be that a recommendation need not be developed.] |
RECOMMENDATION: |
[Recommendation: How to make the condition equal the criteria.] |
This chapter provides instructions to complete the SACWIS Overview, Process, and New Initiatives Sections of the Guide (Appendices A, B, and C). As previously indicated in chapter 2.2.1, the State must send the completed SACWIS Assessment Review Guide to ACF. By organizing the review material as a set of questions supported by citations and narrative explanation, the processes of documenting and creating a "report" are combined in a single electronic document. This should greatly reduce the time required of State and Federal reviewers.
The Guide is divided into three segments:
Appendix A - an Overview Section that describes characteristics of the SACWIS system, including the objectives, applications, and architecture;
Appendix B - a Process Section that describes the system's conformance with mandatory program requirements derived from statute and regulation; and
Appendix C - a New Initiatives Section that describes how the system supports new State and Federal initiatives and demonstration waivers.
States MAY provide the information requested in the SACWIS Assessment Review Guide in an alternative form to the extent that the State already has it available.
Appendix A: Overview Section
This part of the SACWIS Assessment Review Guide provides an overview of the SACWIS system, including the objectives, applications, and architecture. The following section provides guidance and describes the purpose of requesting the information.
All State responses should be typed in the text boxes provided in the instrument.
A. I: covers general identifying information as well as information that will help other States understand the overall size, cost, and programmatic functionality of the system.
Questions 3 and 4 are included for the following reasons:
Question 3. This question on system support staff is primarily intended to help other States assess the suitability of the system for transfer. By understanding the numbers and types of people who operate and maintain the system, other States may be able to make a preliminary assessment about transfer potential. This question is geared to normal support staff, not staff available for major system modifications. If this information is not available because a contractor maintains the system and the information is considered proprietary, please provide the available information and note the exception.
Question 4. The question on the number of system users is also intended to help other States assess suitability for transfer. The State should specify the number of current on-line users.
A. II is for summary technical information. A paragraph or two for each question should be sufficient.
A. III is for identification of each of the contractors that participated in the planning, development, implementation, and operation of the SACWIS system. Please identify contractor by project phase and role within that phase.
A. IV is for financial information. This information should identify all developmental costs (Planning, Design, Development, and Implementation) and should correspond to the information in the approved APD.
A. V is for "lessons learned." Because of the value of this section to other States, ACF strongly encourages States to spend some time completing this section. The section is for information only, and States will not be penalized for the content.
A. VI lists required enclosures to be submitted with the completed Guide. Note that the information provided will be cross-referenced to responses in Appendix B.
The Summary Table located in Appendix B is updated by ACF as the report is finalized and reflects the findings recorded in the questionnaire by the Federal review team. As findings are resolved by the State, this table will display the status of each functional component, whether the component has an Action Plan associated with it, and the page number where the component is discussed. This table will allow the reader to focus on unresolved findings, as well as identify those components with an approved Action Plan, which requires tracking in the State’s Advance Planning Document.
The table makes use of the “bookmark” functionality in Word. The State should not type in this table. As the State completes the questionnaire, the page reference numbers in the table will not match the actual pages in the questionnaire. If the State wishes to update the page number on the Summary Table, the writer may type “Control A” (the “Ctrl” key and the “A” key) and then push the “F9” key. This will refresh the page reference numbers on the table. The State does not need to refresh the page numbers.
The Table will appear as follows when the State initially completes the questionnaire.
ACF Only |
|||
Requirement Number |
Conforms? |
Action Plan? |
Page Reference # |
1 |
|
|
B-# |
2 |
|
|
B-# |
3 |
|
|
B-# |
After ACF completes its part of the draft report, the table will be refreshed and will appear as follows.
ACF Only |
|||
Requirement Number |
Conforms? |
Action Plan? |
Page Reference # |
1 |
Y |
|
B-# |
2 |
N |
|
B-# |
3 |
C |
|
B-# |
Once the report is finalized, the table will appear as follows.
ACF Only |
|||
Requirement Number |
Conforms? |
Action Plan? |
Page Reference # |
1 |
Y |
|
B-# |
2 |
Y |
N |
B-# |
3 |
Y |
Y |
B-# |
This table indicates that SACWIS requirements have been satisfied for the identified requirements. Component number two has been totally resolved, while component number three has an approved Action Plan that will be followed though the State’s Advance Planning Document.
Summary table definitions:
Requirement Number refers to the sequential numbering of each of the SACWIS requirements.
Conforms indicates if the component conforms to the specified SACWIS requirement.
Action Plan indicates if ACF has approved the State’s plans for bringing the specific SACWIS component into compliance at some point in the future. A “Y” in this column indicates that an Action Plan has been provided to ACF and has subsequently been approved. An “N” indicates that a compliance issue was resolved without an Action Plan (e.g., the State clarified existing functionality or completed an enhancement that resolved the identified issue). An “NA” or blank in this column would indicate that the component was found to be compliant at the time of the initial review and an Action Plan was not necessary.
Page Reference # identifies the page number of the report where the finding is discussed. The “#” symbol represents an integer. Page reference numbers, as do all “bookmarked” items, must be periodically refreshed. ACF is responsible for this task.
Appendix B: Process Section
The Process Section is intended to establish the conformance of the State's SACWIS system to the functional requirements established in the SACWIS Action Transmittals, Program Instructions, program regulations and State contract documentation. The section numbering corresponds to the sections in the February 24, 1995 SACWIS Action Transmittal (number ACF-OISM-001). The sequential numbering is used for ease of discussion. Note that the functional requirements are stated as open-ended questions. In response to each, the State will describe (in the State Response section) how the functionality is met.
In the State Response section, the State will provide a brief, but complete narrative description of how the SACWIS supports the required and optional functional components approved in the APD. Each response should include a self-assessment describing the extent to which the State's SACWIS system fulfills the functionality. If the State determines that a requirement is incomplete, the State should describe:
the differences or "gaps" between what is required and what has been implemented;
the reasons for the gaps;
actual or potential effect on the State's child welfare program; and
planned actions for closing the gaps to meet all requirements.
Required functional components are identified with an asterisk (*) after the title. Optional functional components are followed by a question that asks if the State elected to include the functionality in the SACWIS system ("Was this Function selected as an Option in the State’s Approved APD? Yes or No"). The State indicates whether the optional functionality was a "State Selected Option" by checking the appropriate box. Optional functional components, for which the State was approved funding, become mandatory with APD approval.
The Guide uses a table to cross-reference the system functionality to the documentation provided by the State in response to Appendix A - VI: Enclosures. The cross-reference tables are completed for each of the functional sub-components (e.g., Intake & Screening).
The following Cross-Reference table is used for all functional sub-components:
Cross Reference Box for the “XXXX” Section |
|||
Reference Sources (Document/Chapter): |
|||
Screen Identifier |
Alert Identifier |
Notice Identifier |
Report Identifier |
|
|
|
|
Cross-reference table definitions:
The Reference Sources is used to list the titles and page numbers of the enclosed documents (such as a User Manual) that provide supporting evidence or additional detail on how the system meets the functional requirements.
The Screen Identifier is used to identify the screen(s) utilized to meet the functional requirement. This can either be cross-referenced by number to a "screen print" included as an attachment or to another resource (such as a User Manual) provided by the State that identifies the use of the screen(s).
The Alert Identifier, Notice Identifier, and Report Identifier are for identifying (as applicable) the alerts/ticklers, notices, or reports that support a functional sub-component. The response should be cross-referenced by number (such as Alert No. 1) to the lists of alerts, notices, and reports provided by the State (Appendix A. VII: Enclosures).
The term "Alert" is used in the Review Guide. It refers to an automated or worker generated tickler (reminder) of the need to complete an important task.
A Notice can refer to system-generated correspondence, task lists, reports or word processing templates that are integrated into the SACWIS application.
Appendix C: New Initiatives Section
The New Initiatives Section examines how the State's SACWIS system supports new Federal and State initiatives in the Child Welfare program. The State should describe how its SACWIS supports the program areas identified in Appendix C. If the system does not currently support one or more of the identified program initiatives, the State should describe its plans for modifying the system to support the applicable initiatives. Whether or not the system supports a new initiative, the response should describe the processes used by the State to satisfy the requirement.
The following is provided as an example of how a State should complete the Process Section of the Guide. The information presented in this example is for illustration only. The State's narrative (State Response) should be brief, but complete, and rely on the Documentation References for the supporting details. The Documentation References should correspond to the numbering sequence of the Reference Lists provided by the State at the time the Guide was submitted to ACF (see section 2.2.1).
Appendixes A, B, and C are WORD templates that will enable you to complete them on your computer screen. The answer fields will expand to accommodate the State response.
The “answer fields” appear as rectangles . Please contact ACF staff if you have questions about using the form.
As previously described, the State would use the selected word processing program to add text to the appropriate sections. The word wrapping capabilities of the word processing program will expand the document to fit the need.
Example –
A: Intake
Goal: To provide an automated entry point into the child welfare services agency for children and families who seek services, are referred for services, and/or who are reported to the agency.
Cross Reference Box for the INTAKE Section |
|||
Reference Sources (Document/Chapter): User Manual - Chapter 4 & Training Manual - Chapter 11 |
|||
Screen Identifier |
Alert Identifier |
Notice Identifier |
Report Identifier |
Screens 24 through 28 Note to the reader: The numbers used in this example correspond to the list of screen prints that would have been provided with the Guide. |
Alerts 2, 45, & 67 Note to the reader: The numbers used in this example correspond to the list of alerts that would have been provided with the Guide. |
None. Note to the reader: If one or more notices had been generated by this functional sub-component, they would have been listed here. The numbers would have corresponded to the list of notices that would have been provided with the Guide. |
Reports 15, 16, 34 & 37 Note to the reader: The numbers used in this example correspond to the list of reports (on-line and paper) that would have been provided with the Guide. |
1 - Intake Management - Section A: Intake
1. (1) Record contact/referral * — Describe how the automated system
records initial contacts regarding allegations of abuse or neglect, and/or
provides for the input of a formal referral for protective services, voluntary placement services, juvenile corrections, and other services.
The State response should also indicate when the “Intake Report” is frozen in the system.
State Response:
Our SACWIS includes functionality that supports the State's child/abuse and neglect component. The State has a 24-hour Intake Unit located at the CMI Building. That unit is responsible for receiving and processing all public and private referrals and telephone calls regarding allegations of abuse or neglect. 1.a. Initial contacts regarding allegations of abuse or neglect are captured on the system's 5 intake screens. These screens capture basic information about the caller, victim, family, initial contact and allegation. After the initial information is assessed through the system and the finding approved by the supervisor, the system generates a referral in the form of a “task” for the appropriate unit (e.g., Investigations, Voluntary Services, or the Community Service Office). 1.b. Inquires and requests for voluntary services are captured on the 'contact' screen and are handled by the Community Service Office nearest the family. The Community Services Office uses the same service referral functionality as the Continuing Services Unit. The system prompts the social worker to refer the case to the applicable services. All referrals are generated by the system. The system records all referrals in the database. The system supports the referral process to the Investigation Worker after the case has completed the screening process. If the “incident” is screened-in, the system refers the case to the investigation unit and sends an alert to the supervisor and the unit clerk. 1.c. The intake referral is frozen (or locked) at the point the supervisor approves the initial finding. The system allows additional information to be attached to the initial referral as Post-Referral Additions. The summary screen will display the initial referral and any subsequent Post-Referral Additions. Based on State defined parameters, Post-Referral Additions are frozen two days after entry or when the Investigation is completed (which ever comes first).All SACWIS requirements have been met. |
2. (2) Collect intake/referral information * — Describe how the automated system
allows for the input of available situation and demographic information,
including the cross-referencing of relationships among participants and
the reason for referral.
State Response:
2.a. The Intake Unit located at the CMI Building records situational and demographic information on the 5 intake screens. Additional information is collected during assessment and investigation of the allegation or request for services. Information can be recorded in the applicable data fields as a caller provides it, or the system allows the worker to use structured text fields on each of the Intake screens to capture free flowing information. If the text fields are used, the data fields are completed once the call is finished. Where applicable, the system uses pick-lists, checklist, and radio buttons to simplify data entry. 2.c. The Reason for Referral is captured on the initial contact screen. 2.b. To the extent that it is known, individuals are linked to other members of the family during the initial Intake. According to State policy, the system records the relationship of all (extended) family members to the alleged victim. The system allows inquires based on the family or the individual. Furthermore, individuals can be linked to more than one family and family members to more than one victim. Where it is possible, the system calculates reciprocal relationships when the cross-referenced relationship is established (e.g., if Marayna is Caroline’s sister, therefore, Caroline is Marayna’s sister). While we believe that the system meets all SACWIS Requirements, the current process for linking family members to the family unit is cumbersome. We will be examining alternatives to the current process. A budget and schedule will be submitted in the next APD Update. |
4. (4) Record "information only" requests — Describe how the automated system records calls or contacts which do not involve a specific allegation or a referral concerning abuse and/or neglect.
Was this Function selected as an Option in the State’s Approved APD? YES NO X .
If Yes - State Response:
|
Note to the reader: This is an example of how a State would respond to a question about an optional component that was not included in the State's approved APD.
1 - Intake Management - Section B: Screening
2. (6) Record the results of the screening evaluation * — Describe how the automated system provides for the recording of the determination resulting from the screening process.
State Response:
Once the supervisor approves the screening recommendation, the system automatically records the determination on the case status screen. The system’s business rules and the Social Worker’s assessment support the recommendation. All SACWIS requirements have been met. |
Note to the reader: This example is intended to show that some of the responses can be very brief. The purpose of the narrative is to briefly answer the questions and to provide a framework for evaluating the cross-referenced material.
Specific ACF comments will be documented in the space provided after the "State Response" section.
ACF Comments for Requirement: [1] |
|||||
Conforms? Y/C/N |
|
Action Plan? Y/N/Blank |
|
Resolution Date |
|
Finding Summary Worksheet Completed? Yes or Blank |
|
||||
|
|||||
Table definitions:
Conforms? Y/C/N - The Federal team records the results of the functional review on the ACF portion of the Guide. For each mandatory functional requirement and each funded optional requirement, the team notes whether the system conforms (Y), conditionally conforms (C), or does not conform (N). See section 2.4.3 for additional information. This field is “bookmarked” to the Summary table found on page two of Appendix B.
Action Plan? Y/N/Blank – The Federal team will use this area of the report to indicate if it has approved the State’s plans to bring the system into conformance at some point in future (Y). An “N” in this field indicates that the initial issue has already been resolved and that the system is SACWIS compliant in this area (an Action Plan is not necessary). A “blank” field indicates that the component was found SACWIS compliant in the initial assessment. Approved Action Plans are tracked in the Advance Planning Document through their completion. This field is “bookmarked” to the Summary table found on page two of Appendix B.
Finding Summary Worksheet Completed? Yes or Blank - The Finding Summary Worksheet (see Exhibit 2-1 on page 9) should be completed if there are significant deficiencies that must be corrected in order for the system to meet the requirements established in the SACWIS AT or the State's approved APD. Use of the Finding Summary Worksheet ordinarily will only be necessary when State and Federal staff cannot reach agreement on a finding. The Finding Summary Worksheet will be prepared after the on-site review is completed and the draft report has been given to the State. This section is used to indicate that a Finding Summary Worksheet was completed.
Resolution Date - Used to record the date that an issue was resolved or an action plan proposed by the State was accepted by ACF.
ACF Comments - ACF draft and final comments regarding the system.
The State should use the version of the draft SACWIS Assessment Review (SAR) report provided by ACF to respond to the identified concerns. If the State believes that it is necessary to update one of its original responses, it should do so by adding additional text separated by a header-line that indicates the date when the new text was added. The project staff should call the ACF Team Leader if there are any questions about responding to the draft report.
As an example:
State Response:
Original response remains.
MM/DD/YYYY (This date represents the State’s subsequent response. If additional responses were needed, they would also be preceded by a date header.) Additional information is provided to address any concerns and/or to propose an action plan to address the concern. |
Following is a list of “lessons learned” from previous SARs. This information is intended to help States prepare for their review. The guidance is organized into three categories: preparing for the review, completing the Guide, and suggestions for the on-site review.
Preparing for the Review:
Careful preparation for the assessment review will save time, help all parties understand the critical issues for the review, and ensure that all have a common understanding of the purpose and goals of the review.
Ensure that the approved Implementation APD is complete and accurate. ACF uses the approved APD, contract documents, and requirements listed in program regulations, Action Transmittals, and Program Instructions to measure the system’s scope and determine compliance. It is therefore critical that the approved Implementation APD accurately reflects system functionality, including optional functions the State has chosen to implement. ACF recommends that States regularly review the approved APD to ensure it accurately reflects the intended system design and functionality. If the scope of the system's functionality is not reflected in the approved APD, including optional functions originally planned for the system design, submit an APD Update to explain the variations and changes in system functionality. Submit the APD Update before the review.
Communicate frequently with the Federal team leader. Use conference calls, telephone calls, and e-mail to answer documentation questions, understand and define on-site review expectations, finalize the agenda, and complete site visit logistics. Use the opportunity to ask ACF questions posed by staff who will be participating in the review. This preparation will help the State further understand the review scope and help identify critical issues well in advance of the on-site visit.
Keep State staff informed. The project manager should regularly share what is learned from communications with ACF with the management and line staff who will be involved in the review. Clearly define the roles staff will play during the review so they know what to expect and are prepared.
Learn from other States. The project manager should contact other States that have had an assessment review to benefit from their insights. ACF can provide a list of States that have had an assessment review. ACF User Group Meetings also provide opportunities to network with States and hear their perspective on the assessment process.
Understand ACF goals for the assessment review. In general, ACF’s goals for the review are: to determine State compliance with the State APD, contract documents, and federal requirements; assist the State in correcting any deficient areas; identify “best practices” which can be shared with other States’ SACWIS efforts; and identify areas for future technical assistance to States.
Completing the Guide:
Appendices A, B, and C of the Guide and the supporting documentation (please see Section B.1. in Chapter Two of these instructions) supplied by the State are essential to the final assessment review report. These materials are critical to understanding the SACWIS and assessing compliance. If any of the supporting documentation cannot be provided, include a brief explanation why it was not available; do not create new documentation solely for this review. States are encouraged to consult with ACF about supplying the requested information in existing or alternative formats.
When responding to questions in the Guide, the State should consider the following:
Completely answer the entire question that is posed in each section of the Guide. Several of the original requirements were defined by a ten-State workgroup, and have multiple parts; please answer each part of the question.
Ensure responses are complete, clear, and concise. Each answer should stand on its own, without reference to responses in other parts of the Guide.
Fully explain potential or existing system related problems and incomplete functionality. ACF will work with the State to develop action plans to bring the system into compliance. This action plan will be incorporated into the report so it is clear how each problem is addressed.
Cross-reference responses to documentation provided in support of the review. The cross-referenced documentation will enrich ACF’s understanding of the system by supplementing the written response.
Use State and county staff with a detailed knowledge of the system. Such staff will be able to write concrete, precise, and knowledgeable responses and show how the system meets the needs of end users.
Work on the document in Sections, but edit it for overall consistency. As many staff will contribute to the document, the project manager should review if for unnecessary repetition and statements that may appear contradictory and require clarification.
Remember that these documents could eventually become public. Responses should provide sufficient detail to address the questions and be clear to readers lacking detailed knowledge of the SACWIS system.
By following the above guidance, States will obviate the need to revise and re-submit responses.
The State is reminded to submit copies of the documentation to both the Regional and Central ACF Offices.
Suggestions for the On-Site Review:
Focus the on-site system demonstration on functional areas that reflect the processing of a typical child welfare case. It is very helpful for the Federal team to understand how the system is used to support local and State offices business processes, and see it used as it is in day-to-day operations. Clearly outline the steps and processes that will be demonstrated and how they interrelate. As the demonstration proceeds, specify the functional area being demonstrated to the Federal team.
Use State and county staff with a detailed knowledge and experience with the system to demonstrate the system. Experienced staff will convey to the Federal team a sense of how the system is used in the field and can quickly navigate between different interfaces, reports, and ticklers.
The Federal team will ask questions during the demonstration. Although the demonstration is a presentation of the system’s functionality, it is an interactive presentation. The Federal team will ask questions of the presenters throughout the demonstration and refer to the Guide responses and supporting documentation. The questions are to help clarify understanding and do not mean something is “wrong” with the system.
Child welfare programmatic staff and information systems staff should be present during the demonstration. As the demonstration covers the system broadly, questions will arise on a wide variety of child welfare and technical issues such as: State policy, eligibility determination, work flow processes, case management supervision, system security, edit checks, and database design. The presentation should be attended by State staff knowledgeable of the systems technical details as well as representatives familiar with how work is done in the field and how reports and ticklers are used.
The local and State office interviews should not be pre-scheduled; interviews will be conducted at the caseworker’s desk. In order not to impede normal daily operations and to provide maximum flexibility for interviews, the review team will interview local office staff as they are available over the course of the day. Caseworkers will be interviewed at their desks so that the Federal team can observe how they use the system. The State project team member accompanying the Federal review team should help coordinate the interviews by determining when individuals are available for interviews (i.e., who will be in the office at what time).
The local office interviews require logistical coordination by the State. The local office interviews typically take an entire business day and require visits to between two and ten of the various types of offices that use the system. The time and distance involved requires careful advance planning by the State. The selection of offices to visit will be made during pre-visit conference calls.
The Federal team may need to interview private provider’s staff. If the State requires private providers to use the SACWIS, the Federal review team will interview the private providers' staff at one or more of their offices. The State will coordinate the logistics of these interviews.
In summary, SARs are a cooperative process between the State and Federal teams. ACF encourages the State to ask questions and seek clarification on any issues before, during, or after the review.
1See ACF Action Transmittal number ACF-OISM-001 for discussion of alternative approaches to a single statewide system.
| File Type | application/msword |
| File Title | OMB Control No: 0970-0159 |
| Author | Terry Watt |
| Last Modified By | dowens |
| File Modified | 2008-03-04 |
| File Created | 2008-03-04 |