|
This report contains the Content
Unrelated to a Module for the following comments:
2,4,5,6,8,10,11,12,13,16,21,23,24,25,26,28,31,33,34,35,37,38,40,41,43,45,46,51,52,54,55,57,58,59,60,61,62,65,66,67,69,70,71,72,74,75,78,79,80,81,82,83,84,85,86,88,90,91,94,95,96,97,98,99,101,104,105,108,109,110,111,112,113,114,116,118,119,120,121,122,124,125,127,128,129,133,134
|
|
Comment 2 ( Clarity, Impact And
Performance Project Steering Committee; RPIC in ID)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility
* Cost effectiveness of
the new system: The information obtained from this report that
will be meaningful and useful at a local and/or national level
will not justify the effort required to complete the data
collection and submission for the Annual Report. From Region 10’s
response from the first comment period: “Despite the fact
that RNG has essentially taken us from 6 national goals to 2 (with
the civic engagement goal from the infographic apparently being
included under both the individual/family goals and the community
goal and not a separate third goal), the data collection and
reporting have become more complex. If the 6 NPIs and the past
reporting process weren’t getting us data that was reliable,
how does increasing the complexity help? More data doesn’t
equate to more effectiveness, especially given the questions
around the capacity to collect and report data with integrity.”
* Since the close of the first response period,
changes were made to some of the indicators, however, the
complexity remains high. With complexity comes room for a variety
of interpretations. Will CAAs in each state, much less around the
country be able to agree on the meaning of each indicator? A
shared understanding and meaning of the data is required in order
to have any value.
* We understand that social
indicators have been removed, for now, from RNG. These indicators
are used to describe the conditions in local communities and
should be left to the federal, state and local jurisdictions
currently responsible for gathering and maintaining this data and
information.
2. The accuracy of the agency’s
estimate of the burden of the proposed collection of information
* OCS provided the network with an estimate of how
long it will take a CAA to complete this new report. It would be
helpful to know the methodology used to arrive at the estimate to
determine if there is a basis in the reality that mirrors the very
busy, day to day operations of a community action agency.
*
Beyond this estimated time to complete the survey, has any effort
been made to analyze the capacity of the network to report data
with the level of complexity outlined in the proposed reporting
format with any degree of validity?
* Staff
responsible for reporting using the current IS Survey process
estimate that the new format will at least double and probably
triple the time it takes to do a good job with this proposed
process. That increased time comes with no additional offsetting
resources, and will take time and resources away from providing
mission driven services in communities.
* We do not
believe that a majority of the CAAs in our 3 states have the
capacity (staff, financial, technical) to report with this level
of complexity. We are particularly concerned about rural agencies.
For some regions in our states, the CAA is the primary providers
of services to low-income individuals. The burden of these
reporting requirements will significantly reduce the time and
resources to provide services.
* A majority of the
CAAs in our 3 states do not have the capacity to report
unduplicated numbers of people receiving services by domain. Head
Start, energy and weatherization, housing and homeless services
have separate data reporting systems in Oregon, for example. Asset
building has yet another. Service Point is used by Continuum of
Care and VA programs. This issue is likely not unique to the 53
agencies represented by this response.
3. The quality,
utility, and clarity of the information to be collected; and
*
The cart is out in front of the horse. For the data to be valuable
at the national level, and to warrant the time and resources
required, there must be an understanding of clear and shared
definitions. There has not been enough time devoted to developing
this understanding at the local or national level. It would be
impractical and unwise to implement RNG and then attempt to get
the national network in agreement about what the data elements
mean and how data should be collected.
* A common
understanding of each Community Measure will be required if the
data from these measures will provide any valuable information.
How will we as a statewide or regional network of CAAs come to a
common understanding? Will Region 10’s understanding be the
same as other Regions? If not, how will the data gathered be
useful (see bullet #1)?
* Without a shared
understanding, with agreed upon definitions, CAAs will spend days
and weeks collecting data from multiple sources in order to fill
in the blanks and check the boxes. As IT/data staff will tell you,
“garbage in, garbage out.”
As stated in
the response to #2, the quality of data will be undermined by the
sheer volume of systems that the data must come from. If the data
quality is mediocre at best, then the utility and clarity is
compromised.
4. Ways to minimize the burden of the
collection of information on respondents, including through the
use of automated collection techniques or other forms of
information technology.
* It is possible to collect
information from clients of CAAs by using iPads or laptop
computers, located at a reception or work desk. This works best
for clients who have received a very specific service (e.g. energy
assistance, rental assistance, budgeting classes) and are able to
relate the questions to that specific service. At a minimum this
requires:
1. That staff understand the importance of
the survey and gently encourage clients to take it;
2.
That clients are willing to take it;
3. That staff are
available to assist clients who may have a language or reading
barrier, without contaminating the survey process;
4.
That staff are available to monitor the devices;
5.
That staff are available to deal with problems that may arise with
the iPad or laptop;
6. That funds are available to
purchase and maintain the equipment.
* OCS provided
the network with an estimate of how long it will take a CAA to
complete this new report. It would be helpful to know the
methodology used to arrive at the estimate to determine if there
is a basis in the reality that mirrors the very busy, day to day
operations of a community action agency.
Our estimates
for agencies who have under 100 employees and those who have over
100 employees are as follows:
Hours per week: Over 100
Staff Hours per week: Less than 100
Year 1 Year 2 Ongoing
Year 1 Year 2 Ongoing
51 59.5 48.5 40 43.5 36
Hours
per year: Over 100 Staff Hours per year: Less than 100
Year
1 Year 2 Ongoing Year 1 Year 2 Ongoing
2652 3094 2522
2080 2262 1862
* A majority of the CAAs in our 3
states do not have the capacity to report unduplicated numbers of
people receiving services by domain.
* The OMB could
require that the VA, HUD, OCS and the Department of Education
allow crosswalks to be developed so that OMB can look at outcomes
across programs without placing this burden on Community Action.
* Rather than spend time and resources on rolling out
Roma Next Gen, the OMB could require that all government funders
of social service programs collect data on the same indicators (or
at least some of the same indicators).
* To test the
viability of the RNG indicators, OMB might require OCS to start
out with a small sample, with a focus on one area of service (e.g.
housing stability).
|
|
Comment 4 ( Sacramento Employment And
Training Agency (Seta) ; Other/Unknown in CA)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility
Most of the information
collection proposed in the June 2016 Community Services Block
Grant Annual Report, is already collected from 19 CSBG delegate
agencies, internal CSBG program staff, multiple competitive grant
and foundation-funded programs, SETA and delegate Head Start sites
serving over 6,000 children daily, refugee and youth services
throughout Sacramento County, National Emergency Grant activities,
13 Workforce Innovation and Opportunity Act (WIOA) America’s
Job Centers serving nearly 40,000 individuals annually, and the
leveraged resources of regional, public and local entities in the
Greater Sacramento Area. In each of the programs noted above, data
collection is designed to efficiently collect and analyze data
elements requested by each independent funder and to provide
reasonable oversight and accountability, ensuring that the primary
goal of providing community services or resources is achieved.
The desire to overlay a single standardized system of
data collection, compilation, analysis, reporting, and identifying
unduplicated households accessing multiple services across
disparate programs administered by individual CAAs, to the Office
of Community Services (OCS), is understandable. What is less clear
is the need to add to existing data collection processes and
systems that already meet the data reporting requirements of the
various federal, state and other programs.
CSBG
resources represent less than 2% of SETA’s annual budget.
The proposed data reporting requirements would significantly
increase the cost of data reporting to the CSBG program and reduce
the levels of service provided to the community. Furthermore, the
proposed data reporting requirements would discourage
community-based organizations and other service providers from
participating in the CSBG program thereby reducing the program’s
ability to leverage other resources.
The Annual Burden
Estimates noted in the OCS Dear Colleague Letter dated June 17,
2016, might seem adequate for a CAA mature and fully-developed
reporting process across few systems, but it does not represent
the burden of proposed collection of information for SETA. The
estimates do not include the increased time and effort in
coordinating and reconciling data collection across multiple
information systems, ongoing training and technical assistance for
over 50 service locations, and the increased administrative costs
of information technology, software, equipment and maintenance
necessary to manage such a system.
SETA is unable to provide
any quantifiable estimate of the Annual Burden of the proposed
reporting system, but it is clear that the annual burden would
greatly exceed the suggested 242 hours.
Suggested
options include: (1) using an intelligent software solution that
compiles data from multiple data reporting systems, and (2) a
centralized federal approach that deconstructs NPI aligned data
from national governmental reporting systems and delivers it to
the CAAs
|
|
Comment 5 (Action For Better
Community ; Eligible Entity/Local Agency in NY)
|
|
The practical utility and
necessity of the collection of the data requested is extremely
limited. The agency currently completes both quarterly and annual
data reports for the state oversight agency (NYS Department of
State, Office of Community Services). The program outcomes in
those reports as they relate to the agency’s community
action plan are very useful, indeed, necessary for agency quality
assurance and needs assessment. The data that are used for the
national CSBG-IS report, however, are not used by the agency for
any reason other than required completion of the report itself.
The agency has been collecting and reporting data
agency wide as well as at the program level for decades; it is
fairly straightforward to estimate the burden of the proposed
information collection with some accuracy.
First, the
agency does not foresee any reduction in data collection and
reporting as described in the proposal. The online interface to
collect data by the states, which may be made available to
grantees, does NOT promise to offer any time-reduction or lesser
burden to report information. The agency currently uses such an
interface to enter its annual performance data with the state. The
proposed interface would either replace that tool or be used in
addition to it. Due to the volume of data required by the
proposal, either a replacement or addition of this new reporting
tool would result in more of a reporting burden.
More
concretely, the agency estimates very conservatively that it
utilizes well over $75,000 in annual resources to support the
collection and reporting of agency (non-fiscal) data. This figure
includes a full-time staff person who coordinates agency wide data
reporting, staff time collecting and reporting data internally and
annual costs of several different agency databases.
The
agency will respond to the issue of resource burden by estimating
the ADDITIONAL costs it anticipates as a result of the proposed
changes to the ROMA report:
* Staff time to collect
intake data would increase slightly due to the requirement to
define and track “new” customers. Estimated staff
affected = 15 staff. Estimated hours of additional time per year =
4 hrs. Estimated cost to the agency (@ $20/hr.) = 15 x 4 x 20 =
$1,200.
* Staff time to transfer data from existing
agency tools into the new database/reporting system will increase
due to the significant increase in the number of data points being
collected. Estimated staff affected = 15 staff. Estimated hours of
additional time per year = 12 hrs. Estimated cost to the agency (@
$20/hr.) = 15 x 12 x 20 = $3,600.
* Staff time to
develop new training materials on the indicators and to provide
training (this would likely be a “one-time” expense as
such training is already part of the agency’s reporting
budget). Estimated staff affected = 15 staff. Estimated hours of
additional time per year = 8 hrs. Estimated cost to the agency (@
$20/hr.) = 15 x 8 x 20 = $2,400.
* Staff time to
perform quality assurance of data would increase due to the
increased number of measures as well as the increased complexity
of the community indicators. Estimated staff affected = 2 staff.
Estimated hours of additional time per year = 10 hrs. Estimated
cost to the agency (@ $40/hr.) = 15 x 8 x 40 = $4,800.
*
Staff time to track data will increase due to increased categories
and complexity. Estimated staff affected = 15 staff. Estimated
hours of additional time per year = 4 hrs. Estimated cost to the
agency (@ $20/hr.) = 15 x 4 x 20 = $1,200.
* Staff
time to establish community baselines for multiple community
impact initiatives required in Module 3. Estimated staff affected
= 4 staff. Estimated hours of additional time per year = 15 hrs.
Estimated cost to the agency (@ $40/hr.) = 4 x 15 x 40 = $2,400.
* Staff time to update internal tools for collecting
and aggregating data will be required on a one-time basis to
transition to the new requirements. Estimated staff affected = 3
staff. Estimated hours of additional time per year = 20 hrs.
Estimated cost to the agency (@ $40/hr.) = 3 x 20 x 40 = $2,400.
Note that the above expenses do not foresee updated
databases, software purchases or other expenses that might be
required to support data collection. This estimate conservatively
assumes existing infrastructure can suffice.
The sum
of the above expenses is $18,000. The agency considers this to be
a very conservative estimate of NEW expenses resulting from the
proposal. The agency would not choose, voluntarily, to spend this
amount of money to implement this system – it simply does
not offer significant returns to the agency given the fiscal
resources required to support it.
3. The quality,
utility, and clarity of the information to be collected; and
The
quality and utility of the data have been addressed previously,
but to reiterate some key points:
* Number of new
customers is not useful information. Instead, changes in the
patterns of services received or in the outcomes experienced are
more actionable for the agency.
* Changes in # of new
customers can potentially reflect contradictory trends with no
indication which is in effect: an increase in the number of new
customers could reflect harsher local conditions that require more
people to seek services, OR it could reflect more effective
outreach and collaboration by service providers to connect people
with needed services.
* Community level indicators are
wrought with complexity. They are often not available for the
population of interest. More importantly, the complexity of
community level issues means single rates or percentages are
capturing the impact of multiple forces. Relying on one data point
when one knows that several variable drive that number is not
sound practice.
|
|
Comment 6 (Action For Boston
Community Development Inc.; Other/Unknown in MA)
|
|
Many of the changes in the
Annual Report requirements will call for modifications in the
orientation and day-to-day workflow of agencies, in the interest
of greater rigor in data collection. To the extent that these
changes are manageable, they are welcome; however, many will
create increased time and cost burdens.
* Estimated
time and cost. The time and cost burden of reporting appears to be
substantially underestimated by OCS at 242 hours per sub-grantee
(individual Community Action Agencies). For example, Action for
Boston Community Development, Inc. (ABCD) estimates an annual time
burden of 771 hours, excluding the data gathering activities
required of front-line staff. When the work of these front-line
staff members is included, there is a 116% increase in the overall
cost of maintaining the systems required to support production of
the Annual Report (see attached Table 1)
The overall
scope of data to be reported has been dramatically enlarged in the
proposed Annual Report, which is the major driver of these time
and cost increases for ABCD. However, for many Community Action
Agencies the escalation in time and cost burdens will be much
higher.
* Investment in data warehousing. ABCD is in
the relatively unusual position, among its agency peers, of having
invested in data warehousing systems which support data collection
across multiple program-specific databases. This is a necessity
for meeting the Annual Report requirements. For an organization
managing multiple programs to establish such a data warehousing
system is a major capital expenditure—and it is a larger
expense for the organizations which have built the strongest
arrays of programming. For example, ABCD has expended several
hundred thousand dollars in creating and upgrading its internal
data warehousing capacity. This was an especially complex process,
given ABCD’s size and the evolving nature of the systems;
however, even a modestly sized data warehousing project represents
a major cost (see attached Table 2)
It is important to
note that such data warehousing systems provide important benefits
for individual sub grantees, and may be a practical necessity for
reporting unless and until major federally-funded human services
programs utilize interoperable automated client tracking and
reporting tools (or at the least, share a common data dictionary).
However, they require substantial new resources to establish and
manage, and no new funding has been identified.
*
OLDC. The projected Online Data Collection system (OLDC) is
unlikely to reduce these times and cost burdens, and in fact it is
much more probable that OLDC will require additional investment
from individual agencies.
Question 3: The quality,
utility and clarity of the information to be collected
There
is a crucially important distinction to be made among the various
information requirements in the proposed Annual Report.
*
Improvement in family and individual outcomes. The individual and
family outcomes articulated in Module 4 represent, for the most
part, dramatic improvement in the quality and meaningfulness of
data that can be developed from local agency reporting (there are
some specific issues with the items included, which are detailed
below.)
However, there are important methodological
problems associated with reporting requirements in other areas.
* Issues with “new” clients. As noted
above, the inclusion of reporting for “Characteristics of
New Individuals and Households” in Module 4, Section A: Data
Entry Form is unlikely to be of utility to local grantees, and
will produce meaningless numbers, at best, if aggregated on a
national level.
* Issues with community measures.
Similarly, the way in which “community-level” data is
used in Module 3, Section A is very likely to produce reported
numbers which are of limited use in program management, and
inappropriate for national aggregation.
Question 4:
Ways to minimize the burden of the collection of information on
respondents, including through the use of automated collection
techniques or other forms of information technology.
The
most important immediate action that HHS can take to reduce the
burden of data collection is to eliminate the elements of the
proposed Annual Report which will generate data that has little or
no utility.
However, this review process highlights
the need for HHS-funded programs to develop solutions to issues of
data integration—whether by creating interoperable program
reporting systems, or by providing assistance to individual local
agencies in the development of data warehousing systems.
Table
1: TIME AND COST ESTIMATE FOR ANNUAL REPORTING
ITEM CURRENT Hours/
Cost PROPOSED Hours/ Cost OVERALL CHANGE
1.CLIENT DATA
MANAGEMENT
a. Case Worker Level (Gathering and
entering data on households)
b. Planning Staff Level
(Cleaning data, coding)
c. Data Warehouse (Adding and
updating merge algorithms, other maintenance) a. Staff currently
report 2 hrs./wk./person @ $20/hr. for 40 persons = $83,200 b.
Staff currently report 22 hrs./mo. @ $28/hr. = $7,392 c. Current
contract cost averages $10,000/yr. TOTAL COST $100,592 a. Test
suggests increase to 4.5 hrs./wk./person @ $20/hr. for 40 persons
= $187,200 b. Test suggests increase to 26 hrs./mo. @ $28/hr. =
$8,736 c. Contract cost estimated at $16,000/yr. TOTAL COST
$211,936 $ 111,344
2. STAFF TRAINING
a.
Case Worker Level
b. Planning Staff Level
c.
Data Warehouse a. Staff currently receive 2.5 hrs. Annually @
$20/hr. for 40 persons = $2,000 b. Staff currently average 4 hours
annually @ $28/hr. for 3 persons = $336 c. No current charges
TOTAL COST $2,336 a. Projected at 3.5 hrs. annually @ $20/hr. for
40 persons = $2,800 b. Projected at 8 hours annually @ $28/hr. for
3 persons = $672 c. Additional costs of approximately $4,000/year
projected, associated with community program tracking and
reporting TOTAL COST $7,472 $5,136
3. DEVELOPMENT AND
TRACKING OF COMMUNITY MEASURES
a. Identification of
community a. No current charges a. Projected at 12 hours baselines
b. Tracking of community projects activity and impact b. No
current charges TOTAL COST $0 annually @ $28/hr. for 3 persons
=$1,008 b. Projected at 16 hours annually @ $28/hr. for 3 persons
=$1,344 TOTAL COST $2,352 $2,352
4. PREPARATION AND
SUBMISSION OF ANNUAL REPORT
a. Communication and
coaching with internal departments and
delegate agencies
b.
Collation of reported data
c. Drafting required
narrative information
d. Drafting required
Organizational Standards information
e. Data quality
assurance
f. Entry into state portal a. Staff
currently estimate 5.5 hrs. annually @ $28/hr. for each of 12
reporting units =$1,848 b. Staff currently estimate 6 hours
annually @ $28/hr. =$168 c. Staff currently estimate 6 hours
annually @ $28/hr. =$168 d. No current charges e. Staff currently
estimate 6 hours annually @ $28/hr. =$164 f. e. Staff currently
estimate 6 hours annually @ $28/hr. = $168 TOTAL COST $2,520 a.
Projected at 12.5 hrs. annually @ $28/hr. for each of 12 reporting
units =$4,200 b. Projected at 14 hours annually @ $28/hr. =$392 c.
Projected at 10 hours annually @ $28/hr. =$252 d. Projected at 18
hours annually @ $28/hr. =$504 e. Projected at 22 hours annually @
$28/hr. =$616 f. e. Projected at 10 hours annually @ $28/hr. =$280
TOTAL COST $6244 $3,724
TOTAL INCREASE $122,556
Table 2: TIME AND COST ESTIMATE FOR DATA WAREHOUSE
CAPACITY DEVELOPMENT
TASK INPUTS COST
Non-Recurring
Costs
1. INITIAL REVIEW AND COLLATION OF DATABASE
ELEMENTS AND REPORTING REQUIREMENTS a. Internal MIS Staff 10 hrs.
@ $40/hr. = $400. b. Internal Planning Staff 4 hrs.@$28/hr. =
$112. $512.
2. CONTRACTING AND SPECIFICATION FOR DATA
WAREHOUSE PRODUCT AND SERVICES a. Internal MIS Staff 18 hrs.
@ $40/hr. = $720. b. Internal Legal Staff 2 hrs. @ $45/hr. = $90.
$810.
3. PERMISSIONING ACCESS FROM
DATABASE
MANAGERS a. Internal MIS Staff 18 hrs. @ $40/hr. = $720. $720.
3. LICENSING OF DATA
WAREHOUSE PRODUCT a.
Internal MIS Staff 6 hrs. @ $40/hr. = $240 b. Licensing
Fee/Purchase @ $50,000 $50,240.
4. MERGING DATA SETS
a. Consulting Services by Data Warehouse Vendor @
$50,000. $50,000.
5. STAFF TRAINING
* SQL
Training for MIS Staff
* Training for Planning
End-Users
a. Internal MIS Staff 6 hrs. @ $40/hr. =$240
b. SQL Training @ $2,500 $2,740.
TOTAL NON-RECURRING
COST $105,022.
Recurring Costs
1. DATA
STORAGE (web-based) a. Web-based Storage @ $2,000/mo. For 12 mos.
= $24,000 $24,000
2.DATA WAREHOUSE REVISION AND UPDATE
a. Consulting Services by Data Warehouse Vendor @ $10,000. $10,000
3.REPORT WRITING a. Internal MIS Staff 6 hrs./mo. @
$40/hr. for 12 mos. = $2,880 $2,880
TOTAL RECURRING
COST $36,880
|
|
Comment 8 (Ashtabula County Community
Action Agency; Eligible Entity/Local Agency in OH)
|
|
Based on our calculations, the
cost for ROMA Next Gen implementation is conservatively estimated
to be $24,000 per year. We receive about $240,000 annual CSBG
funding, so this represents a minimum of 10% of our budget spent
tracking and reporting. We are concerned that to accomplish the
required collection and maintenance of data, the proposed system
would divert resources from our direct service for low-income
customers.
We do not have the
capacity either technically or financially to develop the
electronic tracking system that will be needed to report the
proposed data – so the above estimate requires that the
responsibility for that is assumed by our state CSBG office (Ohio
Development Services Agency).
|
|
Comment 10 (Blue Valley Community
Action Partnership; Other/Unknown in NB)
|
|
I recognize I have not
identified each section within the standards, but I simply did not
have the time to review all items and comment each time a concern
popped up – I do believe ROMA needs improvement and I
believe multi-year tracking and coordination are laudable
objectives, as are results, but saying it doesn’t make it
happen. Federal grants management may need to be involved in
adjusting line items to reflect programmatic goals and
accomplishments need to include “in process”
accomplishments.
|
|
Comment 11 (California Community
Action Partnership Association; Other/Unknown in CA)
|
|
On behalf of the 60 member
agencies that comprise the California Community Action Partnership
Association, I am writing to respectfully submit comments on
behalf of our collective membership regarding the proposed
Community Services Block Grant (CSBG) reporting requirements
familiarly known as “ROMA Next Generation”.
Your office has
received many responses from our individual member agencies so I
will not be voluminous in presenting the concerns of our member
agencies, however, it is important that the main issues are
reiterated in order for our Federal partners to understand the
tremendous burden this will potentially place on Community Action
Agencies at the local level. The issues I will note are reflective
of the feedback CALCAPA has gathered from our network agencies.
Additionally, I write from the perspective of a
professional who has provided leadership at the Municipal, County,
and State levels in implementing performance-measurement and
management tools and systems. I helped launch Baltimore City Stat
to measure and improve performance of City Departments and service
delivery. As the Assistant Chief of Juvenile Probation for Dallas
County Texas I created JD Stat as a performance management tool
for the largest Juvenile Justice agency in Texas. In my role as
Executive Director of the Community Services Administration for
the State of Maryland, I served as a leader on the team assembled
to implement the Maryland State Stat Program which received
national acclaim.
All of our agency leaders agree
wholeheartedly that we must utilize data to better measure
effectiveness as well as to tell Community Action’s story
more poignantly and effectively to policy makers and the public at
large, however, our members have expressed serious concerns about
ROMA Next Generation being the tool to accomplish that goal! Some
of the major issues you have already been made aware of that give
our member agencies pause in moving forward with ROMA Next
Generation and the proposed CSBG reporting requirements are:
1.
COST AND FINACIAL BURDEN ON LOCAL AGENCIES
One of our
larger more complex agencies after conducting an extremely
thorough analysis of what it would cost to purchase and maintain a
system to collect, collate, and report unduplicated data across
our 14 programs, 49 funding sources, 101 contracts, and 42 funding
periods was estimated in excess of $270,000 per year.
2.
THE ADMINISTRATIVE TIME BURDEN
As well as the
intentions may be with the proposed reporting requirements, it
does not factor in the burdensome time to collect the desired
data. One of our smaller Limited Purpose Agencies recently
concluded after a deep diving analysis it would require
12,509
hours for staff to compile the required data.
3. DOES
NOT FACTOR THE UNIQUENESS OF EACH COMMUNITY ACTION AGENCY
It
is often stated that if you mirror any two Community Action
Agencies side by side, you will NEVER find two Community Action
Agencies that are identical! It seems clear that the underlying
agenda behind the introduction of ROMA Next Generation is in
essence contrary to the authorizing language of the Economic
Opportunity Act of 1964 and the successive re-authorizations of
that Act. The framers of the Act understood inherently that each
community is different and therefore they have different issues
which require innovative and customized approaches to address. Now
is the time to innovate, now is the time to open the immense pools
of thought and ideas to demonstrate the value and worth of
Community Action. Now is the time to herald our core values –
the uniqueness of each community and the leveraging of CSBG
dollars to innovate new localized solutions with greater impact
and superior outcomes. Each local jurisdiction should determine
and prioritize its own needs, then create informed, effective, and
tailored responses addressing those needs. Now more than ever is
not the time to turn back from the framers intent.
4.
ADDED BUREAUCRACY AND OVER REGULATION
Often we hear
this point echoed in the private sector but this also is an ever
increasing reality for Community Action Agencies, particularly if
this reporting framework moves forward. As government adds to the
layers of administration and bureaucracy entailed in Community
Action, we face a perilous slippery slope towards stripping
agencies of the local control that is at the core of our success.
Each community, each region in our nation is unique. It is the
hallmark of Community Action to empower local leaders and consult
local residents to design and develop customized programs and
initiatives to fight poverty.
At a time of reduced
funding for poverty efforts at the Federal level and harsher
conditions facing people with low incomes, it is unconscionable to
prioritize data collection and the increased time and expense that
will result from the implementation of ROMA Next Generation. The
current system is more than adequate. The United States needs a
Community Action Network that addresses local needs with a laser
focus on innovation and the leveraging of all possible resources
to create and introduce opportunities that lead to independence
and prosperity among our disadvantaged populations.
For
50 years we as a nation have waged a War on Poverty. As an
Association we share the insight of Napoleon Bonaparte, French
Military and Political Leader when he stated, “War is ninety
percent information”. It is clear that we must more
effectively utilize data in our War on Poverty. However, we also
share the thoughts of famed Physicist Albert Einstein when he said
“Not everything that can be counted counts and not
everything that counts can be counted”.
Thank
you for the opportunity to weigh in on behalf the 60 California
Community Action Agencies who are fighting every day to empower
those who count to lift themselves out of the deep recesses of
poverty to enjoy the promise and opportunity that is America!
|
|
Comment 12 (Cap Service ;
Other/Unknown in WI)
|
|
I am the President and CEO of
CAP Services, now recognizing its 50th year of service like so
many in the community action network. We are proud of the
long-term powerful impact generated in the communities we serve,
currently through nearly 30 different programs to advance our
mission "to transform people and communities to advance
social and economic justice." In 2015, we served
approximately 2,500 households in our five-county footprint in
Central and East Central Wisconsin and have done it transparently
and with accountability to our constituents, donors, investors,
community partners and volunteers.
A diverse
organization like ours is critical to addressing the complex
issues of poverty. And while we have been very successful in
securing the resources to create and sustain innovative programs,
the requirements of our funders are also expansive. Many require
significant reporting, often funder-specific reporting tools.
Examples of those with such include Head Start and Early Head
Start, the Low-Income Weatherization Assistance Program,
AmeriCorps, DOL-Youth build, WIOA, SBA, HUD and even more.
We
recognize this as a part of doing business but do also ask that
you recognize the implications of more expansive reporting for
CSBG, including areas not previously required. CSBG is important
to our organization but at the same time it is a funding source,
not a program. As I posed in my previous comments, this diversity
of sources requires significant reporting, often times in slightly
different ways.
We have looked at very expensive data
systems, and have yet to find one that would adequately,
efficiently and cost-effectively move to a central system for
tracking outcomes. As such we collect and aggregate information
for CSBG reporting, duplicating effort almost without exception
because other funders will not waive the requirements to report in
their systems (many web-based and not compatible with other
systems).
We agree with the Partnership and others in
our network that while there are elements of ROMA Next Gen that
may be helpful in strengthening our ability to communicate our
impact, the collection of information does not meet the threshold
of necessity or utility. The package as proposed in its entirety
is overly burdensome, contains reporting tools that will provide
useless information, and is outside the reach of the Community
Services Block Grant (CSBG). I question who you identify as the
target audience for the report, and whether what is collected both
clear and concise enough not to raise questions and doubt the
capacity or deliverables by the network (i.e., questions about
collective impact or community indicators). I daresay, the breadth
of the proposed report would not be particularly helpful to the
Congressional delegation nor for the public at large, particularly
when you get past individual or household indicators.
We
have engaged in local community impact initiatives and see mixed
results in terms of true collaboration, relevant and meaningful
sharing of data and continuity for the purpose of longer-term
measurement. These groups take time to storm, form, norm —
and with changes in leadership and time, can fade, or at least
lose some of the meaning or spirit for which they were created. To
create a system of reporting of the same, without better network
planning, analysis and benefit seems short-sighted and ill-advised
investment of precious resources.
However, with
modifications, a system for collection of information can be
developed that improves the performance management of the agency
and has practical utility. Detail is provided here outlining where
changes need to be made to reach a level of workability and
utility.
|
|
Comment 13 (Cattaraugus Community
Action, Inc.; Eligible Entity/Local Agency in NY)
|
|
1: Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility.
We have carefully
reviewed the talking points issued by the National Community
Action Partnership and submitted to OCS. We strongly agree with
those talking points with respect to the necessity and utility of
some of the data elements. We also endorse their recommendations
for modification of Organizational Standards, removal of the
reporting on characteristics for NEW Individuals and Households,
Removal of percentage/rate indicates for Community Outcomes;
removal of Collective Impact reporting, the removal of stability
indicators, and re-introduction ROMA goals centered on agency
capacity.
2: The accuracy of the agency’s
estimate of the burden of the proposed collection of information.
We have reviewed available information on time and
cost burden associated with implementation of ROMA Next Gen. While
we believe that the estimates provided thus far are essential to
understanding the scope of the problem, but we further believe
that it is not possible to accurately assess the increased time
and cost burdens associated with proposed changes. It is apparent
that many agencies lack current capacity to report on proposed
ROMA Next Gen data points. Correcting this problem will require
new/updated reporting software. It is our understanding that HHS
is developing software needed to submit state reports, but that
this software may or may not be available to local agencies.
Without this knowledge, it is not possible to accurately estimate
the costs of software design/redesign.
It is also
apparent that the new ROMA Next Gen data points will place an
increased burden on agency resources with respect to staff time
spent in gathering and entering data, monitoring data quality, and
generating and submitting reports. Again, it is not possible to
place a dollar amount on staff time without a full picture of the
data collection system. We can anticipate substantially increased
costs associated with staff training in ROMA Next Gen, as well as
in data collection, data entry and reporting, but placing a dollar
amount on these costs is pre-mature.
3: The quality,
utility, and clarity of the information to be collected
We
strongly agree with the National Community Action Partnership with
respect to their comments on the quality, clarify and utility of
the data to be collected and refer you to that document for
detailed information. We wish to emphasize that tying a community
indicator directly to the work of a Community Action Agency
grossly ignores the fact that multiple factors can impact any one
community condition. It also sets Community Action agencies up to
fail because, despite our best efforts, community factors beyond
our control can negatively impact achievement of community based
outcomes.
4: Ways to minimize the burden of the
collection of information on respondents, including through the
use of automated collection techniques or other forms of
information technology.
Again, we agree with the
National Community Action Partnerships suggestions. We further
suggest that OCS confer with the vendors of the various report
systems currently in the Community Action network, and gather
their thoughts on cost estimates and ways that data collection and
reporting can be streamlined
|
|
Comment 16 (Central Missouri
Community Action ; Other/Unknown in MO)
|
|
Whether the
proposed collection of information is necessary for the proper
performance of the functions of the agency, including whether the
information shall have practical utility
the overall
concept of the information to be gathered for the purpose of
validating the work of community action is absolutely necessary.
Central Missouri Community Action is prepared to meet new
requirements of information collection and appreciates the work
that has been done thus far. Although we as a state feel
comfortable with the idea of ROMA Next Generation, we are
concerned with the overall impact to the network if other
agencies and states do not have the resources or training to
collect information within the proposed framework of indicators
and requirements.
2. The accuracy of the agency’s
estimate of the burden of the proposed collection of information
the time burden seems underestimated. The level of
expertise to conduct some of the proposed data collection and
analysis is heightened as well. Additional skill sets may be
needed, which would increase the overall cost burden associated
with the CSBG Annual Report.
3. The quality, utility,
and clarity of the information to be collected; and
Quality:
The proposed set of indicators seems to focus more on quantity
rather than quality. Our recommendation would be that the numbers
of general indicators are decreased (although we know not all
agencies must report on each one). If a smaller number of
indicators were used to aggregate data to the state and national
level, this may be more meaningful to decision makers. Agencies
could then use local indicators to track additional outcomes
based on their unique programs. For example, if there were 2-3
indicators relating to each of the domains of CSBG these could
provide meaningful aggregated data. The local analysis
expectation could then be strengthened to fuel the ROMA cycle
within the evaluation and assessment portion.
Utility:
To the point above, we feel the information would be most useful
in fewer, more meaningful indicators that can show national
impact of community action in a more concise way. Also, it would
seem appropriate to use the language of other indicators that are
already established through Head Start or HUD to promote further
consistency. A request from our network is to be able to access
the local, state, and national comparison data after compiled to
use and share locally. Clarity: We respect the strengthened focus
on community and believe it is necessary. However, many of the
proposed community indicators could be confused with individual
or family level indicators. The meaning tends to be similar, as
well as the unit of measurement, while the wording is different.
Examples include:
• Community Level, Education
#9 = Family Level, Education #8, 9, 10.
•
Community Level, Education #5 = Family Level, Education #7.
Next, there are terms we have identified that would
need to clarified definitions before moving forward to ensure the
network is collecting similar information:
•
Low-income
• Community
• Living
wage
• “New” in characteristics
report
Lastly, the variation of percentages, numbers,
and rates seems confusing and inconsistent. We understand the
need to examine percentages when looking at an increase or
decrease and numbers are helpful in other circumstances. The way
some indicators are written it causes confusion on what is being
counted or measured. The rates should most definitely be
examined. However, we believe that considerations should be given
to including this information in the community needs assessment
to be analyzed over the years to see trends up or down.
4.
Ways to minimize the burden of the collection of information on
respondents, including through the use of automated collection
techniques or other forms of information technology.
We
consider ourselves fortunate to have a statewide data system that
is a priority of the State CSBG administrators through
discretionary funding. Many do not have this resource, which will
be necessary for receiving an accurate unduplicated count.
Training and technical assistance (T/TA) will be necessary on
counting with consistent definitions and systems/procedures for
efficient data collection. We also know the web based data tool
in the community commons platform will become even more of an
asset. The 19 Community Action Agencies in Missouri utilize this
tool to collect and analyze secondary data sets for a number of
purposes. The indicators that are available through this tool at
the state level should be considered in relation to the proposed
indicators.
|
|
Comment 21 (Combined Community Action
; Eligible Entity/Local Agency in TX)
|
|
I am writing to voice my
concerns about the new reporting and cost as it relates to the
ROMA Next Generation. Because of all the unknowns, it is almost
impossible to estimate the cost associated with the new reporting.
I do know with all certainty that there will be a huge spike in
the cost for reporting and upgrades to our client tracking
software to collect all the data. We will be spending more time
reporting then actually helping the customers in our service area.
We are a very small community action agency, but well known and
respected in the community. Our annual CSBG budget is $200,000, so
any increase in reporting (manpower) and software cuts into that
budget significantly.
The CSBG is a unique federal
program aimed at alleviating poverty by affording local eligible
entities flexibility in decision making about the use of the funds
for a broad range of purposes, provides emphasis on community
level work and local decision making about how best to meet local
needs also requires maximum participation of residents of the low
income community in the decision making process. By implementing
these new reporting requirements agencies are being told how to
spend the money which is to pay more staff people and upgrade
software for reporting purposes instead of helping the local
community. We all need to read the CSBG Act again and remember why
it was enacted. Many agencies will be put out of business because
of these rules and reporting requirements. Local county officials
who are very supportive of our agency will be very unhappy to hear
about these new requirements. I plan to meet with each of my
county judges and explain the situation so they are not surprised
when a large portion of our CSBG budget is being used to upgrade
software and hiring additional staff instead of helping the local
community
|
|
Comment 23 (Community Action Agency
Of Beaver County ; Eligible Entity/Local Agency in PA)
|
|
As a public CAAP agency the
overall requirements would be a very heavy cost burden, which we
would not have the funds and also the time would be astronomical
and again as a public CAAP we do not have the capacity or the cost
to add staff. Our subcontractors would also be in the same
situation.
The Community Action Council of Portage
County (Ohio) has a relatively small CSBG grant of approximately
$281,000 in FY 2016. We estimate that the implementation of ROMA
Next Gen would require an investment of CSBG funding of
approximately 15.4% of our total CSBG allocation, or approximately
$44,000 per year, assuming no significant additional software,
hardware, or related costs. This amount reflects predominantly
staff time. If the current scheduled implementation date holds
true, we estimate that it would require an additional initial
investment as well. Our conservative estimate of this cost would
approximate $12,000.
This amount above assumes that
our state CSBG office (Ohio Development Services Agency) will be
responsible for development of the electronic data tracking system
that will be needed for reporting purposes, including bearing the
costs to do so.
Community Action Committee of
Victoria, Texas (CACVT) supports comments filed by the Texas
Department of Housing and Community Affairs regarding proposed
changes to the CSBG Annual Report.
CACVT feels staff
will be required to capture such a large amount of information and
to work with such a complicated reporting system that there won’t
be time to actually help the people that have come to us for
assistance to transition them into self-sufficiency.
Combined
Community Action, Inc., Giddings, TX supports comments filed by
the Texas Department of Housing and Community Affairs regarding
proposed changes to the CSBG Annual Report. I sent comments last
week regarding the CSBG Annual Report.
Community
Action Commission of Erie, Huron, & Richland Counties Inc. has
determined that the attached costs will be $53,418.31, which is
10.72% of our annual CSBG allocation of $498,495. However, there
is the compounded aspect of systems compatibility that need to be
factored and cannot be ascertained until the actual
implementation.
1.
Whether the proposed collection of information is necessary for
the proper performance of the functions of the agency, including
whether the information shall have practical utility. The proposed
collection of information is necessary for proper performance of
function of the agency in that it will enhance the ability to
effectively quantify outcomes and measure that are not currently
reflected due to some data reporting limitations in the ROMA
environment.
2. The accuracy of the agency’s
estimate of the burden of the proposed collection of information.
CACEHR anticipates the initial burden of decreased efficiency
attributed to learning curve and ability to transition into
competence with the updated environment. However, after initial
implementation actual burden instance is expected to level off or
decr4ase because of environment proficiency.
3. The
quality, utility, and clarity of the information to be collected;
and The quality, utility, and clarity of the collected information
will depend heavily on the new workflow of the environment. ROMA’s
new updates appear to focus on providing the ability to more
accurately provide pertinent detailed outcomes and measures
related to the four module protocols. (i.e. Mod 1-State
Administration, Mod 2-Agency Expenditure, Capacity, and Resources,
Mod 3-Communtiy Level, Mod 4-Individual and Family Level)
4.
Ways to minimize the burden of the collection of information on
respondents, including through the use of automated collection
techniques or other forms of information technology. This area is
difficult to adequately determine due to ROMA Next Generation
actual performance and data inputs not being available in
conjunction by agency personnel and sister agencies, as well as
lack of empirical result and outcome. More time is needed to gage
additional improvements.
To Whom It May
Concern:
Based on our calculations, the cost for ROMA
Next Gen implementation is conservatively estimated to be $24,000
per year. We receive about $240,000 annual CSBG funding, so this
represents a minimum of 10% of our budget spent tracking and
reporting. We are concerned that to accomplish the required
collection and maintenance of data, the proposed system would
divert resources from our direct service for low-income customers.
South Plains Community Action Association supports
comments filed by the Texas Department of Housing and Community
Affairs regarding proposed changes to the CSBG Annual Report and
we feel this unfunded mandate will spike expenses for additional
staff and upgraded systems in order to capture the enormous amount
of data as proposed. Local staff will be steered away from helping
people to capturing a vast amount of information and navigating a
complicated reporting system which may yield less impressive
results of the CSBG program because staff is re-directed from
direct services to paperwork. Indicators won’t necessarily
capture the variety of Community Action achievements and projects
and won’t tell the true CSBG story. Cost estimates are
difficult to gage due to many unknowns at this point in time.
As
the CEO of the Multi-Service Center, a community action agency in
Washington state, I am writing to provide my feedback on the
proposed changes to reporting for CSBG and the cost estimates
provided by HHS related to these changes. I am very concerned
about the implications of the proposed changes as well as the
accuracy of estimates provided by HHS.
· The
information obtained from the report that is being proposed will
not have a significant increase in usefulness or being meaningful,
at least not sufficient to justify the effort required to complete
data collection and submission of the annual report. If the 6 NPIs
and the past reporting process weren’t providing
sufficiently reliable data, I struggle to see how increasing the
complexity helps in this matter. More data doesn’t
necessarily equate to effectiveness, especially given the concerns
about capacity to collect and report data with integrity. In
addition, with increased complexity come increased possibility for
ambiguity and multiple interpretations. How will CAAs in each
state not to mention around the country come to mutual agreement
regarding the meaning of each indicator? A shared understanding of
the indicators is necessary for uniformity and for the data to be
meaningful and valuable.
· I am unsure as to
what methodology OCS used to arrive at the estimate regarding
length of time to complete the new report. Therefore, it is
difficult for me to determine if this estimate truly reflects the
realities of day to day operations of a community action agency.
Furthermore, I did not see any information or analysis of the
existing capacity of CAAs to collect and report data with the
level of complexity outlined in the proposed reporting format.
Certainly it is impossible to assert that every CAA is at the same
level of capacity in this regard as it relates to factors
including but not limited to software/database capacity and
staffing resources.
· Staff responsible for
using the current report estimate that the new format will double
or triple the time it currently takes in order to do a good job
with the proposed process. Given that there are no additional
resources being provided to offset these costs, the increased time
required to complete the report will take time and resources away
from providing the services that fulfill the mission of CAAs in
our communities.
I feel it is critical that OCS take
more time to develop shared definitions of indicators and to
investigate more thoroughly whether or not CAAs currently have the
capacity and resources to meet the new data requirements. At the
very least, it would be helpful to know what methodology was used
by OCS to create its cost estimates. It would also be helpful if
OCS could advocate with other government entities to allow
information to be shared or cross walked.
Ultimately,
we feel strongly that data is a critical element in telling our
story and evidencing the effectiveness of community action in
alleviating and eradicating poverty and we appreciate OCS’
efforts to improve data collection and reporting. We simply want
to ensure that the data being collected is, in fact, accurate,
meaningful, and useful and that collecting and reporting data does
not negatively impact service delivery and outcomes as a result of
not having sufficient capacity to carry out the reporting process.
|
|
Comment 24 (Community Action Agency
Of Butte County Inc. ; Other/Unknown in CA)
|
|
Thank you for conducting the
process that invites comments on the Annual Report proposed as pan
of the ROMA Next Generation (RNG) data collection, reporting and
management system. The RNG concept has been reviewed by many
interested
parties with significant input offel•ed
during its' development process over the last 12 months. From the
national network of field practitioners there have been both
varying degrees of agreement and disagreement with various
elements of the content. Because it is the nature of community
action to be forthright on the actions necessary to improve the
conditions and causes of poverty then some of that will be
included here. At this time I also think it appropriate to make
clear that I am submitting these remarks in my role as the chief
executive of a local member of the community action network and
not in any capacity of leadership in -that network.
We
find ourselves in the midst of much change as we review this
proposed method to help us to be better The evolution of language
and terminology, of the manner and methods of evaluation should
not be cause for us to do nothing. Rather we can use it to help us
to be mindful that what we do now should bear the marks of what is
enduring. I fit lends itself to varying or changing
interpretations then we should lead lightly since ours is a
strategy still evolving into a type of discipline. In my
38
years of work in the field of community action at the local,
state, regional and national levels I have been committed to
seeking excellence in our work. I remain committed to the notion
that there is nothing like community action to serve as a tool for
our individual and collective conscience in pursuing that desired
state for our communities and those who live in those communities.
It is both process and product; communities and community members
working together to maximize opportunities for all.
Comments
Re Next Cienu•atien Annual Report Input to follow the four
questions
I
will provide comment on the four (4) questions listed in the Dear
Colleague Letter. It will not be in the format that was requested
i.e. "by module, section, and page number as appropriate"
because that approach does not acknowledge or provide for some
very real concerns that are inherent but not obvious when you're
focused on the end result i.e. the Annual Report. Certain comments
reflect the continuing concern with some of the basis for the
Annual Report that is not clearly evident in the modules but of
definite importance when you drill down into the "how"
of how the proposed Annual Report must be completed to be of a
sufficient utility so that it offsets the investment of time,
funds and more. The Annual Report reflects a significant revision
to the type of data to be collected, the costs and methods of
collection of that data and the manner in which it is used. It is
my assessment that the full implications of these revisions have
not given adequate consideration such that the federal government
and network of agencies would be better served to put off approval
of the proposed ROMA Next Generation proposal until sufficient
analysis, understood and agreement has been achieved.
Here
I address multiple points including: a) use of social indicators,
b) Baseline Measures Report, c) Collective Impact measures and d)
the lack of cost effectiveness of the effort to complete the
necessary data collection for the Annual Report to be of
sufficient off-setting utility at the local level and
beyond.
Under-resourced Implementation to hinder
achievement
A concern is now expressed that the design
of the implementation of the proposed Annual Report does not
include adequate resources to ensure its efficient and
cost-effective intent. Having experienced the effort to roll-out
the first generation of the ROMA system of management and
reporting I can say that, from a national perspective, it was
under-resourced. It took approximately 10 years (to approximately
2003) for the main points of ROMA to become well-enough known for
there to be common or shared understandings amongst CAA's and the
state and national systems of oversight and support. It then took
an additional 10 years (to approximately 2013) for the national
network to embrace the ROMA system as a management tool with the
importance originally intended. The reasons for this extended
period of embrace, I believe, are the inconsistent provision of
resources for training and technical assistance throughout the
national network.
That
situation was remedied to a major extent with the current
administration over the last 4 years.
However such a
vision for support before, during and after implementation is
absent from the proposal.
We understand that this is
not the intent of the Agency. Be that as it may, without the
planned resources for a roll-out, the chances are great that it
will suffer the same fate, thereby negating the beliefs
anticipated in its conceptualization and design.
Collection
of such data is an entirely different matter calling for the
development of methods of collection, entry and management. In
each such case it will require that a unique definition of the
data must occur. It is critical that there be common
understandings and definitions agreed to, documented and shared in
order for meaning to be drawn from data. But it does not stop
there since then the common understanding then has to be
communicated to others in different times and in different spaces
without guarantee that it will be accepted. This creates an
untenable situation where data is collected without value because
of the lack of shared understanding and meaning.
I
believe that the agency's estimate of the burden of the proposed
collection of information is understated. Local CAA's vary
tremendously in there. Capacity to respond to data collection and
reporting requirements. For some time now CAA’s have
operated under in an environment that demands restrictions on data
collection and sharing. These restrictions are inherent in grant
agreements and other program support guidance. The result has been
a very inefficient approach to the important
aspect of data
collection, management and reporting. One fix could be for OMB to
impose overarching requirements on grant programs but even that
would have much to overcome since the sheer amounts of required
data currently being collected due to multiple grant program
requirements from as many different federal agencies through a
myriad of software systems is overwhelming.
The
proposed Annual Report does not do anything to reduce this
existing burden.
Future fixes amount to unknown costs
and impacts
In the short term the NPI structure
includes potential new data fields that will require new efforts
to design existing software solutions. However these will be
one-time efforts, with associated costs, that should not continue
to impact future data collection costs. The proposed Annual Report
presentations include extended timelines to accommodate the
development of solutions for efficient use of technology that will
be necessary for the Annual Report to be reality. While the
inclusion of an extended time is warranted it is also cause for
concern that the fixes envisioned and, nessacary will be in the
future at an unknown cost to financial and human resources. The
result can easily be seen as a drain on existing and future
resumes intended for mission-related activities.
As
to the estimate of the burden it has been my experience that most
agencies in the network will involve a combination Of
specialists/managers and support staff to assist in the collection
and review of the data that represents their compliance and
performance efforts i.e. not an unimportant task. The more
proficient agencies will utilize standard data quality testing
methods at identified intervals to ensure accuracy and
reliability. Reconciliations and/or interpretations are generally
handled by management to ensure understanding of system issues or
program issues. This assumes that existing systems are adequate to
the task of collecting under a new NPI structure. The nature of
systems is change. Therefore the cost and effects of systems
tweaking or outright change will need to be considered. In no
small number of cases a significant amount of resources may be
needed since NPI's require data to be collected of various sorts
to be able to estimate change or accomplishment over time.
The
proposed Annual Report relies upon a reporting framework that
utilizes a new Theory of Change (TOC). The inclusion of a TOC is
an important new development in the community action field and
establishes the work of the network as one worthy of
consideration. However the proposed TOC excludes the critical
development of a strong national organizational structure that
will be the agent of change in helping to deliver relevant and
catalytic activities including programming, problem measurement,
new collaborative systems designs, evaluation etc.
It
is the contention of many in and out of the network that the
proposed TOC should include a goal for the Agency. This change
would ensure that local communities focus on both the desired end
of increased opportunities for all as well as the important
development, maintenance and strengthening of the means by which
the desired end is accomplished. The end is commonly accepted that
local conditions for low-income persons be improved; opportunities
maximized and low-income persons achieve improved
self-sufficiency. The commitment to improvement (i.e. the means as
embodied in a CSBG Eligible Entity) will not embody all of the
improvement desired but it can safely be said that the end will
not happen without the means.
Many local institutions
exist for broad reasons in the life of a local community. In the
Private sphere these institutions are primarily focused on
targeted constituencies whether they are the: religious, aged,
artists, youth groups, recreation groups etc. Local government
exists for all local constituencies. CAA's
exist to focus on
the conditions and circumstances of poverty i.e. the sole purpose
of the Community Services Block Grant. Any assumption that a focus
on strengthening local CAA's is not needed as concept-delivery
mechanisms to maintain and grow the efforts that will be required
to accomplish change over time is unrealistic, Therefore it is
highly recommended that the role of Agency be emphasized more by
establishing a goal for it in the TOC.
In summary
conclude that the field of community action is one that is ripe
for continued exploration of ways to improve various aspects. It
is this commitment to seeking new ways to achieve our goal of
maximizing opportunity for all including low-income residents that
is a driving force in conducting pilot projects, pursuing trends
in relevant evaluation and, where possible, scalability. To
varying degrees CEE's have been a community-based laboratory of
Service-delivery and innovative approaches for many years. That is
why did not respond to the three questions in the. Federal
Register notice regarding the data elements. That discussion
should be the subject of subsequent efforts to ensure the maximum
amount of efficiency is designed into the collection of key
data.
The proposed Annual Report, while commendable in
concept, requires too much work to get it to the point where its
cost-effectiveness warrants implementation because of the benefit
that it poses to the problem. The problem is how to improve the
reporting of a national network that will enable greater
understanding of the impact that community action is achieving as
it has since its inception in 1964. The proposed Annual Report is
not the solution to that problem. It is a valiant attempt at
addressing important issues that deserve attention but its
overreaches for solutions poses more challenges than the problem.
Some of the challenges may be overcome, in the course of time,
with appropriate approaches to problem-solving.
To reiterate that the
capability is present the national network of CEE's is glad to
have been integral participants in the companion efforts to
improve accountability via the creation and implementation of
national Organizational Standards for all local CSBG Eligible
Entities. We are also glad to have been an integral part of the
effort to create State and Federal Accountability Measures since
these then addresses the full complement of responsible parties to
the unique mission that is community action.
It is my
sincere hope that these comments are taken with the intended
purpose; to be candid comments on a concept with far-reaching
implications. It is also my hope that what is clear through these
comments is that I share a vision of the community action network
that will always be characterized by the desire and commitment for
the work that we do at all levels to be viewed as excellent,
valuable, accountable and a valid expression of this country's
commitment to promoting opportunity for all who call it home.
Thank you, again, for this opportunity to share my perspectives on
this very important
issue with you.
|
|
Comment 25 (Community Action Agency
Of North Alabama ; Other/Unknown in AL)
|
|
Reduce the amount of CSBG
funds we spend providing direct services and use that money to add
to our workforce an additional full time equivalent Resource
Specialist to assist our Compliance, Planning and Resource
Specialist (CPR Specialist). ◦This additional staff member
will be assisting the CPR Specialist in the compilation of data
and support documentation for electronic and hard copy files,
aggregating reports from Child plus, Counselor ax and other data
systems with FACspro data to ensure accurate counts and do our
best to ensure no duplication of services, analyzing whole agency
data and creating dashboards for the Leadership Team and Board to
use as they lead the agency and complete the new IS Report and
others created in response to ROMA NG.
•Carve out
approximately $50,000 annually (salary + 30% costs for benefit
package) for the additional CPR Resource Specialist
What
must be done nationally to accommodate ROMA NG in its current form
is to make changes to allow CSBG funding to be spent on staff time
to gather, aggregate and analyze data required from non-CSBG
funded programs with CSBG funded programs. Currently CSBG an
inadequate amount of funding is available for this task and
non-CSBG funded programs do not have the funding to pay for this
task. If CSBG wants whole agency data reporting, they must make
provisions to cover the costs to glean that data for reporting.
Our issue is - to accommodate all that ROMA NG
requires will have a negative impact on our agency….in
added cost due to the enormous amount of data gathering across the
agency and the loss of direct service results due to a higher
percentage of CDBG dollars to be used as staff support.
|
|
Comment 26 (Community Action Agency
Of Northwest Alabama, Inc.; Eligible Entity/Local Agency in AL)
|
|
Views represent the Agency
Executive Director and staff following an Agency staff meeting
August 5, 2016. The HHS table estimating the hours needed, on
average, is below. According to HHS they are still working on
associating dollars with this table. Number of Respondents 52
Grantees 1,035 Sub-Grantees Number of responses per respondent 1
Average burden hours per response 164 242 Total burden hours CSBG
Annual Report 8,551 250,585 Items to include in cost estimate of
yearly, post-adoption ROMA NG work TIME 1. The amount of time that
intake and other workers will spend obtaining information that is
not required by other programs. Given the expansive ROMA NG
measures, we estimate the documentation/follow-up time to be
double per worker. Currently a worker has 2,080 work hours per
year. While this number will not change, what shall change is the
number of individuals and families assisted. Excluding LIHEAP we
currently work with 900-1,000 households each year overall. ROMA
NG will cut in half the number of emergency needs that can be
addressed due to the burden of over 1,000 new measures and
follow-up on each one.
2. The typical amount of time
required for each worker to be trained and refreshed on the skills
of data entry, or data acquisition, including new and continuing
staff. Initial training will be extensive. First the certified
ROMA trainers will need to be trained along with the executive
director, followed by the staff and board. Software training will
be a crucial step for the responsible staff to understand the
changes and to train agency-wide staff. ROMA Trainers time is
estimated at 80 hours each for two Trainers = 160 hours + 8 hours
training staff + 4 hours with Board = 172 hours. Executive
Director = 40 hours. Software staff = 120 hours + 104 hours in
training/tech support for staff. All other staff = 20 hours x 13
staff = 260 hours + additional hours individual tech support. 8
hours x 13 staff = 104
Total Agency time in initial
change = 800 hours or more (if no new employees are hired) 3. The
amount of time your team will spend on data quality assurance, as
well as quality assurance on the final report. In the beginning
reports will need to be looked at weekly with the ROMA Trainers
and ED then reported to staff where we are failing to track data.
Following this, monthly reports at staff meeting and monitoring by
lead staff will need to take place to prevent falling behind.
Final reporting will be time consuming to ensure all data is
correct and contact employees to make corrections as necessary.
Executive Director/ROMA Trainers weekly monitoring/staff support =
10 hours. Staff support = 3 hours weekly. Final reporting = 12
hours Total quality assurance = 25 hours 4. The amount of time
staff will spend tracking individuals and families, tracking
progress of community projects and recording the progress in the
format required in modules 3 and 4. Our Agency provides services
to 900-1,000 households each year not including LIHEAP. It is
simply unrealistic to assume current staff can track this number
for 180 days after a service is provided. Answering the ‘measures’
may force the Agency to eliminate some clients believed to not
meet the level we seek for achieving outcomes. 1,040 hours year x
8 staff = 8,320 hours 5. The amount of time staff will spend
gathering external demographic information required in module 4.
This will mainly be Executive Director and Admin Assistant who are
responsible for final reporting. 80 hours year 6. The amount of
time spent tracking volunteers, board members and their
commitments to organizations and projects that are not run by your
Community Action Agency. This simply cannot be tracked. We would
not even try. 7. The new reports to the state required for each of
the 50 organizational standards, with additional analysis required
on any standards that your agency has not achieved in full [module
1.] More reporting time = 8 hours 8. The placement of data into
the state reporting software, including checking and answers to
the narratives required by the state for module 1, as well as the
local information on funds, people and projects. After reviewing
the format suggested for State use it will be time consuming to
hand enter this information unless the state shares the on-line
tool. Time to collect information will need to be on-going yearly
= 40 hours for reporter Writing narratives = 40-60 hours yearly 9.
The Cost of Time = in excess of $150,000. This change would
roughly entail ¼ of the Agency current staff time x salary
for Agency staff. O Based on the full cost of the different levels
of staff contribution time.
|
|
Comment 28 (Community Action Agency
Of Northwest Alabama, Inc. ; Eligible Entity/Local Agency in AL)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility Certainly not all indicators are
practical for every community action. Many of the proposed
indicators are totally unrealistic for a small agency and in the
end will only be a number not having effect on the daily
operations. The levels of information to be collected in many
areas cannot in good faith be obtained creating the illusion of
failure. ROMA NG disregards the Standards which are so rigid and
control much of the agency activities. The Standards ensure
agencies must provide/produce proof of actual documentation of
activities. With this documentation comes the assurance work is
being done in the community and for the agency. It is not
necessary to add multiple layers using many documents to this
already cumbersome process. Perhaps the Standards and ROMA NG
should have been rolled into one process and streamlined for
reporting.
2. The accuracy of the agency’s
estimate of the burden of the proposed collection of information.
What is being proposed (242 hours per response) is considerably
less than our estimates which would in many ways equate to almost
½ of a staff time for training, collection and reporting.
3. The quality, utility, and clarity of the
information to be collected; and It is unrealistic to think staff
feeling the constant pressure to perform would gather quality
data. When it becomes more about ‘meeting an indicator’
than meeting a client need the data gathering or long-term
reporting could suffer. We are expected to report on all funding
resources/revenue regardless of having administration funds to do
so. Many of our resources have no admin at all. We try to work
effectively and efficiently within the funding sources guidelines
to help our community. To be required to add additional reporting
of data collected is non-productive.
4. Ways to
minimize the burden of the collection of information on
respondents, including through the use of automated collection
techniques or other forms of information technology. Consider what
useful information for reporting purposes is. If it’s only
use is on the local level why collect this? It would certainly be
difficult to aggregate some of the data into useful state-wide
report s.
Items to include in cost estimate of yearly,
post-adoption ROMA NG work TIME 1. The amount of time that intake
and other workers will spend obtaining information that is not
required by other programs. Given the expansive ROMA NG measures,
we estimate the documentation/follow-up time to be double per
worker. Currently a worker has 2,080 work hours per year. While
this number will not change, what shall change is the number of
individuals and families assisted. Excluding LIHEAP we currently
work with 900-1,000 households each year overall. ROMA NG will cut
in half the number of emergency needs that can be addressed due to
the burden of over 1,000 new measures and follow-up on each one.
2. The typical amount of time required for each worker
to be trained and refreshed on the skills of data entry, or data
acquisition, including new and continuing staff. Initial training
will be extensive. First the certified ROMA trainers will need to
be trained along with the executive director, followed by the
staff and board. Software training will be a crucial step for the
responsible staff to understand the changes and to train
agency-wide staff. ROMA Trainers time is estimated at 80 hours
each for two Trainers = 160 hours + 8 hours training staff + 4
hours with Board = 172 hours. Executive Director = 40 hours.
Software staff = 120 hours + 104 hours in training/tech support
for staff. All other staff = 20 hours x 13 staff = 260 hours +
additional hours individual tech support. 8 hours x 13 staff = 104
Total Agency time in initial change = 800 hours or
more (if no new employees are hired) 3. The amount of time your
team will spend on data quality assurance, as well as quality
assurance on the final report. In the beginning reports will need
to be looked at weekly with the ROMA Trainers and ED then reported
to staff where we are failing to track data. Following this,
monthly reports at staff meeting and monitoring by lead staff will
need to take place to prevent falling behind. Final reporting will
be time consuming to ensure all data is correct and contact
employees to make corrections as necessary. Executive
Director/ROMA Trainers weekly monitoring/staff support = 10 hours.
Staff support = 3 hours weekly. Final reporting = 12 hours Total
quality assurance = 25 hours 4. The amount of time staff will
spend tracking individuals and families, tracking progress of
community projects and recording the progress in the format
required in modules 3 and 4. Our Agency provides services to
900-1,000 households each year not including LIHEAP. It is simply
unrealistic to assume current staff can track this number for 180
days after a service is provided. Answering the ‘measures’
may force the Agency to eliminate some clients believed to not
meet the level we seek for achieving outcomes. 1,040 hours year x
8 staff = 8,320 hours 5. The amount of time staff will spend
gathering external demographic information required in module 4.
This will mainly be Executive Director and Admin Assistant who are
responsible for final reporting. 80 hours year 6. The amount of
time spent tracking volunteers, board members and their
commitments to organizations and projects that are not run by your
Community Action Agency. This simply cannot be tracked. We would
not even try. 7. The new reports to the state required for each of
the 50 organizational standards, with additional analysis required
on any standards that your agency has not achieved in full [module
1.] More reporting time = 8 hours 8. The placement of data into
the state reporting software, including checking and answers to
the narratives required by the state for module 1, as well as the
local information on funds, people and projects. After reviewing
the format suggested for State use it will be time consuming to
hand enter this information unless the state shares the on-line
tool. Time to collect information will need to be on-going yearly
= 40 hours for reporter Writing narratives = 40-60 hours yearly 9.
The Cost of Time = in excess of $150,000. This change would
roughly entail ¼ of the Agency current staff time x salary
for Agency staff. O Based on the full cost of the different levels
of staff contribution time.
|
|
Comment 31 (Community Action
Commission Of Erie, Huron, & Richland Counties ; Eligible
Entity/Local Agency in N/A)
|
|
Community Action Commission of
Erie, Huron, & Richland Counties Inc. has determined that the
attached costs will be $53,418.31, which is 10.72% of our annual
CSBG allocation of $498,495. However, there is the compounded
aspect of systems compatibility that need to be factored and
cannot be ascertained until the actual implementation.
1.
Whether the proposed collection of information is necessary for
the proper performance of the functions of the agency, including
whether the information shall have practical utility. The proposed
collection of information is necessary for proper performance of
function of the agency in that it will enhance the ability to
effectively quantify outcomes and measure that are not currently
reflected due to some data reporting limitations in the ROMA
environment.
2. The accuracy of the agency’s
estimate of the burden of the proposed collection of information.
CACEHR anticipates the initial burden of decreased efficiency
attributed to learning curve and ability to transition into
competence with the updated environment. However, after initial
implementation actual burden instance is expected to level off or
decr4ase because of environment proficiency.
3. The
quality, utility, and clarity of the information to be collected;
and The quality, utility, and clarity of the collected information
will depend heavily on the new workflow of the environment. ROMA’s
new updates appear to focus on providing the ability to more
accurately provide pertinent detailed outcomes and measures
related to the four module protocols. (i.e. Mod 1-State
Administration, Mod 2-Agency Expenditure, Capacity, and Resources,
Mod 3-Communtiy Level, Mod 4-Individual and Family Level)
4.
Ways to minimize the burden of the collection of information on
respondents, including through the use of automated collection
techniques or other forms of information technology. This area is
difficult to adequately determine due to ROMA Next Generation
actual performance and data inputs not being available in
conjunction by agency personnel and sister agencies, as well as
lack of empirical result and outcome. More time is needed to gage
additional improvements.
|
|
Comment 33 (Community Action Council
; Eligible Entity/Local Agency in KY)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility.
The Council is in
favor of creating a universal data reporting system that
accurately demonstrates the outcomes of the work conducted by
CAAs. The Network would benefit from a comprehensive data system
that could show the true impact of the services CAAs provide to
local communities, and accurate, reliable data is the best way to
accomplish this outcome. However, the proposed reporting structure
is not feasible as existing funding is not sufficient to allow for
a complete overhaul of each CAA’s reporting system. The
proposed method of data collection will require each CAA to
develop a new database system—and hire additional front line
and management staff—to ensure data accuracy. If these
elements are not in place, the information that is reported will
be inaccurate and will not reflect the true outcomes of the CAAs
and the Network.
O The Council agrees with the
Partnership in that the package as proposed in its entirety is
overly burdensome, contains reporting tools that will provide
useless information, and is outside the reach of the Community
Services Block Grant (CSBG). However, with modifications, a system
for collection of information can be developed that improves the
performance management of the agency and has practical utility.
Detail is provided here outlining where changes need to be made to
reach a level of workability and utility.
O 1.1 –
The Council agrees with the Partnership in that State Agencies
should have the option to report the number of agencies at a
variety of thresholds that still delineate strong performance or
allow for a more accurate reflection of the training and technical
assistance needs. For instance, State Agencies should be able to
report the number of local agencies that have met 100%, 90%, 80%,
70%, 60%, 50%, and less than 50% of the standards. This will allow
for the collection of more accurate information; better tracking
of results over time; and better, more efficient investment of
training and technical assistance.
O 1.2 – The
Council agrees with the Partnership in that, in addition, many
agencies may either lack management information systems with the
capacity to disaggregate this type of data from its overall
customer database, or struggle to integrate the required data
given their use of multiple reporting systems. In addition, as
many local agencies serve families over time, with no definition
of “new” being provided for review, the CSBG Network
will be unable to implement this report consistently. Defining an
individual or family as new becomes challenging when they may
receive different services over many years.
2. The
accuracy of the agency’s estimate of the burden of the
proposed collection of information
o Concerns
1.
(2.1) – The Council agrees with the Partnership in that the
current IS survey includes 300+ data elements to report. The
proposed CSBG Annual Report includes more than 1,000. This
increased burden of reporting comes with no new resources.
2.
(2.2) – The Council agrees with the Partnership in that, to
comply with the draft, CSBG Annual Report local agencies will need
to modify existing systems.
3. (2.3) – The
Council agrees with the Partnership in that many agencies lack
management information systems with the capacity to provide the
required data. Data fields required for ROMA Next Generating
reporting may not be mirrored in those used by agencies. In
addition, many agencies use multiple reporting systems which add
to the time necessary to provide the data.
> Option
selected describing what the Council will need to report the
requested ROMA Data:
1. The agencies’ estimated
time burden of 242 hours per sub-grantee (local Community Action
Agency) is significantly under estimated. Based on our
calculations, the burden to our local agency would be 1440 hours.
The cost of this time burden is estimated to be $21,000.00.
>
The Council agrees with the Partnership in that sub-grantees will
need systems that track customers, services, and outcomes, and,
given the requirement of many funding sources, additional and
sometime separate data collection systems are required. State
grantees do not have the funding or capacity, for the most part,
to create systems or support local agencies to develop systems
that will streamline or decrease the burden outlined here. Staff
will need to be trained on the new online system.
3.
The quality, utility, and clarity of the information to be
collected; and
o Module 3, Section A: The Council
agrees with the Partnership in that, first, Community-level
indicators are influenced by a wide number of sources that would
most likely overwhelm the impact of even successful community
initiatives, a fact that significantly limits their utility for
providing meaningful information about outcomes. Secondly, any
meaningful data about the outcomes of community-level initiatives
would require a rigorous program evaluation and could not be
determined on the data collected through the Annual Report alone.
4. Ways to minimize the burden of the collection of
information on respondents, including through the use of automated
collection techniques or other forms of information technology.
It would be very beneficial to local CAAs to have a
universal system that is created and maintained at the national
level. In the current proposal, however, only states will have
access to the national database. As States simply compile the
information provided by local CAAs, data monitoring under the
current proposal will be less efficient. If local CAAs were given
access to the database, they could perform data entry, reducing
the burden on States and improving data accuracy and efficiency.
Local CAA access to a universal database system will streamline
the process by ensuring that all CAAs are using the same format
and data collection methods.
O The Council agrees with
the Partnership in that sub-grantees will still need systems that
track customers, services, outcomes, and, given the requirement of
many funding sources, requires additional and sometime separate
data collection systems. The Council further agrees with the
Partnership in that new state-level systems will do little to
address the needs of local agencies for improved management
information systems with the capacity to collect and report the
data required by ROMA Next Generation.
o The Council
agrees with the Partnership in that DHHS should examine its own
practices to ensure that all DHHS-funded programs in the field can
use data collection systems that can be easily integrated and
“talk” to one another to produce reports that meet the
needs of local agencies, states, and OCS. Data systems need to be
streamlined across all DHHS-funded programs.
|
|
Comment 34 (Community Action Council
Of Lexington-Fayette, Bourbon, Harrison And Nicholas Counties ;
Eligible Entity/Local Agency in KY)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility
The Council is in favor of
creating a universal data reporting system that accurately
demonstrates the outcomes of the work conducted by CAAs. The
Network would benefit from a comprehensive data system that could
show the true impact of the services CAAs provide to local
communities, and accurate, reliable data is the best way to
accomplish this outcome. However, the proposed reporting structure
is not feasible as existing funding is not sufficient to allow for
a complete overhaul of each CAA’s reporting system. The
proposed method of data collection will require each CAA to
develop a new database system—and hire additional front line
and management staff—to ensure data accuracy. If these
elements are not in place, the information that is reported will
be inaccurate and will not reflect the true outcomes of the CAAs
and the Network.
O The Council agrees with the
Partnership in that the package as proposed in its entirety is
overly burdensome, contains reporting tools that will provide
useless information, and is outside the reach of the Community
Services Block Grant (CSBG). However, with modifications, a system
for collection of information can be developed that improves the
performance management of the agency and has practical utility.
Detail is provided here outlining where changes need to be made to
reach a level of workability and utility.
O 1.1 –
The Council agrees with the Partnership in that State Agencies
should have the option to report the number of agencies at a
variety of thresholds that still delineate strong performance or
allow for a more accurate reflection of the training and technical
assistance needs. For instance, State Agencies should be able to
report the number of local agencies that have met 100%, 90%, 80%,
70%, 60%, 50%, and less than 50% of the standards. This will allow
for the collection of more accurate information; better tracking
of results over time; and better, more efficient investment of
training and technical assistance.
O 1.2 – The
Council agrees with the Partnership in that, in addition, many
agencies may either lack management information systems with the
capacity to disaggregate this type of data from its overall
customer database, or struggle to integrate the required data
given their use of multiple reporting systems. In addition, as
many local agencies serve families over time, with no definition
of “new” being provided for review, the CSBG Network
will be unable to implement this report consistently. Defining an
individual or family as new becomes challenging when they may
receive different services over many years.
2. The
accuracy of the agency’s estimate of the burden of the
proposed collection of information
o Concerns
1.
(2.1) – The Council agrees with the Partnership in that the
current IS survey includes 300+ data elements to report. The
proposed CSBG Annual Report includes more than 1,000. This
increased burden of reporting comes with no new resources.
2.
(2.2) – The Council agrees with the Partnership in that, to
comply with the draft, CSBG Annual Report local agencies will need
to modify existing systems.
3. (2.3) – The
Council agrees with the Partnership in that many agencies lack
management information systems with the capacity to provide the
required data. Data fields required for ROMA Next Generating
reporting may not be mirrored in those used by agencies. In
addition, many agencies use multiple reporting systems which add
to the time necessary to provide the data.
> Option
selected describing what the Council will need to report the
requested ROMA Data:
1. The agencies’ estimated
time burden of 242 hours per sub-grantee (local Community Action
Agency) is significantly under estimated. Based on our
calculations, the burden to our local agency would be 1440 hours.
The cost of this time burden is estimated to be $21,000.00.
>
The Council agrees with the Partnership in that sub-grantees will
need systems that track customers, services, and outcomes, and,
given the requirement of many funding sources, additional and
sometime separate data collection systems are required. State
grantees do not have the funding or capacity, for the most part,
to create systems or support local agencies to develop systems
that will streamline or decrease the burden outlined here. Staff
will need to be trained on the new online system.
3.
The quality, utility, and clarity of the information to be
collected; and
o Module 3, Section A: The Council
agrees with the Partnership in that, first, Community-level
indicators are influenced by a wide number of sources that would
most likely overwhelm the impact of even successful community
initiatives, a fact that significantly limits their utility for
providing meaningful information about outcomes. Secondly, any
meaningful data about the outcomes of community-level initiatives
would require a rigorous program evaluation and could not be
determined on the data collected through the Annual Report
alone.
4. Ways to minimize the burden of the collection
of information on respondents, including through the use of
automated collection techniques or other forms of information
technology.
It would be very beneficial to local CAAs
to have a universal system that is created and maintained at the
national level. In the current proposal, however, only states will
have access to the national database. As States simply compile the
information provided by local CAAs, data monitoring under the
current proposal will be less efficient. If local CAAs were given
access to the database, they could perform data entry, reducing
the burden on States and improving data accuracy and efficiency.
Local CAA access to a universal database system will streamline
the process by ensuring that all CAAs are using the same format
and data collection methods.
O The Council agrees with
the Partnership in that sub-grantees will still need systems that
track customers, services, outcomes, and, given the requirement of
many funding sources, requires additional and sometime separate
data collection systems. The Council further agrees with the
Partnership in that new state-level systems will do little to
address the needs of local agencies for improved management
information systems with the capacity to collect and report the
data required by ROMA Next Generation.
o The Council
agrees with the Partnership in that DHHS should examine its own
practices to ensure that all DHHS-funded programs in the field can
use data collection systems that can be easily integrated and
“talk” to one another to produce reports that meet the
needs of local agencies, states, and OCS. Data systems need to be
streamlined across all DHHS-funded programs.
|
|
Comment 35 (Community Action Council
Of Portage County; Other/Unknown in OH)
|
|
To Whom it May Concern:
The
Community Action Council of Portage County (Ohio) has a relatively
small CSBG grant of approximately $281,000 in FY 2016. We estimate
that the implementation of ROMA Next Gen would require an
investment of CSBG funding of approximately 15.4% of our total
CSBG allocation, or approximately $44,000 per year, assuming no
significant additional software, hardware, or related costs. This
amount reflects predominantly staff time. If the current scheduled
implementation date holds true, we estimate that it would require
an additional initial investment as well. Our conservative
estimate of this cost would approximate $12,000.
This
amount above assumes that our state CSBG office (Ohio Development
Services Agency) will be responsible for development of the
electronic data tracking system that will be needed for reporting
purposes, including bearing the costs to do so.
We
appreciate the opportunity to provide you this information.
Thank You.
|
|
Comment 37 (Community Action
Partnership Of Kern ; Other/Unknown in CA)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility Collection of data by individual
programs is necessary to evaluate the success of specific program
services and often required to justify funding for the program. In
many cases, individual programs are required to collect specific
data through specific data collection process and platforms.
Sharing of data is not always feasible or even allowable depending
on the program and federal parent agency. Collectively gathering
of data from multiple programs, with the goal of providing
unduplicated service counts is appealing, but not always possible.
Our agency has performed service distribution at a
high level for a long time under the current and previous
reporting systems. We recognize the importance of providing
unduplicated counts and family level measurement of movement
toward self-sustainability when possible. Requiring unduplicated
counts across all service types and through the combination of
data platforms are not always practical. As an example, linking
services provided to a family by CAPK VITA (Volunteer Income Tax
Assistance), CAPK Youth Services and the CAPK Food Bank will
require data from each user. Under the proposed reporting system,
the Food Bank will be required to obtain at least 9 pieces of
personal information during emergency food distribution efforts.
In any case, our Food Bank will be unable to report approximately
half of the food distribution customer base because they are
served by private food pantries where we are not involved in
direct distribution activity. If we are placed in a position of
requiring pantry partners to obtain this additional demographic
information from each client, we will lose many of our pantry
sites. The accompanying backlash and bad publicity would not look
good. We think this type of cascade effect could end up creating a
situation we are trying to avoid by implementing strict reporting
standards. Other question 1 responses:
· Module
1: State Agencies should be able to report the number of local
agencies that have met 100%, 70%, and less than 50% of the
organization standards, not just 100%. · Module 4: Remove
the Characteristics for NEW Individuals and Households Report.
many agencies may either lack management information systems with
the capacity to disaggregate this type of data from its overall
customer database, or struggle to integrate the required data
given their use of multiple (and sometimes incompatible) reporting
systems.
· Module 3: Having individual agencies
report on rate change at the community level on social indicators
does not provide information that is usable.
·
Module 4: Propose a suggested stability indicator scale where
appropriate and tailored to meet regional needs. A one size fits
rule does not apply in this case.
· Include a
4th ROMA Goal: Agencies Increase their capacity to achieve
results. Make it clear to network agencies that improvements are
expected to take place and CSBG dollars may be used to meet the
needs of ROMA Next Generation reporting and tracking requirements.
Recognize the potential for short term decreases in service at the
expense of making the agency improvements.
2. The
accuracy of the agency’s estimate of the burden of the
proposed collection of information If CAPK continues partial
funding of several programs with CSBG dollars, reporting and
tracking requirements associated with the new ROMA standards will
increase dramatically. Tracking clients after they receive
services is a large part of the reason for this increase. All CSBG
funded programs will require significant additional training prior
to compliance with new tracking and reporting requirements. The
time spent on this translates into the equivalent of more than a
full time position spread over those programs. A wide variety of
employees are involved with reporting and tracking. The OCS
estimated time burden of 242 hours per sub-grantee (local
Community Action Agency) is significantly under estimated. Based
on our calculations, the burden to our local agency would be
closer to 2000 hours. The cost of this time burden is estimated to
be $75,000. An agency of our size and complexity may be forced to
severely limit CSBG funding to one or two very specific programs
that are easy to report and track, and do not involve proprietary
data concerns.
Other question 2 responses:
·
The current IS survey includes 300+ data elements to report. The
proposed CSBG Annual Report includes more than 1,000. This
increased burden of reporting comes with no new resources.
·
Data fields required for ROMA Next Generating reporting may not be
mirrored in those used by agencies. In addition, CAPK is required
to use multiple (and sometimes incompatible) reporting systems
which significantly adds to the time necessary to provide the
data.
· The OLDC or online automated system may
benefit State Lead Agencies (grantees), but will most likely not
benefit local Community Action Agencies (sub-grantees).
Sub-grantees will still need systems that track customers,
services, outcomes, and given the requirement of many funding
sources (including other federal Department of Health and Human
Services funding) requires additional and sometime separate data
collection systems.
3. The quality, utility, and
clarity of the information to be collected; and
We
agree that unduplicated counts and tracking services are critical
to determining the effectiveness of agency services when
conducting an analysis of movement out of poverty. Reported data
will only be as good as the ability of programs to share the data.
If programs cannot readily share data, multiple redundant data
input systems will be required leading to additional time spent
(see costs associated with #2 above).
The additional
reporting and tracking requirements and heavy emphasis on specific
(guaranteed?) outcomes, may lead to a significant decrease in the
quantity of services provided to our community. In order to prove
that our quality of services will bring a family out of poverty,
the quantity of services provided to the public will decrease. If
this is the goal, please spell it out and make it clear so there
is no confusion in the future when agencies report back with far
fewer client contacts and drastic drops in some service areas.
Take for instance the estimated 90,000 or so unduplicated users of
the Food Bank. If our pantry partners do not agree to new data
collection, we could lose half of the people that traditionally
obtained CAPK Food Bank resources. We would be in a position of
reporting half as many clients receiving services. How will that
look in the short term? CAPK may be able to regain many of those
lost clients through expansion of our own distribution system, but
that could take years. The push for data on emergency food
distribution could backfire for larger agencies like CAPK and end
up making the State, Federal and network CSBG program look bad.
Everything about ROMA leads us to believe the goal is
to show how we bring people and families out of poverty. This will
require a much more targeted approach uniting all programs (ROMA
Next Generation), as opposed to past business as usual (silo
services). If this is truly the intent of this process, recognize
the anticipated short term decreases in service that are
unavoidable for larger agencies trying to meet these requirements.
Other question 3 responses:
·
Several elements of data to be collected under this proposal have
no meaning at the state and federal levels, and only have value,
meaning, and usefulness at the local level.
·
Comparing overall percentages of community indicators (e.g. the
unemployment rate, violent crime rate) to individual local
agencies outcomes is misleading.
· Module 4,
Section A: Characteristics for NEW Individuals and Households -
Data Entry
Form should be removed. The large number of
“one time” customers in programs such as LIHEAP will
skew any meaningful interpretation of the data. Aside from the
challenge of defining who constitutes a new customer, agencies
that use multiple reporting systems will also have the added
burden of tracking new customers across all programs and services.
· Module 3, Section A: Any meaningful data
about the outcomes of community-level initiatives would require a
rigorous program evaluation and could not be determined on the
data collected through the Annual Report alone.
4.
Ways to minimize the burden of the collection of information on
respondents, including through the use of automated collection
techniques or other forms of information technology.
We
already know that several major Federal programs do not allow easy
sharing of data for reporting outside of their individual program.
Since CAPK deals in many program services funded by many Federal
agencies (e.g. USDA, USHHS, USDOE, IRS), developing a data system
capable of collecting required data is not only cost redundant,
but cost ineffective. We are exploring the system requirements to
meet the reporting and tracking need and estimate it could cost as
much as $30,000. This is on top of the approximate $90,000
currently needed to maintain existing reporting software and
associated hardware.
One significant burden to larger
agencies is the need to provide full unduplicated reporting and
tracking of services for programs receiving even $1 of CSBG
funding. This requirement will effectively direct CSBG funding
into one or two programs, removing the flexibility of CSBG that is
so special to providers. The CAPK Food Bank is heavily funded by
CSBG dollars and will not be able to comply with the proposed
reporting and tracking standards without massive short term loses
in clients and partner organizations. CAPK WIC may only receive a
small amount of CSBG funding during a year, but meeting reporting
and tracking issues create a burden. Consider a start point for
funding levels associated with suggested reporting and tracking,
maybe when a program exceeds 49% CSBG funding.
·
To minimize the burden on respondents, DHHS should examine its own
practices to ensure that all DHHS-funded programs in the field can
use data collection systems that can be easily integrated and
“talk” to one another to produce reports that meet the
needs of local agencies, states, and OCS.
|
|
Comment 38 (Community Action
Partnership Of Orange County; Eligible Entity/Local Agency in CA)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility
o The practical utility
of the information is unknown at this time. Much of the data is
collected for other funding sources utilizing reporting forms and
systems mandated by those funders. The rollout for Community
Action Agencies should be on a pilot basis that includes
representation of large and small CAAs, urban & rural, public
& private, etc.: rather than requiring the full implementation
by the whole network. We simply don’t know how well ROMA NG
will work. Full implementation will lead to constant disagreement
and tinkering nationwide in an attempt to implement changes and
corrections (meaning lots of time & money).
2. The
accuracy of the agency’s estimate of the burden of the
proposed collection of information
The agency’s’
estimated time burden of 242 hours per sub-grantee (local
Community Action Agency) is significantly under estimated. Based
on our calculations, the burden to our local agency would be 3 FTE
Staff (approximately 6,000 hours).The cost of this time burden is
estimated to be approximately $150,000. The estimate is based on a
presumption that 3 data/evaluation coordinators will be hired; 1
for each of the agency’s 3 operations departments. Estimate
for development of a computer program for centralized intake and
coordinated reporting is for a one time purchase from $60,000 to
$80,000 (plus unspecified ongoing annual maintenance costs). The
centralized intake form will be in addition to the actual
application form required by specific programs. This will be
necessary to reliably collate data from 16 major programs in
addition to CSBG. Most of these programs have their own unique
reporting systems which are not easily transferable or directly
compatible with the CSBG reporting.
3. The quality,
utility, and clarity of the information to be collected; and
o
We anticipate differences in definitions from program to program
which may complicate the reporting process.
O The
information/data for the proposed report is technical and
complicated. Initially, it would be better to simplify reporting
and systems rather than adding complicated layers and categories.
As better understanding and acceptance occurs, we can improve
reporting accuracy and sophistication, over time, helping to
ensure competency and compliance throughout the network.
4.
Ways to minimize the burden of the collection of information on
respondents, including through the use of automated collection
techniques or other forms of information technology.
Decrease
and simplify the number of proposed reporting elements. We will
attempt to be responsive with information for all programs
however; we do not currently collect all the data necessary to
complete all categories identified for reporting.
Develop
an efficient and cost effective way to report and utilize data
already collected for other federal and standardized programs in
their current format.
|
|
Comment 40 (Community Action
Partnership Of San Luis Obispo County, Inc.; Eligible Entity/Local
Agency in CA)
|
|
I am writing as CEO of
Community Action Partnership of San Luis Obispo County, Inc.
located in central California. I serve on the Partnership National
Board and serve as Third Vice President currently. I am also a
CALCAPA member here in California. You will be receiving
communication from both organizations in addition to NCAF,
community action's advocacy organization.
CAPSLO has
been readying our agency for the past 3 years in data collection
improvement. We currently use a combination of ClientTrac and
ChildPlus in combination. I am concerned that many agencies in the
Network are not equipped for the robust data collection system
without additional funding. We need a year simultaneously to see
how the states and agencies at the local level come through
Organizational Standards.!
I know the rationale behind
taking out "Agency Goal". I believe and have so stated
publicly that not all agencies are ready to move forward with this
new reporting and all the requirements. Agencies need
infrastructure goals and boards need to be on top of this new
phase in community action. Please leave Agency Goal in as Goal
#4.
Lastly, much training is needed at local and state
levels to do this ROMA NG well! Please include a pilot
phase...many of us in the field would volunteer...my Board
supports CAPSLo volunteering for this huge opportunity. I am
concerned to know how the state agencies nationwide are going to
be trained for this endeavor. The readiness varies tremendously
among the 50 states and territories.
· We have
major concerns about the States being able to train/educate the
CAPs on how to properly report out with these new requirements.
We've been reporting on the existing NPIs for a long time, and
when we have a question re the NPIs it is hard to get a definitive
answer from our state agency on the proper way to report out. From
our past experience, the answer can change from one field rep to
the next based on how they interpret the category. With the new
proposed ROMA Next Gen report being more complicated, I'm
concerned about how this will roll-out and how much information
they will have in order to train/educate us on how to properly
report out. There will need to be in depth written instructions
(with examples) from the Feds, and trainings from State/Fed reps
to explain how to complete the reporting. I have concerns with
"collective Impact" that I will defer to the Partnership
comments.
Thank you for taking all of the input from
the field! I look forward to the next phase of review!
|
|
Comment 41 (Community Action Planning
Council Of Jefferson County, Inc.; Eligible Entity/Local Agency in
NY)
|
|
Thank you for allowing our
Community Action Network to provide comments on the proposed OMB
PRA Clearance FRN #1. We applaud you for striving to improve our
performance management system.
Community Action
Planning Council of Jefferson County, NY (CAPC) has reviewed the
draft documents for ROMA Next Generation and the CSBG Annual
Report and would like to make the following comments. We believe
the progress made to date since the January 2011 is significant,
and we want to keep moving forward. We want to get the development
work behind us so we can move forward together with the
implementation of ROMA Next Generation. In general, the
requirements as written do not lend themselves to value at all
levels; they are not feasible given limited financial resources,
technological capacity, diversity of the Network (small/large,
public/private, urban/rural/suburban), and limited T/TA fund. Our
agency feels that requirements will place a significant undue
burden on us as a local agency in staff time and costs associated
with reporting.
Please find below our detailed
comments:
Our concern, as well as our entire networks,
is focused in five core areas including:
• Module
1: Reporting on CSBG Organizational Standards and Technical
Assistance Plans/Quality Improvement Plans for local agencies with
unmet standards.
• Module 3: Community Outcome
Indicators and Collective Impact
• Module 4:
Characteristics for NEW Individuals and Households
•
Module 4: Stability Measures/Indicators
• Modules
3 and 4-ROMA Goals: Lack of inclusion of an Agency Goal
Question
1. Whether the proposed collection of information is necessary for
the proper performance of the functions of the agency, including
whether the information shall have practical utility.
While
there are elements that may be helpful, there are a number of
areas detailed below where the collection of information does not
meet the threshold of necessity or utility. The package as
proposed in its entirety is overly burdensome, contains reporting
tools that will provide useless information, and is outside the
reach of the Community Services Block Grant (CSBG). However, with
modifications, a system for collection of information can be
developed that improves the performance management of the agency
and has practical utility. Detail is provided here outlining where
changes need to be made to reach a level of workability and
utility.
1. Module 1: Modify the options CSBG State
Lead Agencies/grantees have when reporting to OCS on CSBG
Organizational Standards to allow for a broader range of
responses. As noted below, there are significant challenges with
the way this information is being reported and creates a misuse of
the Organizational Standards. Expecting constant perfection is
unrealistic and does not have practical utility for local agencies
or the State CSBG Lead Agency. As proposed, the single metric the
State Agencies will report on will be the number of eligible
entities that meet 100% of the CSBG Organizational Standards. To
increase utility of the information, State Agencies should have
the option to report the number of agencies at a variety of
thresholds that still delineate strong performance or allow for a
more accurate reflection of the training and technical assistance
needs. For instance State Agencies should be able to report the
number of local agencies that have met 100%, 90%, 80%, 70%, 60%,
50%, and less than 50% of the standards. This will allow for the
collection of more accurate information; better tracking of
results over time; and better, more efficient investment of
training and technical assistance.
2. Module 4: Remove
the Characteristics for NEW Individuals and Households Report.
This report has no practical utility at the local level given the
time and expense to create it at each local agency. This is not
necessary for agency performance, and the intentions outlined by
OCS for its use are not reflective of how local agencies and
states look at their communities to assess needs, develop
programs, and report progress. It also lacks utility at the state
level given that data such as this rolled up to the state level
will be skewed by large population centers and decrease the
practical utility of the data even further. In addition, many
agencies may either lack management information systems with the
capacity to disaggregate this type of data from its overall
customer database, or struggle to integrate the required data
given their use of multiple (and sometimes incompatible) reporting
systems. In addition, as many local agencies serve families over
time, with no definition of “new” being provided for
review, the CSBG Network will be unable to implement this report
consistently. Defining an individual or family as new becomes
challenging when they may receive different services over many
years (e.g. Early Head Start, child care, after school programs,
youth programs, job training, WIC, food boxes, LIHEAP, etc.).
3.
Module 3: Remove the Community Outcome Indicators that include
percent or rates. This inclusion is inappropriate and will provide
meaningless data when aggregated beyond individual initiatives.
Such data has validity for geographic areas much larger in than
those addressed or targeted by local agency projects. This
validity is lost when the scale is smaller and accepted data
comparison points become unavailable. Having individual agencies
report on rate change at the community level on social indicators
does not provide information that is usable. It is more
appropriate to give agencies the option of reporting outcome data
and the flexibility to define which indicators they use.
4.
Module 3: Remove Collective Impact from all reports. Collective
Impact is simply one set of strategies for doing community-level
work and while CSBG eligible entities are engaged in such efforts,
privileging one approach given the broader range of options for
doing community-level work is inappropriate and unnecessary. This
should be removed and included in training and technical
assistance endeavors.
5. Module 4: Remove the Stability
Indicators. The indicators selected and included in the clearance
package are not helpful or practical when reporting customers
stabilized as a result of the eligible entity’s work. One
option duplicates data already collected and the second uses a
tool that defines financial stability based on a construct the
uses assumptions drawn from the general population, not low-income
individuals and families specifically. Because financial well-
being means something very different for the low-income population
in comparison to the general population, use of the scale will
produce data of limited utility.
6. Address the unique
issues of Public entities that receive CSBG. Public agencies
(sub-grantees) that sub-contract most or all of their CSBG funding
to other service providers may face added challenges of requiring
their grantees to collect data in accordance with ROMA Next
Generation requirements since it may limit agencies to contracting
with providers that have the requisite data collection capacity or
require additional technical assistance to meet the reporting
requirements. This challenge is not addressed in the proposed
reporting package.
7. Include a 4th ROMA Goal: Agencies
Increase their capacity to achieve results. Reinstate
this
national goal that addressed the critical role the
network of local agencies serving 99% of US counties plays in
addressing poverty. Reasons for exclusion have been inadequate to
date. Because agency capacity is critical to the ultimate success
of programs and services, it is important to clearly state this
objective in the overall National Goals. This also highlights the
broader need of the Network for ongoing training, technical
assistance, and general capacity- building supports.
Question
2: The accuracy of the agency’s estimate of the burden of
the proposed collection of information.
Community
Action is very concerned about the additional staff time and
training for the data collection proposed.
•
Concerns include:
1. The current IS survey includes
300+ data elements to report. The proposed CSBG Annual Report
includes more than 1,000. This increased burden of reporting comes
with no new resources.
2. The burden on local agencies
is impacted by the implementation of the Online Data Collection
system (OLDC). To comply with the draft CSBG Annual Report local
agencies will need to modify existing systems.
3. Many
agencies lack management information systems with the capacity to
provide the required data. Data fields required for ROMA Next
Generating reporting may not be mirrored in those used by
agencies. In addition, many agencies use multiple (and sometimes
incompatible) reporting systems which adds to the time necessary
to provide the data. Our agency, CAPC, has recently signed a
contract with a Management Information for-profit company that we
felt had the capacity to help us report outcomes and demographics
for our programs. We have been conducting research for 8 years to
find the closest match in reporting in our programs and to raise
funds to purchase any system. If we are required to meet the
proposed requirements as they have been written and need to
include additional/revised tracking methods, it will add to the
already significant cost of a new data system; and add to the
burden of staff to ensure the correct information is reported.
4.
The proposed estimated time burden of 242 hours per local
Community Action Agency is profoundly under estimated. Based on
our calculations, the burden to CAPC would be at minimum, 1,280
hours. The cost of this time burden is estimated to be $26,880.
This calculation is based on an annual minimum of 80 hours per
each program plus 160 hours Administrative costs. This does not
include any time spent with our new software company.
•
In addition, it is not clear how the burden for local agencies is
decreased by the online data collection system described in the by
OCS. OCS’ Dear Colleague Letter of June 17, 2016, states,
“Aware of the reporting burden the proposed Annual Report
will entail, OCS will decrease that burden by establishing an
on-line automated system for use by the states and – at the
discretion of the states – the local agencies, for reporting
that allows in many instances auto-populating data from one year
to the next and a variety of data entry processes (e.g. manual
entry and automatic upload).” It is important to note, this
online automated system may benefit State Lead Agencies
(grantees), but will most likely not benefit local Community
Action Agencies (sub-grantees). Sub-grantees will still need
systems that track customers, services, outcomes, and given the
requirement of many funding sources (including other federal
Department of Health and Human Services funding) requires
additional and sometime separate data collection systems. State
grantees do not have the funding or capacity for the most part, to
create systems or support local agencies to develop systems that
will streamline or decrease the burden outlined here.
Question
3: The quality, utility, and clarity of the information to be
collected
As noted below in the detailed comments by
module, several elements of data to be collected under this
proposal have no meaning at the state and federal levels, and only
have value, meaning, and usefulness at the local level.
•
Several of the elements to be collected as described are not
available in smaller geographic areas usually targeted by local
agencies in their community work and if created by local agencies,
lose validity. The social or population level indicators being
requested are not meant for single agency reporting and are best
served for community-wide scorecards or utilized as part of a
community needs assessment. For example, comparing overall
percentages of community indicators (e.g. the unemployment rate,
violent crime rate) to individual local agencies outcomes is
misleading.
• Module 4, Section A: Characteristics
for NEW Individuals and Households - Data Entry Form is an
unnecessary document that will not produce quality data; it will
have little utility at a local level; and it lends itself to
meaningless data at a state and national level. It should be
removed. Because demographic data alone does not include
information about the needs of new customers, this information
fails to help agencies determine if they are addressing issues
identified in their community assessments. Furthermore, the large
number of “one time” customers in programs such as
LIHEAP will skew any meaningful interpretation of the data. Aside
from the challenge of defining who constitutes a new customer,
agencies that use multiple reporting systems will also have the
added burden of tracking new customers across all programs and
services.
• Module 3, Section A: Many of the
indicators used here are inappropriate for reporting by local
sub-grantees to States and by States to OCS and will produce data
of limited utility. Several challenges underscore this point.
First, community-level data on indicators is not always available
at the level (e.g. regional or zip code/census tract) or
population (e.g. low-income children aged 0-5) that matches the
area and population targeted by the agency. Second,
community-level indicators are influenced by a wide number of
sources that would most likely overwhelm the impact of even
successful community initiatives, a fact that significantly limits
their utility for providing meaningful information about outcomes.
Third, any meaningful data about the outcomes of community-level
initiatives would require a rigorous program evaluation and could
not be determined on the data collected through the Annual Report
alone.
Question 4: Ways to minimize the burden of the
collection of information on respondents, including through the
use of automated collection techniques or other forms of
information technology.
As noted above, OCS’ Dear
Colleague Letter of June 17, 2016, states, “Aware of the
reporting burden the proposed Annual Report will entail, OCS will
decrease that burden by establishing an on-line automated system
for use by the states and – at the discretion of the states
– the local agencies, for reporting that allows in many
instances auto-populating data from one year to the next and a
variety of data entry processes (e.g. manual entry and automatic
upload).” It is important to note, this online automated
system may benefit State Lead Agencies (grantees), but will most
likely not benefit local Community Action Agencies (sub-grantees).
Sub-grantees (local agencies) will still need systems that track
customers, services, outcomes, and given the requirement of many
funding sources (including other federal
DHHS Services
funding) requires additional and sometime separate data collection
systems. State grantees do not have the funding or capacity for
the most part, to create systems that will streamline or decrease
the burden outlined here. Furthermore, new state-level systems
will do little to address the needs of local agencies for improved
management information systems with the capacity to collect and
report the data required by ROMA Next Generation.
•
OCS and the federal government provide funding to local
sub-grantees that require separate systems and prohibit the
aggregation of data. To minimize the burden on respondents, DHHS
should examine its own practices to ensure that all DHHS-funded
programs in the field can use data collection systems that can be
easily integrated and “talk” to one another to produce
reports that meet the needs of local agencies, states, and OCS.
|
|
Comment 43 (Community Renewal Team ;
Eligible Entity/Local Agency in CT)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility
As a community action
agency leader, there are several areas detailed in the following
sections where the collection of information proposed does not
meet what we need to collect and has little to no usefulness. The
proposed package is overly burdensome and is requiring the agency
to collect information outside the reach of CSBG and data that we
need to conduct our business operations.
We share your
commitment in cultivating improved performance management. In
addition we also agree that state and national reports must show
the impact of the CSBG network and demonstrate results as well.
We propose the changes as follows:
a.
Module 1: Modify the options CSBG State Lead Agencies/grantees
have when reporting to OCS on CSBG Organizational Standards to
allow for a broader range of responses.
There are
significant challenges with the way this information is being
reported and potential creates a misuse of the Organizational
Standards. As proposed, the metric the State Agencies will report
on will be the number of eligible entities that meet 100% of the
CSBG Organizational Standards. State agencies should have the
option to report the number of agencies at a variety of thresholds
(100%, 90%, 80%, 70%, 60 %...) that show strong performance or the
training and technical needs of the agencies. Presenting this type
of information in this way will reflect a more accurate picture of
agency performance will allow for better tracking over time and
assist the State agencies in more efficient investment of training
and technical assistance.
b. Module 4: Remove the
Characteristics for NEW Individuals and Households Report.
There
is not practical use at the local level given the time and expense
to create it at our agency. This information is not necessary for
agency performance and not reflective of how local agencies and
states look at their communities to assess needs, report progress
and develop new programming. In addition, we serve families over
time. With no definition of “new” being provided for
review, it will be difficult to implement this report
consistently. It becomes even more challenging when individuals or
families received different services over many years (Early Head
Start, Head Start, Youth Programs, LIHEAP, etc.)
c. Module
3: Remove the Community Outcome Indicators that include percent or
rates.
Inclusion of the rates in the Community Outcome
indicators is inappropriate and will provide meaningless data when
aggregated beyond individual initiatives. Having individual
agencies report on rate change at the community level on social
indicators does not provide information that is usable. It is more
appropriate to give agencies the option of reporting outcome data
and the flexibility to define which indicators they use.
d.
Module 3: Remove Collective Impact from all reports. Collective
Impact is simply one set of strategies for doing community-level
work. Giving priority to one approach when there are other options
for doing community level work is not necessary. We recommend
removing this and including it in the training and technical
assistance section.
e. Module 4: Remove the Stability
Indicators. The indicators selected and included in the clearance
package are not helpful or practical when reporting customers
stabilized as a result of an agency’s work. The first option
duplicates data already being collected. The second option which
uses a tool (CFPB Financial Well Being Scale) to define financial
stability, using assumptions drawn from the general population,
but not specifically from low income individuals and families.
Financial well-being has different meaning for the low-income
population in comparison to the general population.
f.
Include a 4th ROMA Goal: Agencies Increase their capacity to
achieve results. Reinstate this national goal that addressed the
critical role the network of local agencies serving 99% of US
counties plays in addressing poverty. Increasing agency capacity
is a very critical factor for achieving success in our programs
and services.
2. The accuracy of the agency’s
estimate of the burden of the proposed collection of information
We have grave concerns regarding the burden of the
proposed collection of information in light of a number of
factors:
a. The current
Information Services survey includes approximately 300+ reporting
data elements. The proposed CSBG Annual Report includes more than
1,000 data elements. This significantly increases the reporting
burden without providing new resources to assist the agencies in
meeting this requirement.
b. The burden on local
agencies is not impacted by the implementation of the On-Line Data
Collection System. To comply with this would require significant
modifications to our existing system.
c. Our
management information system currently has the data elements
required for CSBG IS reporting and some additional elements we as
an agency have chosen to collect. To comply with ROMA NG Annual
Report requirements would require significant (and costly)
modifications to our existing system. In addition, our agency uses
multiple and (and sometimes incompatible) funder required
reporting systems which adds significant time to collect and
report the data.
CRT’s Estimated Time Burden
The agency’s’ estimated time burden of 242
hours per sub-grantee (local Community Action Agency) is
significantly under estimated. Based on our calculations, the
burden to our local agency would be 72,086 hours. The cost of this
time burden is estimated to be $2,772,282.
CRT is a
large community action agency – with a workforce of 700
employees, annual budget of $60 million; annual client service of
44,256 families and 112,206 individuals and around 60 different
programs. Because of funder mandates and complexity of program
services, the agency utilizes multiple databases to comply with
program regulations and annual reporting requirements. Integration
of these myriad of systems makes annual reporting more timely and
costly. Increased reporting requirements and follow-up will impact
our program and staff resources greatly.
3. The
quality, utility, and clarity of the information to be collected
As noted below and in the detailed comments by module,
several elements of data to be collected under this proposal have
no meaning at the state and federal levels, and only have value,
meaning, and usefulness at the local level.
a. Several
elements to be collected are not readily available and are the
result of community wide efforts and are best served for
community-wide scorecards or utilized as part of a community needs
assessment. For example, comparing overall percentages of
community indicators (e.g. the unemployment rate, violent crime
rate) to individual local agencies outcomes is misleading
b.
Module 4, Section A: Characteristics for NEW Individuals and
Households - Data Entry Form is an unnecessary document that will
not produce quality data; it has little utility at an agency/local
level; and it lends itself to meaningless data at a state and
national level. It should be removed. The large number of “one
time” customers in programs such as LIHEAP will skew any
meaningful interpretation of the data. Aside from the challenge of
defining who constitutes a new customer, our agency uses multiple
reporting systems and it will have the added burden of tracking
new customers across all programs and services.
c.
Module 3, Section A: Many of the indicators used here are
inappropriate for reporting by local sub-grantees to States.
First, community-level data on indicators is not always available
at the level (e.g. regional or zip code/census tract) or
population (e.g. low-income children aged 0-5) that matches the
area and population targeted by the agency. Second,
community-level indicators are influenced by a wide number of
sources that would most likely overwhelm the impact of even
successful community initiatives, a fact that significantly limits
their utility for providing meaningful information about outcomes.
Third, any meaningful data about the outcomes of community-level
initiatives would require a rigorous program evaluation and could
not be determined on the data collected through the Annual Report
alone.
4. Ways to minimize the burden of the
collection of information on respondents, including through the
use of automated collection techniques or other forms of
information technology.
a. While this online automated
system may benefit State Lead Agencies it will not be likely to
benefit CRT as a sub grantee. We will still need systems that
track customers, services, and outcomes. Given that many of our
funding sources requires additional and sometime separate data
collection systems and reporting systems that also increases the
complexity of data collection.
b. To minimize the
burden on respondents, DHHS should examine its own practices to
ensure that all DHHS-funded programs in the field can use data
collection systems that can be easily integrated and “talk”
to one another to produce reports that meet the needs of local
agencies, states, and OCS.
|
|
Comment 45 (Del Norte Senior Center,
Inc.; Eligible Entity/Local Agency in CA)
|
|
I am writing to respectfully
submit my comments regarding the proposed Community Services Block
Grant (CSBG) reporting requirements familiarly known as “ROMA
Next Generation.”
While I am specifically
providing feedback on behalf of my own agency, I feel that my
concerns are relevant to any CSBG-funded agency, especially one as
small as mine.
Initially,
I would like to comment that the development of the proposed
reporting requirements was done very much without the ongoing
input or participation of the individual agencies that make up the
Community Action Network. Unlike the development of the
organizational standards, which were created in a very open,
inclusive and responsive process, the ROMA Next Generation product
is the result of a closed and secretive process that has been
completely unresponsive to the legitimate concerns of the CSBG
network. I do not feel the agencies that are going to be bound by
these cumbersome and unworkable requirements were given any
meaningful opportunity to influence the outcome.
Overall,
the ROMA Next Generation requirements have all the earmarks of
having been developed primarily by theoreticians and researchers.
The new indicators, demographic data collection and follow-up
requirements appear to be a wish list of everything that it would
be nice to know about individuals in poverty and the services they
access. No thought appears to have
been given as to whether
there is a realistic possibility that the data could actually be
collected.
Even rigorous scientific research only uses
sample data. We as CSBG entities are being asked to collect
research-quality data on every single person who is touched by
CSBG funds. It is simply
outside of the mission of the
program and beyond the capacity of most agencies to achieve.
Thank you for this opportunity to comment, though it
feels much too late in the process. My specific comments on
elements of the proposed requirements are as follows:
1.
Whether the proposed collection of information is necessary for
the proper performance of the functions of the agency, including
whether the information shall have practical utility.
The
level of detail being required in both the community level
initiatives and the family/individual NPI characteristics, and the
amount of information required to be collected from beneficiaries
is of limited use in the day-to-day functioning of the agency. It
is not necessary for proper performance and has little to no
practical utility.
Programs for which a significant level of
detail is intrinsically necessary to provide the service will
already have the collection of that data built into them. Programs
that do not need extensive client demographic or follow-up
information will be unduly hindered by excess data collection and
reporting requirements. CLIENTS WILL AVOID SERVICES AND AGENCIES
WILL AVOID PARTNERING WHEN FACED WITH THE HIGH
AMOUNT OF DATA
COLLECTION REQUIRED.
The existing NPI’s and
client characteristics reports allow for sufficient data
collection to inform the agency’s planning and performance
evaluation activities and to provide program effectiveness data
for federal purposes. There is no practical utility to having to
report the level of detail being contemplated to a state or
federal agency.
2. The accuracy of the agency’s
estimate of the burden of the proposed collection of
information
The agency’s cost of the reporting
burden of the proposed system is significantly underestimated,
even for my small agency with no current sub recipients. As has
been made
abundantly clear in the comments that have been
made to date, there is no single, unifying data collection system
that can accurately and efficiently record and report the data
currently
being required, much less the much more detailed
demographic, follow-up and narrative information being proposed.
Our agency currently tracks CSBG data by hand, using client
sign-in sheets and manual calculations. In an effort to reduce the
reporting burden, we have sought out proposals for client tracking
databases. Our agency is relatively simple, yet we still use three
separate client tracking databases that report information across
four distinct programs.
The estimated cost to purchase
and set up a database system that would allow collection of
unduplicated client information and integrate with the systems
already in place was between $85,000 and $100,000 for my agency
alone. Spending that much on a data collection system is
completely unreasonable, given that this amount could fund half of
my entire Senior Nutrition program for a full year. If CSBG funds
were used to fund sub recipients, the cost would be similar for
each agency added to the system.
Absent a single
unifying database, the burden to report data to every program in
which a client could participate, then re-report it to CSBG
multiplies. The table below illustrates our estimated labor burden
to process our existing service population through all of the
programs in which they participate, and to re-enter that
information for CSBG’s proposed reporting
requirements.
Total Hours Hourly Staff Rate
Cost
Initial Intake 1,221 21.56 $ 26,324.76 $
Service
Contacts 10,188 12.55 $ 127,853.88 $
Duplicate Entry
900 12.55 $ 11,295.00 $
Program Reporting 200 28.33 $
5,666.00 $
Total Hours 12,509 171,139.64 $
TOTAL
DATA COLLECTION AND REPORTING BURDEN
Assumes 30 minutes
intake for each client annually and approximately 5 minutes of
initial data entry per client service event. *Duplicate Entry
estimates the hours required to enter data into a new CSBG data
collection system that has already been entered into the data
collection systems of other programs.
3. The quality,
utility, and clarity of the information to be collected; and
The quality and clarity of
the information to be collected is questionable. Most clients and
staff persons are likely to accurately view the data collection
process as simply a bureaucratic barrier to receiving and
providing services. The utility of collecting and reporting
individually identifiable data for every client for every service
is also questionable. As was
stated in earlier comments, even
pure research activities use sample data. Requiring
research-quality data on every client is unreasonable when our
primary goal is not research, but service delivery.
4.
Ways to minimize the burden of the collection of information on
respondents, including through the use of automated collection
techniques or other forms of information technology.
The
people most in need of services to alleviate poverty are also the
least likely to be able to access and use automated data
collection. In addition, in an atmosphere where many programs are
trying to minimize the barriers to service access, the collection
of personally identifiable information is seen by many clients as
being intrusive, especially among the hard to reach homeless and
those with mental health challenges. Asking questions about the
sources of family income, race, employment status, household
composition, etc. will often make people turn away from
assistance.
|
|
Comment 46 (Delaware Opportunities
Inc.; Eligible Entity/Local Agency in DE)
|
|
I. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility.
Module 3: Remove the
Community Outcome Indicators that include percent or rates.
Percent and rate of what? Without this information, (i.e. what is
the base...where is it defined...when or where is it consistent)
the result is meaningless. It will not mean much to us at the
local level and when the data is aggregated at the state or
national level it will mean absolutely nothing.
Module
4: Remove the Characteristics for NEW Individuals and Households
Report. This report has no practical utility at this agency. We
lack the data collection and reporting software to maintain the
report and the cost for doing so is expected to be substantial
initially and ongoing. We do not feel that the collection and
reporting of this data will have any impact on our performance in
the short or long term. We have not yet even grasped how to define
one client as "new" and another as continuing. Our data
system has households going back to at least 1990. If we served
them as a Head Start child then and they are now applying for HEAP
are they "new" now or were they "new"
then?
2. The accuracy of the agency's estimate of the
burden of the proposed collection of information
We
believe that the estimate of the burden is substantially
understated. First, it does not recognize that our existing data
collection and reporting system will need to be vastly expanded or
newly created. We anticipate that for the first year we will
require an expenditure of approximately $10,000 in CSBG funds
just
for the data collection and reporting tool plus a first year
maintenance support cost of $6,000 or $16,000. We expect that
there will be an ongoing cost of at least $6,000 each year for the
maintenance of the new tool. The reporting tool referenced for use
by the state will have limited and probably negligible use by an
agency such as ours even if such a tool is eventually made
available. All of the fields in such a tool would have to be
uploaded from our local data base which would have to have the
same fields. Hence the collection of data, organization, and
reporting functions would need to be finished at the local level
prior to uploading to a (probably superlative) reporting tool.
Beyond this, we see a data collection, data entry, and data
reporting requirement at 3 minutes per individual served. For this
organization, this translates to 3 minutes times 10,000
unduplicated persons = 30,000 minutes or 500 hours. This does not
include service log entries, and staff training. Ata minimum, this
translates to an extra burden of $15, 188 (Average hourly wage of
data collection/reporting staff @$22.50 plus fringe benefits at
35% = $30.375 x 500 hours), plus the ongoing $6,000 indicated
above. For the first year this translates to a cost of $31,188 or
13% of our CSBG annual budget and Or about 9% charge of our
current CSBG budget (more if there is sequestration or other
budgetary reductions). The items to be tracked have increased by
330%. It has to be recognized that there is an additional cost
associated with this and it is unlikely that anyone wants to
increase budgets by upwards of 10% solely for administrative
purposes. The result will be a reduction in services that the data
collection is trying so hard to report upon.
3. The
quality, utility, and clarity of the information to be collected;
and
Module 4, Section A: Characteristics for NEW
Individuals and Households — The data entry form, we believe
is unnecessary, will produce no useful information and will be
costly to implement. As noted earlier, the definition of "new"
is subject to local interpretation and given the ambiguity of the
term can
produce no information at the state or national
level. Because demographic data alone does not include information
about the needs of new customers, this information fails to help
us determine if we are addressing issues identified in our
community assessments. Furthermore, the large number of "one
time" customers in programs such as LIHEAP will skew any
meaningful interpretation of the data. Aside from the challenge of
defining who constitutes a new customer, we already use multiple
reporting systems and will have the added burden of tracking new
customers across all programs and services.
4. Ways to
minimize the burden of the collection of information on
respondents, including through the use of automated collection
techniques or other forms of information technology.
The
OCS comments which accompany the reporting tool indicate that the
reporting burden on states will be lessened with the availability
of an online reporting mechanism. That system, if allowed by New
York State for use by us, will have little to no utility or
purpose unless we develop a system which actually collects,
aggregates, and summarizes in fields suitable for uploading to the
online system. All of the burden for this will remain with us, the
local agency. The auto populating feature OCS describes may be
useful for State reports, but the ability to auto populate data on
an agency by agency basis is probably not present. Since local
agencies report to the State, the online system would have to be
the State's system for data collection, or is it being proposed
that this online reporting feature would be present for local
agencies to use and obviate the need for the state collection and
aggregation of data?
|
|
Comment 51 (Dickinson Community
Action Partnership ; Other/Unknown in ND)
|
|
Community Action Partnership
of Dickinson, North Dakota (CAP) is respectfully submitting the
following general comments regarding the CSBG Annual Report OMB
Clearance Process. The overall opinion of our Agency regarding
ROMA NG is that it is overly and unnecessarily burdensome,
expensive, and does very little to enhance the experience of
clients and their efforts to obtain self-sufficiency
While
there are elements that may be helpful, there are a number of
areas detailed below where the general collection of information
does not meet the threshold of necessity or utility. The package
as proposed in its entirety is overly burdensome, contains
reporting tools that will provide useless information, and is
outside the reach of the Community Services Block Grant (CSBG).
However, with modifications, a system for collection of
information can be developed that improves the performance
management of the agency and has practical utility. Detail is
provided here outlining where changes need to be made to reach a
level of workability and utility.
Address the unique
issues of Public entities that receive CSBG. Public agencies
(sub-grantees) that sub-contract most or all of their CSBG funding
to other service providers may face added challenges of requiring
their grantees to collect data in accordance with ROMA Next
Generation requirements since it may limit agencies to contracting
with providers that have the requisite data collection capacity or
require additional technical assistance to meet general, agency
goal the reporting requirements. This challenge is not addressed
in the proposed reporting package.
Include a 4th ROMA
Goal: Agencies Increase their capacity to achieve results.
Reinstate this national goal that addressed the critical role the
network of local agencies serving 99% of US
counties plays in
addressing poverty. Reasons for exclusion have been inadequate to
date. Because agency capacity is critical to the ultimate success
of programs and services, it is important to clearly state this
objective in the overall National Goals. This also highlights the
broader need of the Network for ongoing training, technical
assistance, and general capacity-building supports.
2.
The accuracy of the agency's estimate of the burden of the
proposed collection of information Regarding the accuracy of the
agency estimate of the cost burden, CAP concerns include:
general
The current IS survey includes 300+
data elements to report. The proposed CSBG Annual Report includes
more than 1,000. This increased burden of reporting comes with no
new resources.
The burden on local agencies is impacted
by the implementation of the Online Data Collection system (OLDC).
To comply with the draft CSBG Annual Report local agencies will
need to modify existing systems. An initial investment in new
software for North Dakota was quoted at over $50,000 which only
covers the set-up and one year of training. This does not include
any of the costs associated with adding necessary modules such
as
Weatherization or Head Start.
Many agencies
lack management information systems with the capacity to provide
the required data. Data fields required for ROMA Next Generating
reporting may not be mirrored in those used by agencies. In
addition, many agencies use multiple (and sometimes incompatible)
reporting systems which adds to the time necessary to provide the
data.
In addition, it is not clear how the burden for
local agencies is decreased by the online data collection system
described in the by OCS. OCS' Dear Colleague Letter of June 17,
2016, states, "Aware of the reporting burden the proposed
Annual Report will entail, OCS will decrease that burden by
establishing an on-line automated system for use by the states and
— at the discretion of the states — the local
agencies, for reporting that allows in many instances
auto-populating data from one year to the next and a variety of
data entry processes (e.g. manual entry and automatic upload)."
It is important to note, this online automated system may benefit
State Lead Agencies (grantees), but will most likely not benefit
local Community Action Agencies (sub-grantees). Sub-grantees will
still need systems that track customers, services, outcomes, and
given the requirement of many funding sources (including other
federal Department of Health and Human Services funding) requires
additional and sometime separate data collection systems. State
grantees do not have the funding or capacity for the most part, to
create systems or support local agencies to develop systems that
will streamline or decrease the burden outlined here.
3.The
quality, utility, and clarity of the information to be
collected
As noted below in the detailed comments by
module, several elements of data to be collected under this
proposal have no meaning at the state and federal levels, and only
have value, meaning, and usefulness at the local level.
•
As noted above, OCS' Dear Colleague Letter of June 17, 2016,
states, "Aware of the reporting burden the proposed Annual
Report will entail, OCS will decrease that burden by establishing
an on-line automated system for use by the states and — at
the discretion of the states — the local agencies, for
reporting that allows in many instances auto-populating data from
one year to the next and a variety of data entry processes (e.g.
manual entry and automatic upload)." It is important to note,
this online automated system may benefit State Lead Agencies
(grantees), but will most likely not benefit local Community
Action Agencies (sub-grantees). Sub-grantees (local agencies) will
still need systems that track customers, services, outcomes, and
given the requirement of many funding sources (including other
federal Department of Health and Human Services funding) requires
additional and sometime separate data collection systems. State
grantees do not have the funding or capacity for the most part, to
create systems that will streamline or decrease the burden
outlined here. Furthermore, new state-level systems will do little
to address the needs of local agencies for improved management
information systems with the capacity to collect and report the
data required by ROMA Next Generation.
OCS and the
federal government provide funding to local sub-grantees that
require separate systems and prohibit the aggregation of data. To
minimize the burden on respondents, DHHS should examine its own
practices to ensure that all DHHS-funded programs in the field can
use data collection systems that can be easily integrated and
"talk" to one another to produce reports that meet the
needs of local agencies, states, and OCS.
|
|
Comment 52 (Dickinson Iron Community
Service Agency ; Eligible Entity/Local Agency in MI)
|
|
We at DICSA, a community
action agency serving the rural counties of Dickinson and Iron in
Michigan’s Upper Peninsula, are concerned that the ROMA NG
package is overly burdensome and not attainable within the scope
of our Community Services Block Grant. For instance, the goals of
this report don’t seem to consider smaller agencies with no
urban center and the minimum level of funding. Our staff is
stretched to its capacity and cannot possibly take on full scale
case management or rework our community partnerships to fit into
the specific strategies of Collective Impact. Instead of helping
our agency performance, the report serves instead to draw
essential staff away from impacting low income individuals’
self-sufficiency in order to adapt what we do to fit the rubrics
and metrics of another set of reporting guidelines.
The
estimated burden of the proposed collection of information is
incredibly challenging to quantify. How much will this report
change what our intake staff collect and input into our data
systems? Will these systems have the capacity to gather the
information and to effectively sort and report the data? How much
time will be spent on learning the new reporting data elements,
connecting them with the current elements, and putting them into
quantifiable and meaningful terms? Will any of this help us do a
better job serving our local residents? The estimated time burden
of 242 hours per Community Action Agency is likely under
estimated. Those 242 hours are probably what will be needed for
one person to fully understand the new system, train others on
what is needed to be gathered, and eventually be able to collect,
calculate, decipher, formulate and report the data. The cost of
this time burden to our agency for that one person would be
$8,983. We fully expect that numerous additional hours will be
expended by intake staff and program managers, adding to the
burden. This increased burden of reporting comes with no new
resources. Smaller agencies like ours simply do not have the cash
flow or general fund to support non-budgeted expenses.
|
|
Comment 54 (East Missouri Action
Agency, Inc.; Eligible Entity/Local Agency in MO)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility.
I agree that the
collection of data is necessary to substantiate the work that we
do, and prove that our work is valid. I also feel that the
community action network in Missouri is ready to begin the process
of ROMA Next Gen, but I am worried that many states are not up to
the task yet, which can potentially be very harmful to community
action, overall. If the other states do not have the resources
needed to collect the required information, how will that affect
the entire network when legislators and other officials review the
work we have done?
2. The accuracy of the agency’s
estimate of the burden of the proposed collection of
information.
Considering that the current IS report
includes 300+ data elements to report, and the proposed CSBG
Annual Report includes more than 1,000 data elements, I believe
that OCS’s estimated time burden of 242 hours per
sub-grantee (local Community Action Agency) is somewhat under
estimated. Based on our calculations, the burden to our agency
would be 396 hours. The cost of this time is estimated to be
$8,740. Please note, this estimation does not include the T/TA
that will be needed for our employees to understand how to
complete the new report.
3. The quality, utility, and
clarity of the information to be collected.
If the
proposed number of general indicators were decreased, for example,
2-4 general indicators for each of the domains, then the aggregate
data collected from the entire network would be more focused and
refined. Which would then be much more meaningful data to present
to legislators and funders? To show more localized impact,
agencies should still be allowed to track other outcomes using
local indicators more specific to their unique programs and
communities.
For more clarity in the information
collected, I feel that the data would have more impact if the
indicators were worded similarly to other programs, such as Head
Start and HUD. If all programs reported using similar indicators,
then, community action would have more impact on a national
level.
4. Ways to minimize the burden of the collection
of information on respondents, including through the use of
automated collection techniques or other forms of information
technology.
In Missouri, we have been using a
state-wide data system for many years. Several other states do not
have this resource, which is absolutely necessary for collecting
accurate data. The proposed on-line automated system is a nice
thought, but much of our agency’s reporting comes from many
different data tracking systems. Our other programs, Head Start,
Women’s Wellness, Housing/HUD, and Weatherization all have
their own systems. Unless OCS can develop a system that can be
used in conjunction with all of the other systems, the proposed
on-line automated system will not work for local agency.
Therefore, the burden of the collection of information at the
local level will not be decreased in any way. It is still going to
be a large and expensive burden on our agency.
|
|
Comment 55 (Florida Department Of
Economic Opportunity ; Other/Unknown in FL)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility
2. The accuracy of the
agency’s estimate of the burden of the proposed collection
of information
3. The quality, utility, and clarity of
the information to be collected; and
4. Ways to
minimize the burden of the collection of information on
respondents, including through the use of automated collection
techniques or other forms of information technology.
*
It is this agency’s opinion that overall, the proposed ROMA
Next Generation changes are positive and a move in the right
direction. However, it is also my belief that some of the
additional indicators will be difficult to track as it can be
almost impossible to measure some of the conditions so closely due
to external situations that are beyond our agency’s control.
* Implementation Timelines – being that we are a
public agency, our turnaround time on such a project will very
likely be slower than private agencies. All aspects of our
department are required to be reviewed and scrutinized by other
areas such as budget & management, procurement, etc. We may
not always be able to meet our deadlines due to situations that
are beyond our control.
* Staff Training Needs –
it goes without saying that these changes are detailed and labor
intensive program. Additional training will be required.
|
|
Comment 57 (Fresno Economic
Opportunities Commission ; Other/Unknown in CA)
|
|
We are writing to respectfully
request that the proposed ROMA Next Generation (ROMA NG) be
deferred for introduction to the new administration, its review
and comment, with a concurrent, much more thorough network
assessment, comment, and road test. We have attended all available
webinars hosted by OCS and NASCSP and, to my knowledge, are not
aware – until this time – that there has been a
formal, public demonstration of the software ROMA NG intends for
us to use for the collection of data. That alone causes
considerable concern; we are not prepared to provide complete and
informed comments without having experienced the data collection
systems ROMA NG proposes.
While we offer a succinct
summary of our assessment and deep concerns over the proposed
changes to ROMA below, it seems clear that the underlying agenda
behind the introduction of ROMA NG is contrary to the authorizing
language of the Economic Opportunity Act of 1964 and successive
re-authorizations of that Act.
The Act emphasizes the
local focus of Community Action, something virtually foreign to
existing government trends that centralize and control more and
more of the activities of our communities. The Community Action
Network is successful precisely because each jurisdiction is given
the mandate and freedom to assess local needs, and direct its
resources to needs prioritized by the input of local residents,
leaders appointed by elected officials representing the
jurisdiction, and local constituents elected by local residents.
As government adds to the layers of administration and bureaucracy
entailed in community action and the expenditure of CSBG funds to
carry out the work of Community Action, a perilous, slippery slope
towards stripping agencies of the local control that is at the
core of our success will occur. Each community, each region in our
nation is unique. It is the hallmark of Community Action to
empower local leaders and consult local residents to design and
develop customized programs and initiatives to fight poverty.
Furthermore, the timing of the introduction of ROMA NG
and the short timeframes to comment on an initiative that adds
significant components and layers to the existing system make it
appear that the administration is eager to install the new system
before it leaves office. Our concerns over the extraordinarily
burdensome and unnecessary new requirements of the system require
that the intent to usher in ROMA NG be deferred until the new
administration, working hand-in-hand with the network, can
reassess the need for a revised ROMA system. The system in its
current configuration satisfies requests for information made by
our elected leaders, funders, and the residents of our
communities.
While we understand and agree entirely
with the need for accountability, updated by the administration in
2011 in the form of the Government Performance and Results Act
(GPRA), the proposed ROMA NG requirements are simply overbearing,
unnecessary, and introduce inordinate time and expense commitments
to the detriment of the energy and focus that our mandate to
create opportunity and reduce poverty requires.
The
data to be collected by ROMA NG is significantly more than the
existing ROMA requirements which, for an agency as complex as
Fresno EOC, would be extraordinarily burdensome, expensive, and
time consuming. The same concerns would apply for smaller agencies
that would not have the capacity, staffing, infrastructure,
funding, or time to complete the required reports. A sampling of
added elements in ROMA NG gives rise to our concern.
d.
After extensive research and retaining qualified consultants to
advise us in the use of electronic systems to collect, collate,
and report unduplicated data across our 14 programs, 49 funding
sources, 101 contracts, and 42 funding periods, it became apparent
the cost to purchase and maintain such a system would be
prohibitive. Estimates were in excess of $270,000 per year, all of
which would have to be absorbed by CSBG and would reduce program
activity to report activity only CSBG requires. Our board elected
not to move forward with the plan. Any reliable collection
methodology to meet the increased burden of ROMA NG reporting,
whether manual or electronic, will add significant cost, funds we
would much rather deploy for direct services and/or leverage for
greater impact.
2. Those who are familiar with the
kinds of information, and volume of information Congress seeks
from the network report that much of the data ROMA NG would
collect is superfluous. Congressional requests tend to be more
general, such as, “How many families did you move out of
poverty?”
3. Will implementing ROMA NG help
agencies do their job better? The work of community action is
fluid and extremely challenging. It requires that we be
resourceful, nimble, and innovative. Is it data that generally
informs new directions or new responses to urgent needs? Not
always. Perhaps a better barometer of need is information drawn
directly from residents living in our disadvantaged communities.
The involvement of our residents in the work of community action
is a core value of community action; in our experience, low-income
residents provide us with the best guidance on need, priorities,
and resources they require.
Recommendations
We
would suggest a program and process-based approach to document
activities and outcomes, a much simpler format aligned with the
way in which data is more commonly collected by agencies. With
this approach agencies would provide data and anecdotal highlights
for major programs they operate. Data would be drawn from reports
the agency already submits to funders, stakeholders, and
constituents.
Reports would be unique to each
community action agency which, after all, is the hallmark of
community action. We are among the few remaining federal programs
that enjoy bipartisan support precisely because, at our core,
Community Action is about local control, it requires and honors
the involvement and wishes of local leaders – including
representatives of low-income communities – rather than
foisting a predetermined agenda and menu of services to report on.
To merge the thinking and actions of agencies toward a raft of
established activities and outcomes runs the risk of derailing
efforts to respond to local need in innovative ways.
The
system of measurement and accountability should drive agencies
with the urgency to think broadly, ever seeking innovative
solutions that meet local needs. Responding to a prescribed menu
of activities and expected outcomes limits innovation, it
threatens to suffocate innovation and the creative advances that
demonstrate new hope and opportunity for those living in poverty.
In other words, the more defined, the more extensive and
prescriptive the measuring tool, the less likely innovation will
occur.
Now is the time to innovate, now is the time to
open the vast reservoirs of thought and ideas to demonstrate the
value and worth of community action. Now is the time to herald our
core values – the uniqueness of each community and the
leveraging of CSBG dollars to innovate new
|
|
Comment 58 (Greater Bergen Community
Action; Eligible Entity/Local Agency in NJ)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility
On an individual program
level, most of the information is already collected and analyzed
to ensure that the program is necessary, efficient, and changing
the lives of the individuals enrolled. The many funders of CAA
programs already require outcome reporting, but, of course, in
their own format. Compiling all the information for this annual
report is a goal to reach for, but currently not practical. Most
state and federal funders of programs run by CAAs do not share
client data files with anyone – gathering information from
each program is time consuming when possible. The cost-effective
technology needed is simply not available to individual
agencies.
Module 3 I think falls short of the goal of
being rolled-up to better present a nation-wide picture of
community action. If we each define our community, the rates and
percentages have no meaning when combined on a national level –
how can a 10% increase on a block level be aggregated on a state
level? The tools indicated are for project management, and should
be a tool for agency use, not a required report.
2.
The accuracy of the agency’s estimate of the burden of the
proposed collection of information
Without a definition
of what specifically is included in the estimate of the burden,
and the assumption that it should include the preparation required
throughout the fiscal year to allow for one employee to collate
the information required for the report, the estimate falls very
short of the time needed. Our current costs to complete the IS
Survey, gather all the information throughout the year and
maintain the technology required is around $6800. That covers 222
employee hours. Our estimate for the burden for the proposed
report is $24,200, including 658 hours. The hours will almost
triple due to the increase in the amount of information required
to be tracked for community projects, collected and analyzed for
all client outcomes, training on new technology needed for
compliance, time needed to complete the more complex reporting
forms, etc. The database technology this makes necessary increases
that cost substantially.
3. The quality, utility, and
clarity of the information to be collected; and
The
Individual and Family level NPIs have been well-though out and
improved. The community level reporting has issues with
definitions, rates and percentages that need to be more fully
explained. The report on services delivered is a list of things we
do, not outcomes. Although useful for an agency to know, it
doesn’t advance our national story, and should not be in the
report. The same thing applies to the list of community
strategies.
Ways to minimize the burden of the
collection of information on respondents, including through the
use of automated collection techniques or other forms of
information technology.
The interactive spread sheets
will minimize the burden after the first year. Getting all the
federal and state databases to allow agencies to download and/or
upload information into another database would reduce the data
collection burden tremendously.
|
|
Comment 59 (Greater Bergen Community
Action ; Eligible Entity/Local Agency in NJ)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility
On an individual program
level, most of the information is already collected and analyzed
to ensure that the program is necessary, efficient, and changing
the lives of the individuals enrolled. The many funders of CAA
programs already require outcome reporting, but, of course, in
their own format. Compiling all the information for this annual
report is a goal to reach for, but currently not practical. Most
state and federal funders of programs run by CAAs do not share
client data files with anyone – gathering information from
each program is time consuming when possible. The cost-effective
technology needed is simply not available to individual
agencies.
Module 3 I think falls short of the goal of
being rolled-up to better present a nation-wide picture of
community action. If we each define our community, the rates and
percentages have no meaning when combined on a national level –
how can a 10% increase on a block level be aggregated on a state
level? The tools indicated are for project management, and should
be a tool for agency use, not a required report.
2.
The accuracy of the agency’s estimate of the burden of the
proposed collection of information
Without a definition
of what specifically is included in the estimate of the burden,
and the assumption that it should include the preparation required
throughout the fiscal year to allow for one employee to collate
the information required for the report, the estimate falls very
short of the time needed. Our current costs to complete the IS
Survey, gather all the information throughout the year and
maintain the technology required is around $6800. That covers 222
employee hours. Our estimate for the burden for the proposed
report is $24,200, including 658 hours. The hours will almost
triple due to the increase in the amount of information required
to be tracked for community projects, collected and analyzed for
all client outcomes, training on new technology needed for
compliance, time needed to complete the more complex reporting
forms, etc. The database technology this makes necessary increases
that cost substantially.
3. The quality, utility, and
clarity of the information to be collected; and
The
Individual and Family level NPIs have been well-though out and
improved. The community level reporting has issues with
definitions, rates and percentages that need to be more fully
explained. The report on services delivered is a list of things we
do, not outcomes. Although useful for an agency to know, it
doesn’t advance our national story, and should not be in the
report. The same thing applies to the list of community
strategies.
Ways to minimize the burden of the
collection of information on respondents, including through the
use of automated collection techniques or other forms of
information technology.
The interactive spread sheets
will minimize the burden after the first year. Getting all the
federal and state databases to allow agencies to download and/or
upload information into another database would reduce the data
collection burden tremendously.
|
|
Comment 60 (Hinds County Human
Resources Agency ; Other/Unknown in MS)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility
The information has
practical value for the local agencies but offers no practical
value to combine at the state or federal level. The data collected
now has not been used for any other purpose than reporting. The
increased information assumes the purpose of Community Action
Agencies to be uniform or follow the vision of one individual.
This counters the original purpose of Community Action to be
locally driven to fits the needs of each community. While there
are elements that may be helpful, there are a number of areas
detailed below where the collection of information does not meet
the threshold of necessity or utility. The package as proposed in
its entirety is overly burdensome, contains reporting tools that
will provide useless information, and is outside the reach of the
Community Services Block Grant (CSBG). However, with
modifications, a system for collection of information can be
developed that improves the performance management of the agency
and has practical utility. Detail is provided here outlining where
changes need to be made to reach a level of workability and
utility.
Furthermore, the introduction of “Collective
Impact” has been annoying a lot of marginalized communities
because it is yet another example of the mainstream community
“discovering” something that has been around for a
long time, slapping some academic labels and concepts on it,
positioning it as new, and then getting all the attention and
resources. Long before CI came out as a formal concept, community
action in diverse communities were already working together,
sharing information, creating coordinating mechanisms. We have
been doing Collective Impact way before it was called that, and we
continue to.
This new community format also
perpetuates Trickle-Down Community Engagement: this is where “we
bypass the people who are most affected by issues, engage and fund
larger organizations to tackle these issues, and hope that
miraculously the people most affected will help out in the effort,
usually for free.” Without meaning to, CI backbone
organizations have become some of the biggest perpetrators of this
terrible and destructive practice. Some backbone organizations’
gravity is so strong that it begins gatekeeping, spewing out tiny
amounts back to the organizations and communities most affected by
the challenges the CI efforts are trying to address. Grassroots
organizations that used to be able to solicit significant
participation directly from people of low income are now told from
whatever juggernaut CI effort is ruling the landscape.
Due
to all the gatekeeping, inequitable funding allocation process,
TDCE, and other factors, communities of color are oftentimes left
behind by CI efforts. We are not funded on the same level, and yet
are asked to provide input, do outreach, mobilize our communities,
etc., and because we believe in the goals of the CI efforts, and
sometimes because we are backed into corners, we’ll be
involved. This leads to the CI effort being seen as inclusive,
when in reality, it is tokenizing. This creates a shell of our
purpose, our need and our ability to give voice to those we serve.
2. The accuracy of
the agency’s estimate of the burden of the proposed
collection of information
The agency’s’
estimated time burden of 242 hours per sub-grantee (local
Community Action Agency) is significantly under estimated. Based
on our calculations, the burden to our local agency would be 520
hours. The cost of this time burden is estimated to be $24,000.
The current IS survey includes 300+ data elements to report. The
proposed CSBG Annual Report includes more than 1,000. This
increased burden of reporting comes with no new resources. The
burden on local agencies is not impacted by the implementation of
the Online Data Collection system (OLDC). To comply with the draft
CSBG Annual Report local agencies will need to modify existing
data collection systems.
This proposed data
information especially in the community level section uses an
incorrect perception of the Community Action Network purpose and
therefor drives the agencies burden in a completely different
direction then developed over the lifetime of the agency.
3.
The quality, utility, and clarity of the information to be
collected; and
As noted below in the detailed comments
by module, several elements of data to be collected under this
proposal have no meaning at the state and federal levels, and only
have value, meaning, and usefulness at the local level. The
continue vision of Community Action Agencies as some sort of
research institution does not allow for the rigorous amount of
stabilization and self-sufficiency case management or community
development work that is needed for outcomes for everyday low
income individuals. The quality of data listed in this new system
creates a fantasy albeit also dangerous collection process.
Experimenting on impoverished people was never the intent of CSBG.
It is develop real solutions with local induvial
development.
Several of the elements to be collected as
described are not available in smaller geographic areas usually
targeted by local agencies in their community work and if created
by local agencies, lose validity. The social or population level
indicators being requested are not meant for single agency
reporting and are best served for community-wide scorecards or
utilized as part of a community needs assessment. For example,
comparing overall percentages of community indicators (e.g. the
unemployment rate, violent crime rate) to individual local
agencies outcomes is misleading and extremely dangerous data
collection.
This will hold agencies accountable for
such factors in which they have little control over.
Take
a state such as Mississippi that continues to reject multiple
systems and programs that would greatly impact those in which we
serve. This state rejected Medicaid expansion while being the
unhealthiest state. This state reduced education funding while
being rank 50th in multiple education factors. This state
continues to limit all forms of birth control and birth control
education while leading the leading persons in poverty includes
young single mothers. Therefor the work of our local agencies
become that of much survival and small but mighty impact on the
individual level while also allowing many of communities to not
just fully collapse under the pressure of rising cost with low
wages. Crime rates rise as a means of poverty survival and even
serving as the biggest CSBG recipient in the state, we have to
focus on the individuals we can change and not always research of
collective impacts.
· Module 4, Section A:
Characteristics for NEW Individuals and Households - Data Entry
Form is an unnecessary document that will not produce
quality data; it will have little utility at a local level; and it
lends itself to meaningless data at a state and national level. It
should be removed. Because demographic data alone does not include
information about the needs of new customers, this information
fails to help agencies determine if they are addressing issues
identified in their community assessments. Furthermore, the large
number of “one time” customers in programs such as
LIHEAP will skew any meaningful interpretation of the data. Aside
from the challenge of defining who constitutes a new customer,
agencies that use multiple reporting systems will also have the
added burden of tracking new customers across all programs and
services.
· Module 3, Section A: Many of the
indicators used here are inappropriate for reporting by local
sub-grantees to States and by States to OCS and will produce data
of limited utility. Several challenges underscore this point.
First, community-level data on indicators is not always available
at the level (e.g. regional or zip code/census tract) or
population (e.g. low-income children aged 0-5) that matches the
area and population targeted by the agency. Second,
community-level indicators are influenced by a wide number of
sources that would most likely overwhelm the impact of even
successful community initiatives, a fact that significantly limits
their utility for providing meaningful information about outcomes.
Third, any meaningful data about the outcomes of community-level
initiatives would require a rigorous program evaluation and could
not be determined on the data collected through the Annual Report
alone.
4. Ways to minimize the burden of the
collection of information on respondents, including through the
use of automated collection techniques or other forms of
information technology.
As noted above, OCS’
Dear Colleague Letter of June 17, 2016, states, “Aware of
the reporting burden the proposed Annual Report will entail, OCS
will decrease that burden by establishing an on-line automated
system for use by the states and – at the discretion of the
states – the local agencies, for reporting that allows in
many instances auto-populating data from one year to the next and
a variety of data entry processes (e.g. manual entry and automatic
upload).” It is important to note, this online automated
system may benefit State Lead Agencies (grantees), but will most
likely not benefit local Community Action Agencies (sub-grantees).
Sub-grantees (local agencies) will still need systems that track
customers, services, outcomes, and given the requirement of many
funding sources (including other federal Department of Health and
Human Services funding) requires additional and sometime separate
data collection systems. State grantees do not have the funding or
capacity for the most part, to create systems that will streamline
or decrease the burden outlined here. Furthermore, new state-level
systems will do little to address the needs of local agencies for
improved management information systems with the capacity to
collect and report the data required by ROMA Next Generation.
OCS and the federal government provide funding to
local sub-grantees that require separate systems and prohibit the
aggregation of data. To minimize the burden on respondents, DHHS
should examine its own practices to ensure that all DHHS-funded
programs in the field can use data collection systems that can be
easily integrated and “talk” to one another to produce
reports that meet the needs of local agencies, states, and OCS.
|
|
Comment 61 (Illinois Association
Community Action Agency ; Other/Unknown in IL)
|
|
General Comments
This
letter provides detailed comments and recommendations based on the
format requested in the Federal Register. Overarching concerns
with the currently proposed CSBG Annual Report include:
Reporting requirements that exceed agency capacity for
data collection, integration, and analysis.
Many
agencies struggle under the burden of collecting and reporting
data for multiple programs that use and require different —
and often incompatible — software and systems. The CSBG
Annual Report as currently proposed is unduly burdensome and
underestimates the time required at the local level to collect,
integrate, and report data at the agency level.
Reporting
requirements that produce data of limited utility for CAAs and
State CSBG Lead Agencies. IACAA strongly believes that all data
collected for the CSBG Annual Report should have a clear purpose
and use for local agencies and State CSBG Lead Agencies. Caution
must be taken if data reporting is included that is not intended
to be aggregated and used at the state and federal
level.
Responses to the Questions
1. Whether
the proposed collection of information is necessary for the proper
performance of the functions of the agency, including whether the
information shall have practical utility
The Illinois
Association of Community Action Agencies (IACAA) agrees with the
Community Action Partnership that while there are elements that
may be helpful, there are a number of areas detailed below where
the collection of information does not meet the threshold of
necessity or utility. The package as proposed in its entirety is
overly burdensome, contains reporting tools that will provide
unnecessary information, and is outside the reach of the Community
Services Block Grant (CSBG). However, with modifications, a system
for collection of information can be developed that improves the
performance management of the agency and has practical
utility.
Detail is provided here outlining where
changes need to be made to reach a level of workability and
utility.
I. Module I: Modify the options CSBG State
Lead Agencies/grantees have when reporting to OCS on CSBG
Organizational Standards to allow for a broader range of
responses. As noted below, there are significant challenges with
the way this information is being reported and creates a misuse of
the Organizational Standards.
Expecting constant
perfection is unrealistic and does not have practical utility for
local agencies or the State CSBG Lead Agency. As proposed, the
single metric the State Agencies will report on will be the number
of eligible entities that meet 100% of the CSBG Organizational
Standards. To increase utility of the information, State Agencies
should have the option to report the number of agencies at a
variety of thresholds that still delineate
Illinois
Association of Community Action Agencies Feedback — CSBG
Annual Report OMB PRA Clearance FRN #1 strong performance or allow
for a more accurate reflection of the training and technical
assistance needs. For instance State Agencies should be able to
report the number of local agencies that have met 100%, 9() %,
80%, 70%, 60%, 50%, less than 50% of the standards. This will
allow for the collection of more accurate information; better
tracking of results over time; and better, more efficient
investment of training and technical assistance.
Module
4: Remove the Characteristics for NEW Individuals and Households
Report. This report has no practical utility at the local level
given the time and expense to create it at each local agency. This
is not necessary for agency performance, and the intentions
outlined by OCS for its use are not reflective of how local
agencies and states look at their communities to assess needs,
develop programs, and report progress. It also lacks utility at
the state level given that data such as this rolled up to the
state level will be skewed by large population centers and
decrease the practical utility of the data even further.
Furthermore, many agencies may either lack management information
systems with the capacity to disaggregate this type of data from
its overall customer database, or struggle to integrate the
required data given their use of multiple (and sometimes
incompatible) reporting systems, In addition, as many local
agencies serve families over time, with no definition of "new"
being provided for review, the CSBG Network will be unable to
implement this report consistently. Defining an individual or
family as new becomes challenging when they may receive different
services over many years (e.g. Early Head
Start, child
care, after school programs, youth programs, job training, WIC,
food boxes, LIHEAP, etc.).
Module 3: Remove the
Community Outcome Indicators that include percent or rates. Their
inclusion is inappropriate and will provide meaningless data when
aggregated beyond individual initiatives. Such data has validity
for geographic areas much larger than those addressed or targeted
by local agency projects. This validity is lost when the scale is
smaller and accepted data comparison points become unavailable.
Having individual agencies report on rate change at the community
level on social indicators does not provide information that is
usable. It is more appropriate to give agencies the option of
reporting outcome data and the flexibility to define which
indicators they use.
Module 3: Remove Collective Impact
from all reports. Collective Impact is simply one set of
strategies for doing community-level work and while CSBG eligible
entities are engaged in such efforts, privileging one approach is
inappropriate and unnecessary given the broader range of options
for doing community-level work. Collective Impact should be
removed and included in training and technical assistance
endeavors as a potential local option. Besides CAAs would need
significant resources and support to engage in Collective Impact
which is presently not the case.
Module 4: Remove the
Stability Indicators. The indicators selected and included in the
clearance package are not helpful or practical when reporting
customers stabilized as a result of the eligible entity's work.
One option duplicates data already collected and the second uses a
tool that is not specifically designed for low-income individuals
and families and has an overly broad survey to define Community
Action customer stability.
Address the unique issues of
Public entities that receive CSBG. Public agencies that
sub-contract most or all of their CSBG funding to other service
providers may face added 3 Illinois Association of Community
Action Agencies Feedback — CSBG Annual Report OMB PRA
Clearance FRN #1 challenges of requiring their grantees to collect
data in accordance with ROMA Next Generation requirements since it
may limit agencies to contracting with providers that have the
requisite data collection capacity or require additional technical
assistance to meet the reporting requirements. This challenge is
not addressed in the proposed reporting package. Include a 4th
ROMA Goal: Agencies Increase their capacity to achieve
results.
Reinstate this national goal that addressed
the critical role the network of local agencies serving 99% of US
counties plays in addressing poverty. Reasons for exclusion have
been inadequate to date. Because agency capacity is critical to
the ultimate success of programs and services, it is important to
clearly state this objective in the overall National Goals. Adding
back an agency goal is a prudent approach. The standards will not
capture or encourage all of the work of CAAs. The best example is
the OCS directive to use bundled services and to examine/address
the impact of toxic stress on young brains. When addressing toxic
stress, CAAs with Head Start programs will be better equipped to
explore and offer some services; the bottom line is that most CAAs
do not currently hold the capacity to address the impact of toxic
stress. Collaborations with early childhood organizations are
essential to move in this direction, as well as partnerships with
universities and other educational institutions. The argument can
be made that the Standards provide some direction, but does not go
far enough to support this work. This will require a goal for
agencies to build capacity and expertise to integrate new
practices that addresses poverty. This requires systemic change
that goes beyond case management. CAAs are exploring bundling
methods that can provide better services, but require unduplicated
counts to multiple funders. This can be challenging and requires
that agencies enhance or change processes and have the ability to
upgrade when technology advances.
This also highlights
the broader need of the Network to commit resources for ongoing
training, technical assistance, and general capacity-building
supports consistent with private sector approaches such as
Continuous Quality Improvement, Lean and Six Sigma.
2.
The accuracy of the agency's estimate of the burden of the
proposed collection of information
While this is an
exciting area for Community Action, there needs to be an
identification of common practices across the entire network.
Collection of data will be a huge undertaking, but at this point,
much of the data analysis will be beholden to the expertise,
either at the agency or association level, which could be varied
and not necessarily rigorous. Standardized processes, such as the
Needs Assessment and a data collection tool, beyond the NPIs,
would go a long way to allow this type of collection and analysis
to occur. Additionally, CAAs may not have adequate resources or
capacity to support the collection, analysis and use of data. At
best, it is not clear how the burden for local agencies is
decreased by the online data collection system described in the by
OCS. Consequently, if the time burden analysis is to be conducted
with the scenario of successful implementation, then it seems
prudent to also perform a scenario analysis of time burden with an
assumption of no new system. In the event of failed
implementation, substantial additional state and federal level
resources would be diverted away from capacity building to support
high-quality programs toward data collection.
Illinois
Association of Community Action Agencies Feedback — CSBG
Annual Report OMB PRA Clearance FRN #1
Concerns include:
The current IS survey includes 300+ data elements to
report. The proposed CSBG Annual Report includes more than 1,000.
This increased burden of reporting comes with no new
resources.
The burden on local agencies is not
alleviated with the implementation of the Online Data Collection
system (OLDC). To comply with the draft CSBG Annual Report local
agencies will need to modify existing systems which will come at a
premium cost with no additional resources.
Many
agencies lack management information systems with the capacity to
provide the required data. Data fields required for ROMA Next
Generating reporting may not be mirrored in those used by
agencies. In addition, many agencies use multiple (and sometimes
incompatible) reporting systems which adds to the time necessary
to provide the data.
3. The quality, utility, and
clarity of the information to be collected; and
As
noted below in the detailed comments by module, several elements
of data to be collected under this proposal have no meaning at the
state and federal levels, and only have value, meaning, and
usefulness at the local level.
Several of the elements
to be collected as described are not available in smaller
geographic areas usually targeted by local agencies in their
community work and if created by local agencies, lose validity.
The social or population level indicators being requested are not
meant for single agency reporting and are best served for
community-wide scorecards or utilized as part of a community needs
assessment. For example, comparing overall percentages of
community indicators (e.g. the unemployment rate, violent crime
rate) to individual local agencies outcomes is misleading
Module
4, Section A: Characteristics for NEW Individuals and Households -
Data Entry Form is an unnecessary document that will not produce
quality data; it will have little utility at a local level; and it
lends itself to meaningless data at a state and national level. It
should be removed. Because demographic data alone does not include
information about the needs of new customers, this information
fails to help agencies determine if they are addressing issues
identified in their community assessments. Furthermore, the large
number of "one time" customers in programs such as
LIHEAP will skew any meaningful interpretation of the data. Aside
from the challenge of defining who constitutes a new customer,
agencies that use multiple reporting systems will also have the
added burden of tracking new customers across all programs and
services.
Module 3, Section A: Many of the indicators
used here are inappropriate for reporting by local sub-grantees to
States and by States to OCS and will produce data of limited
utility. Several challenges underscore this point. First,
community-level data on indicators is not always available at the
level (e.g. regional or zip code/census tract) or population (e.g.
low-income children aged 0-5) that matches the area and population
targeted by the agency. Second, community-level indicators are
influenced by a wide
Illinois Association of Community
Action Agencies Feedback — CSBG Annual Report OMB PRA
Clearance FRN #1 number of sources that would most likely
overwhelm the impact of even successful community initiatives, a
fact that significantly limits their utility for providing
meaningful information about outcomes. Third, any meaningful data
about the outcomes of community-level initiatives would require a
rigorous program evaluation and could not be determined on the
data collected through the Annual Report alone.
4. Ways
to minimize the burden of the collection of information on
respondents, including through the use of automated collection
techniques or other forms of information technology.
As
noted above, OCS' Dear Colleague Letter of June 17, 2016, states,
"Aware of the reporting burden the proposed Annual Report
will entail, OCS will decrease that burden by establishing an
on-line automated system for use by the states and — at the
discretion of the states — the local agencies, for reporting
that allows in many instances auto-populating data from one year
to the next and a variety of data entry processes (e.g. manual
entry and automatic upload)." It is important to note, this
online automated system may benefit State Lead Agencies
(grantees), but will most likely not benefit local Community
Action Agencies (sub-grantees). Sub-grantees (local agencies) will
still need systems that track customers, services, outcomes, and
given the requirement of many funding sources (including other
federal Department of Health and Human Services funding) requires
additional and sometime separate data collection systems. State
grantees do not have the funding or capacity for the most part, to
create systems that will streamline or decrease the burden
outlined here. Furthermore, new state-level systems will do little
to address the needs of local agencies for improved management
information systems with the capacity to collect and report the
data required by ROMA Next Generation.
OCS and the
federal government provide funding to local sub-grantees that
require separate systems and prohibit the aggregation of data. To
minimize the burden on respondents, DHHS should examine its own
practices to ensure that all DHHS-funded programs in the field can
use data collection systems that can be easily integrated and
"talk" to one another to produce reports that meet the
needs of local agencies, states, and OCS.
|
|
Comment 62 (Inca Community Service;
Eligible Entity/Local Agency in OK)
|
|
Question 1: Whether the
proposed collection of information is necessary for the proper
performance of the functions of the agency, including whether the
information shall have practical utility
Response:
The
proposed collection of system in extremely burdensome and
expensive to the point we would be spending more time and money
rendering reports than assisting those in need become
self-sufficient. In reviewing the module we found some useful
tools and reports but there were many which seem redundant and
provided useless information for us to use in improving the agency
and the quality of services we provide.
·
Module 1: Change the reporting on Organizational
Standards
Expecting constant perfection is impossible
and unrealistic. The Organizational Standards are more of a living
document which is always changing, with different standards
becoming due at different times. It is unrealistic to believe that
an agency could be constantly at 100%.
A more
realistic, accurate and useful tool would be for the State
Agencies to report the number of local agencies that have met
100%, 90%, 80%,70%, 60% 50% and less. This would give a clearer
picture of where and how much technical assistance is needed.
·
Module 3: Community Outcome Indicators
Again we looked
at how this information who assist us and found the information
distorted and ineffective. Our service area contains cities and
towns under the population of 5,000 and 100’s of pocket
communities of 100 or less. This data was meaningless and
incomplete since are agencies target area was smaller populations
and spread over many miles. Because of the size of our target
population the data on rate change at the community level on
social indicators could not accurately be displayed or useful.
Some flexibility in choosing the indicators used, using
geographical makeup and description of target service area and
goals would give a more accurate picture.
·
Module 4: Remove the NEW Individual and Household Staff
The
other issue to the time and expense of collecting, sorting, and
ensuring this data is correct. Currently we do not have an
effective system or manpower to handle this type of input or
ensure the data is correct. This is a lot of expense for data
which can easily be distorted or of very minimal use.
Module
4: Remove Stability Indicators
What is stable in a
rural community in Southeast Oklahoma is completely different from
what is stable in New York City. There is also too much difference
in the interpretation of what financial well-being means to put a
one size fits all scale to.
Question 2: The accuracy
of the agency’s estimate of the burden of the proposed
collection of information.
As an agency we are already
feeling the burden and stress of reporting on the current system
(374). It will be overwhelming to nearly triple the now-required
information to 1131 items. This would be an overwhelming addition
to administrative cost, data entry costs and IT costs which would
greatly effective the funds needed to provide the services to
those in need. In a time where we are trying to more efficient and
invest our funds into the projects and families who need them, it
does not make sense to spend so much time and effort in generic
reporting which will not show each Community Action Agency
accurately.
Below is the estimated time of 242 hours
for local Community Action Agencies is significantly
underestimated. Our agency is currently stressed fulfilling the
current ROMA without the extra burden. Based on our calculations
the burden to our local agency would be 1,836 hours before adding
the estimated hours for follow up which would be an additional
1,875 hours = 3,711 hours. The cost of this burden is estimated to
be on an average of $25.00 an hour (wage + benefits) = $45,900.00
with additional follow-up requirement $92,775.00.
Our
Methodology Used to figure cost above what is currently being
conducted:
· The amount of time of intake and
other workers will spend obtaining information that is not
required by other programs. 5 hours a week x 52 = 260
· Amount of
time entering data or transferring it from another program’s
software to the CSBG information system.
Entering Data
- 15 minutes per entry (service) X 4,000 services = 600
hours.
Transferring Data – RSVP (1 hour a month)
x 12 = 12 hours; Wk, RxOklahoma (1hour a month) = 12 hours; RAVE –
(10 hours a month) x 12 = 120 hours; JAMM transit – (30
minutes a day) = 480 per year. Total of all Transferring time =
624 hours
· Typical Training Time spent for each
worker to me trained and refreshed on the skills of data entry, or
data acquisition, including new and continuing staff. Based on
current training time and turn-over in positions. 24 hours
·
Time spent team will spend on data quality assurance, as well as
quality assurance on the final report. 4 hours for Quality Control
personnel, 2 hours for report generation, 2 hours for corrections.
Monthly team meeting review and discussing date: (1 hour X 12 = 12
hours). Final Review and preparation for Board of Director. ( 1
hour per month) = 21 X 12 = 252 hours.
· Time
spent tracking individuals and families, tracking progress of
community projects and recording the progress in the format
required in Modules 3 and 4. Average Estimate time per client
follow=up is 1.5 hours x 1,250 unduplicated service recipients
(estimated # which currently do not have a follow-up process) =
1,875 hours per year. (Used historic data on a current case
management program to estimate amount of time on follow-up on to
come up with the average time). We do not currently do follow-up
on a large portion of our service recipients due to the expense.
· Time spent gathering external demographic
information required in module 4. Estimated 8 hours. (Based on
time spent on gathering data for CAN)
· Time
spent tracking volunteers, board members and their commitments to
organization and project that are not run by your Community Action
Agency. 1 hour per month for Board = 12 hours 2 hours per month
for volunteers = 24 hours. A total of 36 hours.
·
Time spent on additional analysis required to report of the 50
organizational standards. 2 hours per month = 24 hours
·
Time spent reporting to state and answering the narratives
required by the state for module 1, as well as the local
information on funds, people and projects. 8 hours
Question
3: The quality, utility, and clarity of the information to be
collected; and
· As mention above our agency’s
service area covers four counties in rural Oklahoma which does not
have any cities or town above 5000 and many small communities with
populations of 200 or less. Some of the information to be
collected is not available for our geographic areas and incomplete
in other. This would result in even additional time being used to
create the data necessary and ensuring it is accurate. This would
especially be true in the social and community indicators needed.
· Our management team discussed in length how we
would use this information as an agency and the only use we could
determine was whether more individuals needed the service this
year than from a previous year. This could be achieved with just a
check box. All the detailed characteristic information seems
impertinent to the needs of our community or the services we give.
Who is new? Every time someone changes partners are they new? Are
they new to the agency or the program they are participating in?
We believe this will cause the final results to be faulty
information.
Question 4: Ways to minimize the burden of
the collection of information on respondents, including through
the use of automated collection techniques or other forms of
information technology.
Our agency currently uses
tracking systems from 6 different funding sources (Head Start,
RSVP, Oklahoma Department of Transportation, Homeless Program, and
Oklahoma Department of Human Services) using their data input
system. This also includes the data tracking system for our CSBG
program called CAPTAIN. None of these systems “talk”
to each other so we spend time doing double entry putting the
information into the CSBG reporting system. Most of these are
federal program, if a system could be put in place so these could
communicate with each other this would greatly reduce the manpower
time it takes to do double entry.
|
|
Comment 65 (Kceoc Community Action
Partnership; Eligible Entity/Local Agency in KY)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility
Collection of information
is always necessary to measure performance and to ensure
accountability. In the case of the changes, the feedback opens up
whether or not the information collected is beneficial and
necessary to really tell our story. The IS report is about
outcomes, not process; therein lies our part of concerns with the
report changes. There are positive changes; however, there are
others that will create undo hardship on agencies already strapped
for funding and maxed on staff capacity. We are already reporting
on the ROMA report, logic model (reported internally, since
required by the state to collect with the annual proposal), and
the IS survey. Agency wide, we are inputting several thousand
families in our agency wide database. Then, we have inputting for
other programs leveraged/supported by CSBG and other funding.
KCEOC has some staff already inputting information for the same
clients into 4 separate databases. It is overwhelming to say the
least.
As a CSBG Director and Director over all of
these other programs, I am then charged with overseeing all this
data, pulling reports, completing reports, ensuring accuracy,
maintaining compliance, checking files to ensure the data from the
files matches the data from databases; and, this is only a minor
part of my day to day duties. I am also concerned with how
accurate the reporting will be since staff will, for example in
reference to Module 2 Section B, have to choose between properly
serving the client and documenting time spent on administrative
type duties such as planning. In my agency, we always try to put
the client’s needs first. When we do this, documenting hours
for other specific duties can’t always be the priority, but
we are currently giving a faith effort in which I am
confident.
Another comment I would like to make is that
the second biggest reason I see staff seeking other employment is
due to their jobs becoming increasingly overwhelming. With many
funders expecting more and more, and the burden falling on already
overworked staff, I fear we will lose well trained and seasoned
staff that love their work, in exchange for employers that are
more realistic about the strain they put on their staff.
2.
The accuracy of the agency’s estimate of the burden of the
proposed collection of information
Realistically, I
could use at least part time help now to pull everything together.
The burden of this would take at least 1 full time person, since I
would also have this person to take over doing the community needs
assessment, assisting with strategic planning, organizational
standard compliance/reporting, logic model completion, ROMA entry
of programs who do not allow this with their funds (currently we
hire someone part time to do this over the course of several
months). The total estimated cost (including fringe/indirect)
would be approximately $60,000 per year, plus additional basic
support costs for staff related expenses, including training
(estimated at $10,000 each year).
There would also be
additional expenses staff (includes salary/fringe/indirect) such
as additional data collection/research time, training, tracking
families, involved in their data collection & input. This is
estimated at $59,000 per year. And/or, hiring more staff to take
over their some of their current duties so they have more time for
reporting/tracking clients/all other duties that will come with
this. This could potentially have an additional cost of $70,000 to
$140,000, depending on the number of staff needed.
Please
note, these are just estimates and the costs could run much higher
in the first and subsequent years. KCEOC uses the state
associations system; therefore, the cost for this is unknown. No
system currently exists in the state that can integrate all the
systems utilized by KCEOC.
3. The quality, utility, and
clarity of the information to be collected; and
Without
help to create a position to be our “Assessment
Coordinator,” all we can do is our best and hope we can meet
these criteria in the reporting. See additional comments
below.
4. Ways to minimize the burden of the collection
of information on respondents, including through the use of
automated collection techniques or other forms of information
technology.
Technology is great, but you still have to
have people to input and collect the data.
|
|
Comment 66 (Lakes And Prairies
Community Action Partnership ; Other/Unknown in MN)
|
|
Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility.
While there are elements
that may be helpful, the package as proposed in its entirety is
overly burdensome, contains reporting tools that will provide
useless information, and is outside the reach of the Community
Services Block Grant (CSBG). However, with modifications, a system
for collection of information can be developed that improves the
performance management of the agency and has practical utility.
Detail is provided below that outlines where changes need to be
made to reach a level of workability and utility.
The
accuracy of the agency's estimate of the burden of the proposed
collection of information
The agency's' estimated time
burden of 242 hours per sub-grantee (local Community Action
Agency) is significantly under estimated. Based on our
calculations, the burden to our local agency would be 2,360 hours.
The cost of this time burden is estimated to be $50,400. Included
in this estimate is staff time for training, data entry, and
report collection for 4,000 clients annually. Although there is a
new online data collection
system proposed, this online
automated system will most likely not benefit us as a local
Community Action Agency. Our agency will in fact need more complex
systems that track customers, services, outcomes, given that the
current IS survey includes 300+ data elements to report, and the
proposed CSBG Annual Report includes more than 1,000, this
increased burden of reporting comes with no new resources.
The
quality, utility, and clarity of the information to be
collected
As noted below in the detailed comments by
module, several elements of data to be collected under this
proposal have no meaning at the state and federal levels, and only
have value, meaning, and usefulness at the local level. Several of
the elements to be collected as described are not available in
smaller geographic areas usually targeted by local agencies in
their community work and if created by local agencies, lose
validity. The social or population level indicators being
requested are not meant for single agency reporting and are best
served for community-wide scorecards or utilized as part of a
community needs assessment. For example, comparing overall
percentages of community indicators (e.g. the unemployment rate,
violent crime rate) to individual local agencies outcomes is
misleading.
Ways to minimize the burden of the
collection of information on respondents, including through the
use of automated collection techniques or other forms of
information technology.
OCS' Dear Colleague Letter of June
17, 2016, states, "Aware of the reporting burden the proposed
Annual Report will entail, OCS will decrease that burden by
establishing an on-line automated system for use by the states and
— at the discretion of the states — the local
agencies, for reporting that allows in many instances
auto-populating data from one year to the next and a variety of
data entry processes (e.g. manual entry and automatic upload)."
It is important to note, this online automated system may benefit
State Lead Agencies (grantees), but will most likely not benefit
local Community Action Agencies (sub-grantees). Sub-grantees
(local agencies) will still need systems that track customers,
services, outcomes, and given the requirement of many funding
sources (including other federal Department of Health and Human
Services funding) requires additional and sometime separate data
collection systems. State grantees do not have the funding or
capacity for the most part, to create systems that will streamline
or decrease the burden outlined here. Furthermore, new state-level
systems will do little to address the needs of local agencies for
improved management information systems with the capacity to
collect and report the data required by ROMA Next Generation. OCS
and the federal government provide funding to local sub-grantees
that require separate systems and prohibit the aggregation of
data. To minimize the burden on respondents, DHHS should examine
its own practices to ensure that all DHHS-funded programs in the
field can use data collection systems that can be easily
integrated and "talk" to one another to produce reports
that meet the needs of local agencies, states, and OCS.
|
|
Comment 67 (Lorain County Community
Action Agency ; Other/Unknown in OH)
|
|
It takes 1 weeks’ worth
of time on the Director to create this report:
1 day
(8 hours) to extract data (numbers) from multiple systems (Ohio’s
OCEAN System Head Start data, manual Excel systems).
1
day (8 hours) to analyze/compile data and input into OCEAN
2
days (16 hours) in compiling narratives which include
explanations, notes, write-up of outcomes for the various
programs.
1 day (8 hours) to input the narratives into
to OCEAN
In addition to this time, we approximately
have 8 hours each from others who are forwarding data to the
Director. There are individuals from Head Start, Food Service,
Youth Programming and CSBG Program Coordinator. This is an
additional 32 hours
Therefore; we calculate the
current quarterly expense to be 72 hours at $50.00 per hour to be
$3,600.00. This quarterly number, times 4 equals $14,400.00
annually.
Now to go further looking at the ROMA Next
Generation’s requirements, we calculate the work, detailed
above, to be 3 times the current work load. The annual cost will
rise in labor to $43,200.00. This is the cost of a full-time
worker per year just to perform the ROMA Next Generation
reporting.
We calculated 3 times the current cost for
the following reasons:
· There are many more
categories that will involve more specific differentiating of the
data being collected,
· In these categories,
more judgment in collecting the data will be required, and
·
To collect the data more systems will be needed at the collecting
points to ensure the data is inputted properly.
Basically,
we foresee a new data collection and reporting system will need to
be created to handle the data. Will the benefits of having this
data justify the cost?
As an agency, we do not believe
that the additional time and cost improve programming and
outcomes.
Thank you for considering what we have
presented. Should you need to contact us, you may contact myself
at (440) 204-3153 or our President and CEO Jackie Boehnlein at
(440)204-3150
|
|
Comment 69 (Lycoming-Clinton Counties
Commission For Community Action (Step), Inc.; Eligible
Entity/Local Agency in PA)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility Each grant administered by STEP has
specific requirements, often the outcomes are able to integrate
into the CSBG IS, but sometimes they are not. Having the
opportunity to insert your own NPIs is important. The information
is valuable, but the extent of what needs to be gathered should
not over burden the Community Action Agency. The report should NOT
be called the CSBG Annual Report – rather it should be
called the COMMUNITY ACTION ANNUAL REPORT. CSBG mandates
collection of data for all program activities, not just for ones
utilizing CSBG funds. Thus calling it CSBG Annual Report is
completely inaccurate and makes it confusing!
2. The
accuracy of the agency’s estimate of the burden of the
proposed collection of information From an agency-wide
perspective, the amount of time that needs to be focused on
CSBG-related items is intense. Considering the Community Needs
Assessment, Organizational Standards, Reporting for CSBG-IS, plus
all the requirements and tasks that go along with these items –
the overall time burden is at least one FTE staff member. This
staff member is often Administrative in nature with a higher
capacity to understand the unique inter-related nature of
Community Action. STEP’s Management Team made up Chief
Operations Officer, Chief Financial Officer, and President &
CEO are also heavily engaged in all CSBG-related items. Yet, the
funds received through CSBG are only approximately $330,000. When
our budget is $24 Million and everything falls under CSBG/ROMA
fundamentals, it is a lot of effort related to such a small amount
of funds. STEP has recently developed its own online data
collection and reporting system through Client Track and invested
significantly in it – any changes to this system will be
costly. On the State level many of the systems in use cannot be
migrated to Client Track due to State agency requirements, so
multiple systems still have to be utilized.
3. The
quality, utility, and clarity of the information to be collected
Many of the indicators are the same or similar. The biggest area
that there are issues with is the Community Level Indicators.
There is no way to successfully determine percentages for what is
being asked with making significant assumptions about how
Community Action Agency programs have sole responsibility for
community change. This is unrealistic; community change only comes
through partnership and collaboration with other entities.
Community Action Agencies are often the catalyst for community
change, but not the sole responsible party.
4. Ways to
minimize the burden of the collection of information on
respondents, including through the use of automated collection
techniques or other forms of information technology. Over the last
five years, STEP has invested significant time in terms of
information technology to support the CSBG-IS Report. Over
$200,000 has been invested between purchase and training time for
the Client Track software deployed in 2014. STEP has had
significant challenges with getting State entities able to
transfer client information into our system, thus multiple systems
are still in use throughout the agency. A new system provided by
OCS will only complicate what we are currently doing. Client Track
has been developed to mirror the requirements of CSBG-IS on
individual/family level and provide Outcome Matrices for each with
trackable goal plans.
|
|
Comment 70 (M.E.O.A.G. Inc.; Eligible
Entity/Local Agency in OH)
|
|
Nancy the time and effort for
Lou and I to come up with the cost for ROMA next generation would
be extensive of itself, but what part I have seen, I would
estimate new software, hardware, staff time and training if needed
we probably will have around 20% of our 2 year grant of $445,000
if not more. Our cost for the new technology will be the same
price as those agencies with bigger CSBG grants then ours. Thus
the burden of implementation will hit us harder financially. These
are just my thoughts with no facts to base my comments on.
|
|
Comment 71 (Maine Community Action
Association ; Other/Unknown in NE)
|
|
Upon reviewing the proposed
new requirements for the CSBG Annual Report, MCAA was disappointed
to find that recommendations by the National Community Action
Foundation (NCAF), the National Community Action Partnership
(NCAP), and state and local agencies were not included in the
revised system requirements. Instead, the reporting requirements
are triple what the current survey necessitates; forcing CAP's to
spend more time on administrative tasks. CAP agencies, state
government and the federal government would be better served by
streamlined reporting, such as pared-down ROMA NG system
recommended by NCAF in their April I 3, 2016 comments. The
proposed new reporting standards also lack flexibility, making it
difficult for Caps to capture some of their achievements.
Paradoxically, while the new standards have more reporting
categories, they may not allow an agency to fully capture the
benefits of some of their programs. In the case of CAP agencies,
one size does not fit all.
The proposed system for
information collection will be overly burdensome to our agencies.
We share ACF's commitment to increasing accountability on the part
of CAP agencies. We agree that the goal of our services must be to
make material improvements in the lives of the families we serve.
However, we are concerned the new reporting requirements under
ROMA-NG will greatly expand the work of CAP staff without
improving the reports ability to measure that improvement.
The
current survey requires reports on 374 data points, while the new
proposed report includes more than 1,100.Additionally, there are
no new resources provided to agencies to cover the cost to compile
all of this extra information. This result will not be improved
services, but fewer services as staff time is diverted to address
reporting requirements.
An additional cost will be
implementing data and reporting systems to manage and submit the
new ROMA NG reports. The additional expense for software and
training of staff are not covered by this initiative. We are very
concerned meeting these reporting requirements may require double
entry of data, an inefficient and expensive option. We strive
every day to maximizing our staff members' time and focus on
service delivery over reporting. We would prefer that our staff
not be needlessly bogged down in administrative requirements, and
that our funding be used where is it most necessary ...helping the
individuals that we serve.
MCAA endorses the
recommendations brought forward by NCAF and NCAP to improve
reporting requirements without making them excessively burdensome
to agencies.
We appreciate your attention to these
concerns and hope that you will consider them carefully as you
finalize the requirements for the proposed ROMA Next Generation
system.
|
|
Comment 72 (Massachusetts Community
Action Agency Planners’ Group; State Association in MA)
|
|
The Mass CAP Planners’
Group met on Friday, March 25 from 10am to 2pm, in order to
discuss ROMA Next Generation materials distributed by OCS. Since
the time available was short, the group focused its discussion on
higher-level responses, while recognizing that many elements of
the materials distributed deserve more detailed exploration.
OVERALL RESPONSE TO THE ROMA-NG INITIATIVE. The
Community Action staff involved in the March 25 meeting, while
recognizing the complexity of the ROMA-NG program and the need for
further work, reached the following general conclusions.
·
This initiative is vitally important. ROMA-NG is an initiative of
great strategic importance to the Community Action network. It
responds to challenges which the network has experienced in
measuring its impacts and in communicating those impacts to
stakeholders. As such, it is very valuable.
·
This is a complex and costly process—and CAAs will need help
with the required investments. ROMA-NG will require changes in the
orientation and day-to-day workflow of agencies, in the interest
of greater rigor in data collection. To the extent that these
changes are manageable, they are welcome; however, many will
create increased time and cost burdens. As the process of
implementation goes forward, agencies will need support in
handling new costs, training requirements, and associated
conversion issues.
· More time for
consideration and discussion would be very helpful. The
participants in the discussion greatly appreciated the opportunity
to offer input. However, the volume of information provided by OCS
and the complexity of some of the proposed changes make it
difficult to create a complete response in the timetable allowed.
We respectfully request a two week extension (until April 15),
which we hope would balance the timetable needs of OCS with the
imperative of thorough and responsible review by the CAA network.
· A carefully phased implementation process
will be crucial to success. A very complex set of systems changes
like ROMA-NG is obviously challenging to roll out over a wide
range of diverse organizations. As we know from other systems
change efforts, introducing new information and new requirements
in a stepwise fashion can help to support adoption and
satisfaction. The ROMA-NG program has the advantage of being
readily adaptable to “staging” For example,
o
The Theory of Change is readily understood, and does not require
agencies to change their day-to-day operations. This tool can be
provided to agencies quickly.
O Individual and Family
NPIs, while they may benefit from some adjustment, incorporate a
straightforward set of outcomes measures (as well as a small
number which are problematic.) The overall approach used will be
readily understood by anyone familiar with outcomes-based
management. With the exception of a small number of measures which
need further refinement, these NPIs can be implemented on a fairly
short timetable.
O The Community NPIs includes a
mixture of some measures that are straightforward and easy to use,
and some that present serious methodological challenges. While a
subset of measures proposed could be implemented readily, some
deserve further study or need to be reframed. More time may be
needed to work through these issues.
O Demographic
Reporting, with the changes as proposed, introduces the powerful
but challenging concept of longitudinal tracking. This is likely
to require rethinking client tracking and data collection systems
for many, if not most, agencies. There will also be a need for
significant new investment in technology. Consequently, this is
likely to be the element that requires the most training,
consultation and technical assistance in advance of
implementation.
COMMENTS ON PRESENTATION AND
ORGANIZATION. Most of the comments provided in this summary are
substantive-- in the sense that they relate to the content of the
materials made available for review, rather than the organization
of the documents or the language used. However, a number of
concerns emerged in discussion which could be reduced by
relatively straightforward changes in presentation. These are
identified below.
· Clarify that organizations
are not required to report on all NPIs. While at present most
agencies feel no pressure to report on all of the available NPIs,
the level of change required by ROMA-NG has led some agencies to
ask if they are “expected” to use all of the new
measures. Clarifying that no specific NPIs would be “required,”
or seen as necessary to a positive evaluation of agency
performance, would be helpful.
· Clarify
whether organizations would be asked to roll up reporting on NPIs
to include multiple programs. At present some organizations
separate reporting on NPIs by program—an approach which
might become more difficult using the new standards. Clarifying
that multiple program outcomes could be reported under each
NPI—and perhaps offering a comment on how agencies could
break out program-by-program reporting for their internal
use--would be helpful.
· Crosswalk NPIs with
services and strategies. A general rubric connecting services and
strategies to some possible NPIs associated with them would be of
practical use to planners and program staff (although it would
need to be understood as a guide for thinking about program
outcomes, not as a rigid framework.)
·
Crosswalk “new” and “old” NPIs. Similarly,
showing a potential framework for relating the new standards to
the old reporting framework would be helpful (although clearly it
would not be a simple matching process; some “old”
NPIs would not translate to the new measures, and many “new”
NPIs have no directly corresponding measure in use now.)
·
Clarify how the Community level NPIs are linked to the reporting
framework for Collective Impact. The use of a collective impact
tracking framework is potentially useful, but it is unclear how it
would relate to specific NPIs. An explanatory statement would be
helpful.
· Provide an overview of how the
various elements of the proposed reporting structures fit
together. It would be useful to show more distinctly, perhaps in
schematic form, how the building blocks of National Goals, NPIs,
Services and Strategies listings fit together. More broadly, it
would be helpful for planners and other stakeholders to have an
outline of the connections between the new measures, the existing
systems for reporting using the Annual Work Plan, and the
Community Action Plan/Strategic Plan requirements.
·
Revisit the numeration of NPIs. While an apparently small point,
the numbering of NPIs caused some confusion about how reporting
would work. Specifically, there is a logical problem with how the
numbered NPIs relate to their subsets; in some cases, the
subcategories that would provide an appropriate parallel structure
are missing. In addition, it may be useful to adopt a numeration
system which incorporates both Individual/Family and Community
indicators.
· Make the option of reporting on
“other” indicators and services more evident
throughout. The group noted with appreciation that an option of
“other” was provided in many places throughout the
materials; however, it is not consistently present.
COMMENTS
ON ‘THEORY OF CHANGE.’ The TOC graphically depicts the
essential elements of community action. It embraces the people we
serve and reinforces that we are all part of the same community.
However, while the language used does articulate the importance of
fostering stability and economic mobility as well as the overall
health and vibrancy of the community, it does not mention our
commitment to reducing poverty. While we do need to add layers of
meaning to our discussion of our work in terms of the need to
address inequality and share prosperity, we do ourselves, and more
importantly, the people we serve, a disservice when we leave out
the word ‘poverty’--as it is the fundamental condition
we must address in the richest country in the world.
COMMENTS
ON INDIVIDUAL /FAMILY LEVEL NPIs. The proposed Individual/Family
Level NPIs cover an appropriately wide range of program
interventions, and offer a much-improved framework for reporting.
It is notable that all of these measures are outcomes, as opposed
to the existing NPI structure, which includes many output
measures. The long-term benefit of these proposed changes is
likely to be significant and positive—supporting increased
credibility for the Community Action system as a whole, and
allowing for increased comparability with other publicly-funded
programs (e.g., Head Start, WIOA).
Specific
Observations. While the overall thrust of the NPIs is extremely
helpful, a number of specific issues suggest the need for
continued refinement.
· A need for interim
measures was identified in some domains. This could potentially be
addressed using a scales-and-ladders approach.
·
Specific concerns were identified with the focus on “Living
Wage” jobs. Since access to living wage jobs is largely
determined by macroeconomic conditions over which individual CAAs
have little control, this was seen as a double-edged sword for
reporting purposes. In a related note, the MIT Living Wage
Calculator was seen by some as unduly complicated for use in this
context.
· The absence of a measure
specifically related to gains in income was seen as a weakness. It
was noted that income gains are more frequently demonstrable than
acquisition of a living wage job, and represent an intuitively
meaningful standard which non-specialist stakeholders understand.
· The effort to align measures with those in
use in other systems (Head Start, WIOA), was appreciated. There
may be a need to further check this alignment, and to ensure that
terminology and definitions are consistent. A particular instance
noted was Items 3 and 4 under Health and Social/Behavioral
Development, which appear to diverge from Head Start performance
measures.
· Several NPIs were considered weak
in terms of their level of specificity and/or probable consistency
across agencies reporting. These included the Item 2 under Health
and Social/Behavioral Development, “number and percent of
individuals who demonstrate improved health and well-being,”
Item 1 under Income and Asset Building, “number and percent
of individuals who achieve and maintain capacity to meet basic
needs” and the similar “basic needs” measure
under Individual and Family Stability. For each of these items, a
listing of potentially valid objective standards would be helpful
as guidance
(Option Three under Individual and Family
Stability addresses this issue.)
· The options
offered under Individual and Family Stability raise larger issues
related to data collection and analysis. While Option One reflects
our shared interest in beginning to quantify the differential
impacts of bundled services, it appears out of place as an
NPI—this could be profitably considered in
conjunction with changes to the
Demographics report,
since it requires a structured process for longitudinal
assessment. (The inference that multiple outcomes are equivalent
to “stability” might also be questioned.) In Item 2(b)
under Option Two, there is a reference to multi-year tracking,
which is a very important goal—but one for which many
agencies do not have appropriate data systems. This item is
followed by a discussion of cross-tabulation of outcomes with
numbers of service contacts, which is also likely to be a
technical challenge for a large proportion of agencies. Finally,
there is a problem with outcome statements under both options in
that it is unclear what the denominator would be for “number
and percent of individuals”. Overall, these NPIs raise
important questions for which Community Action needs to develop
answers—but the technical framework through which they can
be answered is, at best, still under development.
COMMENTS
ON COMMUNITY LEVEL NPIs. The development of an ambitious set of
Community
Level National Performance Indicators, and
the introduction of the Community Level of Effort Progress Tool,
are welcome changes which have the potential to create powerful
statements of impact. They support the strategic imperative of
elevating the visibility of the community work done by CAAs.
Flexibility is key here; it is important allow CAAs to describe
the work they already do, and not to tie them to a set of
indicators which are unrealistic or unduly rigid.
Specific
Observations. While the strategic direction represented by the
Community NPIs is very positive, there are significant issues with
some of the proposed indicators.
· There are
fundamental problems with indicators which use changes in
population-level ratios as the measure (e.g., “percent
increase in children who are kindergarten-ready” or
“increase in adult literacy rate.”) These measures,
which are appropriate for large-scale public health interventions
and efforts of similar scope, are unlikely to produce useful
information in the context of a single agency’s actions over
one year. The group is aware of the OCS guidance which allows for
individual definition of “community”—however,
this is unlikely to eliminate the problem. If “community”
is defined as a service population (which would be necessary in
most cases in order to report meaningful outcomes), the measure is
essentially reporting on the success of a particular CAA program;
this is information that fits more appropriately under Individual
and Family Level NPIs.
· Conversely, most of
the NPIs which use scalar measures (e.g., “number of jobs
created”, “number of safe and affordable housing units
developed”) are workable.
They may require
further definitions or clarifications (some of which are noted
below).
· For a number of Community NPIs, there
may be need for further guidance on the conditions under which it
is appropriate for agencies to report that jobs, housing units, or
other resources are “maintained”. This type of measure
can lack meaning unless there are specific standards which ensure
that the reporting CAA is actively engaged in activities which
support “maintenance”.
· In a
number of Community NPIs, the phase “number and percent
increase/decrease” is used; this is generally not meaningful
unless the denominator in question is defined. In most cases,
these NPIs would be clearer without the “percent”.
·
The organization of Community NPIs related to “Civic
Engagement and Community Involvement” is confusing. The
intent to separate contributions by “people with low
incomes” and others is understood, but splitting them under
two National Goals creates a conceptual barrier for the reader.
The indicators listed under National Goal 2 (non-low-income
contributors) are generally weak (they represent number of donors,
number of volunteers, and the number of people who come to
meetings, none of which may be very meaningful.) However, the
Indicators under National Goal 3 (low-income persons) are clear
and succinct.
· The introduction of the
Community Level Efforts in Progress Tool was seen as a positive
step. This tool will require additional training and technical
assistance for agencies, but it provides an initial effort to
extend the agencies’ reporting horizons to efforts which
stretch across multiple years. There was some confusion, however,
about how (and whether) this tool would be linked with specific
community NPIs.
COMMENTS ON INDIVIDUAL/FAMILY
STRATEGIES AND COMMUNITY LEVEL STRATEGIES. The listings of
strategies and services for both Individual/Family and Community
are extensive and largely non-problematic. There was, however,
some need for more explanation of how this data would be reported,
and whether it would be integrated with NPIs.
COMMENTS
ON DEMOGRAPHIC REPORTING. For the most part, changes regarding the
specific data elements to be collected for demographic reporting
were seen as non-problematic. However, the approach to measuring
CAA population changes against a baseline was seen as seriously
flawed --for reasons which relate to both basic statistical
methodology, and to the practical limitations faced by individual
agencies. The following are some key points made in the
discussion.
· The goal of gathering information
on the experience of CAA consumers over time is extremely
important, and should be pursued energetically. The network is
interested in conducting long-term evaluations but it will require
a more fully developed research model.
· The
two-point process shown here will not produce meaningful data, for
reasons including the following.
O To create
meaningful group comparisons, the two groups being compared must
be composed of the same individuals (e.g., a cohort.) To create a
cohort composed of ALL of a CAA’s consumer base, data
systems need to exist which will consistently preserve a “starting
point” data set and then match it with a “second point
in time” data set. Most client tracking systems now used by
CAAs overwrite data as it is updated, making this close to
technically impossible. While it is possible to create a system
which can produce this result, it is very likely that such a
system does not exist now.
O If such a system is to be
developed, the data points being compared need to be thoughtfully
selected; the appropriate time frame also needs to be discussed.
While we may be interested in employment and education changes,
for example, tracking family composition and age may produce
mostly “noise.” It is also unlikely that real change
will be seen, for most households, in one year. Multi-year
tracking will be needed, but it brings with it additional
technical complications.
O A longitudinal tracking
model will require broad operational changes for many CAAs.
Currently, many CAA consumers receive services only once a year.
More generally, CAAs may not have systems in place for
re-assessment. Changes in this regard could be very positive, but
they will take time and money.
|
|
Comment 74 (Merced County Community
Action Agency ; Other/Unknown in CA)
|
|
I am writing to give our input
on ROMA NG and the challenges it possesses for the network. Not
only is ROMA NG being rolled out at the same time that the network
has adopted and begun to report on Organizational Standards; which
is time-consuming, but there is little evidence of the real value
it has to the network, outside monitoring agencies, or the
communities we serve.
Not Community Driven:
ROMA
NG seems to be rooted in a false pretense that the Agencies need
more help in planning and reporting on outcomes, when in fact, the
Agencies across this nation are excellent at planning the needed
resources and strategies for their communities. This is outlined
in our Community Action Plans which cumulate data from the
community leaders, the low-income, and the agencies themselves.
The key here is that CSBG dollars are designed to be flexible
because the community, as well as the needs of the low-income, is
constantly evolving. ROMA NG specifically states that plans must
be followed and benchmarks must be met, or extensive narratives
will be required to justify the changes. Data is being used to
lead us in the community services we provide. To that end agencies
will look better, if they only choose strategies that are easily
trackable and fall within specific categories, not the needs of
the community.
Too Expensive:
The
additional reporting requirements (funds) that are required for
ROMA NG will come directly out of services to the low-income.
Currently, we are being told that states will have the option of
increasing (doubling) their percentage of administration in order
to comply with ROMA NG, a return on investment report will be
contracted to the tune of almost $7 million, and an annual
Helping People and Changing Lives Since 1965 report
will be done at a cost of $350,000. These figures do not include
the cost to the local organizations in additional staffing to
track and report on case management and input data (2 FTE
employees). Lastly, most agencies will have to invest in software
that produces unduplicated client counts. Those estimates run from
$75,000-$250,000 depending on the program and the size of the
software system. None of this money will be hitting the ground and
bettering people's lives; but deepening the community's distrust
of federal bureaucracy when services are cut.
Not
Evidence Based:
My question is, to what end? History
has proven that when tasked with additional reporting
requirements, little, or nothing is done with the data. As a
matter of fact, 20 years ago we began reporting under ROMA
standards only to be told four years ago, that the data was hard
to understand and had very little usable information. A waste of
16 years; only to be augmented with ROMA NG which does very much
the same thing, with more emphasis on narratives, long range
planning, unduplicated counts, and case management. The people
that designed the ROMA NG reporting system do not comprehend the
work involved in changing reporting systems, tracking thousands of
people who live in a constant state of crisis that are accessing
our services through hundreds of different doors. Imagine,
stopping each person in a line for food in order to record basic
demographics on them. The wait gets longer and the service is
slower. The current reporting system disincentives emergency
services which are critical in each community, because they are
hard to case manage, and plan for. ROMA NG is not proven to
produce better outcomes. Since ROMA NG is proprietary in nature,
it is very hard to explain to elected officials, community
leaders, and staff. The verbiage is not standardized in the
industry and is difficult to understand. The verbiage used to
measure success is not inclusive of all of the services we provide
and forces you to make a decision that may or may not be accurate.
No Network Support:
I am urging you to
reconsider the investment of both time and money on what this
"updated" reporting system will cost. Overwhelmingly,
the network has spoken up against ROMA NG just to be told it is
moving forward. With the network opposed to the implementation as
it currently stands, it has very little chance of proper
implementation, or success in the field.
|
|
Comment 75 (Michigan Community Action
; Eligible Entity/Local Agency in MI)
|
|
The collection of information
does not meet the threshold of necessity or utility. The package
is costly, burdensome, provides useless information, and requests
data on outcomes that are beyond the capacity of the Community
Services Block Grant (CSBG). OCS' development of CSBG Performance
Standards has been a model of positive steps for insuring
accountability and capacity to maximize the potential of the CSBG.
Changes to the CSBG Annual Report are unnecessary.
If
there is a need for additional data and information on CSBG
programs, an alternative approach to changing the CSBG Annual
Report would be to commission an annual research study to measure
the impact and effectiveness of typical CSBG interventions. Most
CSBG grantees could export a wide variety of data that would allow
experienced researchers to answer narrowly drawn research
questions and to respond to data needs from OCS and other
stakeholders.
Address the unique issues of Public
entities that receive CSBG. Public agencies (sub-grantees) that
sub-contract most or all of their CSBG funding to other service
providers may face added challenges of requiring their grantees to
collect data in accordance with ROMA Next Generation requirements
since it may limit agencies to contracting with providers that
have the requisite data collection capacity or require additional
technical assistance to meet the reporting requirements. This
challenge is not addressed in the proposed reporting package. In
Michigan, many Public entities work with faith-based providers and
local cooperatives. These neighborhood based organizations do not
have the resources to report at this level.
Question 2:
The accuracy of the agency's estimate of the burden of
the proposed collection of information.
The current IS
survey includes 300+ data elements to report. The proposed CSBG
Annual Report includes more than 1,000. This increased burden of
reporting comes with no new resources.
The burden on
local agencies is not impacted by the implementation of the Online
Data Collection system (OLDC). To comply with the draft CSBG
Annual Report local agencies will need to modify existing systems
and states routinely pass on reporting burdens to the local level
because of their own financial and technical constraints.
The
agency's' estimated time burden of 242 hours per sub-grantee
(local Community Action Agency) is significantly under estimated.
Based on our calculations, the burden to our local agencies in
Michigan would be 431 hours per subgrantee. The cost of this time
burden is estimated to be $1,437,385 for Michigan. This does not
include burdens on the State of Michigan or Michigan Community
Action. These additional hours are estimated to be at least 231
hours for a cost of $26,565 (training, data clean up, system
design, working with consultants, etc.).
There is no
reduction in burden for local agencies by the online data
collection system described in the by OCS. OCS' Dear Colleague
Letter of June 17, 2016, states, "Aware of the reporting
burden the proposed Annual Report will entail, OCS will decrease
that burden by establishing an on-line automated system for use by
the states and - at the discretion of the states -
the local
agencies, for reporting that allows in many instances auto-
populating data from one year to the next and a variety of data
entry processes (e.g. manual entry and automatic upload)." It
is important to note, this online automated system may benefit
State Lead Agencies (grantees), but will most likely adversely
impact local Community Action Agencies (sub-grantees) with more
state imposed deadline and data requirements.
Question
3: The quality, utility, and clarity of the information to be
collected.
Maintaining a flexible data reporting system
and engaging researchers to mine the vast trove of data already
available is a more cost effective approach than increasing the
data provided through the CSBG Annual Report. As noted below in
the detailed comments by module, several elements of data to be
collected under this proposal have no meaning at the state and
federal levels.
Several of the elements to be collected
as described are not available in smaller geographic areas usually
targeted by local agencies in their community work and if created
by local agencies, lose validity. The social or population level
indicators being requested are not meant for single agency
reporting and are best served for community-wide or regional. For
example, comparing overall percentages of community indicators
(e.g. homelessness, poverty level) to individual local agencies
outcomes is misleading
Module 4, Section A:
Characteristics for NEW Individuals and Households - Data Entry
Form is an unnecessary document that will not produce quality
data; it will have little utility at a local level; and it lends
itself to meaningless data at a state and national level. It
should be removed.
Module 3, Section A: Many of the
indicators used here are inappropriate for reporting by local
sub-grantees to States and by States to OCS and will produce data
of limited utility. Several challenges underscore this
point.
First, community-level data on indicators is not
always available at the level (e.g. regional or zip code/census
tract) or population (e.g. low-income children aged 0-5) that
matches the area and population targeted by the agency.
Second,
community-level indicators are influenced by a wide number of
sources that would most likely overwhelm the impact of even
successful community initiatives, a fact that significantly limits
their utility for providing meaningful information about
outcomes.
Third, any meaningful data about the outcomes
of community-level initiatives would require a rigorous program
evaluation and could not be determined on the data collected
through the Annual Report alone.
Question 4: Ways to
minimize the burden of the collection of information on
respondents, including through the use of automated collection
techniques or other forms of information technology.
I.
OCS' Dear Colleague Letter of June 17, 2016, states, "Aware
of the reporting burden the proposed Annual Report will entail,
OCS will decrease that burden by establishing an on-line automated
system for use by the states and - at the discretion of the states
- the local agencies, for reporting that allows in many instances
auto-populating data from one year to the next and a variety of
data entry processes (e.g. manual entry and automatic upload)."
It is important to note, this online automated system may benefit
State Lead Agencies (grantees), but will most likely not benefit
local Community Action Agencies (sub-grantees). Sub-grantees
(local agencies) will still need systems that track customers,
services, outcomes, and given the requirement of many funding
sources (including other federal Department of Health and Human
Services funding) requires additional and sometime separate data
collection systems. State grantees do not have the funding or
capacity for the most part, to create systems that will streamline
or decrease the burden outlined here. Furthermore, new state-level
systems will do little to address the needs of local agencies for
improved management information systems with the capacity to
collect and report the data required by ROMA Next Generation.
OCS
and the federal government provide funding to local sub-grantees
that require separate systems and prohibit the aggregation of
data. To minimize the burden on respondents, DHHS should examine
its own practices to ensure that all DHHS-funded programs in the
field can use data collection systems that can be easily
integrated and "talk" to one another to produce reports
that meet the needs of local agencies, states, and OCS.
To truly minimize the
burden of the collection of information, DHHS and HUD and DOE
should take 12-24 months to review their combined data
requirements of shared grantees. Utilizing one data system for
federal programs that impact low income families would allow for
more robust data alignment and outcome data without the need to
ask any grantee for additional information.
|
|
Comment 78 (Missouri agency Inc. ;
Other/Unknown in MO)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility.
I agree that the
collection of data is necessary to substantiate the work that we
do, and prove that our work is valid. I also feel that the
community action network in Missouri is ready to begin the process
of ROMA Next Gen, but I am worried that many states are not up to
the task yet, which can potentially be very harmful to community
action, overall. If the other states do not have the resources
needed to collect the required information, how will that affect
the entire network when legislators and other officials review the
work we have done?
2. The accuracy of the agency’s
estimate of the burden of the proposed collection of information.
Considering that the current IS report includes 300+
data elements to report, and the proposed CSBG Annual Report
includes more than 1,000 data elements, I believe that OCS’s
estimated time burden of 242 hours per sub-grantee (local
Community Action Agency) is somewhat under estimated. Based on our
calculations, the burden to our agency would be 396 hours. The
cost of this time is estimated to be $8,740. Please note, this
estimation does not include the T/TA that will be needed for our
employees to understand how to complete the new report.
3.
The quality, utility, and clarity of the information to be
collected.
If the proposed number of general
indicators were decreased, for example, 2-4 general indicators for
each of the domains, then the aggregate data collected from the
entire network would be more focused and refined. Which would then
be much more meaningful data to present to legislators and
funders? To show more localized impact, agencies should still be
allowed to track other outcomes using local indicators more
specific to their unique programs and communities.
For
more clarity in the information collected, I feel that the data
would have more impact if the indicators were worded similarly to
other programs, such as Head Start and HUD. If all programs
reported using similar indicators, then, community action would
have more impact on a national level.
4. Ways to
minimize the burden of the collection of information on
respondents, including through the use of automated collection
techniques or other forms of information technology.
In
Missouri, we have been using a state-wide data system for many
years. Several other states do not have this resource, which is
absolutely necessary for collecting accurate data. The proposed
on-line automated system is a nice thought, but much of our
agency’s reporting comes from many different data tracking
systems. Our other programs, Head Start, Women’s Wellness,
Housing/HUD, and Weatherization all have their own systems. Unless
OCS can develop a system that can be used in conjunction with all
of the other systems, the proposed on-line automated system will
not work for local agency. Therefore, the burden of the collection
of information at the local level will not be decreased in any
way. It is still going to be a large and expensive burden on our
agency.
|
|
Comment 79 (Missouri Community Action
Network (Can); State Association in MI)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility
The overall concept of the
information to be gathered for the purpose of validating the work
of community action is absolutely necessary. The Missouri
Community Action Network is prepared to meet new requirements of
information collection and appreciates the work that has been done
thus far. Although we as a state feel comfortable with the idea of
ROMA Next Generation, we are concerned with the overall impact to
the network if other agencies and states do not have the resources
or training to collect information within the proposed framework
of indicators and requirements.
2. The accuracy of the
agency’s estimate of the burden of the proposed collection
of information
The time burden seems slightly
underestimated. The level of expertise to conduct some of the
proposed data collection and analysis is heightened as well.
Additional skill sets may be needed, which would increase the
overall cost burden associated with the CSBG Annual Report. The
investment in the infrastructure for human resources, expertise,
and systems is not present currently to support the proposed
collection of information.
3. The quality, utility, and
clarity of the information to be collected; and
Quality:
The proposed set of indicators seems to focus more on quantity vs.
quality. Our recommendation would be that the numbers of general
indicators are decreased (although we know not all agencies must
report on each one). If a smaller number of indicators were used
to aggregate data to the state and national level, this may be
more meaningful to decision makers. Agencies could then use local
indicators to track additional outcomes based on their unique
programs (For example, if there were 2-3 indicators relating to
each of the domains of CSBG to provide meaningful aggregated
data). The local analysis expectation could then be strengthened
to fuel the ROMA cycle within the evaluation and assessment
portion.
Utility: To the point above, we feel the
information would be most useful in fewer, more meaningful
indicators that can show national impact of community action in a
more concise way. Also, it would seem appropriate to use the
language of other indicators that are already established through
Head Start or HUD to promote further consistency.
Additionally,
our statewide network would request access to the local, state,
and national comparison data after compiled to use and share
locally.
Clarity: We respect
the strengthened focus on community and believe it is necessary.
However, many of the proposed community indicators could be
confused with individual or family level indicators. The meaning
tends to be similar, as well as the unit of measurement, while the
wording is different. Examples include:
·
Community Level, Education #9 = Family Level, Education #8, 9,
10.
· Community Level, Education #5 = Family
Level, Education #7.
Next, there are terms we have
identified that would need clarified definitions before moving
forward to ensure the network is collecting similar
information:
· Low-income
·
Community
· Living wage
·
“New” in characteristics report
Lastly, the
variation of percentages, numbers, and rates seems confusing and
inconsistent. We understand the need to examine percentages when
looking at an increase or decrease and numbers are helpful in
other circumstances.
The way some indicators are
written causes confusion on what is being counted or measured. The
rates should most definitely be examined, but possibly just in the
community needs assessment to be analyzed for multi-year
trends.
4. Ways to minimize the burden of the
collection of information on respondents, including through the
use of automated collection techniques or other forms of
information technology.
We consider ourselves fortunate
to have a statewide data system capable of producing an accurate
unduplicated count and state CSBG administrators who value this
system as a priority of CSBG discretionary funding. Since 2004,
Missouri has invested more than $1.5 million in our statewide
system, excluding agency staff time. Many states do not have
resources or funds allocated for this. We are increasing our
efforts around consistent use of the system and upgrading to a new
version to prepare for such changes in reporting. Training and
technical assistance (T/TA) will also be necessary to ensure
counting with consistent definitions and systems/procedures for
efficient data collection.
We also know the web based
data tool in the community commons platform will become an
increasingly valuable
asset. Three states currently have a
state level tool to gather community and tract level data from
additional state level data sources. The indicators that are
available through these tools nationally and at the state level
should be considered in relation to the proposed indicators.
Module 2 – Agency Expenditures, Capacity, and Resources:
(pages 23-27)
Overall, we agree with the need to
improve reporting and data collection in community action.
However, it should be manageable with the resources agencies have
available. This is not to say that requirements should not be put
above where many agencies are to obligate improvement, but the
improvement should be possible and with adequate investment via
funds and resources. We know this work is important and do not
want the inability of some to meet the requirements or provide
adequate data to cause community action to appear ineffective. The
NPIs should be able to be aggregated at the national level, be
meaningful to decision makers, and show the effectiveness of
community action.
|
|
Comment 80 (Montgomery County
Community Action Development Commission; Other/Unknown in PA)
|
|
I am writing to provide
comments regarding ROMA Next Generation:
· We
believe that the amount of data that would have to be collected is
burdensome and excessive.
· The cost of
purchasing and implementing the new system would be substantial
and require a significant amount of additional resources. The
start-up costs, training and additional staff time, etc. could
exceed $100,000.
· If we were required to pay
for the new system out of current CSBG funds it would adversely
affect the number of people we help and services we provide. This
in turn would decrease our impact on the community.
·
The challenges associated with designing an integrated local
system that can produce valid client tracking data as it relates
to progress, analyze situations at various points in time and
measure community impact may be insurmountable at this time.
·
In our estimation, the time needed to implement the new system
from the beginning to fully operating far exceeds the proposed
start date of October 1, 2017. We believe it could take at least
18 months if not longer.
Therefore, we ask that you
please you reconsider how the new system will impact all Community
Action Agencies, the people we are trying to help and communities
we are working hard to change prior to implementing ROMA Next
Generation. Thank you.
|
|
Comment 81 (Multi-Service Center
(Msc) ; Eligible Entity/Local Agency in WA)
|
|
As the CEO of the
Multi-Service Center, a community action agency in Washington
state, I am writing to provide my feedback on the proposed changes
to reporting for CSBG and the cost estimates provided by HHS
related to these changes. I am very concerned about the
implications of the proposed changes as well as the accuracy of
estimates provided by HHS.
· The information
obtained from the report that is being proposed will not have a
significant increase in usefulness or being meaningful, at least
not sufficient to justify the effort required to complete data
collection and submission of the annual report. If the 6 NPIs and
the past reporting process weren’t providing sufficiently
reliable data, I struggle to see how increasing the complexity
helps in this matter. More data doesn’t necessarily equate
to effectiveness, especially given the concerns about capacity to
collect and report data with integrity. In addition, with
increased complexity come increased possibility for ambiguity and
multiple interpretations. How will CAAs in each state not to
mention around the country come to mutual agreement regarding the
meaning of each indicator? A shared understanding of the
indicators is necessary for uniformity and for the data to be
meaningful and valuable.
· I am unsure as to
what methodology OCS used to arrive at the estimate regarding
length of time to complete the new report. Therefore, it is
difficult for me to determine if this estimate truly reflects the
realities of day to day operations of a community action agency.
Furthermore, I did not see any information or analysis of the
existing capacity of CAAs to collect and report data with the
level of complexity outlined in the proposed reporting format.
Certainly it is impossible to assert that every CAA is at the same
level of capacity in this regard as it relates to factors
including but not limited to software/database capacity and
staffing resources.
· Staff responsible for
using the current report estimate that the new format will double
or triple the time it currently takes in order to do a good job
with the proposed process. Given that there are no additional
resources being provided to offset these costs, the increased time
required to complete the report will take time and resources away
from providing the services that fulfill the mission of CAAs in
our communities.
I feel it is critical that OCS take
more time to develop shared definitions of indicators and to
investigate more thoroughly whether or not CAAs currently have the
capacity and resources to meet the new data requirements. At the
very least, it would be helpful to know what methodology was used
by OCS to create its cost estimates. It would also be helpful if
OCS could advocate with other government entities to allow
information to be shared or cross walked.
Ultimately,
we feel strongly that data is a critical element in telling our
story and evidencing the effectiveness of community action in
alleviating and eradicating poverty and we appreciate OCS’
efforts to improve data collection and reporting. We simply want
to ensure that the data being collected is, in fact, accurate,
meaningful, and useful and that collecting and reporting data does
not negatively impact service delivery and outcomes as a result of
not having sufficient capacity to carry out the reporting process.
|
|
Comment 82 (National Association For
State Community Service Program ; Other/Unknown in N/A)
|
|
Thank you for the opportunity
to comment on the proposed Community Services Block Grant (CSBG)
Annual Report. NASCSP appreciates the ability to actively
participate in this process through our current cooperative
agreement, Performance Management System Development and Data
Collection Analysis and Reporting for the CSBG Program, with the
Office of Community Services (OCS). As a training and technical
assistance provider and membership organization for State
Administrators of both the CSBG and Weatherization Assistance
Program, NASCSP continues to applaud and actively support the
federal investment and national commitment to reducing poverty in
our nation that the CSBG and weatherization programs represent.
As part of this cooperative process, NASCSP has
convened, participated in, and promoted a wide range of
opportunities for engagement during the process of drafting the
proposed CSBG Annual Report and served as a conduit for soliciting
and aggregating CSBG Network feedback. We appreciate that this is
a collective, network-driven process that has been rich with
engagement and dialogue. The timeline for distribution and comment
between the initial release of information and the start of the
60-day comment period took place in a relatively short timeframe.
This has left challenges and a remaining need for feedback,
revisions, and areas of compromise to ensure a strong final
product for the three year clearance of the CBSG Annual Report. We
have reviewed the proposed CSBG Annual Report to identify both the
strengths and potential weaknesses. Our feedback is provided below
and addresses the key questions posed by OCS, as well as module
specific feedback.
Practical
Utility, Estimated Burden, and Quality and Clarity of Information
to be Collected
As we work with all levels of the CSBG
Network to revise and strengthen our performance management
framework and modify our performance evaluation system, we believe
that the proposed CSBG Annual Report will ultimately help showcase
robust results achieved by the CSBG Network for individuals,
families, and communities with low-incomes. The CSBG Annual Report
will help us meet the latest reforms and expectations for
performance management across the government and private sector in
order to focus on analyzing results and using the data to improve
programs and the outcomes achieved. ROMA Next Generation
introduces new elements into the CSBG Annual Report to support the
analysis of services and strategies provided and the impact of
these services and strategies on individual, family, and community
change.
The modified CSBG Annual Report will provide
clearer descriptive data that will demonstrate outcomes at the
individual, family, and community levels, and foster improved data
analysis at every level of the CSBG Network. We recognize the
significant improvements within the proposed CSBG Annual Report,
namely the addition of the Strategies and Services reports, the
expanded section on community level work, modernized language and
indicators, and the intent to automate State level reporting in
OLDC.
At the same time, NASCSP urges OCS to recognize
and take into account the critical training and technical
assistance elements necessary to successfully implement the CSBG
Annual Report and capture quality, meaningful data. Listed below
is a high level summary of areas of concern as related to the CSBG
Annual Report and is intended to serve as an “At-A-Glance”
version of this letter. These concerns are elaborated upon in the
next section of this letter:
1. Implementation and
Timeline
The completion of the CSBG Annual Report will
require substantial investments in staff time and resources at the
Federal, State, and Community Action Agency (CAA) levels. In order
for this important endeavor to succeed, it is essential for OCS to
modify the projected implementation timeline to allow for the
necessary training and technical assistance and systems upgrades.
It will be necessary to provide significant training to staff at
all levels of the CSBG Network to ensure all understand how the
different pieces work together and how to complete the various
reporting forms included in the new report. It is also critical to
allow sufficient time for the CSBG Network to evaluate not only
the content, but also the system for collection (either in the
On-Line Data Collection (OLDC) system or other system), which has
not yet been developed. We expect this training will be needed on
an ongoing basis and not just one time at implementation.
2.
Estimated Burden
NASCSP recognizes that the burden
estimate is an average estimation across States and CAAs, and
therefore may either over or underestimates the time and cost
burden, depending on variables existing in each State. While we
support the revised/automated forms, the eventual creation of an
online data collection system, and agree they will yield
efficiencies in the States’ processes for completing the
annual report (which replaces the IS Survey), the new process also
increases State accountability and places new demands on systems
and software. For example, the capacity of the system in terms of
interactivity and functionality for data upload, storage, and
usability is an unknown element in the estimations for user
burden. Additionally, the burden on States and CAAs is compounded
by barriers to sharing data from other federally funded programs
(e.g. Head Start, WIC, HUD, SNAP, WIOA, etc.), because few strong
“bridges” currently exist at the Federal level and
legal boundaries prevent data sharing.
Because the
content (in the form of the CSBG Annual report) is still in its
draft form, it is and will be difficult to accurately project
costs and time investments needed to implement the final format.
Given that both the content to be collected and the processes by
which the data will be collected are still under conceptual
development, making an accurate burden estimate is difficult as it
is dependent on certain assumptions about capacity and
implementation. With these outstanding elements, it is extremely
difficult for State offices, CAAs, and the Federal government to
accurately estimate the true burden of this system, the CSBG
Annual Report, and a revised data collection system.
The
current CSBG Act, passed in 1998, establishes requirements for
OCS, States, and Eligible Entities to collect data and report on
the planned and actual uses of CSBG. Each state must submit a
report on performance, expenditure of funds spent by the state and
eligible entities, delivery of direct services, characteristics of
clients served, and training and technical assistance provided to
eligible entities. This has been accomplished using the CSBG IS
Survey. NASCSP commends OCS for the evolution of the CSBG IS
Survey into the CSBG Annual Report and for taking steps to ensure
our CSBG Network is fully prepared to move forward in the
ever-changing work to end poverty; however, the modifications
discussed in this letter are critical in ensuring the CSBG Annual
Report can be successfully implemented. Our recommendations
include the removal of the New Characteristics Report,
simplification of the community measures, and addition of
developmental measures. These revisions will greatly alleviate the
burden of data collection at all levels while still meeting the
requirements of legislation and launching the CSBG Network forward
into a learning culture rather than a compliance and reporting
culture and further the goals laid out in the Act
.
NASCSP Feedback –
FRN #1 OMB PRA Clearance, CSBG Annual Report
NASCSP
urges OCS to extend the proposed implementation timeline to ensure
the CSBG Network’s capacity to accurately report on their
efforts and outcomes. We acknowledge that time is of the essence;
however, in order to fortify administration of CSBG and best
document the performance gains of this CSBG Network, considerable
revisions and extensive vetting are essential to determining what
can realistically be accomplished and done well.
|
|
Comment 83 (National Community Action
Foundation ; Eligible Entity/Local Agency in N/A)
|
|
Thank you for considering this
response to the Federal Register notice # 2016-14229 on behalf of
the National Community Action Foundation, which represents the
Community Action Agency [CAA] network before Congress and federal
agencies. The notice solicits reactions to the proposed
requirements for the form and content of a mandatory annual report
to HHS by State CSBG grantees. It would be based on reports to
states from their local sub-grantees.
Our perspective
is informed by:
1. Our staff and our Director’s
participation in OCS’ intensive and highly participatory
working groups over three years;
2. Our governing
Board members, all of who are executive directors of CAAs that are
elected by their colleagues; and,
3. Over one hundred
in-depth conversations with Community Action Agency leaders during
the current year alone.
We remain deeply
disappointed in the proposed report and in the expanded management
information system that it will require for data collection. In
April, we responded to the OCS Informal Request for Comment by
proposing direct and clear changes and alternatives. HHS has
adopted only a few of the changes we suggested. The scope of and
elements in the report are similar to the original draft, and
there is only minor improvement regarding the extraordinary
increase in the paperwork burden this proposal creates. Changes
that might enhance local agency capacity and free-up resources for
information that measures CAAs local results were not included.
Our colleagues at the Community Action Partnership also
recommended changes, all of which we supported, but few are
included in the revised proposal.
Before outlining our
extensive concerns, it is critical to note that the proposal does
not accommodate the statutory requirement that OCS allows states
to adopt other comparable systems. The proposed templates and the
online federal interface, or “OLDC,” into which they
National Community Action Foundation ROMA NG Final Comment will be
loaded when complete will apparently allow no other format or
approach to dovetail with the OCS data system or other state
reports. The elaboration of the system elements, right down to the
details of the local project management tracking, suggests OCS is
not prepared to accept alternative reports proposed by a state.
Section 676(b) (12) of the CSBG Act does not require
states and eligible entities to use ROMA. It requires the use of
ROMA or an alternative performance measurement system (and, in
fact, the proposed annual report document makes reference to such
an alternative system). However, the proposed CSBG Annual Report
form does not allow for reporting annual performance of a state
and its eligible entities under any alternative system. Indeed,
the statement in the Federal Register comment request that the
proposed CSBG Annual Report incorporates Results Oriented
Management and Accountability as “required in Section 676(b)
12 of the CSBG Act,” completely ignores the fact that the
Act specifies that states may use alternative performance
measurement systems. This draft framework is not “as
required” unless it can be expanded to include a system,
including a federal data management system that will accept state
report data.
While appropriation language adopted for
2016 is different from the statute, it was inserted to give OCS
the authority to issue the CAA Organizational Standards with which
the network has arrived at consensus. It states that “The
Secretary shall issue performance standards for nonprofit
organizations [entities] receiving funds from State and
territorial grantees… [Which] shall assure the
implementation of such standards prior to September 30, 2016
[2017]?”
The CSBG Act states that each State:
Shall ensure that all eligible entities in the state
participate, in a performance measurement system, which may be a
performance measurement system for the Secretary facilitated
development… or an alternative system that the Secretary is
satisfied meets the requirements of [678E9b)]… and that
each state and all eligible entities in the State…will
participate in the Results Oriented Management and Accountability
System CSBG Act section 678E(a)(1)(A), 42 USC
National
Community Action Foundation ROMA NG Final Comment 9917(a)(1)A)
and CSBG Act section 676(b)(12), 42 USC 9908(b)(12).
Further,
§42 USC 9917(b) states that HHS:
Shall facilitate
the development of one or more model performance measurement
systems, which may be used by States and by eligible entities to
measure their performance in carrying out the requirements of [the
CSBG Act] and in achieving the goals of their community action
pans. §42 USC 9917(a)(2) further requires that:
1.
Each state “annually prepare and submit to the Secretary a
report on the measured performance of the State and the eligible
entities in the State;” and,
2. The
annual report that HHS is required to submit to Congress must
include, among other things, “a summary of each State’s
performance results, and the results of the eligible entities, as
collected and submitted by the States…and any additional
information that the Secretary considers to be appropriate to
carry out [the CSBG Act]. If the Secretary informs the States of
the need for such additional information and allows a reasonable
period of time for the States to collect and provide the
information.”
Acknowledging that the proposal
does not accommodate the statutory requirements of §42 USC
9917(a)(2), the following comments demonstrate the extent of
further concerns regarding the value of the proposal, and its
efficacy as a tool for either “telling the CSBG story”
or as the toolkit needed by local agencies to identify and monitor
the outcome of their work. We anticipate that the leaders of
individual agencies will be estimating the burden of time and cost
as part of their own comments.
We have organized the
overview section of the response as follows:
1. A
summary of our concerns;
2. An explanation of the kind
of information system and reports that would work far better as a
performance management system; and,
3. A final summary
of our proposed next steps in updating the ROMA system.
A
second section contains a detailed section-by-section analysis
that outlines our concerns.
OVERVIEW
The
Community Action network and NCAF share the Administration’s
commitment to strengthening a performance-driven culture,
including the development of a better system to track the way
CSBG-supported investments and projects are meeting their goals.
We also agree that the state and national reports must include
better descriptive data about CSBG network activities and
participants. Furthermore, we agree that such data must also
clearly demonstrate impact and results.
The OCS Dear
Colleague Letter, dated June 17, 2016, described the intent of the
proposed system:
This data will serve as a basis for
program analysis at the local, state, and Federal levels to
increase understanding of what combination of services or
strategies produce the best outcomes for specific populations,
family types, and communities.
The new CSBG Annual
Report should be about continuous quality improvement at the
local, state and Federal levels rather than reporting for
reporting’s sake. We do not believe most of the new sections
of this draft system serve the intended purpose.
Our
five major concerns, discussed in depth in the second section,
are:
1. The investment in the proposal’s
required reporting will be wasting limited CSBG resources if done
before changing CAA data storage and integration systems. The
proposal requires that limited CSBG resources, integral to
community programs, be invested in system changes that will not
improve results management.
National Community Action
Foundation ROMA NG Final Comment
The software OCS will
develop for states to file reports does not include tools for
local agencies to collect and integrate data into individual,
family or project files. The promised “OLDC” for
filing state reports is not the system to create and organize the
data, in local CAA databases, that agencies and states need to
track their performance in achieving annual and strategic goals.
Yet to be developed local data systems that can
produce clean, client-linked information will take much longer to
build and will be prohibitively expensive. Yet, such data systems
remain the cornerstones of results-oriented information, and
acquiring them should be the priority for resource allocation in
the coming years. No national or state report can be valid if it
is based on the poorly-integrated information now stored in
overlapping federal and state software systems holding client and
project files as required by CAAs’ multiple funders.
2.
The federal requirement for community project-level datasets is an
inappropriate and unprecedented intervention in local agency
projects. The scale of the draft mandated report is vastly larger
than the current system.
a. In Modules 2-4 there are
1,134 potential items compared to about 340 in the present system.
This does not include the data items left empty for agencies to
fill in with information that is “other.” Module 1
requires lengthy state narratives including the results of local
improvement efforts, which will require local narrative reports.
b. In Module 3, the report requires entry of detailed
community level project management reporting for in-progress
initiatives. In Module 1, the report requires extensive state
evaluation reports [up to 2,500 words] on local organizational
standard implementation. We support the opinion of the Community
Action Partnership that the use of area demographic data and
percentages of such data are unsuited to CAA project goals.
National Community Action Foundation ROMA NG Final
Comment
3. Much of the proposed new content will not
produce data that can be the basis for any evaluative analysis at
the local level, much less when aggregated at the state or federal
level with other agency data. In the second section, we identify
those data with little value and the details of our concerns of
their requirement. Unfortunately, the sections that are only
slightly changed will continue to be populated with the
often-duplicative “snapshot” data that have been the
basis for the first generation CSBG
• Later, we
also provide detailed justifications for eliminating the “new
participant” report.
4. The proposed rollout
schedule is unrealistic. We agree that the incrementally updated
characteristics and expenditures reports can take the place of the
current forms by FY 2018. If the promised development of the state
plan submissions and tracking system, the “OLDC,” is
complete by summer of 2017, then we can imagine that a pared-down
version of Module 1, the state administrative reports on plan
implementation, could be in place by FY 2018. However, OCS has yet
to prepare definitions and guidance for many of the new terms and
newly required report elements. OCS should define new terms prior
to final network review.
5. We support the
recommendation by the Community Action Partnership that the new
proposal should restore and reflect the CSBG goal to increase the
capacity of CAAs to fight poverty.
National Community
Action Foundation ROMA NG Final Comment
Our
Alternative Performance Management System
A new
generation of ROMA can, and should, proceed with significant
changes from the current draft. Our concerns about cost, relevance
or efficacy can be resolved by:
1. Reducing the number
of required components, including the services list, the new
participant report, and the stability measures and tracking
measures, 80% of the Community impacts items, and offering these
proposed elements as a voluntary CSBG toolkit.
2.
Postponing the implementation of the newest elements while testing
updated complex information systems.
3. Investing in
CAA-centric information systems and practices that, in 5 years,
can equip local agencies to establish targets and track
customized, quantitative indicators of success for the work they
have identified as top priorities in their strategic plans.
Simplified state reports would include the progress of local
agencies toward their major strategic goals. Narratives would
present what the goals and targets were in the report year and
provide examples of the indicators of results.
4. Such
reports would derive state and federal reports from the data CAAs
need to measure the success of the year's projects. Narratives
should accompany every priority project. This would replace the
generalized individual, family and community national measurement
with clear indicators of progress, which are appropriate to the
mission of Community Action, and reflect its diversity throughout
the nation. HHS is certainly familiar with the problems of
interoperability in data systems because of its recent experience
with developing health databases and integrating them with some
human services information at the state and federal level. It
would be very helpful if HHS were to devote some of its
Medicaid-funded technical data integration resources to designing
data warehousing for all programs that coordinate multiple
resources toward a single purpose.
Instead, ROMA NG
transfers the burden of creating such a data warehouse to CAAs,
though such a task has yet to be mastered by the federal
government. In fact, federal agencies have not yet created the
interoperability that allows data exchange between programs run by
the same agency. For example, many Head Start programs refuse to
share client data files with the CSBG data managers in the same
agency, citing federal policy. Suffice it to say, there are also
no interoperability nor seamless data exchanges between separate
domestic agencies at the federal or state level.
High-impact,
large CAAs have the greatest data complexity. The smaller the
number of funding sources and participant population, the easier
it is for an agency to integrate program and client files that
have been collected and formatted for those funders. Conversely,
the greater the complexity of funding source, partnerships and
general capacity of an agency, the more difficult it is to design
a data system that integrates and organizes the information from
multiple coordinated investments in participants or communities.
If more CSBG funding must be diverted from crucial community
programming, in order to design an effective, integrated reporting
system, the major investments should begin by testing such a local
system or data warehouse at the largest and most programmatically
complex agencies. The results of this demonstration should inform
the subsequent design of
National Community Action
Foundation ROMA NG Final Comment the management and outcome
measurement systems. Those, in turn, should be the basis for the
reporting system and software adopted for collecting meaningful,
valid state reports.
Thank you for taking the time for
deep consideration of our major concerns. Please do not hesitate
to contact me with any further questions.
|
|
Comment 84 (National Community
Partnership ; Other/Unknown in ND)
|
|
Question 1. Whether the
proposed collection of information is necessary for the proper
performance of the functions of the agency, including whether the
information shall have practical utility.
While there
are elements that may be helpful, there are a number of areas
detailed below where the collection of information does not meet
the threshold of necessity or utility. The package as proposed in
its entirety is overly burdensome, contains reporting tools that
will provide useless information, and is outside the reach of the
Community Services Block Grant (CSBG). However, with
modifications, a system for collection of information can be
developed that improves the performance management of the agency
and has practical utility. Detail is provided here outlining where
changes need to be made to reach a level of workability and
utility.
1. Module 1: Modify the options CSBG State
Lead Agencies/grantees have when reporting to OCS on CSBG
Organizational Standards to allow for a broader range of
responses. As noted below, there are significant challenges with
the way this information is being reported and creates a misuse of
the Organizational Standards. Expecting constant perfection is
unrealistic and does not have practical utility for local agencies
or the State CSBG Lead Agency. As proposed, the single metric the
State Agencies will report on will be the number of eligible
entities that meet 100% of the CSBG Organizational Standards. To
increase utility of the information, State Agencies should have
the option to report the number of agencies at a variety of
thresholds that still delineate strong performance or allow for a
more accurate reflection of the training and technical assistance
needs. For instance State Agencies should be able to report the
number of local agencies that have met 100%, 90%, 80%, 70%, 60%,
50%, and less than 50% of the standards. This will allow for the
collection of more accurate information; better tracking of
results over time; and better, more efficient investment of
training and technical assistance.
2. Module 4: Remove
the Characteristics for NEW Individuals and Households Report.
This report has no practical utility at the local level given the
time and expense to create it at each local agency. This is not
necessary for agency performance, and the intentions outlined by
OCS for its use are not reflective of how local agencies and
states look at their communities to assess needs, develop
programs, and report progress. It also lacks utility at the state
level given that data such as this rolled up to the state level
will be skewed by large population centers and decrease the
practical utility of the data even further. In addition, many
agencies may either lack management information systems with the
capacity to disaggregate this type of data from its overall
customer database, or struggle to integrate the required data
given their use of multiple (and sometimes incompatible) reporting
systems. In addition, as many local agencies serve families over
time, with no definition of “new” being provided for
review, the CSBG Network will be unable to implement this report
consistently. Defining an individual or family as new becomes
challenging when they may receive different services over many
years (e.g. Early Head Start, child care, after school programs,
youth programs, job training, WIC, food boxes, LIHEAP, etc.).
3.
Module 3: Remove the Community Outcome Indicators that include
percent or rates. Their inclusion is inappropriate and will
provide meaningless data when aggregated beyond individual
initiatives. Such data has validity for geographic areas much
larger in than those addressed or targeted by local agency
projects. This validity is lost when the scale is smaller and
accepted data comparison points become unavailable. Having
individual agencies report on rate change at the community level
on social indicators does not provide information that is usable.
It is more appropriate to give agencies the option of reporting
outcome data and the flexibility to define which indicators they
use.
4. Module 3: Remove Collective Impact from all
reports. Collective Impact is simply one set of strategies for
doing community-level work and while CSBG eligible entities are
engaged in such efforts, privileging one approach given the
broader range of options for doing community-level work is
inappropriate and unnecessary. This should be removed and included
in training and technical assistance endeavors.
5.
Module 4: Remove the Stability Indicators. The indicators selected
and included in the clearance package are not helpful or practical
when reporting customers stabilized as a result of the eligible
entity’s work. One option duplicates data already collected
and the second uses a tool that defines financial stability based
on a construct the uses assumptions drawn from the general
population, not low-income individuals and families specifically.
Because financial well-being means something very different for
the low-income population in comparison to the general population,
use of the scale will produce data of limited utility.
6.
Address the unique issues of Public entities that receive CSBG.
Public agencies (sub-grantees) that sub-contract most or all of
their CSBG funding to other service providers may face added
challenges of requiring their grantees to collect data in
accordance with ROMA Next Generation requirements since it may
limit agencies to contracting with providers that have the
requisite data collection capacity or require additional technical
assistance to meet the reporting requirements. This challenge is
not addressed in the proposed reporting package.
7.
Include a 4th ROMA Goal: Agencies Increase their capacity to
achieve results. Reinstate this national goal that addressed the
critical role the network of local agencies serving 99% of US
counties plays in addressing poverty. Reasons for exclusion have
been inadequate to date. Because agency capacity is critical to
the ultimate success of programs and services, it is important to
clearly state this objective in the overall National Goals. This
also highlights the broader need of the Network for ongoing
training, technical assistance, and general capacity-building
supports.
Question 2: The accuracy of the agency’s
estimate of the burden of the proposed collection of
information.
The Partnership urges local agencies and
states to estimate their time and cost burden. Based on feedback
from the Network, we suspect that the current estimation is low
given the increase in the amount of information required and the
challenges many agencies face collecting data from multiple
reporting systems. Agencies that respond to the estimation of
burden question should choose one of the following options and
include an explanation detailing how they arrived at the number
and the impact of this collection on the agency or the state
office. (Please forward your estimate to NCAF and the Partnership
to assist in the estimated burden Network-wide.)
·
Concerns include:
1. The current IS survey includes
300+ data elements to report. The proposed CSBG Annual Report
includes more than 1,000. This increased burden of reporting comes
with no new resources.
2. The burden on local agencies
is not impacted by the implementation of the Online Data
Collection system (OLDC). To comply with the draft CSBG Annual
Report local agencies will need to modify existing systems.
3.
Many agencies lack management information systems with the
capacity to provide the required data. Data fields required for
ROMA Next Generating reporting may not be mirrored in those used
by agencies. In addition, many agencies use multiple (and
sometimes incompatible) reporting systems which adds to the time
necessary to provide the data.
In addition, it is not
clear how the burden for local agencies is decreased by the online
data collection system described in the by OCS. OCS’ Dear
Colleague Letter of June 17, 2016, states, “Aware of the
reporting burden the proposed Annual Report will entail, OCS will
decrease that burden by establishing an on-line automated system
for use by the states and – at the discretion of the states
– the local agencies, for reporting that allows in many
instances auto-populating data from one year to the next and a
variety of data entry processes (e.g. manual entry and automatic
upload).” It is important to note, this online automated
system may benefit State Lead Agencies (grantees), but will most
likely not benefit local Community Action Agencies (sub-grantees).
Sub-grantees will still need systems that track customers,
services, outcomes, and given the requirement of many funding
sources (including other federal Department of Health and Human
Services funding) requires additional and sometime separate data
collection systems. State grantees do not have the funding or
capacity for the most part, to create systems or support local
agencies to develop systems that will streamline or decrease the
burden outlined here.
Question 3: The quality,
utility, and clarity of the information to be collected.
As
noted below in the detailed comments by module, several elements
of data to be collected under this proposal have no meaning at the
state and federal levels, and only have value, meaning, and
usefulness at the local level.
· Several of the
elements to be collected as described are not available in smaller
geographic areas usually targeted by local agencies in their
community work and if created by local agencies, lose validity.
The social or population level indicators being requested are not
meant for single agency reporting and are best served for
community-wide scorecards or utilized as part of a community needs
assessment. For example, comparing overall percentages of
community indicators (e.g. the unemployment rate, violent crime
rate) to individual local agencies outcomes is misleading
·
Module 4, Section A: Characteristics for NEW Individuals and
Households - Data Entry Form is an unnecessary document that will
not produce quality data; it will have little utility at a local
level; and it lends itself to meaningless data at a state and
national level. It should be removed. Because demographic data
alone does not include information about the needs of new
customers, this information fails to help agencies determine if
they are addressing issues identified in their community
assessments. Furthermore, the large number of “one time”
customers in programs such as LIHEAP will skew any meaningful
interpretation of the data. Aside from the challenge of defining
who constitutes a new customer, agencies that use multiple
reporting systems will also have the added burden of tracking new
customers across all programs and services.
·
Module 3, Section A: Many of the indicators used here are
inappropriate for reporting by local sub-grantees to States and by
States to OCS and will produce data of limited utility. Several
challenges underscore this point. First, community-level data on
indicators is not always available at the level (e.g. regional or
zip code/census tract) or population (e.g. low-income children
aged 0-5) that matches the area and population targeted by the
agency. Second, community-level indicators are influenced by a
wide number of sources that would most likely overwhelm the impact
of even successful community initiatives, a fact that
significantly limits their utility for providing meaningful
information about outcomes. Third, any meaningful data about the
outcomes of community-level initiatives would require a rigorous
program evaluation and could not be determined on the data
collected through the Annual Report alone.
Question 4:
Ways to minimize the burden of the collection of information on
respondents, including through the use of automated collection
techniques or other forms of information technology.
·
As noted above, OCS’ Dear Colleague Letter of June 17, 2016,
states, “Aware of the reporting burden the proposed Annual
Report will entail, OCS will decrease that burden by establishing
an on-line automated system for use by the states and – at
the discretion of the states – the local agencies, for
reporting that allows in many instances auto-populating data from
one year to the next and a variety of data entry processes (e.g.
manual entry and automatic upload).” It is important to
note, this online automated system may benefit State Lead Agencies
(grantees), but will most likely not benefit local Community
Action Agencies (sub-grantees). Sub-grantees (local agencies) will
still need systems that track customers, services, outcomes, and
given the requirement of many funding sources (including other
federal Department of Health and Human Services funding) requires
additional and sometime separate data collection systems. State
grantees do not have the funding or capacity for the most part, to
create systems that will streamline or decrease the burden
outlined here. Furthermore, new state-level systems will do little
to address the needs of local agencies for improved management
information systems with the capacity to collect and report the
data required by ROMA Next Generation.
· OCS
and the federal government provide funding to local sub-grantees
that require separate systems and prohibit the aggregation of
data. To minimize the burden on respondents, DHHS should examine
its own practices to ensure that all DHHS-funded programs in the
field can use data collection systems that can be easily
integrated and “talk” to one another to produce
reports that meet the needs of local agencies, states, and OCS.
|
|
Comment 85 (Nek-Cap, Inc. (Northeast
Kansas Community Action Agency); Eligible Entity/Local Agency in
KS)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility:
I support the collection
of data outlined in each Module and believe it is necessary in
informing Congress and all funders the results, outcomes and
impacts of Community Action services and strategies. The baseline
data elements are already collected and will present no additional
burden. Specifically tracking and determining which specific
services actually produce the final expected outcomes are more
difficult at both the family and community level. To determine
long term impact based on interaction with a CSBG funded service
is currently not possible through existing software and would be
very difficult and costly to measure without significant consumer
self -reporting and community alignment with agency initiatives.
2. The accuracy of the agency’s estimate of the
burden of the proposed collection of information:
The
estimated cost to NEK-CAP, Inc., for the full implementation of
ROMA Next Generation and proposed CSBG Annual Report is
$264,220.00. This amount reflects approximately one-half of all
CSBG funds currently received by the agency and would , therefore,
need to supplemented with at a minimum an additional $76,000.00 In
order to properly collect, analyze and evaluate data as required
per the ROMA cycle. The estimate concluded that at a minimum
approximately 8,880 hours of staff time would need to be devoted
to the overall process. Furthermore, the agency would need to hire
additional staff, restructure the job duties and services provided
by currently CSBG funded staff and acquire new software modules
either through CAP 60 or another software vendor with the
capability of integrating all agency reported data. These
estimates are based on current and projected costs associated with
agency CAP 60 user’s time and wages, staffing patterns,
training needs, software costs, and the potential for contracted
services.
3. The quality, utility, and clarity of the
information to be collected:
The quality of the data
is only as good as the staff input and the accuracy and reporting
capability of the data software system. CAP 60 does not produce
quality reports and there are significant questions about validity
and reliability. The information to be collected is quite clear
but the utility of collecting such data has the potential to fail
to demonstrate the overall impact of the utilization of CSBG
funded strategies due to the inability to track long term outcomes
and the systemic challenges, endured by low-income populations and
communities, which are beyond the control of Community Action
Programs.
4. Ways to minimize the burden of the
collection of information on respondents, including through the
use of automated collection techniques or other forms of
information technology.
A software system which can
integrate/extrapolate data from multiple systems would be ideal.
Duplicate entry into various systems is a waste of resources and
can affect the unduplicated count. A state supported system such
as HMIS could potentially be adapted to meet the reporting
requirements; however, there would more than likely be a
significant cost with the redesign of such a system. Develop an
mobile application which interfaces with agency tracking software
that can be utilized by consumers for reporting purposes
|
|
Comment 86 (New England Community
Action Partnership (Necap) ; State Association in CT)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility.
There are numerous
areas where the collection of information does not meet the
threshold of necessity or utility. The package as proposed is
burdensome, contains reporting tools that will provide useless
information, and is outside the reach of the Community Services
Block Grant (CSBG). However, with modifications, a system for
collection of information can be developed that improves the
performance management of the agency and has practical utility.
Detail is provided below that outlines where changes need to be
made to reach a level of workability and utility.
·
Module 1: Modify the options CSBG State Lead Agencies/grantees
have when reporting to OCS on CSBG Organizational Standards to
allow for a broader range of responses. As noted below, there are
significant challenges with the way this information is being
reported and creates a misuse of the Organizational Standards.
Expecting constant perfection is unrealistic and does not have
practical utility for local agencies or the State CSBG Lead
Agency. As proposed, the single metric the State Agencies will
report on will be the number of eligible entities that meet 100%
of the CSBG Organizational Standards. To increase utility of the
information, State Agencies should have the option to report the
number of agencies at a variety of thresholds that still delineate
strong performance or allow for a more accurate reflection of the
training and technical assistance needs. For instance State
Agencies should be able to report the number of local agencies
that have met 100%, 90%, 80%, 70%, 60%, 50%, and less than 50% of
the standards. This will allow for the collection of more accurate
information; better tracking of results over time; and better,
more efficient investment of training and technical assistance.
· Module 4: Remove the Characteristics for NEW
Individuals and Households Report. This report has no practical
utility at the local level given the time and expense to create it
at each local agency. This is not necessary for agency
performance, and the intentions outlined by OCS for its use are
not reflective of how local agencies and states look at their
communities to assess needs, develop programs, and report
progress. It also lacks utility at the state level given that data
such as this rolled up to the state level will be skewed by large
population centers and decrease the practical utility of the data
even further. In addition, many agencies may either lack
management information systems with the capacity to disaggregate
this type of data from its overall customer database, or struggle
to integrate the required data given their use of multiple (and
sometimes incompatible) reporting systems. In addition, as many
local agencies serve families over time, with no definition of
“new” being provided for review, the CSBG Network will
be unable to implement this report consistently. Defining an
individual or family as new becomes challenging when they may
receive different services over many years (e.g. Early Head Start,
child care, after school programs, youth programs, job training,
WIC, food boxes, LIHEAP, etc.).
· Module 3:
Remove the Community Outcome Indicators that include percent or
rates. Their inclusion is inappropriate and will provide
meaningless data when aggregated beyond individual initiatives.
Such data has validity for geographic areas much larger in than
those addressed or Community Action Partnership ROMA Next
Generation Talking Points Memo 4 targeted by local agency
projects. This validity is lost when the scale is smaller and
accepted data comparison points become unavailable. Having
individual agencies report on rate change at the community level
on social indicators does not provide information that is usable.
It is more appropriate to give agencies the option of reporting
outcome data and the flexibility to define which indicators they
use.
· Module 3: Remove Collective Impact from
all reports. Collective Impact is simply one set of strategies for
doing community-level work and while CSBG eligible entities are
engaged in such efforts, privileging one approach given the
broader range of options for doing community-level work is
inappropriate and unnecessary. This should be removed and included
in training and technical assistance endeavors.
·
Module 4: Remove the Stability Indicators. The indicators selected
and included in the clearance package are not helpful or practical
when reporting customers stabilized as a result of the eligible
entity’s work. One option duplicates data already collected
and the second uses a tool that defines financial stability based
on a construct the uses assumptions drawn from the general
population, not low-income individuals and families specifically.
Because financial wellbeing means something very different for the
low-income population in comparison to the general population, use
of the scale will produce data of limited utility.
·
Include a 4th ROMA Goal: Agencies Increase their capacity to
achieve results. Reinstate this national goal that addressed the
critical role the network of local agencies serving 99% of US
counties plays in addressing poverty. Reasons for exclusion
have been inadequate to date. Because agency capacity is critical
to the ultimate success of programs and services, it is important
to clearly state this objective in the overall National Goals.
This also highlights the broader need of the Network for ongoing
training, technical assistance, and general capacity-building
supports
2. The accuracy of the agency’s
estimate of the burden of the proposed collection of information.
Many agencies have expressed concerns about the cost
and time of modifying and managing current systems comply with the
draft CSBG Annual Report. NECAP echoes these concerns. Estimated
costs will be reflected in agency-specific comments.
3.
The quality, utility, and clarity of the information to be
collected.
Several elements of data to be collected
under this proposal have no meaning at the state and federal
levels, and only have value, meaning, and usefulness at the local
level. It also mandates the use of limited outcome indicators in
order to standardize and collapse the rich variety of Community
Action Agency achievements and projects into a national summary.
· Several of the elements to be collected as
described are not available in smaller geographic areas usually
targeted by local agencies in their community work and if created
by local agencies, lose validity. The social or population level
indicators being requested are not meant for single agency
reporting and are best served for community-wide scorecards or
utilized as part of a community needs assessment. For example,
comparing overall percentages of community indicators (e.g. the
unemployment rate, violent crime rate) to individual local
agencies outcomes is misleading
· Module 4,
Section A: Characteristics for NEW Individuals and Households -
Data Entry Form is an unnecessary document that will not produce
quality data; it will have little utility at a local level; and it
lends itself to meaningless data at a state and national level. It
should be removed. Because demographic data alone does not include
information about the needs of new customers, this information
fails to help agencies determine if they are addressing issues
identified in their community assessments. Furthermore, the large
number of “one time” customers in programs such as
LIHEAP will skew any meaningful interpretation of the data. Aside
from the challenge of defining who constitutes a new customer,
agencies that use multiple reporting systems will also have the
added burden of tracking new customers across all programs and
services.
· Module 3, Section A: Many of the
indicators used here are inappropriate for reporting by local
sub-grantees to States and by States to OCS and will produce data
of limited utility. Several challenges underscore this point.
First, community-level data on indicators is not always available
at the level (e.g. regional or zip code/census tract) or
population (e.g. low-income children aged 0-5) that matches the
area and population targeted by the agency. Second,
community-level indicators are influenced by a wide number of
sources that would most likely overwhelm the impact of even
successful community initiatives, a fact that significantly limits
their utility for providing meaningful information about outcomes.
Third, any meaningful data about the outcomes of community-level
initiatives would require a rigorous program evaluation and could
not be determined on the data collected through the Annual Report
alone.
4. Ways to minimize the burden of the
collection of information on respondents, including through the
use of automated collection techniques or other forms of
information technology.
· The draft report
nearly triples the amount of now-required information. There are
more than 1,100 items for local reports (and even more when
counting "other" items) versus the over 350 in the
current system. This means CAAs will be forced to contribute on
many of the new state data items. Agencies will have to contribute
because States must now investigate and report narrative details
on local agency achievement of organizational standards. While
OCS’s Dear Colleague Letter of June 17, 2016, states, “Aware
of the reporting burden the proposed Annual Report will entail,
OCS will decrease that burden by establishing an on-line automated
system for use by the states and – at the discretion of the
states – the local agencies, for reporting that allows in
many instances auto-populating data from one year to the next and
a variety of data entry processes (e.g. manual entry and automatic
upload),” it is important to note these concerns. Also, this
online automated system may benefit State Lead Agencies
(grantees), but will most likely not benefit local Community
Action Agencies (sub-grantees). Sub-grantees (local agencies) will
still need systems that track customers, services, outcomes, and
given the requirement of many funding sources (including other
federal Department of Health and Human Services funding) requires
additional and sometime separate data collection systems. State
grantees do not have the funding or capacity for the most part, to
create systems that will streamline or decrease the burden
outlined here. Furthermore, new state-level systems will do little
to address the needs of local agencies for improved management
information systems with the capacity to collect and report the
data required by ROMA Next Generation.
· OCS
and the federal government provide funding to local sub-grantees
that require separate systems and prohibit the aggregation of
data. To minimize the burden on respondents, DHHS should examine
its own practices to ensure that all DHHS-funded programs in the
field can use data collection systems that can be easily
integrated and “talk” to one another to produce
reports that meet the needs of local agencies, states, and OCS.
|
|
Comment 88 (New York State Community
Action Association; Eligible Entity/Local Agency in NY)
|
|
Question 1. Whether the
proposed collection of information is necessary for the proper
performance of the functions of the agency, including whether the
information shall have practical utility.
The proposed
collection of information includes some useful elements but in
whole is quite burdensome and includes information that is of
limited use and well beyond the parameters of the Community
Services Block Grant (CSBG.) Suggested modifications which could
result in a useful and effective system of information collection
are noted below.
1. Module 1: Allow CSBG State Lead
Agencies/grantees to report on the eligible entities’
compliance with standards at a range of levels beyond 100%, for
example, 50%, 75% 90% and 100%. The Organizational Standards were
intended to promote excellence not merely the lowest level of
compliance. It is therefore reasonable and important that the
metrics include those agencies that meet less than 100% to provide
a more accurate assessment of the training and technical
assistance needs of eligible entities and to track CAA’s
progress in meeting the Standards over time. Requiring that only
100% compliance be reported undermines the intention of the
standards to promote excellence and is of limited to utility for
the CAAs, State lead agencies and NASCSP and OCS.
2.
Module 3: Make the reporting of community level data on social
indicators optional and provide flexibility in defining which
indicators to use and report. The inclusion of Community Outcome
data with percentages or rates provides little meaningful data on
the initiatives undertaken by local agencies. Their inclusion is
inappropriate and will provide meaningless data when aggregated
beyond individual initiatives.
3. Module 3: Collective
Impact is only one of a variety of options for community-level
work and should therefore be removed. As one in a menu of options,
collective Impact should be removed as including it suggests a
strong preference for this approach as opposed to other possible
community-level strategies.
4. Module 4: Remove the
Characteristics for NEW Individuals and Households Report. There
is no definition of “New” included here nor is there a
widely accepted standard for identifying a “new”
customer. For example when is a customer who receives multiple
services over many years “New”? And what utility does
collecting this data have for state agencies or OCS when it is
rolled up and aggregated? The purpose identified by OCS for the
collection of this data is to inform the needs assessment and
program development process. Agencies already have a process for
identifying needs and developing services with little to be gained
by expending the time and resources needed to collect this data.
Furthermore, many agencies lack the technical capacity and
resources to collect and analyze this data and many are challenged
by the barriers created by the mandated use of multiple, often
incompatible, reporting systems.
5. Module 4: The
Stability Indicators included here are neither practical nor
useful and should be removed. One option included here uses a tool
for defining financial stability that is based on assumptions
drawn from the general population and is therefore of very limited
utility when applied to the low income people served by CAAs.
Another option duplicates data already collected.
6.
Reinstate the ROMA Goal: Agencies Increase their capacity to
achieve results. This goal reflects the vital role of the national
network of local CAAs and there is little sound reason to remove
it. Agency capacity is critical to the success of the services
provided and should be identified as such in the National Goals.
Inclusion of Agency Capacity is also important to highlight the
ongoing need for training, technical assistance and other capacity
building services for the CAA network.
Question 2: The
accuracy of the agency’s estimate of the burden of the
proposed collection of information.
While this is a
question that can only be answered by individual CAAs, this
Association is concerned about the burden of this enormous
increase in the collection of information with no additional
resources provided to meet this requirement. The increase in the
data elements included in the proposed CSBG Annual Report from 300
to more than 1,000 poses potentially great burdens with no
additional resources. This is particularly problematic given the
challenges already faced by agencies that are required to collect
and report data using multiple, often incompatible data
systems.
It should also be noted that the on-line
system described by OCS in the Dear Colleague Letter of June 17,
2016, may be of benefit to State Lead Agencies but does little to
address the issues that eligible entities face in tracking and
reporting data for multiple agencies using multiple information
and reporting systems.
Question 3: The quality,
utility, and clarity of the information to be collected.
A
number of the elements to be collected are primarily useful as
components of community-wide scorecards and have very limited
utility or are not available for single agency reporting or as
data to describe the work done in the smaller geographic areas
served by local agencies, particularly regarding community-level
work. The use of such data to assess and inform individual agency
outcomes is at best limited and may in fact be misleading if used
to compare agency outcomes with community indicators.
·
Module 3, Section A: The indicators used here are to report
community-level data to States and by States to OCS and will be of
limited utility. The available community-level data often do not
match the geographic area and population targeted by the agency,
for example by census track or age. It is also extremely difficult
to assess the impact of an agency’s intervention as part of
a community-level initiative since many external factors influence
the impact of such an undertaking and could erase or minimize the
impact of community-level initiatives. Evaluating the outcomes of
community-level initiatives requires a formal program evaluation
and cannot be accomplished solely through the use of data
collected in the Annual Report.
· Module 4,
Section A: Characteristics for NEW Individuals and Households -
Data Entry Form should be removed. As noted above the collection
of this information is both difficult and costly and of little use
at the local level. This data does not include information about
the needs and challenges facing “new” customers and
does little to help agencies assess whether they are meeting their
needs. Additionally, this data fails to account for the many “one
time” customers who access services such as LIHEAP and thus
makes interpretation of the data difficult and inaccurate.
Further, as previously noted, because many agencies are required
to use multiple, incompatible data collection and reporting
systems, this will pose a significant burden on agencies with no
resources available to support this work.
Question 4:
Ways to minimize the burden of the collection of information on
respondents, including through the use of automated collection
techniques or other forms of information technology.
As
previously noted, the on-line system referenced by OCS in the June
17, 2016 Dear Colleague Letter will do little to help agencies
track customers, services and outcomes. The online automated
system may be an asset to State Lead Agencies, but offers little
for local Community Action Agencies which will still be required
to track and report data using multiple information management
systems with different, often incompatible requirements. It is
unlikely that the State Lead Agencies will have sufficient
resources to enhance the network’s data collection capacity
nor will the proposed on-line system address the need for improved
information management capacity as required by ROMA Next Gen. To
minimize the burden and maximize the utility of the data
collected, the federal government should assess its data
collection requirements and find a way to use information
management systems that can be integrated to meet the needs of the
multiple government funding and regulatory agencies.
|
|
Comment 90 (North Central Community
Action Program; Eligible Entity/Local Agency in WI)
|
|
Thank you for your work on
ROMA Next Generation. I appreciate the overwhelming task this is
to develop a system that captures the great work that is performed
across the country by community action agencies as well as
demonstrate quality service and data collection.
First of all, I am
concerned about the burden of reporting and tracking of all of the
data. We are a small community action agency. Our organizational
structure will make my job much more complicated because I do much
of the reporting. The requirements in the proposed ROMA NG will be
extremely challenging for me. I did like some of the changes that
came with the first revisions to ROMA NG but I still believe that
you are asking for too much information, some of which is not very
practical or useful. We do not have the financial resources to put
into place all of the required data collection components, let
along all of the time needed, to address everything in ROMA NG.
A few areas that stand out as unrealistic include:
-Collecting over 1,000 data elements in the proposed
IS Survey. The current survey is challenging and only has a little
over 300.
-The data requested will require us to
implement new data collection methods that are cost prohibitive.
-The time necessary to respond will go way beyond the
estimated hour’s projections.
|
|
Comment 91 (North Dakota Community
Action ; State Association in ND)
|
|
While there are elements that
may be helpful, there are a number of areas detailed below where
the collection of information does not meet the threshold of
necessity or utility. The package as proposed in its entirety is
overly burdensome, contains reporting tools that will provide
useless information, and is outside the reach of the Community
Services Block Grant (CSBG). However, with modifications, a system
for collection of information can be developed that improves the
performance management of the agency and has practical utility.
Detail is provided here outlining where changes need to be made to
reach a level of workability and utility.
I. Module 1:
Modify the options CSBG State Lead Agencies/grantees have when
reporting to OCS on CSBG Organizational Standards to allow for a
broader range of responses. As noted below, there are significant
challenges with the way this information is being reported and
creates a misuse of the Organizational Standards. Expecting
constant perfection is unrealistic and does not have practical
utility for local agencies or the State CSBG Lead Agency. As
proposed, the single metric the State Agencies will report on will
be the number of eligible
entities that meet 100% of the CSBG
Organizational Standards. To increase utility of the information,
State Agencies should have the option to report the number of
agencies at a variety of thresholds that still delineate strong
performance or allow for a more accurate reflection of the
training and technical assistance needs.
For instance
State Agencies should be able to report the number of local
agencies that have met 100%, 90%, 80%, 70%, 60%, 50%, and less
than 50% of the standards. This will allow for the
collection
of more accurate information; better tracking of results over
time; and better, more efficient investment of training and
technical assistance.
Module 4: Remove the
Characteristics for NEW Individuals and Households Report. This
report has no practical utility at the local level given the time
and expense to create it at each local
agency. This is not
necessary for agency performance, and the intentions outlined by
OCS for its use are not reflective of how local agencies and
states look at their communities to assess needs, develop
programs, and report progress. It also lacks utility at the state
level given that data such as this rolled up to the state level
will be skewed by large population centers and decrease the
practical utility of the data even further. In addition, many
agencies may either lack management information systems with the
capacity to disaggregate this type of data from its
overall
customer database, or struggle to integrate the required data
given their use of multiple (and sometimes incompatible) reporting
systems. In addition, as many local agencies serve families over
time, with no definition of "new" being provided for
review, the CSBG Network will be unable to implement this report
consistently. Defining an individual or family as new becomes
challenging when they may receive different services over many
years (e.g. Early Head Start, child care, after school programs,
youth programs, job training, WIC, food boxes, LIHEAP, etc.).
Module 3: Remove the Community Outcome Indicators that include
percent or rates. Their inclusion is inappropriate and will
provide meaningless data when aggregated beyond individual
initiatives. Such data has validity for geographic areas much
larger in than those addressed or
targeted by local agency
projects. This validity is lost when the scale is smaller and
accepted data comparison points become unavailable. Having
individual agencies report on rate change at the community level
on social indicators does not provide information that is usable.
It is more appropriate to give agencies the option of reporting
outcome data and the flexibility to define which indicators they
use.
Module 3: Remove Collective Impact from all
reports. Collective Impact is simply one set of strategies for
doing community-level work and while CSBG eligible entities are
engaged in such efforts, privileging one approach given the
broader range of options for doing community-level work is
inappropriate and unnecessary. This should be removed and included
in training and technical assistance endeavors.
Module
4: Remove the Stability Indicators. The indicators selected and
included in the clearance package are not helpful or practical
when reporting customers stabilized as a result of the
eligible
entity's work. One option duplicates data already collected and
the second uses a tool that defines financial stability based on a
construct the uses assumptions drawn from the general population,
not low-income individuals and families specifically. Because
financial well-being means something very different for the
low-income population in comparison to the
general
population, use of the scale will produce data of limited
utility.
Address the unique issues of Public entities that
receive CSBG. Public agencies (sub-grantees) that sub-contract
most or all of their CSBG funding to other service providers may
face added challenges of requiring their grantees to collect data
in accordance with ROMA Next Generation requirements since it may
limit agencies to contracting with providers that have the
requisite data collection capacity or require additional technical
assistance to meet the reporting requirements. This challenge is
not addressed in the proposed reporting package.
Include
a 4th ROMA Goal: Agencies Increase their capacity to achieve
results. Reinstate this national goal that addressed the critical
role the network of local agencies serving 99% of US
counties
plays in addressing poverty. Reasons for exclusion have been
inadequate to date. Because agency capacity is critical to the
ultimate success of programs and services, it is important to
clearly state this objective in the overall National Goals. This
also highlights the broader need of the Network for ongoing
training, technical assistance, and general capacity-building
supports.
2. The accuracy of the agency's estimate of
the burden of the proposed collection of information Regarding the
accuracy of the agencVs estimate of the cost burden, CAPND
concerns include:
The current IS survey includes 300+
data elements to report. The proposed CSBG Annual Report includes
more than 1,000. This increased burden of reporting comes with no
new resources.
The burden on local agencies is not
impacted by the implementation of the Online Data Collection
system (OLDC). To comply with the draft CSBG Annual Report local
agencies will need to modify existing systems. An initial
investment in new software for North Dakota was quoted at over
$50,000 which only covers the set-up and one year of training.
This does include any of the costs association with adding
necessary modules such as Weatherization or Head Start.
Many
agencies lack management information systems with the capacity to
provide the required data. Data fields required for ROMA Next
Generating reporting may not be mirrored in those used by
agencies. In addition, many agencies use multiple (and sometimes
incompatible) reporting systems which adds to the time necessary
to provide the data.
In addition, it is not clear how
the burden for local agencies is decreased by the online data
collection system described in the by OCS. OCS' Dear Colleague
Letter of June 17, 2016, states,
"Aware of the reporting
burden the proposed Annual Report will entail, OCS will decrease
that burden by establishing an on-line automated system for use by
the states and — at the discretion
of the states —
the local agencies, for reporting that allows in many instances
auto-populating data from one year to the next and a variety of
data entry processes (e.g. manual entry and automatic upload)."
It is important to note, this online automated system may benefit
State Lead Agencies (grantees), but will most likely not benefit
local Community Action Agencies (sub-grantees). Sub-grantees will
still need systems that track customers, services, outcomes, and
given the requirement of many funding sources (including other
federal Department of Health and Human Services funding) requires
additional and sometime separate data collection systems. State
grantees do not have the funding or capacity for the most part, to
create systems or support local agencies to develop systems that
will streamline or decrease the burden outlined here.
3.The
quality, utility, and clarity of the information to be
collected
As noted below in the detailed comments by
module, several elements of data to be collected under this
proposal have no meaning at the state and federal levels, and only
have value, meaning, and usefulness at the local level.
•
Several of the elements to be collected as described are not
available in smaller geographic areas usually targeted by local
agencies in their community work and if created by local agencies,
lose validity. The social or population level indicators being
requested are not meant for single agency reporting and are best
served for community-wide scorecards or utilized as
part of a
community needs assessment. For example, comparing overall
percentages of community indicators (e.g. the unemployment rate,
violent crime rate) to individual local agencies outcomes is
misleading
Module 4, Section A: Characteristics for NEW
Individuals and Households - Data Entry Form is an unnecessary
document that will not produce quality data; it will have little
utility at a local level; and it lends itself to meaningless data
at a state and national level. It should be removed. Because
demographic data alone does not include information about the
needs of new customers, this information fails to help agencies
determine if they are addressing issues identified in their
community assessments. Furthermore, the large number of "one
time"
customers in programs such as LIHEAP will skew any
meaningful interpretation of the data. Aside from the challenge of
defining who constitutes a new customer, agencies that use
multiple reporting systems will also have the added burden of
tracking new customers across all programs and services.
Module
3, Section A: Many of the indicators used here are inappropriate
for reporting by local sub-grantees to States and by States to OCS
and will produce data of limited utility. Several challenges
underscore this point. First, community-level data on indicators
is not always available at the level (e.g. regional or zip
code/census tract) or population (e.g. low-income children aged
0-5) that matches the area and population targeted by the agency.
Second, community-level indicators are influenced by a wide number
of sources that would most likely overwhelm the impact of even
successful community initiatives, a fact that significantly limits
their utility for providing meaningful information about outcomes.
Third, any meaningful data about the outcomes of community-level
initiatives would require a rigorous program evaluation and could
not be determined on the data collected through the Annual Report
alone.
4. Ways to minimize the burden of the collection of
information on respondents, including through the use of automated
collection techniques or other forms of information technology.
•
As noted above, OCS' Dear Colleague Letter of June 17, 2016,
states, "Aware of the reporting burden the proposed Annual
Report will entail, OCS will decrease that burden by establishing
an on-line automated system for use by the states and — at
the discretion of the states — the local agencies, for
reporting that allows in many instances auto-populating data from
one year to the next and a variety of data entry processes (e.g.
manual entry and automatic upload)."
It Is
important to note, this online automated system may benefit State
Lead Agencies (grantees), but will most likely not benefit local
Community Action Agencies (sub-grantees). Sub-grantees (local
agencies) will still need systems that track customers, services,
outcomes, and given the requirement of many funding sources
(including other federal Department of Health and Human Services
funding) requires additional and sometime separate data collection
systems. State grantees do not have the funding or capacity for
the most part, to create systems that will streamline or decrease
the burden outlined here. Furthermore, new state-level systems
will do little to address the needs of local agencies for improved
management information systems with the capacity to collect and
report the data required by ROMA Next Generation. OCS and the
federal government provide funding to local sub-grantees that
require separate systems and prohibit the aggregation of data. To
minimize the burden on respondents, DHHS should examine its own
practices to ensure that all DHHS-funded programs in the field can
use data collection systems that can be easily integrated and
"talk" to one another to produce reports that meet the
needs of local agencies, states, and OCS.
|
|
Comment 94 (Northeast Kansas
Community Action Agency; Eligible Entity/Local Agency in KS)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility:
I support the collection
of data outlined in each Module and believe it is necessary in
informing Congress and all funders the results, outcomes and
impacts of Community Action services and strategies. The baseline
data elements are already collected and will present no additional
burden. Specifically tracking and determining which specific
services actually produce the final expected outcomes are more
difficult at both the family and community level. To determine
long term impact based on interaction with a CSBG funded service
is currently not possible through existing software and would be
very difficult and costly to measure without significant consumer
self -reporting and community alignment with agency initiatives.
2. The accuracy of the agency’s estimate of the
burden of the proposed collection of information:
The
estimated cost to NEK-CAP, Inc., for the full implementation of
ROMA Next Generation and proposed CSBG Annual Report is
$264,220.00. This amount reflects approximately one-half of all
CSBG funds currently received by the agency and would , therefore,
need to supplemented with at a minimum an additional $76,000.00 In
order to properly collect, analyze and evaluate data as required
per the ROMA cycle. The estimate concluded that at a minimum
approximately 8,880 hours of staff time would need to be devoted
to the overall process. Furthermore, the agency would need to hire
additional staff, restructure the job duties and services provided
by currently CSBG funded staff and acquire new software modules
either through CAP 60 or another software vendor with the
capability of integrating all agency reported data. These
estimates are based on current and projected costs associated with
agency CAP 60 user’s time and wages, staffing patterns,
training needs, software costs, and the potential for contracted
services.
3. The quality, utility, and clarity of the
information to be collected:
The quality of the data
is only as good as the staff input and the accuracy and reporting
capability of the data software system. CAP 60 does not produce
quality reports and there are significant questions about validity
and reliability. The information to be collected is quite clear
but the utility of collecting such data has the potential to fail
to demonstrate the overall impact of the utilization of CSBG
funded strategies due to the inability to track long term outcomes
and the systemic challenges, endured by low-income populations and
communities, which are beyond the control of Community Action
Programs.
4. Ways to minimize the burden of the
collection of information on respondents, including through the
use of automated collection techniques or other forms of
information technology.
A software system which can
integrate/extrapolate data from multiple systems would be ideal.
Duplicate entry into various systems is a waste of resources and
can affect the unduplicated count. A state supported system such
as HMIS could potentially be adapted to meet the reporting
requirements; however, there would more than likely be a
significant cost with the redesign of such a system. Develop a
mobile application which interfaces with agency tracking software
that can be utilized by consumers for reporting purposes.
|
|
Comment 95 (Northeast Pennsylvania
Community Action Agency; Other/Unknown in PA)
|
|
The proposed collection of
information is necessary as it pertains to those programs that are
operated by each entity. The information collected can be used as
a tool for perspective agencies to track areas where there is an
increase or decrease in demand for services. It can also be used
as a way to determine if entities should seek additional funding
opportunities in particular areas. Although it is nearly
impossible to provide an estimate on the true cost per
agency,
the Health and Human Services average burden hours seems to
underestimate this massive undertaking. We have included with
these comments, a starting point budget justification, to compare
to Health and Human Services. This identifies just how drastically
different a Sub-Grantees burden hours would be in relation to the
average figures calculated by HHS. Not taken into consideration
are varied costs and time associated with software development
which includes development of reporting tool'.
The
burden on the agencies data collection will be substantial. The
process will first begin with the computer/client tracking
software that the agency currently uses. It will have to be
determined if this mechanism will meet the new data collection
standards and if the software is capable of handling the increased
quantity of information. The software will not only have to be
updated for the end users but also new reporting documents will
have to be developed. If the current tracking mechanism is found
unsuitable, then the appropriate staff will have to begin
researching and reviewing suitable alternatives. If any changes
are made to the software and/or reporting tools, this will require
retraining both end users and those staff responsible for
analyzing reports. With the increase of reportable areas, this
will increase the amount of time
those responsible for
aggregating this data will spend on this task. Some areas may also
require manual tracking if they cannot be integrated into a new or
existing software system.
The information collected is
only as good as the corresponding questions that are asked. The
questions need to be clear and concise. They should not be phrased
in ways that may lead to misinterpretation or open endedness. If
any agency is not currently using a form of information
technology, it will be near impossible to collect, analyze and
report on all the data to be collected. Whether a current software
system is in place or not, there will be an increase to each
entity that will be charged with undertaking this process.
We
appreciate the opportunity to express our views on the Community
Services Block Grant (CSBG) Annual report. We are hopeful the
information contained in these comments will allow the Office of
Community Services to obtain insight on the daily processes of
those who will be responsible for administering, collecting and
reporting on this vital program.
Community Services
Block Grant (CSBG)
ROMA Next Generation Implementation
Cost Estimate
OPERATING COSTS:
Full Cost of
the different levels of staff contribution time (approx. 30 staff
x 5%Of time)
Contracted Professional services and
training (approx. I staff x 10% of time)
Maintaining
software, integrating mandated data (approx. 2 staff x 5% of time)
Tracking projects (approx. 2 staff x 5% of time)
Reporting progress on organizational standards
(approx. I staff x 5% of time)
Support and train staff
who use software (approx. 1 staff x 10% of time)
ROMA
training ( approx. 1 staff x 5 % of time)
External Data
Acquisition ( approx. 15 staff x 5% of time)
Total
Operating
GRAND TOTAL
Rate
$15.00
$15.00
$15.00
$15.00
$15.00
$15.00
$15.00
$15.00
$15.00
Hours
on Project
2925
195
195 195
97.5
195
97.5
1462.5
53625
$43,875
$2,925
$2,925
$2,925
$1 ,463
$2,925
$1463
$21 ,938
$80,438
$80,438
|
|
Comment 96 (Northern Tier Community
Action Corp. ; Eligible Entity/Local Agency in PA)
|
|
Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility
There are individual parts
of this effort that will be of benefit in reporting, however when
you offset this with the additional burden of data collection and
reporting, it is difficult to justify.
Operational
costs increase continually, with no increase in CSBG funding. CSBG
is the core funding for this agency to leverage and facilitate a
variety of programs that could not stand alone. These critical
programs and services for low-income families are in many cases
the only thing that keeps families from being, unemployed, hungry
and homeless. As it stands now our staff typically manages several
programs at one time and in the process of managing these
programs, serves as case managers helping families to hold it all
together. Now we are to consider adding an even greater burden
with tracking additional elements and reducing the time they have
to do the work of service provision and case management.
The
accuracy of the agency estimate of the burden of the proposed
collection of information
It is understood that the
time estimated for reporting would decrease to some extent as the
process is perfected, however it will still be an unbearable
burden on the minimally funded and staffed agencies. This would
require additional staff just to provide data collection and
reporting. We have calculated the burden of proposed collection
based on the current time spent collecting and compiling data for
reports and then completing the reports.
Northern Tier
Community Action Corporation is an Equal Opportunity Employer and
Provider of Services Programs and services made possible with
funding by the Community Services Block Grant through the
Department of Community and Economic Development
The current reporting
for the Pennsylvania quarterly Community Organization Planning
&Outcomes System (COPOS) reports and the Organizational
Standards reporting we provide to DCED requires an average of 237
hours annually. The burden of increasing this threefold (711hrs.
and possibly over $20,000 annually) for agencies such as ours that
are minimally staffed and underfunded is overwhelming; and to say
the least cost prohibitive.
The quality, utility, and
clarity of the information to be collected; and
Data
for some of the indicators in the report is not available in many
of our areas and will require establishing additional arrangements
with other organizations that are also overburdened, understaffed
and underfunded, trying to do the best they can with what they
have to serve the needs of our communities. Then if you look at
the level of data collected and reported in many cases is of
little value in our planning and evaluation efforts.
Ways
to minimize the burden of the collection of information on
respondents,
4. Including through the use of automated
collection techniques or other forms of information technology.
It
has always seemed a simple solution to reducing the reporting
burden on local agencies would be for OCS to create a national and
statewide client/program tracking system where every client/
program would be entered into the system with the indicator and
outcomes data collected and entered during the intake, service
provision and outcome process. Once data is entered into the
system, OCS and State Agencies could pull reports on a continual
basis. We understand the massive scope of such a project as
creating this system, but once implemented, it would free up local
agency staff to do the important job of serving the people and not
spending an inordinate amount of their time collecting data and
preparing reports.
Typically each program funding
agency has requirements for reporting each of their programs (i.e.
Homeless Prevention Programs, Foreclosure Prevention Programs
require reporting to HUD, Energy and Weatherization Programs are
required to report to DOE, etc.) We have a wealth of human capital
that truly cares about taking care of the needs of low-income
families and helping them to achieve self-sufficiency, but much of
their time and abilities is lost on data collection and reporting.
|
|
Comment 97 (Northwest New Jersey
Community Action ; Eligible Entity/Local Agency in NJ)
|
|
COST FACTORS
I would like
to focus my comments on the financial burden that I believe the
system will cost. The BIGGEST issue is that funding sources like
their own proprietary software and won’t share information
or allow us to import export data to our central database. As you
can imagine, with minimal funding for our programs, we don’t
have the staff time to double enter all the data. (Currently,
these federal funding sources will not cooperate: WIC,
Weatherization, LIHEAP Energy Assistance, Child Care Block Grant,
Head Start, Early Head Start and HUD.) Additionally, there are
many State programs that will not do it either. The irony is, in
our own state (NJ) our CSBG cognizant agency (NJ Department of
Community Affairs) requires us to have a central database, and
they run some of these programs (LIHEAP, WX HUD), but they won’t
allow the data to be exported!
So what if we were
forced to enter all this information, what would it cost? Here’s
my conservative estimate:
· It now takes about
a half hour to do a family. There are 300 factors going to 1000
(as I understand it), but let’s just say it’s now an
hour for round terms.
· We don’t know how
many unduplicated clients we have because of the funding source
software issue previously mentioned. We believe that there are
about 35,000 clients, but let’s low ball it at 20,000; so
that’s 20,000 hours.
· Assume data clerks
make (with payroll taxes and fringe) $50,000 a year.
·
Assume they work 40 hours a week.
· 20,000 hours
represent 500 weeks.
· That’s 10 data
clerks x $50,000 = $500,000.
· Add in the cost
of the software, about $5,000 per year annual fee.
·
Add in an internal quality management/analysis person at $105,000
(includes health and fringe)
· TOTAL = $610,000
· I would ask:
o Is it worth the
cost?
O I’d rather put that money into services.
O Who is going to pay it?
O Will I have to
pull program staff out and not get the services done at the
expense of reporting?
I appreciate what you are trying
to do and on a theoretical basis, I endorse it, BUT until the
practical side of this is addressed, it will render us ineffective
because we will be focused on reporting and not service. Community
action absolutely needs to tell a better story and it has been one
of our weaknesses. We are complex organizations making profound
changes to our communities and we always walk a tightrope trying
to do well on reduced budgets. I’m with you in the “data
collection spirit,” but not on practical implementation
side.
|
|
Comment 98 (Norwescap; Eligible
Entity/Local Agency in NJ)
|
|
Thank you for the opportunity
to comment on the ROMA Next Generation concepts. While there are a
number of items in ROMA NG that don’t have much value, I
know that you are receiving comments from the National CAP
Association that mirror my feelings on them. So please consider my
unwritten comments a “ditto” of theirs when it comes
to the burden of collecting information that may be difficult to
get and at the same time have no value to the overall management
and direction of the agency.
COST FACTORS
I
would like to focus my comments on the financial burden that I
believe the system will cost. The BIGGEST issue is that funding
sources like their own proprietary software and won’t share
information or allow us to import export data to our central
database. As you can imagine, with minimal funding for our
programs, we don’t have the staff time to double enter all
the data. (Currently, these federal funding sources will not
cooperate: WIC, Weatherization, LIHEAP Energy Assistance, Child
Care Block Grant, Head Start, Early Head Start and HUD.)
Additionally, there are many State programs that will not do it
either. The irony is, in our own state (NJ) our CSBG cognizant
agency (NJ Department of Community Affairs) requires us to have a
central database, and they run some of these programs (LIHEAP, WX
HUD), but they won’t allow the data to be exported!
So
what if we were forced to enter all this information, what would
it cost? Here’s my conservative estimate:
·
It now takes about a half hour to do a family. There are 300
factors going to 1000 (as I understand it), but let’s just
say it’s now an hour for round terms.
· We don’t
know how many unduplicated clients we have because of the funding
source software issue previously mentioned. We believe that there
are about 35,000 clients, but let’s low ball it at 20,000;
so that’s 20,000 hours.
· Assume data
clerks make (with payroll taxes and fringe) $50,000 a year.
·
Assume they work 40 hours a week.
· 20,000 hours
represent 500 weeks.
· That’s 10 data
clerks x $50,000 = $500,000.
· Add in the cost
of the software, about $5,000 per year annual fee.
·
Add in an internal quality management/analysis person at $105,000
(includes health and fringe)
· TOTAL = $610,000
· I would ask:
o Is it worth the
cost?
O I’d rather put that money into services.
O Who is going to pay it?
O Will I have to
pull program staff out and not get the services done at the
expense of reporting?
I appreciate what you are trying
to do and on a theoretical basis, I endorse it, BUT until the
practical side of this is addressed, it will render us ineffective
because we will be focused on reporting and not service. Community
action absolutely needs to tell a better story and it has been one
of our weaknesses. We are complex organizations making profound
changes to our communities and we always walk a tightrope trying
to do well on reduced budgets. I’m with you in the “data
collection spirit,” but not on practical implementation
side.
If you have questions about this, please contact
me at the number below and PLEASE be realistic about your
expectations. Thanks.
|
|
Comment 99 (Northeast Community
Action Agency ; Other/Unknown in FL)
|
|
No Comment
|
|
Comment 101 (Ohio Association Of
Community Action Agencies; Other/Unknown in OH)
|
|
Ohio’s fifty community
action agencies have reviewed the new ROMA Next Generation Modules
and determined that the cost would be, on average, equal to
anywhere from 10% to 16% of their total CSBG award. Of those sent
to me, most were closer to the 16 % range. This is burdensome, to
say the least, and not as Congress intends the money to be spent.
We are also concerned about the requirements in the
various modules and rather than write to all of the points, I
think it would be more efficient to tell you that we agree with
the comments of the Community Action Partnership. The only
exception would be the points made about public agencies, as Ohio
has only private, nonprofit Community Action Agencies.
|
|
Comment 104 (People Incorporated Of
Virginia; Other/Unknown in VA)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility
While the proposed Annual
Report contains some elements that will be useful, particularly at
the Individual and Family levels, there are a number of areas and
indicators that cause the Board and staff of People Incorporated
grave concern. Specific areas are detailed below, and focus
largely on the Community level indicators. In its current
iteration, the proposed CSBG Annual Report is overly burdensome,
does not provide for consistent data across agencies and states,
and requests extensive information outside the mandate and scope
of the Community Services Block Grant. Much of the information
requested is inappropriate and has limited, if any practical
utility.
2. The accuracy of the agency’s
estimate of the burden of the proposed collection of information
The agency’s’ estimated time burden of 242
hours per sub-grantee (local Community Action Agency) is
significantly under what we estimated. Based on staff
calculations, the burden to People Incorporated would be 2,652
hours in the year before reporting begins, 3,094 hours in the
first year of reporting and 2,522 hours per year thereafter. The
cost of this time burden is estimated to be $82,812, $74,138 and
$60,622 in years one, two and thereafter. This accounts only for
staff time; the new IT system necessary to track and report the
new data elements would likely double this cost.
The
current IS survey includes 300+ data elements to report. The
proposed CSBG Annual Report includes more than 1,000. This
increased burden of reporting comes with no new resources.
3.
The quality, utility, and clarity of the information to be
collected; and
People Incorporated's greatest concern
in regard to this proposed report are in the area of the Community
level indicators. The social or population level indicators being
requested in Module 3 are inappropriate for individual agency
reporting and are best utilized for community-wide scorecards or
as part of a community needs assessment. Comparing overall
percentages of community indicators to individual local agencies
outcomes is misleading. As an example, this agency does extensive
work in economic development, including job creation, but in the
last year, the gains staff have been able to make in job creation
in SW Virginia are more than offset by mine and plant closures,
making reporting on the rate of unemployment meaningless in terms
of our agency performance; the hundreds of jobs People
Incorporated has created are subsumed in the larger number of
layoffs when utilizing the unemployment rate as an indicator. This
is true for many, if not all of the proposed social indicators,
which include, but are not limited to the change in
communities' rates of infant mortality, childhood obesity, adult
literacy, recidivism, high school graduation, homeownership,
foreclosure, et cetera.
4. Ways to minimize the burden of
the collection of information on respondents, including through
the use of automated collection techniques or other forms of
information technology.
OCS’ Dear Colleague
Letter of June 17, 2016, states, “Aware of the reporting
burden the proposed Annual Report will entail, OCS will decrease
that burden by establishing an on-line automated system for use by
the states and – at the discretion of the states – the
local agencies, for reporting that allows in many instances
auto-populating data from one year to the next and a variety of
data entry processes (e.g. manual entry and automatic upload).”
This online system and any new state-level systems will do little
to address the needs of local agencies for improved management
information systems with the capacity to collect and report the
data required by ROMA Next Generation.
The online
automated system may benefit State Lead Agencies (grantees), but
will most likely not benefit local Community Action Agencies
(sub-grantees). Sub-grantees will still need systems that track
customers, services, outcomes, and given the requirement of many
funding sources (including other federal Department of Health and
Human Services funding) requires additional and often separate
data collection systems. People Incorporated is required by
funders to use more than 10 discrete data systems, no two of which
are designed to communicate readily with an outside system.
OCS and the federal
government provide funding to local sub-grantees that require
separate systems and prohibit the aggregation of data. To minimize
the burden on respondents, DHHS should examine its own practices
to ensure that all DHHS-funded programs in the field can use data
collection systems that can be easily integrated and “talk”
to one another to produce reports that meet the needs of local
agencies, states, and OCS.
|
|
Comment 105 (Proaction Of Steuben And
Yates Inc.; Eligible Entity/Local Agency in NY)
|
|
Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility.
While there are elements
that may be helpful, there are a number of areas detailed below
where the collection of information does not meet the threshold of
necessity or utility. The package as proposed in its entirety is
overly burdensome, contains reporting tools that will provide
useless information, and is outside the reach of the Community
Services Block Grant (CSBG). However, with modifications, a system
for collection of information can be developed that improves the
performance management of the agency and has practical utility.
Detail is provided here outlining where changes need to be made to
reach a level of workability and utility.
1. Module 1:
Modify the options CSBG State Lead Agencies/grantees have when
reporting to OCS on CSBG Organizational Standards to allow for a
broader range of responses. As noted below, there are significant
challenges with the way this information is being reported and
creates a misuse of the Organizational Standards. Expecting
constant perfection is unrealistic and does not have practical
utility for local agencies or the State CSBG Lead Agency. As
proposed, the single metric the State Agencies will report on will
be the number of eligible entities that meet 100% of the CSBG
Organizational Standards. To increase utility of the information,
State Agencies should have the option to report the number of
agencies at a variety of thresholds that still delineate strong
performance or allow for a more accurate reflection of the
training and technical assistance needs. For instance State
Agencies should be able to report the number of local agencies
that have met 100%, 90%, 80%, 70%, 60%, 50%, and less than 50% of
the standards. This will allow for the collection of more accurate
information; better tracking of results over time; and better,
more efficient investment of training and technical
assistance.
2. Module 4: Remove the Characteristics for
NEW Individuals and Households Report. This report has no
practical utility at the local level given the time and expense to
create it at each local agency. This is not necessary for agency
performance, and the intentions outlined by OCS for its use are
not reflective of how local agencies and states look at their
communities to assess needs, develop programs, and report
progress. It also lacks utility at the state level given that data
such as this rolled up to the state level will be skewed by large
population centers and decrease the practical utility of the data
even further. In addition, many agencies may either lack
management information systems with the capacity to disaggregate
this type of data from its overall customer database, or struggle
to integrate the required data given their use of multiple (and
sometimes incompatible) reporting systems. In addition, as many
local agencies serve families over time, with no definition of
“new” being provided for review, the CSBG Network will
be unable to implement this report consistently. Defining an
individual or family as new becomes challenging when they may
receive different services over many years (e.g. Early Head Start,
child care, after school programs, youth programs, job training,
WIC, food boxes, LIHEAP, etc.).
3. Module 3: Remove the
Community Outcome Indicators that include percent or rates. Their
inclusion is inappropriate and will provide meaningless data when
aggregated beyond individual initiatives. Such data has validity
for geographic areas much larger in than those addressed or
targeted by local agency projects. This validity is lost when the
scale is smaller and accepted data comparison points become
unavailable. Having individual agencies report on rate change at
the community level on social indicators does not provide
information that is usable. It is more appropriate to give
agencies the option of reporting outcome data and the flexibility
to define which indicators they use.
4. Module 3:
Remove Collective Impact from all reports. Collective Impact is
simply one set of strategies for doing community-level work and
while CSBG eligible entities are engaged in such efforts,
privileging one approach given the broader range of options for
doing community-level work is inappropriate and unnecessary. This
should be removed and included in training and technical
assistance endeavors.
5. Module 4: Remove the Stability
Indicators. The indicators selected and included in the clearance
package are not helpful or practical when reporting customers
stabilized as a result of the eligible entity’s work. One
option duplicates data already collected and the second uses a
tool that defines financial stability based on a construct the
uses assumptions drawn from the general population, not low-income
individuals and families specifically. Because financial
well-being means something very different for the low-income
population in comparison to the general population, use of the
scale will produce data of limited utility.
6. Address
the unique issues of Public entities that receive CSBG. Public
agencies (sub-grantees) that sub-contract most or all of their
CSBG funding to other service providers may face added challenges
of requiring their grantees to collect data in accordance with
ROMA Next Generation requirements since it may limit agencies to
contracting with providers that have the requisite data collection
capacity or require additional technical assistance to meet the
reporting requirements. This challenge is not addressed in the
proposed reporting package.
7. Include a 4th ROMA Goal:
Agencies Increase their capacity to achieve results. Reinstate
this national goal that addressed the critical role the network of
local agencies serving 99% of US counties plays in addressing
poverty. Reasons for exclusion have been inadequate to date.
Because agency capacity is critical to the ultimate success of
programs and services, it is important to clearly state this
objective in the overall National Goals. This also highlights the
broader need of the Network for ongoing training, technical
assistance, and general capacity-building supports. Question 2:
The accuracy of the agency’s estimate of the burden of the
proposed collection of information.
The Partnership
urges local agencies and states to estimate their time and cost
burden. Based on feedback from the Network, we suspect that the
current estimation is low given the increase in the amount of
information required and the challenges many agencies face
collecting data from multiple reporting systems. Agencies that
respond to the estimation of burden question should choose one of
the following options and include an explanation detailing how
they arrived at the number and the impact of this collection on
the agency or the state office. (Please forward your estimate to
NCAF and the Partnership to assist in the estimated burden
Network-wide.)
· Concerns include:
1.
The current IS survey includes 300+ data elements to report. The
proposed CSBG Annual Report includes more than 1,000. This
increased burden of reporting comes with no new resources.
2.
The burden on local agencies is not impacted by the implementation
of the Online Data Collection system (OLDC). To comply with the
draft CSBG Annual Report local agencies will need to modify
existing systems.
3. Many agencies lack management
information systems with the capacity to provide the required
data. Data fields required for ROMA Next Generating reporting may
not be mirrored in those used by agencies. In addition, many
agencies use multiple (and sometimes incompatible) reporting
systems which adds to the time necessary to provide the data.
·
The agency’s estimate time burden of 242 hours per
sub-grantee is significantly underestimated. Based on estimated
calculations, the burden to our local agency will be more than
2,000 hours. The cost burden for the first year alone is estimated
to be around $32,000.
· In addition, it is not
clear how the burden for local agencies is decreased by the online
data collection system described in the by OCS. OCS’ Dear
Colleague Letter of June 17, 2016, states, “Aware of the
reporting burden the proposed Annual Report will entail, OCS will
decrease that burden by establishing an on-line automated system
for use by the states and – at the discretion of the states
– the local agencies, for reporting that allows in many
instances auto-populating data from one year to the next and a
variety of data entry processes (e.g. manual entry and automatic
upload).” It is important to note, this online automated
system may benefit State Lead Agencies (grantees), but will most
likely not benefit local Community Action Agencies (sub-grantees).
Sub-grantees will still need systems that track customers,
services, outcomes, and given the requirement of many funding
sources (including other federal Department of Health and Human
Services funding) requires additional and sometime separate data
collection systems. State grantees do not have the funding or
capacity for the most part, to create systems or support local
agencies to develop systems that will streamline or decrease the
burden outlined here.
Question 3: The quality, utility,
and clarity of the information to be collected.
As
noted below in the detailed comments by module, several elements
of data to be collected under this proposal have no meaning at the
state and federal levels, and only have value, meaning, and
usefulness at the local level. Several of the elements to be
collected as described are not available in smaller geographic
areas usually targeted by local agencies in their community work
and if created by local agencies, lose validity. The social or
population level indicators being requested are not meant for
single agency reporting and are best served for community-wide
scorecards or utilized as part of a community needs assessment.
For example, comparing overall percentages of community indicators
(e.g. the unemployment rate, violent crime rate) to individual
local agencies outcomes is misleading
· Module
4, Section A: Characteristics for NEW Individuals and Households -
Data Entry Form is an unnecessary document that will not produce
quality data; it will have little utility at a local level; and it
lends itself to meaningless data at a state and national level. It
should be removed. Because demographic data alone does not include
information about the needs of new customers, this information
fails to help agencies determine if they are addressing issues
identified in their community assessments. Furthermore, the large
number of “one time” customers in programs such as
LIHEAP will skew any meaningful interpretation of the data. Aside
from the challenge of defining who constitutes a new customer,
agencies that use multiple reporting systems will also have the
added burden of tracking new customers across all programs and
services.
· Module 3, Section A: Many of the
indicators used here are inappropriate for reporting by local
sub-grantees to States and by States to OCS and will produce data
of limited utility. Several challenges underscore this point.
First, community-level data on indicators is not always available
at the level (e.g. regional or zip code/census tract) or
population (e.g. low-income children aged 0-5) that matches the
area and population targeted by the agency. Second,
community-level indicators are influenced by a wide number of
sources that would most likely overwhelm the impact of even
successful community initiatives, a fact that significantly limits
their utility for providing meaningful information about outcomes.
Third, any meaningful data about the outcomes of community-level
initiatives would require a rigorous program evaluation and could
not be determined on the data collected through the Annual Report
alone.
Question 4: Ways to minimize the burden of the
collection of information on respondents, including through the
use of automated collection techniques or other forms of
information technology.
· As noted above, OCS’
Dear Colleague Letter of June 17, 2016, states, “Aware of
the reporting burden the proposed Annual Report will entail, OCS
will decrease that burden by establishing an on-line automated
system for use by the states and – at the discretion of the
states – the local agencies, for reporting that allows in
many instances auto-populating data from one year to the next and
a variety of data entry processes (e.g. manual entry and automatic
upload).” It is important to note, this online automated
system may benefit State Lead Agencies (grantees), but will most
likely not benefit local Community Action Agencies (sub-grantees).
Sub-grantees (local agencies) will still need systems that track
customers, services, outcomes, and given the requirement of many
funding sources (including other federal Department of Health and
Human Services funding) requires additional and sometime separate
data collection systems. State grantees do not have the funding or
capacity for the most part, to create systems that will streamline
or decrease the burden outlined here. Furthermore, new state-level
systems will do little to address the needs of local agencies for
improved management information systems with the capacity to
collect and report the data required by ROMA Next Generation.OCS
and the federal government provide funding to local sub-grantees
that require separate systems and prohibit the aggregation of
data. To minimize the burden on respondents, DHHS should examine
its own practices to ensure that all DHHS-funded programs in the
field can use data collection systems that can be easily
integrated and “talk” to one another to produce
reports that meet the needs of local agencies, states, and OCS.
|
|
Comment 108 (Redwood Community Action
Agency ; Eligible Entity/Local Agency in CA)
|
|
This response is provided by a
CSBG eligible entity engaged in day-to-day operations serving
clients through a variety of programs which are leveraged
extensively with other funding sources (and reporting
requirements). We appreciate the opportunity to provide feedback
and comments that may assist in developing the most appropriate
annual report to capture the impact of CSBG programs.
|
|
Comment 109 (Sa Howell, LLC ;
Other/Unknown in GA)
|
|
An agency level ROMA Goal
needs to be included in order to track organizational capacity.
Agency level activities are a fundamental principle of Community
Action, and the inclusion of agency level outcomes is extremely
necessary for tracking purposes.
|
|
Comment 110 (Semcac; Eligible
Entity/Local Agency in MN)
|
|
Question 1. Whether the
proposed collection of information is necessary for the proper
performance of the functions of the agency, including whether the
information shall have practical utility.
While there
are elements that may be helpful, there are a number of areas
detailed below where the collection of information does not meet
the threshold of necessity or utility. The package as proposed in
its entirety is overly burdensome, contains reporting tools that
will provide useless information, and is outside the reach of the
Community Services Block Grant (CSBG). However, with
modifications, a system for collection of information can be
developed that improves the performance management of the agency
and has practical utility. Detail is provided here outlining where
changes need to be made to reach a level of workability and
utility.
1. Module 1: Modify the options CSBG State
Lead Agencies/grantees have when reporting to OCS on CSBG
Organizational Standards to allow for a broader range of
responses. As noted below, there are significant challenges with
the way this information is being reported and creates a misuse of
the Organizational Standards. Expecting constant perfection is
unrealistic and does not have practical utility for local
agencies. As proposed, the single metric the State Agencies will
report on will be the number of eligible entities that meet 100%
of the CSBG Organizational Standards. To increase utility of the
information, State Agencies should have the option to report
the
Dodge County
Fillmore County
Freeborn
County
Houston County
Mower County
Steele
County
Winona
County
634-4350
765-2761
373-1329
725-3677
433-5889
451-7134
452-8396
With
additional programs operated in Goodhue, Olmsted, Rice and Wabasha
Counties
Please remember Semcac programs in your
financial and estate planning. Your legacy is a gift to the
future.
An Equal Opportunity Employer
A
Community Action Agency Serving Southeast Minnesota Since
1966!
204 South Elm, PO Box 549, Rushford, MN
55971-0549
Phone: 507-864-7741 Fax: 507-864-2440 Visit:
www.semcac.org Email: [email protected]
number of
agencies at a variety of thresholds that still delineate strong
performance or allow for a more accurate reflection of the
training and technical assistance needs. For instance State
Agencies should be able to report the number of local agencies
that have met 100%, 90%, 80%, 70%, 60%, 50%, and less than 50% of
the standards. This will allow for the collection of more accurate
information; better tracking of results over time; and better,
more efficient investment of training and technical
assistance.
2. Module 4: Remove the Characteristics for
NEW Individuals and Households Report. This report has no
practical utility at the local level given the time and expense to
create it at each local agency. This is not necessary for agency
performance, and the intentions outlined by OCS for its use are
not reflective of how local agencies and states look at their
communities to assess needs, develop programs, and report
progress. It also lacks utility at the state level given that data
such as this rolled up to the state level will be skewed by large
population centers and decrease the practical utility of the data
even further. In addition, many agencies may either lack
management information systems with the capacity to disaggregate
this type of data from its overall customer database, or struggle
to integrate the required data given their use of multiple (and
sometimes incompatible) reporting systems. In addition, as many
local agencies serve families over time, with no definition of
“new” being provided for review, the CSBG Network will
be unable to implement this report consistently. Defining an
individual or family as new becomes challenging when they may
receive different services over many years (e.g. Early Head Start,
child care, after school programs, youth programs, job training,
WIC, food boxes, LIHEAP, etc.).
3. Module 3: Remove the
Community Outcome Indicators that include percent or rates. Their
inclusion is inappropriate and will provide meaningless data when
aggregated beyond individual initiatives. Such data has validity
for geographic areas much larger in than those addressed or
targeted by local agency projects. This validity is lost when the
scale is smaller and accepted data comparison points become
unavailable. Having individual agencies report on rate change at
the community level on social indicators does not provide
information that is usable. It is more appropriate to give
agencies the option of reporting outcome data and the flexibility
to define which indicators they use.
4. Module 3:
Remove Collective Impact from all reports. Collective Impact is
simply one set of strategies for doing community-level work and
while CSBG eligible entities are engaged in such efforts,
privileging one approach given the broader range of options for
doing community-level work is inappropriate and unnecessary. This
should be removed and included in training and technical
assistance endeavors.
5. Module 4: Remove the Stability
Indicators. The indicators selected and included in the clearance
package are not helpful or practical when reporting customers
stabilized as a result of the eligible entity’s work. One
option duplicates data already collected and the second uses a
tool that defines financial stability based on a construct the
uses assumptions drawn from the general population, not low-income
individuals and families specifically. Because financial
well-being means something very different for the low-income
population in comparison to the general population, use of the
scale will produce data of limited utility.
6. Include
a 4th ROMA Goal: Agencies Increase their capacity to achieve
results. Reinstate this national goal that addressed the critical
role the network of local agencies serving 99% of US counties
plays in addressing poverty. Reasons for exclusion have been
inadequate to date. Because agency capacity is critical to the
ultimate success of
programs and services, it is
important to clearly state this objective in the overall National
Goals. This also highlights the broader need of the Network for
ongoing training, technical assistance, and general
capacity-building supports.
Question 2: The accuracy of
the agency’s estimate of the burden of the proposed
collection of information.
1. The current IS survey
includes 300+ data elements to report. The proposed CSBG Annual
Report includes more than 1,000. This increased burden of
reporting comes with no new resources.
2. The burden on
local agencies is not impacted by the implementation of the Online
Data Collection system (OLDC). To comply with the draft CSBG
Annual Report local agencies will need to modify existing
systems.
3. Many agencies lack management information
systems with the capacity to provide the required data. Data
fields required for ROMA Next Generating reporting may not be
mirrored in those used by agencies. In addition, many agencies use
multiple (and sometimes incompatible) reporting systems which adds
to the time necessary to provide the data.
4. The
agency’s’ estimated time burden of 242 hours per
sub-grantee (local Community Action Agency) is significantly under
estimated. Based on our calculations, the burden to our local
agency would be 322 The cost of this time burden is estimated to
be $6,440.00
· In addition, it is not clear how
the burden for local agencies is decreased by the online data
collection system described in the by OCS. OCS’ Dear
Colleague Letter of June 17, 2016, states, “Aware of the
reporting burden the proposed Annual Report will entail, OCS will
decrease that burden by establishing an on-line automated system
for use by the states and – at the discretion of the states
– the local agencies, for reporting that allows in many
instances auto-populating data from one year to the next and a
variety of data entry processes (e.g. manual entry and automatic
upload).” It is important to note, this online automated
system may benefit State Lead Agencies (grantees), but will most
likely not benefit local Community Action Agencies
(sub-grantees).
Question 3: The quality, utility, and
clarity of the information to be collected.
As noted
below in the detailed comments by module, several elements of data
to be collected under this proposal have no meaning at the state
and federal levels, and only have value, meaning, and usefulness
at the local level.
· The social or population
level indicators being requested are not meant for single agency
reporting and are best served for community-wide scorecards or
utilized as part of a community needs assessment. For example,
comparing overall percentages of community indicators (e.g. the
unemployment rate, violent crime rate) to individual local
agencies outcomes is misleading
· Module 4,
Section A: Characteristics for NEW Individuals and Households -
Data Entry Form is an unnecessary document that will not produce
quality data; it will have little utility at a local level; and it
lends itself to meaningless data at a state and national level. It
should be removed. Because demographic data alone does not include
information about the needs of new customers, this information
fails to help agencies
determine if they are addressing
issues identified in their community assessments. Furthermore, the
large number of “one time” customers in programs such
as LIHEAP will skew any meaningful interpretation of the data.
Aside from the challenge of defining who constitutes a new
customer, agencies that use multiple reporting systems will also
have the added burden of tracking new customers across all
programs and services.
· Module 3, Section A:
Many of the indicators used here are inappropriate for reporting
by local sub-grantees to States and by States to OCS and will
produce data of limited utility. Several challenges underscore
this point. First, community-level data on indicators is not
always available at the level (e.g. regional or zip code/census
tract) or population (e.g. low-income children aged 0-5) that
matches the area and population targeted by the agency. Second,
community-level indicators are influenced by a wide number of
sources that would most likely overwhelm the impact of even
successful community initiatives, a fact that significantly limits
their utility for providing meaningful information about outcomes.
Third, any meaningful data about the outcomes of community-level
initiatives would require a rigorous program evaluation and could
not be determined on the data collected through the Annual Report
alone.
Question 4: Ways to minimize the burden of the
collection of information on respondents, including through the
use of automated collection techniques or other forms of
information technology.
· As noted above, OCS’
Dear Colleague Letter of June 17, 2016, states, “Aware of
the reporting burden the proposed Annual Report will entail, OCS
will decrease that burden by establishing an on-line automated
system for use by the states and – at the discretion of the
states – the local agencies, for reporting that allows in
many instances auto-populating data from one year to the next and
a variety of data entry processes (e.g. manual entry and automatic
upload).” It is important to note, this online automated
system may benefit State Lead Agencies (grantees), but will most
likely not benefit local Community Action Agencies (sub-grantees).
Sub-grantees (local agencies) will still need systems that track
customers, services, outcomes, and given the requirement of many
funding sources (including other federal Department of Health and
Human Services funding) requires additional and sometime separate
data collection systems. State grantees do not have the funding or
capacity for the most part, to create systems that will streamline
or decrease the burden outlined here. Furthermore, new state-level
systems will do little to address the needs of local agencies for
improved management information systems with the capacity to
collect and report the data required by ROMA Next Generation.
·
OCS and the federal government provide funding to local
sub-grantees that require separate systems and prohibit the
aggregation of data. To minimize the burden on respondents, DHHS
should examine its own practices to ensure that all DHHS-funded
programs in the field can use data collection systems that can be
easily integrated and “talk” to one another to produce
reports that meet the needs of local agencies, states, and OCS.
|
|
Comment 111 (South Plain Community
Action Association ; State Association in TX)
|
|
South Plains Community Action
Association supports comments filed by the Texas Department of
Housing and Community Affairs regarding proposed changes to the
CSBG Annual Report and we feel this unfunded mandate will spike
expenses for additional staff and upgraded systems in order to
capture the enormous amount of data as proposed. Local staff will
be steered away from helping people to capturing a vast amount of
information and navigating a complicated reporting system which
may yield less impressive results of the CSBG program because
staff is re-directed from direct services to paperwork. Indicators
won’t necessarily capture the variety of Community Action
achievements and projects and won’t tell the true CSBG
story. Cost estimates are difficult to gage due to many unknowns
at this point in time
|
|
Comment 112 (Southeastern North
Dakota ; Eligible Entity/Local Agency in ND)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility
While there are elements
that may be helpful, there are a number of areas detailed below
where the collection of information does not meet the threshold of
necessity or utility. The package as proposed in its entirety is
overly burdensome, contains reporting tools that will provide
useless information, and is outside the reach of the Community
Services Block Grant (CSBG). However, with modifications, a system
for collection of information can be developed that improves the
performance management of the agency and has practical utility.
Detail is provided here outlining where changes need to be made to
reach a level of workability and utility.
1. Module I:
Modify the options CSBG State Lead Agencies/grantees have when
reporting to OCS on CSBG Organizational Standards to allow for a
broader range of responses. As noted below, there are significant
challenges with the way this information is being reported and
creates a misuse of the Organizational Standards. Expecting
constant perfection is unrealistic and does not have practical
utility for local agencies or the State CSBG Lead Agency. As
proposed, the single metric the State Agencies will report on will
be the number of eligible
entities that meet 100% of the CSBG
Organizational Standards. To increase utility of the information,
State Agencies should have the option to report the number of
agencies at a variety of thresholds that still delineate strong
performance or allow for a more accurate reflection of the
training and technical assistance needs. For instance State
Agencies should be able to report the number of local agencies
that have met 100%, 90%, 80%, 70%, 60%, 50%, and less than 50% of
the standards. This will allow for the collection of more accurate
information; better tracking of results over time; and better,
more efficient investment of training and technical
assistance.
Module 4: Remove the Characteristics for
NEW Individuals and Households Report. This report has no
practical utility at the local level given the time and expense to
create it at each local
agency. This is not necessary for
agency performance, and the intentions outlined by OCS for its use
are not reflective of how local agencies and states look at their
communities to assess needs, develop programs, and report
progress. It also lacks utility at the state level given that data
such as this rolled up to the state level will be skewed by large
population centers and decrease the practical utility of the data
even further. In addition, many agencies may either lack
management information systems with the capacity to disaggregate
this type of data from its overall customer database, or struggle
to integrate the required data given their use of multiple (and
sometimes incompatible) reporting systems. In addition, as many
local agencies serve families over time, with no definition of
"nevi' being provided for review, the CSBG Network will be
unable to implement this report consistently. Defining an
individual or family as new becomes challenging when they may
receive different services over many years (e.g. Early Head Start,
child care, after school programs, youth programs, job training,
WIC, food boxes, LIHEAP, etc.).
Module 3: Remove the
Community Outcome Indicators that include percent or rates. Their
inclusion is inappropriate and will provide meaningless data when
aggregated beyond individual initiatives. Such data has validity
for geographic areas much larger in than those addressed or
targeted by local agency projects. This validity is lost when the
scale is smaller and accepted data comparison points become
unavailable. Having individual agencies report on rate change at
the community level on social indicators does not provide
information that is usable. It is more appropriate to give
agencies the option of reporting outcome data and the flexibility
to define which indicators they use.
Module 3: Remove
Collective Impact from all reports. Collective Impact is simply
one set of strategies for doing community-level work and while
CSBG eligible entities are engaged in such efforts, privileging
one approach given the broader range of options for doing
community-level work is inappropriate and unnecessary. This should
be removed and included in training and technical assistance
endeavors.
Module 4: Remove the Stability Indicators.
The indicators selected and included in the clearance package are
not helpful or practical when reporting customers stabilized as a
result of the
eligible entity's work. One option duplicates
data already collected and the second uses a tool that defines
financial stability based on a construct the uses assumptions
drawn from the general population, not low-income individuals and
families specifically. Because financial well-being means
something very different for the low-income population in
comparison to the general population, use of the scale will
produce data of limited utility.
Address the unique
issues of Public entities that receive CSBG. Public agencies
(sub-grantees) that sub-contract most or all of their CSBG funding
to other service providers may face added challenges of requiring
their grantees to collect data in accordance with ROMA Next
Generation requirements since it may limit agencies to contracting
with providers that have the requisite data collection capacity or
require additional technical assistance to meet the reporting
requirements. This challenge is not addressed in the proposed
reporting package.
Include a 4th ROMA Goal: Agencies Increase
their capacity to achieve results. Reinstate this national goal
that addressed the critical role the network of local agencies
serving 99% of US
counties plays in addressing poverty.
Reasons for exclusion have been inadequate to date. Because agency
capacity is critical to the ultimate success of programs and
services, it is important to clearly state this objective in the
overall National Goals. This also highlights the broader need of
the Network for ongoing training, technical assistance, and
general capacity-building supports.
2. The accuracy of
the agency's estimate of the burden of the proposed collection of
information
Regarding the accuracy of the agency
estimate of the cost burden, CAPND concerns include:
The
current IS survey includes 300+ data elements to report. The
proposed CSBG Annual Report includes more than 1,000. This
increased burden of reporting comes with no new resources.
The
burden on local agencies is not impacted by the implementation of
the Online Data Collection system (OLDC). To comply with the draft
CSBG Annual Report local agencies will need to modify existing
systems. An initial investment in new software for North Dakota
was quoted at over $50,000 which only covers the set-up and one
year of training. This does include any of the costs association
with adding necessary modules such as Weatherization or Head
Start.
Many agencies lack management information
systems with the capacity to provide the required data. Data
fields required for ROMA Next Generating reporting may not be
mirrored in those used by agencies. In addition, many agencies use
multiple (and sometimes incompatible) reporting systems which adds
to the time necessary to provide the data.
In addition,
it is not clear how the burden for local agencies is decreased by
the online data collection system described in the by OCS. OCS'
Dear Colleague Letter of June 17, 2016, states, "Aware of the
reporting burden the proposed Annual Report will entail, OCS will
decrease that burden by establishing an on-line automated system
for use by the states and — at the discretion of the states
— the local agencies, for reporting that allows in many
instances auto-populating data from one year to the next and a
variety of data entry processes (e.g. manual
It is
important to note, this online automated system may entry and
automatic upload)." benefit State Lead Agencies (grantees),
but will most likely not benefit local Community Action Agencies
(sub-grantees). Sub-grantees will still need systems that track
customers, services, outcomes, and given the requirement of many
funding sources (including other federal
Department of Health
and Human Services funding) requires additional and sometime
separate data collection systems. State grantees do not have the
funding or capacity for the most part, to create systems or
support local agencies to develop systems that will streamline or
decrease the burden outlined here.
3.The quality,
utility, and clarity of the information to be collected
As
noted below in the detailed comments by module, several elements
of data to be collected under this proposal have no meaning at the
state and federal levels, and only have value, meaning, and
usefulness at the local level.
Several of the elements
to be collected as described are not available in smaller
geographic areas usually targeted by local agencies in their
community work and if created by local agencies, lose validity.
The social or population level indicators being requested are not
meant for single agency reporting and are best served for
community-wide scorecards or utilized as part of a community needs
assessment. For example, comparing overall percentages of
community indicators (e.g. the unemployment rate, violent crime
rate) to individual local agencies outcomes is misleading
Module
4, Section A: Characteristics for NEW Individuals and Households -
Data Entry Form is an unnecessary document that will not produce
quality data; it will have little utility at a local level; and it
lends itself to meaningless data at a state and national level. It
should be removed. Because demographic data alone does not include
information about the needs of new customers, this information
fails to help agencies determine if they are addressing issues
identified in their community assessments. Furthermore, the large
number of "one time" customers in programs such as
LIHEAP will skew any meaningful interpretation of the data. Aside
from the challenge of defining who constitutes a new customer,
agencies that use multiple reporting systems will also have the
added burden of tracking new customers across all programs and
services.
Module 3, Section A: Many of the indicators
used here are inappropriate for reporting by local sub-grantees to
States and by States to OCS and will produce data of limited
utility. Several challenges underscore this point. First,
community-level data on indicators is not always available at the
level (e.g. regional or zip code/census tract) or population (e.g.
low-income children aged 0-5) that matches the area and population
targeted by the agency. Second, community-level indicators are
influenced by a wide number of sources that would most likely
overwhelm the impact of even successful community initiatives, a
fact that significantly limits their utility for providing
meaningful information about outcomes. Third, any meaningful data
about the outcomes of community-level initiatives would require a
rigorous program evaluation and could not be determined on the
data collected through the Annual Report alone.
4. Ways
to minimize the burden of the collection of information on
respondents, including through the use of automated collection
techniques or other forms of information technology.
As
noted above, OCS' Dear Colleague Letter of June 17, 2016, states,
"Aware of the reporting burden the proposed Annual Report
will entail, OCS will decrease that burden by establishing an
on-line automated system for use by the states and — at the
discretion of the states — the local agencies, for reporting
that allows in many instances auto-populating data from one year
to the next and a variety of data entry processes (e.g. manual
entry and automatic upload)."
It is important to
note, this online automated system may benefit State Lead Agencies
(grantees), but will most likely not benefit local Community
Action Agencies (sub-grantees). Sub-grantees (local agencies) will
still need systems that track customers, services, outcomes, and
given the requirement of many funding sources (including other
federal Department of Health and Human Services funding) requires
additional and sometime separate data collection systems. State
grantees do not have the funding or capacity for the most part, to
create systems that will streamline or decrease the burden
outlined here. Furthermore, new state-level systems will do little
to address the needs of local agencies for improved management
information systems with the capacity to collect and report the
data required by ROMA Next Generation.
OCS and the
federal government provide funding to local sub-grantees that
require separate systems and prohibit the aggregation of data. To
minimize the burden on respondents, DHHS should examine its own
practices to ensure that all DHHS-funded programs in the field can
use data collection systems that can be easily integrated and
"talk" to one another to produce reports that meet the
needs of local agencies, states, and OCS.
|
|
Comment 113 (Southern New Hampshire
Services ; Eligible Entity/Local Agency in NH)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility
The proposed collection
of information represents a significant increase from current
practice, given that the proposed CSBG Annual Report essentially
triples the data elements currently contained in the Information
System Survey. We do not believe that all of the information is
necessary, nor is it likely that a number of elements will have
any practical usefulness, especially when aggregated across the
1,000 Community Action Agencies. Points spelled out below will
identify more specifics in this area.
2. The accuracy
of the agency’s estimate of the burden of the proposed
collection of information
We feel the time burden of
242 hours per agency is underestimated. Based on our calculations
we estimate approximately 400 hours or more, given the amount of
additional information to collect, necessary data collection and
reporting software training, software enhancements, data quality
checks, etc. We conservatively estimate the cost burden to be in
the area of $10,000. This does not include the annual support cost
of approximately $10,000 for the software utilized to generate the
data in the CSBG report. (Based on our agency’s percentage
of clients served by the statewide network supported by the
software.)
3. The quality, utility, and clarity of the
information to be collected; and
Specific comments speaking
to these points will be made below on individual modules.
4.
Ways to minimize the burden of the collection of information on
respondents, including through the use of automated collection
techniques or other forms of information technology.
Automating
the collection process may be beneficial to the states in
aggregating the data, but does not impact the data collection
burden of any local agency. A significant improvement in data
collection efforts at the local level would be facilitated by
funding sources allowing for more flexibility in data collection
and reporting systems, and urging vendors providing data
collection systems to accommodate sharing information between
systems so that local agencies can more easily collect
unduplicated counts of individuals served. Much time and effort in
collecting data could be saved by ensuring that current software
programs, which are frequently proprietary and some cases
mandated, are able to “talk” to data collection
systems at the agency level, so that data does not have to be
gathered from so many different places.
|
|
Comment 114 (Southwest Oklahoma
Community Action ; Other/Unknown in OK)
|
|
Is this a valuable system?
Southwest Oklahoma Community Action Group, Inc. recognizes the
importance of accountability and continuous performance management
improvement; however, the idea that a report that potentially
triples the reporting requirements will somehow improve outcomes
for customers is simply mistaken .First, all federal dollars
require performance management and outcomes reporting. When
Community Action Agencies also report this information in the CSBG
IS report, it is in essence duplicating numbers. For instance, in
our agency we were funded last year to serve 250 children with
Head Start. Those children are being reported in the Program
Information Report (PIR) to HHS, and they are also being reported
in the CSBG IS report; hence duplicated reporting. Staff hours are
being used to aggregate data in the
format required by Head
Start, as well as reporting in a manner proscribed by CSBG.
Southwest Oklahoma Community Action received approximately
$114,000 in CSBG funding last year. Each time the data reporting
is increased, there is less funds for working to reduce the causes
and effects of poverty and to help people achieve
self-sufficiency. Secondly, the models that are followed do not
support a whole client approach across a multi-year platform which
is desperately needed in order to help people break the cycle of
poverty. Having a job for six months does not indicate stability
.Many people can work six months at a time, but they do not make
strides because they are always changing jobs. Agencies should be
finding solutions for why generations of families continue to need
our services. Additionally, agencies have had to add in the
requirements of Organizational Standards for data tracking_ This
alone is cumbersome due to the number of standards and the varying
time frames. One can quickly become out of compliance just by
scheduling difficulties or staff turnover. Some local agencies
have gone so far as to discuss relinquishing their CSBG dollars.
Our opinion is CSBG is the heart of the agency; it is what makes
our organization a Community Action. Yet, we have growing concerns
about the utilization of CSBG for more and more recordkeeping and
less work in the communities we serve.
2. Is the cost
estimate that HHS provided correct? The estimated time burden of
242 hours per sub-grantee (local CAA) is significantly under
estimated. Based on our estimation, the burden to our local agency
would be approximately an additional 1,000 hours for the ROMA Next
Generation reporting and the Organizational Standards maintenance.
This estimate does not include time that other programs spend
collecting and providing information for the CSBG reporting. It is
anticipated that the cost of the 1,000 hours will reach as much as
$35,000 (salaries and fringe) depending on which staff members are
tasked with the additional reporting. This is an additional 30% of
our organization's CSBG funds.
It does not improve an
agency's performance to report outcomes in multiple reporting
formats (CSBG, Head Start, Weatherization, etc.) .This are already
being done in grants management. What makes agencies more
performance driven is how they are collaborating in the community
to solve the problems of that community, what is being done that
is unique and successful, the ability to think creatively and
outside of the box. CSBG has always been the seed money to either
start new things or to address things that didn't have funding or
to help
augment programs to improve success. As always,
Southwest Oklahoma Community Action Group, Inc. will strive to
meet all requirements; however, it must be noted that each time
there are more added requirements, service to the community is
reduced. We appreciate the opportunity to comment and provide
feedback.
|
|
Comment 116 (Texas Association Of
Community Action Agencies, Inc. (Tacaa); Other/Unknown in TX)
|
|
Comments regarding proposed
changes to the CSBG Annual Report:
In our opinion,
this unfunded mandate will spike expenses for additional staff and
upgraded systems in order to capture the enormous amount of data
as proposed. Local staff will be steered away from helping people
to capturing a vast amount of information and navigating a
complicated reporting system which does not necessarily guarantee
impressive results of the CSBG program because staff is
re-directed from direct services to paperwork. Indicators won’t
necessarily capture the variety of Community Action achievements
and projects and won’t tell the CSBG story. Cost estimates
are difficult to gage due to many unknowns at this point in
time.
Please do not over complicate the reporting
process. Helping people is a priority!
|
|
Comment 118 (The State Of New York
Community Action Association ; State Association in NY)
|
|
Question 1. Whether the
proposed collection of information is necessary for the proper
performance of the functions of the agency, including whether the
information shall have practical utility.
The proposed
collection of information includes some useful elements but in
whole is quite burdensome and includes information that is of
limited use and well beyond the parameters of the Community
Services Block Grant (CSBG.) Suggested modifications which could
result in a useful and effective system of information collection
are noted below.
1. Module 1: Allow CSBG State Lead
Agencies/grantees to report on the eligible entities’
compliance with standards at a range of levels beyond 100%, for
example, 50%, 75% 90% and 100%. The Organizational Standards were
intended to promote excellence not merely the lowest level of
compliance. It is therefore reasonable and important that the
metrics include those agencies that meet less than 100% to provide
a more accurate assessment of the training and technical
assistance needs of eligible entities and to track CAA’s
progress in meeting the Standards over time. Requiring that only
100% compliance be reported undermines the intention of the
standards to promote excellence and is of limited to utility for
the CAAs, State lead agencies and NASCSP and OCS.
2.
Module 3: Make the reporting of community level data on social
indicators optional and provide flexibility in defining which
indicators to use and report. The inclusion of Community Outcome
data with percentages or rates provides little meaningful data on
the initiatives undertaken by local agencies. Their inclusion is
inappropriate and will provide meaningless data when aggregated
beyond individual initiatives.
3. Module 3: Collective
Impact is only one of a variety of options for community-level
work and should therefore be removed. As one in a menu of options,
collective Impact should be removed as including it suggests a
strong preference for this approach as opposed to other possible
community-level strategies.
4. Module 4: Remove the
Characteristics for NEW Individuals and Households Report. There
is no definition of “New” included here nor is there a
widely accepted standard for identifying a “new”
customer. For example when is a customer who receives multiple
services over many years “New”? And what utility does
collecting this data have for state agencies or OCS when it is
rolled up and aggregated? The purpose identified by OCS for the
collection of this data is to inform the needs assessment and
program development process. Agencies already have a process for
identifying needs and developing services with little to be gained
by expending the time and resources needed to collect this data.
Furthermore, many agencies lack the technical capacity and
resources to collect and analyze this data and many are challenged
by the barriers created by the mandated use of multiple, often
incompatible, reporting systems.
5. Module 4: The
Stability Indicators included here are neither practical nor
useful and should be removed. One option included here uses a tool
for defining financial stability that is based on assumptions
drawn from the general population and is therefore of very limited
utility when applied to the low income people served by CAAs.
Another option duplicates data already collected.
6.
Reinstate the ROMA Goal: Agencies Increase their capacity to
achieve results. This goal reflects the vital role of the national
network of local CAAs and there is little sound reason to remove
it. Agency capacity is critical to the success of the services
provided and should be identified as such in the National Goals.
Inclusion of Agency Capacity is also important to highlight the
ongoing need for training, technical assistance and other capacity
building services for the CAA network.
Question 2: The
accuracy of the agency’s estimate of the burden of the
proposed collection of information.
While this is a
question that can only be answered by individual CAAs, this
Association is concerned about the burden of this enormous
increase in the collection of information with no additional
resources provided to meet this requirement. The increase in the
data elements included in the proposed CSBG Annual Report from 300
to more than 1,000 poses potentially great burdens with no
additional resources. This is particularly problematic given the
challenges already faced by agencies that are required to collect
and report data using multiple, often incompatible data systems.
It should also be noted that the on-line system
described by OCS in the Dear Colleague Letter of June 17, 2016,
may be of benefit to State Lead Agencies but does little to
address the issues that eligible entities face in tracking and
reporting data for multiple agencies using multiple information
and reporting systems.
Question 3: The quality,
utility, and clarity of the information to be collected.
A
number of the elements to be collected are primarily useful as
components of community-wide scorecards and have very limited
utility or are not available for single agency reporting or as
data to describe the work done in the smaller geographic areas
served by local agencies, particularly regarding community-level
work. The use of such data to assess and inform individual agency
outcomes is at best limited and may in fact be misleading if used
to compare agency outcomes with community indicators.
·
Module 3, Section A: The indicators used here are to report
community-level data to States and by States to OCS and will be of
limited utility. The available community-level data often do not
match the geographic area and population targeted by the agency,
for example by census track or age. It is also extremely difficult
to assess the impact of an agency’s intervention as part of
a community-level initiative since many external factors influence
the impact of such an undertaking and could erase or minimize the
impact of community-level initiatives. Evaluating the outcomes of
community-level initiatives requires a formal program evaluation
and cannot be accomplished solely through the use of data
collected in the Annual Report.
· Module 4,
Section A: Characteristics for NEW Individuals and Households -
Data Entry Form should be removed. As noted above the collection
of this information is both difficult and costly and of little use
at the local level. This data does not include information about
the needs and challenges facing “new” customers and
does little to help agencies assess whether they are meeting their
needs. Additionally, this data fails to account for the many “one
time” customers who access services such as LIHEAP and thus
makes interpretation of the data difficult and inaccurate.
Further, as previously noted, because many agencies are required
to use multiple, incompatible data collection and reporting
systems, this will pose a significant burden on agencies with no
resources available to support this work.
Question 4:
Ways to minimize the burden of the collection of information on
respondents, including through the use of automated collection
techniques or other forms of information technology.
As
previously noted, the on-line system referenced by OCS in the June
17, 2016 Dear Colleague Letter will do little to help agencies
track customers, services and outcomes. The online automated
system may be an asset to State Lead Agencies, but offers little
for local Community Action Agencies which will still be required
to track and report data using multiple information management
systems with different, often incompatible requirements. It is
unlikely that the State Lead Agencies will have sufficient
resources to enhance the network’s data collection capacity
nor will the proposed on-line system address the need for improved
information management capacity as required by ROMA Next Gen. To
minimize the burden and maximize the utility of the data
collected, the federal government should assess its data
collection requirements and find a way to use information
management systems that can be integrated to meet the needs of the
multiple government funding and regulatory agencies.
|
|
Comment 119 (TN Department Human
Services ; State in TN)
|
|
Our biggest concern is the
timing of the rollout of the CSBG Annual Report. It is believed
that if all moves forward as expected that the CSBG Annual Report
could be finalized as early as October 2016, but may not be
approved and finalized until mid-winter. This does not allow
sufficient time for our state to train each of our eligible
entities, and for agencies to submit Community Action Plans based
upon these new reporting standards, especially since Tennessee is
one of five (5) states obligated to begin the reporting process
beginning this coming July 2017 (since we operate on a State
Fiscal Year)
October
2016-January 2017 CSBG Annual Report Finalized
January-March 2017 Develop
and provide training to eligible entities
March-April
2017 Eligible entities receive Community Action Plans to
write
May 2017 Eligible entities submit Community
Action Plans for approval
July 2017 Eligible
entities begin tracking outcomes based upon new CSBG Annual
Reporting
As a network we depend upon our national
partners to lead us in providing training and technical
assistance. I am not sure they will be prepared to provide the
training and technical assistance needed in time, nor will we have
the ability to develop quality training given the short
turnaround. In addition, mother nature is our enemy this time of
year (January-March 2017). Besides the fact that we do not have
any funding allocated to providing training and technical
assistance. Our state has a 95% pass through of funds, and it
would take some time to negotiate with eligible entities an
alternative funding schedule to support funding for the required
training and technical assistance.
On top of that we do
not have any kind of data base ourselves. Seventeen (17) of the
twenty (20) agencies use THO Software, one (1) agency uses
CAPTAIN, one (1) agency uses the e-logic system and one (1) agency
uses excel like us. At this time, without knowing the database
requirements established through ACF and OLDC the burden is
enormous to get all twenty (20) agencies to come to an agreement
on what that system should be, or have the expertise of what that
system should look like, without knowing that the CSBG Annual
Report will be finalized and what those requirements will look
like. Again, the short turnaround is weighing heavily on us. We
don’t want to develop a system that ultimately has to be
scrapped because we cannot get the information we need from
eligible entities and we do not have the ability to upload
information into OLDC as requested. Again, we do not have the
funding allocated to establishing a data base. Our state has a 95%
pass through of funds, and it would take time to negotiate with
eligible entities an alternative funding schedule to support
funding for a state(wide) system.
Finally, we have
concerns with the overall burden of time it will take for agencies
to track and report the required information. It already takes us
nearly three months, from January 1-March 31 every year for
agencies to submit their reports (due December 30), our department
to input the data into the database, and work with the agencies to
address any discrepancies. Knowing that the reporting standards
have been expanded I am sure that those estimates would increase,
and not increase…unless we had a data base that was could
pull information directly from the agencies and be reported
directly through OLDC.
|
|
Comment 120 (Tompkins Community
Action ; Eligible Entity/Local Agency in NY)
|
|
Some of the data operations
essential to providing valid answers about the long-term impact of
poverty reduction investments are, unfortunately, not widely used
in either industry or government at this point. Our methods of
achieving change for individuals, families and communities depend
on integrating and sequencing different services and investments.
The funders of those investments, whether public or private,
typically have their own customized reporting system, including
their own customized data requirements and identifiers for
participants. We, CAAs are positioned at the intersection of many
domestic programs, including state and local. Therefore, we need
to coordinate delivery in a way that is both rational and
effective. Every data requirement attached to investments a CAA
administers must be reconciled with the local agency and only in
the local agency. We report to numerous funders about the same
participant or project. For us, a minimally funded organization,
CSBG only funds the facility to which they come and staff
salaries; one staff for direct service and the other P/T staffer
for compliance and data management.
Addition to the cost of purchasing and implementing new agency
data systems in the first years, significant additions to the
person-hours now devoted to data management and quality assurance
would be needed indefinitely after implementation. The new system,
though more detailed, labor intensive and costly will not serve
its intended purpose of providing a comprehensive tool for CAA
management.
As far as ROMA NG and increased reporting,
the data collection, reporting, staff time, software
upgrades/purchases/training/etc. with limited funds hence limited
staff is not a reasonable expectation. Services associated with
CSBG funds in our agency is a very small portion of our agency’s
overall operations. To require other non-CSBG programs/staff to
increase their time in data collection/reporting is unreasonable
since these programs often face their own challenges with staff
time. Also, the families we serve are faced with being required to
provide information which may not be necessary for the service in
which they are engaged. Furthermore, there will be inconsistent
data collection based on different data collection systems not to
mention problems associated with translating information into
percentages. This is only a number and if not provided with
context can be a very misleading representation of our community
and our work in serving the community. I agree we need to tell our
story, which is unique to each agency, but numbers without
analysis/context or skewed and then possibly use to compare agency
performance one against another, will undermine our work. The
difficulties of implementing an integrated local system for a
multi-program, multi-dimensional approach to ending poverty
appears to have been under estimated in the proposed
implementation plan. ROMA Next Gen requires that each individual’s
file stores data on the dates and nature of services in the format
required by the CSBG report as well as required by all the other
funders - integrating this information requires software and
operations that select information in different funder databases
and transfer it to a central file. Such software operations are
generally known as data warehouse management, which involves a
higher level of IT skill than we possess.
It would be
very helpful if HHS were to devote some of its Medicaid-funded
technical resources to designing data warehousing for all programs
that coordinate multiple resources toward a single purpose. The
cost-effective technology needed to produce valid data tracking
participants’ progress, to compare participant situations at
different points in time; to integrate unduplicated information
across multiple reporting systems and to answer the important
question of how Community Action programs and projects impact
community poverty over time does not exist. Using CSBG dollars for
creating and sustaining this infrastructure, rather than using it
to fund unique activities with a direct impact on the development
of opportunities in local communities, would be a mistake. We
believe that, in addition to start-up costs, additional staff time
would be needed to manage data integration and data quality
indefinitely. If data integration capacity were developed, the
functions of training, checking data collection and entry and
ensuring that the system that moves data from the original places
reported by funder to the file of the individual participant
client is functioning as continuous and more complex than the
current system. Even if it were possible and desirable to
implement the proposed changes, the process would involve software
procurement, installation testing and training that cannot
realistically be completed by the proposed October 1, 2017
commencement date; writing an RFP and evaluating bids will
certainly require at least one year. This best case scenario
leaves a very short time frame for winning bidders to produce
software, install it, test it, correct it and train users
I
conclude my comments with proposals for a pared-down version of
the community-level project measures which have been articulated
by our partner, NCAF and I quote;
“We believe
that a new generation of ROMA can and should proceed, and that the
concerns about cost, relevance or efficacy can be resolved by:
1. Reducing the number of required components while
offering all of the many proposed elements in a voluntary CSBG
toolkit;
2. Postponing the implementation of the
newest elements while testing updated complex information systems;
and,
3. Affording far greater flexibility for state
and local determination of the performance indicators that will be
tracked and reported nationwide. Generally, the proposed ROMA NG
system includes reports on indicators of “how” results
are achieved, or outputs. Collecting this data far exceeds the
scope of state and national annual reports on activities and
results. We believe only elements that collect data on a limited
set of robust outcomes; demographics and expenditures should be
required, while the tools for collecting time specific participant
data are postponed until data management software is tested.
|
|
Comment 121 (Total Action For
Progress ; Other/Unknown in VA)
|
|
As mentioned above, TAP is
fully committed to reporting accurate, relevant, and significant
data that informs OCS, Virginia's Office of Volunteer and
Community Services (OVCS), and other funders, clients, board
members, and other stakeholders of the progress TAP is making in
helping individuals, families, and communities become strong and
stable and to flourish. However, TAP has several concerns about
the proposed CSBG Annual Report requirements:
In order
to do this type of data collection and tracking, it will require
newer and different resources and skills than our agency —
and other CAPs — have.
We will need to develop
and maintain a data analysis department whose sole focus is
tracking, maintaining, analyzing, reporting and ensuring the
quality of the data entered.
CSBG is such a small piece
of our budget — our programs already must report to their
primary funders, many of whom are federal agencies. We are being
asked to coordinate our data reporting when even within
departments of these agencies, there are multiple data systems
which aren't coordinated. O We asks that we be able to use other
reports that federal agencies already
require of us.
O
If this is not possible, we ask that OCS wait until data systems
up to the task have been designed, made available at a cost that
isn't prohibitive, and been tested in a pilot program before
requiring that CAPs use a new system.
The burden of
completing the current funding application and reporting
(particularly the fourth quarter report) is already crippling.
In
order for the Board to be able to ingest, understand, and provide
oversight, they need a report that ALL members of the Board will
be able to use. A snapshot of important outcomes is necessary for
this purpose.
Overall, TAP has the following
recommendations: We recommend that OCS reduce the number of
required components to the most relevant outcomes and outputs and
select performance indicators that are relevant and can be
measured and reported at the state, regional, and local levels.
Further, it is not reasonable to think that CAP agencies will be
able to conform to all the requirements by October 2017. We will
need years to build our capacity to do so, including planning,
budgeting, and training.
2. Is the cost estimate HHS
provided correct?
HHS has estimated that it would take
each sub-grantee (CAP agency) 242 hours at an as yet unspecified
cost to the agency. After careful consideration, TAP estimates
conservatively that it would take at least 5155 hours to fully
comply with the requirements, which is approximately 2473 more
staff hours than it takes to comply with current requirements.
Further, it will cost at least $296,925 to comply with the
reporting requirements, which is more than half of the CSBG
funding that TAP receives annually, funding intended to pay for
services and administrative supports.
While some of the
estimated cost is related to purchase and/or development of a data
management system for the agency, most of the costs would be
ongoing annual expenses and are broken down
below:
ACTIVITY
Amount of time it takes for
staff to collect information not required by other funders
Amount
of time workers will spend entering data or transferring it from
another program's software Amount of time required for each worker
to be trained and refreshed on the skills of data entry or
acquisition Amount of time our team will spend on data quality
assurance, as well as quality assurance on the final report
Amount of time staff
will spend tracking individuals and families, tracking progress of
community projects, and recording the progress in the format
required Amount of time spent tracking volunteers, board members
and their commitments to organizations and projects that are not
run by your Community Action Agency Amount of time spent on new
reports to the state required for each of the 50 organizational
standards, with additional analysis required on any standards that
your agency has not achieved in full
Placement of data
into the state reporting software, including checking and answers
to the narratives required by the state for module 1, as well as
the local information on funds, people and projects Contracted
professional services IT and
COST JUSTIFICATION
and
ESTIMATED HOURS
25 programs x 10 hours
per program x $20/hour (average salary and benefits) 5 minutes per
client served x 5700 clients
25 hours per person x 100
people x 20/hour
Data: 64 hours x $50/hour Final report: 20
hours x
$50/hour
35 hours x 8 staff x 4
quarters x
$50/hour
Salary and benefits for
a full-time staff member
140 hours x $50
40
hours x $50/hour
120 contracted hours
x
TOTAL
COST
5000
28,500
50,000
4200
56,000
50,000
7000
2000
13,200
raining.
Maintaining
the software to keep and integrate mandated data, and to track our
projects and our progress on or additional standards Purchasing
new software Supporting and training staff that use the software
ROMA training — staff time
ROMA training —
training cost
External data acquisition (surveys,
demographics, etc.)
$110/hour
Actual cost
for current software
One-time cost
Trainer
contract
80 hours per staff person x 3 staff x
20/hour
$775 per person to go to the training 26 weeks
x 1 staff x $20/hr. x 40 hours
3100
40,000
10,000
4800
2325
20,800
|
|
Comment 122 (Trehab Community Action
Agency ; Other/Unknown in PA)
|
|
Trehab is a Community Action
Agency that covers 6 counties in Northeast Pennsylvania. Our
Agency is committed to act as a service provider and advocate for
the poor, unemployed, underemployed, elderly and other groups at
risk.
Within the parameter of The Agency budget, the
reporting requirements cause a large; personnel financial burden A
more simplified system could easily relay client reporting
requirements. The time and financial burden take away from efforts
to provide essential services.
The time burden and costs to
implement system estimated annual cost is approx. $108,810 .00.
|
|
Comment 124 (U.S Department Of Health
And Human Service ; Eligible Entity/Local Agency in DC)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility:
The proper performance of
the functions of the eligible entities are currently assessed
annually by our State CSBG Office through the CSBG required onsite
monitoring reviews, CSBG Organizational Standards assessments,
State Accountability Measures, CSBG State Plan and Application
requirements, and monitoring for adherence to the ROMA performance
management cycle of assessment, planning, implementation,
achievement of results and evaluation.
Although some
modules and sections of the proposed CSBG Annual Report would
provide useful or meaningful community, family, and individual
outcomes and information, other areas, as described later in this
document, would require information that is not necessary for the
proper performance of the functions of the eligible entities.
The
accuracy of the agency's estimate of the burden of the proposed
collection of information:
In our opinion, the burden
hours estimated for the State CSBG Office (164 hours) is
significantly underestimated. Our estimation, based on the
processes we conduct in order to complete the current annual
report (CSBG-IS), easily exceeds 270 hours. Those processes
are:
Preparing annual report forms, instructions, and
guidance o (24 hours)
Providing an annual training to
the 17 eligible entities in our state o (4 hours)
Ongoing
training and technical assistance to the 17 eligible entities o
(30 hours) approximately 10 hours a month for 3
months
Individualized training and technical assistance
to new eligible entity staff o (12 hours) 3 staff @ 4
hours
Collecting the data and information from our 17
eligible entities o (16 hours)
Reviewing and conducting
data quality checks on all the data and information submitted by
the 17 eligible entities in order to assure accurate data o (136
hours) 8 hours per agency
Collecting and preparing the
state data and information for the annual report o (8
hours)
Transferring the eligible entities and state
data into the reporting system o (24 hours)
Inputting
our data quality notes into the reporting system o (16
hours)
Given the increase in the amount of data and
information the proposed CSBG Annual Report would require, this
can only add to our burden; and these new requirements come with
no new administrative resources. The following are sections in the
proposed report that contain new reporting requirements. It is
difficult to estimate the required hours for these new
requirements, but they would add burden hours to our estimate
(These new requirements are not included in our estimated burden
hours listed above):
Module 1 -Sections 82, B3, B7 o
Analysis on trends and results from available ROMA data could be a
substantial increase in burden (see B7)
Module I —
Sections G, H, and I
Module 3—
Section
A (Community Level Initiatives in Progress)
Module 3 —
Section A (Community Level Strategies)
Module 4 —
Section A (NEW Characteristics Report)
Module 4 —
Section D (Individual and Family Services)
A number of
State CSBG Offices, including ours, are allotted less than 5% of
their state's annual CSBG allotment for administrative costs, and
do not receive any of the state's CSBG funds for
remainder/discretionary projects. The allotment and availability
of CSBG funds to State CSBG Offices must be considered when
reviewing and deciding on how the new requirements and proposed
changes presented in the proposed report would affect the burden
of State CSBG Offices.
The quality, utility, and clarity of
the information to be collected:
QUALITY
Our
State CSBG Office and our eligible entities agree that the states
and eligible entities should report their performance measures
annually to OCS. We also agree that some of the new requirements
and proposed changes would be improvements to the current CSBG
ROMA performance measurement system. These new requirements and
proposed changes include:
State CSBG Offices reporting
their statewide goals and accomplishments State CSBG Offices
reporting on their CSBG Organizational Standards assessments State
CSBG Offices reporting on their monitoring of eligible
entities
The proposed changes to reporting Eligible
Entity Expenditures, Capacity, and Resources
The new
Community Level Strategies requirements
The proposed
changes to the ALL Characteristics Report
The new
Individual and Family Services requirements
The
proposed changes to the Individual and Family NPls
However,
we also agree that some of the new requirements and proposed
changes would not improve the system, and would likely add to the
burden of State CSBG Offices and eligible entities. Some of those
concerns are:
Summary Analysis (Module 1, Section
B7)
State Linkages and Communications (Module I,
Section G)
State and Eligible Entity Analysis of Data
(Module I, Section 13)
State Feedback on Data
Collection, Analysis, and Reporting (Module I, Section 14)
NEW
Characteristics Report (Module 4, Section A)
The new
requirement that would require eligible entities to report
Community Level Initiative baseline numbers (Module 3, Sections A
and B)
The proposed changes that would require eligible
entities to report their targets for all their NPls (Module 3 and
4)
Our comments and feedback regarding these concerns
can be found on pages 5 and 6, under ADDITIONAL FEEDBACK AROUND
DATA ELEMENTS.
CLARITY
In general, we feel
that the data and information to be collected in the proposed CSBG
Annual Report is mostly clear. Our concerns around clarity are:
Module 3 (Section A) — This is a new section and
it has a number of new data elements and requests information that
is unfamiliar to us. Our State CSBG Office and eligible entities
would need training and technical assistance specific to this
module prior to implementation.
Implementation Timeline
— We understand that full reporting would begin in FY 2018.
However, we understand that State CSBG Offices would be reporting
on some of the new requirements and proposed changes as soon as
March 31, 2017. We do not believe this would give us a sufficient
amount of time to prepare and collect a lot of that
information.
If the proposed CSBG Annual Report is
approved and OCS provides additional guidance, an instruction
manual with definitions, and training and technical assistance
prior to full implementation, there shouldn't be any major issues
around clarity and consistency.
UTILITY
The
sections for collecting and reporting State CSBG Office and
eligible entity performance measures would be useful. However, we
don't believe the proposed CSBG Annual Report would be a quality
tool for eligible entities to use for planning and analysis. Our
eligible entities stated that they use their client and community
needs assessment data, program specific data, and other local data
and information for program planning and implementation; rarely do
they use the data and information from their Community Action NPI
Report when they conduct planning at their agency.
Also,
as stated in question #1 under OCS REQUESTED FEEDBACK, we
currently assess our eligible entities annually through the CSBG
required onsite monitoring reviews, CSBG
Organizational
Standards assessments, State Accountability Measures, CSBG State
Plan and Application requirements, and adherence to the ROMA
performance management cycle of assessment, planning,
implementation, achievement of results and evaluation. Therefore,
to be useful, the proposed report should only be a report for
State CSBG Offices and the eligible entities to report their
performance measures.
Ways to minimize the burden of
the collection of information on respondents, including through
the use of automated collection techniques or other forms of
information technology:
The proposed CSBG Annual
Report should be a report for State CSBG Offices and the eligible
entities to report their performance measures. The report does not
need to be, and will not be used by our eligible entities, as a
planning and analysis tool for continuous quality improvement.
Eligible entities use their client and community needs assessment
data, program specific data, and other local data and information
for program planning and implementation; rarely do they use the
data and information from their Community Action NPIs Report for
planning and analysis. Eligible entities already conduct ROMA
planning and analysis for Eligible entities already conduct ROMA
planning and analysis for continuous quality improvement and are
monitored by our State CSBG Office for those purposes. Therefore,
to minimize burden, we suggest the proposed report focus solely on
collecting meaningful community, family, and individual outcomes
and information from eligible entities, and the necessary
performance measures and information from State CSBG Offices.
|
|
Comment 125 (Vermont Community Action
Partnership ; Other/Unknown in VT)
|
|
First, we want to recognize
the incredible amount of thought, effort, collaboration,
communication, and creativity that went into the development of
this resulting process and document proposed to reflect the next
generation of the ROMA framework. We also want to acknowledge that
a great deal of the proposed process and document is positive,
constructive, and has the potential to be of significant benefit
to the Community Action network and to our efforts to continue
planning, implementing, monitoring, and evaluating the ongoing War
on Poverty and to inform the public about its value and
effectiveness in supporting and lifting up millions of Americans
struggling to survive and create a better life.
That
said, we also feel the need to convey some concerns regarding a
number of aspects of the proposed process and document which we
feel will diminish its effectiveness and actually risk interfering
with the work of our agencies by making the administrative
requirements imposed on us significantly, and unnecessarily, more
burdensome, without commensurate benefits resulting from that
increased burden.
Our concerns are largely focused in
the following 5 key areas:
* Module 1: Reporting on
CSBG Organizational Standards and Technical Assistance Plans /
Quality Improvement Plans for local agencies with unmet standards.
* Module 3: Community Outcome Indicators and
Collective Impact
* Module 4: Characteristics for New
Individuals and Households
* Module 4: Stability
Measures / Indicators
* Modules 3 & 4: ROMA Goals
– Lack of inclusion of an Agency Capacity-Building Goal Q.
1: Whether the proposed collection of information is necessary for
the proper performance of the functions of the agency, including
whether the information shall have practical utility.
While
there are elements of data collection that that may be of value,
there are a number of areas, as indicated below, where the
collection of information does not meet the threshold of necessity
or utility. We have to look at the proposed package as a whole,
and when we view it that way, it is far too burdensome, contains
reporting tools that will provide information of little or no
utility, and is outside the reach of the Community Services Block
Grant (CSBG). However, with some modifications, a system for
collection of information could be developed that has practical
utility and can be used to improve the performance management of
our agencies. We are in substantial agreement with the Community
Action Partnership regarding many of their recommendations for
improving the proposed package, as indicated below, to achieve an
effective level of workability and utility.
1. Module
1: Modify the options State CSBG Lead Agencies and grantees have
when reporting to OCS on CSBG Organizational Standards, to allow
for a broader range of responses. As noted below, there are
significant challenges with the way this information is being
reported that misinterprets and misuses the Organizational
Standards. Expecting constant perfection is unrealistic and does
not have practical utility for either local agencies or the State
CSBG Lead Agency (LAs). As proposed, the single measure the State
Agencies will report on is the number of Eligible Entities (EEs)
that meet 100% of the CSBG Organizational Standards. To make this
measure practical and realistic, and to increase the usefulness of
the information, LAs should have the option to report the number
of agencies at a range of thresholds that still indicate strong
performance as well as allow for a more accurate reflection of the
EEs’ training and technical assistance needs. For example,
reporting the number of local agencies that have met 100%, 90%,
80%, 70%, 60%, 50%, and less than 50% of the standards will allow
for the collection of more accurate information, better tracking
of results over time, and better, more efficient investment of
training and technical assistance resources.
2. Module
4: Remove the Characteristics for New Individuals and Households
Report. This report has very little if any practical utility at
the local level in most cases, and will be a significant net drain
on agency resources, given the time and expense required to create
it at each local agency. This is not necessary for agency
performance (and may in fact contribute to diminishing it), and
the intentions outlined by OCS for its use are not reflective of
how local agencies and states look at their communities to assess
needs, develop programs, and report progress. It also lacks
utility at the state level given that data such as this rolled up
to the state level will often be skewed by large population
centers and will therefore decrease the practical utility of the
data even further. In addition, many agencies, including those in
Vermont, either lack management information systems with the
capacity to disaggregate this type of data from its overall
customer database, or struggle to integrate the required data
given their use of multiple (and often incompatible) reporting
systems required by our disparate funding sources. Moreover, since
our agencies serve many families over time, with no definition of
“new” being provided for review, the CSBG network will
be unable to implement this report consistently. Defining an
individual or family as new becomes challenging when they may
receive different services over many years (e.g. Head Start /
Early HS, child care, afterschool programs, youth services, job
training / readiness, food, fuel, homelessness prevention, small
business support, financial literacy, IDAs, income tax assistance,
senior services, etc.).
3. Module 3: Remove the
Community Outcome Indicators that include percent’s or
rates. Their inclusion is inappropriate and will provide
meaningless or even misleading data when aggregated beyond
individual initiatives. Such data has validity for geographic
areas and populations much larger than those addressed or targeted
by local agency services and initiatives. This validity is lost
when the scale is smaller and generally accepted data comparison
points become unavailable. Having individual agencies report on
rate change at the community level on social indicators does not
provide information that is usable. In fact, it is more likely to
end up being detrimental to our purposes, since it will more often
under-report the actual outcomes and impact on the people the
agencies are actually serving when they are compared to an
inappropriately large community-wide population that most agencies
have no realistic capacity to make a measurable impact on. It is
much more appropriate and useful to give agencies the option of
reporting outcome data and the flexibility to define which
indicators they use.
4. Module 3: Remove Collective
Impact from all reports. Collective Impact is simply one set of
strategies for doing community-level work, and while EEs are
engaged in such efforts, privileging one approach over others,
given the broader range of options for doing community-level work,
is inappropriate and unnecessary. We are suggesting that
Collective Impact is not a worthwhile strategy for many agencies
to engage in. We’re just recognizing the reality that
relatively few of us have sufficient resources and/or are in a
position to mount an effective initiative of that kind;
consequently, making it an expectation in the core reporting tool
will likely create the unfortunate perception that the majority of
agencies are failing to meet a key expectation and thus diminish
the general perception of the entire network’s
effectiveness. This should be removed and instead encouraged as
part of training and technical assistance initiatives.
5.
Module 4: Remove the Stability Indicators. The indicators selected
and included in the clearance package are not helpful or practical
when reporting customers stabilized as a result of the EEs’
work. One option duplicates data already collected and the second
uses a tool that defines financial stability based on a construct
the uses assumptions drawn from the general population, not those
of low-income individuals and families. Because financial
well-being means something very different for the low-income
population in comparison to the general population, use of the
scale will produce data of limited utility and, as with Collective
Impact, may actually distort the perception of our effectiveness.
6. Include a 4th ROMA Goal: Agencies Increase their
Capacity to Achieve Results. Re-instate this current national goal
that addresses the critical role the network of local agencies
serving the vast majority of the country plays in addressing
poverty. The stated reasons for proposing to eliminate this goal
have been inadequate, missing the need to focus efforts and
resources on continuous improvement. Because agency capacity is
critical to the ultimate success of our programs, it is important
to clearly state this objective in the overall National Goals.
This also highlights the broader need of the network for ongoing
training, technical assistance, and general capacity-building
supports.
Q. 2: The accuracy of the agency’s
estimate of the burden of the proposed collection of information.
It has proven to be extremely difficult, if not
impossible, to come up with a realistic estimate of our agencies’
time and cost burden to implement the proposed new ROMA-NG
package, since there is so much of it that is untried and
untested, and largely because the new technology needs and costs –
in agency personnel, IT consultants, software and equipment –
that will be needed to meet the expanded data collection,
analysis, and reporting requirements of ROMA-NG are very
significant and beyond our current capacity to make even a
reasonably accurate rough estimate. What we know for certain is
that, since no additional resources will be provided to EEs to
build our capacity to meet these new and expanded requirements,
the burden will undoubtedly be far greater than most if not all of
we can absorb within our existing infrastructure and resources.
Our concerns are in part based on the following details:
1.
The current IS survey includes around 300 data elements to report.
The proposed CSBG Annual Report includes more than 1,000. This
increased burden of reporting comes with no new resources.
2.
Most if not all of our local agencies lack information management
systems with the capacity to provide the required data; we have
been struggling to find or develop adequate, reliable, and
user-friendly systems just to deal with the current requirements
for generating comprehensive, agency-wide, unduplicated client
data. Based on our collective experience to-date, the data fields
required for ROMA–NG reporting are likely not to match those
currently used by our agencies, and will almost certainly mean
additional significant and complex negotiations, time and costs
associated with integrating our required proprietary data
management systems (e.g., HMIS, Hancock for Weatherization, PROMIS
and other Head Start software, etc.) in order to produce
agency-wide unduplicated data.
3. Despite OCS’s
claim in its June 17th Dear Colleague Letter that the burden for
local agencies will be decreased by the implementation of the
automated Online Data Collection system (OLDC), it’s not at
all clear how that will occur. It may in fact benefit State Lead
Agencies, but will most likely not reduce the time or cost burden
for local agencies. We will still need systems that track
customers, services, and outcomes, and given the specific
requirements of many funding sources (including other federal HHS
divisions), that will mean additional and often separate data
collection systems. To comply with the draft CSBG Annual Report
local agencies will need to modify, expand, and in some cases
replace their existing systems. State LAs do not have the funding
or capacity, for the most part, to create systems or support local
agencies to develop systems that will streamline or decrease the
burden imposed by the proposed package; and what funding they do
have is from the same pool of resources (e.g., CSBG Discretionary
funds) that is now used to support expanded and new, innovative
programs and/or emergency needs of the local agencies, thus
reducing or eliminating current opportunities.
Based
on all of the above concerns, although we are unable to come up
with specific estimates of the additional time and cost that for
our local agency will need to meet the proposed ROMA-NG data
requirements, we feel certain that OCS’s estimated time
burden of 242 hours per local Community Action Agency is
significantly underestimated. We project that at a minimum; the
burden to our local agencies in Vermont would be double or triple
that amount, if not even more. The cost of this time burden is
impossible for us to estimate at this time, but we would like to
note for the record that the additional hours of staff time is
only one element of the additional cost, which will include
significant expenditures for IT consultants, software purchase
and/or modifications, and equipment to enable us to meet the new
requirements.
Q.
3: The quality, utility, and clarity of the information to be
collected.
As noted below in the detailed comments by
module, several elements of data to be collected under this
proposal have no meaning at the state and federal levels, and only
have value, meaning, and usefulness at the local level.
1. Several of the
elements to be collected are not available in smaller geographic
areas and among sub-populations usually targeted by local agencies
in our community work and, if created by local agencies, lack
statistical validity. The social and population level indicators
being requested are not meant for single agency reporting and are
best used as part of a community needs assessment. For example,
comparing overall percentages of community indicators (e.g. the
unemployment rate, violent crime rate) to individual local
agencies’ outcomes is misleading and is likely to have the
unintended consequence of making it look like our services are
ineffective because they’re not producing change on an
impossibly-expected scale.
2. Module 4, Section A:
Characteristics for New Individuals and Households – This
Data Entry Form is an unnecessary document that will not produce
quality data; it will have little usefulness at the local level;
and it lends itself to meaningless and misleading data at state
and national levels; consequently, it should be removed. Because
demographic data alone does not include information about the
needs of new customers, this information fails to help agencies
determine if they are addressing issues identified in their
community assessments. Furthermore, the large number of “one
time” customers in programs such as LIHEAP will skew any
meaningful interpretation of the data. Aside from the challenge of
defining who constitutes a new customer, agencies that use
multiple reporting systems will also have the added, and extremely
challenging, burden of tracking new customers across all programs
and services.
3. Module 3, Section A – Many of
the indicators used here are inappropriate for reporting by local
sub-grantees to States and by States to OCS and will produce data
of limited utility, for a number of reasons. First,
community-level data on indicators is not always available at the
level (e.g. regional or zip code/census tract) or population (e.g.
low-income children aged 0-5) that matches the area and population
targeted by the agency. Second, community-level indicators are
influenced by a wide number of sources that would most likely
overwhelm the impact of even successful community initiatives, a
fact that significantly limits their utility for providing
meaningful information about outcomes that doesn’t
under-report the real impact on the people served by our agencies.
Third, any meaningful data about the outcomes of community-level
initiatives would require a rigorous program evaluation and could
not be determined on the data collected through the Annual Report
alone.
Q. 4: Ways to minimize the burden of the
collection of information on respondents, including through the
use of automated collection techniques or other forms of
information technology.
OCS and other federal agencies
provide funding to EEs that require separate systems and prohibit
the aggregation of data. To minimize the burden on respondents,
DHHS should examine its own practices to ensure that all
DHHS-funded programs in the field can use data collection systems
that can be easily integrated and “talk” to one
another to produce reports that meet the needs of local agencies,
states, and federal agencies. OCS should also seriously consider a
number of options for better supporting local agencies’
efforts to meet the data collection requirements, including:
Providing funding for acquisition, development, and/or
modification of integrated automated data systems; support the
development of a national data management software application
that will be provided to local agencies to use for this purpose;
and/or coordinate the development of uniform specifications for
data management systems that meet the ROMA-NG requirements and
which all software vendors will be required to meet in order for
local agencies to use them for this purpose.
|
|
Comment 127 (Virginia Community
Action Partnership ; State Association in VA)
|
|
1. Whether the proposed
collection of information is necessary for the proper performance
of the functions of the agency, including whether the information
shall have practical utility.
We remain concerned that
there are a number of areas where proposed collection of
information does not meet the standard of either necessity for the
proper performance of agency functions or the standard of
practical utility for the network. Many of these concerns were
raised in comments from Virginia community action agencies in
their preliminary comments, but these suggested changes were not
reflected in the document that we are currently reviewing.
Regarding public community action agencies, many of
these agencies sub-contract their CSBG funding to other service
providers in their communities. In some cases, these subgrantees
may not have the capacity or funds for data collection that would
meet the requirements of ROMA Next Generation. We believe that
this possible challenge should be addressed in the proposed
reporting package.
The ROMA goal that recognized the
importance of agency capacity has not been included in the
proposed CSBG Annual Report formulation. We think that the goal of
increasing agency capacity to achieve results should be addressed
in the Annual Report. Expanding agency capacity is critical to the
success of agency programs and services. Agency capacity is
closely related to the important needs of training, technical
assistance and overall capacity building for the community action
network.
2. The accuracy of the agency’s
estimate of the burden of the proposed collection of information.
OMB review of proposed regulations requires a
systematic identification and weighing of the costs and benefits
of the regulation. This should include the cost of data collection
at each local agency as well as the cost of establishing or
upgrading a data collection and management system at the state
CSBG office level. It would also include a comparison of present
and future costs with the anticipated benefits of the regulation.
Cost-effectiveness analysis also seeks to determine how a given
goal may be achieved at the least cost. Two things seem clear to
us—there has not been a comparison of the current cost
burden of existing ROMA/State Plan requirements so that OMB would
have a baseline to work with, and the costs of compliance with the
proposed protocol will have different costs in different states
and different local agencies. This will largely depend upon the
current state of data collection and management systems at each
agency and in each state. This is a very critical area for local
agencies and state CSBG offices that are dependent upon CSBG core
funding for their operations. I am not aware of any proposal by
the Obama Administration that would provide increased funding to
local agencies or state CSBG offices to help meet these
substantial costs. The Virginia community action network does not
currently have a statewide client data system, and the cost of
establishing such a system for all local agencies will be
considerable. A small group of Virginia local agencies are getting
ready to release an RFP and we will have a better sense of the
cost of this system (both establishing the system and ongoing cost
of administration) when we receive responses to the RFP.
At
this time, we believe that some local agencies lack the management
information systems with the capacity to provide the required
data. The data fields provided for ROMA Next Generation reporting
may not be the same as those used by these agencies at this time.
In addition, agencies with a broad variety of programs and
services use multiple reporting systems (some of which are
incompatible) and the proposed reporting would add significantly
to the time necessary to provide the required data.
The
on-line automated system that will be used by state CSBG offices
may assist State Lead Agencies but most likely will not benefit
local community action agencies in terms of time or ease of
access. Local community action agencies will need systems that
track customers, services, and outcomes. Separate funding sources
(including other U.S. Department of Health and Human Services
funding) require additional and usually separate data collection
systems. State CSBG Offices do not have the funding or capacity
available to assist local agencies in creating systems or to
support local agencies to develop systems that will ease the
burden of time and cost expected to meet the proposed protocol.
4. Ways to minimize the burden of the collection of
information on respondents, including through the use of automated
collection techniques or other forms of information technology.
The only way that the burden could be minimized on
local community action agencies is for funding assistance to be
provided to local agencies to assist in the development of
client-based data management systems for each agency. New
state-level systems will not address the needs of local agencies
and state agencies do not have the funding or capacity to assist
agencies in developing these data systems.
Federal
Office of Community Services and other federal government agencies
could assist in easing the burden on local community action
agencies by ensuring that all federally-funded programs may use
data collection systems that may be easily integrated so that
local community action agencies, state CSBG Offices and federal
OCS may more easily produce reports that meet the needs of all
parties. Having separate systems and not allowing aggregation of
data is a major burden, particularly for community action agencies
which generally have many programs and services dependent upon
different sources/streams of federal, state and local government
funding.
|
|
Comment 128 (Washington-Morgan
Counties ; Eligible Entity/Local Agency in WA)
|
|
We are a rural CAP in
Appalachian Ohio our estimate of the additional cost to collect
the information required by the purposed ROMA next Gen. is
$40,000-$45,000. This represents approximately 16% of our CSBG
allocation.
|
|
Comment 129 (Western New York
Community Action Agency; Eligible Entity/Local Agency in NY)
|
|
I am the Executive Director of
a Western New York Community Action Agency. We are a very diverse
agency that serves Chautauqua County in all services and for
limited services: we work in three states and multiple counties in
those states. We have a customized database that tracks and
measures customer progress across ROMA based continuums of care.
All of our customers are risk assessed and receive comprehensive
services aimed at financial independence and economic security.
Our services are completely integrated and our agency is providing
service at the lowest possible overhead cost through maximum
efficiency and capitalizing on the umbrella of the community
action statute.
With all that being said, I must state
that the requirements being suggested under ROMA Next Generation
are arduous for the local CAA and costly. The amount of material
being collected annually is overwhelming and often greater than
HHS former audits that occurred once every three years. In
addition to collecting this material, mailing it we would still
has quarterly visits that require almost one full day a quarter
for various senior level managers and the Executive Director. The
amount of money received through the CSBG block grant, though
critical in its flexibility and responsiveness to immediate local
need; is less than one percent of our total budget.
In
addition to the time and cost factors of reporting and tracking,
the outcomes that are being proposed will lose value by collapsing
work done into very broad categories. The value of and impact of
local response to need is a critical measure for congress to weigh
as they make appropriations. It is often the key measure that our
congressmen and women are most concerned with.
|
|
Comment 133 (Wsos Community Action;
Eligible Entity/Local Agency in OH)
|
|
We have estimated our agency
costs for ROMA Next Gen implementation to be $65,000 per year, at
a minimum. This equates to 16% of our annual $400,000 CSBG
allocation.
This figure assumes
that our state CSBG office (Ohio Development Services Agency) will
take the responsibility for development of the electronic data
tracking system that will be needed for reporting purposes,
including bearing the costs to do so.
|
|
Comment 134 (York County Community
Action ; Eligible Entity/Local Agency in ME)
|
|
The proposed collection of
information is seen as overly burdensome; however, we are in
agreement that a stronger focus on outcomes rather than outputs is
a desirable goal. To meet that goal, we recommend the following:
The agency’s estimated time burden of 242 hours
per sub-grantee (local Community Action Agency) is underestimated.
Based on our calculations, the burden to our local agency would be
650 hours. The cost of this time burden is estimated to be
$35,000.
While we greatly appreciate the idea of having
an online automated system for use by states, the system will
likely benefit the State Lead Agency and not the sub-grantee
(Community Action Agency). We will still need databases that track
clients, services, outcomes, and many funders require separate
data collection systems. Any new online system may in fact lead to
added work hours.
|
|