Supporting Statement
(Parts A & B)
for
Voluntary Academic and Industry “Partner” Surveys
to Implement Executive Order 12862 and 5 U.S.C. 305
for the Department of Health and Human Services
The Office of Grants and Acquisition Policy and Accountability (OGAPA) which consists of the Division of Acquisition (DA) and the Office of Grants (OG) under the Assistant Secretary for Financial Resources (ASFR), Office of the Secretary, Department of Health and Human Services (DHHS) request that the Office of Management and Budget (OMB) extend its existing approval under Clearance No. 0990-0220 for DHHS to undertake voluntary surveys of DHHS’ partners in academia and industry (e.g., Principal Investigators, business offices, and vendors) for three years. To comply with Executive Order 12862, Setting Customer Service Standards (the EO), DHHS again plans to systematically survey its grant recipients and contractors to compile their evaluations of the Department’s grants and procurement processes, and to improve the way we conduct business with them.
These voluntary surveys will continue to be a collaborative effort, with OGAPA providing leadership, oversight, and a methodology; and the DHHS Operating Divisions (OPDIVs) conducting the surveys for their own operations. Each OPDIV will conduct web-based surveys of its partners to obtain feedback for improving business processes. The grant recipients and contractors to be surveyed are sufficiently familiar with the Department and its OPDIVs to make this feedback extremely useful. These surveys will give OGAPA and each of the OPDIVs an opportunity to understand and evaluate grant and procurement quality standards, as well as to incorporate best industry or public sector standards into OPDIV practices.
As HHS’ functional manager for procurement policy and oversight, DA has given nine decentralized OPDIVs the tools, training, software programs, and guidance they need to compile and examine their own contractor survey data. Further, DA is responsible for: (i) developing the contractor survey instrument; (ii) preparing the web-based notices, alert notices, and other survey reminder notices; (iii) facilitating the conduct of OPDIV surveys; (iv) fostering OPDIV improvement initiatives resulting from the survey efforts; (v) monitoring response rates; (vi) ensuring compliance with the required statistical methodology; and (vii) gauging overall survey administration progress, through regular meetings of the DA’s Acquisition Balanced Scorecard Users Group (consisting of subject matter experts from each OPDIV).
Similarly, OG has assumed the same responsibilities for the grants functional staff in the OPDIVs, with the OG regular Executive Committee on Grants Administration Policy (ECGAP) sub-group meetings providing subject matter experts from each OPDIV.
With this clearly defined methodology, each OPDIV grants and contracting office will: (a) select its own sample; (b) conduct its own web-based surveys using LMI’s electronic survey system; (c) compile its own survey data; and (d) analyze the office-unique survey results.
Essentially, DA’s and OG’s original goals remain the same: to promote this survey effort as a useful self-assessment, self-improvement, and benchmarking tool, while ensuring that data reliability is maintained, by working cooperatively with each of the following eleven OPDIVs: Administration on Aging1, Agency for Children and Families1, Centers for Medicare and Medicaid Services; Program Support Center; Centers for Disease Control and Prevention; Food and Drug Administration; Health Resources and Services Administration; Indian Health Service; National Institutes of Health; Substance Abuse and Mental Health Services Administration; Agency for Healthcare Research and Quality.
To comply with Executive Order 12862, Setting Customer Service Standards (the EO), DHHS again plans to survey its grant recipients and contractors to understand better how they feel about the Department’s grants and procurement processes, and to improve the way we conduct business with them.
In addition to the EO, 5 U.S.C. 305(b), “Systematic agency review of operations” states, “. . . each agency shall review systematically the operations of each of its activities, functions, or organizational units, on a continuing basis.” The purposes of the reviews “. . . include determining the degree of efficiency and economy in the operation of the agency’s activities, functions, or organization units.” These surveys will form a part of that review for acquisition and grants offices within DHHS.
These surveys form an important part of the DHHS Balanced Scorecard (BSC) in these areas. The two surveys (vendors/grant recipients) in each functional area provide data for the four perspectives in the BSC: financial, customer, business processes, and learning and growth. The previous survey data, collected under Clearance No. 0990-0220, were used to gauge the overall health of DHHS’ grants and acquisition processes, as well as to target opportunities for improvement. The results indicated that our contractors and grant recipients are generally satisfied with DHHS’ grants and acquisition processes, but that there is some room for improvement.
Specifically, DHHS and its OPDIVs consider the following three criteria in selecting improvement areas: (1) low performance scores (i.e., performance gap analysis); (2) high importance to management; and (3) likely success (i.e., “low-hanging fruit”). Moreover, we consider the following three factors in developing and implementing our organizational improvement efforts: (1) considering resource availability; (2) forming project improvement teams; and (3) recognizing and rewarding improvement efforts.
The application of the above criteria and factors resulted in the identification of qualitative performance gaps and the selection of appropriate and realistic improvement targets for timeliness, quality, efficiency, and cooperation. This analytical approach culminated in concrete improvements to DHHS’ business processes, such as realigning procurement offices to strengthen partnerships with contractors, streamlining the contract payment process to reduce the time for payment; and expanding post-award orientation meetings to increase productivity. The same methodology will be used in the future to continue organizational momentum for improvement. Further, we will continue to properly use ordinal data and frequency distributions to target improvement efforts and gauge performance trends over time.
The survey information will continue to be used by each OPDlV to improve the efficiency, quality, and timeliness of its processes, as well as to strengthen its partnership with academia and industry. Although the survey instruments are brief—with only basic information requested to measure satisfaction and to obtain feedback on areas that may require improvement—we expect the data, comments, and suggestions offered by our respondents to help improve the performance of OPDIV systems and contain costs. Finally, these surveys will help DHHS comply with the EO and 5 U.S.C. 305.
Under previous data collection efforts, DHHS transitioned from mail surveys to Web-based surveys, which are now the norm. Web-based surveys are easier for our grant recipients and contractors to complete, and they have reduced—and will continue to reduce—the survey administration burdens and costs for our OPDIVs, as well. HHS establishes a unique Internet address for each survey effort. Hard copies of our contractor survey instrument may be found at Enclosures 1 and 2.
We do not have or collect similar information from other sources. DA’s well-established Acquisition Balanced Scorecard Users Group and the similar OG Executive Committee on Grants Administration Policy (ECGAP) serve as independent steering committees to maintain survey administration quality and avoid duplication of effort.
The survey instrument is brief, with only very basic information requested to measure satisfaction and to obtain feedback on areas that may require improvement. To minimize burdens, DHHS has formulated questions simply and directly, used close-ended (not open-ended) questions, made the questionnaire answerable within 15 minutes, grouped questions into categories for ease of response, and pretested the questionnaire to ensure minimal burden. While small entities continue to be important academic and industry partners for our grants and contracting offices, we do not foresee any significant economic impact on them from conducting this survey.
Previous clearances called for acquisition surveys to be completed every 24 to 36 months. Our survey schedule has been revised to provide for less frequent collection of that data, i.e., the surveys will now be done only on a 36-month cycle. The revised schedule allows improvement efforts to be fully implemented before respondents are re-surveyed.
We do not have similar information from other sources. Without this information, we will have no access to the priorities our academic and industry partners place on potential improvement efforts. Absent this data, our OPDIVs would be unable to develop reliable improvement plans for raising satisfaction levels. Only by continuing to collect this data will we be able to identify performance gaps, select improvement targets, streamline our processes, assess improvement efforts, track performance progress and vendor and grant recipient satisfaction, and establish performance benchmarks. Lastly, this data is necessary for compliance with the EO.
We are using a voluntary survey and requesting timely responses, and there are no special circumstances requiring additional justification.
The 60-day Federal Register notice published on March 19, 2010, pg# 13287, vol. 75. There no comments received. In addition, DHHS remains in close contact with industry and the research community to obtain their views on the availability, disclosure, and reporting of survey information. In addition, DHHS has thoroughly pretested the survey instrument with outside potential respondents to ensure that it is clear, reasonable, and free of undue burdens.
Respondents receive no remuneration for completing surveys.
Through DHHS’ survey introduction and the face page of the DHHS survey instrument, each OPDIV grants and contracting office assures survey respondents that their individual responses will not be reported—thus, helping to achieve a high response rate. Response aggregates are adequate for complying with the EO. The surveys are voluntary, and an independent third party, such as the Logistics Management Institute (LMI), administers all web-based surveys. For vendors, background information is limited to business category, the type of product or service offered by the contractor, acquisition method, and the number of years of doing business with us. For grant recipients, background data requests the type of organization, title of respondent, number of competing applications submitted and funded, and number of non-competing continuations submitted.
No sensitive information is requested.
Estimated Annualized Burden Table
Type of Respondent
|
Number of Respondents |
Number of Responses per Respondent |
Average Burden hours per Response |
Total Burden Hours |
Vendors |
1000 |
1 |
12 |
200 |
Grant Recipients |
1667 |
1 |
10 |
278 |
Total |
|
|
|
478 |
For the Acquisition Balanced Scorecard, in line with the actual, previous data collection experience, DHHS expects to conduct a total of nine OPDIV-level surveys over the next 3 years. The total number of vendors to be surveyed is expected to be 1,000, with an average response time of 12 minutes. Given an 80 percent response rate, and an average response time of 12 minutes per respondent, the burden per 12-month cycle is 200 hours. In addition, each respondent is expected to incur an average cost of less than $5.00. The costs to the agency and to the public are considered very low.
B. Estimated Annualized Burden Costs
Type of Respondent
|
Total Burden Hours
|
Hourly Wage Rate
|
Total Respondent Costs
|
Vendors |
200 |
$5.00 |
$1,000.00 |
Grants Recipients |
278 |
$4.25 |
$1,181.50 |
Total |
|
|
$2,181.50 |
None. Since this is an attitudinal survey on partner satisfaction, existing reporting and record keeping practices are more than sufficient. There are no additional records required for this survey, and no record retention effects.
Consistent with our previous data collection effort, ¼ of an existing FTE will continue to be used, per 36-month cycle, to conduct each survey effort (including identifying the universe of potential survey recipients, drawing the sample, compiling and analyzing data, reporting on results, etc). Because we use a web-based survey, no significant printing and mailing costs are associated with each survey effort.
In addition, DHHS expects to incur an average of about $5,000 per survey effort for survey consultant support from the Logistics Management Institute or similar provider—to help us improve the electronic processing and analysis of performance data.
This is an adjustment due to errors both in the respondents estimate and the burden hours in the previous collection. Also, the burden hours have increase by 96 hours.
Because of the sensitivity of the data, each OPDIV collects, analyzes, and maintains its survey data locally. There is no central repository for all OPDIV survey data. Given the limitations of ordinal data, each OPDIV, commonly with contractor support, will analyze the frequency distributions of its survey responses. Those sub-elements showing lower levels of satisfaction will be highlighted for OPDIV management review as prime candidates for our continuous improvement process.
In addition, aggregate survey information will be used to help develop GPRA improvement targets. In addition, each office will issue benchmarking or other reports to both DA/OG and its OPDIV management as survey efforts are conducted and analyzed.
The approval number and expiration date will be placed at the front of the electronic survey or mail survey booklet, as appropriate.
No exceptions are taken to the certification statement.
In line with the previous data collection effort, DHHS expects to conduct web-based surveys on a 36-month cycle. While offices with few members in the population will do a census, larger grants or contracting offices will take a sample of no more than one-half the population to achieve plus or minus 3 percent precision at the 95 percent confidence level; with the largest offices surveying about 1,200 recipients or vendors. All sampling will be based on systematic random sampling—selecting every nth one from an alphabetical list of the population names. These procedures will be repeated for every survey cycle. Stratified sampling will be used whenever answers to survey questions are expected to differ significantly from one stratum to another in the grant recipient or contractor population (e.g., commercial firms vs. non-profit organizations). Each of our OPDIV offices will work with Dr. Dillman’s Tailored Design Method to achieve a high response rate, targeted at 80 percent. Sufficient OPDIV and Departmental resources have been allocated for the efficient and effective management and use of this survey information.
DA’s Acquisition Balanced Scorecard Users Group and OG’s Executive Committee on Grants Administration Policy (ECGAP) will continue to ensure that surveys are conducted in accordance with the required statistical methodology.
In addition, at OMB’s request, we would be glad to provide a copy of LMI’s “Government Manager’s Guide to Satisfaction Surveys and Performance Improvements” [second edition]. It was written by Larry Schwartz –one of DHHS’ survey consultants; and exemplifies the kind and quality of consultant support given to DHHS and its Users Groups. The paper’s guiding principles will help the OPDIVs to conduct the requested contractor and grant recipient surveys.
Each contracting and grants office will continue to follow a hypergeometric distribution by taking a 50 percent sample for populations of 1,000 or fewer, to achieve precision of plus or minus 3 percent at the 95 percent confidence level. A normal distribution would be used for any larger populations. Stratified sampling will be used when needed to ensure a representative sample. However, we believe stratification has limited applicability for our OPDIVs, because survey answers are not expected to differ greatly among population strata. DHHS will call for OPDIV data collection approximately every 36 months to minimize the burden on respondents, obtain timely feedback, address grant recipient or contractor concerns promptly, and encourage continuous improvement efforts.
DHHS and its OPDIV contracting and grants offices will continue to use Dr. Dillman’s Tailored Design Method for web-based surveys, to balance survey costs against the need for adequate response rates. Each of our OPDIVs or its support contractor plans to contact its own survey participants several times to ensure high response rates, targeted at 80 percent. For web-based surveys, our survey consultant—LMI—tracks this information electronically. Survey participants will continue to be contacted as follows:
Alert message, alerting the selected sample to the upcoming survey.
First cover message and survey instrument.
Second cover message and survey instrument.
First reminder and survey instrument.
Second reminder and survey instrument, and
Third reminder and survey instrument.
Post-survey analysis shows that OPDIV response rates under our previous data collection method averaged in the mid-60 percents, and were as high as 83 percent. To help ensure that we receive an 80 percent response rate across the board, we have instituted the following procedures:
(a) We are instructing our OPDIVs to “scrub” their mailing lists on a regular basis, to make them more current, accurate and complete. This will resolve the major obstacle to our achieving higher response rates.
(b) We have added a mandatory email address data element to our Departmental Contract Information System to collect and store email addresses.
Using the two new procedures stated above, we believe that DHHS can meet OMB’s target response rate.
DHHS has pretested its web-based survey instruments to eliminate or revise questions, as necessary. In addition, we used the pretest to add material that the representative respondents strongly believed should be included. Further, DA and OG remain in close contact with industry and the research communities to obtain their views on the availability, disclosure, and reporting of this information. Essentially, DHHS continuously strives to minimize burden on respondents.
We continue to retain the services of the Logistics Management Institute (LMI) to help us properly analyze and use the results of this survey effort. LMI points-of-contact is Isabela Castaneda (703-917-7533).
1 For Grants only. AoA and ACF have no internal contracting office.
| File Type | application/msword |
| File Title | SUPPORTING STATEMENT |
| Author | LMI |
| Last Modified By | DHHS |
| File Modified | 2010-06-30 |
| File Created | 2010-06-28 |