Supporting Statement

Supporting Statement for Paperwork Reduction Act Submission

NASA Benchmarking of Program Office Size, Structure, and Performance

1. Justification

A.1 Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.

In 2004, President Bush gave NASA a defining challenge and an opportunity to undertake a new journey of exploration of the solar system, beginning with the return of humans to the Moon by the end of the next decade. The challenge further requires NASA to establish and implement a long range strategy for exploration that is both sustainable and affordable. One of the elements of our response to that challenge is a reexamination of NASA’s approaches to structuring, sizing and managing our programs. A key component of that examination is developing an understanding of how corporate America implements its successful programs.

A.2 Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.

The NASA Administrator has requested that Program Analysis and Evaluation conduct a study to benchmark the size, structure and performance of programs from a diverse set of American corporations and to understand the factors that enable their approach to program management. NASA management may utilize the information and the analysis of information to shape the size and structure of NASA program offices.

This benchmarking is the second part of a two phase research undertaking. In the initial study, NASA civil servants collected information on government programs and two corporations. This information was presented to the Administrator, who requested that we expand the effort to incorporate a significant number of companies that were not primarily serving as federal suppliers. The contention is that NASA’s program offices are significantly larger than corporate programs offices and that the size and nature of our undertakings may not fully justify the larger sized program offices.

The intent of the benchmarking study is to evaluate whether corporate America has a set of better practices that enables them to manage their programs with fewer people, e.g. more efficiently, effectively and less costly than NASA’s historical approaches to program management. If there are opportunities to make appropriate changes in program office structure and size, then NASA is interested in understanding the enabling factors and evaluating whether the approaches can work in a governmental organization. It is crucial that we understand enough about the context of the organizations to determine whether their practices can be effectively executed under the requirements and the constraints of the federal sector. NASA will receive a summary report from the contractor who is tasked to collect and analyze the corporate data.

The Constellation Program Office is the target initial beneficiary for application of executable better practices. The structure and ultimate size of the program office is of concern, given the “go as you pay” approach to implementing the President’s Vision for Space Exploration. In order to fully realize the benefits of this study, it is critical that we have the results available as soon as possible, since the Program Office is up and running and is in the process of establishing its internal processes and procedures.

NASA has contracted with the Human Systems Knowledge Network, Incorporated (HSKNI) to enroll 30 to 50 companies in this NASA sponsored study. On NASA’s behalf, HSKNI will be dealing directly with the corporations and will be fully responsible for managing the security of any intellectual property or corporately sensitive information that might be provided in the course of this benchmarking. Because the information may be corporately sensitive, HSKNI is collecting and handling the individual company’s information. NASA will not receive individually identifiable information on any of the companies, but rather will receive aggregated information from the entire sample. Private companies will, it is hoped, be willing to be more open with the information provided for the study. HSKNI will provide an integrated report to NASA as a contract deliverable, as noted above. This report will also be made available to corporate participants, along with an individual report detailing the company’s relative ranking against the entire sample.

A.3 Describe whether, and to what extent, the collection of information involves the use of automated electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce the burden.

The research plan is that corporate information will be documented through an enrollment process followed by interviews and a facilitated workshop at each of the companies rather than through issuance of a paper or electronic survey.

The enrollment process will identify specific programs, their cost, their staffing, their organization and their performance. If it appears that there are sufficient data for the company information to support the study, a set of artifacts will be collected and analyzed in preparation for the interview/workshop process. During the interviews or facilitated workshops, HSKNI will initiate conversations about: Managing Strategy, Managing Portfolios, Managing Programs, and Managing Projects. While discussing each of these topics, HSKNI will guide the discussions to cover responses on: Environment, Management, Performance, Governance, Decision Making, Risk, Acquiring Resources and Communications.

NASA has provided a full set of the questions/topics to be pursued in the interviews, per your request, and we would be happy to discuss any and all of these with you. The topics were selected to enable us to understand how the companies move program objectives from strategy through portfolios to the program office and then down to projects. In order to do so, conversations will be initiated on how the corporation distributes/assigns functional responsibilities relative to response to the environment, management, performance, governance, etc. As noted in question A.1, the study must produce data on program office size and structure in context so that NASA can evaluate whether the practices and approaches are appropriate or viable in a federal organization.

In this instance questions about the environment, refer to the environment in which the corporation exists, e.g. its external milieu. Presumably, for example, the size and structure of a program office would be influenced by the stability of the corporate milieu and the frequency and extent of changes to which the program must respond. It would also be important to understand whether other levels of the corporate structure (e.g. strategic or portfolio levels) serve to buffer and direct responses to changes or whether the program operates relatively independently in responding to changes, necessitating devotion of internal resources to monitoring and managing interfaces to external entities and the external environment.

HSKNI will record responses (or dialogue) working against a desired set of target information (questions).These questions were developed based on a Project Management Institute Program Management Standard which has recently been approved. [A full set of these has been supplied separately with the initial submission] which they utilize in the scoring for the benchmarking reports. The following questions provide an example of the type of questions that may be used to initiate the conversation on the management of strategic planning: Are strategic priorities clearly articulated? Are processes in place to anticipate and adapt to changes in strategic direction? Through this conversation HSKNI can provide the participant with a ranking based on a mid-point determined from the Project Management Institute Program Management Standard as a reference with the ranges to be established from analysis of the responses and relative to the rest of the participants and they can provide NASA with a set of best practices, organizational and governance constructs, an understanding of how much of the strategic planning and response to change are managed external to the Program Office versus within the Program Office. This information will be of assistance in understanding the functions and size of the Program Offices in Corporations participating in the benchmarking. Similar types of questions will be utilized in initiating conversations in the other areas.

As these interview data are collected, HSKNI will be “building the bench” for the scoring. This is a stepwise process, with the “bench” being created and then modified throughout the analysis of the corporate interview results. HSKNI assigned scores for each of the target questions will range from 5 to 1 (strongest to weakest). A suggestion of the strongest and weakest response follows each question on the itemization of ~ 160 target information/questions that have been supplied to OMB. Each corporation will be provided a report that displays their ranking relative to the remainder of the respondents. The interview data and collected documentation will be analyzed for a relationship to the size and performance data and the relationships will be included in the report. The report will provide both statistical results (the corporate score on the category relative to the minimum, maximum, and mean and the relationships to structure, size, cost and results) and a descriptive comparison of their responses relative to the better practices identified by the entire sample set. HSKNI will be responsible for aggregating the data and performing the statistical analyses for both the individual corporate reports and for the aggregate report being provided to both NASA and the corporations as a contract deliverable.

NASA programs will be benchmarked by a team of NASA employees and these results will then offer a basis of comparison and provide an identification of opportunities for adoption of better practices. In addition, NASA personnel will develop a set of estimating relationships that enable us to relate the key factors driving size and structure with the size and organizational construct. These algorithms (general structure is the logarithmic y = a + b^x) can then be utilized to either estimate future requirements or to measure progress towards implementation of better practices. Because the “bench” is being developed via the planned data collection and the data are being collected as part of a research undertaking, it is difficult to be more specific about the full set of relationships and analytical methods at this time.

Results will be assessed at the mid-point of the study and a decision might be taken to utilize a focused electronic questionnaire covering a small subset of the topics with the remaining corporations.

A.4 Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in question 2 above.

Prior to undertaking this benchmarking, a literature search was employed to determine whether similar study results had been published. The search produced no similar set of study results.

We were unable to locate anything other than single program case studies that dealt with the topics of structuring or sizing program offices. There were a substantial number of studies on structuring and sizing projects, but not on program offices (managing a collection of projects, by NASA’s definition). The Project Management Institute, upon learning of our intent to conduct this study, has enquired about our thoughts about publication of the results, as sizing and structuring of program offices is not a topic that has been widely studied.

While there are studies on the individual topics, neither NASA nor HSKNI have been successful in finding a comprehensive and systematic study relating the identified factors to program office size, structure and performance. The corporation participants who do their own benchmarking have indicated that they are interested in participating based on the systemic approach and the absence of studies of this nature and scope.

A.5 If the collection of information impacts small businesses or other small entities (question 5 of OMB Form 83-I), describe any methods used to minimize burden.

The targeted corporations are all large entities that engage in project management as a primary means of developing their products and services. By definition, small businesses will not be included in the benchmarking.

A.6 Describe the consequence to Federal program or policy activities of the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

The results that will be reported to NASA and the participants of this study will not disclose any identifying information about the individual participants data. NASA will benefit from the identification of corporate best practices and an understanding of the factors needed to enable those practices as it examines the potential of alternative approaches to program management. The benefit to the corporate participants will be the opportunity to receive a report and assessment of how their organization compares to the entire group of participants and to receive a report of the relevant findings that HSKNI will make to NASA upon completing the study.

In addition to the development of relative ranking in the development of the benchmark, NASA will be utilizing standard statistical methods to develop estimating relationships relating key parameters (interfaces, number of functions performed by the program office, etc.) to the program office staffing levels, based on the data provided by the corporations.

As indicated above, the Constellation Program Office within the Exploration Program is the initial target beneficiary for application of executable better practices. In order to fully realize the benefits of this study, it is critical that we have the results available as soon as possible, since the Program Office is up and running and is in the process of establishing its internal processes and procedures. If the study is not conducted, NASA will not be able to benefit from best practices or alternative approaches to sizing and organizing its programs, which will impact efforts to field an affordable and sustainable program office for the Vision for Space Exploration.

A.7 Explain any special circumstances that would cause an information collection to be conducted in a manner: requiring respondents to report information to the agency more often than quarterly and requiring respondents to submit proprietary or confidential information, etc.

The benchmarking is a one-time study effort and will not be repeated. Information will be collected primarily through interviews and facilitated workshops. Participants in the first phase will not be required to complete any surveys or questionnaires. It is possible that a second phase may issue a targeted electronic questionnaire on a subset of the topics being pursued. Again this would be a one-time effort.

Information provided to NASA by the benchmarking contractor will not contain any proprietary or confidential information and will not enable identification of the individual participants.

A.8 If applicable, provide a copy and identify the date, and page number of the publication of the Federal Register of the agency’s notice, required by 5 CFR 1320.8(d), soliciting comments on the information collection prior to submission to OMB.

A Federal Register notice has been filed (Federal Register Vol. 71, No. 245, Thursday December 21, 2006, Page 76701).

Participation in the research is entirely voluntary and the data collection is a one-time event. No comments have been received to date from the Federal Register notice.

A.9 Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

Participants will receive a report of their ranking relative to the remainder of the benchmarking participants.

As indicated in section A.4 above, a systematic and comprehensive study associating program office size, structure, and performance has not previously been conducted. Beyond a literature search, neither HSKNI (consultants specializing in evaluation of corporate project management practices) nor the Project Management Institute could identify any studies involving the comprehensive look at program offices in the context of the entities strategic management, portfolio management, and project management. As stated above, the uniqueness of the research has proved to be a compelling recruitment tool.

A.10 Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.

The contractor may sign confidentiality agreements with the participant, and will be responsible for maintaining security on materials provided by participants. These materials will not be provided to the government.

A.11 Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private.

No topics of sensitive nature will be involved in this benchmarking study.

A.12 Provide estimates of the hour burden of the information collection.

It is estimated that each corporation will contribute 124 man hours to the interviews or one time facilitated workshop. That is assuming 3 respondents on the first day and 12 respondents on the second day, with an assumed 8 hour day and 4 man hours of follow-up. With 30 participant corporations, the total study would consume 3,720 corporate man hours.

A.13 Provide an estimate for the total annual cost burden to respondents or record keepers resulting from the collection of information.

a) If we assume an hourly rate of ~$150 per hour for 124 hours, the total cost per corporation would be approximately $18,600.00. The total cost of the study, with 30 participant corporations would be $558,000.00. It is estimated that each participating company would have to pay approximately $100,000 to obtain a similar benchmarking result, if it were to hire the contractor to perform this work for the company. It would appear that the participating corporations each have a net benefit from participation of ~$81,000.00.

2. The value of the NASA study contract is $425,000. The contract is a fixed price contract to collect, analyze and report results to participants and NASA.

A.14 Provide estimates of annualized costs to the Federal government . Also provide a description of the method used to estimate cost, which should include quantification of hours, operational expenses, and any other expenses that would not have been incurred without this information collection.

The total cost to the government is $425,000, which is accommodated with funds that were appropriated for use in FY06. Civil service salaries supporting this one time study are part of the Program Analysis and Evaluation base. Existing IT resources will be used to house the contractor’s deliverables.

As indicated in preceding sections, the respondents will be provided a copy of the aggregated summary of the results along with the individual report comparing their company to the benchmark. A decision has not been taken about a broader publication of results.

B. Collections of Information Employing Statistical Methods

B.1 Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used.

The minimum number of respondents has been set at 30 to provide a modicum of statistical significance to the findings. No attempt is being made to statistically extend the benchmarking results to a universal population set. By definition, once enlisted, participants will provide a complete set of responses and there has been no further consideration of response rate. A brief discussion of the analysis of the results has been provided in question A.3.

B.2 Describe the procedures for the collection of information

A goal of including a diverse set of companies, including high technology producers, is being accommodated through the recruitment process. It is unknown at this time whether the responses will stratify by business sector or any of the other factors being considered in this one time study. If the data begin to demonstrate stratification, the standard statistical methods will be employed to determine significance of the results and measures.

Given the purpose of this benchmarking study, and the limited number of participants in the benchmarking no problems of specialized sampling procedures or accuracy are anticipated.

B.3. Describe methods to maximize response rates and to deal with issues of non-response.

Since the study is utilizing interviews and workshops as the primary means of collecting data, response rates are not a consideration. Participants will also provide a set of corporate artifacts that will be utilized to validate inputs and assure reliability of the scoring.

B.4. Describe any tests of procedures or methods to be undertaken.

HSKNI has drawn a substantial number of the ~160 target questions from a database of validated and previously tested questions. The analytical methodology has been demonstrated through the creation and use of a Corporate Practices Questionnaire that has been utilized to evaluate project management practices in the private sector.

B.5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The study is being conducted by the Cost Analysis Division of the Office of Program Analysis and Evaluation. The study lead is Johanna A. Gunderson. Ms Gunderson may be reached at 202-358-2517. The contractor that will be conducting the interviews and workshops and analyzing the corporate results is the Human Systems Knowledge Networks, Inc.