OMB PRA PDEO Support Statement 011807

OMB PRA PDEO Support Statement 011807.doc

Public Diplomacy Evaluation Office: Performance Measurement, Evaluation and Public Diplomacy Program Surveys

OMB: 1405-0158

Document [doc]
Download: doc | pdf

SUPPORTING STATEMENT FOR
PAPERWORK REDUCTION ACT SUBMISSION


Public Diplomacy Evaluation Office (PDEO)

Evaluation Survey Question Bank


OMB Control Number: 1405-0158


Introduction


The Department of the State’s (DOS) Public Diplomacy Evaluation Office (PDEO) is requesting a revision of the currently approved generic clearance for performance measurement, evaluation and customer satisfaction surveys. Included in this request is an expanded collection of questions designed to measure and evaluate the performance of programs, products and services provided by the Bureau of Educational and Cultural Affairs (ECA), the Bureau of International Information Programs (IIP), and the Office of the Under Secretary for Public Diplomacy and Public Affairs, Office of Policy, Planning and Resources (R/PPR). PDEO combines the evaluation and performance measurement staff’s of ECA, IIP and R/PPR.


The revision as requested under PDEO will cover the evaluation and performance research studies that will be conducted by ECA, IIP and R/PPR. PDEO’s mandate is to evaluate individually, all major public diplomacy programs, products and services, and educational and cultural exchange programs. As part of PDEO’s application for revision of the generic clearance previously granted to ECA, (OMB control no. 1404-0158), PDEO is submitting a question bank, or a compendium of questions developed by PDEO to collect evaluation and performance measurement data and information.


In this proposal for a revision of the information collection, PDEO requests that OMB review the procedures and questions for these surveys as a program. Under the procedures proposed here, PDEO will conduct the necessary quality control, including assurances that the individual survey instruments comport with the guidelines of the OMB clearance.

A. Justification

1. The programmatic clearance enables PDEO to conduct evaluations and performance measurement research through survey data collection efforts. This information is being collected to improve the services and products that DOS provides to the public and thus, better carry out part of its mission. The data captured has also been used to help DOS and Public Diplomacy successfully meet organizational performance and accountability goals through the following three legislative mandates:


  • Government Performance and Results Act of 1993 (GPRA), further information can be found at:

http://www.whitehouse.gov/omb/mgmt-gpra/gplaw2m.html;




This information collection permits PDEO to use performance measurement and evaluation data to assess more effectively Public Diplomacy (PD) programming, products and services. The generic clearance re-enforces programmatic compliance with established reporting mandates while ensuring the best value for public resources spent on PD programs. PDEO’s evaluation and measurement efforts provide methodologically rigorous data collection and analyses in place of more subjective, ad hoc, non-standardized anecdotal material.


2. The primary purposes of this generic clearance for information collection is to 1) have a question bank that contains a set of pre-approved questions. These questions emphasize attitudinal and behavior change along with customer satisfaction scales for use on PDEO surveys that have already been cleared by OMB, and 2) include PDEO’s ability to directly assess and measure its customer/respondent base for participation in, and satisfaction with ECA, IIP and R/PPR programs, products and services; their experience and accomplishments during or since participation; and their preferences for existing and future programming, products and services. The information collected also informs and supports budgeting, program management and design, program planning, results reporting, information dissemination and outreach initiatives.


Data collected through the generic clearance will be derived from customer/respondent surveys. The data collection instruments are designed to assess participant/respondent satisfaction and to determine the overall effectiveness of ECA, IIP and R/PPR, programs, products and services in meeting GPRA, PMA and PART legislative requirements. The clearance will cover revision, and program-based evaluations and performance measurement research for ECA exchanges and PD programs, products and services. The customer/respondent-base (or target audience) is limited primarily to participants and alumni of ECA exchange’s and PD programs, users of PD products and services, and to a lesser extent, U.S. and foreign host families, institutions and program administrators in ECA, IIP and R/PPR.


Each PDEO evaluation incorporates general consistency and comparability in research questions and methodological approaches. Because each evaluation relates to a different ECA exchange or PD program, data collection instruments vary as necessary in intent and response choices.



3. More than 95% of the data collection conducted by PDEO uses electronic collection techniques. Technology is used in nearly every survey in which safety, security, programmatic, cultural or political concerns are not of sufficient magnitude to pose a negative impact on the respondent. Survey instruments are distributed via web-based or e-mail technology in PDF format, allowing the respondent to complete the survey and return it anytime during the survey period.


4. PDEO is responsible for the collection, analysis and reporting of performance and evaluation data needed by ECA, IIP and R/PPR to fulfill the requirements of the Department’s annual strategic planning process and annual Congressional budget exercises as part of the GPRA, PART and PMA mandates. By overseeing and coordinating evaluation and measurement research efforts of three evaluation staffs (ECA/P, IIP/P and R/PPR), PDEO helps to avoid duplication and over-surveying of respondents while minimizing data collection costs. The information captured, analyzed and reported by PDEO is not available through any other office at the Department of State.


5. Information collected under this information collection will have no impact on small businesses.


6. If the information is not collected, the Bureau, and the Department of State, will be unable to document the effectiveness, impacts or outcomes of its vital public diplomacy works or exchange programs. In addition, ECA, IIP and R/PPR will not be able to meet accountability requirements or to assess the degree to which programs are meeting their goals. Moreover, the Department will be unable to comply fully with its congressional and executive mandates, including OMB’s mandate to evaluate and report the results of exchange and public diplomacy programs.


7. There are no special circumstances associated with this collection.


8. PDEO occasionally consults with external organizations about survey design, methodology, analysis, data collection approaches and sampling frames. The vendors that PDEO has consulted with include, SRI, Aguirre International, American Institutes for Research (AIR), ForeSee Results, T.E. Systems Inc., and Macro International. These organizations are available for future consultations when needed. A 60-day notice was published in the Federal Register on 12/6/2007. No public comments were received.


9. PDEO does not provide payments or gifts to respondents.

10. PDEO and its external contractors follow all procedures and policies stipulated under the Privacy Act of 1974 to guarantee the privacy of the respondents.

11. Questions are carefully composed and structured to avoid being sensitive or offensive to respondents. On occasion, however, respondents may be asked about their religious or ethnic affiliation, which are optional survey items. This type of sensitive information will be used to provide only demographic output data.

12. It is estimated that the annual hour burden for respondents will be 4,188.3 hours for 8,377 respondents. For the three year period of the generic clearance request the burden will be 12,565 hours for 25,131 respondents.



ITEM


ANUAL TOTAL


3 YEAR TOTAL


Estimated Number of Respondents


8,377


25,131


Estimated Number of Responses


8,377


25,131


Average Hours Per Response1


30 Minutes


30 Minutes


Total Hours


4,188.33


12,565


13. There are no costs incurred by respondents.

14. It costs PDEO (ECA, R/PPR, IIP) $1.1 million in fiscal year 2007 to capture evaluation and performance measurement data needed to conduct its research studies.  Included are estimates of staff labor and research expenditures.


Nine (9) Staff Members (including FTE and Contract Labor) $ 700,000

Research Expenditures (survey translation, data collection, printing, software licenses) $ 400,000

Total $1,100,000

15. One reason for requesting a revision to the ECA generic clearance is ECA’s incorporation into PDEO and the need to convert from the standard OMB clearance to a generic clearance. PDEO is expanding its efforts to systematically collect performance data for analysis and reporting to program managers, senior management, OMB, Congress and other stakeholders. Burden totals will now be calculated on a triennial generic collection basis.


The approved question bank will be a collection of questions that have been pulled together for use on PDEO surveys. The benefits of having a pre-approved question bank will allow PDEO to draft surveys using those questions which will allow surveys to be administered in a more timely and expedited manner (without having to constantly go to OMB for approval), thus, increasing the efficiency of PDEO data collection efforts while also minimizing human and time resources constantly used to approve these surveys on an individual basis.



16. PDEO is responsible for all publications associated with its evaluation and performance measurement efforts, and are considered to be the sole property of the Department of State. PDEO contractors are forbidden contractually from publishing results unless specifically granted an exemption by PDEO. PDEO will not publish this information for public use; however, some aggregate data is made available through OMB’s ExpectMore.gov official website.


17. PDEO will display the OMB expiration date.

18. There are no exceptions requested for this collection.






Section B:


Collections of Information Employing Statistical Methods


1. Respondent Universe


Every effort is made to ensure that the respondent universe will be kept to those individuals that have been identified as appropriate respondents.



PDEO gathers information from participants in ECA and PD programs, selected users of PD products and services, and others engaged in DOS efforts. As a result, the potential respondent universe for a given PDEO assessment, performance measurement, satisfaction study, or evaluation is highly dependent upon the nature and purpose of the specific project.

The size of the respondent universe in past PDEO evaluations, performance measurement and management studies have varied from program to program and project to project, ranging from the teen’s in a management survey, to the thousands (1,000-3,000), for multi-year assessments. Currently, the majority of PDEO evaluations are conducted as a census (all participants surveyed) increasing the size of the respondent universe. In the future we hope to increase our use of sampling methodologies as a way to reduce the number of individuals contacted, the number of responses received, and the respondent burden associated with our survey research. As we move forward, PDEO will investigate the relationship between the size of the respondent universe/population surveyed and response rates.



Response Rates


Response rates for our performance measurement and evaluation studies ranged 18% to 100% for a questionnaire delivered to a focus group.


A factor affecting survey response rates for many of our performance measurement studies was the touch point (pre-program, post-program and follow-up). Survey response rates in 2007, for our performance measurement research, were closely associated with the survey touch point. Pre-program and post-program surveys had substantiality higher response rates than did the follow-up survey. Follow-up surveys are typically conducted 8-10 months after a program ends. The follow-up surveys had an average response rate that was significantly lower than the pre-program and post-program surveys.


Provided in the Table I, are examples of response rates from studies conducted by PDEO in 2007. Included is information on the size of the respondent universe, the actual number of respondents, target audience, data collection methodology, research initiative and method of administration:











Table I


Representative SAMPLE of RESPONSE RATES by SURVEY TYPE,

Method of administration or Research inititative



Name of Study


Potential Respondent Universe



Number of Respondents



Response Rates



Respondent

Selection




Audience


Method of Administration /Research Initiative


Evaluation


International Visitors Leadership Program




95




61




64%



Census



Program

Participants




Evaluation/Paper


Performance Measurement/ Management



Foreign Press Center

(Management)







2000









364









18.2%








Census




Domestic/ Employees








Electronic


Performance Measurement


National Security Language Initiative

(Pre-program)


Foreign Leaders

Exchange Program

(Post-program)




Study of the U.S.

(Follow-Up)






132






1173





149





122






903





70





92.4%






76.9%





46.9%





Census






Census





Census





Domestic/

Program Participants




Foreign/

Program Participants



Foreign/

Program Participants





Electronic






Electronic





Electronic


Public Measurement Data Collection Project

(PMDCP)


541


541


100%


Sampling


Selected Countries Participants


Face-Face /Telephone






  1. Procedures for the Collection of Information


The majority of performance measurement studies conducted by PDEO are census. However, in the conduct of evaluations, PDEO uses standardized and known sampling methodologies including: random sampling, availability sampling, expert sampling and quota sampling. Table II below provides a heuristic device for mapping the data collection methodologies used in the different research initiatives:












Table II



DATA COLLECTION TOOLS/METHODOLOGIES USED BY PDEO

Research

Initiative

Questionnaire

Observation

Focus Groups

Interviews


Performance Measurement/

Management


X





Evaluations


X


X


X


X



Public Measurement Data Collection Project



X






X



X


It is PDEO’s intention to include sample design in its research studies more frequently as we expand our performance measurement and evaluation efforts.



3. Methods to Maximize Response Rate and Quality of Responses


PDEO pre-tests its data collection instruments, proposed procedures and methods when possible. The purpose is to ensure clarity, brevity, relevance, user-friendliness, understandability, and sensitivity to a respondent’s culture and the political climate in which they live.


When pre-tests are conducted, they use the same methodology and procedures that will be implemented in the actual survey process. These processes include distributing the survey by e-mail or regular mail, conducting focus groups, and meeting with contractors/researchers to go over results and re-visit the instruments content. In all cases, pre-tests have been extremely useful for clarifying instructions and questions, refining the response categories, and adding new questions when necessary. We believe that each of these activities help to achieve higher response rates.


All data collection methods are tailored to fit the prevailing political, cultural, safety, security, and accessibility conditions in each country in which participants are involved in an evaluation or performance measurement study. Initial contact with prospective respondents is conducted through e-mails or letters, and, when possible, telephone calls are also made. Follow-up reminders sent periodically to non-respondents encourage them to respond. In combination with pre-testing, we believe that these efforts stimulate response rates.


4. Test of Procedures or Methods


Currently, there are no tests of procedure or methods in place.


5. Consultations on Statistics


PDEO employs two full time statisticians to provide expertise in sampling, survey methodology and statistical analysis.


1 The time to complete a survey ranges from 15-45 minutes. The averaged estimated time to complete a survey is 30 minutes.


6


File Typeapplication/msword
File TitleSUPPORTING STATEMENT FOR
AuthorCrowleyml
Last Modified ByR. A. Ciupek
File Modified2008-02-12
File Created2008-01-18

© 2024 OMB.report | Privacy Policy