OMB PRA PDEO Support Statement Part B 011807

OMB PRA PDEO Support Statement Part B 011807.doc

Public Diplomacy Evaluation Office: Performance Measurement, Evaluation and Public Diplomacy Program Surveys

OMB: 1405-0158

Document [doc]
Download: doc | pdf

SUPPORTING STATEMENT FOR
PAPERWORK REDUCTION ACT SUBMISSION


Public Diplomacy Evaluation Office (PDEO)

Evaluation Survey Question Bank


OMB Control Number: 1405-0158




Section B:


Collections of Information Employing Statistical Methods


1. Respondent Universe


Every effort is made to ensure that the respondent universe will be kept to those individuals that have been identified as appropriate respondents.



PDEO gathers information from participants in ECA and PD programs, selected users of PD products and services, and others engaged in DOS efforts. As a result, the potential respondent universe for a given PDEO assessment, performance measurement, satisfaction study, or evaluation is highly dependent upon the nature and purpose of the specific project.

The size of the respondent universe in past PDEO evaluations, performance measurement and management studies have varied from program to program and project to project, ranging from the teen’s in a management survey, to the thousands (1,000-3,000), for multi-year assessments. Currently, the majority of PDEO evaluations are conducted as a census (all participants surveyed) increasing the size of the respondent universe. In the future we hope to increase our use of sampling methodologies as a way to reduce the number of individuals contacted, the number of responses received, and the respondent burden associated with our survey research. As we move forward, PDEO will investigate the relationship between the size of the respondent universe/population surveyed and response rates.



Response Rates


Response rates for our performance measurement and evaluation studies ranged 18% to 100% for a questionnaire delivered to a focus group.


A factor affecting survey response rates for many of our performance measurement studies was the touch point (pre-program, post-program and follow-up). Survey response rates in 2007, for our performance measurement research, were closely associated with the survey touch point. Pre-program and post-program surveys had substantiality higher response rates than did the follow-up survey. Follow-up surveys are typically conducted 8-10 months after a program ends. The follow-up surveys had an average response rate that was significantly lower than the pre-program and post-program surveys.


Provided in the Table I, are examples of response rates from studies conducted by PDEO in 2007. Included is information on the size of the respondent universe, the actual number of respondents, target audience, data collection methodology, research initiative and method of administration:











Table I


Representative SAMPLE of RESPONSE RATES by SURVEY TYPE,

Method of administration or Research inititative



Name of Study


Potential Respondent Universe



Number of Respondents



Response Rates



Respondent

Selection




Audience


Method of Administration /Research Initiative


Evaluation


International Visitors Leadership Program




95




61




64%



Census



Program

Participants




Evaluation/Paper


Performance Measurement/ Management



Foreign Press Center

(Management)







2000









364









18.2%








Census




Domestic/ Employees








Electronic


Performance Measurement


National Security Language Initiative

(Pre-program)


Foreign Leaders

Exchange Program

(Post-program)




Study of the U.S.

(Follow-Up)






132






1173





149





122






903





70





92.4%






76.9%





46.9%





Census






Census





Census





Domestic/

Program Participants




Foreign/

Program Participants



Foreign/

Program Participants





Electronic






Electronic





Electronic


Public Measurement Data Collection Project

(PMDCP)


541


541


100%


Sampling


Selected Countries Participants


Face-Face /Telephone






  1. Procedures for the Collection of Information


The majority of performance measurement studies conducted by PDEO are census. However, in the conduct of evaluations, PDEO uses standardized and known sampling methodologies including: random sampling, availability sampling, expert sampling and quota sampling. Table II below provides a heuristic device for mapping the data collection methodologies used in the different research initiatives:












Table II



DATA COLLECTION TOOLS/METHODOLOGIES USED BY PDEO

Research

Initiative

Questionnaire

Observation

Focus Groups

Interviews


Performance Measurement/

Management


X





Evaluations


X


X


X


X



Public Measurement Data Collection Project



X






X



X


It is PDEO’s intention to include sample design in its research studies more frequently as we expand our performance measurement and evaluation efforts.



3. Methods to Maximize Response Rate and Quality of Responses


PDEO pre-tests its data collection instruments, proposed procedures and methods when possible. The purpose is to ensure clarity, brevity, relevance, user-friendliness, understandability, and sensitivity to a respondent’s culture and the political climate in which they live.


When pre-tests are conducted, they use the same methodology and procedures that will be implemented in the actual survey process. These processes include distributing the survey by e-mail or regular mail, conducting focus groups, and meeting with contractors/researchers to go over results and re-visit the instruments content. In all cases, pre-tests have been extremely useful for clarifying instructions and questions, refining the response categories, and adding new questions when necessary. We believe that each of these activities help to achieve higher response rates.


All data collection methods are tailored to fit the prevailing political, cultural, safety, security, and accessibility conditions in each country in which participants are involved in an evaluation or performance measurement study. Initial contact with prospective respondents is conducted through e-mails or letters, and, when possible, telephone calls are also made. Follow-up reminders sent periodically to non-respondents encourage them to respond. In combination with pre-testing, we believe that these efforts stimulate response rates.


4. Test of Procedures or Methods


Currently, there are no tests of procedure or methods in place.


5. Consultations on Statistics


PDEO employs two full time statisticians to provide expertise in sampling, survey methodology and statistical analysis.


3


File Typeapplication/msword
File TitleSUPPORTING STATEMENT FOR
AuthorCrowleyml
Last Modified ByR. A. Ciupek
File Modified2008-02-25
File Created2008-02-25

© 2024 OMB.report | Privacy Policy