SECTION B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
Universe and Respondent Selection
The Commercial Service wants to capture user satisfaction data for counseling services provided and to respond efficiently and effectively to our clients’ needs and expectations. User populations will have already received the specific counseling service from a domestic International Trade Specialist.
For most survey instruments currently in use, all clients are surveyed. Due to the small number of clients participating in counseling services, all users of counseling services that have three or more counseling sessions recorded in an International Trade Specialist’s client management system within a six month period will receive a user satisfaction survey. This will enable the CS to understand the overall customer satisfaction with services provided to companies in various industries and sectors. There is also limited data for each International Trade Specialist and domestic office so surveying the whole population will enable the Commercial Service to obtain as much feedback as possible. In addition, the CS wants to follow-up with all clients who receive counseling in order to respond to unmet client needs and expectations and to resolve outstanding client issues
However, if response rates are insufficient, the CS has designed sampling methodologies. Where product/service user populations are large and response rates are sufficiently high, CS favors simple random samples without replacement.
In the pilot program, the CS was instructed by OMB to do nonresponse follow-up if the response rate was below 80% and to track the variation in responses for respondents that were contacted in subsequent rounds. The CS conducted nonresponse interviews with non-respondents and determined that a strong similarity existed between the survey results and responses received in subsequent rounds. This suggests that, since the assumption of the data missing at random holds and is unbiased then the data collected by the CS can be collected at a much lower cost and respondent burden then when nonresponse follow-up is employed. The CS determined why respondents did not respond and found that response bias was not apparent; therefore the generalization to the population can be justified. The CS compared data in hand on respondents and nonrespondents and found that there were no significant differences in the composition of respondents versus nonrespondents. The CS also compared characteristics of early respondents with late respondents (e.g. first and second waves in e-mail surveys) and found no substantial differences.
The CS will continue to track nonresponse bias and conduct nonresponse follow-up if the survey response rate is below 80% to find out why people did not respond and determine the extent of response bias. The CS will compare data in hand on respondents and nonrespondents and compare characteristics of early respondents with late respondents. In cases where nonresponse bias is found, the CS will estimate response bias and use the data to the best extent possible. If the examination of the data reveals obvious abnormalities, then the CS will generalize the results to the respondents only. If the examination of the data reveals no obvious abnormalities the CS will assume there is no response bias and will generalize the results to the population.
The estimated potential respondent universe is 4158 companies and we expect that based on past response rates, the counseling session user satisfaction survey will achieve a 41% response rate or approximately 1700 survey responses. Although the CS achieved only a 27% response rate in the pilot program for the counseling session survey, the CS fully expects this rate to increase in the future as was the case for other CS user satisfaction surveys; which rolled out with a 26% response rate in their first year and then increased to above 40% in the second and subsequent years.
Procedures for Collecting Information
Due to the small number of clients participating in counseling services, all users of counseling services who have three or more counseling sessions entered into an International Trade Specialist’s client management system during a six month period will receive a user satisfaction survey. Clients will not receive more than one counseling session survey in a six-month period.
As the purpose of the counseling session survey is to evaluate client perceptions of quality and value with respect to the counseling services provided to US companies and then to interpret the data to make business process improvements, market research states that user satisfaction data with a high level of statistical accuracy (e.g. statistical error less than or equal to +/-5%) is sufficient to implement policy and business process improvements to address customer satisfaction.
Methods to Maximize Response
The data collection method chosen for the counseling session quality assurance survey is an e-mail message delivering a hot link to a web enabled survey – in part, because this allows for minimum expense and should yield more considered answers. If the client does not respond to the survey within two weeks, another e-mail reminder is sent to the client. A number of efforts have already been undertaken to improve response rates. An Internet delivered survey reduces the burden on the respondents, surveys have been shortened, and open-ended questions requiring narrative responses have been reduced. Ongoing review of the survey instruments and policies may result in further refinements and resubmission of instruments. If changes are deemed necessary to improve response rates or more accurately capture client perceptions, these changes may occur within the three-year approval.
To address non-response rate, when the response rate is lower than recommended, the CS will perform telephone interviews of non-respondents to inquire about their user satisfaction and complete the user satisfaction survey by phone. In July 2006, the CS addressed the non-response rate from the ODO Counseling Session Survey conducted in June of 2006 and determined that of the 2929 non-respondents, 186 were due to incorrect e-mails recorded in the International Trade Specialist’s CMS or spam blockers that intercepted the survey before the client received it. The CS then conducted telephone interviews of non-respondents and determined the variance of responses that were contacted in subsequent rounds to be in the normal range. Therefore, the survey results are unbiased and can be generalized to the population.
Testing Procedures
Questionnaire construction is one of the most critical stages in the survey development process so a great deal of thought has gone into the design issues involved (e.g. development and pretesting) regarding the “Counseling Session Quality Assurance Survey.” In order to prevent the resulting data from being seriously misleading, the CRM Unit has worked with Internal Trade Specialists within the organization to make sure that the concepts be clearly defined and survey questions unambiguously phrased. Furthermore, designing a suitable questionnaire requires attention be given to its length and the order in which questions are asked – long questionnaires are apt to induce respondent fatigue and errors arising from inattention, incomplete answers, and refusals. In addition to making sure that the questionnaire is the suitable length, all questions in the survey have been pretested to ensure that they are understood by the respondents and do not adversely affect survey cooperation. The survey was based on a modified version of a proven survey instrument. The core of the survey will entail questions of customer satisfaction and perception of quality of counseling services provided by the CS. CS also tests on external clients whenever appropriate.
Contacts for Statistical Aspects of Data Collection
Individuals who will be or have been consulting on the statistical aspects of the design of the counseling session satisfaction survey, as well as those individuals who will actually collect and/or analyze the information, are:
Suzan Price
Customer Relationship Management Unit
U.S. Commercial Service
U.S. Department of Commerce
(202) 482-6042 fax: (202) 482-2599
Joseph P. Carter
Customer Relationship Management Unit
U.S. Commercial Service
U.S. Department of Commerce
(202) 482-3342 fax: (202) 482-2599
The Burke Institute
Professors Tony Zahorik and Jim Kershaw
805 Central Avenue
Cincinnati, OH 45202
Jon Cioffi, Senior Consultant
CFI Group USA, LLC
625 Avis Drive
Ann Arbor, MI 48108
(734) 623-5424
Bernie Lubran
Federal Consulting Group – American Customer
Satisfaction Index
1700 G Street, NW
Washington, DC 20552
(202) 906-5642
LEGAL AUTHORITY: 15 U.S. C 4724
| File Type | application/msword |
| File Title | SECTION B |
| Author | Jacqueline Harris |
| Last Modified By | Jacqueline Harris |
| File Modified | 2006-11-22 |
| File Created | 2006-11-22 |