0625-0275 - Supporting Statement Part B for User Satisfaction Impact Needs 09-01-21a

SUPPORTING STATEMENT

U.S. Department of Commerce

International Trade Administration

Surveys for User Satisfaction, Impact, and Needs

OMB Control No. 0625-0275

2. Collections of Information Employing Statistical Methods

   1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

All clients who have received assistance from ITA will have the opportunity to fill out a transactional survey (i.e. comment card).  The transactional comment card survey is not mandatory or predicated on receiving ITA assistance.  The estimated number of clients that will have the opportunity to fill out a transaction comment card survey will be 30,000.  Moreover, the estimated potential respondent universe will be around 6,300 based on a historic average response rate of 21 percent. Please see the table below for respondent universe and historic response rates.

 

Fiscal Year	Surveys Sent	Responses	Percentage
2020	57,541	5,519	10%
2019	15,349	2,399	16%
2018	17,026	2,919	17%
2017	12,277	2,760	22%
2016	10,826	2,218	20%
2015	10,655	2,512	24%
2014	6,030	1,462	24%
2013	5,062	1,528	30%
2012	6,904	1,427	21%
2011	9,107	2,108	23%
2010	20,187	4,758	24%
2009	19,190	4,486	23%
2008	10,878	2,551	23%

In addition, clients will be randomly selected to receive an annual survey.  The sampling methodology will be explained in further detail in question 2 below. The annual survey is also voluntary and sent to an average of 17,583 clients.  The average response rate is 14 percent, based on previous years’ annual survey results. Please see the table below for respondent universe and historic response rates.

Fiscal Year	Surveys Sent	Responses	Percentage
2020	26,619	2,459	9%
2019	27,787	2,279	8%
2018	16,110	1,119	7%
2017	12,338	1,646	13%
2016	15,029	4,057	27%
2015	7,616	1,263	17%%

2. Describe the procedures for the collection of information including:

Only those selected to receive the annual survey will consist of a sample selection. Clients that have received ITA assistance during the fiscal year will be selected to receive the annual survey. The sampling frame will use a random sample at the overall organizational level and then use a stratified, random sample plan that takes into account the relevant components of its clients such as industry, size of business (perhaps number of employees or revenues), and/or geographic location. To provide a simple example, please see Table 1 (page 9). Assume that the relevant components are industry and size. Further, assume that there are four industries and two sizes of business per industry. ITA will need to know what the proportions of these four industries and two business sizes are across its sampling frame. For purposed of this example, assume that Industry 1 represents 50% of the client base and Industries 2, 3, and 4 represent 25%, 15% and 10%, respectively.

Perhaps another relevant indicator of interest may be the size of the business or organization. Size could be an indication of the number of employees a client has, or it could be total revenues, assuming clients are willing to report the number or it is publicly available. ITA will need to know the distribution of business sizes by industry – or if it chooses other variables to segment its clients. For purposes of this simplified example, assume the distribution is as shown in Table 1.

The extent to which ITA will be in a position to enhance the accuracy of the estimates depends on the selection of the segments it defines. Serious consideration needs to be given to a geographic component in the segmentation scheme since ITA manages its accounts and cases from geographically dispersed locations. Externalities unique to geographic areas may have an effect on accounts/cases and should be taken into consideration. In addition, having data elements such as the congressional district identified and associated with each account will facilitate decisions about how to respond to requests for results at the congressional district level.

The sample pool will be randomly selected and stratified using the following parameters: industry, size of business (number of employees or revenues), and geographic location. Each parameter will have equal weight and be proportional to the overall sample population. Additionally, the sampling procedure will adjust for over sampling to ensure that the appropriate segments (industry, business size and geographic location) are statistically represented in the respondent sample size. The annual survey will conclude once the sample size is statistically valid at the 95 percent confidence level.

All other surveys will not be using any sampling procedures and ITA hopes to achieve a 14 percent response rate, which is in line with its historic averages. ITA will be taking the following steps to improve its response rates: reducing the length of the Comment Card and ensuring that all other surveys are as short as possible, revising the content of the messages that will contain the survey links, changing the subject line and sending a pre-survey message prior to the survey being distributed.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

ITA will be committed to employing the following best practices to increase response rates:

• Letting clients know that ITA will be soliciting their feedback at the beginning of a “case” because ITA values their perspective to inform its continuous improvement efforts.

• Engaging the client in determining an appropriate timeframe for capturing their feedback after a meaningful service has been delivered.

• Facilitating the client providing the feedback via a method of their choice – for example, via telephone interview, or email invitation to take a web survey.

• Providing pre-notification that a customer feedback survey is coming – this can be via a phone call (if the client has indicated a preference for doing the telephone interview) and/or via an advance email letting the client know when to expect the invitation and what the SUBJECT of the email will be and what FROM address will be indicated. (In the private sector, this pre-notification has been shown to be effective when it comes from a known or trusted source.)

• Ensuring that the ‘invitation’ language explains why ITA is conducting the survey, why it is important the selected client participate in the survey, how it intends to use the data collected and when the aggregated results will be shared.

• Sending reminders to encourage the client to participate.

• Sharing the aggregated results with all clients – for example, the results can be shared periodically on a website or via an email.

• Communicating changes/improvements that are being made in response to customer feedback surveys.

For the annual survey, the sampling procedure will adjust for over sampling in order to yield "reliable" data that can be generalized to the universe studied. That is, ITA will continue to send additional surveys from its sample pool until the appropriate response rate is attained to be statistically valid at the 95 percent confidence level for the universe studied.

ITA will conduct an analysis of nonresponse bias for the survey and for each important question on the survey. If significant bias is identified, ITA will propose changes to the survey to accommodate this bias and/or correct the results in a systematic manner. However, ITA has not observed any bias in its survey responses and has not conducted a nonresponse bias analysis for this survey. ITA will conduct this analysis as soon as it’s warranted.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

A cross-tabulation and regression analysis will be employed for the annual survey results only. The purpose of this test is to determine drivers of customer satisfaction and will be for internal use only. More specifically, ITA will quantitatively test the questionnaire itself in terms of:

• gaining insight into whether there are apparent relationships between/among satisfaction/use again/recommend questions and the needs and impact questions

• the effectiveness of trying to capture all three types of data (satisfaction/use again/ recommend, needs and impact) in one survey questionnaire

• utility of individual questions,

• understanding if there are topics that customers think should be included that were not in the pilot questionnaire

• obtaining feedback from respondents about the questionnaire experience itself

• gaining understanding about the willingness/ability of respondents to report on impact results

1. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Individuals who will be or have been consulting on the statistical aspects of the design of the advocacy user satisfaction survey, as well as those individuals who will actually collect and/or analyze the information, are:

Joe Carter, International Trade Administration, U.S. Department of Commerce: (303) 844-5656

John Seo, International Trade Administration, U.S. Department of Commerce: (202) 809-5977

LEGAL AUTHORITY: PUBLIC LAW 15 U.S.C. et seq and 15 U.S.C. 171 et seq

Table 1

(NOTE: All data are for illustrative purposes only)

	Industry 1		Industry 2		Industry 3		Industry 4
Proportion of All Clients (A)	50%		25%		15%		10%
Business Size	% Large	% Small	% Large	% Small	% Large	% Small	% Large	% Small
Proportion within the Industry (B)	90	10	80	20	13	87	8	92
Proportion for Stratified, Random Sample (C)	45.00%	5.00%	20.00%	5.00%	1.95%	13.05%	0.80%	9.20%
Proportion of Respondents Reporting Increased Revenues (D)	35.00%	30.00%	40.00%	52.00%	32.00%	18.00%	22.00%	17.00%
Total Amt of Increased Revenues Reported (E)	$110,000,000	$20,000,000	$15,000,000	$8,000,000	$75,000,000	$25,000,000	$8,817,000	$27,000,451	$288,817,451
Total Respondents Reporting Revenue Increases (F)	20	30	19	31	18	32	17	37	204
Average Amt of Increased Revenues for this Segment (G)	$5,500,000	$666,667	$789,474	$258,065	$4,166,667	$781,250	$518,647	$729,742
Total Number of Clients for this Segment or "Cell" (H)	4500	500	2000	500	195	1305	80	920	(S) 10,000
Total Number to Apply Avg to (J)	1575	150	800	260	62	235	18	156
Total Estimated Revenue Increased for Segment (K)	$8,662,500,000	$100,000,000	$631,578,947	$67,096,774	$260,000,000	$183,515,625	$9,128,188	$114,131,636	$10,027,951,171

Calculations in support of Table 1

Total number of clients in sampling frame x proportion industry is of total sampling frame x proportion business size is of the industry = number of clients in that segment (S x A x B = H)

Total amount of Increased Revenue (or number of Jobs Created or number of Jobs Safeguarded) reported / number or respondents reporting Increased Revenue (or Jobs Created or Jobs Safeguarded) = average Increased Revenue (or Jobs Created or Jobs Safeguarded) among those reporting Increased Revenue (or Jobs Created reported or Jobs Safeguarded reported)(E/F=G)

Total number of clients for the segment x proportion of reporting Increased Revenues (or Jobs Created or Jobs Safeguarded) = total number to apply the average amount of Increased Revenue (or Jobs Created or Jobs Safeguarded) to (H x D = J)

Average amount of Increased Revenues for this segment (or Jobs Created or Jobs Safeguarded) x total number to apply the average to = total estimated Increased Revenue (or Jobs Created or Jobs Safeguarded) (G x J = K)

Once the total estimated Increased Revenues (or Jobs Created or Jobs Safeguarded) are determined for each segment (or cell), sum those to have Total Estimated Revenue Increased (or Jobs Created or Jobs Safeguarded)

4