Baldrige National Quality Program Research: Phase I Survey Supporting Statement

OMB Control No. 0693-0031 – NIST Generic Request for Customer Service-Related Data Collections

BALDRIGE NATIONAL QUALITY PROGRAM RESEARCH: PHASE I SURVEY

FOUR STANDARD SURVEY QUESTIONS

1. Explain who will be surveyed and why the group is appropriate to survey.

In order to revitalize the Baldrige brand, it is necessary to collect the currently held perceptions and attitudes of people for whom the brand is most likely to be relevant. Because the Baldrige Performance Excellence Program and Award are focused on the quality and performance excellence in a variety of types of businesses and organizations, it is necessary to gather information from individuals who are responsible for these dimensions in their companies and organizations. Participants will be decision-makers at different levels in their organization who have some quality or performance improvement responsibilities: mid-level or senior managers, senior executives, school superintendents and principals, hospital administrators, etc. This data collection consists of a Phase I survey conducted online, followed by a series of in-depth telephone interviews, and a Phase II survey conducted online, with people participating in no more than a single research session. This document covers the Phase I survey.

In Phase I, the sample will be equally split to represent n=300 current or lapsed customers and n=300 potential customers or prospects, which will allow us to better understand the feelings of those who have had past contact with Baldrige (and will have more familiarity with the organization) versus those who have never participated in the Baldrige programs or taken advantage of the products or services it offers. Current customers are defined as those who have used any Baldrige services in the past two years, while lapsed customers are defined as those who have used any Baldrige services more than two years ago (i.e., applied for or won award, used Criteria, downloaded website materials, served as a Baldrige Examiner, attended Baldrige events or Quest for Excellence Conference). Potential customers are those who may or may not be aware of Baldrige, but have not been involved in the Award process or used the materials offered by Baldrige.

A natural fall-out of n=300 Baldrige customers, either current or lapsed, will be obtained, meaning there will be no forced quotas for each group. Respondents will be surveyed from companies in varied industries, including manufacturing, small business, health care, services, education, and nonprofit. A mix of for-profit businesses and non-profit or government organizations will also be obtained during the research process. Aeffect, the marketing research firm contracted for this study, will also obtain a mix of respondents by business size: large (500+ employees), mid-size (100-499 employees), and small (under 100 employees). By obtaining the views of current customers, lapsed customers and potential customers across industries and company size, our analysis and recommendations will be based on feedback from a representative mix of key audience members.

The same sample criteria apply to all phases of this research. This audience is appropriate to survey because all have had at least some experience with making quality and performance decisions at their company. Therefore, they are qualified to answer questions about quality and performance excellence. It is also important that we obtain a mix of respondents from different industry and business types. This effort will reduce bias so that our sample will represent different groups of current and potential customers who are providing valuable input.

2. Explain how the survey was developed including consultation with interested parties, pre-testing, and responses to suggestions for improvement.

The survey was developed after a thorough review of the Baldrige website, Baldrige Criteria for Performance Excellence, and previous Baldrige research studies. We also included lines of questions used in previous Aeffect research, some of which have already gone through the OMB clearance process for other clients. In addition, a kick-off meeting (conference call) with Aeffect and Baldrige Program officials at NIST was held to garner a better understanding of the research objectives, to review past research materials (in order to leverage previously acquired learning), and to discuss and review the project scope, key phases of the study, and the purpose of each phase. Aeffect then submitted a draft of the Phase I online survey (along with materials for the other phases) to the Baldrige Program.

All NIST feedback was incorporated into the revised version of the test material, which will be pre-tested once the study is approved by OMB, via a “soft launch” to the Phase I online survey. The soft launch means that Aeffect will e-mail invitations to a portion of the sample in order to obtain about n=30 completed surveys. The soft launch allows Aeffect to test for any programming or other unforeseen issues, so that they can be fixed before sending the survey to the larger intended audience.

3. Explain how the survey will be conducted, how customers will be sampled if fewer than all customers will be surveyed, expected response rate, and actions your agency plans to take to improve the response rate.

This research consists of an online survey that will be used to collect perceptions and feelings related to the Baldrige brand as well as insights on the customers’ needs and requirements. This information will be used to develop branding materials, namely a series of positioning statements for the brand, as well as to help the program develop more effective products and services.

Care is taken to ensure that the survey can be completed easily and quickly by participants. The survey is timed repeatedly so we understand how long it will take the average person to complete the task. Our goal is to collect quality information without fatiguing the respondent. We also look for ways to present the questions and response options in a manner that is easy to understand. This effort includes using language that the participants understand and that they consider their own.

In addition, the online methodology has been selected in this phase to collect the bulk of the feedback because it makes it easier for people to participate. They will be able to complete the survey when it is convenient for them to do so, according to their own personal schedule, either at work or at home, early or late in the day, etc. Further, this audience of businesspeople and leaders in various types of organizations is extremely likely to have online access at both work and at home, so no one will be left out due to a lack of technology.

Data Collection

All of Aeffect’s data collection and sample management occurs in-house and is managed by an internal Information Technology staff. Aeffect has developed an email cover letter to invite business decision-makers to participate. This letter is included on the first page of the Phase I survey submitted in the OMB package. The NIST logo will be embedded in the invitation to the customer sample list to aid participation. The email will contain a link to the online survey being hosted on Aeffect servers. After reaching the initial survey web page, respondents will be asked to verify their email address and/or provide a password to ensure non-duplicate respondents and then be screened to confirm that they are a senior executive, department head, or decision-maker in their company. Once these screening criteria have been met, respondents will begin the actual survey. Data collection is anticipated to take about 2-3 weeks. All quotas, such as customer and industry type and business size, will be monitored during data collection.

Sample

In the first phase of research, Aeffect will conduct an Internet survey with the Baldrige target audience, with a total sample of n=600 respondents. The sample will be equally split to represent n=300 current or lapsed customers and n=300 potential customers or prospects. At the 95% confidence level, a sample size of n=600 yields a maximum margin of error of +/-4%, which is appropriate for a study of this nature. A natural fall-out of customers, current users versus lapsed users, will be obtained. Data can later be weighted, if needed, to equally represent both user groups in the sub-sample. According to NIST, the database of current and lapsed customers includes several thousand individuals from organizations throughout the U.S., and would be provided to the contractor. The contractor (Aeffect) will secure the sample for potential customers.

To reach current Baldrige customers, that is, people who have participated in the Award process or have attended a Baldrige conference, Aeffect will utilize the sample list provided by NIST which will include respondent names, e-mails, (some) phone numbers, and addresses of current customers. To obtain customer prospects, that is, people without a direct association with Baldrige, Aeffect will purchase additional sample through an online panel list provider.

Respondents will be surveyed from companies in varied industries, and to obtain a representative mix, quotas will be established by industry type; specifically: Business (n=150), Education (n=150), Health Care (n=150), and Government or Non-Profit (n=150). Aeffect monitors data collection on an on-going basis and programs the software to terminate respondents once a quota has been met. A mix of for-profit businesses and non-profit organizations will be obtained. Aeffect will also obtain a mix of business respondents by number of employees: 33% large (500+ employees), 33% mid-size (100-499 employees), and 33% small (under 100 employees). Participants will be decision-makers at different levels in their organization with some quality and performance improvement responsibilities: mid-level or senior managers, C-level executives, the highest level of management within a company (e.g., Chief Executive Officer (CEO); Chief Operations Officer (COO), or Chief Financial Officer (CFO)); school superintendants and principals; hospital administrators; etc. It should be noted that all participants will come from companies and organizations located in the United States.

The individuals on both the Baldrige customer list and the obtained list of non-customers will be selected randomly to receive an invitation to participate if the entire list is not used in the initial wave of electronic invitations. For the online survey, reminder emails will be sent after the first wave of invitations are delivered in order to give these people the additional opportunity to participate.

The audience of Baldrige customers is specialized in that there is a limited universe of people in the Baldrige database, which consists of people who have participated in the process. Once these people have been approached and “used up,” there is no other sample available to acquire.

Contingency Plans

We plan to send out email invitations in waves rather than send the entire sample at once. If response rates are lower than expected, we will send out additional reminder emails with different copy or appeals or send follow-up waves of invitations earlier than anticipated, as well as purchase additional non-customer sample. The customer definition makes it difficult to acquire customer lists independent of what Baldrige can deliver. However, Baldrige will work with Aeffect to identify other suitable external lists if needed.

4. Describe how the results of the survey will be analyzed and used to generalize the results to the entire customer population.

After the data has been prepared, the task order leader and account executive team at Aeffect will begin analysis by running basic descriptive frequencies. This step will provide percentages of the total respondent sample for all questions contained in the survey. From this output, analysis will reveal the basic makeup of the sample, including break-outs by subgroups. Aeffect uses a proprietary software system (TableIt) to create data tabulation banners that identify statistically significant differences between sub-segments within the data. For example, it is anticipated that responses by each of the sub-segments will be compared, such as current customers versus prospects, or large versus small businesses.

If necessary, the data can be weighted to better represent the entire population. If we find that a group is underrepresented, we can weight the data on that dimension in order to provide a more balanced view of the findings. Aeffect regularly weights the data it collects for government agencies and other clients to match the profile of the U.S. population as a whole. We could weight the Baldrige data to better align with business profiles outlined in U.S. Labor statistics.

Results will be generalized to the target population as a whole, as well as to specific sub-groups. The findings from Phase I will guide us in developing the branding materials (i.e., positioning statements) as well as a detailed set of customer needs and requirements. The branding materials will be evaluated in the Phase II online survey. Ultimately, this research will allow us to select a preferred approach to branding Baldrige and to clearly identify a validated set of customer needs and requirements.

Aeffect will utilize univariate statistics, such as means, standard deviations, etc. to help further profile the composition of the target audiences, including wants, needs, perceptions, and attitudes. Comparative approaches such as testing of sub-group means or proportions for significant differences allow for comparisons between different sub-groups. These sub-groups may include target audiences specified by NIST, comparison of business decision-makers from various U.S. regions, those aware of Baldrige versus those unaware, business type, etc. Aeffect will work closely with NIST to identify any further sub-groups of special interest for analysis.

Aeffect will also employ more advanced multivariate statistics, such as correlation and factor analyses, which allows researchers to examine relationships in data, understanding how one factor may affect or influence another. Similarly, statistical approaches such as regression and CHAID analyses will be employed to understand what brand characteristics are predictive of a consumer’s level of satisfaction or purchase intent. For example, a CHAID analysis may help identify the predictors to enrolling in the Baldrige Program. Classification statistics, such as cluster and discriminant analyses, are used to examine and explore what dimensions can help differentiate one group of consumers from another.

Aeffect will also utilize a perceptual mapping technique to examine brand relationships. Perceptual mapping is a useful tool for demonstrating the similarities and differences between a set of brands. More specifically, it visually depicts how strongly certain brands are associated with various attributes and other brands. In the maps, brands are spaced more closely together when they are perceived to be similar to each other and they are spaced farther apart when there are perceived differences between them. Brands are differentiated on perceptual maps based upon their proximity to the included associations, which are represented by vectors. When the brand’s plot point falls close to the end of a vector, it is highly associated with this characteristic. In addition, the longest vectors represent those measures or dimensions with the greatest differentiating power.

As an example, Aeffect would create a map for NIST that would illustrate how certain brands are perceived in relation to the Baldrige brand, and would allow a visual graphic that offers perspective on brand perceptions and how they relate to different brands. Another analytical technique that will be applied is a brand equity analysis. Aeffect’s brand equity analysis utilizes a technique called LOGIT transformation to assess brand strengths and vulnerabilities versus the competition. This analysis helps to predict how a brand should perform based on how respondents evaluate all brands in the market across key characteristics of the brand. In turn, the actual performance can then be compared against the predicted performance, identifying areas of strength (where a brand outperforms the expectation) as well as areas of vulnerability (where a brand fails to perform up to the level of expectation).

After data collection is complete and the data have been analyzed, Aeffect will submit a summary report of findings to the NIST technical Point of Contact for his or her review. The report will include a description of background and objectives for the project, methodology used, a summary of findings, a detailed findings section with charts and graphs, and conclusions and recommendations. The report will provide NIST with an overview of customer brand perceptions, and customer requirements, needs, and expectations. Aeffect will also provide relevant findings by subgroups, and will work with NIST to determine any other sections to be included.

6

O MB Control No. 0693-0031 – NIST Generic Request for Customer Service-Related Data Collections for Baldrige National Quality Program Research conducted by Aeffect, Inc.