Imagine yourself judging a chili cooking contest, with long tables lined with big steaming pots of homemade chili. Given that each pot of chili has been stirred well, you are able to taste just one spoonful to determine how the whole pot tastes. That's because the small sample accurately represents the entire pot. Now imagine if you had to eat half of the pot in order to make the same determination. There's not enough antacid to relieve that heartburn.
A lot of home builders are giving themselves heartburn by insisting on more survey responses when accurate results can be obtained from a much smaller number of returned questionnaires. Making matters worse, there are a number of research companies tying to woo builders away from their current research program by claiming you need 80 percent of surveys filled out.
Surveys: Quantity vs. Quality
Why this overemphasis on quantity instead of quality? Besides lack of knowledge, it could be human nature. No one likes bad news, so there's a natural response not to believe it, and many people will try to explain it away. "How can this be?" they wonder. "Clearly, the survey was only filled out by our dissatisfied customers. All of our happy customers must have thrown it away." Such reactions, however, are nothing more than misleading thoughts by those grasping to make sense of results they prefer not to accept.
A quality survey administered properly will yield accurate results with less than 80 percent response rate. Best of all, the response rate needed to achieve results that are 90 percent accurate is often a lot smaller than you may think.
The key is to give your program time to reach the targeted response rate and you will have the same group as if you received all of the surveys.
If you built more than 125 homes you do not need 80 percent of the surveys returned to accurately gauge your performance in customer satisfaction. That's because the power of randomization and avoiding bias comes into play. Just like that stirred up pot of chili, all you need is a tablespoon to get proper taste, given that the pot is big enough. If the pot were too small you might scoop up a burnt edge that doesn't reflect the true quality of the chili.
Sure, the first surveys to pour in might be from angry customers anxious for an opportunity to vent their frustrations. Given time, however, the total number of returned surveys will be a proportional representation of the entire customer base. The key is to give your program time to reach the targeted response rate and you will have the same group as if you received all of the surveys.
Survey Target Response Rates
Assuming that you are surveying all of your customers, what response rate should you aim for? Generally, as the population size increases, the needed response rate goes down. In our experience, most larger builders can be reasonably confident in the overall company results with just a 20 percent response rate. Though higher response rates might give you more peace of mind, the fact is that the results don't change significantly. Once you reach 40–50 percent, you are as good as its going to get with more than 125 homes. The chart above shows the necessary number of surveys needed given the number of homes built. Assuming we want to be 90 percent confident in the results and the maximum scores could vary being only +/- 6%. (on a 6 point scale, this would mean scores could go up or down +/- 0.36; on a 10 point scale it would mean scores could go up or down +/- 0.6).
A quality survey administered properly will yield accurate results with less than 80 percent response rate.
NRS once conducted a survey for a builder that generated a 50 percent response rate with more than 400 closings in a year. The client, however, wasn't happy with the negative results and insisted that only the unhappy customers responded, despite showing him the response rate chart. Rather than face reality, he paid us to drive the response rate up to 75 percent. Now, if the builder's theory were true, we should have seen some change in the results. Yet, despite having a 75 percent response rate, the survey results stayed the same.
This home builder is not alone in its thinking. A lot of companies overemphasize the importance of huge survey responses and waste precious time chasing surveys instead of satisfaction. Look at how often sales staff or construction supers chase customers to get the survey response rates up and you can see why this has become a real problem for many home builders. Unfortunately, while a lot of home builders spend a great deal of time and energy to increase the number of survey responses, those resources are not being used to address the customer satisfaction problems identified in the survey results.
RELATED
- Road Map for a Great Customer Journey
- Creating a Culture of Customer Commitment
- How Do You Measure the Customer Experience?
Business Improvement and Customer Satisfaction
Imagine what would happen, however, if home builders stopped chasing higher survey response rates and instead turned their attention to improving the areas of their businesses that are disappointing customers? The results won't change significantly — even if you got 100 percent participation. Meanwhile, the things that could really improve customer satisfaction and create delighted loyal customers are being ignored.
This fact hasn't been lost for the best in customer satisfaction. Pulte Homes, which is the leader in homebuyer satisfaction in the U.S, with 16 divisions ranked #1. Pulte knows the importance customer satisfaction plays in improving its performance. So, what response rate does it use for its customer satisfaction surveys? A respectable 40–50 percent. Centex Homes ranked #1 in 10 divisions uses also uses a different survey with a 40–50 percent response, John Laing Homes the #1 builder in North America in customer satisfaction, recipient of the 2005 NRS Diamond Award presented by Professional Builder receives a 50–60 percent response rate. The list of leading companies is long who understand that focusing satisfaction with customers is more important than being distracted with unnecessary survey response rates. Make sure your company isn't wasting resources chasing surveys in lieu of customer satisfaction and delight.
Response Rates
90% confidence
+/-6% error
| Number of Homes Built | Response Rate Needed |
| 125 | 60% |
| 225 | 46% |
| 325 | 37% |
| 500 | 27% |
| 1000 | 16% |
| 10,000 | 2% |
Paul Cardis is CEO of NRS Corp., a leading research and consulting firm specializing in customer satisfaction for the home building industry. He can be reached at [email protected].