Take One Survey'

And Salt Lightly Before Believing

There's nothing like an election year to immerse ourselves in surveys. This, despite the fact that more often than not, they tell us less than we want to know, and even less than we should believe.

Everybody, it seems, wants to know the future, and it appears that the closest you can get to knowing the future is to accumulate a bunch of guesses. Great for a parlor game, but not much use for making important decisions. Everybody wants to know what other people are doing about important matters, which is a more valid reason for business surveys. Sometimes the survey informs business decision, and keeps a business from operating in the dark. Sometimes the results offer emotional salvation (For example ' According to the survey, I make more than the average of my profession.)

Surveys come in different forms, of course. There's the big political survey that seeks to tell us where the voters stand on each candidate. There's the advertising survey that seeks to tell us the efficacy of a new ad campaign, and there's the brand name survey ('Have You ever heard of this product? What do you think of it?'). These are the kind of surveys mostly done by professional survey and market research companies.

These companies are generally pretty good. They use sophisticated modeling techniques and computerized formulae to tell us (with a margin of error of four or five points either way) who's going to win the next election and how high skirts are going to be next year. Highly scientific stuff ' but, unfortunately not always right. When they're wrong, they claim, often quite rightly, that the world moves faster, and opinions change faster, then they can survey. The problem is that they don't usually claim it until after they're proven wrong. Which is why you have to take those surveys with a grain of salt.

Then there are the attitude surveys (Why do customers (and clients) buy? What do you think of my service?). These are a problem, because too many people don't really know what they think about a lot of things. ('I hired this firm because I needed a lawyer. But I chose this firm over that firm because '). Here's where you get into trouble. The intellectual factors (I like their litigation track record) is only part of the answer. The other part is emotional (the managing partner reminded me of my father) is usually too Freudian to tell you much you can use in a business context. That survey requires a box of salt.

Then there are the localized, one firm, internally done surveys of client satisfaction. These, too, must be taken with a lot of salt. Or as one respondent said, 'If I didn't think I was getting my money's worth I'd have fired you.' These are too often done by marketing people with little sense of the science of surveys. The results are mostly useless. Surveys like these are popular marketing devices, and not just the push-pulls, which load the questions to favor the sponsors' products or services. Some surveys are reasonably accurate, but too often they're the lazy marketer's way to avoid original thinking. If you really want to know what your clients think of you, hire a professional who will sit down with clients for an intensive one on one discussion. Then you're most likely to get substantive information.

Probably the most difficult aspects of surveys are that too many people are not always objective in the answers they give. Many reasons for that ' they don't really know how they feel, they don't want to appear unqualified to answer, they have their own agenda, they don't always know the real reasons they do things and so forth. Many surveys delve into areas in which the answers are so rooted in the subconscious that the spoken response bears no resemblance to the hidden response. Look back on your own choices in buying services. Are your reasons always the right ones? Do you really believe your brand of tomato juice is better than the other brands, or is it just that it's the brand your mother bought? Clearly, the most suspect surveys are those that try to fathom the depth of motivations. We don't always know why we do things, but we'll answer those questions anyway.

Now, I grant you that surveys, even the home grown ones, are not universally bad. Some surveys are thoughtful and useful, with answers you can reasonably rely on. But most of them should be taken carefully, an art in itself.

If you read the results of a survey just for a general idea of how people think about a subject, the inaccuracy of the results doesn't really matter. But if you make decisions based on the results, then the survey had better be an accurate reflection of what the surveyed really think or did. Those are the times when your intuition or experience supersedes even the most professional surveys.

Still, many surveys serve a purpose, if they're taken with that famous grain of salt. Never, never, in judging a survey, abdicate your own intelligence, experience and intuition.

A survey may be useful if:

• You understand the methodology. How was the survey done? Never trust a survey in which the methodology isn't made clear. Written surveys are the most suspect, because too many valid respondents don't bother, or don't take it seriously. In-person responses that arise from intensive personal interviews by trained professional interviewers are at the other end of the validity spectrum ' the responses you can most trust.
• If it's a survey by a reputable publication or website in which participants are invited to submit answers to a specific question.
• If the respondents' qualifications are strong. The best way to judge advertising, for example, is whether respondents are willing to make buying decisions based on it, or by name recognition. The worst way is on aesthetics, which only experienced professionals can judge with any reasonable validity.
• If the sample is valid. It should be broad and based on the nature of respondents. For example, managing partners know about partner compensation (but don't always tell the truth, which is why you need a very broad sample). They know about plans for the future, but sometimes hedge their responses for competitive reasons.
• If the responses give you a valid sense of trend. A sample that's too small is useless, regardless of responses.
• If you can trust the objectivity of the survey's sponsor.
• If you can trust the survey result sufficiently to use it in your own planning.
• And ultimately, if the survey has only a toe in the past and two feet into the future.

In any endeavor, it's invaluable to know what others think and what others are doing. But two things to keep in mind:

1. A survey, no matter how good or how well founded, isn't gospel. Its greatest value is as a guide to your own thinking and decision making. But it should never be more than one factor of many in your decision.
2. Your own experience, intelligence and intuition may be informed by a survey's results, but the final decision should be yours.

Then the survey might be really useful.

And Salt Lightly Before Believing

There's nothing like an election year to immerse ourselves in surveys. This, despite the fact that more often than not, they tell us less than we want to know, and even less than we should believe.

Everybody, it seems, wants to know the future, and it appears that the closest you can get to knowing the future is to accumulate a bunch of guesses. Great for a parlor game, but not much use for making important decisions. Everybody wants to know what other people are doing about important matters, which is a more valid reason for business surveys. Sometimes the survey informs business decision, and keeps a business from operating in the dark. Sometimes the results offer emotional salvation (For example ' According to the survey, I make more than the average of my profession.)

Surveys come in different forms, of course. There's the big political survey that seeks to tell us where the voters stand on each candidate. There's the advertising survey that seeks to tell us the efficacy of a new ad campaign, and there's the brand name survey ('Have You ever heard of this product? What do you think of it?'). These are the kind of surveys mostly done by professional survey and market research companies.

These companies are generally pretty good. They use sophisticated modeling techniques and computerized formulae to tell us (with a margin of error of four or five points either way) who's going to win the next election and how high skirts are going to be next year. Highly scientific stuff ' but, unfortunately not always right. When they're wrong, they claim, often quite rightly, that the world moves faster, and opinions change faster, then they can survey. The problem is that they don't usually claim it until after they're proven wrong. Which is why you have to take those surveys with a grain of salt.

Then there are the attitude surveys (Why do customers (and clients) buy? What do you think of my service?). These are a problem, because too many people don't really know what they think about a lot of things. ('I hired this firm because I needed a lawyer. But I chose this firm over that firm because '). Here's where you get into trouble. The intellectual factors (I like their litigation track record) is only part of the answer. The other part is emotional (the managing partner reminded me of my father) is usually too Freudian to tell you much you can use in a business context. That survey requires a box of salt.

Then there are the localized, one firm, internally done surveys of client satisfaction. These, too, must be taken with a lot of salt. Or as one respondent said, 'If I didn't think I was getting my money's worth I'd have fired you.' These are too often done by marketing people with little sense of the science of surveys. The results are mostly useless. Surveys like these are popular marketing devices, and not just the push-pulls, which load the questions to favor the sponsors' products or services. Some surveys are reasonably accurate, but too often they're the lazy marketer's way to avoid original thinking. If you really want to know what your clients think of you, hire a professional who will sit down with clients for an intensive one on one discussion. Then you're most likely to get substantive information.

Probably the most difficult aspects of surveys are that too many people are not always objective in the answers they give. Many reasons for that ' they don't really know how they feel, they don't want to appear unqualified to answer, they have their own agenda, they don't always know the real reasons they do things and so forth. Many surveys delve into areas in which the answers are so rooted in the subconscious that the spoken response bears no resemblance to the hidden response. Look back on your own choices in buying services. Are your reasons always the right ones? Do you really believe your brand of tomato juice is better than the other brands, or is it just that it's the brand your mother bought? Clearly, the most suspect surveys are those that try to fathom the depth of motivations. We don't always know why we do things, but we'll answer those questions anyway.

Now, I grant you that surveys, even the home grown ones, are not universally bad. Some surveys are thoughtful and useful, with answers you can reasonably rely on. But most of them should be taken carefully, an art in itself.

If you read the results of a survey just for a general idea of how people think about a subject, the inaccuracy of the results doesn't really matter. But if you make decisions based on the results, then the survey had better be an accurate reflection of what the surveyed really think or did. Those are the times when your intuition or experience supersedes even the most professional surveys.

Still, many surveys serve a purpose, if they're taken with that famous grain of salt. Never, never, in judging a survey, abdicate your own intelligence, experience and intuition.

A survey may be useful if:

• You understand the methodology. How was the survey done? Never trust a survey in which the methodology isn't made clear. Written surveys are the most suspect, because too many valid respondents don't bother, or don't take it seriously. In-person responses that arise from intensive personal interviews by trained professional interviewers are at the other end of the validity spectrum ' the responses you can most trust.
• If it's a survey by a reputable publication or website in which participants are invited to submit answers to a specific question.
• If the respondents' qualifications are strong. The best way to judge advertising, for example, is whether respondents are willing to make buying decisions based on it, or by name recognition. The worst way is on aesthetics, which only experienced professionals can judge with any reasonable validity.
• If the sample is valid. It should be broad and based on the nature of respondents. For example, managing partners know about partner compensation (but don't always tell the truth, which is why you need a very broad sample). They know about plans for the future, but sometimes hedge their responses for competitive reasons.
• If the responses give you a valid sense of trend. A sample that's too small is useless, regardless of responses.
• If you can trust the objectivity of the survey's sponsor.
• If you can trust the survey result sufficiently to use it in your own planning.
• And ultimately, if the survey has only a toe in the past and two feet into the future.

In any endeavor, it's invaluable to know what others think and what others are doing. But two things to keep in mind:

1. A survey, no matter how good or how well founded, isn't gospel. Its greatest value is as a guide to your own thinking and decision making. But it should never be more than one factor of many in your decision.
2. Your own experience, intelligence and intuition may be informed by a survey's results, but the final decision should be yours.

Then the survey might be really useful.