Key Considerations in Choosing a Survey

A presenter at a recent law firm management seminar started his discussion with a graphic of lemmings rushing off a cliff – and floating upward rather than crashing onto the rocks below. The speaker offered this illustration to represent the comfort that many law firms take from making decisions based on what other firms do.

Legal survey participation often reflects a lemming mentality. “Of course we will participate. Everyone else does. It's part of the annual ritual. Inertia rules!”

There are, however, excellent non-lemming reasons for survey participation. Are you keenly interested to learn if your firm is well managed and competitive relative to comparable firms? Or, if you're not inclined to view firm management competitively, are you at least interested in obtaining financial and operational comparisons to help you run your firm efficiently? In either case, you'll most likely want to participate in one or more law firm surveys.

But which surveys? Law firms receive surveys every week from national consulting firms, legal publications and local professional associations.

Moreover, participating in any survey costs time and often a substantial fee. The “yea” or “nea” decision on participating in a survey should therefore be made before the form arrives on your desk ' not when the Managing Partner forwards the questionnaire, or when you receive the annoying reminder from the sponsor that the deadline has been extended.

Key Criteria

So how does one determine which surveys will provide meaningful information and be worth the time and expense to complete? By asking the following types of questions:

Comparability of firms. Will the peer group(s) be comparable to your firm? A firm of 125 attorneys with three satellite offices will not derive valid comparisons if the average size of the participants is 30 timekeepers. Similarly, a Los Angeles partnership will not be well served by an associate compensation survey with respondents mostly from cities like Topeka, Des Moines and Omaha.

Helpful hints:

• Find out if firms you consider peers are participating.
• Contact the sponsor to learn the demographics of firms participating in past surveys.
• Negotiate a price reduction if participation in your key comparison group does not meet a minimum level.

Applicability of published results. Will the reported survey findings actually be useful? Seemingly small reporting variations can materially impact survey results ' and your managerial decisions. For example, in calculating per capita revenues or expenses based on “full time equivalent” headcounts, are partners on sabbatical counted? How are the costs of administrative support functions reported if they are outsourced?

Helpful hints:

• Carefully analyze the survey's instructions.
• Call the sponsor to discuss the strengths and weaknesses of the published results.

Reliability of findings. Does the survey design suggest that collected data will be reliable, and its interpretation objective? Does the publisher use adequate quality control, and filter the data for anomalies?

Specialist organizations, for example, may be less objective in their reporting. Smaller survey publishers may lack the resources to produce statistically valid reports.

Helpful hint:

• Use different surveys to supplement and cross-check each other. For example, you might use salary and benefit data from the ALA annual compensation survey (and ALA chapter surveys) to cross-check surveys published by organizations representing functional specialists.

Statistical granularity and consistency. Will the statistical measurements include just medians and means, or will quartiles and other more detailed statistical breakdowns be provided? Does the survey break out findings for branch offices or just aggregate them? Is information available for practice groups? How will the survey findings be presented? Are data reported consistently across categories and over time?

Helpful hints:

• Review several past copies of the survey results.
• Critique reporting categories of high interest to the firm, both for adequate detail and for consistency from year to year.

Support for all your constituencies. How will the report be used? Each of the major surveys and many of the small reports emphasize particular management areas. Your firm's Chief Operating Officer, Managing Partner, Executive Committee, Director of Human Resources and some individual department managers may all have a strong interest in survey findings. To support all of them with meaningful and reliable information, you'll probably need to participate in multiple surveys.

Synchronization with firm's calendar. Relative to the firm's annual schedule, what is the timing of the survey's data submission and anticipated report publication? Does the survey questionnaire have to be completed during the firm's busiest time of the year? Will the published results be received 2 weeks after the annual budget must be adopted, or after the strategic plan must be reviewed? If so, another survey may prove more practical.

Anti-trust risks. While the national surveys have anti-trust counsel review their survey instruments and reports for compliance with federal anti-trust statutes, this necessary practice is by no means universal.

With regional and local surveys, you need to exercise caution: you need to inquire seriously, “If I participate, will I receive a visit from a Federal Trade Commission representative? Will I go to jail?”

As the ALA's excellent Antitrust Guide states: “There is no organization too small or too localized to escape the possibility of a civil or criminal antitrust suit. The federal government has brought civil or criminal actions against such small organizations as the Maine Lobstermen, a Virginia audio-visual association, Bakersfield Plumbing Contractors… and local barbers' associations.”

Note that ignorance of the applicable laws – on your part or by the survey sponsor ' is not an excuse.

Conclusion: Need for Discipline

Surveys are a key part of the financial tool kit for legal managers, be they attorneys or administrators. A cynic might query, “Why do I need to participate in an expensive and time-consuming survey published by a major accounting firm to tell me what I already know ' that my overhead is too high and my receivables are too old?” The rejoinder is that without comparative figures, it is difficult to draw valid conclusions about whether your staff ratios are reasonable, whether the firm's capital structure is responsible, or whether the amount spent on marketing is within appropriate limits. Most well managed firms therefore find legal surveys a great help in establishing targets or benchmarks; good surveys certainly are more reliable than “gut instinct” in setting financial goals.

A law firm should decide what major annual surveys are cost justified, will provide continuity, have a stable participation base, fit the firm's schedule, adapt to changing organizational trends in the profession and report the most useful information in a meaningful format. Specialty surveys such as those on technology, departmental practices and practice structure may also be helpful on a periodic basis. All other surveys that arrive in the mail should be relegated to the recycling bin.

The Association of Legal Administrators sponsored a panel discussion entitled “Law Firm Surveys – The Good, The Bad, and The Ugly” at their 2003 annual convention in San Diego. The panel included Don Williams (Chief Operating Officer – Schwabe Williamson & Wyatt, P.C.), Pete Peterson (former Legal Administrator for several Midwest law firms and current law firm consultant with the Law Firm Business Institute), and Bill Bachman (Executive Director – Bingham McCutchen LLP). Mike Short (Director – Hildebrandt International) moderated the discussion. Among other topics, the panelists advised attendees on the surveys available, key considerations when choosing a survey, data comparability issues, and survey data interpretation.

Part One of this article offered a general overview and comparison of the major surveys currently available for US law firms (Altman Weil's “The Survey of Law Firm Economics”; Hoffman Alvary's “The Flash Report on Law Firm Economics”; and Pricewaterhouse Coopers' “The Law Firm Statistical Survey”). Part Two offers advice on how to select a survey, and Part Three will explain how to get the most out of survey participation.

Rational Survey Selection

A presenter at a recent law firm management seminar started his discussion with a graphic of lemmings rushing off a cliff – and floating upward rather than crashing onto the rocks below. The speaker offered this illustration to represent the comfort that many law firms take from making decisions based on what other firms do.

Legal survey participation often reflects a lemming mentality. “Of course we will participate. Everyone else does. It's part of the annual ritual. Inertia rules!”

There are, however, excellent non-lemming reasons for survey participation. Are you keenly interested to learn if your firm is well managed and competitive relative to comparable firms? Or, if you're not inclined to view firm management competitively, are you at least interested in obtaining financial and operational comparisons to help you run your firm efficiently? In either case, you'll most likely want to participate in one or more law firm surveys.

But which surveys? Law firms receive surveys every week from national consulting firms, legal publications and local professional associations.

Moreover, participating in any survey costs time and often a substantial fee. The “yea” or “nea” decision on participating in a survey should therefore be made before the form arrives on your desk ' not when the Managing Partner forwards the questionnaire, or when you receive the annoying reminder from the sponsor that the deadline has been extended.

Key Criteria

So how does one determine which surveys will provide meaningful information and be worth the time and expense to complete? By asking the following types of questions:

Comparability of firms. Will the peer group(s) be comparable to your firm? A firm of 125 attorneys with three satellite offices will not derive valid comparisons if the average size of the participants is 30 timekeepers. Similarly, a Los Angeles partnership will not be well served by an associate compensation survey with respondents mostly from cities like Topeka, Des Moines and Omaha.

Helpful hints:

• Find out if firms you consider peers are participating.
• Contact the sponsor to learn the demographics of firms participating in past surveys.
• Negotiate a price reduction if participation in your key comparison group does not meet a minimum level.

Applicability of published results. Will the reported survey findings actually be useful? Seemingly small reporting variations can materially impact survey results ' and your managerial decisions. For example, in calculating per capita revenues or expenses based on “full time equivalent” headcounts, are partners on sabbatical counted? How are the costs of administrative support functions reported if they are outsourced?

Helpful hints:

• Carefully analyze the survey's instructions.
• Call the sponsor to discuss the strengths and weaknesses of the published results.

Reliability of findings. Does the survey design suggest that collected data will be reliable, and its interpretation objective? Does the publisher use adequate quality control, and filter the data for anomalies?

Specialist organizations, for example, may be less objective in their reporting. Smaller survey publishers may lack the resources to produce statistically valid reports.

Helpful hint:

• Use different surveys to supplement and cross-check each other. For example, you might use salary and benefit data from the ALA annual compensation survey (and ALA chapter surveys) to cross-check surveys published by organizations representing functional specialists.

Statistical granularity and consistency. Will the statistical measurements include just medians and means, or will quartiles and other more detailed statistical breakdowns be provided? Does the survey break out findings for branch offices or just aggregate them? Is information available for practice groups? How will the survey findings be presented? Are data reported consistently across categories and over time?

Helpful hints:

• Review several past copies of the survey results.
• Critique reporting categories of high interest to the firm, both for adequate detail and for consistency from year to year.

Support for all your constituencies. How will the report be used? Each of the major surveys and many of the small reports emphasize particular management areas. Your firm's Chief Operating Officer, Managing Partner, Executive Committee, Director of Human Resources and some individual department managers may all have a strong interest in survey findings. To support all of them with meaningful and reliable information, you'll probably need to participate in multiple surveys.

Synchronization with firm's calendar. Relative to the firm's annual schedule, what is the timing of the survey's data submission and anticipated report publication? Does the survey questionnaire have to be completed during the firm's busiest time of the year? Will the published results be received 2 weeks after the annual budget must be adopted, or after the strategic plan must be reviewed? If so, another survey may prove more practical.

Anti-trust risks. While the national surveys have anti-trust counsel review their survey instruments and reports for compliance with federal anti-trust statutes, this necessary practice is by no means universal.

With regional and local surveys, you need to exercise caution: you need to inquire seriously, “If I participate, will I receive a visit from a Federal Trade Commission representative? Will I go to jail?”

As the ALA's excellent Antitrust Guide states: “There is no organization too small or too localized to escape the possibility of a civil or criminal antitrust suit. The federal government has brought civil or criminal actions against such small organizations as the Maine Lobstermen, a Virginia audio-visual association, Bakersfield Plumbing Contractors… and local barbers' associations.”

Note that ignorance of the applicable laws – on your part or by the survey sponsor ' is not an excuse.

Conclusion: Need for Discipline

Surveys are a key part of the financial tool kit for legal managers, be they attorneys or administrators. A cynic might query, “Why do I need to participate in an expensive and time-consuming survey published by a major accounting firm to tell me what I already know ' that my overhead is too high and my receivables are too old?” The rejoinder is that without comparative figures, it is difficult to draw valid conclusions about whether your staff ratios are reasonable, whether the firm's capital structure is responsible, or whether the amount spent on marketing is within appropriate limits. Most well managed firms therefore find legal surveys a great help in establishing targets or benchmarks; good surveys certainly are more reliable than “gut instinct” in setting financial goals.

A law firm should decide what major annual surveys are cost justified, will provide continuity, have a stable participation base, fit the firm's schedule, adapt to changing organizational trends in the profession and report the most useful information in a meaningful format. Specialty surveys such as those on technology, departmental practices and practice structure may also be helpful on a periodic basis. All other surveys that arrive in the mail should be relegated to the recycling bin.