Breaking Bad: The Pitfalls of Using Research and How to Avoid Them

Unreliable Data Sources

Invariably, one of the first questions to ask when reviewing any key piece of data, statistic or finding is, says who? The source of information can lend ' or detract from ' credibility. In an age when virtually any entity can conduct, publish and disseminate information it is easy to become overwhelmed and confused; or to be tempted to rely on a less than credible source (especially when the finding supports a position or it is the only available source for a hard-to-find statistic).

While knowing the source is not a foolproof method for avoiding bad data, it lends insight into whether the force behind the research has a specific agenda to push, or motivation for results to pan out a certain way (e.g., a technology company that touts its own study saying they are the top choice for CRM). Similarly, some organizations are simply more reputable or trustworthy than others. Research institutes, government organizations, and academic institutions, for example, might fall into this category. The more challenging question comes when the source is unfamiliar to you. New research reports pop up all the time and not all is shoddy. How, then, do you test the validity of their information?

A good starting point is to examine the methodology and approach section of any documented research report and then perform a Google search to see if the organization has been quoted or referenced in the mainstream press. Learn what you can about the company, individual or association who conducted the study. And don't be afraid to go with your gut. If the organization's website is littered with advertisements or looks hokey, feel free to steer clear. Better safe than sorry, especially if the data will be used to support an investment of firm time or resources. Maintain a list of go-to resources to help streamline data gathering efforts in the future.

Poorly Designed Surveys

It is amazing how virtually everyone believes they are qualified to design a research study. Invitations to participate in a “quick online poll” are routine, as are impromptu firm-sponsored studies. While the stakes are low in gathering insights about one's “love profile” or favorite color, when it comes to making business decisions, survey design is critical to mitigating risk and obtaining accurate results. Just as an experienced litigator knows how to effectively ask questions to persuade rather than confuse the jury, researchers know how to create questions to obtain objective, reliable results. Survey questionnaire design is a science, one that many analysts spent untold numbers of dedicated hours studying in college and beyond. Poorly designed questions can lead respondents to answer in a certain manner (as can questions intentionally designed to elicit bias). Relatedly, question order can impact perceptions, sentiment and likelihood of responding (in a certain way or at all).

Consider, for example, the well-intentioned question, “Is diversity important to you?” What might the results tell you? Typically, the underlying objective behind a question like this is to ascertain whether having more diversity at a firm will help win more business. To that end, this question tells you absolutely nothing. Of course diversity is important. You will be hard-pressed to find a sophisticated buyer of legal services who would not answer that question affirmatively. The design of the question is inherently flawed. (Incidentally, this scenario also holds true for questions like “would you recommend our firm?”) There are a number of more effective ways to ask this question, including a forced rank of pre-screened criteria for selection grounded in a specific historical experience (e.g., Which of the following criteria, in order of importance with 1 being most important, helped you to distinguish the law firm you last hired for a complex matter from other law firms you were considering? Please leave criteria you did not factor into your decision blank.)

Always request a copy of the questionnaire, if available. If building your own study, bring in an expert perspective to ensure the research on which you are about to embark delivers meaningful results. And, perhaps most importantly, avoid the trap far too many law firms fall into ' designing by committee. Too many voices contributing to the survey will only hamper the process and muddy the results.

Response Bias

Anyone who has taken a statistics course has likely heard this term, and perhaps the terms “random sampling,” “sample bias” or “control group.” Each of these are tools statisticians use (random sampling, control groups) or results they avoid (sample bias) to ensure the results of a study are objective, reliable and relevant in statistical analysis. The truth is that in the business world, it is often uneconomical, impractical and, at times, impossible to design a study that strictly adheres to the principles essential to scientific studies (such as those examining the efficacy of drugs or mortality rates associated with disease). Skilled researchers do their best to mitigate bias as much as possible.

Transparency with respect to the survey population (who participated in the research) and how the individuals were selected and invited to participate, is a key characteristic to look for in any research. Acknowledgement of the potential shortcomings of the respondent group is a good sign that the authors considered potential bias in reporting their findings. When establishing your own target samples, consider the overall population you are aiming to represent, your objectives and the level of reliability you need to make a decision. Not every study needs statistically reliable sampling. In fact, most business needs benefit from more qualitative approaches. The key is in knowing the overarching goal and accurately interpreting ' and portraying ' the information that results.

Incomplete Data Sets

In the past decade, a number of resources and research tools have come online to help law firms make better business decisions using their own internal data (e.g., Aderant's Redwood Analytics, Thomson Elite) or externally aggregated data (e.g., Thomson's MonitorSuite, Lexis AtVantage, Lex Machina, Pitchbook, Capital IQ, etc.). In addition, long'relied-upon tools such as Bloomberg, Hoovers and First Research have made concerted efforts to improve their own utility for law firm users. This is good progress. Law firms now have the ability to analyze trends and perform competitive analyses in ways they could not do just a decade ago.

These tools, however, are not the panacea some hold them out to be. Though able to offer a wealth of information and, used properly, to provide valuable insights and detailed data on a number of critical business decisions, many of them simultaneously pose a threat in the hands of a less skilled or untrained user. Many of these data sources rely heavily on electronically reported information, e.g., electronically filed litigation, publicly filed transactions and similar data.

Though it may seem hard to believe in this day and age, numerous courts, jurisdictions and other data-gathering bodies have either not yet moved exclusively to electronic filing, or do not make databases of their electronic information available to others. Similarly, though some private transactions report certain information (e.g., which law firm handled the transaction), others don't. What this shortcoming translates to for a law firm is the potential for a sophisticated analysis based on a less sophisticated (or incomplete) data set. For example, a law firm's analysis of asbestos litigation trends may be missing hundreds or thousands of court filings initiated at the county level. Or a competitive analysis of private equity transactions may include law firm representation in only 42% of cases (and favor the larger law firms who have the resources and place an emphasis on reporting for publicity reasons).

Map out the best-use case scenarios for each of the CI tools to help provide parameters and guidance to those less familiar with their functionality and vulnerabilities. Establish strong, direct lines of communication with the support teams at each of the vendors (admittedly, some are better than others) and use these to test hypotheses and ask questions where data sources or completeness is unclear. Most of all, include a clear explanation of what the data can and cannot tell the firm with the presentation of any data analysis ' one of many best practices we will explore further in the next installment.

Next Stop, Wonderland

Heeding this advice to help ferret out the best quality sources of information, establish good data collection practices and navigate away from misinformation is the first step in making better data-driven decisions.

Marcie Borgal Shunk, a member of this newsletter's Board of Editors, is a Senior Consultant at LawVision Group and member of the Legal Marketing Association's Board of Directors.

'

SPECIAL OFFER: Twitter, LinkedIn, Facebook and Google+ followers can get an online subscription to Marketing the Law Firm for only $299. Click here, select Digital Only and use promo code MLFOL299 at checkout. This offer is valid for new subscribers only.

'