Overcoming Barriers to Technology-Assisted Review

This article explores five barriers to adopting technology assisted review, and considerations that may influence decisions to change.

Barrier 1: We've Always Done It This Way

A firm's incumbent review process is likely the most significant barrier to the adoption of advanced technologies for review. Change is difficult and disruptive, so naturally the impetus to expend time, money and effort to adopt new methods and workflows must be driven by factors compelling enough to convince decision makers that the effort is necessary or worthwhile.

One such factor is the ever-mounting volume of electronically stored information (ESI). Manually intensive methods are hobbled by it; they are becoming too expensive, inconsistent and time-consuming to manage. Selecting keywords to cull the volume isn't the answer, and this method is increasingly scrutinized by courts. Alternative methods bolstered by more sophisticated technologies and expertise thus begin to beckon.

Lawyers who aren't aware of limitations of commonly used methods for document review could be in for a surprise the next time they go to court. The opposition may have acquired the technology and skills ' or at least sufficient knowledge of them ' to challenge less technologically advanced methods in what may very well become the new norm in high-stakes litigation.

Another factor that might drive change is the recognition that technology assisted methods can reduce both cost and risk: cost because lawyers will be able to focus on the data that matters, thus using fewer hours more efficiently; and risk because TAR can result in greater accuracy. (See, “Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review,” Maura R. Grossman & Gordon V. Cormack, Richmond Journal of Law and Technology, Vol. XVII, Issue 3, http://jolt.richmond.edu/v17i3/article11.pdf.)

Both cost and risk reduction are possible, but only if the right tools are engaged with the proper expertise to attend them. If done poorly, however, TAR may reduce neither cost nor risk ' or reduce one at the expense of the other. Reducing volume may reduce costs, for example, but review tools in the wrong hands can yield deficient productions that the opposition, by evaluating protocols, may now be able to detect. If the opposition knows that relevant (or even material) information is left in the garbage pile because of poor sampling or iteration protocols, the result could be impossible to defend and enormously costly to re-do.

Lawyers need to understand the alternatives and their repercussions so that they can make the most informed recommendations for their clients and their firms.

Barrier 2: I Went to Law School, Not MIT

Since most lawyers are not technologists, it is no wonder that agreeing on a set of keywords at a meet-and-confer may still seem to be the path of least resistance ' until the opposing party has other ideas.

Cooperation certainly has its place, but it is likely that discussion about technology assisted review methods and protocols will begin to creep into the more cantankerous side of the meet-and-confer; being prepared for it is only prudent. Fortunately, numerous CLEs, webinars and other educational offerings are now devoted to this topic and can help get lawyers up to speed on the basics. It remains to be seen, however, whether only rudimentary knowledge will be enough to prevail on the e-discovery battlefield.

When electronic information contains the evidence in a case, lawyers need to understand what review pitfalls could prevent them from producing or finding it to support their own case. If they lack requisite knowledge, they need to harness the right expertise. Judge Peck's reference to bringing “your geek to court” may prove to be more sobering than humorous if counsel has certified a production without knowing whether the data population was properly collected, indexed or sampled, or whether it was iteratively refined.

There is safety in knowing the right questions to ask and asking them, even if one needs a geek by their side. Knowing what expertise is required is half the battle. An ESI expert may seem to be the expert you need, but when it comes to document review or e-discovery search, it's search expertise that's required. One thing is certain: There is an evolution in the types of e-discovery challenges that may be raised. Lawyers owe it to themselves and their clients to be prepared for this change.

Barrier 3: You're Selling What?

In the face of marketing hype for technology assisted review tools, hesitancy about being an “early adopter” and confusion about the offerings can certainly be a barrier to change. While it may be tempting to wait for the fallout from other cases, it is still prudent to be prepared. There are enough tools and methods for TAR ' each with their own strengths and limitations ' to create confusion in the market, and there is a definite learning curve for these tools.

One source of confusion relates to the term “predictive coding,” which is a tool that uses machine learning algorithms to predict the likelihood that a document is relevant. Because predictive coding was the tool at issue in the cases mentioned above, many have conflated it with technology assisted review, assuming it to be the only method available. But there are other TAR methods that don't rely on machine learning algorithms. Tools that predict responsiveness using machine learning techniques are very different, and often less accurate, than those which determine it using more transparent tools and methods. It behooves the practitioner to ask the questions that will help him or her understand the differences among various TAR methods.

To ensure defensibility, buyers should ask software vendors or service providers to show the recall and precision statistics for their methodology and to explain how they were determined. Recall and precision are measurements that reflect the accuracy of a review, and claims about them should be carefully scrutinized. If this isn't done at the outset, it may need to be done later under less favorable conditions (i.e., in court).

Fortunately, impartial studies that have tested different search methods and tools for accuracy are available as a resource. For example, the NIST-sponsored TREC Legal Track (http://trec-legal.umiacs.umd.edu) is an example of a scientifically endorsed study that has created a protocol and evaluated the effectiveness of various search methods in reviewing large datasets.

The bottom line for the buyer? Understand what's being offered, and how it differs from other tools and services. Also consider cost in terms of the price that will be paid in time, effort and resources. A tool or process that may appear to be cost-effective based on sticker price won't be such a bargain if it takes too much time to learn, too many people to deploy and supervise, or if it doesn't perform as expected.

Barrier 4: Confusion in the Courtroom

If there is confusion in the marketplace, recent cases suggest that this confusion may have carried over to the courtroom.

For example, in Da Silva Moore, much of the testimony about technology assisted review (predictive coding in this case) came from a panoply of experts, all speaking in their own tongues to the court ' and often disagreeing with one another. The bulk of the testimony was about statistical sample sizes and iterations and acceptable levels of recall and precision (accuracy). Judge Peck's request to one expert to “dumb it down” goes to show it has its prickly side. One tech-savvy lawyer, an in-house e-discovery expert for a large firm, admitted to dragging out a statistics books to try to figure it all out.

Revisiting Statistics 101 may not be necessary, but counsel should at the very least know how well a technology assisted review method has performed. In federal court, after all, it is the lawyer who must provide certifications about reasonableness of process to ensure that discovery is complete and correct. Counsel must be able, in other words, to verify what Judge Peck was after in the Da Silva Moore protocol: adequate precision and recall.

Understandably, most lawyers probably don't want to be in the position of having to defend a method or tool that hasn't quite shaken out. Although courtroom confusion and noise will likely subside as knowledge increases, postponing change until the bench figures it all out may not be the best approach.

Barrier 5: Profits and Revenue Streams

For billable hour law firms, linear manual review has historically been a moneymaker, and technology assisted review ' or any method that reduces time spent reviewing documents ' may not necessarily be a welcome change.

With the billable hour in the line of fire, maintaining document review revenue probably isn't the most compelling argument for sticking to the status quo. These economic times have upped the ante for law firms to provide their clients the most bang for their litigation buck by providing legal rather than document review expertise. The smart firm looks for ways to exploit technological advances by developing expertise in review alternatives and looking for the highest quality technology driven methods.

The use of technology assisted review methods can enable law firms to deliver more value to their clients by enhancing the quality and accuracy of review. Quality means defensibility, and accuracy means that a firm reviews the smaller set of responsive information, lending focus to the actual evidentiary needs of the case. Together, they mean reduced time, increased efficiency and less risk of privilege waiver and sanctions; in short, less cost and less risk.

Conclusion

Given the momentum that seems to be building toward adoption of technology assisted review and the costs and risks of being unprepared to do so, it may be worth your while to consider how to address these barriers to change, and to make an informed decision whether ' or when ' to take the next step.

In law, nothing could go more against the grain for those who are inherently risk averse and precedent-trained than having to brave new approaches to the familiar, seemingly tried and true task of document review.

But recent matters have pushed the topic of technology assisted review (TAR) into the judicial limelight, generating much discussion ' and confusion ' in the legal community. Judge Peck's endorsement of TAR in Da Silva Moore v. Publicis Groupe & MSL Group, 11 Civ. 1279 (S.D.N.Y. 2012) (http://bit.ly/IialKX) and Judge Chamblin's order requiring the use of predictive coding in Global Aerospace, Inc. v. Landow Aviation, No. CL 61040 (Loudoun Cty, VA, April 23, 2012) (http://bit.ly/RQCMd1), illustrate that technology deployed for document review is gaining acceptance from the bench, and may over time dislodge the more linear or manual review processes that most lawyers still use today.

Many forward-looking law firms and their corporate clients have already dispatched e-discovery experts and procurement personnel to collect details and prices for TAR offerings, and they're no doubt returning from the field with mixed messages and a less than full understanding of the facts. Although the use of TAR is probably inevitable, there are barriers that law firms and legal departments face as the need to change becomes more pressing.

This article explores five barriers to adopting technology assisted review, and considerations that may influence decisions to change.

Barrier 1: We've Always Done It This Way

A firm's incumbent review process is likely the most significant barrier to the adoption of advanced technologies for review. Change is difficult and disruptive, so naturally the impetus to expend time, money and effort to adopt new methods and workflows must be driven by factors compelling enough to convince decision makers that the effort is necessary or worthwhile.

One such factor is the ever-mounting volume of electronically stored information (ESI). Manually intensive methods are hobbled by it; they are becoming too expensive, inconsistent and time-consuming to manage. Selecting keywords to cull the volume isn't the answer, and this method is increasingly scrutinized by courts. Alternative methods bolstered by more sophisticated technologies and expertise thus begin to beckon.

Lawyers who aren't aware of limitations of commonly used methods for document review could be in for a surprise the next time they go to court. The opposition may have acquired the technology and skills ' or at least sufficient knowledge of them ' to challenge less technologically advanced methods in what may very well become the new norm in high-stakes litigation.

Another factor that might drive change is the recognition that technology assisted methods can reduce both cost and risk: cost because lawyers will be able to focus on the data that matters, thus using fewer hours more efficiently; and risk because TAR can result in greater accuracy. (See, “Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review,” Maura R. Grossman & Gordon V. Cormack, Richmond Journal of Law and Technology, Vol. XVII, Issue 3, http://jolt.richmond.edu/v17i3/article11.pdf.)

Both cost and risk reduction are possible, but only if the right tools are engaged with the proper expertise to attend them. If done poorly, however, TAR may reduce neither cost nor risk ' or reduce one at the expense of the other. Reducing volume may reduce costs, for example, but review tools in the wrong hands can yield deficient productions that the opposition, by evaluating protocols, may now be able to detect. If the opposition knows that relevant (or even material) information is left in the garbage pile because of poor sampling or iteration protocols, the result could be impossible to defend and enormously costly to re-do.

Lawyers need to understand the alternatives and their repercussions so that they can make the most informed recommendations for their clients and their firms.

Barrier 2: I Went to Law School, Not MIT

Since most lawyers are not technologists, it is no wonder that agreeing on a set of keywords at a meet-and-confer may still seem to be the path of least resistance ' until the opposing party has other ideas.

Cooperation certainly has its place, but it is likely that discussion about technology assisted review methods and protocols will begin to creep into the more cantankerous side of the meet-and-confer; being prepared for it is only prudent. Fortunately, numerous CLEs, webinars and other educational offerings are now devoted to this topic and can help get lawyers up to speed on the basics. It remains to be seen, however, whether only rudimentary knowledge will be enough to prevail on the e-discovery battlefield.

When electronic information contains the evidence in a case, lawyers need to understand what review pitfalls could prevent them from producing or finding it to support their own case. If they lack requisite knowledge, they need to harness the right expertise. Judge Peck's reference to bringing “your geek to court” may prove to be more sobering than humorous if counsel has certified a production without knowing whether the data population was properly collected, indexed or sampled, or whether it was iteratively refined.

There is safety in knowing the right questions to ask and asking them, even if one needs a geek by their side. Knowing what expertise is required is half the battle. An ESI expert may seem to be the expert you need, but when it comes to document review or e-discovery search, it's search expertise that's required. One thing is certain: There is an evolution in the types of e-discovery challenges that may be raised. Lawyers owe it to themselves and their clients to be prepared for this change.

Barrier 3: You're Selling What?

In the face of marketing hype for technology assisted review tools, hesitancy about being an “early adopter” and confusion about the offerings can certainly be a barrier to change. While it may be tempting to wait for the fallout from other cases, it is still prudent to be prepared. There are enough tools and methods for TAR ' each with their own strengths and limitations ' to create confusion in the market, and there is a definite learning curve for these tools.

One source of confusion relates to the term “predictive coding,” which is a tool that uses machine learning algorithms to predict the likelihood that a document is relevant. Because predictive coding was the tool at issue in the cases mentioned above, many have conflated it with technology assisted review, assuming it to be the only method available. But there are other TAR methods that don't rely on machine learning algorithms. Tools that predict responsiveness using machine learning techniques are very different, and often less accurate, than those which determine it using more transparent tools and methods. It behooves the practitioner to ask the questions that will help him or her understand the differences among various TAR methods.

To ensure defensibility, buyers should ask software vendors or service providers to show the recall and precision statistics for their methodology and to explain how they were determined. Recall and precision are measurements that reflect the accuracy of a review, and claims about them should be carefully scrutinized. If this isn't done at the outset, it may need to be done later under less favorable conditions (i.e., in court).

Fortunately, impartial studies that have tested different search methods and tools for accuracy are available as a resource. For example, the NIST-sponsored TREC Legal Track (http://trec-legal.umiacs.umd.edu) is an example of a scientifically endorsed study that has created a protocol and evaluated the effectiveness of various search methods in reviewing large datasets.

The bottom line for the buyer? Understand what's being offered, and how it differs from other tools and services. Also consider cost in terms of the price that will be paid in time, effort and resources. A tool or process that may appear to be cost-effective based on sticker price won't be such a bargain if it takes too much time to learn, too many people to deploy and supervise, or if it doesn't perform as expected.

Barrier 4: Confusion in the Courtroom

If there is confusion in the marketplace, recent cases suggest that this confusion may have carried over to the courtroom.

For example, in Da Silva Moore, much of the testimony about technology assisted review (predictive coding in this case) came from a panoply of experts, all speaking in their own tongues to the court ' and often disagreeing with one another. The bulk of the testimony was about statistical sample sizes and iterations and acceptable levels of recall and precision (accuracy). Judge Peck's request to one expert to “dumb it down” goes to show it has its prickly side. One tech-savvy lawyer, an in-house e-discovery expert for a large firm, admitted to dragging out a statistics books to try to figure it all out.

Revisiting Statistics 101 may not be necessary, but counsel should at the very least know how well a technology assisted review method has performed. In federal court, after all, it is the lawyer who must provide certifications about reasonableness of process to ensure that discovery is complete and correct. Counsel must be able, in other words, to verify what Judge Peck was after in the Da Silva Moore protocol: adequate precision and recall.

Understandably, most lawyers probably don't want to be in the position of having to defend a method or tool that hasn't quite shaken out. Although courtroom confusion and noise will likely subside as knowledge increases, postponing change until the bench figures it all out may not be the best approach.

Barrier 5: Profits and Revenue Streams

For billable hour law firms, linear manual review has historically been a moneymaker, and technology assisted review ' or any method that reduces time spent reviewing documents ' may not necessarily be a welcome change.

With the billable hour in the line of fire, maintaining document review revenue probably isn't the most compelling argument for sticking to the status quo. These economic times have upped the ante for law firms to provide their clients the most bang for their litigation buck by providing legal rather than document review expertise. The smart firm looks for ways to exploit technological advances by developing expertise in review alternatives and looking for the highest quality technology driven methods.

The use of technology assisted review methods can enable law firms to deliver more value to their clients by enhancing the quality and accuracy of review. Quality means defensibility, and accuracy means that a firm reviews the smaller set of responsive information, lending focus to the actual evidentiary needs of the case. Together, they mean reduced time, increased efficiency and less risk of privilege waiver and sanctions; in short, less cost and less risk.

Conclusion

Given the momentum that seems to be building toward adoption of technology assisted review and the costs and risks of being unprepared to do so, it may be worth your while to consider how to address these barriers to change, and to make an informed decision whether ' or when ' to take the next step.