Court Praises Predictive Coding, Then Rejects It

Over the past few years, much ink has been spilled by judges, commentators, and e-discovery service providers opining on the merits and drawbacks of predictive coding. (As we noted in our article, “'Seed Set' Documents Should Not Be Discoverable,” 251 NYLJ 2 (Feb. 4, 2014), in which we provide background on predictive coding and review a number of judicial decisions on the topic, predictive coding, also often referred to as technology assisted review, is the use of computer-generated algorithms to supplement and extend the work of human reviewers in the discovery document review process.) Topics have included when predictive coding should and should not be used, which of the many competing predictive coding technologies and processes are best, whether predictive coding is more accurate than human review, and exactly how, if at all, predictive coding should be incorporated into discovery protocols.

In one of the best-known court battles over predictive coding, Moore v. Publicis Groupe, 287 F.R.D. 182 (S.D.N.Y. 2012), the court concluded that a judge could incorporate predictive coding into electronically stored information (ESI) protocols over the objection of one of the parties. But what happens when the parties have already agreed to an ESI protocol, which does not include predictive coding, and one party wants to change this protocol, over the objection of the other party, by incorporating predictive coding? Can one party unilaterally alter the method of discovery, or does the agreed-upon ESI protocol take precedence?

This is the question addressed by the court in Progressive Casualty Ins. v. Delaney, No. 2:11-cv-00678, 2014 WL 3563467 (D. Nev. July 18, 2014). In answering this question,'the court sheds light on the reasons why parties have been reluctant to accept predictive coding, the need for cooperation and transparency with one's adversary, the resulting risks of this cooperation, and highlights a key debate over best practices ' whether search terms can be used to first limit the universe of documents before predictive coding is employed.

Progressive Casualty Ins. v. Delaney

In this litigation, Progressive was seeking a declaratory judgment that its Directors and Officers Company Liability Insurance Policy for Financial Institutions issued to various failed banks did not provide coverage for lawsuits initiated by the Federal Deposit Insurance Corporation (FDIC) against former directors, officers, and employees of the failed banks.

In the course of this litigation, on Oct. 24, 2013, Progressive and the FDIC agreed to a joint ESI protocol (ESI protocol) which, inter alia, prescribed the process for searching, reviewing and producing ESI. This ESI protocol, which followed the same template used by the parties in five other related lawsuits, was formalized as a court order. Id. at 5.

Progressive collected 1.8 million documents in its discovery efforts. The parties agreed to a set of search terms to filter this full set of documents, which the FDIC “spent months narrowing at Progressive's insistence” and which yielded 565,000 potentially relevant documents. Id . Pursuant to the ESI protocol, Progressive could either turn over all 565,000 documents without reviewing them for responsiveness, or it could choose to undertake a review and withhold any privileged or non-responsive documents. And, as the court noted, the “ESI protocol addressed inadvertent disclosure of privileged materials and contained clawback provisions pursuant to Fed. R. Civ. P. 26(b)(5)(B) and Fed. R. Evid. 502(d).” Id. at 7.

Progressive was supposed to produce documents on a rolling basis. An initial set of documents was due to be produced by the end of September 2013. When Progressive missed that deadline, it told the FDIC that it would produce an initial set of documents in mid-October. Again, Progressive did not produce any documents. As of Dec. 27, 2013, the FDIC had yet to receive the vast majority of Progressive's production and had not received any production of ESI, even though discovery was set to close less than three months later.

This delay was due to Progressive's choice, as was specified as an allowable option in the ESI protocol, to review the 565,000 documents using a traditional process of having dedicated review attorneys manually review each document for responsiveness. After eight contract attorneys had reviewed 125,000 of the documents, Progressive, recognizing that discovery deadlines were looming, determined that the manual review was too time intensive and expensive and began exploring alternative options. Progressive turned to predictive coding to speed up its review. Notably, however, predictive coding was never part of the initially agreed-upon ESI protocol and Progressive never informed the court or the FDIC that it was employing predictive coding; only after the FDIC filed its motion to compel on Dec. 27, 2013 did the parties meet and confer regarding the possibility of its use. Id. at 4.

During meet-and-confers in January 2014, the parties could not reach a resolution regarding the use of predictive coding. This was in part due to Progressive's reluctance to provide detailed information about how it planned to apply predictive coding. Progressive's counsel refused to provide such details based on a belief that the FDIC's request was part of a “pattern of making unreasonable discovery demands.” Id. at 9. After the meet-and-confers, the FDIC opposed Progressive's proposed use of predictive coding, arguing that it would complicate discovery and lead to numerous satellite disputes. To the extent the court was inclined to allow predictive coding, however, the FDIC set forth an alternative proposal. While Progressive wanted to apply predictive coding only to the 565,000 documents that resulted from the application of the agreed upon search terms, the FDIC wanted to apply predictive coding to all 1.8 million documents, plus any additional ESI later collected by Progressive. The FDIC maintained that it would not have initially narrowed its search terms if it had known that Progressive would employ predictive coding because predictive coding achieves similar benefits as narrowed search terms. Additionally, the FDIC insisted upon incorporating the recommended practices of Equivio, Progressive's predictive coding technology vendor, into the protocol, which was lacking from Progressive's proposal. Id. at 4.

Therefore, the court had to determine whether Progressive could employ predictive coding and, if so, the contours of its usage. If the court concluded that Progressive could not employ predictive coding, then it would have to determine what Progressive's obligations were regarding these 565,000 documents.

The court held that Progressive could not employ predictive coding and, as contemplated by the ESI protocol, ordered Progressive to produce all 565,000 documents, responsive or not, to the FDIC. Id. at 11. The court reached this decision after noting that “[t]he cases which have approved technology assisted review of ESI have required an unprecedented degree of transparency and cooperation among counsel,” which the court found lacking in this case. Id. at 4. The court acknowledged that there are many reasons litigators are unwilling to engage in this level of transparency and cooperation, such as:

[M]ethodological decisions reveal work product; discovery about discovery exceeds the scope of Rule 26 of the Federal Rules of Civil Procedure; revealing documents nonresponsive to discovery requests exposes the producing party to unnecessary litigation risks; and the Federal Rules of Civil Procedure only require parties to conduct a reasonable search for responsive documents. Id. at 10.

Additionally, the court highlighted that Progressive's proposed plan for predictive coding was lacking due to its failure to incorporate the best practices recommended by Equivio, including Progressive's proposal to “give its counsel exclusive responsibility for training the predictive coding software.” Id. The court also disapproved of Progressive's attempt to apply predictive coding after search terms had already been run. The court stated:

Progressive's proposal would relieve it of the burden of manual review of ESI according to the ESI protocol it stipulated to and allow it to adopt a predictive coding alternative to a small subset of the universe of ESI collected. Id.

The court also agreed with the FDIC that employing predictive coding would lead to numerous satellite disputes, such as whether a particular document is relevant or not. Id. at 11.

Regarding privilege, the court allowed Progressive to apply privilege filters to the documents and withhold documents these filters identified as “more likely privileged.” Id. Progressive then had 90 days to compile a privilege log for these privileged documents, though there was a clawback agreement in place in the event of an inadvertent disclosure.

In light of the shortcomings of Progressive's predictive coding plan and in the absence of an agreement to modify the ESI protocol, the protocol as originally ordered maintained primacy. The court noted that “Progressive began utilizing predictive coding techniques to review ESI without the Defendants' agreement to amend the parties' stipulated ESI protocol Order (DKT # 67), and without seeking leave of the court to amend the ESI order.” Id. at 2. Since, pursuant to this protocol, neither party had agreed to the use of predictive coding, the court was not inclined to incorporate predictive coding at this stage of the litigation over the FDIC's objections.

Lessons Learned

Despite the court's rejection of predictive coding in this specific case, it did nonetheless look favorably upon predictive coding when employed in a manner more aligned with recommended practices. The court stated: “Predictive coding has emerged as a far more accurate means of producing responsive ESI in discovery. Studies show it is far more accurate than human review or keyword searches which have their own limitations.” Id. at 8. It went so far as to say that “[h]ad the parties worked with their e-discovery consultants and agreed at the onset of this case to a predictive coding based ESI protocol, the court would not hesitate to approve a transparent, mutually agreed upon ESI protocol.” Id. at 9.

Therefore, litigants who think predictive coding may be needed in the course of discovery should be cooperative and transparent in the discovery process. As demonstrated by Progressive, this includes working with the court and one's adversary regarding discovery methods and not unilaterally choosing a course of discovery, especially if doing so violates a mutually agreed-upon, court-ordered discovery protocol.

Ultimately, although it is an endorsement of predictive coding when used under the right circumstances, Progressive serves as a reminder that complications still exist when attempting to employ predictive coding, including debates over best practices, resulting satellite disputes, and the need to cooperate with one's adversary.

H. Christopher Boehning and Daniel J. Toal are partners in the Litigation Department of Paul, Weiss, Rifkind, Wharton & Garrison LLP. Boehning's practice includes complex commercial and civil litigation matters, insurance counseling and litigation, regulatory inquiries, internal investigations and international arbitrations. He can be reached at [email protected]. Toal's practice concentrates in securities litigation, employment law, environmental litigation, and complex commercial disputes. He can be reached at [email protected].

Over the past few years, much ink has been spilled by judges, commentators, and e-discovery service providers opining on the merits and drawbacks of predictive coding. (As we noted in our article, “'Seed Set' Documents Should Not Be Discoverable,” 251 NYLJ 2 (Feb. 4, 2014), in which we provide background on predictive coding and review a number of judicial decisions on the topic, predictive coding, also often referred to as technology assisted review, is the use of computer-generated algorithms to supplement and extend the work of human reviewers in the discovery document review process.) Topics have included when predictive coding should and should not be used, which of the many competing predictive coding technologies and processes are best, whether predictive coding is more accurate than human review, and exactly how, if at all, predictive coding should be incorporated into discovery protocols.

In one of the best-known court battles over predictive coding, Moore v. Publicis Groupe, 287 F.R.D. 182 (S.D.N.Y. 2012), the court concluded that a judge could incorporate predictive coding into electronically stored information (ESI) protocols over the objection of one of the parties. But what happens when the parties have already agreed to an ESI protocol, which does not include predictive coding, and one party wants to change this protocol, over the objection of the other party, by incorporating predictive coding? Can one party unilaterally alter the method of discovery, or does the agreed-upon ESI protocol take precedence?

This is the question addressed by the court in Progressive Casualty Ins. v. Delaney, No. 2:11-cv-00678, 2014 WL 3563467 (D. Nev. July 18, 2014). In answering this question,'the court sheds light on the reasons why parties have been reluctant to accept predictive coding, the need for cooperation and transparency with one's adversary, the resulting risks of this cooperation, and highlights a key debate over best practices ' whether search terms can be used to first limit the universe of documents before predictive coding is employed.

Progressive Casualty Ins. v. Delaney

In this litigation, Progressive was seeking a declaratory judgment that its Directors and Officers Company Liability Insurance Policy for Financial Institutions issued to various failed banks did not provide coverage for lawsuits initiated by the Federal Deposit Insurance Corporation (FDIC) against former directors, officers, and employees of the failed banks.

In the course of this litigation, on Oct. 24, 2013, Progressive and the FDIC agreed to a joint ESI protocol (ESI protocol) which, inter alia, prescribed the process for searching, reviewing and producing ESI. This ESI protocol, which followed the same template used by the parties in five other related lawsuits, was formalized as a court order. Id. at 5.

Progressive collected 1.8 million documents in its discovery efforts. The parties agreed to a set of search terms to filter this full set of documents, which the FDIC “spent months narrowing at Progressive's insistence” and which yielded 565,000 potentially relevant documents. Id . Pursuant to the ESI protocol, Progressive could either turn over all 565,000 documents without reviewing them for responsiveness, or it could choose to undertake a review and withhold any privileged or non-responsive documents. And, as the court noted, the “ESI protocol addressed inadvertent disclosure of privileged materials and contained clawback provisions pursuant to Fed. R. Civ. P. 26(b)(5)(B) and Fed. R. Evid. 502(d).” Id. at 7.

Progressive was supposed to produce documents on a rolling basis. An initial set of documents was due to be produced by the end of September 2013. When Progressive missed that deadline, it told the FDIC that it would produce an initial set of documents in mid-October. Again, Progressive did not produce any documents. As of Dec. 27, 2013, the FDIC had yet to receive the vast majority of Progressive's production and had not received any production of ESI, even though discovery was set to close less than three months later.

This delay was due to Progressive's choice, as was specified as an allowable option in the ESI protocol, to review the 565,000 documents using a traditional process of having dedicated review attorneys manually review each document for responsiveness. After eight contract attorneys had reviewed 125,000 of the documents, Progressive, recognizing that discovery deadlines were looming, determined that the manual review was too time intensive and expensive and began exploring alternative options. Progressive turned to predictive coding to speed up its review. Notably, however, predictive coding was never part of the initially agreed-upon ESI protocol and Progressive never informed the court or the FDIC that it was employing predictive coding; only after the FDIC filed its motion to compel on Dec. 27, 2013 did the parties meet and confer regarding the possibility of its use. Id. at 4.

During meet-and-confers in January 2014, the parties could not reach a resolution regarding the use of predictive coding. This was in part due to Progressive's reluctance to provide detailed information about how it planned to apply predictive coding. Progressive's counsel refused to provide such details based on a belief that the FDIC's request was part of a “pattern of making unreasonable discovery demands.” Id. at 9. After the meet-and-confers, the FDIC opposed Progressive's proposed use of predictive coding, arguing that it would complicate discovery and lead to numerous satellite disputes. To the extent the court was inclined to allow predictive coding, however, the FDIC set forth an alternative proposal. While Progressive wanted to apply predictive coding only to the 565,000 documents that resulted from the application of the agreed upon search terms, the FDIC wanted to apply predictive coding to all 1.8 million documents, plus any additional ESI later collected by Progressive. The FDIC maintained that it would not have initially narrowed its search terms if it had known that Progressive would employ predictive coding because predictive coding achieves similar benefits as narrowed search terms. Additionally, the FDIC insisted upon incorporating the recommended practices of Equivio, Progressive's predictive coding technology vendor, into the protocol, which was lacking from Progressive's proposal. Id. at 4.

Therefore, the court had to determine whether Progressive could employ predictive coding and, if so, the contours of its usage. If the court concluded that Progressive could not employ predictive coding, then it would have to determine what Progressive's obligations were regarding these 565,000 documents.

The court held that Progressive could not employ predictive coding and, as contemplated by the ESI protocol, ordered Progressive to produce all 565,000 documents, responsive or not, to the FDIC. Id. at 11. The court reached this decision after noting that “[t]he cases which have approved technology assisted review of ESI have required an unprecedented degree of transparency and cooperation among counsel,” which the court found lacking in this case. Id. at 4. The court acknowledged that there are many reasons litigators are unwilling to engage in this level of transparency and cooperation, such as:

[M]ethodological decisions reveal work product; discovery about discovery exceeds the scope of Rule 26 of the Federal Rules of Civil Procedure; revealing documents nonresponsive to discovery requests exposes the producing party to unnecessary litigation risks; and the Federal Rules of Civil Procedure only require parties to conduct a reasonable search for responsive documents. Id. at 10.

Additionally, the court highlighted that Progressive's proposed plan for predictive coding was lacking due to its failure to incorporate the best practices recommended by Equivio, including Progressive's proposal to “give its counsel exclusive responsibility for training the predictive coding software.” Id. The court also disapproved of Progressive's attempt to apply predictive coding after search terms had already been run. The court stated:

Progressive's proposal would relieve it of the burden of manual review of ESI according to the ESI protocol it stipulated to and allow it to adopt a predictive coding alternative to a small subset of the universe of ESI collected. Id.

The court also agreed with the FDIC that employing predictive coding would lead to numerous satellite disputes, such as whether a particular document is relevant or not. Id. at 11.

Regarding privilege, the court allowed Progressive to apply privilege filters to the documents and withhold documents these filters identified as “more likely privileged.” Id. Progressive then had 90 days to compile a privilege log for these privileged documents, though there was a clawback agreement in place in the event of an inadvertent disclosure.

In light of the shortcomings of Progressive's predictive coding plan and in the absence of an agreement to modify the ESI protocol, the protocol as originally ordered maintained primacy. The court noted that “Progressive began utilizing predictive coding techniques to review ESI without the Defendants' agreement to amend the parties' stipulated ESI protocol Order (DKT # 67), and without seeking leave of the court to amend the ESI order.” Id. at 2. Since, pursuant to this protocol, neither party had agreed to the use of predictive coding, the court was not inclined to incorporate predictive coding at this stage of the litigation over the FDIC's objections.

Lessons Learned

Despite the court's rejection of predictive coding in this specific case, it did nonetheless look favorably upon predictive coding when employed in a manner more aligned with recommended practices. The court stated: “Predictive coding has emerged as a far more accurate means of producing responsive ESI in discovery. Studies show it is far more accurate than human review or keyword searches which have their own limitations.” Id. at 8. It went so far as to say that “[h]ad the parties worked with their e-discovery consultants and agreed at the onset of this case to a predictive coding based ESI protocol, the court would not hesitate to approve a transparent, mutually agreed upon ESI protocol.” Id. at 9.

Therefore, litigants who think predictive coding may be needed in the course of discovery should be cooperative and transparent in the discovery process. As demonstrated by Progressive, this includes working with the court and one's adversary regarding discovery methods and not unilaterally choosing a course of discovery, especially if doing so violates a mutually agreed-upon, court-ordered discovery protocol.

Ultimately, although it is an endorsement of predictive coding when used under the right circumstances, Progressive serves as a reminder that complications still exist when attempting to employ predictive coding, including debates over best practices, resulting satellite disputes, and the need to cooperate with one's adversary.

H. Christopher Boehning and Daniel J. Toal are partners in the Litigation Department of Paul, Weiss, Rifkind, Wharton & Garrison LLP. Boehning's practice includes complex commercial and civil litigation matters, insurance counseling and litigation, regulatory inquiries, internal investigations and international arbitrations. He can be reached at [email protected]. Toal's practice concentrates in securities litigation, employment law, environmental litigation, and complex commercial disputes. He can be reached at [email protected].