Ensuring Consistent Quality in Electronic Discovery

Electronic evidence is now so prevalent, it's no surprise that most litigation requires its collection and review. But, how do you know that you've collected everything you intended to harvest? How do you verify that what is going out the door to opposing counsel is enough to satisfy their demand, but isn't overbroad and an invitation for fishing? On that point, zoom out for an even more macrocosmic view and ask yourself how you can manage your vendors to ensure consistent quality over the long term.

Ultimately, it's the responsibility of the client representative, either a member of the corporate law department or a member of outside counsel's case team, to ensure the quality of the collection and processing effort. Don't assume that just because the evidence is digital, it doesn't need quality control to the same extent as paper. Electronic evidence is daunting and pushes most people's computer-phobia buttons, but the need for high-level, top-shelf, efficient and effective quality control (QC) in handling electronic evidence, in all matters, can't be ignored.

No specific single formula applies to all situations, and even though every case is different, some basic rules apply across the board.

Know What You're Collecting

Different types of electronic evidence require distinct types of collection and processing methods. For example, the collection and processing methods for e-mail backup tapes are different from those for database extracts.

As a first step, review the discovery demand to see where the likely relevant information is housed. Then, call the IT person who is in charge of the client's data store, and start asking questions. If you are being asked to collect e-mail, be sure to ask the requesting party or parties:

• What kind of e-mail system is in use?
• On what kind of server is it housed?
• What is its backup schedule?
• Are files stored locally as well as on the network?
• What is the e-mail backup medium?
• What is the tape-rotation schedule?

Be sure to take these steps, too:

• Document the answers to all of the above questions, getting as many technical specifications as possible.
• Pair technical information with network diagrams, because this will be source material if you need to explain or defend the collection method.
• Request a network diagram of each repository that interests you. A site visit is often invaluable, because what the in-house IT group thinks it has, and what it really has, can often be markedly different. Expect that the scope of the collection will expand as you go forward.
• Discourage the internal IT group from carrying out self-collection. Although an attractive option at the outset because it is the most inexpensive solution, self-collection is risky because non-forensically trained custodians may unwittingly make errors in the collection process. Normally, it's very easy for an experienced attorney to tear these untrained custodians' processes apart and make a spoliation claim. At a minimum, have an attorney supervise the IT person doing the collection, or preferably, employ a third-party vendor to perform the collection. Involving a vendor costs more in the short term, but will yield a much more defensible evidentiary set in the long term.

Karl Lomberk of Datum Legal, a litigation-support vendor, has witnessed the consequences of firms taking the burdens of data collection upon themselves. 'Law firms have expertise in practicing law, not necessarily in collecting and managing data. There are vendors who do data collection every day, and have streamlined and perfected the process,' he explains.

Involve litigation-support personnel or in-house experts as early in the project as possible. Having the expertise of someone in-house who is familiar with the technical issues and working with vendors will prove immeasurably valuable. If your firm or corporate legal department doesn't have an in-house person, then designate someone to be a single point of contact (SPOC) with the vendor. Eliminating conflicting instructions will increase the overall speed and quality of the vendor's work product.

Predetermine Quality Control Specifications

Document the quality-control specifications ahead of time. John Rubens of The Oliver Group, an electronic-evidence harvesting firm, suggests having a vendor-services agreement in place before collection. 'It avoids confusion and sets a standard for the entire process,' he advises.

Start with defining the scope of the collection and processing. In the case of e-mail, first have the vendor make a working copy. Secure the original data in a fireproof safe. Then have the vendor deduplicate or near-deduplicate the data set. Deduplication is an important phase of the data-collection process because it removes exact copies of e-mails. Near-deduplication removes similar, but not exact, copies. Choose exact deduplication if a wider production set is desired, and choose near-deduplication if you want a more restrained production. Also, determine whether the deduplication is to be performed across the entire collection, or just within custodian mailboxes.

The document-culling and filtering parameters must also be specified. Culling is based on criteria such as time and date ranges, custodians or specific subjects within the e-mail. Filtering, on the other hand, is based on file type. Certain files that exist within the data set can be excluded, such as system or log files. Employing deduplication, culling and filtering at the outset will limit all the downstream production, review and quality-control costs.

Next, specify the output format ' normally either native files or .TIFF images. In the event that the data is being imaged and housed in a review tool, be sure that the contract also specifies that a specific load-file format ' the format of the file with the particular load of information to be processed ' will be provided with the production. If a non-working load file is produced, then the contract should provide for the entire batch to be sent back and a working load file provided with it. Look at it this way: Think in terms of having the vendor provide a plug-and-play data set for review and production; no staff time should be spent fixing data that has been received from the vendor ” if there is, then there's a quality-control problem, and the vendor should fix the error immediately.

Quality Review

Review the production as it progresses and verify that it meets expectations.

Carolyn B. Southerland, special counsel for Baker Botts LLP, suggests requiring vendors to produce stan-dard quality-control reports throughout the process. For example:

• If backup tapes are being restored and reviewed, then check the results;
• Ask whether the number of responsive e-mails is larger or smaller than you anticipated;
• Ask what the file sizes are; and
• Ask whether you're hitting databases within the set.

Southerland says that seeing what the process uncovers early will save on downstream processing and quality-control costs. She also suggests targeting the 'low-hanging fruit' first; that is, by identifying which stores are likely to be the most responsive and processing those first, one increases the chances of satisfying the discovery demand without having to engage in unduly burdensome or expensive discovery.

Another step: Request the before-and-after logs of the processed files. Check and see whether what was culled out is reasonable. See also whether the files that remain don't seem to fit the parameters. Where a full-text index was used to deduplicate e-mail, verify that the correct terms and roots were used to focus the production to the relevant documents. If you're still not getting to what interests you, then expand or change the search terms, because very often, it's a process of finding the needles in the haystack, and sometimes you must focus or expand your search to find them all. Ideally, this is a team, and participatory, process, so don't be afraid to discuss modifying the search criteria with the vendor.

Vendor Communication

Don't be afraid to ask questions. It's normal to feel intimidated by the technical aspects of electronic-evidence production at the outset. A good vendor will take the time to explain its approach and the decisions that go into processing the production. The good, responsible and conscientious vendor will also explain the technical underpinnings of the process ' so, if you don't understand a term or a concept, be sure to ask about it. If you get pushback from a vendor and find that the designated representative ' or the one for whom you must search ' won't take the time to explain it, or if you get a 'you don't need to know' sort of response, that's a sure sign of trouble. Good vendors always engage in a transparent, consistent and documentable process ' after all, the process must be transparent so that the attorneys always have a complete understanding of what's going on and so that they can be consistent to ensure accurate results.

Don't shy away, either, from discussing the vendor's use of production scripts, which are a kind of process checklist. This step ensures that the same process is employed each time it's undertaken. The process must be able to be documented, because it must be defensible if questioned. Quality vendors understand this and will provide complete chain-of-custody logs for the data in your case file.

Assistance

And a word to the wise (or the soon-to-be wise): Overcome the frequently felt super-person instinct that leads to trying to manage everything, because that instinct usually leads to super problems. The hard-fact lowdown is this: Involve a professional vendor if you need one. As a general rule, the larger the collection or the higher the damage threshold, the greater the chances that engaging an e-discovery expert will improve your chances of success. In complex cases, where the evidentiary request includes multiple stores of e-mail, documents, databases and legacy systems, the technical decisions will multiply as the process continues. Having an expert in place to help set and control the process can be invaluable and is some of the best insurance against a spoliation claim.

Opposing Dialogue

Engage in conversations with opposing counsel early in the collection process. Though litigation is historically a bunker-mentality business, Southerland suggests contacting opposing counsel to discuss the proposed collection process and seeing whether a compromise can be reached that satisfies plaintiffs but is not unduly burdensome to the client. For example, Southerland recommends that you 'contact the IT person in charge of a database and see what pre-existing reports are available; then contact opposing counsel and see if those reports will satisfy their needs.'

She points out that this can lessen the time needed to collect relevant information by using the resources the client already has, as well as the computing power of the data store itself.

'If those reports aren't sufficient, then a more customized type of data collection is in order, which should include testing the results prior to review to see whether the process accomplished the objective,' she says, adding that 'the biggest issue is overabundance of responsive data. This excess severely impacts the client from the cost perspective.'

If a repository is culled and there is still 60% of it left, then re-examine the culling procedure. Introduce more limits on the process, such as date, custodian or file-type restrictions ' or expand ones that were already set. By determining with opposing counsel what can be deemed responsive, you can eliminate more files of limited or no evidentiary value. This represents a win-win for both sides, because it lowers production costs and the time to review the documents.

Planning

Plan according to the needs of the project, and give your vendors time to work. Processing takes time, as does the quality-control process itself. Talk to the vendor ahead of time to find out how much time is needed to do a particular job. If possible, do a rolling production, which will allow flexibility to give the vendor the maximum time to process the data and apply quality-control criteria to it. From the vendor's perspective, Datum's Lomberk notes that the planning and ramp-up time is the hardest stage, but that it's the most important because every project is different. He recommends that clients involve the vendor at the earliest point possible for QC, because the vendor will be held accountable for the integrity of the data, and producing the right data the right way takes time.

'It takes time to QC and reprocess .TIFF images that do not convert properly,' Lomberk says. 'We recommend checking every image to guarantee quality as a critically important step.'

In the end ' and usually in the beginning, too ' one thing you can count on is that challenging issues will always arise with electronic-evidence collection. But with the right preplanning and quality-assurance measures in place ahead of time, you can be confident of the quality of information being collected, reviewed and produced. For the process to work most effectively, though, there must be a true partnership between the legal professionals and the vendor. Open lines of communication and clear management of expectations will go a long way toward ensuring a timely, responsive and cost-effective evidentiary production.

Electronic evidence is now so prevalent, it's no surprise that most litigation requires its collection and review. But, how do you know that you've collected everything you intended to harvest? How do you verify that what is going out the door to opposing counsel is enough to satisfy their demand, but isn't overbroad and an invitation for fishing? On that point, zoom out for an even more macrocosmic view and ask yourself how you can manage your vendors to ensure consistent quality over the long term.

Ultimately, it's the responsibility of the client representative, either a member of the corporate law department or a member of outside counsel's case team, to ensure the quality of the collection and processing effort. Don't assume that just because the evidence is digital, it doesn't need quality control to the same extent as paper. Electronic evidence is daunting and pushes most people's computer-phobia buttons, but the need for high-level, top-shelf, efficient and effective quality control (QC) in handling electronic evidence, in all matters, can't be ignored.

No specific single formula applies to all situations, and even though every case is different, some basic rules apply across the board.

Know What You're Collecting

Different types of electronic evidence require distinct types of collection and processing methods. For example, the collection and processing methods for e-mail backup tapes are different from those for database extracts.

As a first step, review the discovery demand to see where the likely relevant information is housed. Then, call the IT person who is in charge of the client's data store, and start asking questions. If you are being asked to collect e-mail, be sure to ask the requesting party or parties:

• What kind of e-mail system is in use?
• On what kind of server is it housed?
• What is its backup schedule?
• Are files stored locally as well as on the network?
• What is the e-mail backup medium?
• What is the tape-rotation schedule?

Be sure to take these steps, too:

• Document the answers to all of the above questions, getting as many technical specifications as possible.
• Pair technical information with network diagrams, because this will be source material if you need to explain or defend the collection method.
• Request a network diagram of each repository that interests you. A site visit is often invaluable, because what the in-house IT group thinks it has, and what it really has, can often be markedly different. Expect that the scope of the collection will expand as you go forward.
• Discourage the internal IT group from carrying out self-collection. Although an attractive option at the outset because it is the most inexpensive solution, self-collection is risky because non-forensically trained custodians may unwittingly make errors in the collection process. Normally, it's very easy for an experienced attorney to tear these untrained custodians' processes apart and make a spoliation claim. At a minimum, have an attorney supervise the IT person doing the collection, or preferably, employ a third-party vendor to perform the collection. Involving a vendor costs more in the short term, but will yield a much more defensible evidentiary set in the long term.

Karl Lomberk of Datum Legal, a litigation-support vendor, has witnessed the consequences of firms taking the burdens of data collection upon themselves. 'Law firms have expertise in practicing law, not necessarily in collecting and managing data. There are vendors who do data collection every day, and have streamlined and perfected the process,' he explains.

Involve litigation-support personnel or in-house experts as early in the project as possible. Having the expertise of someone in-house who is familiar with the technical issues and working with vendors will prove immeasurably valuable. If your firm or corporate legal department doesn't have an in-house person, then designate someone to be a single point of contact (SPOC) with the vendor. Eliminating conflicting instructions will increase the overall speed and quality of the vendor's work product.

Predetermine Quality Control Specifications

Document the quality-control specifications ahead of time. John Rubens of The Oliver Group, an electronic-evidence harvesting firm, suggests having a vendor-services agreement in place before collection. 'It avoids confusion and sets a standard for the entire process,' he advises.

Start with defining the scope of the collection and processing. In the case of e-mail, first have the vendor make a working copy. Secure the original data in a fireproof safe. Then have the vendor deduplicate or near-deduplicate the data set. Deduplication is an important phase of the data-collection process because it removes exact copies of e-mails. Near-deduplication removes similar, but not exact, copies. Choose exact deduplication if a wider production set is desired, and choose near-deduplication if you want a more restrained production. Also, determine whether the deduplication is to be performed across the entire collection, or just within custodian mailboxes.

The document-culling and filtering parameters must also be specified. Culling is based on criteria such as time and date ranges, custodians or specific subjects within the e-mail. Filtering, on the other hand, is based on file type. Certain files that exist within the data set can be excluded, such as system or log files. Employing deduplication, culling and filtering at the outset will limit all the downstream production, review and quality-control costs.

Next, specify the output format ' normally either native files or .TIFF images. In the event that the data is being imaged and housed in a review tool, be sure that the contract also specifies that a specific load-file format ' the format of the file with the particular load of information to be processed ' will be provided with the production. If a non-working load file is produced, then the contract should provide for the entire batch to be sent back and a working load file provided with it. Look at it this way: Think in terms of having the vendor provide a plug-and-play data set for review and production; no staff time should be spent fixing data that has been received from the vendor ” if there is, then there's a quality-control problem, and the vendor should fix the error immediately.

Quality Review

Review the production as it progresses and verify that it meets expectations.

Carolyn B. Southerland, special counsel for Baker Botts LLP, suggests requiring vendors to produce stan-dard quality-control reports throughout the process. For example:

• If backup tapes are being restored and reviewed, then check the results;
• Ask whether the number of responsive e-mails is larger or smaller than you anticipated;
• Ask what the file sizes are; and
• Ask whether you're hitting databases within the set.

Southerland says that seeing what the process uncovers early will save on downstream processing and quality-control costs. She also suggests targeting the 'low-hanging fruit' first; that is, by identifying which stores are likely to be the most responsive and processing those first, one increases the chances of satisfying the discovery demand without having to engage in unduly burdensome or expensive discovery.

Another step: Request the before-and-after logs of the processed files. Check and see whether what was culled out is reasonable. See also whether the files that remain don't seem to fit the parameters. Where a full-text index was used to deduplicate e-mail, verify that the correct terms and roots were used to focus the production to the relevant documents. If you're still not getting to what interests you, then expand or change the search terms, because very often, it's a process of finding the needles in the haystack, and sometimes you must focus or expand your search to find them all. Ideally, this is a team, and participatory, process, so don't be afraid to discuss modifying the search criteria with the vendor.

Vendor Communication

Don't be afraid to ask questions. It's normal to feel intimidated by the technical aspects of electronic-evidence production at the outset. A good vendor will take the time to explain its approach and the decisions that go into processing the production. The good, responsible and conscientious vendor will also explain the technical underpinnings of the process ' so, if you don't understand a term or a concept, be sure to ask about it. If you get pushback from a vendor and find that the designated representative ' or the one for whom you must search ' won't take the time to explain it, or if you get a 'you don't need to know' sort of response, that's a sure sign of trouble. Good vendors always engage in a transparent, consistent and documentable process ' after all, the process must be transparent so that the attorneys always have a complete understanding of what's going on and so that they can be consistent to ensure accurate results.

Don't shy away, either, from discussing the vendor's use of production scripts, which are a kind of process checklist. This step ensures that the same process is employed each time it's undertaken. The process must be able to be documented, because it must be defensible if questioned. Quality vendors understand this and will provide complete chain-of-custody logs for the data in your case file.

Assistance

And a word to the wise (or the soon-to-be wise): Overcome the frequently felt super-person instinct that leads to trying to manage everything, because that instinct usually leads to super problems. The hard-fact lowdown is this: Involve a professional vendor if you need one. As a general rule, the larger the collection or the higher the damage threshold, the greater the chances that engaging an e-discovery expert will improve your chances of success. In complex cases, where the evidentiary request includes multiple stores of e-mail, documents, databases and legacy systems, the technical decisions will multiply as the process continues. Having an expert in place to help set and control the process can be invaluable and is some of the best insurance against a spoliation claim.

Opposing Dialogue

Engage in conversations with opposing counsel early in the collection process. Though litigation is historically a bunker-mentality business, Southerland suggests contacting opposing counsel to discuss the proposed collection process and seeing whether a compromise can be reached that satisfies plaintiffs but is not unduly burdensome to the client. For example, Southerland recommends that you 'contact the IT person in charge of a database and see what pre-existing reports are available; then contact opposing counsel and see if those reports will satisfy their needs.'

She points out that this can lessen the time needed to collect relevant information by using the resources the client already has, as well as the computing power of the data store itself.

'If those reports aren't sufficient, then a more customized type of data collection is in order, which should include testing the results prior to review to see whether the process accomplished the objective,' she says, adding that 'the biggest issue is overabundance of responsive data. This excess severely impacts the client from the cost perspective.'

If a repository is culled and there is still 60% of it left, then re-examine the culling procedure. Introduce more limits on the process, such as date, custodian or file-type restrictions ' or expand ones that were already set. By determining with opposing counsel what can be deemed responsive, you can eliminate more files of limited or no evidentiary value. This represents a win-win for both sides, because it lowers production costs and the time to review the documents.

Planning

Plan according to the needs of the project, and give your vendors time to work. Processing takes time, as does the quality-control process itself. Talk to the vendor ahead of time to find out how much time is needed to do a particular job. If possible, do a rolling production, which will allow flexibility to give the vendor the maximum time to process the data and apply quality-control criteria to it. From the vendor's perspective, Datum's Lomberk notes that the planning and ramp-up time is the hardest stage, but that it's the most important because every project is different. He recommends that clients involve the vendor at the earliest point possible for QC, because the vendor will be held accountable for the integrity of the data, and producing the right data the right way takes time.

'It takes time to QC and reprocess .TIFF images that do not convert properly,' Lomberk says. 'We recommend checking every image to guarantee quality as a critically important step.'

In the end ' and usually in the beginning, too ' one thing you can count on is that challenging issues will always arise with electronic-evidence collection. But with the right preplanning and quality-assurance measures in place ahead of time, you can be confident of the quality of information being collected, reviewed and produced. For the process to work most effectively, though, there must be a true partnership between the legal professionals and the vendor. Open lines of communication and clear management of expectations will go a long way toward ensuring a timely, responsive and cost-effective evidentiary production.