Survey Says: Tips on Getting Over the Daubert Hurdle

There are several key aspects to ensuring that a survey addresses consequential at-issue facts, including assessing whether it:

• Provides evidence for the relevant inquiry (in EMove, Inc. v. Hire a Helper LLC (2018), the court excluded a survey that only measured consumer confusion, instead of addressing the more pertinent legal question of public injury as a result of such confusion);
• Targets the relevant stage(s) of consumers' decision making (in Select Comfort Corp. v. Baxter (2016), the court admitted a survey that measured initial-interest confusion but commented that the plaintiffs would have to provide additional evidence for the at-issue confusion at the time of purchase);
• Is applicable to the relevant timeframe (in O'Reilly Automotive Stores v. Bearing Technologies (2018), a survey conducted in 2017 was challenged for being used to ascertain the secondary meaning of a mark in 2010);
• Is conducted by an expert with relevant qualifications (in Fish v. Kobach (2018), the court excluded an expert for lack of relevant academic experience)

Don't overlook opportunities to inform

Surveys that provide tangential information may still be deemed relevant if the information aids in deciding the at-issue facts and provides some kind of supporting insight into the ultimate at-issue disputes. The court in Townsend v. Monster Beverage Corp., a false advertising case, admitted the plaintiff's survey that provided clarity on how respondents understood some of the at-issue product labels, even though it did not measure the impact of the at-issue claims on consumers' ultimate purchasing decisions.

Play devil's advocate: Think ahead and explain your methodology

Anticipating the types of questions that may arise in a challenge and taking proactive steps to fully explain the survey methodology can also help head off relevance issues. The court in O'Reilly Automotive admitted the plaintiff's survey despite a challenge to its relevance because the survey results adequately demonstrated that there was no meaningful difference between mechanics who worked during the at-issue period and those who started later. The challenge that a survey conducted in 2017 could not measure consumer perception as far back as 2010 was denied, and the court allowed the jury to determine whether the survey results were persuasive.

Takeaway: To avoid exclusion of survey evidence as irrelevant, identify the most consequential factual disputes in the case and carefully consider how your survey addresses them.

To Be Reliable, Make Sure You're Focused on the Key Elements of the Decision-Making Process

The foundation for a reliable survey is the use of a representative sample of the relevant "target population" (i.e., those who make decisions potentially influenced by the challenged conduct), a purchase environment that reflects real-world decision making, and carefully designed survey stimuli that test only the impact of the at-issue attribute(s). These basic survey design elements give rise to some guiding principles.

Make it real (or as real as possible)

Reliable surveys mimic or account for key aspects of the target population's decision-making environment as closely as necessary. In Hain Blueprint v. Blueprint Coffee, a trademark infringement case, the court excluded the plaintiff's survey for failing to approximate actual grocery store settings because, among other things, it ignored the relative in-store location of the plaintiff's pre-bottled beverages and the defendant's bagged whole-bean coffee.

By contrast, in Bimbo Bakeries v. Sycamore, a trade dress infringement case, the court found that the plaintiff's online survey was admissible because the survey replicated the look of the product, even though it was not administered in an actual bakery environment. The court explained that such a survey might be excluded "[i]f the central issue at the heart of the case were the smell or touch of the product," but "[i]n a case where the look of the product is at issue, the party seeking to exclude must show what makes the method of online surveying unreliable."

Minimize the differences

Proper stimuli, whether visual or descriptive aids, are also key to a reliable survey. When comparing specific features of a product, the experimental and control stimuli — that is, the different images or descriptions provided to respondents — should share as many characteristics as possible, except for the attribute(s) being studied (Diamond, 2011), and should offer no cues to artificially steer or bias respondents (Diamond, 2012). These best practices allow the expert to identify differences in responses driven just by the at-issue attribute(s).

For example, in In re: Whirlpool Corp. Front-Loading Washer Products Liability Litigation, the court excluded the defendant's mall survey asking respondents to choose between washers presented only on printed brochures and an actual, physical washing machine, explaining that "a study cannot reliably measure consumer preferences where the available options are unreasonably one-sided."

Takeaway: To ensure that your survey is reliable, identify the relevant marketplace conditions and avoid introducing differences that can distract respondents or bias their responses.

Next Challenge: Weight in Court

After overcoming the hurdle of a Daubert challenge, surveys must then ensure that they are accorded weight in the court's final decision. The next article in the Daubert series, "Weighing the Benefits: How Much Weight Will Your Survey Have in Court?" will provide more tips on how to handle this second hurdle.

*****

Rebecca Kirk Fair is a managing principal in the Boston office of Analysis Group, supporting antitrust litigation, merger and agency competition review, intellectual property, false advertising, tax, and class certification issue. She can be reached at [email protected]. Peter Hess is a vice president in the Menlo Park office of Analysis Group, supporting a wide variety of litigation, business, and policy issues. He can be reached at [email protected]. Vendela Fehrm is a manager in the San Francisco office of Analysis Group, supporting a wide variety of litigation cases, including matters concerning securities, antitrust, telecommunications, and false advertising. She can be reached at [email protected]. Joe Bourdage, Stephen Cacciola and Genna Liu of Analysis Group contributed to the article.