Weighing the Benefits: How Much Weight Will Your Survey Have in Court?

Although following generally accepted survey practices may fortify your survey against critiques, our review of recent U.S. court rulings highlighted several vulnerabilities that can result in your survey evidence being afforded less weight even after being admitted.

Critiques That May Harm Your Survey's Weight

Who's in Your Universe?

Echoing the Reference Manual on Scientific Evidence (Diamond, 2011) and Trademark and Deceptive Advertising Surveys: Law, Science, and Design (Diamond and Swann, 2012), it is crucial to properly define your survey's universe according to the legal matter at hand — that is, to make sure that the population of survey respondents is qualified to provide meaningful responses to the question at hand. This key component may make or break your survey's evidentiary weight.

Take, for example, the trademark infringement case Friesland Brands v. Vietnam National Milk Co a/k/a Vinamilk. Vinamilk challenged Friesland's survey over its "overbroad" universe for surveying consumers of condensed milk instead of consumers of sweetened condensed milk only, who make up Vinamilk's main customer base. The court ruled that "a survey with an 'overbroad' universe may still have probative value," and so the survey was admissible. However, admissibility is only one hurdle for the survey to take: shortcomings such as the survey universe being not perfectly representative of Vinamilk's customer base give the trier of fact the "opportunity to evaluate the weight that should accorded" to the evidence — a hurdle that may be as challenging for a case as a successful motion to exclude the survey evidence.

Conversely, in the trademark dispute case Koninklijke Philips Electronics v. Hunt Control Systems, although Hunt challenged the universes of Philips' surveys for being underinclusive compared to Hunt's actual customer base, the court ultimately found the survey results to be informative because the surveys' target populations already captured "a significant part of the universe of consumers who would select or purchase Hunt products." Put simply, the more accurately defined your survey universe is, the more weight your survey evidence will likely carry.

What's in Your World?

In Classic Foods International v. Kettle Foods, Inc., the defendant employed a survey that was intended to test consumer confusion over an alleged mark by using a coupon to shop for a particular product. The plaintiff moved to exclude the survey evidence, alleging the survey was "unrealistic" and failed to "simulate 'real world conditions'"

In the defendant's survey, a coupon and the related product (the stimuli) were shown to respondents sequentially and so, according to the plaintiff's expert, failed to mirror the real world, where shoppers take coupons with them to the store and simultaneously observe products. The court ruled that the evidence was admissible; however, it also suggested that the plaintiff may include this failure during cross-examination in an attempt to reduce the survey's evidentiary weight.

This is not to say courts expect surveys to perfectly mimic real-life conditions, a feat typically impossible for surveys due to their nature of asking questions about specific slices of real life rather than quietly observing the relevant issues in a nonintrusive way. Consequently, experts should strive to simulate the real-world experience as closely as necessary given the questions they are seeking to address.

For example, experts can try to design surveys that put respondents into a real-world environment, that show real-world advertisements and products, or that motivate "respondents to seriously engage with hypothetical choice tasks" (Hainmueller, Hangartner, and Yamamoto, "Validating Vignette and Conjoint Survey Experiments Against Real-World Behavior," 2015) — that is, to make choices and decisions as they would in real life.

Are You Comparing Apples to Apples?

Failure to use a proper control is yet another hurdle that may affect your survey's weight. Controls are used in surveys to establish a baseline and are critical for properly isolating the causal effects, if any, of the at-issue claim or mark. In Spangler Candy Company v. Tootsie Roll Industries, LLC, a packaging infringement dispute, Tootsie Roll challenged the plaintiff's control on two accounts: using "control products" instead of "control groups" (that is, showing a different product to the same group, rather than showing the same product with one change specific to the at-issue claim to a different group); and a lack of resemblance to the at-issue products.

The court ruled that "while the controls here … bear 'blatantly obvious differences' to the packages at issue, the survey will not be excluded on this ground alone." However, these deficiencies left the survey exposed to the opposing party's critiques during trial.

Designing a proper control stimulus, when a control group is required, can be challenging but is not impossible. Keeping this in mind, and following the tips discussed in our previous article, experts should select a control stimulus that "shares as many characteristics with the experimental stimulus as possible, with the key exception of the characteristic whose influence is being assessed" (Diamond, 2011).

Is Unfamiliarity Breeding Contempt?

In Charlene Dzielak, et al. v. Whirlpool Corp., Whirlpool challenged the plaintiff's conjoint analysis, alleging its methodology is "a little-known proprietary method that is rarely used –if at all" and "has been peer-reviewed in only one article, authored by its inventor." Despite its unfamiliarity with the methodology, the court found it was "sufficiently reliable and relevant" to be admissible, adding that the methodology was more appropriately challenged through cross-examination and rebuttal evidence.

Next: Which Method Is for You?

To help shield survey evidence from such critiques and give it as much weight as possible, experts can buttress their methodological choices with a thorough literature review and follow best practices, such as those laid out in the Reference Guide on Survey Research.

In fact, the appropriate survey methodology will depend on the legal question being addressed, and each methodology comes with its own set of best practices. The next article in this Daubert series will present the different methodologies available and provide tips on selecting and applying them properly.

*****

Rene Befurt is a vice president in the Boston office of Analysis Group, who specializes in developing survey experiments and choice modeling approaches in consumer surveys, and has served as an expert witness in survey and sampling matters. Anthony Nasr is a senior research professional in the Montreal office of Analysis Group, supporting commercial disputes, securities damages, and intellectual property. He can be reached at [email protected]. Marie Warchol is a senior analyst in the Boston office of Analysis Group, supporting commercial disputes and intellectual property. She can be reached at [email protected].