Cognitive Encapsulation: Thinking Inside the Box

More than a decade ago, the Honorable Stephen Hjelt, the presiding Administrative Law Judge for the California Office of Administrative Hearings in San Diego, wrote what might best be described as an open letter to psychologists (Hjelt, S. (2000). Professional psychology: A view from the bench. Register Report, 26:1, 8-13.) Judge Hjelt observed: “[Y]our profession has strong roots as a discipline that has a foundation in the scientific method. However, some of you simply stopped using it” (p. 12).

The failure of some psychologists to employ the scientific method is seen in the selection by evaluators of assessment instruments that do not meet the evidentiary demands of forensic work. For example, data collected by Quinnell and Bow (Quinnell, F. A. & Bow, J. N. (2001). Psychological tests used in child custody evaluations. Behavioral Sciences and the Law, 19, 491-501) led them to conclude that “projective instruments, which generally lack adequate psychometric properties, continue to be widely used despite the absence of empirical support” (p. 500).

Surveys such as that conducted by Quinnell and Bow are not designed to collect and analyze the perspectives that underlie the decisions concerning what assessment techniques to employ. Anita Trubitt, who promotes the use of various projective techniques in forensic evaluations, has unambiguously expressed her awareness that methods used in forensic evaluations “must ' be supported by scientific evidence” that methods endorsed by her (most notably, drawings and interpretations of children's sand tray play) do not meet “reliability and validity criteria”; but that those who feel, as she does, that these methods “tell us ' how the child sees himself and his relationship with parents” should use those methods “for [their] own edification ' [but not] mention [their] use at all in the report to the court.” (Words taken from an outline for a workshop entitled “To play or not to play in child custody evaluations: Recent challenges in current practice,” presented 10/13/06 at the Association of Play Therapists conference in Toronto, Canada.)

Information gathered by evaluators for what Trubitt refers to as their “edification” is information that, from the perspective of an attorney, constitutes an element of “the facts or data ' upon which an expert bases an opinion or inference” (FRE 703) and which the expert may “be required to disclose. ' ” (FRE 705). Put simply: Trubitt advocates using certain assessment methods, knowing as they are being used, that their use will not be disclosed.

Most of us describe something as reliable when we wish to communicate the view that it can be relied upon. It is important to recognize that, as the term is used in the field of psychology, it relates to two different measures of consistency. The first is the degree to which different users of a particular assessment device will interpret in a reasonably similar manner the data that the device yields. This is known as the instrument's inter-judge reliability. The second is the likelihood that data obtained from a test-taker today will be reasonably similar to data obtained from the same test-taker at a specified time in the future. This is known as the instrument's test/re-test reliability. Psychologists are also concerned with an assessment instrument's validity ' the likelihood that it is measuring what its users presume it is measuring.'

It should not be presumed that great weight should be given to data obtained through the use of instruments the reliability of which has been established and that produce scores. Among the attorneys and judges who seek advisory input from forensic mental health professionals, there are many who presume that instruments producing scores can be counted on to contribute clarity and objectivity to what otherwise might be an arcane process, vitiated by subjectivity. This belief leads many evaluators, attorneys, litigants, and judges to attach to test scores far more weight than is appropriate. Scores often create an aura of precision that misleads decision-makers.

Privacy v. Disclosure

In the course of receiving education and training in providing treatment services, notions of privacy are instilled in practitioners in all the mental health professions. Treatment providers become protective of the privacy of those with whom they professionally interact and often take the position that certain notes taken by them (usually referred to as process notes) should be protected from disclosure.

Close to a decade ago, I observed that “the integrity of the evaluative process is dependent upon transparency. ' ” [Martindale, D. A. (2004). Integrity and transparency: A commentary on record-keeping in child custody evaluations. J Child Custody: Research, Issues, and Practices, 1:1, 33-42, p. 34]. Model Standard 3.2(b) of the Association of Family and Conciliation Court's Model Standards for Custody Evaluation states that records “shall be created in reasonable detail, shall be legible, shall be stored in a manner that makes expeditious production possible, and shall be made available in a timely manner to those with the legal authority to inspect them or possess copies of them.”

Every once in awhile, an evaluator will assert that contemporaneously taken notes should be protected from disclosure, and, every once in a while, a judge will agree. In 2002, in a litigated custody dispute being heard in Westchester County, NY, a trial court judge opined that the notes of a “court appointed neutral forensic psychologist contain the unfiltered, immediate impressions of the psychologist, jotted down in haste and without the benefit of reflection” and, for that reason, should be protected from disclosure (Ochs v. Ochs, 193 Misc.2d 502, at 508 (NY Sup.Ct. Westchester Co. 2002)). The court added: “There is simply nothing on the face of the report or in the facts that have been related with respect to its development that would justify inquiry into its foundations” (at 510).

The position of the Ochs court, that inquiry into the foundations that underlie an evaluator's opinions must be justified by a showing of “special circumstances” (at 503), is at odds with the United State Supreme Court's decision in Daubert v. Merrell Dow Pharmaceuticals, 509 U.S. 579 (1993). The Court (at 592-593) stressed the importance of examining the underlying reasoning employed by an expert. Explorations of the “unfiltered, immediate impressions” of evaluators often shed much needed light on the reasoning that forms the foundation for the opinions expressed by evaluators in their reports and in their testimony and upon which many courts will subsequently rely.

Efforts by evaluators to shield their notes from disclosure are likely to reflect their desire to insulate themselves from legitimate scrutiny. In some cases, however, evaluators seek to protect the identities of individuals from whom they have obtained collateral source information. A recently reviewed report contained the following phrases: 1) “by most accounts, ' ” 2) “ by all reports, ' ” 3) “ By most reports, '” 4) “a previous therapist asserted ' ” and 5) “ Collateral sources also described. ' ” The American Psychological Association's Specialty Guidelines for Forensic Psychology admonishes psychologists to “ identify the source of each piece of information that was considered and relied upon in formulating a particular conclusion, opinion or other professional product” (Guideline 11.03).

Contractual Interpersonal Dynamics

In the patient-psychotherapist relationship, an implicit contract governs the interpersonal dynamics. The patient provides information to the therapist so that the therapist can use that information to assist the patient. When mental health professionals are performing evaluations in the context of litigated custody disputes, they are expected to obtain information from those being evaluated (often, with some difficulty; often, mixed with misinformation) and use that information to assist the courts that have requested the evaluations.

When evaluations have been concluded, reports are prepared for use by the courts. These reports should be descriptive; specifically, they should contain: 1) descriptions of the strengths and deficiencies of the parties that relate to parenting issues; 2) articulations of the bases for the opinions that certain strengths and deficiencies are tied to parenting, and, only where sought by the court; 3) placement/access recommendations clearly linked to the descriptions and articulations in the reports.

For some mental health professionals whose past experience has been in the provision of health services, old habits are hard to break. They continue to operate under the terms of the unwritten patient-therapist contract: “You have provided me with information. I owe it to you to use that information to formulate opinions concerning ways in which you can improve your life.” There is often a failure to recognize that what, in a therapy context, was viewed as helpful, may, in an evaluative context, be viewed as unneeded, unwanted, and intrusive.

Consider the following: A psychologist who has accepted the task of evaluating a family in connection with a litigated custody dispute includes in his report the recommendation that the father (who is obese) undergo bariatric surgery, so he will be able to play ball with his son. I will ignore the most obvious issue; namely, that psychologists should not be opining on the advisability of medical procedures. Though the foregoing is an extreme example of providing answers to questions that were not posed, in reviewing evaluators' reports, it is not uncommon to encounter recommendations concerning ways in which the litigants can improve their own lives and the lives of the children who are at the center of the litigation. Reports that include recommendations aimed at improving the lives of those whose disputes have brought them to court, are referred to by me as prescriptive.

Though the involvement of two parents in custody litigation constitutes evidence of their inability to construct a parenting plan on their own, it should not be presumed that they lack the requisite competence to make other parenting decisions. The evaluative task is descriptive in nature and when evaluators prescribe ways in which litigating parents can improve their lives and the lives of their children, the evaluators are expressing opinions, the foundations for which are likely to lie in personal values and beliefs, rather than in empirical outcome data.

Bases for Opinions

With only rare exceptions, in a treatment context, the only person who must be satisfied with the practitioner's work is the patient. Opinions expressed by therapists to their patients may or may not be digested and may or may be acted upon, but rarely are they directly challenged. In contrast, forensic practitioners must be prepared to explain and defend their methods, decisions, and opinions. Ethical standards that were not relevant to the provision of treatment must be learned and honored.

Among the ethical transgressions encountered in reviews of custody evaluators' reports or transcripts of their testimony, is the expression of “opinions of the psychological characteristics of individuals [in the absence of] an examination of the individuals adequate to support [those] statements or conclusions” [wording taken from Ethical Standard 9.01(b)]. The confidence with which such opinions are often expressed is a source of bewilderment to many reviewers.

Readers are encouraged to give thought to the manner in which all of us develop the feeling (whether it be realistic or not) that we possess a particular skill. From early childhood, the confident feeling that we are doing something well arises in response to reinforcement. Even reasonably introspective adults are inclined not to dissect reinforcement. (Perhaps those who are telling me that I am doing X well are not good judges of X. Perhaps I should not conclude that my ability to do X well in context A indicates that I will be able to do X well in context B.) With great regularity, treatment providers receive strong reinforcement when they offer their professional opinions of the personality characteristics of individuals with whom they have never interacted.

People in individual psychotherapy discuss problematic interpersonal interactions with spouses, children, neighbors, bosses, subordinates, in-laws, etc. In their attempts to assist their patients, therapists generate hypotheses concerning the personality characteristics of the spouses, children, etc. being discussed. The hypotheses are developed based entirely on information and perspectives provided to the therapists by their patients. At some point in the therapeutic process, therapists are likely to share with their patients the hypotheses that have been developed. Predictably, when therapists share with their patients the hypotheses that have been formulated based upon input from the patients, the patients confirm those hypotheses.

The circularity of this dynamic is exquisite, yet its symmetry is likely not to be noticed by the participants. 1) A patient provides information to her therapist about her spouse; 2) based on that information, the therapist develops hypotheses concerning the personality characteristics of the spouse; 3) one day, the patient relates something exasperating that the spouse has done; 4) the therapist offers an explanation, predicated on the hypotheses that she has generated; 5) the patient expresses congratulatory amazement at the depth of the therapist's understanding of the spouse. Therapists who have traveled in this circle a sufficient number of times are at risk for becoming confident that, employing information provided by others, they can formulate accurate opinions concerning individuals with whom they have not interacted.

After years of being praised for the accuracy of their descriptions of the personality characteristics of individuals whom they have not met, many therapists apparently become confident that they can do well that which reasonable minds know cannot be done. The skill that treatment providers have developed is not the ability to make diagnoses in absentia; rather, it is the ability to listen with care and draw reasonable inferences based upon what they have been told. The accuracy of the inferences is being judged by those who provided the information on which they are based. It is, therefore, predictable that the inferences will be deemed to be accurate.

Conclusion

This article was the second of two, addressing the ways in which education, training, and experience in providing health services often creates a cognitive box from which practitioners must extricate themselves if they wish to perform forensic services. Issues emphasized in this article were the importance of selecting assessment methods of established reliability and validity; recognizing the demands of the discovery process, by creating reasonably detailed and legible notes; focusing attention on securing information directly related to the psycholegal issues in dispute; and, refraining from expressing opinions of the psychological characteristics of individuals who have not been evaluated.

'

'

David Martindale, Ph.D, A.B.B.P., a member of this newsletter's Board of Editors, is board certified in forensic psychology by the American Board of Professional Psychology. He served as the Reporter for the Association of Family and Conciliation Courts' Model Standards for Child Custody Evaluation and is the co-author, with Jon Gould, of The Art and Science of Child Custody Evaluations. E-mail: [email protected].

'

This article is the second of two that address the ways in which education, training, and experience in providing health services often creates a cognitive box from which practitioners must extricate themselves if they wish to perform forensic services. The first article, published in our March 2013 Issue, addressed the ways in which backgrounds in providing health services shape interviewing methods and create a desire to assist those with whom mental health practitioners professionally interact. It presented the position that forensic interviews differ from those conducted in treatment contexts and that forensic evaluators cannot advise or assist those who are being evaluated in forensic contexts.

This second article addresses the selection of assessment instruments, issues of privacy and disclosure, evaluator-evaluee interpersonal dynamics, and the need for providers of forensic evaluation services to examine with care the bases for the opinions expressed by them.

Assessment

More than a decade ago, the Honorable Stephen Hjelt, the presiding Administrative Law Judge for the California Office of Administrative Hearings in San Diego, wrote what might best be described as an open letter to psychologists (Hjelt, S. (2000). Professional psychology: A view from the bench. Register Report, 26:1, 8-13.) Judge Hjelt observed: “[Y]our profession has strong roots as a discipline that has a foundation in the scientific method. However, some of you simply stopped using it” (p. 12).

The failure of some psychologists to employ the scientific method is seen in the selection by evaluators of assessment instruments that do not meet the evidentiary demands of forensic work. For example, data collected by Quinnell and Bow (Quinnell, F. A. & Bow, J. N. (2001). Psychological tests used in child custody evaluations. Behavioral Sciences and the Law, 19, 491-501) led them to conclude that “projective instruments, which generally lack adequate psychometric properties, continue to be widely used despite the absence of empirical support” (p. 500).

Surveys such as that conducted by Quinnell and Bow are not designed to collect and analyze the perspectives that underlie the decisions concerning what assessment techniques to employ. Anita Trubitt, who promotes the use of various projective techniques in forensic evaluations, has unambiguously expressed her awareness that methods used in forensic evaluations “must ' be supported by scientific evidence” that methods endorsed by her (most notably, drawings and interpretations of children's sand tray play) do not meet “reliability and validity criteria”; but that those who feel, as she does, that these methods “tell us ' how the child sees himself and his relationship with parents” should use those methods “for [their] own edification ' [but not] mention [their] use at all in the report to the court.” (Words taken from an outline for a workshop entitled “To play or not to play in child custody evaluations: Recent challenges in current practice,” presented 10/13/06 at the Association of Play Therapists conference in Toronto, Canada.)

Information gathered by evaluators for what Trubitt refers to as their “edification” is information that, from the perspective of an attorney, constitutes an element of “the facts or data ' upon which an expert bases an opinion or inference” (FRE 703) and which the expert may “be required to disclose. ' ” (FRE 705). Put simply: Trubitt advocates using certain assessment methods, knowing as they are being used, that their use will not be disclosed.

Most of us describe something as reliable when we wish to communicate the view that it can be relied upon. It is important to recognize that, as the term is used in the field of psychology, it relates to two different measures of consistency. The first is the degree to which different users of a particular assessment device will interpret in a reasonably similar manner the data that the device yields. This is known as the instrument's inter-judge reliability. The second is the likelihood that data obtained from a test-taker today will be reasonably similar to data obtained from the same test-taker at a specified time in the future. This is known as the instrument's test/re-test reliability. Psychologists are also concerned with an assessment instrument's validity ' the likelihood that it is measuring what its users presume it is measuring.'

It should not be presumed that great weight should be given to data obtained through the use of instruments the reliability of which has been established and that produce scores. Among the attorneys and judges who seek advisory input from forensic mental health professionals, there are many who presume that instruments producing scores can be counted on to contribute clarity and objectivity to what otherwise might be an arcane process, vitiated by subjectivity. This belief leads many evaluators, attorneys, litigants, and judges to attach to test scores far more weight than is appropriate. Scores often create an aura of precision that misleads decision-makers.

Privacy v. Disclosure

In the course of receiving education and training in providing treatment services, notions of privacy are instilled in practitioners in all the mental health professions. Treatment providers become protective of the privacy of those with whom they professionally interact and often take the position that certain notes taken by them (usually referred to as process notes) should be protected from disclosure.

Close to a decade ago, I observed that “the integrity of the evaluative process is dependent upon transparency. ' ” [Martindale, D. A. (2004). Integrity and transparency: A commentary on record-keeping in child custody evaluations. J Child Custody: Research, Issues, and Practices, 1:1, 33-42, p. 34]. Model Standard 3.2(b) of the Association of Family and Conciliation Court's Model Standards for Custody Evaluation states that records “shall be created in reasonable detail, shall be legible, shall be stored in a manner that makes expeditious production possible, and shall be made available in a timely manner to those with the legal authority to inspect them or possess copies of them.”

Every once in awhile, an evaluator will assert that contemporaneously taken notes should be protected from disclosure, and, every once in a while, a judge will agree. In 2002, in a litigated custody dispute being heard in Westchester County, NY, a trial court judge opined that the notes of a “court appointed neutral forensic psychologist contain the unfiltered, immediate impressions of the psychologist, jotted down in haste and without the benefit of reflection” and, for that reason, should be protected from disclosure ( Ochs v. Ochs , 193 Misc.2d 502, at 508 (NY Sup.Ct. Westchester Co. 2002)). The court added: “There is simply nothing on the face of the report or in the facts that have been related with respect to its development that would justify inquiry into its foundations” (at 510).

The position of the Ochs court, that inquiry into the foundations that underlie an evaluator's opinions must be justified by a showing of “special circumstances” (at 503), is at odds with the United State Supreme Court's decision in Daubert v. Merrell Dow Pharmaceuticals , 509 U.S. 579 (1993). The Court (at 592-593) stressed the importance of examining the underlying reasoning employed by an expert. Explorations of the “unfiltered, immediate impressions” of evaluators often shed much needed light on the reasoning that forms the foundation for the opinions expressed by evaluators in their reports and in their testimony and upon which many courts will subsequently rely.

Efforts by evaluators to shield their notes from disclosure are likely to reflect their desire to insulate themselves from legitimate scrutiny. In some cases, however, evaluators seek to protect the identities of individuals from whom they have obtained collateral source information. A recently reviewed report contained the following phrases: 1) “by most accounts, ' ” 2) “ by all reports, ' ” 3) “ By most reports, '” 4) “a previous therapist asserted ' ” and 5) “ Collateral sources also described. ' ” The American Psychological Association's Specialty Guidelines for Forensic Psychology admonishes psychologists to “ identify the source of each piece of information that was considered and relied upon in formulating a particular conclusion, opinion or other professional product” (Guideline 11.03).

Contractual Interpersonal Dynamics

In the patient-psychotherapist relationship, an implicit contract governs the interpersonal dynamics. The patient provides information to the therapist so that the therapist can use that information to assist the patient. When mental health professionals are performing evaluations in the context of litigated custody disputes, they are expected to obtain information from those being evaluated (often, with some difficulty; often, mixed with misinformation) and use that information to assist the courts that have requested the evaluations.

When evaluations have been concluded, reports are prepared for use by the courts. These reports should be descriptive; specifically, they should contain: 1) descriptions of the strengths and deficiencies of the parties that relate to parenting issues; 2) articulations of the bases for the opinions that certain strengths and deficiencies are tied to parenting, and, only where sought by the court; 3) placement/access recommendations clearly linked to the descriptions and articulations in the reports.

For some mental health professionals whose past experience has been in the provision of health services, old habits are hard to break. They continue to operate under the terms of the unwritten patient-therapist contract: “You have provided me with information. I owe it to you to use that information to formulate opinions concerning ways in which you can improve your life.” There is often a failure to recognize that what, in a therapy context, was viewed as helpful, may, in an evaluative context, be viewed as unneeded, unwanted, and intrusive.

Consider the following: A psychologist who has accepted the task of evaluating a family in connection with a litigated custody dispute includes in his report the recommendation that the father (who is obese) undergo bariatric surgery, so he will be able to play ball with his son. I will ignore the most obvious issue; namely, that psychologists should not be opining on the advisability of medical procedures. Though the foregoing is an extreme example of providing answers to questions that were not posed, in reviewing evaluators' reports, it is not uncommon to encounter recommendations concerning ways in which the litigants can improve their own lives and the lives of the children who are at the center of the litigation. Reports that include recommendations aimed at improving the lives of those whose disputes have brought them to court, are referred to by me as prescriptive.

Though the involvement of two parents in custody litigation constitutes evidence of their inability to construct a parenting plan on their own, it should not be presumed that they lack the requisite competence to make other parenting decisions. The evaluative task is descriptive in nature and when evaluators prescribe ways in which litigating parents can improve their lives and the lives of their children, the evaluators are expressing opinions, the foundations for which are likely to lie in personal values and beliefs, rather than in empirical outcome data.

Bases for Opinions

With only rare exceptions, in a treatment context, the only person who must be satisfied with the practitioner's work is the patient. Opinions expressed by therapists to their patients may or may not be digested and may or may be acted upon, but rarely are they directly challenged. In contrast, forensic practitioners must be prepared to explain and defend their methods, decisions, and opinions. Ethical standards that were not relevant to the provision of treatment must be learned and honored.

Among the ethical transgressions encountered in reviews of custody evaluators' reports or transcripts of their testimony, is the expression of “opinions of the psychological characteristics of individuals [in the absence of] an examination of the individuals adequate to support [those] statements or conclusions” [wording taken from Ethical Standard 9.01(b)]. The confidence with which such opinions are often expressed is a source of bewilderment to many reviewers.

Readers are encouraged to give thought to the manner in which all of us develop the feeling (whether it be realistic or not) that we possess a particular skill. From early childhood, the confident feeling that we are doing something well arises in response to reinforcement. Even reasonably introspective adults are inclined not to dissect reinforcement. (Perhaps those who are telling me that I am doing X well are not good judges of X. Perhaps I should not conclude that my ability to do X well in context A indicates that I will be able to do X well in context B.) With great regularity, treatment providers receive strong reinforcement when they offer their professional opinions of the personality characteristics of individuals with whom they have never interacted.

People in individual psychotherapy discuss problematic interpersonal interactions with spouses, children, neighbors, bosses, subordinates, in-laws, etc. In their attempts to assist their patients, therapists generate hypotheses concerning the personality characteristics of the spouses, children, etc. being discussed. The hypotheses are developed based entirely on information and perspectives provided to the therapists by their patients. At some point in the therapeutic process, therapists are likely to share with their patients the hypotheses that have been developed. Predictably, when therapists share with their patients the hypotheses that have been formulated based upon input from the patients, the patients confirm those hypotheses.

The circularity of this dynamic is exquisite, yet its symmetry is likely not to be noticed by the participants. 1) A patient provides information to her therapist about her spouse; 2) based on that information, the therapist develops hypotheses concerning the personality characteristics of the spouse; 3) one day, the patient relates something exasperating that the spouse has done; 4) the therapist offers an explanation, predicated on the hypotheses that she has generated; 5) the patient expresses congratulatory amazement at the depth of the therapist's understanding of the spouse. Therapists who have traveled in this circle a sufficient number of times are at risk for becoming confident that, employing information provided by others, they can formulate accurate opinions concerning individuals with whom they have not interacted.

After years of being praised for the accuracy of their descriptions of the personality characteristics of individuals whom they have not met, many therapists apparently become confident that they can do well that which reasonable minds know cannot be done. The skill that treatment providers have developed is not the ability to make diagnoses in absentia; rather, it is the ability to listen with care and draw reasonable inferences based upon what they have been told. The accuracy of the inferences is being judged by those who provided the information on which they are based. It is, therefore, predictable that the inferences will be deemed to be accurate.

Conclusion

This article was the second of two, addressing the ways in which education, training, and experience in providing health services often creates a cognitive box from which practitioners must extricate themselves if they wish to perform forensic services. Issues emphasized in this article were the importance of selecting assessment methods of established reliability and validity; recognizing the demands of the discovery process, by creating reasonably detailed and legible notes; focusing attention on securing information directly related to the psycholegal issues in dispute; and, refraining from expressing opinions of the psychological characteristics of individuals who have not been evaluated.

'

'

David Martindale, Ph.D, A.B.B.P., a member of this newsletter's Board of Editors, is board certified in forensic psychology by the American Board of Professional Psychology. He served as the Reporter for the Association of Family and Conciliation Courts' Model Standards for Child Custody Evaluation and is the co-author, with Jon Gould, of The Art and Science of Child Custody Evaluations. E-mail: [email protected].

'