Computer-Generated Test Interpretation in Custody Matters

The next wave of developments included the addition to basic scoring of test interpretation: In addition to getting the basic scale-scores sent back, the practitioner could request narrative print-outs offering ideas about the possible meaning of the scores that had been calculated (e.g., “Test takers with this score profile tend to experience abiding anxiety and occasional panic attacks. There are indications, however, that she will be a poor candidate for treatment due to her lack of insight and the fact that denial as a defense is a well-engrained coping mechanism for her.”). This addition of CGTI to the assessment process has both ethical implications for the forensic practitioner and legal advocacy implications for attorneys representing parents or children.

Types of CGTI

There are two primary types of CGTI. Graham, 2006. MMPI: Assessing personality and psychopathology. New York: Oxford Press. In “actuarial CGTI” the output that comes back after the test is scored contains only interpretive statements where research has clearly established that a particular score is associated with a particular behavior (e.g., “Scores this high on scale 6 tend to be associated with high suspiciousness about others.”). This approach is the “just-the-facts Ma'am” approach. If there is a research-established behavior associated with the score, it is reported with no elaboration.

However, virtually all scoring systems for personality instruments represent the second type of CGTI: “automated interpretation.” With this type, the generation of interpretative statements about a score or combination of scores is based on available research and the clinical experience of the authors of the scoring service (e.g., “Scores this high on scale 6 tend to be associated with high suspiciousness about others. There tends to be a high level of distrust in intimate relationships. These individuals often struggle in their romantic relationships and their defenses are difficult to penetrate in psychotherapy, suggesting that the prognosis for change is poor.”).

How CGTI Works

The basic workings of CGTI are as follows: Once a practitioner (including forensic custody evaluators) administers a test, a choice is made whether to order a simple scoring of the test (often referred to as a profile service) or a full interpretive report. For some tests (primarily the MMPI-2), a choice can also be made to ask for, in addition to a comparison of the parent to the general population, a comparison of the results to those of certain sub-groups (like custody litigants). The test-scoring company has entered into its computer system a list of internal algorithms that operate essentially like decision-rules. The algorithms are mathematical formulas for how various scores should be weighted in selecting what sentences to print in the report. These specify, for example, that when scale A is above a T-score of 70 and scale B is above 80, certain sentences should be printed out for the practitioner about the person who was tested. A report is generated with multiple pages of assertions about the likely characteristics of the test-taker.

The sophisticated practitioner knows that these assertions are just hypotheses to be checked against other data. The unsophisticated (or careless) practitioner lifts the phrases verbatim and inserts those he or she agrees with in a report, with little cautious analysis.

Forensic Practitioners:
Assessing the Pros and Cons of CGTI

The possible advantages of CGTI include: 1) a reduction in the subjectivity involved in choosing what interpretations of a parent's score profile to consider from the many available; 2) consistency in interpretive output across practitioners, because the same scores always yield the same interpretations; 3) a reduction in the effect of biasing factors associated with more subjective interview approaches (such as the tendency to have positive impressions make one miss seriously dysfunctional traits); 4) some insurance that important, established clinical correlates will not be missed by the evaluator (Pope, 2006. The MMPI, MMPI-2, and MMPI-A in Court. Washington: American Psychological Association).

On the other hand, there are possible disadvantages to the use of CGTI.

Psychologists have an ethical obligation to select scoring and interpretation services after considering the evidence for the validity of the scoring and interpretation program. American Psychological Association, 2002. Ethical principles of psychologists and code of conduct. American Psychologist. 57(12), 1060-1073. The reality, however, is that the critical underlying algorithms that form the basis and source of the interpretative statements are proprietary secrets about which the evaluator has no (or virtually no) specific knowledge. Flens, 2005. The responsible use of psychological testing in child custody evaluations: Selection of tests. J Child Custody, volume 2. Therefore, in most circumstances, evaluators cannot really know if they agree with the weightings that were applied to the scores, with the formulas applied, or with the more nebulous clinical experience-based inferences of the developers of the scoring system. This also means that there is no way for the evaluator, or any other behavioral scientist, to confirm the validity and reliability of the interpretation that arrives in the mail.

Issues for Legal Advocacy

As a result of the problems associated with the use of CGTI, under cross-examination, evaluators who have used it as a primary basis for a conclusion they have reached about a parent are almost always unable to provide the foundational basis for their conclusion or evidence for its validity and reliability. In addition there is the following challenge: While basic scoring services are controlled by copyright strictures, this is not true of services that provide interpretations of the scores obtained. Any psychologist, even those quite inexperienced with a given test, can develop lists of personally preferred interpretations of various scale elevations, express them mathematically, and charge for offering interpretative print-outs (Graham, 2006).

In addition, because the use of CGTI involves a delegation of forensic judgment to un-identifiable proprietary sources, it also invokes important admissibility problems. Most central to this issue is the fact that the expert offering testimony based on proprietary interpretation algorithms cannot meet the evidentiary requirement for extrinsic proof of reliability, despite the fact that interpretive services may be generally accepted as a method in the witness's discipline. Tippins, 2007, Jan. 2, Custody Evaluations ' Part XXII: Computer Generated Test Interpretation. NYLJ, p. 3, col. 1. In most circumstances, evaluators will not need a CGTI for basic scale interpretations and will be on solid ground simply referencing relevant research or test-related treatises. However, for more nuanced inferences about test patterns, the risk for sole reliance on CGTI increases, as do the evidentiary problems connected to related testimony.

Practice Implications

When an interpretation of a test result that has implications for one's client is proclaimed as based on a CGTI, be aware that this can open the door to a number of avenues of attack with regard to hearsay and admissibility. It will be critical during trial preparation to do the following:

• Attempt to gain access to a copy of the evaluator's entire file.
• Compare verbiage in the test results section of the evaluator's report with the verbiage in any CGTI in the file. Evaluators sometimes lift and insert entire sections.
• Look for any conclusions about your client that appear solely based on the CGTI ' these may be most open to evidentiary challenges.
• Pay close attention to verbiage in the CGTI that the evaluator chose not to use in their final report ' cherry-picking for negative assertions that fit the evaluators “take” on a parent can, at times, represent confirmatory hypothesis testing at its worst.
• Confirm that the evaluator took advantage of the chance to order a comparison of the parent's responses to those of the custody-litigant population. Not doing so can mean that the evaluator may have offered a negative interpretation for findings that are quite normal among litigants.

Positive Changes

Despite the evidentiary problems presented by the proprietary nature of the algorithms underlying CGTI's, there is some notable movement in the right direction. For example, the interpretive reports for MMPI-2-RF (A substantially revised and re-normed version of the MMPI-2 ) contain the basic cut-offs and supportive research references for many interpretative statements and even identify when an interpretation is the result of the author's logical/clinical analysis in the absence of supportive research. Ben-Porath, Y.S. and Tellegen, A. MMPI-2-RF: Manual for Administration, Scoring, and Interpretation. Minneapolis: University of Minnesota Press. This should increase the chances that, at least among the evaluators that use this particular test (a small percentage at present), the sometimes life-altering test-interpretations offered in custody reports will be grounded more often in empirically verified findings rather than in proprietary and inaccessible decision-rules or flawed reasoning.

Jeffrey P. Wittmann, PhD, a member of this newsletter's Board of Editors, is a licensed psychologist and trial consultant at The Center for Forensic Psychology in Albany. He provides peer review services and forensic training in New York and nationally, and is the author of Custody Chaos, Personal Peace (Penguin, 2001). He can be reached at [email protected].

By Jeffrey P. Wittmann

hat is Computer-Generated Test Interpretation (CGTI)? There was a time when psychologists scored test results the old-fashioned way: by hand. Test-takers filled in dots on a sheet, choosing their responses to various questions posed by such tests as the MMPI (Minnesota Multiphasic Personality Inventory) or the MCMI (Millon Clinical Multiaxial Inventory). Then the psychologist, taking on a tedious and lengthy task, laid plastic scoring templates for a long list of scales and sub-scales over the answer sheet, counting true and false responses to yield raw scores. This long list of scores was then used to find “standard scores” using tomes filled with tables of numbers. The clinician then sat back and, using various treatises and clinical experience, developed interpretations of the results with respect to the test-takers personality traits, abilities or dispositions.

As tests became more complicated and the number of sub-scales skyrocketed, and as computers became a more easily accessible tool, the testing industry began to offer basic scoring services. One could scan, fax or mail the client's responses and a computer easily and quickly did the scoring work that used to take hours, eliminating various forms of human error.

The next wave of developments included the addition to basic scoring of test interpretation: In addition to getting the basic scale-scores sent back, the practitioner could request narrative print-outs offering ideas about the possible meaning of the scores that had been calculated (e.g., “Test takers with this score profile tend to experience abiding anxiety and occasional panic attacks. There are indications, however, that she will be a poor candidate for treatment due to her lack of insight and the fact that denial as a defense is a well-engrained coping mechanism for her.”). This addition of CGTI to the assessment process has both ethical implications for the forensic practitioner and legal advocacy implications for attorneys representing parents or children.

Types of CGTI

There are two primary types of CGTI. Graham, 2006. MMPI: Assessing personality and psychopathology. New York: Oxford Press. In “actuarial CGTI” the output that comes back after the test is scored contains only interpretive statements where research has clearly established that a particular score is associated with a particular behavior (e.g., “Scores this high on scale 6 tend to be associated with high suspiciousness about others.”). This approach is the “just-the-facts Ma'am” approach. If there is a research-established behavior associated with the score, it is reported with no elaboration.

However, virtually all scoring systems for personality instruments represent the second type of CGTI: “automated interpretation.” With this type, the generation of interpretative statements about a score or combination of scores is based on available research and the clinical experience of the authors of the scoring service (e.g., “Scores this high on scale 6 tend to be associated with high suspiciousness about others. There tends to be a high level of distrust in intimate relationships. These individuals often struggle in their romantic relationships and their defenses are difficult to penetrate in psychotherapy, suggesting that the prognosis for change is poor.”).

How CGTI Works

The basic workings of CGTI are as follows: Once a practitioner (including forensic custody evaluators) administers a test, a choice is made whether to order a simple scoring of the test (often referred to as a profile service) or a full interpretive report. For some tests (primarily the MMPI-2), a choice can also be made to ask for, in addition to a comparison of the parent to the general population, a comparison of the results to those of certain sub-groups (like custody litigants). The test-scoring company has entered into its computer system a list of internal algorithms that operate essentially like decision-rules. The algorithms are mathematical formulas for how various scores should be weighted in selecting what sentences to print in the report. These specify, for example, that when scale A is above a T-score of 70 and scale B is above 80, certain sentences should be printed out for the practitioner about the person who was tested. A report is generated with multiple pages of assertions about the likely characteristics of the test-taker.

The sophisticated practitioner knows that these assertions are just hypotheses to be checked against other data. The unsophisticated (or careless) practitioner lifts the phrases verbatim and inserts those he or she agrees with in a report, with little cautious analysis.

Forensic Practitioners:
Assessing the Pros and Cons of CGTI

The possible advantages of CGTI include: 1) a reduction in the subjectivity involved in choosing what interpretations of a parent's score profile to consider from the many available; 2) consistency in interpretive output across practitioners, because the same scores always yield the same interpretations; 3) a reduction in the effect of biasing factors associated with more subjective interview approaches (such as the tendency to have positive impressions make one miss seriously dysfunctional traits); 4) some insurance that important, established clinical correlates will not be missed by the evaluator (Pope, 2006. The MMPI, MMPI-2, and MMPI-A in Court. Washington: American Psychological Association).

On the other hand, there are possible disadvantages to the use of CGTI.

Psychologists have an ethical obligation to select scoring and interpretation services after considering the evidence for the validity of the scoring and interpretation program. American Psychological Association, 2002. Ethical principles of psychologists and code of conduct. American Psychologist. 57(12), 1060-1073. The reality, however, is that the critical underlying algorithms that form the basis and source of the interpretative statements are proprietary secrets about which the evaluator has no (or virtually no) specific knowledge. Flens, 2005. The responsible use of psychological testing in child custody evaluations: Selection of tests. J Child Custody, volume 2. Therefore, in most circumstances, evaluators cannot really know if they agree with the weightings that were applied to the scores, with the formulas applied, or with the more nebulous clinical experience-based inferences of the developers of the scoring system. This also means that there is no way for the evaluator, or any other behavioral scientist, to confirm the validity and reliability of the interpretation that arrives in the mail.

Issues for Legal Advocacy

As a result of the problems associated with the use of CGTI, under cross-examination, evaluators who have used it as a primary basis for a conclusion they have reached about a parent are almost always unable to provide the foundational basis for their conclusion or evidence for its validity and reliability. In addition there is the following challenge: While basic scoring services are controlled by copyright strictures, this is not true of services that provide interpretations of the scores obtained. Any psychologist, even those quite inexperienced with a given test, can develop lists of personally preferred interpretations of various scale elevations, express them mathematically, and charge for offering interpretative print-outs (Graham, 2006).

In addition, because the use of CGTI involves a delegation of forensic judgment to un-identifiable proprietary sources, it also invokes important admissibility problems. Most central to this issue is the fact that the expert offering testimony based on proprietary interpretation algorithms cannot meet the evidentiary requirement for extrinsic proof of reliability, despite the fact that interpretive services may be generally accepted as a method in the witness's discipline. Tippins, 2007, Jan. 2, Custody Evaluations ' Part XXII: Computer Generated Test Interpretation. NYLJ, p. 3, col. 1. In most circumstances, evaluators will not need a CGTI for basic scale interpretations and will be on solid ground simply referencing relevant research or test-related treatises. However, for more nuanced inferences about test patterns, the risk for sole reliance on CGTI increases, as do the evidentiary problems connected to related testimony.

Practice Implications

When an interpretation of a test result that has implications for one's client is proclaimed as based on a CGTI, be aware that this can open the door to a number of avenues of attack with regard to hearsay and admissibility. It will be critical during trial preparation to do the following:

• Attempt to gain access to a copy of the evaluator's entire file.
• Compare verbiage in the test results section of the evaluator's report with the verbiage in any CGTI in the file. Evaluators sometimes lift and insert entire sections.
• Look for any conclusions about your client that appear solely based on the CGTI ' these may be most open to evidentiary challenges.
• Pay close attention to verbiage in the CGTI that the evaluator chose not to use in their final report ' cherry-picking for negative assertions that fit the evaluators “take” on a parent can, at times, represent confirmatory hypothesis testing at its worst.
• Confirm that the evaluator took advantage of the chance to order a comparison of the parent's responses to those of the custody-litigant population. Not doing so can mean that the evaluator may have offered a negative interpretation for findings that are quite normal among litigants.

Positive Changes

Despite the evidentiary problems presented by the proprietary nature of the algorithms underlying CGTI's, there is some notable movement in the right direction. For example, the interpretive reports for MMPI-2-RF (A substantially revised and re-normed version of the MMPI-2 ) contain the basic cut-offs and supportive research references for many interpretative statements and even identify when an interpretation is the result of the author's logical/clinical analysis in the absence of supportive research. Ben-Porath, Y.S. and Tellegen, A. MMPI-2-RF: Manual for Administration, Scoring, and Interpretation. Minneapolis: University of Minnesota Press. This should increase the chances that, at least among the evaluators that use this particular test (a small percentage at present), the sometimes life-altering test-interpretations offered in custody reports will be grounded more often in empirically verified findings rather than in proprietary and inaccessible decision-rules or flawed reasoning.

Jeffrey P. Wittmann, PhD, a member of this newsletter's Board of Editors, is a licensed psychologist and trial consultant at The Center for Forensic Psychology in Albany. He provides peer review services and forensic training in New York and nationally, and is the author of Custody Chaos, Personal Peace (Penguin, 2001). He can be reached at [email protected].