The Failure of Peer Review

To the best of our knowledge, no defendant in a criminal trial has been successful in insisting on a jury of 'his neighbors, fellows, associates, [and] persons having the same legal status in society as that which he holds.' If put to the test, the inaccuracies in the Court's definition would quickly become apparent. A prisoner who commits murder while incarcerated is not likely to be tried by a jury of fellow convicts ' his 'neighbors, fellows, associates [and] persons having the same legal status in society as that which he holds.' It seems that in law, the notion of a peer has been loosely construed.

Peer Review on the Periphery

In the process of establishing expertise, professionals whose publications have appeared in peer-reviewed journals or in texts that have included a peer review element in the editorial procedure are quick to point this out. In doing so, they indirectly suggest that peer review is a procedure that yields scholarly contributions. It turns out that this is true only if their peers are scholars. The rights of the uninformed and misinformed to publish are protected by our Constitution.

When individuals organize for the purpose of publishing a journal, it is more likely than not that they will select an editorial board whose members are interested in the same professional issues as those who initiated the project and who hold similar perspectives on those issues. The editorial board and the executive editor of the journal often assume responsibility for choosing colleagues who will conduct reviews of the manuscripts that are being considered for publication in the journal.

The editorial board, the executive editor, and the reviewers function as gatekeepers with regard to manuscripts that are submitted. The first level of gate-keeping lies with the editor who receives a manuscript and must decide whether to forward the manuscript to one or more reviewers for comment. The editor presumably performs an initial review of submitted manuscripts and rejects those that are deemed unworthy of consideration. There are no standards or guidelines about how to determine which manuscripts are appropriate for further review and which are immediately rejected. The quality of gate-keeping at this initial stage is determined by several factors, only some of which will be identified here.

Editors may not read submitted manuscripts thoroughly. Editors do not always have the requisite scholarly background. Editors and editorial board members may have political agendas that influence their decision-making. Author names and affiliations appear on the manuscripts that are reviewed by editors, affecting the manner in which the gate operates. When manuscripts are passed along to blind reviewers, the personal and academic shortcomings that affect editors are likely to affect the assigned reviewers.

Just as mainstream biologists and geologists tend to view creationist explanations of the development of homo sapiens, the Grand Canyon, etc., as unsupported and unsupportable, mainstream mental health professionals tend to view certain concepts as equally unsupportable. Among mental health professionals, some believe that from the moment of birth all life events are stored in the brain in what can most easily be described as photographic form. Those who hold this view believe that in the eidetic memory of every child are images of all the child's life experiences.

If a group of such professionals were to establish the Society for the Study of Infantile Eidetic Memory and if the Society were to publish a journal, it is quite likely that all those reviewing contributions to the journal would be believers in infantile eidetic memory. Believers might deem acceptable a research article that non-believers would reject. At the Web site of the Eidetic Therapy Institute, www.eidetictherapy.com, readers are informed that an eidetic image 'contains an accurate record of important life experiences, reveals the source of long standing problems and uncovers the pathways to health and well being.'

We delude ourselves when we conceptualize peer reviewed journals as scholarly periodicals requiring that each article submitted for publication be judged by an independent panel of scholarly or scientifically-minded peers, who will reject submitted articles that are lacking in scholarship. The lofty goals of the peer review process (sometimes referred to as refereeing) often are not achieved.

The purpose of peer review is to require that authors meet the standards of their respective disciplines, but what if those who run a particular journal and those who submit articles are all members of a generally unrecognized discipline? Won't contributors to the Journal of Unsupported Theories, the Journal of Sophistry & Silliness, and the Journal of the Wildly Whimsical all assert that their published works have been peer reviewed? Members of the editorial boards of journals are ordinarily well-respected members of their respective fields of study; however, not all fields of study are worthy of respect.

Peer Review and the Mainstream

Even among psychologists who are not members of the American Psychological Association (APA), it would undoubtedly be recognized that the APA represents mainstream psychology. One would be likely to presume that texts published by the APA have been peer-reviewed and that in the peer review process factual inaccuracies would have been identified and removed prior to publication. Not so.

In 2003, the APA released a text, authored by Benjamin and Gollan, outlining custody evaluation procedures in which the authors opine that the erasure of tapes made during interviews 'prevents the opposing counsel from using contemporaneous material out of context during a later cross-examination at deposition or trial.' We believe that mainstream psychologists (and the APA) hold the view that in the field of forensic psychological assessment, methodology is dictated by the philosophy that everything an evaluator does must be open to scrutiny ' even by those who might be motivated by a desire to discredit the evaluator's findings. Destruction of records enables experts to insulate themselves from appropriate scrutiny.

Perhaps the stream is being re-directed. Psychology's national organization has elected to remove from its ethics code a previously strong statement about the creation and preservation of records. In its now obsolete 1992 ethics code, Ethical Standard 1.23(b) reads as follows: 'When psychologists have reason to believe that records of their professional services will be used in legal proceedings ' they have a responsibility to create and maintain documentation in the kind of detail and quality that would be consistent with reasonable scrutiny in an adjudicative forum.' No similar admonition appears in the current (2002) version of the ethics code.

There is reason for concern that the APA is about to take another step back from its 1992 position. APA's 1994 Guidelines for Child Custody Evaluations in Divorce Proceedings will soon be replaced by a document entitled Guidelines for Evaluating Parental Responsibility. A draft of the document, made available for comment, contains the following statement: 'When working in a judicial forum, psychologists are typically [emphasis added] expected to preserve evidence that underlies opinions.' The 1994 guidelines contained the following unambiguous statement: 'All raw data and interview information are recorded with an eye towards their possible review by other psychologists or the court, where legally permitted.'

Recently, in the peer-reviewed Journal of Forensic Psychology Practice (JFPP), Benjamin and his colleagues asserted that the protocol outlined in the 2003 text alluded to above 'does not compel a departure from firmly established legal precedents across jurisdictions for evaluating parental capacity ' ' It seems to us that the destruction of records does 'compel a departure from firmly established legal precedents.' Additionally, in the JFPP article, Benjamin and his colleagues remind readers that their evaluative methodology limits 'the children's involvement in the process only to the structured parent-child observation sessions.' Children are not interviewed. Benjamin practices in a state (Washington) in which, by statute, decisions in custody and access matters must take into account 'the wishes of a child who is sufficiently mature to express reasoned and independent preferences as to his or her residential schedule.'

In adjudicating disputes concerning custody and access, the courts in 39 states, in the District of Columbia, and in Puerto Rico consider the preference of children who are deemed to be of sufficient age and maturity. Additionally, the Uniform Marriage and Divorce Act (UMDA) mandates consideration of children's wishes with respect to their custodians. Though children's expressed preferences are not determinative, they are an important factor that cannot be considered unless children have been interviewed.

If psychologists heed the admonition of the APA, as expressed in its 1994 custody guidelines, that '[t]he focus of the evaluation is on parenting capacity, the psychological and developmental needs of the child, and the resulting fit,' it logically follows that children must be evaluated. It is generally recognized that one child may benefit from the parenting strengths of a particular parent and not be adversely affected by that parent's deficiencies, while another child may be harmed by that same parent's deficiencies and may be unlikely to benefit significantly from that parent's strengths.

Several reviewers of the 2003 Benjamin and Gollan text called attention to significant factual errors and, in strong terms, expressed concerns about the recommended procedures for custody evaluation, including, but not limited to, destroying records, not interviewing children of pre-adolescent age and younger, and crafting preliminary reports prior to reviewing third party information and interviewing collateral sources. In the JFPP article alluded to above, Benjamin and his colleagues re-stated the indisputably inaccurate position that the protocol outlined in 2003 does not require a departure from legal precedents. In the 2007 JFPP article, discredited procedures are extolled and neither the assertions nor the opinions of critics are addressed. The peer review process failed. We are left to wonder how often the process fails.

Supply and Demand

The effectiveness of the peer review process is dramatically influenced by the principle of supply and demand. First, and most obviously, the publication of professional journals is a capitalistic endeavor: The objective is income production for the publishers. Journals require a consistent flow of manuscript submissions in order to survive and those manuscripts must be interesting to the readership. Though dumbing-down is not usually a problem, as it sometimes is when professionals write articles intended for those outside their respective professions, all journal editors are concerned with the interests and perspectives of their subscribers. When members of editorial boards rate a submitted manuscript, one of the criteria is the suitability of the manuscript for the specific journal to which it has been submitted.

The realities of production schedules influence the quality of that which appears in peer-reviewed professional journals, particularly when journals produce theme issues. When editors wish to create a journal issue in which all the articles will address a specific topic, the editors typically contact professionals whose previous work suggests that they have an interest in and possess expertise in the topic that is to be the focus of the planned issue. Once an issue has been planned and it is anticipated that quality submissions will be arriving from authors A, B, C, D, E, and F. If author D's submission arrives close to a non-flexible deadline and if reviewers suggest extensive revisions, attending to those revisions may not always be practical in light of the publisher's often unyielding production schedule. The sage advice of the author's peers ' that significant modifications be made ' may be subordinated to the publisher's demand that all manuscripts intended for the theme issue be print-ready by a rapidly approaching deadline.

What Do They Pay You to Review This Stuff?

Nothing. So what's the inducement? Reviewers get to be the first to read manuscripts that address issues in the reviewers' particular areas of interest. For the most part, this contributes to effective reviewing. Reviewers are likely to be reading manuscripts that deal with topics with which the reviewers are intimately involved and in which the reviewers have an active interest. There is, however, a disadvantage. Reviewers, whose objectivity is at least somewhat dependent upon not knowing whose works they are reading, often find that clues to authorship abound.

The Blind Are Often Perceptive

Obstacles placed in the paths of the blind are usually detected. When blind reviewers wish to identify the authors whose manuscripts they are reviewing, the only obstacle between curious reviewers and the goal of identifying writers is that writers' names do not appear on manuscripts. Unless writers are new to the field, clues to their identities are likely to be plentiful. Just as artists can often identify unsigned works created by colleagues with whose work they are familiar, writers in the mental health field can often be identified by their writing style, the treatises that they cite, and the positions that they take on issues that are the subject of debate. The fact that reviewers can often identify the authors whose manuscripts they are reviewing decreases the objectivity of the peer review process.

Conclusion

It is not our contention that the peer review emperors have no clothes ' just that their sartorial splendor is not what they would have us believe. Constructive skepticism should not be abandoned in the face of the words 'peer reviewed.'

David Martindale, a member of this newsletter's Board of Editors, board certified in forensic psychology by the American Board of Professional Psychology, served as the Reporter for the Association of Family and Conciliation Courts' Model Standards for Child Custody Evaluation and is the co-author, with Jonathan Gould, of The Art and Science of Child Custody Evaluations. He can be contacted at [email protected]. Jonathan Gould is board certified in forensic psychology by the American Board of Professional Psychology. His practice is focused on family law related matters. He is a principal, with David Martindale, in the national consulting group, Child Custody Consultants. He can be contacted at [email protected].

In this article on peer review, we hope to create for the reader a healthy skepticism about the process, and shed light on assumptions that we believe are often made by colleagues, attorneys and judges about the academic rigor and scientific integrity of the endeavor. We begin with a fundamental truth that may appear to many readers to have been borrowed from the collected quotes of Yogi Berra: The quality of peer review is inextricably tied to the quality of the peers conducting the review.

Who Is a Peer?

In 1986, in Batson v. Kentucky , 476 U.S. 79, the United States Supreme Court weighed in on the meaning of 'peer' as used in the jury selection process. In doing so, the Court drew upon a decision handed down in 1879. In Strauder v. West Virginia , 100 U.S. 303 (1879), the Court defined peers as 'equals of the person whose rights [the jury] is selected or summoned to determine; that is, [a jury] of his neighbors, fellows, associates, persons having the same legal status in society as that which he holds' (at 308).

To the best of our knowledge, no defendant in a criminal trial has been successful in insisting on a jury of 'his neighbors, fellows, associates, [and] persons having the same legal status in society as that which he holds.' If put to the test, the inaccuracies in the Court's definition would quickly become apparent. A prisoner who commits murder while incarcerated is not likely to be tried by a jury of fellow convicts ' his 'neighbors, fellows, associates [and] persons having the same legal status in society as that which he holds.' It seems that in law, the notion of a peer has been loosely construed.

Peer Review on the Periphery

In the process of establishing expertise, professionals whose publications have appeared in peer-reviewed journals or in texts that have included a peer review element in the editorial procedure are quick to point this out. In doing so, they indirectly suggest that peer review is a procedure that yields scholarly contributions. It turns out that this is true only if their peers are scholars. The rights of the uninformed and misinformed to publish are protected by our Constitution.

When individuals organize for the purpose of publishing a journal, it is more likely than not that they will select an editorial board whose members are interested in the same professional issues as those who initiated the project and who hold similar perspectives on those issues. The editorial board and the executive editor of the journal often assume responsibility for choosing colleagues who will conduct reviews of the manuscripts that are being considered for publication in the journal.

The editorial board, the executive editor, and the reviewers function as gatekeepers with regard to manuscripts that are submitted. The first level of gate-keeping lies with the editor who receives a manuscript and must decide whether to forward the manuscript to one or more reviewers for comment. The editor presumably performs an initial review of submitted manuscripts and rejects those that are deemed unworthy of consideration. There are no standards or guidelines about how to determine which manuscripts are appropriate for further review and which are immediately rejected. The quality of gate-keeping at this initial stage is determined by several factors, only some of which will be identified here.

Editors may not read submitted manuscripts thoroughly. Editors do not always have the requisite scholarly background. Editors and editorial board members may have political agendas that influence their decision-making. Author names and affiliations appear on the manuscripts that are reviewed by editors, affecting the manner in which the gate operates. When manuscripts are passed along to blind reviewers, the personal and academic shortcomings that affect editors are likely to affect the assigned reviewers.

Just as mainstream biologists and geologists tend to view creationist explanations of the development of homo sapiens, the Grand Canyon, etc., as unsupported and unsupportable, mainstream mental health professionals tend to view certain concepts as equally unsupportable. Among mental health professionals, some believe that from the moment of birth all life events are stored in the brain in what can most easily be described as photographic form. Those who hold this view believe that in the eidetic memory of every child are images of all the child's life experiences.

If a group of such professionals were to establish the Society for the Study of Infantile Eidetic Memory and if the Society were to publish a journal, it is quite likely that all those reviewing contributions to the journal would be believers in infantile eidetic memory. Believers might deem acceptable a research article that non-believers would reject. At the Web site of the Eidetic Therapy Institute, www.eidetictherapy.com, readers are informed that an eidetic image 'contains an accurate record of important life experiences, reveals the source of long standing problems and uncovers the pathways to health and well being.'

We delude ourselves when we conceptualize peer reviewed journals as scholarly periodicals requiring that each article submitted for publication be judged by an independent panel of scholarly or scientifically-minded peers, who will reject submitted articles that are lacking in scholarship. The lofty goals of the peer review process (sometimes referred to as refereeing) often are not achieved.

The purpose of peer review is to require that authors meet the standards of their respective disciplines, but what if those who run a particular journal and those who submit articles are all members of a generally unrecognized discipline? Won't contributors to the Journal of Unsupported Theories, the Journal of Sophistry & Silliness, and the Journal of the Wildly Whimsical all assert that their published works have been peer reviewed? Members of the editorial boards of journals are ordinarily well-respected members of their respective fields of study; however, not all fields of study are worthy of respect.

Peer Review and the Mainstream

Even among psychologists who are not members of the American Psychological Association (APA), it would undoubtedly be recognized that the APA represents mainstream psychology. One would be likely to presume that texts published by the APA have been peer-reviewed and that in the peer review process factual inaccuracies would have been identified and removed prior to publication. Not so.

In 2003, the APA released a text, authored by Benjamin and Gollan, outlining custody evaluation procedures in which the authors opine that the erasure of tapes made during interviews 'prevents the opposing counsel from using contemporaneous material out of context during a later cross-examination at deposition or trial.' We believe that mainstream psychologists (and the APA) hold the view that in the field of forensic psychological assessment, methodology is dictated by the philosophy that everything an evaluator does must be open to scrutiny ' even by those who might be motivated by a desire to discredit the evaluator's findings. Destruction of records enables experts to insulate themselves from appropriate scrutiny.

Perhaps the stream is being re-directed. Psychology's national organization has elected to remove from its ethics code a previously strong statement about the creation and preservation of records. In its now obsolete 1992 ethics code, Ethical Standard 1.23(b) reads as follows: 'When psychologists have reason to believe that records of their professional services will be used in legal proceedings ' they have a responsibility to create and maintain documentation in the kind of detail and quality that would be consistent with reasonable scrutiny in an adjudicative forum.' No similar admonition appears in the current (2002) version of the ethics code.

There is reason for concern that the APA is about to take another step back from its 1992 position. APA's 1994 Guidelines for Child Custody Evaluations in Divorce Proceedings will soon be replaced by a document entitled Guidelines for Evaluating Parental Responsibility. A draft of the document, made available for comment, contains the following statement: 'When working in a judicial forum, psychologists are typically [emphasis added] expected to preserve evidence that underlies opinions.' The 1994 guidelines contained the following unambiguous statement: 'All raw data and interview information are recorded with an eye towards their possible review by other psychologists or the court, where legally permitted.'

Recently, in the peer-reviewed Journal of Forensic Psychology Practice (JFPP), Benjamin and his colleagues asserted that the protocol outlined in the 2003 text alluded to above 'does not compel a departure from firmly established legal precedents across jurisdictions for evaluating parental capacity ' ' It seems to us that the destruction of records does 'compel a departure from firmly established legal precedents.' Additionally, in the JFPP article, Benjamin and his colleagues remind readers that their evaluative methodology limits 'the children's involvement in the process only to the structured parent-child observation sessions.' Children are not interviewed. Benjamin practices in a state (Washington) in which, by statute, decisions in custody and access matters must take into account 'the wishes of a child who is sufficiently mature to express reasoned and independent preferences as to his or her residential schedule.'

In adjudicating disputes concerning custody and access, the courts in 39 states, in the District of Columbia, and in Puerto Rico consider the preference of children who are deemed to be of sufficient age and maturity. Additionally, the Uniform Marriage and Divorce Act (UMDA) mandates consideration of children's wishes with respect to their custodians. Though children's expressed preferences are not determinative, they are an important factor that cannot be considered unless children have been interviewed.

If psychologists heed the admonition of the APA, as expressed in its 1994 custody guidelines, that '[t]he focus of the evaluation is on parenting capacity, the psychological and developmental needs of the child, and the resulting fit,' it logically follows that children must be evaluated. It is generally recognized that one child may benefit from the parenting strengths of a particular parent and not be adversely affected by that parent's deficiencies, while another child may be harmed by that same parent's deficiencies and may be unlikely to benefit significantly from that parent's strengths.

Several reviewers of the 2003 Benjamin and Gollan text called attention to significant factual errors and, in strong terms, expressed concerns about the recommended procedures for custody evaluation, including, but not limited to, destroying records, not interviewing children of pre-adolescent age and younger, and crafting preliminary reports prior to reviewing third party information and interviewing collateral sources. In the JFPP article alluded to above, Benjamin and his colleagues re-stated the indisputably inaccurate position that the protocol outlined in 2003 does not require a departure from legal precedents. In the 2007 JFPP article, discredited procedures are extolled and neither the assertions nor the opinions of critics are addressed. The peer review process failed. We are left to wonder how often the process fails.

Supply and Demand

The effectiveness of the peer review process is dramatically influenced by the principle of supply and demand. First, and most obviously, the publication of professional journals is a capitalistic endeavor: The objective is income production for the publishers. Journals require a consistent flow of manuscript submissions in order to survive and those manuscripts must be interesting to the readership. Though dumbing-down is not usually a problem, as it sometimes is when professionals write articles intended for those outside their respective professions, all journal editors are concerned with the interests and perspectives of their subscribers. When members of editorial boards rate a submitted manuscript, one of the criteria is the suitability of the manuscript for the specific journal to which it has been submitted.

The realities of production schedules influence the quality of that which appears in peer-reviewed professional journals, particularly when journals produce theme issues. When editors wish to create a journal issue in which all the articles will address a specific topic, the editors typically contact professionals whose previous work suggests that they have an interest in and possess expertise in the topic that is to be the focus of the planned issue. Once an issue has been planned and it is anticipated that quality submissions will be arriving from authors A, B, C, D, E, and F. If author D's submission arrives close to a non-flexible deadline and if reviewers suggest extensive revisions, attending to those revisions may not always be practical in light of the publisher's often unyielding production schedule. The sage advice of the author's peers ' that significant modifications be made ' may be subordinated to the publisher's demand that all manuscripts intended for the theme issue be print-ready by a rapidly approaching deadline.

What Do They Pay You to Review This Stuff?

Nothing. So what's the inducement? Reviewers get to be the first to read manuscripts that address issues in the reviewers' particular areas of interest. For the most part, this contributes to effective reviewing. Reviewers are likely to be reading manuscripts that deal with topics with which the reviewers are intimately involved and in which the reviewers have an active interest. There is, however, a disadvantage. Reviewers, whose objectivity is at least somewhat dependent upon not knowing whose works they are reading, often find that clues to authorship abound.

The Blind Are Often Perceptive

Obstacles placed in the paths of the blind are usually detected. When blind reviewers wish to identify the authors whose manuscripts they are reviewing, the only obstacle between curious reviewers and the goal of identifying writers is that writers' names do not appear on manuscripts. Unless writers are new to the field, clues to their identities are likely to be plentiful. Just as artists can often identify unsigned works created by colleagues with whose work they are familiar, writers in the mental health field can often be identified by their writing style, the treatises that they cite, and the positions that they take on issues that are the subject of debate. The fact that reviewers can often identify the authors whose manuscripts they are reviewing decreases the objectivity of the peer review process.

Conclusion

It is not our contention that the peer review emperors have no clothes ' just that their sartorial splendor is not what they would have us believe. Constructive skepticism should not be abandoned in the face of the words 'peer reviewed.'

David Martindale, a member of this newsletter's Board of Editors, board certified in forensic psychology by the American Board of Professional Psychology, served as the Reporter for the Association of Family and Conciliation Courts' Model Standards for Child Custody Evaluation and is the co-author, with Jonathan Gould, of The Art and Science of Child Custody Evaluations. He can be contacted at [email protected]. Jonathan Gould is board certified in forensic psychology by the American Board of Professional Psychology. His practice is focused on family law related matters. He is a principal, with David Martindale, in the national consulting group, Child Custody Consultants. He can be contacted at [email protected].