Law Reviews, Peer Review, and Cheese Knives

According to Professor Christopher Zorn of Empirical Legal Studies, law reviews are “terrible.” And he presents a list of grievances. I’ll get to those, and I’ve already discussed some of the recent “State of the Law Reviews” discussion here and here. But before the grievances, a more fundamental point: Professor Zorn doesn’t make clear his understanding of the purpose of law reviews so it’s impossible to judge whether law reviews are “terrible” or “great” at satisfying that purpose. You can’t call a cheese knife “terrible” just because you have a hard time cutting steak with it.

Let’s say the purposes of law reviews are to (1) get a bunch of ideas out into the universe, with a reliance on post-publication evaluation and sorting, (2) provide information that is useful to the bench, the bar, and scholars in some ratio, (3) have an article’s sourcing and arguments thoroughly checked by student editors, who effectively serve as volunteer student research assistants, and (4) give law students the opportunity to work closely with professors on current scholarship. On those fronts, I would say the law reviews are doing a decent job. But here are Five Reasons Law Reviews Are Terrible, according to Professor Zorn, with my comments:

Terrible Thing #1: “Carpet-bomb submissions” which is when “[o]ne submits one’s paper to literally hundreds of journals at the same time.”

Why is this terrible? Because science, history, and philosophy journals require single submissions. Okay, perhaps the fact that we do things differently is a sign that we should be skeptical of our methods. But Professor Zorn doesn’t explain what terribleness results from legal academics’ “carpet-bomb submissions.” Does it lead to lower quality? A less-practical focus? Worse writing? I’m not sure and he doesn’t explain. If anything, I’d imagine the carpet-bomb approach gets things published faster than a seriatim approach.

Terrible Thing #2: “Publication ‘review’” in which law reviews privilege the work of federal judges, professors at top-20 law schools and famous-ish law professors.

This criticism has some teeth. If, indeed, student-edited law reviews are favoring the “lesser” work of certain prestigious names and ignoring the “better” work of less-advantaged authors, then that sounds like a real problem. Maybe. Given the possibility of post-publication review, the “better” work will likely get picked up by some law review, so it will make it out into the marketplace of ideas where it can be read and discussed by other professors, practitioners, etc. But that brings us to….

Terrible Thing #3: Expediting, which is when professors leverage an offer at a “lower” journal into a fast review and possible acceptance at a “higher” journal.

Professor Zorn notes the process is inefficient. And it sure is. But think about why the process exists at all: Law professors want to move up in the rankings because they want their pieces to get published in more prestigious, highly ranked, big-name journals.

This fact should make us all skeptical that a switch to peer-review would eliminate “Terrible Thing #2”—an undesirable focus on prestige in article selection. Though law professors undoubtedly know more about the law, I’m not at all confident that we are more immune to the biases of prestige, status, and fame. [Quote removed from original post.] [Added text: If anything, students might be more willing to publish articles that upset the prevailing orthodoxy among scholars, the bench, the bar, society, etc.]

Terrible Thing #4: Someone once rejected a brilliant article because of a misplaced comma.

Or something like that. I’m really not sure. Dealing with student editors might sometimes be annoying—I’ve had long debates about Congress’ vs. Congress’s. But I’ve had those debates with professor editors and student editors alike.

Terrible Thing #5: The Bluebook is a “terrible style, seemingly designed for a long-gone era.”

Okay, lots of people hate the Bluebook. I happen to like the Bluebook. But if a professor-author is solely concerned with the quality of the ideas in her piece, then why not just focus above the line and let the editors do what they want with the footnotes/citations. Problem solved. Nothing terrible here.

It seems, however, that Professor Zorn does care about the formatting below the line because his complaint is not so much “I don’t want to worry about this stuff” but is more “I do worry about this stuff and they are getting it all wrong.” So, for example, he claims “cites to numbered references would be far superior” to “Ibid.” in law reviews.

Two things: First, a debate regarding the relative desirability of different methods for referring to already-cited work? As a lover of Bluebook minutia, I’m all for it. But I’m not sure this particular complaint cuts in favor of Professor Zorn’s claim that the peer-review process would be more painless than the student-editor process. Second, there’s no “ibid.” in the Bluebook; only “id.” (Yes, that’s sort of a Ghostbusters reference.)

Terrible Thing Bonus: “One sad consequence of all this terribleness is that law professors and law schools are not taken as seriously by other members of the university community as they might be.”

That may be true. I don’t know. But if it is true, then here’s what that tells me: Other professors are judging law professors by the perceived status of our journal process and not by the quality of our work. If the goal is to focus on the content of scholarship and not on secondary bells and whistles, then it seems these other academics have a ways to go.


3 thoughts on “Law Reviews, Peer Review, and Cheese Knives

  1. That you think peer review, especially double blind peer review, is a “secondary bell and whistle” of the production of knowledge leads me to question whether you know the precise point of peer review in publication. The basic fact is that peer review is considered in most every single academic discipline to be a key aspect of the production of knowledge. I know being a typical law prof you don’t get this, but the process actually improves the work being done. It’s not just having your work edited, law review style (checking cites and commas) by other professors, but instead having the substance and methods critically evaluated by other experts. This is considered critical by people in the social and hard sciences, as well as the humanities. Hell, even business reviews are run that way.

    But to my real point. You seem to think the first/primary point of law reviews is
    “(1) get a bunch of ideas out into the universe, with a reliance on post-publication evaluation and sorting.”

    Please, do tell me: where can I go to find these post-publication evaluations and sorting? I haven’t yet found “The Harvard Review of the Harvard Law Review”… is it obscure and not on Lexis-Nexis?

    If you tell me citations, well, then, that would be one of the exact same ways every other academic discipline does things, and there isn’t really any defense of law journals on this other than “but then more stuff gets out there!” In which case, get a blog or post it on SSRN.

    Most work sucks. Gatekeeper functions (though not perfect) separate the wheat from the chaff. Double blind peer review would also seriously mitigate other critiques, such as status being used as shorthand by those who are woefully inadequate to review articles (i.e. law students). But your quote from Nancy Leong just proves you’re not paying attention: in real academic disciplines a department doesn’t have its own journal (it pulls reviewers from everywhere), and in most cases the journals are double blind (so that the reviewers are white men don’t matter). But furthermore, you miss the point of the critique. It isn’t that you’re immune from biases of prestige, it’s that you’re in fact a qualified expert on the topic, which the students just aren’t. So whether they are “open to new ideas” (which is itself not an unalloyed good) is frankly irrelevant. Maybe they’re open to new ideas because they don’t have the training to discern good ideas from bad ideas, or an effective research design from one poorly executed?

    But we get it. You like easy publications, and the idea of not having to defend your ideas to other experts in the peer review process. And shunting a critical aspect of academic work onto law students. This is why even today people studying law and courts from academic disciplines consider publishing in law reviews either the kiss of death (pre-tenure, signal you can’t get your work taken seriously by experts) or a signal of throwaway work, either mediocre ideas or poorly executed research.

    The last point is true: we don’t take your non-peer reviewed work seriously, because instead of passing it through multiple rounds of reviews and revisions with experts, it gets the “review” of a few unqualified 24 year olds. When academics who don’t realize law journals–to my knowledge uniquely as a field–operate this way, their perception of the rigor of law professors plummets. Because peer review is intimately related to the quality of the work. Which means that the law profs whose work is good and rigorous enough to make it past peer review suffer (non-disciplinary) reputation costs. Of course, this is not all of them: more and more profs (even in your department!) are publishing in peer reviewed outlets, a lot of them trained not only in law. But fact of the matter is, when they’re honest, they are happy to admit they send their “good” or “real” (their words, not mine) work to peer-reviewed, academic journals. Which says all it needs to say.

    • Thanks for the comment Brad. And yes, I admit, I perhaps got a little carried away with the hyperbole with that “bells and whistles” line. Too cute by half.

      Anyway, on the merits, a few thoughts. First, there is already a lot of informal peer review going on before submission to student-edited journals. But let’s assume that a pre-publication gatekeeping/screening peer-review process resulted in fewer “bad” articles getting published. (Aside, I’m not even sure that’s true; what we’d really need to do is reduce the number of law journals and reduce the faculty/school-imposed requirements/expectations that professors publish. Otherwise, those journal slots are going to get filled with something! But in any event…) I’m just not sure what’s so bad about all those “bad” articles getting published. You used to be able to say that fewer trees would get killed for the paper, but nowadays everything is online anyway.

      And no, the post-publication review is not the Harvard Review of the Harvard Law Review. But you’re not that far off! The post-publication review just means that you put your article out there. People read it. They comment on it. They share it with their colleagues. They bash it. They write a response. Etc.

      Take a look at the Harvard Law Review webpage: Over on the right there’s an online forum where other scholars are invited to respond to and critique the articles previously published in the Harvard Law Review. So, yes, I know you were joking. But there sort of is a Harvard Review of the Harvard Law Review.

      Moreover, practitioners might actually read the article. Or legislators. Or folks working in government. Whatever. If they think it might be useful, they use it. If not, they ignore it. If they cite it their court filings, the judge either adopts it or doesn’t. That’s all a review of sorts. It’s not a gatekeeping review, but it’s a review.

      Anyway, it’s not as if the current system results in nobody having any sense of which law professors are doing interesting, insightful work and which are not. (Of course, we might not all agree on that, but that’s another matter.) Law professors have to defend their work to their peers, it’s just that more of that defense happens post-publication than pre-.

      The point is, if there’s some dud out there and it gets ignored, I don’t see the harm. It’s not like the New England Journal of Medicine publishing something that says “Drink Antifreeze to Cure the Common Cold.” That could cause a problem. A law professor writing a theory of the Erie doctrine that’s boring or trivial or glosses over some of the details? Eh. I’m not up in arms. I’m just going to skip it.

      All else being equal, sure, why not switch to peer review? But all else isn’t equal. I like the fact that law students, as part of their education, have a crucial role in the review and creation of legal scholarship. Cutting them out of that process would be a loss for them, and at this point that loss doesn’t seem worth the potential gains.

      • But not a single one of those actually gets to the key point drawn from effectively Every Other Academic Discipline: peer review makes research better. Now, if you want to say that almost all law scholarship is effectively just hand-waving and as such it can’t really be made better… well that’s you saying it, not me.

        Again: the “post-publication review” you cite is absolutely no different than any other field. Of course things that get read get thought about, and occasionally even written about or cited. You can’t consider that to be the equivalent of actual pre-publication peer review, or even a substitute. Again: in no way is the legal field’s form of post-publication review any different, more rigorous, or stringent than other disciplines (and given the lack of empirics I would actually argue it’s less rigorous and less stringent–where’s the replication–though through no fault of its own).

        But you can’t have it both ways with regards to lack of harm of student review. You can’t say “it’s ok if shit gets through because it’s not the NEJM,” and simultaneously say you want practitioners and judges to be reading law review articles (to presumably affect how they practice and decide). One or the other, not both. (It also doesn’t get to the secondary point that tons of scholarly fields that aren’t involved in life-threatening research have peer review.)

        Also, the “outlet for law profs to publish.” Give me a break. This is the most self-serving shit ever. “Hey, we shouldn’t have to actually compete for slots at journals, so let’s have tons of journals and no peer review.” Combined with the fact that law profs (1) don’t actually publish that much compared to the most comparable disciplines, and (2) teach a lot less, it’s doubly self-serving.

        The point about students is bunk, and I can’t believe law profs are still making it. Somehow, students in graduate school learn how to write, do research, and in fact typically publish that research without serving as glorified copy editors. But, if you think that this is in fact integral to the legal education, then it logically follows that ALL law students should have the opportunity, no? If serving on law review is in fact a great and worthy, critical and important aspect of law school, then the fact that it is used as a selection mechanism so the top echelon of students (primarily based on 1L grades, which clearly are directly related to editing ability) can pad their resumes makes your reliance on this point problematic at best, hypocritical at worst.

        All else being equal? People are going to put quotes around the words “academic” and “scholarly” when it comes to law professors and the legal discipline as long as it uniquely maintains a form of publication that other fields consider–for a variety of good reasons–to be inferior and in fact anathema to the production of knowledge.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s