As both an author and member of a number programme committees, I’ve been reflecting on the recent decision of various academic conferences including ICSE 2018, to opt for double blinding of the review process. Essentially this means both the identify of the reviewer and the author are hidden; in the case of triple binding, which has been mooted, even the identity of one’s fellow reviewers is hidden.
But there remain many potential revealing factors e.g. the use of British or US English, the use of a comma or period as a decimal point indicator, choice of word processor and the need to cite and build upon one’s own work. Why not quadruple or even quintuple blinding?!! Should the editor or programme chair be known? What about the dangers of well-established researchers declining to serve on PCs for second-tier conferences? Perhaps there should just be a pool of anonymous papers randomly assigned to anonymous reviewers that will be randomly allocated to conferences?
Personally, I’m strongly opposed to any blinding in the review process. And here’s why.
Instinctively I feel that openness and transparency lead to better outcomes than hiding behind anonymity. Be that as it may, let’s try to be a little more analytical. First, what kinds of bias are we trying to address? There seem to be five types of bias derived from:
– personal animosity
– characteristics of the author e.g. leading to misogynist or racist bias
– the alignment / proximity to the reviewer’s research beliefs and values
– citation(s) of the reviewer’s work
– the reviewer’s narcissism and the need for self-aggrandisement
It seems that only the first two biases could be addressed through blinding, and this of course assumes that the blinding is successful, which in small fields may be difficult. Although I would never seek to actively discover the identity of the authors of a blinded paper, in many cases I am pretty certain as to who the authors are. And it doesn’t matter.
In my opinion, double blinding is a distraction, but one with some negative side effects. The blinding process harms the paper. As an author I’m asked to withhold supplementary data and scripts because this might reveal who I am. Furthermore sections must be written in an extremely convoluted fashion so I don’t refer to my previous work, reference work under review or any of the other perfectly natural parts of positioning a new study. It promotes the idea that each piece of research is in some sense atomic.
Why not do the opposite and make reviewers accountable for their opinions by requiring them to disclose who they are. Journal editors or programme chairs are known so why not the reviewers too?
Open reviewing would reduce negative and destructive reviews. It might also help deal with the situation where the reviewer demands additional references be added all of which seem tangential to the paper but, coincidentally, are authored by the reviewer! The only danger I foresee, might be that reviews become more anodyne as reviewers do not wish to be publicly controversial. But then this supposes I, as a reviewer, wish to have a reputation as a bland yes-person. I would be unlikely to want this, so I’m unconvinced by this argument.
So whilst I accept that the motivation for double blind reviewing is good, and I also accept I seem to be in a minority (see the excellent investigation of attitudes in software engineering by Lutz Prechelt, Daniel Graziotin and Daniel Fernández) but I think it’s unfortunate.
Thoughts on Empirical Software Engineering 
Thank you for taking the time to write about this important issue. I (a strong supporter of double-blind review) would like to respond to some of the points you made in your post. I found that this post conflates many issues unrelated to double-blind review with double-blind review, and I’d like to tease them out. A key point to understand is that most SE conferences used to use single-blind review, and the question before us (which has now been answered in the affirmative) was whether to switch to double-blind review. You can, of course, argue that we should use some other form of review, but that argument is in many ways orthogonal to the argument between whether author identities should or should not be hidden during review (a.k.a. single- vs. double-blind review).
> But there remain many potential revealing factors e.g., the use of British or US English,
> the use of a comma or period as a decimal point indicator, choice of word processor and
> the need to cite and build upon one’s own work. Why not quadruple or even quintuple
> blinding?!!
Yes, there are many factors that can reveal author identities. Blinding a paper is an imperfect process, but it obviously does a better job than not trying to blind a paper at all. Is the argument that if we cannot do something perfectly then we shouldn’t do it at all? That’s silly.
Whatever you mean by triple and quadruple reviewing is not relevant to the question of “should we hide author identities during review?” You can talk about hiding other information, but just because you reason another piece of information should not be hidden has no effect on whether hiding author identities has value.
> What about the dangers of well-established researchers declining to serve on PCs for
> second-tier conferences?
You and I can argue about the value of second-tier conferences, but what does this have to do with hiding author identities? Nothing.
> Instinctively I feel that openness and transparency lead to better outcomes than hiding
> behind anonymity.
Unless openness leads to bias. You know this.
> It seems that only the first two [of five] biases could be addressed through blinding,
I didn’t understand your five types of bias. But that’s irrelevant. Suppose we can only address two of five biases. Is your argument that we shouldn’t address those two because we cannot address the other three? That’s silly. And wrong.
> this of course assumes that the blinding is successful, which in small fields may be difficult.
Blinding is imperfect, but fairly effective.
“We find that [at ASE 2016, OOPSLA 2016, and PLDI 2016] 74%–90% of reviews contain no correct [author identity] guess.” [http://people.cs.umass.edu/~brun/pubs/pubs/LeGoues17cacm.pdf]
> As an author, I’m asked to withhold supplementary data and scripts because this
> might reveal who I am.
No, you are not. ICSE 2018 (and other conferences) allows you to submit anonymized data, scripts, code, etc. Yes, it’s more work to create these than not to create these. It’s also more work to write the paper than not to write it. You can similarly complain that conferences make you write a paper to get your work accepted. No one is asking you to withhold anything.
> Furthermore, sections must be written in an extremely convoluted fashion so I don’t
> refer to my previous work, reference work under review or any of the other perfectly
> natural parts of positioning a new study.
This is categorically false. It is no harder to write “We build on our prior work [7]” than it is to write “We build on Smith and Nguyen’s work [7].” You can cite references under review the exact same way you normally would (just omit author names). There is nothing that blinding your paper requires you to do differently, other than use your identity or institution identity to adversely affect the fairness of the review process. I’ll admit that it takes a little more work to blind a paper, but double-blind review in no way hurts you from properly representing the quality of your work.
> It promotes the idea that each piece of research is in some sense atomic.
Absolutely not.
> Why not do the opposite and make reviewers accountable for their opinions
> by requiring them to disclose who they are.
Many, many reasons, none of which are relevant to double-blind reviewing.
Exposing reviewer identities encourages quid pro quo. Junior (and perhaps non-junior) reviewers are unlikely to feel comfortable being honest in reviewing work of the colleagues on whose recommendation letters they rely to get tenure and promotion. And so on. But, again, completely irrelevant to the question of using single- vs. double-blind review.
The bottom line is, unless the burden of blinding a paper is so huge as to adversely affect the dissemination of research (it is certainly not) — and perhaps even if it were — the key thing that matters is “Does revealing author identities during the review process adversely affect the fairness of review?” The answer has been derived by science again and again, and it is a resounding “Yes!” When reviewers knew author identities, review scores for papers with male first authors were 19% higher, and for papers with female first authors 4% lower [S. G. Roberts and T. Verhoef. Double-blind reviewing at EvoLang 11 reveals gender bias. J. of Language Evolution, 1(2):163–167, 2016.] Reviewers with author information were 1.76× more likely to recommend acceptance of papers from famous authors, and 1.67× more likely recommend acceptance of papers from top institutions [A. Tomkins, M. Zhang, and W. D. Heavlin. Single versus double blind reviewing at WSDM 2017. CoRR, abs/1702.00502, 2017]. US reviewers were more likely to recommend acceptance of papers from US-based institutions [Gastroenterology, Bethesda, MD, USA. US and non-US submissions: An analysis of reviewer bias. JAMA, 280(3):246–247, July 1998]. (While these studies are not on software engineering venues, you either have to accept that these are representative of our field, or to make an argument that for some reason, software engineering reviewers are fundamentally more evolved beyond the effects of subconscious bias than reviewers in other disciplines, including other disciplines in computer science.) And if we accept these studies’ findings that bias exists and that double-blind review helps mitigate it, what possible argument can we give not to use double-blind review? I have to remove names from code before submitting it? It takes a little bit a effort to change “our” to “Nguyen and Smith”? I write behavior with a u? I think not.
I encourage you to take a look at our recent study of the effectiveness of double-blind review at ASE 2016, OOPSLA 2016, and PLDI 2016 [C. Le Goues, Y. Brun, S. Apel, E. Berger, S. Khurshid, and Y. Smaragdakis. Effectiveness of anonymization in double-blind review. Communications of the ACM, in press. http://people.cs.umass.edu/~brun/pubs/pubs/LeGoues17cacm.pdf Again, double-blind review is imperfect, and our study doesn’t look at whether acceptance outcomes were affected, but double-blind review is clearly far better at hiding author identities than single-blind review, and study after study has shown that bias stemming from knowing author identities corrupts the review process.
LikeLike
Excuse me if I will not be as long as the post itself, but nonetheless I would like to make my point.
There will always be a bias towards old, seasoned, well-known researchers: they will want to run the status quo as ever, their papers and reputation to be put before their work. That type of researchers will be much more likely to contrast the double blind review process at any cost.
The junior researchers (and from less high-up institutions) will be less likely to adverse this innovation, or process improvement. It seems much fairer, doesn’t take that much to implement and it shows that they’ll have a higher chance to get accepted.
Oh, and I am a junior researcher, from an OK institution, and I can see the bias towards of papers and authors in journals and conferences. Apologies if I make a complex issue into a old/new generation matter
LikeLike
Hi Sue! Thanks for this point. It’s very hard to know if this bias exists but it’s certainly a possibility. The idea with open review is that if the reviewer’s comments are ill-founded and revolve around the lack of seniority of the authors (rather than the weakness of authors’ ideas) then this would be exposed.
A related problem is suppose the reviewer is junior and the author senior. The lack of blinding might expose the junior reviewer to vindictive behaviour in the future (and frankly I don’t have a good answer for this possibility).
LikeLike