You’re right Anita, data and software based workflows are quite quickly deanonymised. One option would be to drop anonymity (and have authors publish a preprint before review). Until everybody can agree on that, I think AGILE’s current approach is a compromise: the regular reviewer still can decide not to dig too deep to not impact their view, while the reproducibility reviewer can look into the details.
Hi, I’m late to the discussion, but as I am preparing my short paper submission I was asking myself the same question. Inluding a link to the project code on github in the paper will reveal my identity in the URL itself, and even mentioning the name of the project will reveal my identity with a single Google search.
Should such information be removed for the initial submission? As far as I understood, the reproducibility review takes place in a second step after the scientific reviews, so this could be then done on the basis of the de-anonymized version?
Regarding the general discussion: Improving scientific practices is great, but when it comes at the price of sacrificing other best practices that were introduced for a reason (blind reviews), on should maybe hesitate for a moment and ask whether that is a good idea. Maybe the problem could be solved with a two step review process (anonymous scientific review, then de-anonymized reproducability review with veto only for such issues), which I thought was how this was handled anyway.
Hi Florian! Thanks for joining the AGILE Discourse!
Short papers will not be reviewed for reproducibility - we don’t have the bandwidth for that yet.
Regarding the anonymity: even for full papers, authors do not provide all information to stay anonymous, that is why we have the reproducibility review after the submissions are accepted. Anita’s case just shows that is not trivial to create an anonymous deposit, and the authors should have been more careful.
More generally: I understand where you’re coming from, but the advantages of double blind reviews are seen quite critically by more and more researchers. Authors can often easily be deanonymised (via previous work, for example) especially in small disciplines, and it has been shown that reviewers are more friendly and fair when they cannot hide behind anonymity. That said, I am a fan of the idea that tenured researchers should sign their reviews, while early career researchers may stay anonymous and have the opportunity to call out bad work when they might fear bad outcomes because they review a senior persons work.
Thanks for the quick reply, now I understand the process (or could confirm that my previous understanding was in fact correct, but incomplete)
Also thanks for the comment re the general discussion - I think I am with you on this. It would be interesting to continue this discussion IRL sometimes…
Happy to continue a discussion here as well. I think it would be great to discuss the anonymity in the reviews at AGILE. In fact, I have many ideas how AGILE could serve its community even better and following more modern scholarly communication practices (clear preprint/postprint policy etc.). So, if you’d like to bring your question towards the Council for example, count me in for the discussion!
I like that idea. I wonder if it would also have other effects, e.g. if senior researchers hand down review work to juniors researchers, would they still do so if they have to put their name on it?
I have ambiguous view on the handing over of reviews to junior researchers. I’d say this behaviour is great if it is transparent, i.e. the editor knows who contributed how to the review. That way the editor may in the future directly invite the junior person. But I do think that the senior researcher in this scenario should take be responsible to give advise and ensure the review’s quality.