A news article in the upcoming edition of Nature highlights a new kid on the block in the battle for the publishing system of the future – it’s called Rubriq, a company that wants to “outsource” the review process from the normal publication process. Very similar to what Peerage of Science provides, but as a commercial author-pays model. The article reports that
Charging authors an estimated US$500–700 for its service, the firm plans to offer a standard-format anonymized review, and is currently testing its concept with publishers including Public Library of Science (PLoS), Karger, F1000Research and Wiley, as well as more than 500 reviewers.
Of that money, $100 each will go to the reviewers, the rest is used for administration (and profits, I suppose).
I don’t really know how I feel about “commercializing” the review process (Peerage of Science might ADD: or might not, see comments be a better model in that respect), but I do think it makes a lot of sense to dissect the review process from the journals themselves. Multiple review rounds have become the norm rather than the exception, leading to an immense waste of time in the current system. Plus, paying for the review might even have the positive side effect that reviewers and the review editors feel more strongly obliged to deliver quality, I would imagine that authors won’t be happy to pay $500 to receive sloppy or factually wrong reviews. I wonder if you can submit a new version if you don’t like your review …
Still, it remains to be see whether editors and publishers accept the new service – the article cites a source as saying that
journal editors who were wedded to their own peer-review standards “seemed more likely to use one another’s toothbrushes than their review formats”
theoretical ecology 
Peerage of Science also commercializes the review process, it’s just that they charge the fee to journals or publishers rather than to authors.
And while authors might well not be happy to pay money to Rubriq to get cursory or sloppy reviews, I don’t see how this provides an incentive to Rubriq reviewers to be conscientious. Rubriq very much emphasizes speed and standarization, and as far as I can tell only requires reviewers to fill out a standardized form (though they can also make comments if they wish). Personally, if I were a reviewer for Rubriq, I’d do exactly what I was required to do in order to get paid. I’d do that much honestly and conscientiously–but I wouldn’t do a jot more. My “incentive” as a Rubriq reviewer is to do the minimum required of me (and if authors don’t like that, too bad for them!) It’s early days, and maybe I’ll turn out to be very wrong. But I think Rubriq authors would be well-advised to keep in mind the dictum that “you get what you pay for”! And if you’re only paying $100/review, you’re not going to get much, certainly not from people wouldn’t do the review unless they were getting paid.
This is something Owen Petchey and I were often asked about when we were developing our PubCreds proposal. Our response was that, in a PubCreds world, editors requesting reviews had the power to deny “payment” (in the form of PubCreds) for any review they found too sloppy, superficial, or slow to be useful. So Owen and I were not trying to speed up or standardize peer reviews, or reduce peer review to filling out a checkbox-type form. And we retained an impartial arbiter–the editor–to determine which reviews are good enough to merit “payment”.
I doubt that I’m alone in feeling this way. It’s a well-established result in behavioral economics and psychology that, if you give people a monetary incentive to do something they previously used to do out of a sense of generosity or obligation, they mostly stop feeling that generosity or obligation and just do it for the money. Indeed, I suspect that even a PubCreds-type system would at least somewhat degrade the remaining professional obligation people feel to do their “fair share” of reviewing. Which is why Owen and I tried to design PubCreds to *force* people to do their fair share, and to do it well. If you’re going to propose a monetary (or monetary-like) system to replace professional norms and obligations, I think you need to propose one that will produce equally good outcomes. I’m skeptical that Rubriq will do that, but of course time will tell. I’ve been wrong before!
One more thought. The Rubriq people seem to think that differences in review formats among journals are a problem. But in trying to standardize review formats, I have the strong impression that they’re also trying to *change* review formats to a format that’s very quick and easy for reviewers to complete and that doesn’t require reviewers to make lots of comments. Which strikes me as complaining that different gourmet restaurants have different menus–and suggesting that all gourmet restaurants be replaced by McDonald’s.
LikeLike
Hi Jeremy,
thanks for your thoughts.
Seems you’re right about the commercial model of POS, I corrected the text in this respect. It’s really hard to understand how this exactly works though. They say something about being a for-profit company on their website, but I couldn’t find any information about how much they actually charge, and how (per-article or flat rate). Seems the journals pay an undisclosed fee? Any information on that?
About your other points: I would like to split what Rubriq is doing into several aspects:
1) Dissecting the review process from the particular journal
2a) Authors pays for review
2b) Reviewers get money
I think 1) is a sensible idea, for the obvious reasons that it saves a lot of time, I don’t see any problem with that.
About 2a, 2b (which seems to be your main critique): I think you have some points here, but I also think a lot of the problems you raise (McDonald) are not connected to the payment structure as such, but rather to a problem many new and general (OA) systems are facing, namely that they don’t have a big reputation and therefore the general quality of submissions and reviews will be lower than, say, what nature or EL are getting. So, to get this out of our mind, let’s for the moment assume that Ecology Letters would implement 2a, 2b, how would we think this plays out?
About 2b, rewarding reviewers: these psychology studies on, e.g. charging late-coming parents in the kindergarten leading to more late-comers are definitely something to keep in mind, but I’m not totally convinced that a financial reward is necessarily detrimental to the motivation of reviewers. I think it would depend a lot on how this reward is communicated. If you say: “here’s your payment” then this might go in the wrong direct, but if you said “your service is invaluable to the scientific community, nevertheless, we want to give you a small present to show our appreciation”, then I think you perceive the money in a quite different light and it can be motivating. In general, I don’t think the reviews received by EL would get worse if 2b) they payed people $100 compensation for their time.
About 2a, author pays: here, I actually see more potential problems. Paying could be a filter against submitting premature papers – I have heard people saying “I don’t really have good ideas about how to improve this paper, let’s submit it to journal X, I don’t think they’ll take it but maybe we get some good ideas from the reviewers”. If people have little money, you might reduce this behavior. However, if they have a lot and 2a) is implemented, more people might feel it’s OK to submit a paper without any chances of acceptance to EL, pay $500 for that, and in return get some good reviews. I’m not sure how common this behavior would be, but I think it’s something to watch out for.
I wonder whether PubCreds are so different from money in that aspect, wouldn’t the same psychological arguments also apply to PubCreds, doing a minimum effort to get your cred?
I know many economic journals have 2a (and partly 2b) installed for quite a while (e.g. http://www.econ.ucsb.edu/~tedb/jfees.html ), would be interesting to hear from economists about the experience with that.
Finally, regarding how to incentivize quality: I agree that some sort of feedback about amount and the quality of the review would be good, aside of money, and PubCreds would at least solve the amount problem. Not sure about the quality. Also, who’s to judge quality? Authors will tend to bring in their feelings about the evaluation into their assessment of quality. On the other hand, I’m not sure whether an editor can really assess the quality of a review either – some people are really good at writing convincing and eloquent nonsense in reviews, and it is often more easy to spot this for the authors or other people that have read the paper in detail than for an editor that doesn’t have time to dive into every submission he’s handling. So, maybe give it to the crowd, by some possibility to comment and rate reviews, like in Stack Exchange?
LikeLike
Hello Florian and Jeremy,
I am one of the founders of Peerage of Science (and a behavioural ecology postdoc), and principal designer of the service. Apologies if the website is not clear about the business model; a complete website redesign is coming out this month, but in the meantime I am happy to explain:
Peerage of Science is organized as a company, yes. The service is absolutely free for scientists (company is legally required by its own rules to seek best interest of scientists). The business model is to charge a fee from subscribing journals, just like the old-style submission system providers (e.g. Manuscript Central), but with more rational pricing model. Journals get access to the system, can track any peer review process, and can send direct publishing offers to authors. Because the service is billed by a pre-paid quota of publishing offers that are “consumed” only when authors accept a publishing offers, the system is cost-effective for journals too.
How much Peerage of Science charges depends on the contract – the maximum “list price” is 400€ / offer (remember that in the old system journal has to process at least 5 manuscripts in order to get one to publish, while in Peerage of Science they only pay when they get stuff to publish). However, Peerage of Science gives big discounts in return for mutually beneficial collaboration – if the journal endorses the service on their own website or if they declare it a preferred channel for submissions. If a journal is visionary enough to ditch the old systems completely, and use exclusively Peerage of Science (and thus, among other things, does not demand you as an author to give them exclusive right of consideration upon submission like journals currently do), then prices will be much, much lower.
However, as an author you can always create a link and one-time username giving access to peer reviews of your manuscript, and give that link and username to any editor of any journal you choose when submitting the traditional way. Of course it is up to the editors whether they wish to use the reviews. This service is free, for both the author and for the journal receiving the link and password from you.
About standardizing review formats: I obviously have a vested interest so I should not comment the “science of the scorecard” of our competitor. But I will comment regarding the standardization used in Peerage of Science. I recommend you go ahead and try being a reviewer in Peerage of Science: the review has to be written in the form of a coherent, readable essay, max 1000 words, must be organized under headings “Introduction”, “Merits”, “Critique” and “Discussion”, and it will be judged and scored by other reviewers. Now, in addition to that your are free, and encouraged, to add “Additional Comments” -section where more informal suggestions, spelling errors, line-numbered comments etc go, with no word-limit.
If you do try being a reviewer, and get a decent PEQ score from your colleagues, and still feel that it was a quick and easy McDonalds-version of proper peer review, then I am willing to consider the proposition that standardizing peer review is not a great idea. Until then, I shall continue to hold my view that it is a big improvement over the hastily scribbled informal lines of “peer review”, which I am sure you too have sometimes received from established journals.
Jeremy, your old invitation still stands. Florian, I will send you one via email now, you are most welcome.
Kind regards,
Janne
LikeLike
Hi Janne,
thanks for your comments and clarifications – and thanks also for the invitation, I’m curious to give it a go … in general, I think it’s positive that so many new things are tried out at the moment (hope the best ideas stick). As I said, the main innovation I see in both POS and Rubriq is that they separate the reviews from the journals, if that would work and enough journals accept the reviews it would really be a big time saver for the community.
LikeLike