By Prof Zoltan Dienes

In April we launched Peer Community In Registered Reports (PCI RR), where a Registered Report in any discipline can be submitted (by linking to a preprint), and refereed, editorially evaluated and (potentially) eventually accepted  – “recommended” in PCI speak. The process and outcome are free to authors and free to readers. No institution need to pay a fee for these freedoms; freedoms we take to reflect the openness that is the sine qua non of science. See here for a talk by Corina Logan about PCI RR.

Because PCI RR links to pre-prints, the recommended manuscript is not published by PCI RR (though you will have a citable PCI RR recommendation with a DOI). That means an author may (if they wish, though they may not) have the same manuscript published in a mainstream journal. Indeed, we have 19 PCI RR-friendly journals, which guarantee acceptance of our recommended manuscripts without further peer or editorial review (given certain technical conditions are met, such as being in the right discipline and payment of any APCs that the journal would normally charge). Notice it is the author that has the choice of journal, reversing typical power arrangements.

A Registered Report is an article type where in-principle acceptance (or rejection) of a manuscript occurs before results are known. This ordering of acceptance relative to results is aimed at dealing with various biases that can corrupt the scientific process. When the chain of reasoning from theory to predictions and from data to predictions is constructed in light of the data, distortions can occur: for example, arbitrary predictions that do not actually follow from a theory can be stated (because they happen to match the data); or p-hacking or B-hacking (Bayes factor hacking) can be used to create pseudo-evidence for a preferred outcome. How many papers have you read where the “predictions” at the end of the introduction seemed somewhat arbitrary with respect to theory and yet were strangely confirmed? How many correlations with small N were just significant? A further bias is avoided by Registered Reports: that of editors or reviewers rejecting a paper because they did not like the results. As Chris Chambers often puts it: what part of the scientific process should we never have control over? Yet what part of the scientific process often determines publication – and hence can make or break careers? The results.

One advantage of the Registered Report is that reviewers and editors are collaboratively involved in designing the study – at a point in time when their expert advice can still be seriously considered and acted on if useful. Relatedly, once the procedure and analyses have been in principle accepted, a publication is virtually guaranteed. One does not have to chase various journals for months or years because of some aspect of the study that is regarded as weak by those asked to review. In my experience, the fact that advice from reviewers and the editors can inform all aspects of design and analysis (in the absence of knowledge of results), really does change the dynamics of the relation between authors, editor and reviewers.

Do Registered Reports actually reduce bias? In a recent pre-registered study, Registered Reports confirmed the stated predictions less than 50% of the time – yet matched non-Registered Reports confirmed predictions over 90% of the time. Further, Registered Reports are associated with a higher reproducibility of main results from the original data than regular articles; and methodological rigour and quality of Registered Reports are rated substantially higher than non-Registered Reports. I take all this to be preliminary evidence that Registered Reports do in practice reduce biases in the scientific process.

PCI RR overcomes some weaknesses in the current way Registered Reports are managed in journals. As previously mentioned, Registered Reports are completely open and free in PCI RR. Notice this addresses some concerns about equity: financial resources arising from social status (because of race, gender, or institution) are not necessary for having one’s research publicly recommended by the same Registered Reports editors who work for leading journals. (How easy is it for you to pay the APCs for prestigious journals?) PCI RR introduces some other unique key innovations.

With a programmatic Registered Report, a series of related studies can be accepted in principle before the results are in. Each of these studies can then be conducted without further ado, each potentially leading to its own paper. This considerably speeds the process by which a programme of research can be published as a series of Registered Reports.

With a scheduled review, the authors submit a short one-page template-based “snapshot” of their proposed study. The recommender then sends the snapshot to potential reviewers and organises the review process for a fixed future date nominated by the authors. During the intervening time the authors prepare the full manuscript. The aim is to substantially reduce the initial review time.

We operate a graded system of bias control. That means that while the full level of bias control applies to cases where the data does not yet exist, many of the benefits of the Registered Reports process may be still applicable even when the data exist – but for example, cannot be accessed yet; or can be accessed in principle but has not in fact yet been accessed by the researchers, and so on. In each case, we try to control bias as much as possible, while explicitly recognizing there are different risks in different situations.

People interested in becoming a PCI RR Recommender take a two-hour test after reading the extensive guidelines for Recommenders. As far as we know, formal training of Recommenders (editors) is a unique feature of PCI RR.

So far we have had 10 submissions, which I think is a considerably higher rate than for Registered Reports at any journal. If you have a good idea for a Registered Report, or would like to become involved as a Recommender, you know who to call!

Zoltan Dienes is a Professor of Experimental Psychology at the University of Sussex. He is the author of the influential book “Understanding Psychology as a Science,” and is regularly invited to provide Bayes and Registered Report workshops all over the world. He produced the first online Bayes factor calculator in 2008, and has published extensively on Bayes factors including how to get the most out of non-significant results.

Find out more about the School’s Psychological Methods Strategic Focus Area and our commitment to Open Science.