This commentary highlighted the lack of reproducibility as a long-lasting, systematical issue in ecology and evolutionary biology, where results heavily depend on statistical modeling and numerical simulations. The authors suggested a comprehensive guideline to include code review in the pre-submission, peer review, and post-publication process to ensure the validity and robustness of scientific conclusions. Together with other pioneers who advocate for reproducible research in psychology and computer science, the authors proposed a valuable and practical framework (i.e., 4Rs, flowchart for peer reviewers).

As an advocate for reproducible research myself, I agree entirely with the necessity and urgency to change the status quo in the publication process, which emphasizes and incentivizes too much on the scientific novelties, but comparably too little on the reproducibility. Despite being out of the scope of this paper, in an ideal world, scientific discoveries should be published only after stringent fact-checking and careful examination of the method. Before sweeping through the entire field of biology, the discipline of ecology and evolution and other computationally extensive disciplines could be a pioneering test ground for incorporating code review as a standard publication practice.

In light of the well-versed rationales and the recommendations in this paper and my alignment with the preference for reproducible research, I do not have any major "concerns" overall. Before going to minor technical comments, I would like to share some general thoughts that may interest the authors to discuss or further develop in the revised manuscript.

1. In the section Are results reproducible?
 

Although each R in the 4R guidelines is indispensable, the intrinsic demands are increasing from the first to the last R. A fair and feasible implementation of these principles could vary by discipline/subdisciplines. Reproducible results have, of course, the uttermost importance and should be ensured whenever possible. Yet, in some cases, fully reproducing results is practical. 

For example, evolutionary biology has relied on and/or gradually will rely more on insights obtained from high-throughput sequencing (comparative genomics, transcriptomics, and other omics). Due to the nature of its high-volume, sometimes high-dimensionality, computations involved are expansive not only in terms of computational time but also involves in the accessibility and availability of resources, including High-Performance Cluster (HPC), # of CPUs, memories, and storage. If reviewers were required to reproduce results in the peer review process, installation and configuration of a substantial fraction of bioinformatic software are not trivial, even if resources are provided/subsidized for the reviewer.

One possible solution to this (which may be outside the scope of this paper) is to watermark (e.g., using MD5 sum) all intermediate results during the computation and only examine the reproducibility of the "scaled-down" version of results.

2. In the section Is the code Reliable
 

Besides the reasons mentioned by the authors, many errors in code could often appear in user-defined analysis or helper functions. Toy examples, if not unit tests in the engineering fields, should at least be included to test if those functions are doing what they are expected to do.

3. On incentives for code-review club and reviewers
 

As most researchers are scientists (some may be statisticians) by training, requiring them to know and implement the best practices in coding could be too demanding, especially for the first author, who happens to be a bench scientist working on a highly independent project. Besides recommendations proposed by this paper (e.g., code review club, discussing up front potential authorship for code reviewers), I would go beyond and argue that code reviewers are entitled to authorship, especially if they contributed significantly in making code reviewable (Fig. 3)

On the other hand, given the current unfair workload that reviewers commonly face, reviewers should be offered the authorship opportunity as one of the incentives, especially if fact-checking is time-consuming. Although (again) may be out of the scope of this paper, this could be facilitated by journals that adopt a double-blind reviewing process. Authors' consent on granting anonymous reviewers should be asked before the manuscript is sent out to external reviewers, and editors must withhold the information from reviewers until the final acceptance decision on the manuscript to ensure the objectivity of reviewing process. To mitigate the possibility that authors are exploiting the reviewer's fact-checking work, authors should also agree that the reviewer will be entitled to authorships if they (1) find major discrepancies in the code that change the direction of the corresponding conclusions, (2) provides the evidence of their workflow (3) authors subsequently can reproduce the issues that reviewers postulated.

As the authors pointed out, designated "Data Editors" have become standard practices in some journals. Designated "Data Reviewers" may be on the horizon in the future. In such cases, authorship to data reviewers as incentives could be discussed, and it might be favorable not to grant authorship to data reviewers if they do not assess the manuscript's scientific novelty.

Minor comments:

4. Consider switching the orders of Fig.2 and Fig. 3 and updating the in-text reference
5. Line 183-184 https://github.com/pditommaso/awesome-184pipeline not found
6. Line 276-277 https://github.com/SORTEE/peer-277code-review/issues/8 not found
7. Line 352: add in-text reference "(Fig. 1)" after if code is indeed adhering to the R’s listed above.
8. Line 358: missing full citation of Stodden 2011, Light et al., 2014
9. Fig 1: "Code must be error-free": the language is vague, and I suggest revising it to "Is the code doing what it is supposed to do?"
 

The manuscript describes a process by which code-review in the field of ecology and evolution could take place. The authors suggest following the 4 R's (Reported, Run, Reliable, Reproducible). 

Throughout the paper the authors provide some suggestions for reproducibility and effective managment of package versions. A tool not yet mentioned that my be useful is the R package "packrat" which stores packages specific to the version as they are installed, can reload them across different machines and can be shared across users. I recommend adding this type of package to the manuscript as this (version control/reproducibility) is what it was built for. It would be nice to see a packrat snapshot submitted along with code to fully reproduce an analyses. 

Another tool which can greatly aid in reproduciblity is containers and/or the containerization of a research project. Here is a paper describing their use in data science https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1008316 
And more information on the reproducibility of containers here: https://carpentries-incubator.github.io/docker-introduction/reproduciblity/index.html

Beyond that I found the paper to be well written and well structured and feel it will be a helpful guide for the community. Congratulations to the authors and thank you for putting this together!

I would love to see journals take code review serieously and hire someone to code review alongside the work performed by the editors and reviewers. Perhaps some of the society journals should take the lead here?