Code Reviews are more than Pull Requests

Code review is a well-known best practice for software development, with varying degrees of adoption, consistency, and value in industry and research software, and yet the varying kinds of review and the diverse values it brings to the process and people involved have rarely been articulated. When talking about code review many people think of GitHub or GitLab and pull/merge requests. However, especially for research software, code review does not automatically mean reviewing someone else’s changes proposed via a pull request and approving it to be merged into the main branch. In fact, there are several different types of review and some of them are applicable almost exclusively to research software.

The DHTech Community Code Review Working Group (see Damerow et al. 2024) has developed a community code review system for the digital humanities to provide an opportunity for people writing code for digital humanities projects to get feedback on their work. At the time of writing, the group has executed nine code reviews. The submitted code for each review has varied in “completeness” as did the type of code and the goal of the project. Reviewed code has included Python packages, JavaScript visualizations, and agent-based models. Some projects were more or less complete, others were still in active development. Code reviews differed based on the status of the code, the reasons a review was requested, and what the ultimate goal of the reviews and the code authors were. In the following, we summarize the different types of code review we have encountered in the project and our professional practice. Some of them are unique to the research setting, which makes them valuable case studies for considering the ways that research software development diverges from traditional software development.

Types of Code Review

In our experience, there are several factors to consider when planning or executing a code review:

• The status of the code to be reviewed.
• The intended purpose of the code.
• The goal of the review.
• The availability and schedule of code author and reviewer.

These considerations guide the process as well as the review itself. Review of code that is work in progress can have a much greater impact on the quality of the code than review of code that is considered “done.” For work-in-progress code, for instance, a reviewer can make suggestions for how to make code more readable or more efficient, which the code author can incorporate into their development since the project is still active. In contrast, for code that is submitted as part of an article submission, reviewers should focus on the correctness of the results the code produces, rather than on the readability and sustainability of the code.
Reflecting on our experiences and discussions, a categorization emerged that we have found useful to utilize when thinking about the concept of code reviews in research. In the following we will discuss each type of review and its role in the research process.

Continuous Code Review
This type of code review was termed “modern code review” by Bacchelli and Bird (2013) and is the de-facto standard in traditional software development teams. In continuous or modern code review, code is reviewed frequently (for example, for every change to the code base); tools such as GitHub pull requests are used; and review is rather informal, not requiring large team meetings or formal processes (Bacchelli and Bird 2013). A critical requirement for this type of code review is a team in which everyone is familiar with the code base. Typically, developers use separate branches when making changes to the code and “suggest” those changes to be incorporated (“merged”) into the main code base using a pull (or merge) request. Another developer can then review such a request and provide feedback. This kind of workflow satisfies the four-eyes principle (everything needs a second set of eyes), increasing the chance that errors in the code are found before they are incorporated into the main code base. It also supports readability and maintainability of code, since reviewers can point out sections that are hard to understand or that could cause issues with the sustainability of the code.

Additionally, ongoing code review fosters knowledge sharing in a team and can be used as a teaching and mentoring tool. This is especially useful when onboarding new team members or training less experienced developers. Senior members of the team can provide feedback before changes are merged into the main development branch ensuring consistency across the code base.

Code Review for Publication
A type of code review very specific to research software is the review of code during the peer review process of an article. While not yet common, there are a few journals that require any custom code used to generate the results described in a publication to be submitted and reviewed alongside the manuscript. The Journal of Computational Literary Studies (JCLS) and Nature Human Behaviour are two examples requiring the submission of code for review.

Given the context of this type of code review, the focus in these cases is on the results of the code rather than the code itself. The two leading questions for this kind of review are typically “Does the code produce the results described?” and “Is there enough documentation and explanation to reproduce the results?”. The reviewer might identify areas of improvement in regards to maintainability, performance, or sustainability, but the likelihood that those would be addressed by the code authors is low, especially if reworking the code increases the chances of introducing errors into the code.

One challenge for this type of code review is the dual requirement of evaluating the manuscript alongside the code. This requires either reviewers with both domain and technical expertise who are able to review both manuscript and code, or it requires splitting up the work of reviewing across multiple people with the required expertise to review and evaluate the code. Either way, it can increase the review and response time of the publication process. The CODECHECKS project (Nüst and Eglen 2021) presents a flexible approach for journals who are interested in including code review into their publication process. In the CODECHECKS workflow, a codechecker reviews a piece of code that was submitted alongside a manuscript to a publisher for reproducibility by running the code based on the supplied instructions. Codechecks are executed after a manuscript has been accepted for publication avoiding the issue of requiring anonymization in single- or double-blind reviews. Code that has passed a CODECHECK receives a certificate that is published in Zenodo. This process “increase[s] transparency of the checked papers and can contribute to building trust in research findings” (Nüst and Eglen 2021, p. 14).

Review of “Completed” Software
This type of review is similar to the previous category of code review for publications with the key difference that the focus is on the software rather than the manuscript. There are a few journals for publishing research software such as the Journal of Open Source Software (JOSS) or the Journal of Open Research Software (JORS). Typically, these journals require a short technical write-up to accompany the software submission. The review process evaluates both the manuscript and software, focusing on repository availability, licensing, installation success, and the adequacy of documentation.

Repository Review
The less formal version of the type of review journals like JOSS or JORS provide, is the repository review. During a repository review, the code is evaluated against specific requirements, including the presence of a README file, licensing, and existence of installation and usage documentation. This type of review is sometimes offered by RSEs or RSE groups in organizations to help researchers improve their research software (Comi 2023).

Issue-specific Reviews
In an issue-specific code review, the reviewer takes a different approach in the sense that they review the code to answer a specific question or to solve a specific problem. This is often done in a less formal manner but as a request between colleagues to “take a look” and see if they can “spot the problem with the code.” While this type of review might not be what most developers define as code review, it follows the same conceptual process: someone who is not the code author is reviewing the code. It is critical to realize that even though we might not initially think of this as code review, it is an important process that happens regularly in many development teams. However, research software, especially in the humanities, is all too often developed in teams of one with no technical colleagues with the expertise needed to “take a look,” which means the code author is left to solve their coding problem by themselves. This situation is comparable to a researcher writing a paper in a language none of their colleagues speak, so they are unable to receive any feedback or improvement suggestions.

Review in-progress work
Review of code that is under active development has the potential for the greatest impact but can also be the hardest to do effectively. For reviewer feedback to be useful, the reviewing requires an accelerated timeline and must be completed quickly to avoid holding up development. Or the code author might choose to continue development without a review, potentially rendering the review useless when it is completed later, since the code submitted for review is now out of date and the original goals for the review may no longer be relevant. However, review in-progress can be tremendously impactful as it has the potential to make foundational recommendations that impact both the code and coding processes far into the future. For instance, imagine a code author who does not use a linter or follow a specific code style. If a reviewer early on recommends to do just that, the code author can incorporate a linter into their workflow and fix issues identified by the linter as the code is developed rather than checking later and resulting in a list of issues too long to be addressed without substantial extra work.

Code Review for Research Software

Code review can plug into the research process at various stages. Before attempting to integrate code review, it is important to think through the goals of the review and what kind of feedback will have the most impact. This is especially important with generative AI tools that are now capable of both generating and reviewing code. AI tooling raises questions about the role of the developer in the coding process. We believe that code review should be conceptionalized as human-centered: the leading question should be what is most helpful to the author of the code and how does it advance their research in both the short and long term? This might be radically different from the goals of code review in an industry setting, where the focus is on the final product and its development rather than the people maintaining it.

Research software is different from commercially developed software. The people who develop research software often have different backgrounds and motivations than professional software developers in the industry. Those differences should be reflected in the processes and tools that are being employed. We need to remember this when we apply industry best practices to our work and adapt them as necessary to ensure the quality and trustworthiness of our work.

References

Bacchelli, Alberto, and Christian Bird (2013). “Expectations, Outcomes, and Challenges of Modern Code Review.” 2013 35th International Conference on Software Engineering (ICSE), 712–21. https://doi.org/10.1109/ICSE.2013.6606617.

Comi, Troy (2023). “Repo Review Consultations.” US-RSE’23. https://drive.google.com/file/d/14a0EszHMeJJylMowktqrDYhFSsuin3IF/view.

Damerow, Julia, Rebecca Sutton Koeser, Jeffrey C. Carver, and Malte Vogl (2024). “Code Review in Digital Humanities.” Digital Scholarship in the Humanities 40 (Supplement_1): i18–26. https://doi.org/10.1093/llc/fqae052.

Nüst, Daniel, and Stephen J. Eglen (2021). CODECHECK: An Open Science Initiative for the Independent Execution of Computations Underlying Research Articles during Peer Review to Improve Reproducibility. July 20. https://doi.org/10.12688/f1000research.51738.2.