Project Governance / Engineering Steering Committee

The Gerrit project has an engineering steering committee (ESC) that is in charge of:

  • Gerrit core (the gerrit project) and the core plugins

  • defining the project vision and the project scope

  • maintaining a roadmap, a release plan and a prioritized backlog

  • ensuring timely design reviews

  • ensuring that new features are compatible with the project vision and are well aligned with other features (give feedback on new design docs within 30 calendar days)

  • approving/rejecting designs, vetoing new features

  • assigning mentors for approved features

  • accepting new plugins as core plugins

  • making changes to the project governance process and the contribution processes

The steering committee has 5 members:

  • 3 Googlers that are appointed by Google

  • 2 non-Google maintainers, elected by non-Google maintainers for the period of 1 year (see below)

Refer to the project homepage for the list of current committee members.

The steering committee should act in the interest of the Gerrit project and the whole Gerrit community.

For decisions, consensus between steering committee members and all other maintainers is desired. If consensus cannot be reached, decisions can also be made by simple majority in the steering committee (should be applied only in exceptional situations).

The steering committee is empowered to overrule positive/negative votes from individual maintainers, but should do so only in exceptional situations after attempts to reach consensus have failed.

As an integral part of the Gerrit community, the steering committee is committed to transparency and to answering incoming requests in a timely manner.

Election of non-Google steering committee members

The election of the non-Google steering committee members happens once a year in June. Non-Google maintainers can nominate themselves by posting an informal application on the non-public community manager mailing list when the call for nominations is sent to the maintainers list by a community manager.

The list with all candidates will be published at the beginning of the voting period.

Keeping the candidates private during the nomination phase and publishing all candidates at once only at the start of the voting period ensures that:

  • people do not start voting before all candidates are known and the voting period has started

  • candidates that announce their candidacy early do not have an advantage

  • people are not discouraged to candidate when there are already other candidates

By applying to be steering committee member, the candidate confirms to be able to dedicate the time that is needed to fulfill this role (also see steering committee member).

Each non-Google maintainer can vote for 2 candidates. The voting happens by posting on the maintainer mailing list. The voting period is 14 calendar days from the start of the voting.

Google maintainers do not take part in this vote, because Google already has dedicated seats in the steering committee (see section steering committee).

If a non-Google seat on the steering committee becomes vacant before the current term ends, an exceptional election is conducted in order to replace the member(s) leaving the committee. The election will follow the same procedure as regular steering committee elections. The number of votes each maintainer gets in such exceptional election matches the number of seats to be filled. The term of the new member of the steering committee ends at the end of the current term of the steering committee when the next regular election concludes.

Contribution Process

See here.

Design Doc Review

See here.

Semantic versioning

Gerrit follows a light semantic versioning scheme MAJOR.MINOR[.PATCH[.HOTFIX]] format:

  • MAJOR is incremented when there are substantial incompatible changes and/or new features in Gerrit.

  • MINOR is incremented when there are changes that are typically backward compatible with the earlier minor version. Features can be removed following the feature deprecation process. Dependencies can be upgraded according to the libraries upgrade policy.

  • PATCH is incremented when there are backward-compatible bug fixes in Gerrit or its dependencies. When PATCH is zero, it can be omitted.

  • HOTFIX is present only when immediately after a patch release, some urgent fixes in the code or the packaging format are required but do not justify a new patch release.

For every MAJOR.MINOR release there is an associated stable branch that follows well defined rules of development.

Within a stable branch, there are multiple MAJOR.MINOR.PATCH tags created associated to the bug-fix releases of that stable release.

Examples:

  • Gerrit v3.0.0 contains breaking incompatible changes in the functionality because the ReviewDb storage has been totally removed.

  • Gerrit v2.15 contains brand-new features like NoteDb, however, still supports the existing ReviewDb storage for changes and thus is considered a minor release.

  • Gerrit v2.14.20 is the 20th patch-release of the stable Gerrit v2.14.* and thus does not contain new features but only bug-fixes.

Development in stable branches

As their name suggests stable branches are intended to be stable. This means that generally only bug-fixes should be done on stable branches, however this is not strictly enforced and exceptions may apply:

  • When a stable branch is initially created to prepare a new release the Gerrit community discusses on the mailing list if there are pending features which should still make it into the release. Those features are blocking the release and should be implemented on the stable branch before the first release candidate is created.

  • To stabilize the code before doing a major release several release candidates are created. Once the first release candidate was done no more features should be accepted on the stable branch. If more features are found to be required they should be discussed with the steering committee and should only be allowed if the risk of breaking things is considered to be low.

  • Once a major release is done only bug-fixes and documentation updates should be done on the stable branch. These updates will be included in the next minor release.

  • For minor releases new features could be acceptable if the following conditions are met:

    • they are result of a new feature introduced through a merge of an earlier stable branch

    • they are justified for completing, extending or fixing an existing feature

    • does not involve API, user-interface changes or data migrations

    • is backward compatible with all existing features

    • the parts of the code in common with existing features are properly covered by end-to-end tests

    • is important to the Gerrit community and no Gerrit maintainers have raised objections.

  • In cases of doubt or conflicting opinions on new features, it’s the responsibility of the steering committee to evaluate the risk of new features and make a decision based on these rules and opinions from the Gerrit community.

  • The older a stable branch is the more stable it should be. This means old stable branches should only receive bug-fixes that are either important or low risk. Security fixes, including security updates for third party dependencies, are always considered as important and hence can always be done on stable branches.

Examples:

  • Gerrit v3.0.0-rc1 and v3.0.0-rc2 may contain new features and API changes without notice, even if they are both cut on the same stable-3.0 branch.

  • Gerrit v2.14.8 introduced the support for ElasticSearch as a new feature. This was an exception agreed amongst the Gerrit maintainers, did not touch the Lucene indexing code-base, was supported by container-based E2E tests and represents a completion of an high-level feature.

Backporting to stable branches

From time to time bug fix releases are made for existing stable branches.

Developers concerned with stable branches are encouraged to backport or push fixes to these branches, even if no new release is planned. Backporting features is only possible in compliance with the rules above.

Fixes that are known to be needed for a particular release should be pushed for review on that release’s stable branch. They will then be included into the master branch when the stable branch is merged back.

Dealing with Security Issues

If a security vulnerability in Gerrit is discovered, we place an embargo on it until a fixed release or mitigation is available. Fixing the issue is usually pursued with high priority (depends on the severity of the security vulnerability). The embargo is lifted and the vulnerability is disclosed to the community as soon as a fix release or another mitigation is available.

How to report a security vulnerability?

To report a security vulnerability file a security issue in the Gerrit issue tracker. The visibility of issues that are created with the Security Issue template is automatically restricted to Gerrit maintainers and a few long-term contributors. This means as a reporter you may not be able to see the issue once it is created. Security issues are created on the ESC component so that they will be discussed at the next meeting of the Engineering Steering Committee which takes place biweekly.

If an existing issue is found to be a security vulnerability it should be turned into a security issue by:

  1. Setting the component to ESC

  2. Adding the labels Security and NonPublic

In case of doubt, or if an issue cannot wait until the next ESC meeting, contact the Engineering Steering Committee directly by sending them an email.

If needed, the ESC will contact the reporter for additional details.

The Embargo

Once an issue has been identified as security vulnerability, we keep it under embargo until a fixed release or a mitigation is available. This means that the issue is not discussed publicly, but only on issues with restricted visibility (see above) and at the mailing lists of the ESC, community managers and Gerrit maintainers. Since the repo-discuss mailing list is public, security issues must not be discussed on this mailing list while the embargo is in place.

The reason for keeping an embargo is to prevent attackers from taking advantage of a vulnerability while no fixed releases are available yet, and Gerrit administrators cannot make their systems secure.

Once a fix release or mitigation is available, the embargo is lifted and the community is informed about the security vulnerability with the advise to address the security vulnerability immediately (either by upgrading to a fixed release or applying the mitigation). The information about the security vulnerability is disclosed via the repo-discuss mailing list.

Handling of the Security Vulnerability

  1. Engineering Steering Committee evaluates the security vulnerability:

    The ESC discusses the security vulnerability and which actions should be taken to address it. One person, usually one of the Gerrit maintainers, should be appointed to drive and coordinate the investigation and the fix of the security vulnerability. This coordinator doesn’t need to do all the work alone, but is responsible that the security vulnerability is getting fixed in a timely manner.

    If the security vulnerability affects multiple or older releases the ESC should decide which of the releases should be fixed. For critical security issue we also consider fixing old releases that are otherwise not receiving any bug-fixes anymore.

    It’s also possible that the ESC decides that an issue is not a security issue and the embargo is lifted immediately.

  2. Filing a CVE

    For every security issue a CVE that describes the issue and lists the affected releases should be filed. Filing a CVE can be done by any maintainer that works for an organization that can request CVE numbers (e.g. Googlers). The CVE number must be included in the release notes. The CVE itself is only made public after fixed released have been published and the embargo has been lifted.

  3. Implementation of the security fix:

    To keep the embargo intact, security fixes cannot be developed and reviewed in the public gerrit repository. In particular it’s not secure to use private changes for implementing and reviewing security fixes (see general notes about security-fixes).

    Instead security fixes should be implemented and reviewed in the non-public gerrit-security-fixes repository which is only accessible by Gerrit maintainers and Gerrit community members that work on security fixes.

    The change that fixes the security vulnerability should contain an integration test that verifies that the security vulnerability is no longer present.

    Review and approval of the security fixes must be done by the Gerrit maintainers.

    Once a security fix is ready and submitted, it should be cherry-picked to all branches that should be fixed.

  4. CI validation of the security fix:

    The validation of the security fixes does not happen on the regular Gerrit CI, because it would compromise the confidentiality of the fix and therefore break the embargo.

    The release manager maintains a private branch on the gerrit-ci-scripts repository which contains a special build pipeline with special visibility restrictions.

    The validation process provides feedback, in terms of Code-Style, Verification and Checks, to the incoming security changes. The links associated with the build logs are exposed over the Internet but their access limited to only those who are actively participating in the development and review of the security fix.

    The maintainers that are willing to access the links to the CI logs need to request a time-limited (maximum 30 days) nominal X.509 certificate from a CI maintainer, which allows to access the build logs and analyze failures. The release manager may help obtaining that certificate from CI maintainers.

  5. Creation of fixed releases and announcement of the security vulnerability:

    A release manager should create new bug fix releases for all fixed branches.

    The new releases should be tested against the security vulnerability to double-check that the release was built from the correct source that contains the fix for the security vulnerability.

    Before publishing the fixed releases, an announcement to the Gerrit community should be prepared. The announcement should clearly describe the security vulnerability, which releases are affected and which releases contain the fix. The announcement should recommend to upgrade to fixed releases immediately.

    Once all releases are ready and tested and the announcement is prepared, the releases should be all published at the same time. Immediately after that, the announcement should be sent out to the repo-discuss mailing list.

    This ends the embargo and any issue that discusses the security vulnerability should be made public.

  6. Publish the CVE

  7. Follow-Up

    The ESC should discuss if there are any learnings from the security vulnerability and define action items to follow up in the issue tracker.

Core Plugins

See here.

Upgrading Libraries

Changes that add new libraries or upgrade existing libraries require an approval on the Library-Compliance label. For an approval the following things are checked:

  • The library has a license that is suitable for use within Gerrit.

  • If the library is used within Google, the version of the library must be compatible with the version that is used at Google.

Only maintainers from Google can vote on the Library-Compliance label. The Gerrit team at Google uses this change query to find changes that require a Library-Compliance approval.

Gerrit’s library dependencies should only be upgraded if the new version contains something we need in Gerrit. This includes new features, API changes as well as bug or security fixes. An exception to this rule is that right after a new Gerrit release was branched off, all libraries should be upgraded to the latest version to prevent Gerrit from falling behind. Doing those upgrades should conclude at the latest two months after the branch was cut. This should happen on the master branch to ensure that they are vetted long enough before they go into a release and we can be sure that the update doesn’t introduce a regression.

Escalation channel to Google

If anything urgent is blocking that requires the attention of a Googler you may escalate this by writing an email to Han-Wen Nienhuys: hanwen@google.com

Deprecating features

Gerrit should be as stable as possible and we aim to add only features that last. However, sometimes we are required to deprecate and remove features to be able to move forward with the project and keep the code-base clean. The following process should serve as a guideline on how to deprecate functionality in Gerrit. Its purpose is that we have a structured process for deprecation that users, administrators and developers can agree and rely on.

General process:

  • Make sure that the feature (e.g. a field on the API) is not needed anymore or blocks further development or improvement. If in doubt, consult the mailing list.

  • If you can provide a schema migration that moves users to a comparable feature, do so and stop here.

  • Mark the feature as deprecated in the documentation and release notes.

  • If possible, mark the feature deprecated in any user-visible interface. For example, if you are deprecating a Git push option, add a message to the Git response if the user provided the option informing them about deprecation.

  • Annotate the code with @Deprecated and @RemoveAfter(x.xx) if applicable. Alternatively, use // DEPRECATED, remove after x.xx (where x.xx is the version number that has to be branched off before removing the feature)

  • Gate the feature behind a config that is off by default (forcing admins to turn the deprecated feature on explicitly).

  • After the next release was branched off, remove any code that backed the feature.

You can optionally consult the mailing list to ask if there are users of the feature you wish to deprecate. If there are no major users, you can remove the feature without following this process and without the grace period of one release.

Part of Gerrit Code Review