Gerrit Code Review - Accounts

Starting from 2.15 Gerrit accounts are fully stored in NoteDb.

The account data consists of a sequence number (account ID), account properties (full name, display name, preferred email, registration date, status, inactive flag), preferences (general, diff and edit preferences), project watches, SSH keys, external IDs, starred changes and reviewed flags.

Most account data is stored in a special All-Users repository, which has one branch per user. Within the user branch there are Git config files for the account properties, the account preferences and the project watches. In addition there is an authorized_keys file for the SSH keys that follows the standard OpenSSH file format.

The account data in the user branch is versioned and the Git history of this branch serves as an audit log.

The external IDs are stored as Git Notes inside the All-Users repository in the refs/meta/external-ids notes branch. Storing all external IDs in a notes branch ensures that each external ID is only used once.

The starred changes are represented as independent refs in the All-Users repository. They are not stored in the user branch, since this data doesn’t need versioning.

The reviewed flags are not stored in Git, but are persisted in a database table. This is because there is a high volume of reviewed flags and storing them in Git would be inefficient.

Since accessing the account data in Git is not fast enough for account queries, e.g. when suggesting reviewers, Gerrit has a secondary index for accounts.

The All-Users repository is a special repository that only contains user-specific information. It contains one branch per user. The user branch is formatted as refs/users/CD/ABCD, where CD/ABCD is the sharded account ID, e.g. the user branch for account 1000856 is refs/users/56/1000856. The account IDs in the user refs are sharded so that there is a good distribution of the Git data in the storage system.

A user branch must exist for each account, as it represents the account. The files in the user branch are all optional. This means having a user branch with a tree that is completely empty is also a valid account definition.

Updates to the user branch are done through the Gerrit REST API, but users can also manually fetch their user branch and push changes back to Gerrit. On push the user data is evaluated and invalid user data is rejected.

To hide the implementation detail of the sharded account ID in the ref name Gerrit offers a magic refs/users/self ref that is automatically resolved to the user branch of the calling user. The user can then use this ref to fetch from and push to the own user branch. E.g. if user 1000856 pushes to refs/users/self, the branch refs/users/56/1000856 is updated. In Gerrit self is an established term to refer to the calling user (e.g. in change queries). This is why the magic ref for the own user branch is called refs/users/self.

A user branch should only be readable and writeable by the user to whom the account belongs. To assign permissions on the user branches the normal branch permission system is used. In the permission system the user branches are specified as refs/users/${shardeduserid}. The ${shardeduserid} variable is resolved to the sharded account ID. This variable is used to assign default access rights on all user branches that apply only to the owning user. The following permissions are set by default when a Gerrit site is newly installed or upgraded to a version which supports user branches:

The user branch contains several files with account data which are described below.

In addition to the user branches the All-Users repository also contains a branch for the external IDs and special refs for the starred changes.

Also the next available value of the account sequence is stored in the All-Users repository.

There are several situations in which Gerrit needs to query accounts, e.g.:

Accessing the account data in Git is not fast enough for account queries, since it requires accessing all user branches and parsing all files in each of them. To overcome this Gerrit has a secondary index for accounts. The account index is either based on Lucene or Elasticsearch.

Via the Query Account REST endpoint generic account queries are supported.

Accounts are automatically reindexed on any update. The Index Account REST endpoint allows to reindex an account manually. In addition the reindex program can be used to reindex all accounts offline.

A user branch contains several Git config files with the account data:

In addition it contains an authorized_keys file with the SSH keys of the account.

External IDs are used to link identities, such as the username and email addresses, and external identies such as an LDAP account or an OAUTH identity, to an account in Gerrit.

External IDs are stored as Git Notes in the All-Users repository. The name of the notes branch is refs/meta/external-ids.

As note key the SHA-1 of the external ID key is used, for example the key for the external ID username:jdoe is e0b751ae90ef039f320e097d7d212f490e933706. This ensures that an external ID is used only once (e.g. an external ID can never be assigned to multiple accounts at a point in time).

The following commands show how to find the SHA-1 of an external ID:

The note content is a Git config file:

Once SHA-1 of an external ID is known the following command can be used to show the content of the note:

The config file has one externalId section. The external ID key, which consists of scheme and ID in the format '<scheme>:<id>', is used as subsection name.

The accountId field is mandatory. The email and password fields are optional.

Note that git will automatically nest these notes at varying levels. If refs/meta/external-ids:7c/2a55657d911109dbc930836e7a770fb946e8ef is not found then check refs/meta/external-ids:7c/2a/55657d911109dbc930836e7a770fb946e8ef and so on.

The external IDs are maintained by Gerrit. This means users are not allowed to manually edit their external IDs. Only users with the Access Database global capability can push updates to the refs/meta/external-ids branch. However Gerrit rejects pushes if:

Starred changes allow users to mark changes as favorites and receive email notifications for them.

Each starred change is a tuple of an account ID, a change ID and a label.

To keep track of a change that is starred by an account, Gerrit creates a refs/starred-changes/YY/XXXX/ZZZZZZZ ref in the All-Users repository, where YY/XXXX is the sharded numeric change ID and ZZZZZZZ is the account ID.

A starred-changes ref points to a blob that contains the list of labels that the account set on the change. The label list is stored as UTF-8 text with one label per line.

Since JGit has explicit optimizations for looking up refs by prefix when the prefix ends with '/', this ref format is optimized to find starred changes by change ID. Finding starred changes by change ID is e.g. needed when a change is updated so that all users that have the default star on the change can be notified by email.

Gerrit also needs an efficient way to find all changes that were starred by an account, e.g. to provide results for the is:starred query operator. With the ref format as described above the lookup of starred changes by account ID is expensive, as this requires a scan of the full refs/starred-changes/* namespace. To overcome this the users that have starred a change are stored in the change index together with the star labels.

When reviewing a patch set in the Gerrit UI, the reviewer can mark files in the patch set as reviewed. These markers are called ‘Reviewed Flags’ and are private to the user. A reviewed flag is a tuple of patch set ID, file and account ID.

Each user can have many thousands of reviewed flags and over time the number can grow without bounds.

The high amount of reviewed flags makes a storage in Git unsuitable because each update requires opening the repository and committing a change, which is a high overhead for flipping a bit. Therefore the reviewed flags are stored in a database table. By default they are stored in a local H2 database, but there is an extension point that allows to plug in alternate implementations for storing the reviewed flags. To replace the storage for reviewed flags a plugin needs to implement the AccountPatchReviewStore interface. E.g. to support a cluster setup with multiple primary servers handling write operations where reviewed flags should be replicated between the primary nodes one could implement a store for the reviewed flags that is based on MySQL with replication.

The next available account sequence number is stored as UTF-8 text in a blob pointed to by the refs/sequences/accounts ref in the All-Users repository.

Multiple processes share the same sequence by incrementing the counter using normal git ref updates. To amortize the cost of these ref updates, processes increment the counter by a larger number and hand out numbers from that range in memory until they run out. The size of the account ID batch that each process retrieves at once is controlled by the notedb.accounts.sequenceBatchSize parameter in the gerrit.config file.

To replicate account data the following branches from the All-Users repository must be replicated:

Part of Gerrit Code Review