Gerrit writes log files in the $site_path/logs/ folder tracking requests, background and plugin activity and errors. By default logs are written in text format, optionally in JSON format. By default log files are compressed at server startup and then daily at 11pm and rotated every midnight.

Time format

For all timestamps the format [yyyy-MM-dd’T’HH:mm:ss,SSSXXX] is used. This format is both ISO 8601 and RFC3339 compatible.

Logs

The following logs can be written.

HTTPD Log

The httpd log tracks HTTP requests processed by Gerrit’s http daemon and is written to $site_path/logs/httpd_log. Enabled or disabled via the httpd.requestLog option.

Format is an enhanced NCSA combined log, if a log field is not present, a "-" is substituted:

  • host: The IP address of the HTTP client that made the HTTP resource request. If you are using a reverse proxy it depends on the proxy configuration if the proxy IP address or the client IP address is logged.

  • [thread name]: name of the Java thread executing the request.

  • remote logname: the identifier used to link: identify the client making the HTTP request, Gerrit always logs a dash -.

  • username: the username used by the client for authentication. "-" for anonymous requests.

  • [date:time]: The date and time stamp of the HTTP request. The time that the request was received.

  • request: The request line from the client is given in double quotes.

    • the HTTP method used by the client.

    • the resource the client requested.

    • the protocol/version used by the client.

  • statuscode: the HTTP status code that the server sent back to the client.

  • response size: the number of bytes of data transferred as part of the HTTP response, not including the HTTP header.

  • latency: response time in milliseconds.

  • referer: the Referer HTTP request header. This gives the site that the client reports having been referred from.

  • client agent: the client agent which sent the request.

  • total_cpu: total CPU time, CPU time in milliseconds to execute command.

  • user_cpu: user mode CPU time, CPU time in user mode in milliseconds to execute command. CPU time in kernel mode is total_cpu - user_cpu.

  • memory: memory allocated in bytes to execute command. -1 if the JVM does not support this metric.

  • command status: the overall result of the git command over HTTP. Currently populated only for the transfer phase of git-upload-pack commands. Possible values:

    • -1: The git-upload-pack transfer was ultimately not successful

    • 0: The git-upload-pack transfer was ultimately successful

Example:

12.34.56.78 [HTTP-4136374] - johndoe [28/Aug/2020:10:02:20 +0200] "GET /a/plugins/metrics-reporter-prometheus/metrics HTTP/1.1" 200 1247498 1900 - "Prometheus/2.13.1"

SSHD Log

The sshd log tracks ssh requests processed by Gerrit’s ssh daemon and is written to $site_path/logs/sshd_log. Enabled or disabled via option sshd.requestLog.

Log format:

  • [date time]: The time that the request was received.

  • sessionid: hexadecimal session identifier, all requests of the same connection share the same sessionid. Gerrit does not support multiplexing multiple sessions on the same connection. Grep the log file using the sessionid as filter to get all requests from that session.

  • [thread name]: name of the Java thread executing the request.

  • username: the username used by the client for authentication.

  • a/accountid: identifier of the Gerrit account which is logged on.

  • operation: the operation being executed via ssh.

    • LOGIN FROM <host>: login and start new SSH session from the given host.

    • AUTH FAILURE FROM <host> <message>: failed authentication from given host and cause of failure.

    • LOGOUT: logout and terminate SSH session.

    • git-upload-pack.<projectname>: git fetch or clone command for given project.

    • git-receive-pack.<projectname>: git push command for given project.

    • Gerrit ssh commands which may be logged in this field are documented here.

  • wait: command wait time, time in milliseconds the command waited for an execution thread.

  • exec: command execution time, time in milliseconds to execute the command.

  • status: status code. 0 means success, any other value is an error.

  • total_cpu: total CPU time, CPU time in milliseconds to execute command.

  • user_cpu: user mode CPU time, CPU time in user mode in milliseconds to execute command. CPU time in kernel mode is total_cpu - user_cpu.

  • memory: memory allocated in bytes to execute command. -1 if the JVM does not support this metric.

The git-upload-pack command provides the following additional fields after the exec and before the status field. All times are in milliseconds. Fields are -1 if not available when the upload-pack request returns an empty result since the client’s repository was up to date:

  • time negotiating: time for negotiating which objects need to be transferred.

  • time searching for reuse: time jgit searched for deltas which can be reused. That is the time spent matching existing representations against objects that will be transmitted, or that the client can be assumed to already have.

  • time searching for sizes: time jgit was searching for sizes of all objects that will enter the delta compression search window. The sizes need to be known to better match similar objects together and improve delta compression ratios.

  • time counting: time jgit spent enumerating the objects that need to be included in the output. This time includes any restarts that occur when a cached pack is selected for reuse.

  • time compressing: time jgit was compressing objects. This is observed wall-clock time and does not accurately track CPU time used when multiple threads were used to perform the delta compression.

  • time writing: time jgit needed to write packfile, from start of header until end of trailer. The transfer speed can be approximated by dividing total bytes by this value.

  • total time in UploadPack: total time jgit spent in upload-pack.

  • bitmap index misses: number of misses when trying to use bitmap index, -1 means no bitmap index available. This is the count of objects that needed to be discovered through an object walk because they were not found in bitmap indices.

  • total deltas: total number of deltas transferred. This may be lower than the actual number of deltas if a cached pack was reused.

  • total objects: total number of objects transferred. This total includes the value of total deltas.

  • total bytes: total number of bytes transferred. This size includes the pack header, trailer, thin pack, and reused cached packs.

  • client agent: the client agent and version which sent the request.

Example: a CI system established a SSH connection, sent an upload-pack command (git fetch) and closed the connection:

[2020-08-28 11:00:01,391 +0200] 8a154cae [sshd-SshServer[570fc452]-nio2-thread-299] voter a/1000023 LOGIN FROM 12.34.56.78
[2020-08-28 11:00:01,556 +0200] 8a154cae [SSH git-upload-pack /AP/ajs/jpaas-msg-svc.git (voter)] voter a/1000056 git-upload-pack./demo/project.git 0ms 115ms 92ms 1ms 0ms 6ms 0ms 0ms 7ms 3 10 26 2615 0 git/2.26.2
[2020-08-28 11:00:01,583 +0200] 8a154cae [sshd-SshServer[570fc452]-nio2-thread-168] voter a/1000023 LOGOUT

Error Log

The error log tracks errors and stack traces and is written to $site_path/logs/error_log.

Log format:

  • [date time]: The time that the request was received.

  • [thread name]: : name of the Java thread executing the request.

  • level: log level (ERROR, WARN, INFO, DEBUG).

  • logger: name of the logger.

  • message: log message.

  • stacktrace: Java stacktrace when an execption was caught, usually spans multiple lines.

GC Log

The gc log tracks git garbage collection running in a background thread if enabled and is written to $site_path/logs/gc_log.

Log format:

  • [date time]: The time that the request was received.

  • level: log level (ERROR, WARN, INFO, DEBUG).

  • message: log message.

Plugin Logs

Some plugins write their own log file. E.g. the replication plugin writes its log to $site_path/logs/replication_log. Refer to each plugin’s documentation for more details on their logs.

Part of Gerrit Code Review