GitLab Logs

Connecting GitLab logs to your Panther Console

Overview

Panther supports onboarding GitLab logs using Data Transport mechanisms. Audit logs can be ingested with the HTTP Source, while other GitLab logs can be ingested with Amazon Web Services (AWS) S3 and SQS.

How to onboard GitLab Audit Streaming logs to Panther

This process outlines how to onboard GitLab Audit logs. To onboard other types of GitLab logs, such as API, Exceptions, Integrations, Git, and Production logs, follow the separate How to onboard non-Audit GitLab logs to Panther process below.

Step 1: Create an HTTP Source in Panther

1. In the left-hand navigation bar of your Panther Console, click Configure > Log Sources.

2. Click Create New.

3. Search for “GitLab,” then click its tile.

   • In the slide-out panel, the Transport Mechanism dropdown in the upper right corner will be pre-populated with the HTTP option.

4. Click Start Setup.

5. Follow Panther's instructions for configuring an HTTP Source.

   • You will be required to use shared secret authentication. This is the only method of authentication GitLab supports.

   • The Header Name associated with your Secret Key Value will be locked with a value of x-panther-gitlab.

Step 2: Set up Audit Log Streaming in GitLab.

1. In your GitLab console, navigate to your Group.

2. Click Security and Compliance > Audit Events.

3. In the tab at the top of the page, click Streams, then Add Streaming Destination.

4. Enter the information from the HTTP source you created in Step 1:

   • The URL provided by Panther.

   • The header name x-panther-gitlab and the secret you configured in Panther.

5. Click Add to finish creating the Streaming Destination.

How to onboard non-Audit GitLab logs to Panther

The process below outlines how to onboard non-Audit GitLab logs to Panther, such as API, Exceptions, Git, Integrations, and Production logs. If you'd like to onboard Audit logs, follow the separate How to onboard GitLab Audit Streaming logs to Panther process above.

To connect these logs into Panther:

1. Log in to the Panther Console.

2. In the left sidebar, click Configure > Log Sources.

3. Click Create New.

4. Search for the log type you want to onboard, then click its tile.

5. Select the data transport method you wish to use for this integration, then follow Panther's instructions for configuring the method:

   • AWS S3 bucket

   • AWS SQS

6. Configure GitLab to push logs to the Data Transport source.

   • See GitLab's documentation for instructions on pushing logs to your selected Data Transport source.

Supported log types

Required fields in all the tables are in bold.

GitLab.API

Panther uses the latest version of GitLab API logs. Some fields differ from the official documentation.

Reference: GitLab Documentation on API JSON Logs.

Column	Type	Description
time	timestamp	The request timestamp
severity	string	The log level
duration_s	float	The time spent serving the request (in seconds)
db_duration_s	float	The time spent quering the database (in seconds)
view_duration_s	float	The time spent rendering the view for the Rails controller (in seconds)
status	smallint	The HTTP response status code
method	string	The HTTP method of the request
path	string	The URL path for the request
params	[{ "key":string, "value":string }]	The URL query parameters
host	string	Hostname serving the request
ua	string	User-Agent HTTP header
route	string	Rails route for the API endpoint
remote_ip	string	The remote IP address of the HTTP request
user_id	bigint	The user id of the request
username	string	The username of the request
gitaly_calls	bigint	Total number of calls made to Gitaly
gitaly_duration_s	float	Total time taken by Gitaly calls
redis_calls	bigint	Total number of calls made to Redis
redis_duration_s	float	Total time to retrieve data from Redis
correlation_id	string	Request unique id across logs
queue_duration_s	float	Total time that the request was queued inside GitLab Workhorse
meta_user	string	User that invoked the request
meta_project	string	Project associated with the request
meta_root_namespace	string	Root namespace
meta_caller_id	string	Caller ID
p_event_time	timestamp	Panther added standardized event time (UTC)
p_parse_time	timestamp	Panther added standardized log parse time (UTC)
p_log_type	string	Panther added field with type of log
p_row_id	string	Panther added field with unique id (within table)
p_source_id	string	Panther added field with the source id
p_source_label	string	Panther added field with the source label
p_any_ip_addresses	[string]	Panther added field with collection of ip addresses associated with the row
p_any_domain_names	[string]	Panther added field with collection of domain names associated with the row
p_any_trace_ids	[string]	Panther added field with collection of context trace identifiers
p_any_usernames	[string]	Panther added field with collection of usernames associated with the row

GitLab.Audit

Multi-use schema for GitLab audit events, from both self-hosted audit log files and GitLab's audit event streaming feature.

For more information, see GitLab's documentation on Audit JSON Logs and GitLab's documentation on audit event streaming.

Column	Type	Description
severity	string	The log level. Present only in audit log files.
time	timestamp	The event timestamp. Present only in audit log files.
author_id	bigint	User id that made the change
entity_id	bigint	Id of the entity that was modified
entity_type	string	Type of the modified entity
change	string	Type of change to the settings. Present only in audit log files.
from	string	Old setting value. Present only in audit log files.
to	string	New setting value. Present only in audit log files.
author_name	string	Name of the user that made the change
target_id	bigint	Target id of the modified setting
target_type	string	Target type of the modified setting
target_details	string	Details of the target of the modified setting
created_at	timestamp	Timestamp when event was triggered. Present only in audit event streaming
details	json	JSON object containing additional metadata. Present only in audit event streaming
entity_path	string	Full path of the entity affected by the auditable event. Present only in audit event streaming
event_type	string	String representation of the type of audit event. Present only in audit event streaming
id	bigint	Unique identifier for the audit event. Present only in audit event streaming
ip_address	string	IP address of the host used to trigger the event. Present only in audit event streaming
p_event_time	timestamp	Panther added standardized event time (UTC)
p_parse_time	timestamp	Panther added standardized log parse time (UTC)
p_log_type	string	Panther added field with type of log
p_row_id	string	Panther added field with unique id (within table)
p_source_id	string	Panther added field with the source id
p_source_label	string	Panther added field with the source label

GitLab.Exceptions

GitLab log file containing changes to group or project settings

Reference: GitLab Documentation on Exceptions for JSON logs.

Column	Type	Description
severity	string	The log level
time	timestamp	The event timestamp
correlation_id	string	Request unique id across logs
extra_server	{ "os":{ "name":string, "version":string, "build":string }, "runtime":{ "name":string, "version":string } }	Information about the server on which the exception occurred
extra_project_id	bigint	Project id where the exception occurred
extra_relation_key	string	Relation on which the exception occurred
extra_relation_index	bigint	Relation index on which the exception occurred
exception_class	string	Class name of the exception that occurred
exception_message	string	Message of the exception that occurred
exception_backtrace	[string]	Stack trace of the exception that occurred
p_event_time	timestamp	Panther added standardized event time (UTC)
p_parse_time	timestamp	Panther added standardized log parse time (UTC)
p_log_type	string	Panther added field with type of log
p_row_id	string	Panther added field with unique id (within table)
p_source_id	string	Panther added field with the source id
p_source_label	string	Panther added field with the source label
p_any_trace_ids	[string]	Panther added field with collection of context trace identifiers

GitLab.Git

GitLab log file containing all failed requests from GitLab to Git repositories.

Reference: GitLab Documentation on Git for JSON Logs.

Column	Type	Description
severity	string	The log level
time	timestamp	The event timestamp
correlation_id	string	Unique id across logs
message	string	The error message from git
p_event_time	timestamp	Panther added standardized event time (UTC)
p_parse_time	timestamp	Panther added standardized log parse time (UTC)
p_log_type	string	Panther added field with type of log
p_row_id	string	Panther added field with unique id (within table)
p_source_id	string	Panther added field with the source id
p_source_label	string	Panther added field with the source label
p_any_trace_ids	[string]	Panther added field with collection of context trace identifiers

GitLab.Integrations

GitLab log with information about integrations activities such as Jira, Asana, and Irker services.

Reference: GitLab Documentation on Integrations for JSON Logs.

Column	Type	Description
severity	string	The log level
time	timestamp	The event timestamp
service_class	string	The class name of the integrated service
project_id	bigint	The project id the integration was running on
project_path	string	The project path the integration was running on
message	string	The log message from the service
client_url	string	The client url of the service
error	string	The error name if an error has occurred
p_event_time	timestamp	Panther added standardized event time (UTC)
p_parse_time	timestamp	Panther added standardized log parse time (UTC)
p_log_type	string	Panther added field with type of log
p_row_id	string	Panther added field with unique id (within table)
p_source_id	string	Panther added field with the source id
p_source_label	string	Panther added field with the source label
p_any_ip_addresses	[string]	Panther added field with collection of ip addresses associated with the row
p_any_domain_names	[string]	Panther added field with collection of domain names associated with the row

GitLab.Production

GitLab log for Production controller requests received from GitLab

Reference: GitLab Documentation on Production for JSON Logs.

Column	Type	Description
method	string	The HTTP method of the request
path	string	The URL path for the request
format	string	The response output format
controller	string	The Production controller class name
action	string	The Production controller action
status	bigint	The HTTP response status code
time	timestamp	The request timestamp
params	[{ "key":string, "value":string }]	The URL query parameters
remote_ip	string	The remote IP address of the HTTP request
user_id	bigint	The user id of the request
username	string	The username of the request
ua	string	The User-Agent of the requester
queue_duration_s	float	Total time that the request was queued inside GitLab Workhorse
gitaly_calls	bigint	Total number of calls made to Gitaly
gitaly_duration_s	float	Total time taken by Gitaly calls
redis_calls	bigint	Total number of calls made to Redis
redis_duration_s	float	Total time to retrieve data from Redis
redis_read_bytes	bigint	Total bytes read from Redis
redis_write_bytes	bigint	Total bytes written to Redis
correlation_id	string	Request unique id across logs
cpu_s	float	Total time spent on CPU
db_duration_s	float	Total time to retrieve data from PostgreSQL
view_duration_s	float	Total time taken inside the Rails views
duration_s	float	Total time taken to retrieve the request
meta_caller_id	string	Caller ID
location	string	(Applies only to redirects) The redirect URL
exception_class	string	Class name of the exception that occurred
exception_message	string	Message of the exception that occurred
exception_backtrace	[string]	Stack trace of the exception that occurred
etag_route	string	Route name etag (on redirects)
p_event_time	timestamp	Panther added standardized event time (UTC)
p_parse_time	timestamp	Panther added standardized log parse time (UTC)
p_log_type	string	Panther added field with type of log
p_row_id	string	Panther added field with unique id (within table)
p_source_id	string	Panther added field with the source id
p_source_label	string	Panther added field with the source label
p_any_ip_addresses	[string]	Panther added field with collection of ip addresses associated with the row
p_any_trace_ids	[string]	Panther added field with collection of context trace identifiers
p_any_usernames	[string]	Panther added field with collection of usernames associated with the row