Links

GitLab Logs

Connecting GitLab logs to your Panther Console

Overview

Panther supports onboarding GitLab logs using Data Transport mechanisms. Audit logs can be ingested with the HTTP Source, while other GitLab logs can be ingested with Amazon Web Services (AWS) S3 and SQS.

How to onboard GitLab Audit Streaming logs to Panther

This process outlines how to onboard GitLab Audit logs. To onboard other types of GitLab logs, such as API, Exceptions, Integrations, Git, and Production logs, follow the separate How to onboard non-Audit GitLab logs to Panther process below.

Step 1: Create an HTTP Source in Panther

  1. 1.
    In the left-hand navigation bar of your Panther Console, click Configure > Log Sources.
  2. 2.
    Click Create New.
  3. 3.
    Search for “GitLab,” then click its tile.
    • In the slide-out panel, the Transport Mechanism dropdown in the upper right corner will be pre-populated with the HTTP option.
  4. 4.
    Click Start Setup.
    On the GitLab page of the source setup process in Panther, there is a Transport Mechanism dropdown field with a value of HTTP. To its right is a Start Setup button.
  5. 5.
    • You will be required to use shared secret authentication. This is the only method of authentication GitLab supports.
    • The header associated with your shared secret value will be locked with a value of x-panther-gitlab.

Step 2: Set up Audit Log Streaming in GitLab.

  1. 1.
    In your GitLab console, navigate to your Group.
  2. 2.
    Click Security and Compliance > Audit Events.
  3. 3.
    In the tab at the top of the page, click Streams, then Add Streaming Destination.
  4. 4.
    Enter the information from the HTTP source you created in Step 1:
    • The URL provided by Panther.
    • The header name x-panther-gitlab and the secret you configured in Panther.
  5. 5.
    Click Add to finish creating the Streaming Destination.

How to onboard non-Audit GitLab logs to Panther

The process below outlines how to onboard non-Audit GitLab logs to Panther, such as API, Exceptions, Git, Integrations, and Production logs. If you'd like to onboard Audit logs, follow the separate How to onboard GitLab Audit Streaming logs to Panther process above.
To connect these logs into Panther:
  1. 1.
    Log in to the Panther Console.
  2. 2.
    In the left sidebar, click Configure > Log Sources.
  3. 3.
    Click Create New.
  4. 4.
    Search for the log type you want to onboard, then click its tile.
  5. 5.
    Select the data transport method you wish to use for this integration, then follow Panther's instructions for configuring the method:
  6. 6.
    Configure GitLab to push logs to the Data Transport source.
    • See GitLab's documentation for instructions on pushing logs to your selected Data Transport source.

Supported log types

Required fields in all the tables are in bold.

GitLab.API

Panther uses the latest version of GitLab API logs. Some fields differ from the official documentation.
Column
Type
Description
time
timestamp
The request timestamp
severity
string
The log level
duration_s
float
The time spent serving the request (in seconds)
db_duration_s
float
The time spent quering the database (in seconds)
view_duration_s
float
The time spent rendering the view for the Rails controller (in seconds)
status
smallint
The HTTP response status code
method
string
The HTTP method of the request
path
string
The URL path for the request
params
[{ "key":string, "value":string }]
The URL query parameters
host
string
Hostname serving the request
ua
string
User-Agent HTTP header
route
string
Rails route for the API endpoint
remote_ip
string
The remote IP address of the HTTP request
user_id
bigint
The user id of the request
username
string
The username of the request
gitaly_calls
bigint
Total number of calls made to Gitaly
gitaly_duration_s
float
Total time taken by Gitaly calls
redis_calls
bigint
Total number of calls made to Redis
redis_duration_s
float
Total time to retrieve data from Redis
correlation_id
string
Request unique id across logs
queue_duration_s
float
Total time that the request was queued inside GitLab Workhorse
meta_user
string
User that invoked the request
meta_project
string
Project associated with the request
meta_root_namespace
string
Root namespace
meta_caller_id
string
Caller ID
p_event_time
timestamp
Panther added standardized event time (UTC)
p_parse_time
timestamp
Panther added standardized log parse time (UTC)
p_log_type
string
Panther added field with type of log
p_row_id
string
Panther added field with unique id (within table)
p_source_id
string
Panther added field with the source id
p_source_label
string
Panther added field with the source label
p_any_ip_addresses
[string]
Panther added field with collection of ip addresses associated with the row
p_any_domain_names
[string]
Panther added field with collection of domain names associated with the row
p_any_trace_ids
[string]
Panther added field with collection of context trace identifiers
p_any_usernames
[string]
Panther added field with collection of usernames associated with the row

GitLab.Audit

Multi-use schema for GitLab audit events, from both self-hosted audit log files and GitLab's audit event streaming feature.
Column
Type
Description
severity
string
The log level. Present only in audit log files.
time
timestamp
The event timestamp. Present only in audit log files.
author_id
bigint
User id that made the change
entity_id
bigint
Id of the entity that was modified
entity_type
string
Type of the modified entity
change
string
Type of change to the settings. Present only in audit log files.
from
string
Old setting value. Present only in audit log files.
to
string
New setting value. Present only in audit log files.
author_name
string
Name of the user that made the change
target_id
bigint
Target id of the modified setting
target_type
string
Target type of the modified setting
target_details
string
Details of the target of the modified setting
created_at
timestamp
Timestamp when event was triggered. Present only in audit event streaming
details
json
JSON object containing additional metadata. Present only in audit event streaming
entity_path
string
Full path of the entity affected by the auditable event. Present only in audit event streaming
event_type
string
String representation of the type of audit event. Present only in audit event streaming
id
bigint
Unique identifier for the audit event. Present only in audit event streaming
ip_address
string
IP address of the host used to trigger the event. Present only in audit event streaming
p_event_time
timestamp
Panther added standardized event time (UTC)
p_parse_time
timestamp
Panther added standardized log parse time (UTC)
p_log_type
string
Panther added field with type of log
p_row_id
string
Panther added field with unique id (within table)
p_source_id
string
Panther added field with the source id
p_source_label
string
Panther added field with the source label

GitLab.Exceptions

GitLab log file containing changes to group or project settings
Column
Type
Description
severity
string
The log level
time
timestamp
The event timestamp
correlation_id
string
Request unique id across logs
extra_server
{ "os":{ "name":string, "version":string, "build":string }, "runtime":{ "name":string, "version":string } }
Information about the server on which the exception occurred
extra_project_id
bigint
Project id where the exception occurred
extra_relation_key
string
Relation on which the exception occurred
extra_relation_index
bigint
Relation index on which the exception occurred
exception_class
string
Class name of the exception that occurred
exception_message
string
Message of the exception that occurred
exception_backtrace
[string]
Stack trace of the exception that occurred
p_event_time
timestamp
Panther added standardized event time (UTC)
p_parse_time
timestamp
Panther added standardized log parse time (UTC)
p_log_type
string
Panther added field with type of log
p_row_id
string
Panther added field with unique id (within table)
p_source_id
string
Panther added field with the source id
p_source_label
string
Panther added field with the source label
p_any_trace_ids
[string]
Panther added field with collection of context trace identifiers

GitLab.Git

GitLab log file containing all failed requests from GitLab to Git repositories.
Column
Type
Description
severity
string
The log level
time
timestamp
The event timestamp
correlation_id
string
Unique id across logs
message
string
The error message from git
p_event_time
timestamp
Panther added standardized event time (UTC)
p_parse_time
timestamp
Panther added standardized log parse time (UTC)
p_log_type
string
Panther added field with type of log
p_row_id
string
Panther added field with unique id (within table)
p_source_id
string
Panther added field with the source id
p_source_label
string
Panther added field with the source label
p_any_trace_ids
[string]
Panther added field with collection of context trace identifiers

GitLab.Integrations

GitLab log with information about integrations activities such as Jira, Asana, and Irker services.
Column
Type
Description
severity
string
The log level
time
timestamp
The event timestamp
service_class
string
The class name of the integrated service
project_id
bigint
The project id the integration was running on
project_path
string
The project path the integration was running on
message
string
The log message from the service
client_url
string
The client url of the service
error
string
The error name if an error has occurred
p_event_time
timestamp
Panther added standardized event time (UTC)
p_parse_time
timestamp
Panther added standardized log parse time (UTC)
p_log_type
string
Panther added field with type of log
p_row_id
string
Panther added field with unique id (within table)
p_source_id
string
Panther added field with the source id
p_source_label
string
Panther added field with the source label
p_any_ip_addresses
[string]
Panther added field with collection of ip addresses associated with the row
p_any_domain_names
[string]
Panther added field with collection of domain names associated with the row

GitLab.Production

GitLab log for Production controller requests received from GitLab
Column
Type
Description
method
string
The HTTP method of the request
path
string
The URL path for the request
format
string
The response output format
controller
string
The Production controller class name
action
string
The Production controller action
status
bigint
The HTTP response status code
time
timestamp
The request timestamp
params
[{ "key":string, "value":string }]
The URL query parameters
remote_ip
string
The remote IP address of the HTTP request
user_id
bigint
The user id of the request
username
string
The username of the request
ua
string
The User-Agent of the requester
queue_duration_s
float
Total time that the request was queued inside GitLab Workhorse
gitaly_calls
bigint
Total number of calls made to Gitaly
gitaly_duration_s
float
Total time taken by Gitaly calls
redis_calls
bigint
Total number of calls made to Redis
redis_duration_s
float
Total time to retrieve data from Redis
redis_read_bytes
bigint
Total bytes read from Redis
redis_write_bytes
bigint
Total bytes written to Redis
correlation_id
string
Request unique id across logs
cpu_s
float
Total time spent on CPU
db_duration_s
float
Total time to retrieve data from PostgreSQL
view_duration_s
float
Total time taken inside the Rails views
duration_s
float
Total time taken to retrieve the request
meta_caller_id
string
Caller ID
location
string
(Applies only to redirects) The redirect URL
exception_class
string
Class name of the exception that occurred
exception_message
string
Message of the exception that occurred
exception_backtrace
[string]
Stack trace of the exception that occurred
etag_route
string
Route name etag (on redirects)
p_event_time
timestamp
Panther added standardized event time (UTC)
p_parse_time
timestamp
Panther added standardized log parse time (UTC)
p_log_type
string
Panther added field with type of log
p_row_id
string
Panther added field with unique id (within table)
p_source_id
string
Panther added field with the source id
p_source_label
string
Panther added field with the source label
p_any_ip_addresses
[string]
Panther added field with collection of ip addresses associated with the row
p_any_trace_ids
[string]
Panther added field with collection of context trace identifiers
p_any_usernames
[string]
Panther added field with collection of usernames associated with the row