# GitLab Logs ## Overview Panther supports onboarding GitLab logs using [Data Transport](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/data-onboarding/data-transports) mechanisms. Audit logs can be ingested with the [HTTP Source](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/data-onboarding/data-transports/http), while other GitLab logs can be ingested with Amazon Web Services (AWS) [S3](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/data-onboarding/data-transports/aws/s3) and [SQS](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/data-onboarding/data-transports/aws/sqs). ## How to onboard GitLab Audit Streaming logs to Panther This process outlines how to onboard GitLab [Audit](#gitlab.audit) logs. To onboard other types of GitLab logs, such as [API](#gitlab.api), [Exceptions](#gitlab.exceptions), [Integrations](#gitlab.integrations), [Git](#gitlab.git), and [Production](#gitlab.production) logs, follow the separate [How to onboard non-Audit GitLab logs to Panther](#how-to-onboard-non-audit-gitlab-logs-to-panther) process below. ### Step 1: Create an HTTP Source in Panther 1. In the left-hand navigation bar of your Panther Console, click **Configure** > **Log Sources**. 2. Click **Create New**. 3. Search for “GitLab,” then click its tile. * In the slide-out panel, the **Transport Mechanism** dropdown in the upper right corner will be pre-populated with the **HTTP** option. 4. Click **Start Setup**.\ ![On the GitLab page of the source setup process in Panther, there is a Transport Mechanism dropdown field with a value of HTTP. To its right is a Start Setup button.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2FHoEYhhV6Aevm6ZMJg1np%2Fgitlab%20setup.png?alt=media\&token=266aa724-794c-4c51-b173-332d8c5f8974) 5. Follow [Panther's instructions for configuring an HTTP Source](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/data-onboarding/data-transports/http). * You will be required to use [shared secret authentication](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/data-transports/http#shared-secret). This is the only method of authentication GitLab supports. * The **Header Name** associated with your **Secret Key Value** will be locked with a value of `x-panther-gitlab`. ### Step 2: Set up Audit Log Streaming in GitLab. 1. In your GitLab console, navigate to your Group. 2. Click **Security and Compliance** > **Audit Events**. 3. In the tab at the top of the page, click **Streams**, then **Add Streaming Destination**. 4. Enter the information from the HTTP source you created in Step 1: * The URL provided by Panther. * The header name `x-panther-gitlab` and the secret you configured in Panther. 5. Click **Add** to finish creating the Streaming Destination. ## How to onboard non-Audit GitLab logs to Panther The process below outlines how to onboard non-Audit GitLab logs to Panther, such as [API](#gitlab.api), [Exceptions](#gitlab.exceptions), [Git](#gitlab.git), [Integrations](#gitlab.integrations), and [Production](#gitlab.production) logs. If you'd like to onboard [Audit](#gitlab.audit) logs, follow the separate [How to onboard GitLab Audit Streaming logs to Panther](#how-to-onboard-gitlab-audit-streaming-logs-to-panther) process above. To connect these logs into Panther: 1. Log in to the Panther Console. 2. In the left sidebar, click **Configure > Log Sources**. 3. Click **Create New**. 4. Search for the log type you want to onboard, then click its tile. 5. Select the data transport method you wish to use for this integration, then follow Panther's instructions for configuring the method: * [AWS S3 bucket](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/data-onboarding/data-transports/aws/s3) * [AWS SQS](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/data-onboarding/data-transports/aws/sqs) 6. Configure GitLab to push logs to the Data Transport source. * See GitLab's documentation for instructions on pushing logs to your selected Data Transport source. ## Supported log types {% hint style="info" %} Required fields in all the tables are in **bold.** {% endhint %} ### GitLab.API Panther uses the latest version of GitLab API logs. Some fields differ from the official documentation. Reference: [GitLab Documentation on API JSON Logs. ](https://docs.gitlab.com/ee/administration/logs.html#api_jsonlog) | Column | Type | Description | | --------------------- | ---------------------------------------- | --------------------------------------------------------------------------- | | **`time`** | `timestamp` | The request timestamp | | **`severity`** | `string` | The log level | | **`duration_s`** | `float` | The time spent serving the request (in seconds) | | `db_duration_s` | `float` | The time spent quering the database (in seconds) | | `view_duration_s` | `float` | The time spent rendering the view for the Rails controller (in seconds) | | **`status`** | `smallint` | The HTTP response status code | | **`method`** | `string` | The HTTP method of the request | | **`path`** | `string` | The URL path for the request | | `params` | `[{ "key":string, "value":string }]` | The URL query parameters | | **`host`** | `string` | Hostname serving the request | | `ua` | `string` | User-Agent HTTP header | | **`route`** | `string` | Rails route for the API endpoint | | `remote_ip` | `string` | The remote IP address of the HTTP request | | `user_id` | `bigint` | The user id of the request | | `username` | `string` | The username of the request | | `gitaly_calls` | `bigint` | Total number of calls made to Gitaly | | `gitaly_duration_s` | `float` | Total time taken by Gitaly calls | | `redis_calls` | `bigint` | Total number of calls made to Redis | | `redis_duration_s` | `float` | Total time to retrieve data from Redis | | `correlation_id` | `string` | Request unique id across logs | | `queue_duration_s` | `float` | Total time that the request was queued inside GitLab Workhorse | | `meta_user` | `string` | User that invoked the request | | `meta_project` | `string` | Project associated with the request | | `meta_root_namespace` | `string` | Root namespace | | `meta_caller_id` | `string` | Caller ID | | **`p_event_time`** | `timestamp` | Panther added standardized event time (UTC) | | **`p_parse_time`** | `timestamp` | Panther added standardized log parse time (UTC) | | **`p_log_type`** | `string` | Panther added field with type of log | | **`p_row_id`** | `string` | Panther added field with unique id (within table) | | `p_source_id` | `string` | Panther added field with the source id | | `p_source_label` | `string` | Panther added field with the source label | | `p_any_ip_addresses` | `[string]` | Panther added field with collection of ip addresses associated with the row | | `p_any_domain_names` | `[string]` | Panther added field with collection of domain names associated with the row | | `p_any_trace_ids` | `[string]` | Panther added field with collection of context trace identifiers | | `p_any_usernames` | `[string]` | Panther added field with collection of usernames associated with the row | ### GitLab.Audit Multi-use schema for GitLab audit events, from both self-hosted audit log files and GitLab's audit event streaming feature. For more information, see [GitLab's documentation on Audit JSON Logs](https://docs.gitlab.com/ee/administration/logs.html#audit_jsonlog) and [GitLab's documentation on audit event streaming](https://docs.gitlab.com/ee/administration/audit_event_streaming.html). | Column | Type | Description | | -------------------- | ----------- | ---------------------------------------------------------------------------------------------- | | **`severity`** | `string` | The log level. Present only in audit log files. | | **`time`** | `timestamp` | The event timestamp. Present only in audit log files. | | **`author_id`** | `bigint` | User id that made the change | | **`entity_id`** | `bigint` | Id of the entity that was modified | | **`entity_type`** | `string` | Type of the modified entity | | **`change`** | `string` | Type of change to the settings. Present only in audit log files. | | **`from`** | `string` | Old setting value. Present only in audit log files. | | **`to`** | `string` | New setting value. Present only in audit log files. | | **`author_name`** | `string` | Name of the user that made the change | | **`target_id`** | `bigint` | Target id of the modified setting | | **`target_type`** | `string` | Target type of the modified setting | | **`target_details`** | `string` | Details of the target of the modified setting | | **`created_at`** | `timestamp` | Timestamp when event was triggered. Present only in audit event streaming | | **`details`** | `json` | JSON object containing additional metadata. Present only in audit event streaming | | **`entity_path`** | `string` | Full path of the entity affected by the auditable event. Present only in audit event streaming | | **`event_type`** | `string` | String representation of the type of audit event. Present only in audit event streaming | | **`id`** | `bigint` | Unique identifier for the audit event. Present only in audit event streaming | | **`ip_address`** | `string` | IP address of the host used to trigger the event. Present only in audit event streaming | | **`p_event_time`** | `timestamp` | Panther added standardized event time (UTC) | | **`p_parse_time`** | `timestamp` | Panther added standardized log parse time (UTC) | | **`p_log_type`** | `string` | Panther added field with type of log | | **`p_row_id`** | `string` | Panther added field with unique id (within table) | | `p_source_id` | `string` | Panther added field with the source id | | `p_source_label` | `string` | Panther added field with the source label | ### GitLab.Exceptions GitLab log file containing changes to group or project settings Reference: [GitLab Documentation on Exceptions for JSON logs.](https://docs.gitlab.com/ee/administration/logs.html#exceptions_jsonlog) | Column | Type | Description | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | | **`severity`** | `string` | The log level | | **`time`** | `timestamp` | The event timestamp | | `correlation_id` | `string` | Request unique id across logs | | `extra_server` | `{ "os":{ "name":string, "version":string, "build":string }, "runtime":{ "name":string, "version":string } }` | Information about the server on which the exception occurred | | `extra_project_id` | `bigint` | Project id where the exception occurred | | `extra_relation_key` | `string` | Relation on which the exception occurred | | `extra_relation_index` | `bigint` | Relation index on which the exception occurred | | **`exception_class`** | `string` | Class name of the exception that occurred | | **`exception_message`** | `string` | Message of the exception that occurred | | `exception_backtrace` | `[string]` | Stack trace of the exception that occurred | | **`p_event_time`** | `timestamp` | Panther added standardized event time (UTC) | | **`p_parse_time`** | `timestamp` | Panther added standardized log parse time (UTC) | | **`p_log_type`** | `string` | Panther added field with type of log | | **`p_row_id`** | `string` | Panther added field with unique id (within table) | | `p_source_id` | `string` | Panther added field with the source id | | `p_source_label` | `string` | Panther added field with the source label | | `p_any_trace_ids` | `[string]` | Panther added field with collection of context trace identifiers | ### GitLab.Git GitLab log file containing all failed requests from GitLab to Git repositories. Reference: [GitLab Documentation on Git for JSON Logs.](https://docs.gitlab.com/ee/administration/logs.html#git_jsonlog) | Column | Type | Description | | ------------------ | ----------- | ---------------------------------------------------------------- | | **`severity`** | `string` | The log level | | **`time`** | `timestamp` | The event timestamp | | `correlation_id` | `string` | Unique id across logs | | **`message`** | `string` | The error message from git | | **`p_event_time`** | `timestamp` | Panther added standardized event time (UTC) | | **`p_parse_time`** | `timestamp` | Panther added standardized log parse time (UTC) | | **`p_log_type`** | `string` | Panther added field with type of log | | **`p_row_id`** | `string` | Panther added field with unique id (within table) | | `p_source_id` | `string` | Panther added field with the source id | | `p_source_label` | `string` | Panther added field with the source label | | `p_any_trace_ids` | `[string]` | Panther added field with collection of context trace identifiers | ### GitLab.Integrations GitLab log with information about integrations activities such as Jira, Asana, and Irker services. Reference: [GitLab Documentation on Integrations for JSON Logs.](https://docs.gitlab.com/ee/administration/logs.html#integrations_jsonlog) | Column | Type | Description | | -------------------- | ----------- | --------------------------------------------------------------------------- | | **`severity`** | `string` | The log level | | **`time`** | `timestamp` | The event timestamp | | **`service_class`** | `string` | The class name of the integrated service | | **`project_id`** | `bigint` | The project id the integration was running on | | **`project_path`** | `string` | The project path the integration was running on | | **`message`** | `string` | The log message from the service | | **`client_url`** | `string` | The client url of the service | | `error` | `string` | The error name if an error has occurred | | **`p_event_time`** | `timestamp` | Panther added standardized event time (UTC) | | **`p_parse_time`** | `timestamp` | Panther added standardized log parse time (UTC) | | **`p_log_type`** | `string` | Panther added field with type of log | | **`p_row_id`** | `string` | Panther added field with unique id (within table) | | `p_source_id` | `string` | Panther added field with the source id | | `p_source_label` | `string` | Panther added field with the source label | | `p_any_ip_addresses` | `[string]` | Panther added field with collection of ip addresses associated with the row | | `p_any_domain_names` | `[string]` | Panther added field with collection of domain names associated with the row | ### GitLab.Production GitLab log for Production controller requests received from GitLab Reference: [GitLab Documentation on Production for JSON Logs.](https://docs.gitlab.com/ee/administration/logs.html#production_jsonlog) | Column | Type | Description | | --------------------- | ---------------------------------------- | --------------------------------------------------------------------------- | | **`method`** | `string` | The HTTP method of the request | | **`path`** | `string` | The URL path for the request | | `format` | `string` | The response output format | | `controller` | `string` | The Production controller class name | | `action` | `string` | The Production controller action | | **`status`** | `bigint` | The HTTP response status code | | **`time`** | `timestamp` | The request timestamp | | `params` | `[{ "key":string, "value":string }]` | The URL query parameters | | `remote_ip` | `string` | The remote IP address of the HTTP request | | `user_id` | `bigint` | The user id of the request | | `username` | `string` | The username of the request | | `ua` | `string` | The User-Agent of the requester | | `queue_duration_s` | `float` | Total time that the request was queued inside GitLab Workhorse | | `gitaly_calls` | `bigint` | Total number of calls made to Gitaly | | `gitaly_duration_s` | `float` | Total time taken by Gitaly calls | | `redis_calls` | `bigint` | Total number of calls made to Redis | | `redis_duration_s` | `float` | Total time to retrieve data from Redis | | `redis_read_bytes` | `bigint` | Total bytes read from Redis | | `redis_write_bytes` | `bigint` | Total bytes written to Redis | | `correlation_id` | `string` | Request unique id across logs | | `cpu_s` | `float` | Total time spent on CPU | | `db_duration_s` | `float` | Total time to retrieve data from PostgreSQL | | `view_duration_s` | `float` | Total time taken inside the Rails views | | **`duration_s`** | `float` | Total time taken to retrieve the request | | `meta_caller_id` | `string` | Caller ID | | `location` | `string` | (Applies only to redirects) The redirect URL | | `exception_class` | `string` | Class name of the exception that occurred | | `exception_message` | `string` | Message of the exception that occurred | | `exception_backtrace` | `[string]` | Stack trace of the exception that occurred | | `etag_route` | `string` | Route name etag (on redirects) | | **`p_event_time`** | `timestamp` | Panther added standardized event time (UTC) | | **`p_parse_time`** | `timestamp` | Panther added standardized log parse time (UTC) | | **`p_log_type`** | `string` | Panther added field with type of log | | **`p_row_id`** | `string` | Panther added field with unique id (within table) | | `p_source_id` | `string` | Panther added field with the source id | | `p_source_label` | `string` | Panther added field with the source label | | `p_any_ip_addresses` | `[string]` | Panther added field with collection of ip addresses associated with the row | | `p_any_trace_ids` | `[string]` | Panther added field with collection of context trace identifiers | | `p_any_usernames` | `[string]` | Panther added field with collection of usernames associated with the row |