GitLab Logs

Connecting GitLab logs to your Panther Console

Overview

Panther supports onboarding GitLab logs using Data Transport mechanisms.

This page describes two processes: the onboarding process for GitLab audit logs, and the onboarding process for all other GitLab log types. These processes differ because audit logs are ingested through GitLab audit event streaming, while non-audit logs are pulled via the GitLab API.

Audit logs can be ingested with the HTTP Source, while other GitLab logs can be ingested with Amazon Web Services (AWS) S3 and SQS.

To ingest GitLab audit logs into Panther using audit event streaming as described below, you must have GitLab Ultimate.

How to onboard GitLab audit streaming logs to Panther

This process outlines how to onboard GitLab Audit logs. To onboard other types of GitLab logs, such as API, Exceptions, Integrations, Git, and Production logs, follow the separate How to onboard non-audit GitLab logs to Panther process below.

Step 1: Create an HTTP Source in Panther

1. In the left-hand navigation bar of your Panther Console, click Configure > Log Sources.

2. Click Create New.

3. Search for “GitLab,” then click its tile.

   • In the slide-out panel, the Transport Mechanism dropdown in the upper right corner will be pre-populated with the HTTP option.

4. Click Start Setup.

5. Follow Panther's instructions for configuring an HTTP Source, beginning at Step 5.

   • You will be required to use shared secret authentication. This is the only method of authentication GitLab supports.

   • The Header Name associated with your Secret Key Value will be locked with a value of x-panther-gitlab.

   • Payloads sent to this source are subject to the payload requirements for all HTTP sources.

   • Do not proceed to the next step until the creation of your HTTP endpoint has completed.

Step 2: Set up audit log streaming in GitLab

• In the GitLab documentation, follow the Add a new HTTP destination process.

  • In the Destination field, enter the URL you generated in Step 1.

  • Add a header with the name x-panther-gitlab and the secret you configured in Panther in Step 1.

How to onboard non-audit GitLab logs to Panther

The process below outlines how to onboard non-audit GitLab logs to Panther, such as API, Exceptions, Git, Integrations, and Production logs. If you'd like to onboard audit logs, follow the separate How to onboard GitLab audit streaming logs to Panther process above.

To connect these logs into Panther:

1. Log in to the Panther Console.

2. In the left sidebar, click Configure > Log Sources.

3. Click Create New.

4. Search for the log type you want to onboard, then click its tile.

5. Select the data transport method you wish to use for this integration, then follow Panther's instructions for configuring the method:

   • AWS S3 bucket

   • AWS SQS

6. Configure GitLab to push logs to the Data Transport source.

   • See GitLab's documentation for instructions on pushing logs to your selected Data Transport source.

Supported log types

GitLab.API

Panther uses the latest version of GitLab API logs. Some fields differ from the official documentation.

Reference: GitLab documentation on API JSON logs.

schema: GitLab.API
description: |-
    GitLab log for API requests received from GitLab.
    NOTE: We are using the latest version of GitLab API logs. Some fields differ from the official documentation
referenceURL: https://docs.gitlab.com/ee/administration/logs/#api_jsonlog
fields:
    - name: time
      required: true
      description: The request timestamp
      type: timestamp
      timeFormats:
        - rfc3339
      isEventTime: true
    - name: severity
      required: true
      description: The log level
      type: string
    - name: duration_s
      required: true
      description: The time spent serving the request (in seconds)
      type: float
    - name: db_duration_s
      description: The time spent querying the database (in seconds)
      type: float
    - name: view_duration_s
      description: The time spent rendering the view for the Rails controller (in seconds)
      type: float
    - name: status
      required: true
      description: The HTTP response status code
      type: smallint
    - name: method
      required: true
      description: The HTTP method of the request
      type: string
    - name: path
      required: true
      description: The URL path for the request
      type: string
    - name: params
      description: The URL query parameters
      type: array
      element:
        type: object
        fields:
            - name: key
              required: true
              description: Query parameter name
              type: string
            - name: value
              description: Query parameter value
              type: json
    - name: host
      required: true
      description: Hostname serving the request
      type: string
      indicators:
        - hostname
    - name: ua
      description: User-Agent HTTP header
      type: string
    - name: route
      required: true
      description: Rails route for the API endpoint
      type: string
    - name: remote_ip
      description: The remote IP address of the HTTP request
      type: string
      indicators:
        - ip
    - name: user_id
      description: The user id of the request
      type: bigint
    - name: username
      description: The username of the request
      type: string
      indicators:
        - username
    - name: gitaly_calls
      description: Total number of calls made to Gitaly
      type: bigint
    - name: gitaly_duration_s
      description: Total time taken by Gitaly calls
      type: float
    - name: redis_calls
      description: Total number of calls made to Redis
      type: bigint
    - name: redis_duration_s
      description: Total time to retrieve data from Redis
      type: float
    - name: correlation_id
      description: Request unique id across logs
      type: string
      indicators:
        - trace_id
    - name: queue_duration_s
      description: Total time that the request was queued inside GitLab Workhorse
      type: float
    - name: meta.user
      description: User that invoked the request
      type: string
      indicators:
        - username
    - name: meta.project
      description: Project associated with the request
      type: string
    - name: meta.root_namespace
      description: Root namespace
      type: string
    - name: meta.caller_id
      description: Caller ID
      type: string

GitLab.Audit

Multi-use schema for GitLab audit events, from both self-hosted audit log files and GitLab's audit event streaming feature.

For more information, see GitLab's documentation on audit JSON logs and GitLab's documentation on audit event streaming.

schema: GitLab.Audit
description: 'Multi-use schema for GitLab audit events both from self-hosted audit log files, as well as GitLab''s audit event streaming feature: https://docs.gitlab.com/ee/administration/audit_event_streaming/'
referenceURL: https://docs.gitlab.com/ee/administration/logs/#audit_jsonlog
fields:
    - name: severity
      description: The log level. Present only in audit log files.
      type: string
    - name: time
      description: The event timestamp. Present only in audit log files.
      type: timestamp
      timeFormats:
        - rfc3339
      isEventTime: true
    - name: author_id
      required: true
      description: User id that made the change
      type: bigint
    - name: entity_id
      required: true
      description: Id of the entity that was modified
      type: bigint
    - name: entity_type
      required: true
      description: Type of the modified entity
      type: string
    - name: change
      description: Type of change to the settings. Present only in audit log files.
      type: string
    - name: from
      description: Old setting value. Present only in audit log files.
      type: string
    - name: to
      description: New setting value. Present only in audit log files.
      type: string
    - name: author_name
      required: true
      description: Name of the user that made the change
      type: string
    - name: target_id
      required: true
      description: Target id of the modified setting
      type: bigint
    - name: target_type
      required: true
      description: Target type of the modified setting
      type: string
    - name: target_details
      required: true
      description: Details of the target of the modified setting
      type: string
    - name: created_at
      description: Timestamp when event was triggered. Present only in audit event streaming
      type: timestamp
      timeFormats:
        - rfc3339
      isEventTime: true
    - name: details
      description: JSON object containing additional metadata. Present only in audit event streaming
      type: json
    - name: entity_path
      description: Full path of the entity affected by the auditable event. Present only in audit event streaming
      type: string
    - name: event_type
      description: String representation of the type of audit event. Present only in audit event streaming
      type: string
    - name: id
      description: Unique identifier for the audit event. Present only in audit event streaming
      type: bigint
    - name: ip_address
      description: IP address of the host used to trigger the event. Present only in audit event streaming
      type: string
      indicators:
        - ip

GitLab.Exceptions

GitLab log file containing changes to group or project settings

Reference: GitLab documentation on exceptions for JSON logs.

schema: GitLab.Exceptions
description: GitLab log file containing changes to group or project settings
referenceURL: https://docs.gitlab.com/ee/administration/logs/#exceptions_jsonlog
fields:
    - name: severity
      required: true
      description: The log level
      type: string
    - name: time
      required: true
      description: The event timestamp
      type: timestamp
      timeFormats:
        - rfc3339
      isEventTime: true
    - name: correlation_id
      description: Request unique id across logs
      type: string
      indicators:
        - trace_id
    - name: extra.server
      description: Information about the server on which the exception occurred
      type: object
      fields:
        - name: os
          description: Server OS info
          type: object
          fields:
            - name: name
              description: OS name
              type: string
            - name: version
              description: OS version
              type: string
            - name: build
              description: OS build
              type: string
        - name: runtime
          description: Runtime executing gitlab code
          type: object
          fields:
            - name: name
              description: Runtime name
              type: string
            - name: version
              description: Runtime version
              type: string
    - name: extra.project_id
      description: Project id where the exception occurred
      type: bigint
    - name: extra.relation_key
      description: Relation on which the exception occurred
      type: string
    - name: extra.relation_index
      description: Relation index on which the exception occurred
      type: bigint
    - name: exception.class
      required: true
      description: Class name of the exception that occurred
      type: string
    - name: exception.message
      required: true
      description: Message of the exception that occurred
      type: string
    - name: exception.backtrace
      description: Stack trace of the exception that occurred
      type: array
      element:
        type: string

GitLab.Git

GitLab log file containing all failed requests from GitLab to Git repositories.

Reference: GitLab documentation on git for JSON logs.

schema: GitLab.Git
description: GitLab log file containing all failed requests from GitLab to Git repositories.
referenceURL: https://docs.gitlab.com/ee/administration/logs/#git_jsonlog
fields:
    - name: severity
      required: true
      description: The log level
      type: string
    - name: time
      required: true
      description: The event timestamp
      type: timestamp
      timeFormats:
        - rfc3339
      isEventTime: true
    - name: correlation_id
      description: Unique id across logs
      type: string
      indicators:
        - trace_id
    - name: message
      required: true
      description: The error message from git
      type: string

GitLab.Integrations

GitLab log with information about integrations activities such as Jira, Asana, and Irker services.

Reference: GitLab documentation on integrations for JSON logs.

schema: GitLab.Integrations
description: GitLab log with information about integrations activities such as Jira, Asana, and Irker services.
referenceURL: https://docs.gitlab.com/ee/administration/logs/#integrations_jsonlog
fields:
    - name: severity
      required: true
      description: The log level
      type: string
    - name: time
      required: true
      description: The event timestamp
      type: timestamp
      timeFormats:
        - rfc3339
      isEventTime: true
    - name: service_class
      required: true
      description: The class name of the integrated service
      type: string
    - name: project_id
      required: true
      description: The project id the integration was running on
      type: bigint
    - name: project_path
      required: true
      description: The project path the integration was running on
      type: string
    - name: message
      required: true
      description: The log message from the service
      type: string
    - name: client_url
      required: true
      description: The client url of the service
      type: string
      indicators:
        - url
    - name: error
      description: The error name if an error has occurred
      type: string

GitLab.Production

GitLab log for Production controller requests received from GitLab

Reference: GitLab documentation on production for JSON logs.

schema: GitLab.Production
description: GitLab log for Production controller requests received from GitLab
referenceURL: https://docs.gitlab.com/ee/administration/logs/#production_jsonlog
fields:
    - name: method
      required: true
      description: The HTTP method of the request
      type: string
    - name: path
      required: true
      description: The URL path for the request
      type: string
    - name: format
      description: The response output format
      type: string
    - name: controller
      description: The Production controller class name
      type: string
    - name: action
      description: The Production controller action
      type: string
    - name: status
      required: true
      description: The HTTP response status code
      type: bigint
    - name: time
      required: true
      description: The request timestamp
      type: timestamp
      timeFormats:
        - rfc3339
      isEventTime: true
    - name: params
      description: The URL query parameters
      type: array
      element:
        type: object
        fields:
            - name: key
              required: true
              description: Query parameter name
              type: string
            - name: value
              description: Query parameter value
              type: json
    - name: remote_ip
      description: The remote IP address of the HTTP request
      type: string
      indicators:
        - ip
    - name: user_id
      description: The user id of the request
      type: bigint
    - name: username
      description: The username of the request
      type: string
      indicators:
        - username
    - name: ua
      description: The User-Agent of the requester
      type: string
    - name: queue_duration_s
      description: Total time that the request was queued inside GitLab Workhorse
      type: float
    - name: gitaly_calls
      description: Total number of calls made to Gitaly
      type: bigint
    - name: gitaly_duration_s
      description: Total time taken by Gitaly calls
      type: float
    - name: redis_calls
      description: Total number of calls made to Redis
      type: bigint
    - name: redis_duration_s
      description: Total time to retrieve data from Redis
      type: float
    - name: redis_read_bytes
      description: Total bytes read from Redis
      type: bigint
    - name: redis_write_bytes
      description: Total bytes written to Redis
      type: bigint
    - name: correlation_id
      description: Request unique id across logs
      type: string
      indicators:
        - trace_id
    - name: cpu_s
      description: Total time spent on CPU
      type: float
    - name: db_duration_s
      description: Total time to retrieve data from PostgreSQL
      type: float
    - name: view_duration_s
      description: Total time taken inside the Rails views
      type: float
    - name: duration_s
      required: true
      description: Total time taken to retrieve the request
      type: float
    - name: meta.caller_id
      description: Caller ID
      type: string
    - name: location
      description: (Applies only to redirects) The redirect URL
      type: string
    - name: exception.class
      description: Class name of the exception that occurred
      type: string
    - name: exception.message
      description: Message of the exception that occurred
      type: string
    - name: exception.backtrace
      description: Stack trace of the exception that occurred
      type: array
      element:
        type: string
    - name: etag_route
      description: Route name etag (on redirects)
      type: string