Ctrlk
Knowledge BaseRelease NotesRequest Demo
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Cloud Storage (GCS) Source

Onboarding GCS as a Data Transport log source in the Panther Console

Overview

Panther supports configuring Google Cloud Storage (GCS) as a Data Transport to pull log data directly from GCS buckets, write rules, and run queries on this processed data. Panther uses Pub/Sub to be notified of new data in your bucket that is ready to be consumed.

Panther can authenticate against your source using Google Cloud Workload Identity Federation or a service account.

Data can be sent compressed (or uncompressed). Learn more about compression specifications in Ingesting compressed data in Panther.

How to set up a GCS log source in Panther

Step 1: Begin creating the GCS source in Panther

  1. In the left-hand navigation bar of your Panther Console, click Log Sources.

  2. In the upper-right corner, click Create New.

  3. Click the Google Cloud Storage tile.

  4. On the Basic Info page, fill in the fields:

    • Name: Enter a descriptive name for the GCS log source.

    • Prefixes & Schemas: Define combinations of prefixes, schemas, and exclusion filters, according the structure of your data storage in GCS.

      • To attach one or more schemas to all data in the bucket, leave the GCS Prefix field blank. This will create a wildcard (*) prefix.

  5. Click Setup.

  6. On the Log Format page, select the stream type of the incoming logs:

    • Auto

    • Lines

    • JSON

    • JSON Array

  7. Click Continue.

Step 2: Create required Google Cloud Platform (GCP) infrastructure

Before creating GCP infrastructure, you'll need to decide:

If you'd like Panther to authenticate using a Google Cloud service account, follow the instructions in one of the tabs below.

To create GCP infrastructure using Terraform (authenticating with a service account):

  1. On the Infrastructure & Credentials page, click the Service Account tab. Under an "Infrastructure & Credentials" header, a tab labeled "Service Account" is circled.

  2. Click Terraform Template to download the Terraform template.

  3. Fill out the fields in the panther.tfvars file with your configuration.

    • Set authentication_method to "service_account".

  4. Initialize a working directory containing Terraform configuration files and run terraform init.

  5. Copy the corresponding Terraform Command provided and run it in your CLI.

  6. Generate a JSON key file by copying the gcloud Command provided, replacing the value for your service account email address, and running it in your CLI.

    • You can find the service account email in the output of the Terraform Command.

To create the GCP infrastructure components manually in the GCP console (authenticating with a service account):

  1. In your Google Cloud console, determine which bucket Panther will pull logs from.

  2. Create a topic for the notifications.

    • You can create a topic using the gcloud CLI tool with the following command format: gcloud pubsub topics create $TOPIC_ID

  3. Configure the bucket to send notifications for new files to the topic you created.

    • You can create a notification using the gcloud CLI tool with the following command format: gsutil notification create -t $TOPIC_NAME -e OBJECT_FINALIZE -f json gs://$BUCKET_NAME

    • Note: Panther only requires the OBJECT_FINALIZE type.

  4. Create a subscription to be used with the topic you created.

    • You can create a subscription using the gcloud CLI tool with the following command format: gcloud pubsub subscriptions create $SUBSCRIPTION_ID --topic $TOPIC_ID --topic-project $PROJECT_ID

This subscription should not be used by any other service or source.

  1. Enable the IAM API.

  2. Create a new Google Cloud service account. To create the account using the gcloud CLI tool, use the following command format:

    • Make sure to take note of the account email address, as Panther will use this to access the infrastructure created for this GCS integration.

  3. Assign the required IAM roles to the account.

    • The following permissions are required for the project where the Pub/Sub subscription and topic lives:

      Permissions required

      Role

      Scope

      storage.objects.get

      storage.objects.list

      roles/storage.objectViewer

      bucket-name

      pubsub.subscriptions.consume

      roles/pubsub.subscriber

      subscription-name

      pubsub.subscriptions.get

      roles/pubsub.viewer

      subscription-name

      monitoring.timeSeries.list

      roles/monitoring.viewer

      project

      • Note: You can set conditions or IAM policies on permissions for specific resources. This can be done either in the IAM page of the service account (as seen in the example screenshot below) or in the specific resource's page. On the IAM page of the service account, under the header "Grant this service account access to project," there are fields for Rule and Condition.

      • Note: You can create the permissions using the gcloud CLI tool:

        • gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" --role="roles/storage.objectViewer"

        • gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" --role="roles/pubsub.subscriber"

        • gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" --role="roles/pubsub.viewer"

        • gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" --role="roles/monitoring.viewer"

  4. Generate a JSON key file for the service account, which will be used in Panther to authenticate to the GCP infrastructure.

    • To create a JSON key file using the gcloud CLI tool, run the following command format: gcloud iam service-accounts keys create $KEYFILE_PATH --iam-account=$SERVICE_ACCOUNT_EMAIL

    • Alternatively, you can run the above command in GCP's terminal instead of locally.

      1. Click the 3 dots icon menu in the top right, then click Download.

      2. Click the folder icon for Browse.

      3. Navigate to the key file and select it, then click Download.

If you'd like Panther to authenticate using Google Cloud Workload Identity Federation, follow the instructions in one of the tabs below.

To create the GCP infrastructure components using Terraform (authenticating with Workload Identity Federation):

  1. On the Infrastructure & Credentials page, click the Workload Identity Federation tab. Under an "Infrastructure & Credentials" header, a tab labeled "Workload Identity Federation" is circled.

  2. Click Terraform Template to download the Terraform template.

  3. Fill out the fields in the panther.tfvars file with your configuration.

    • Set authentication_method to "workload_identity_federation".

    • Provide values for panther_workload_identity_pool_id, panther_workload_identity_pool_provider_id, and panther_aws_account_id.

  4. Initialize a working directory containing Terraform configuration files and run terraform init.

  5. Copy the corresponding Terraform Command provided and run it in your CLI.

  6. Generate a credential configuration file for the pool by copying the gcloud Command provided, replacing the value for the project number, pool ID, and provider ID, and running it in your CLI.

    • You can find the project number, the pool ID and the provider ID in the output of the Terraform Command.

To create the GCP infrastructure components manually in the GCP console (authenticating with Workload Identity Federation):

  1. In your Google Cloud console, determine which bucket Panther will pull logs from.

Uniform bucket-level access must be enabled on the target bucket in order to grant Workload Identity Federation entities access to cloud storage resources.

  1. Create a topic for the notifications.

    • You can create a topic using the gcloud CLI tool with the following command format: gcloud pubsub topics create $TOPIC_ID

  2. Configure the bucket to send notifications for new files to the topic you created.

    • You can create a notification using the gcloud CLI tool with the following command format: gsutil notification create -t $TOPIC_NAME -e OBJECT_FINALIZE -f json gs://$BUCKET_NAME

    • Note: Panther only requires the OBJECT_FINALIZE type.

  3. Create a subscription to be used with the topic you created.

    • You can create a subscription using the gcloud CLI tool with the following command format: gcloud pubsub subscriptions create $SUBSCRIPTION_ID --topic $TOPIC_ID --topic-project $PROJECT_ID

This subscription should not be used by any other service or source.

  1. Enable the IAM API.

  2. Configure Workload Identity Federation with AWS by following the Configure Workload Identity Federation with AWS or Azure documentation.

    1. As you are defining an attribute mapping(s) and condition, take note of the following examples:

      • Example attribute mappings:

        Google
        AWS

        google.subject

        assertion.arn.extract('arn:aws:sts::{account_id}:')+":"+assertion.arn.extract('assumed-role/{role_and_session}').extract('/{session}')

        attribute.account

        assertion.account

      • Example attribute condition: attribute.account=="<PANTHER_AWS_ACCOUNT_ID>"

The value of the google.subject attribute cannot exceed 127 characters. You may use Common Expression Language (CEL) expressions to transform or combine attributes from the token issued by AWS. The expression suggested in the table above takes this limit into account, and is an attempt at transforming the ARN into a value that uniquely identifies Panther entities. For more information on the AWS attributes, see "Example 2 - Called by user created with AssumeRole" on this AWS documentation page.

b. When you are adding a provider to your identity pool, select AWS.

  1. Assign the required IAM roles to the account.

    • The following permissions are required for the project where the Pub/Sub subscription and topic lives:

Permissions required

Role

Scope

storage.objects.get

storage.objects.list

roles/storage.objectViewer

bucket-name

pubsub.subscriptions.consume

roles/pubsub.subscriber

subscription-name

pubsub.subscriptions.get

roles/pubsub.viewer

subscription-name

monitoring.timeSeries.list

roles/monitoring.viewer

project

  • Note: You can set conditions or IAM policies on permissions for specific resources. This can be done either in the IAM section in GCP (as seen in the example screenshot below) or in the specific resource's page. In the Google Cloud console, the "IAM" navigation bar item is circled. In a slide-out panel, sections titled "Add principals" and "Assign roles" are circled.

  • Note: You can create the permissions using the gcloud CLI tool, where the $PRINCIPAL_ID may be something like: principalSet://iam.googleapis.com/projects/<THE_ACTUAL_GOOGLE_PROJECT_NUMBER>/locations/global/workloadIdentityPools/<THE_ACTUAL_POOL_ID>/attribute.account/<THE_ACTUAL_PANTHER_AWS_ACCOUNT_ID>

    • gcloud projects add-iam-policy-binding $PROJECT_ID --member="$PRINCIPAL_ID" --role="roles/storage.objectViewer"

    • gcloud projects add-iam-policy-binding $PROJECT_ID --member="$PRINCIPAL_ID" --role="roles/pubsub.subscriber"

    • gcloud projects add-iam-policy-binding $PROJECT_ID --member="$PRINCIPAL_ID" --role="roles/pubsub.viewer"

    • gcloud projects add-iam-policy-binding $PROJECT_ID --member="$PRINCIPAL_ID" --role="roles/monitoring.viewer"

  1. Download the credential configuration file, which will be used in Panther to authenticate to the GCP infrastructure.

    • To generate a credential configuration file using the gcloud CLI tool, use the following command format: gcloud iam workload-identity-pools create-cred-config projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$POOL_ID/providers/$PROVIDER_ID --aws --output-file=config.json

Step 3: Provide credential file and configuration values to Panther

If you are using a Google Cloud service account to authenticate:

  1. Under Provide pulling configuration & JSON Keyfile, upload your JSON key file.

  2. Enter your GCS Bucket Name and Pub/Sub Subscription ID, found in the Subscriptions section of your Google Cloud account.

  3. Click Setup. You will be directed to a success screen:

    The success screen reads, "Everything looks good! Panther will now automatically pull & process logs from your account"

    • You can optionally enable one or more Detection Packs.

    • If you have not done so already, click Attach or Infer Schemas to attach one or more schemas to the source.

    • The Trigger an alert when no events are processed setting defaults to YES. We recommend leaving this enabled, as you will be alerted if data stops flowing from the log source after a certain period of time. The timeframe is configurable, with a default of 24 hours.

      The "Trigger an alert when no events are processed" toggle is set to YES. The "How long should Panther wait before it sends you an alert that no events have been processed" setting is set to 1 Day

If you are using Google Cloud Workload Identity Federation to authenticate:

  1. On the Infrastructure & Credentials page, if you have not already, click the Workload Identity Federation tab. Under an "Infrastructure & Credentials" header, a tab labeled "Workload Identity Federation" is circled.

  2. Under Provide pulling configuration & credential configuration file, upload your credential configuration file.

  3. Enter your Project ID, GCS Bucket Name, and Pub/Sub Subscription ID found in the Subscriptions section of your Google Cloud account. In a tab titled "Workload Identity Federation," there are various buttons and form fields, including "GCS Bucket Name," "Pub/Sub Subscription ID," and "Project ID."

  4. Click Setup. You will be directed to a success screen:

    The success screen reads, "Everything looks good! Panther will now automatically pull & process logs from your account"

    • You can optionally enable one or more Detection Packs.

    • If you have not done so already, click Attach or Infer Schemas to attach one or more schemas to the source.

    • The Trigger an alert when no events are processed setting defaults to YES. We recommend leaving this enabled, as you will be alerted if data stops flowing from the log source after a certain period of time. The timeframe is configurable, with a default of 24 hours.

      The "Trigger an alert when no events are processed" toggle is set to YES. The "How long should Panther wait before it sends you an alert that no events have been processed" setting is set to 1 Day

Viewing ingested logs

After your log source is configured, you can search ingested data using Search or Data Explorer.

PreviousGoogle Cloud Sources
NextPub/Sub Source

Last updated

Was this helpful?