Google Cloud Storage (GCS) Source
Onboarding Google Cloud Storage (GCS) as a Data Transport log source in the Panther Console

Overview

With Google Cloud Storage (GCS) as a log source, Panther can pull log data directly from GCS buckets, write rules, and run queries on this processed data.

Prerequisites

Panther requires certain configurations within Google Cloud Platform (GCP) to authenticate and pull logs. A bucket and a subscription for a topic set up with notifications are required. Panther ingests new files through Pub/Sub notifications.

How to connect GCS as a Data Transport log source in Panther

  1. 1.
    Log in to the Panther Console.
  2. 2.
    In the left sidebar, click Configure > Log Sources.
  3. 3.
    Click Create New.
  4. 4.
    Under Custom Onboarding via data transports, click Google Cloud Storage, then click Start source setup.
  5. 5.
    On the "Configure your source" page, fill in the fields:
    • Name: Enter a descriptive name for the GCS log source.
    • Log Types: Select the Log Types Panther should use to parse your GCS logs. Note: At least one Log Type must be selected from the dropdown menu.
    • GCS Prefix Filter: Define a prefix to tell Panther which folders to include in the event that your GCS instance contains multiple data types. Leave this field blank if you want to allow ingestion of all files.
  6. 6.
    Click Continue Setup.
  7. 7.
    On the "Infrastructure & Credentials" page, follow the steps to create the infrastructure component with a Terraform template. If you do not want to use a Terraform Template, you can follow our alternative documentation to complete the infrastructure components process manually.
    1. 1.
      Download and complete the Terraform template
      • Download the Terraform Template.
      • Fill out the fields in the production.tfvars file with your configuration.
      • Initialize a working directory containing Terraform configuration files by running the Terraform Command schema provided.
      • Copy the corresponding Terraform of gcloud command schema provided and run it in your CLI.
      • Generate a JSON keyfile by replacing the value for your service account email in the gcloud command code listed.
        • You can find the key file in the output of the Terraform run.
    2. 2.
      Provide pulling configuration & JSON Keyfile
      • Drag and drop or upload the JSON key into the correct field in Step 2.
      • Paste in your GCS Bucket Name and Pub/Sub Subscription ID, found in the Subscriptions section of your Google Cloud account.
  8. 8.
    Click Continue Setup.
  9. 9.
    On the final "Verify Setup" confirmation page, toggle the alarm button to YES if you would like Panther to send you an alert in case your source does not produce any events.
    • Edit the time period fields "Number" and "Period" to select your alert interval.
    • Note: To create new or modify existing alert destinations, see Destinations.

Alternative to Terraform template: Configuring your integration manually in Google Cloud Platform (GCP)

If you choose to create the infrastructure components manually rather than using a Terraform template during the GCS setup above, follow the instructions below.
  1. 1.
    Log in to your Google Cloud console.
  2. 2.
    Determine which bucket Panther will pull logs from.
  3. 3.
    Create a topic for the notifications.
    • You can create a topic using the gcloud CLI tool with the following command format: gcloud pubsub topics create $TOPIC_ID
  4. 4.
    Configure the bucket to send notifications for new files to the topic you created.
    • You can create a notification using the gcloud CLI tool with the following command format: gsutil notification create -t $TOPIC_NAME -e OBJECT_FINALIZE -f json gs://$BUCKET_NAME
    • Note: Panther only requires the OBJECT_FINALIZE type.
  5. 5.
    Create a subscription to be used with the topic you created. Note: This subscription should not be used by any service other than Panther.
    • You can create a subscription using the gcloud CLI tool with the following command format: gcloud pubsub subscriptions create $SUBSCRIPTION_ID --topic $TOPIC_ID --topic-project $PROJECT_ID
  6. 6.
    Create a new Google Cloud service account and take note of the account email address. Panther will use this to be able to access the infrastructure created for this GCS integration.
    • The following permissions are required for the project where the Pub/Sub subscription and topic lives:
      Permissions required
      Role
      Scope
      storage.objects.get
      storage.objects.list
      storage/viewer
      bucket-name
      pubsub.subscriptions.consume
      pubsub/subscriber
      subscription-name
      pubsub.subscriptions.get
      pubsub/viewer
      subscription-name
      monitoring.timeSeries.list
      monitoring/viewer
      project
    • Note: You can set conditions or IAM policies on permissions for specific resources. This can be done either in the IAM page of the service account (as seen in the example screenshot below) or in the specific resource's page.
  7. 7.
    Generate a JSON key file for the service account, which will be used in Panther to authenticate to the GCP infrastructure.
    • You can create a JSON key file using the gcloud CLI tool with the following command format: gcloud iam service-accounts keys create $KEYFILE_PATH --iam-account=$SERVICE_ACCOUNT_EMAIL

View collected logs

After GCS log sources are fully configured, you can search your data in Data Explorer. For more information and for example queries, please see the documentation on Data Explorer.
Copy link
Outline
Overview
Prerequisites
How to connect GCS as a Data Transport log source in Panther
Alternative to Terraform template: Configuring your integration manually in Google Cloud Platform (GCP)
View collected logs