Links

Google Cloud Storage (GCS) Source

Onboarding Google Cloud Storage (GCS) as a Data Transport log source in the Panther Console

Overview

With Google Cloud Storage (GCS) as a log source, Panther can pull log data directly from GCS buckets, write rules, and run queries on this processed data.

Prerequisites

Panther requires certain configurations within Google Cloud Platform (GCP) to authenticate and pull logs. A bucket and a subscription for a topic set up with notifications are required. Panther ingests new files through Pub/Sub notifications.

How to connect GCS as a Data Transport log source in Panther

  1. 1.
    Log in to your Panther Console.
  2. 2.
    In the left sidebar menu click Configure > Log Sources.
  3. 3.
    In the upper right corner, click Create New.
  4. 4.
    Click the Custom Log Formats tile.
  5. 5.
    Click Google Cloud Storage.
  6. 6.
    On the "Configure your source" page, fill in the fields:
    • Source's Name: Enter a descriptive name for the GCS log source.
    • Log Stream Type: Select the log format Panther should use to parse your GCS logs.
    • GCS Prefix Filter: Define a prefix to tell Panther which folders to include in the event that your GCS instance contains multiple data types. Leave this field blank if you want to allow ingestion of all files.
  7. 7.
    Click Continue Setup.
  8. 8.
    On the "Infrastructure & Credentials" page, follow the steps to create the infrastructure component with a Terraform template. If you do not want to use a Terraform Template, you can follow our alternative documentation to complete the infrastructure components process manually.
    1. 1.
      Download and complete the Terraform template
      • Download the Terraform Template.
      • Fill out the fields in the production.tfvars file with your configuration.
      • Initialize a working directory containing Terraform configuration files by running the Terraform Command schema provided.
      • Copy the corresponding Terraform of gcloud command schema provided and run it in your CLI.
      • Generate a JSON keyfile by replacing the value for your service account email in the gcloud command code listed.
        • You can find the key file in the output of the Terraform run.
    2. 2.
      Provide pulling configuration & JSON Keyfile
      • Drag and drop or upload the JSON key into the correct field in Step 2.
      • Paste in your GCS Bucket Name and Pub/Sub Subscription ID, found in the Subscriptions section of your Google Cloud account.
        The image shows the Infrastructure and Credentials page in the Panther Console. There is a button labeled "Terraform Template" on the page and CLI commands. At the bottom, you cam select a JSON Keyfile containing the credentials for a user with the required permissions. There are fields for GCS Bucket Name and Pub/Sub Subscription ID.
  9. 9.
    Click Continue Setup.
    • The page will display a success message that says "Everything looks good!"
      A screen in the Panther Console displays the message "Everything looks good!"
  10. 10.
    To finish the source setup:
    1. 1.
      Optionally configure a log drop-off alarm.
      • Before you finish the setup, we recommend that you create a log drop-off alarm to alert you if data stops flowing from the log source. Be sure to set an appropriate time interval for when you would like Panther to alert you that the log source is not sending data.
        There is a toggle setting to enable the alarm in case the source does not process any events. It is switched to "on."
    2. 2.
      Optionally enable a Detection Pack.
    3. 3.
      Click Finish Setup.

Alternative to Terraform template: Configuring your integration manually in Google Cloud Platform (GCP)

If you choose to create the infrastructure components manually rather than using a Terraform template during the GCS setup above, follow the instructions below.
  1. 1.
    Log in to your Google Cloud console.
  2. 2.
    Determine which bucket Panther will pull logs from.
  3. 3.
    Create a topic for the notifications.
    • You can create a topic using the gcloud CLI tool with the following command format: gcloud pubsub topics create $TOPIC_ID
  4. 4.
    Configure the bucket to send notifications for new files to the topic you created.
    • You can create a notification using the gcloud CLI tool with the following command format: gsutil notification create -t $TOPIC_NAME -e OBJECT_FINALIZE -f json gs://$BUCKET_NAME
    • Note: Panther only requires the OBJECT_FINALIZE type.
  5. 5.
    Create a subscription to be used with the topic you created. Note: This subscription should not be used by any service other than Panther.
    • You can create a subscription using the gcloud CLI tool with the following command format: gcloud pubsub subscriptions create $SUBSCRIPTION_ID --topic $TOPIC_ID --topic-project $PROJECT_ID
  6. 7.
    Create a new Google Cloud service account. Make sure to take note of the account email address, as Panther will use this to access the infrastructure created for this GCS integration.
    • To create the account using the gcloud CLI tool, use the following command format: gcloud iam service-accounts create $PROJECT_ID --description="$DESCRIPTION" --display-name="$DISPLAY_NAME"
  7. 8.
    Assign the required IAM roles to the account.
    • The following permissions are required for the project where the Pub/Sub subscription and topic lives:
      Permissions required
      Role
      Scope
      storage.objects.get
      storage.objects.list
      roles/storage.objectViewer
      bucket-name
      pubsub.subscriptions.consume
      roles/pubsub.subscriber
      subscription-name
      pubsub.subscriptions.get
      roles/pubsub.viewer
      subscription-name
      monitoring.timeSeries.list
      roles/monitoring.viewer
      project
      • Note: You can set conditions or IAM policies on permissions for specific resources. This can be done either in the IAM page of the service account (as seen in the example screenshot below) or in the specific resource's page.
        On the IAM page of the service account, under the header "Grant this service account access to project," there are fields for Rule and Condition.
      • Note: You can create the permissions using the gcloud CLI tool:
        • gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" --role="roles/storage.objectViewer"
        • gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" --role="roles/pubsub.subscriber"
        • gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" --role="roles/pubsub.viewer"
        • gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" --role="roles/monitoring.viewer"
  8. 9.
    Generate a JSON key file for the service account, which will be used in Panther to authenticate to the GCP infrastructure.
    • To create a JSON key file using the gcloud CLI tool, use the following command format: gcloud iam service-accounts keys create $KEYFILE_PATH --iam-account=$SERVICE_ACCOUNT_EMAIL
  9. 10.
    Download the key file.
    1. 1.
      Open the GCP terminal ("Activate Cloud Shell")
    2. 2.
      Click the 3 dots icon menu in the top right, then click Download.
    3. 3.
      Click the folder icon for Browse.
    4. 4.
      Navigate to the key file and select it, then click Download.

View collected logs

After GCS log sources are fully configured, you can search your data in Data Explorer. For more information and for example queries, please see the documentation on Data Explorer.