Cloud Storage (GCS) Source

Onboarding GCS as a Data Transport log source in the Panther Console

Overview

Panther supports configuring Google Cloud Storage (GCS) as a Data Transport to pull log data directly from GCS buckets, write rules, and run queries on this processed data. Panther uses Pub/Sub to be notified of new data in your bucket that is ready to be consumed.

Data can be sent compressed (or uncompressed). Learn more about compression specifications in Ingesting compressed data in Panther.

How to set up a GCS log source in Panther

Step 1: Begin creating the GCS source in Panther

1. In the left-hand navigation bar of your Panther Console, click Configure > Log Sources.

2. In the upper-right corner, click Create New.

3. Click the Custom Log Formats tile.

4. On the Google Cloud Storage tile, click Start.

5. On the Basic Info page, fill in the fields:

   • Name: Enter a descriptive name for the GCS log source.

   • Prefixes & Schemas: Define combinations of prefixes, schemas, and exclusion filters, according the structure of your data storage in GCS.

     • To attach one or more schemas to all data in the bucket, leave the GCS Prefix field blank. This will create a wildcard (*) prefix.

6. Click Setup.

7. On the Log Format page, select the stream type of the incoming logs:

   • Auto

   • Lines

   • JSON

   • JSON Array

8. Click Continue.

Step 2: Create required Google Cloud Platform (GCP) infrastructure

You can set up required GCP infrastructure by applying a Terraform template provided by Panther, or manually, in the GCP console.

Terraform

Using Terraform to configure GCP infrastructure

1. On the "Infrastructure & Credentials" page, click Terraform Template to download the Terraform template.

   • You can also find the Terraform template at this GitHub link.

2. Fill out the fields in the panther.tfvars file with your configuration.

3. Initialize a working directory containing Terraform configuration files by running the Terraform Command schema provided.

4. Copy the corresponding Terraform of gcloud command schema provided and run it in your CLI.

5. Generate a JSON key file by replacing the value for your service account email in the gcloud command code listed.

   • You can find the key file in the output of the Terraform run.

Manual creation in GCP

Configuring your GCP infrastructure manually

To create the GCP infrastructure components manually, rather than using a Terraform template, follow the instructions below.

1. In your Google Cloud console, determine which bucket Panther will pull logs from.

   • If you have not created a bucket yet, please see Google's documentation on creating a bucket.

2. Create a topic for the notifications.

   • You can create a topic using the gcloud CLI tool with the following command format: gcloud pubsub topics create $TOPIC_ID

3. Configure the bucket to send notifications for new files to the topic you created.

   • You can create a notification using the gcloud CLI tool with the following command format: gsutil notification create -t $TOPIC_NAME -e OBJECT_FINALIZE -f json gs://$BUCKET_NAME

   • Note: Panther only requires the OBJECT_FINALIZE type.

4. Create a subscription to be used with the topic you created. Note: This subscription should not be used by any other service or source.

   • You can create a subscription using the gcloud CLI tool with the following command format: gcloud pubsub subscriptions create $SUBSCRIPTION_ID --topic $TOPIC_ID --topic-project $PROJECT_ID

5. Enable the IAM API.

6. Create a new Google Cloud service account. To create the account using the gcloud CLI tool, use the following command format:

   gcloud iam service-accounts create SA-NAME \
       --description="DESCRIPTION" \
       --display-name="DISPLAY_NAME"

   • Make sure to take note of the account email address, as Panther will use this to access the infrastructure created for this GCS integration.

7. Assign the required IAM roles to the account.

   • The following permissions are required for the project where the Pub/Sub subscription and topic lives:

     Permissions required	Role	Scope
     storage.objects.get storage.objects.list	roles/storage.objectViewer	bucket-name
     pubsub.subscriptions.consume	roles/pubsub.subscriber	subscription-name
     pubsub.subscriptions.get	roles/pubsub.viewer	subscription-name
     monitoring.timeSeries.list	roles/monitoring.viewer	project

     • Note: You can set conditions or IAM policies on permissions for specific resources. This can be done either in the IAM page of the service account (as seen in the example screenshot below) or in the specific resource's page.
     • Note: You can create the permissions using the gcloud CLI tool:

       • gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" --role="roles/storage.objectViewer"

       • gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" --role="roles/pubsub.subscriber"

       • gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" --role="roles/pubsub.viewer"

       • gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" --role="roles/monitoring.viewer"

8. Generate a JSON key file for the service account, which will be used in Panther to authenticate to the GCP infrastructure.

   • To create a JSON key file using the gcloud CLI tool, use the following command format: gcloud iam service-accounts keys create $KEYFILE_PATH --iam-account=$SERVICE_ACCOUNT_EMAIL

9. Download the key file.

   1. Open the GCP terminal ("Activate Cloud Shell")

   2. Click the 3 dots icon menu in the top right, then click Download.

      • 
   3. Click the folder icon for Browse.

   4. Navigate to the key file and select it, then click Download.

Step 3: Provide key file and configuration values to Panther

1. Under Provide pulling configuration & JSON Keyfile, upload your JSON key file.

2. Enter your GCS Bucket Name and Pub/Sub Subscription ID, found in the Subscriptions section of your Google Cloud account.
3. Click Setup. You will be directed to a success screen:

   

   • You can optionally enable one or more Detection Packs.

   • If you have not done so already, click Attach or Infer Schemas to attach one or more schemas to the source.

   • The Trigger an alert when no events are processed setting defaults to YES. We recommend leaving this enabled, as you will be alerted if data stops flowing from the log source after a certain period of time. The timeframe is configurable, with a default of 24 hours.

     

Viewing ingested logs

After your log source is configured, you can search ingested data using Search or Data Explorer.