Managing Google Cloud Storage (GCS) Log Sources with Terraform (Beta)

Manage GCS log sources as code in Terraform

Overview

Managing Google Cloud Storage (GCS) log sources with Terraform is in open beta starting with Panther version 1.121, and is available to all customers. Please share any bug reports and feature requests with your Panther support team.

You can define your Google Cloud Storage (GCS) log source in Terraform using the Panther Terraform provider. This allows you to manage your GCS log sources as infrastructure as code, enabling version control and automated deployments.

Other methods to create a GCS log source include using the Panther API directly and manual creation in the Panther Console.

How to define your Panther GCS log source in Terraform

The following sections outline how to define your GCS log source in HashiCorp Configuration Language (HCL).

Prerequisites

• Before starting, ensure you have an API URL and token with the Manage Log Sources permission. This is required to complete Step 3.

  • If needed, follow these instructions for creating an API token in the Panther Console.

• Set up your Google Cloud Platform (GCP) infrastructure following the GCS Source setup guide.

Step 1: Choose an authentication method

Select an authentication method for accessing your GCS bucket:

• Service Account: Uses a Google Cloud service account with a JSON key file

• Workload Identity Federation: Uses Google Cloud Workload Identity Federation with AWS

The authentication method you select will determine the variables you define in Step 2, below.

Step 2: Define variables

Define a variables.tf file with the variables shown in the code block below.

variable "panther_api_token" {
  description = "Panther API token"
  type        = string
  sensitive   = true
}

variable "panther_api_url" {
  description = "Panther API URL"
  type        = string
}

variable "integration_label" {
  description = "The name of the GCS log source integration"
  type        = string
}

variable "gcs_bucket" {
  description = "The name of the GCS bucket to pull logs from"
  type        = string
}

variable "subscription_id" {
  description = "The Pub/Sub subscription ID for GCS bucket notifications"
  type        = string
}

variable "credentials_type" {
  description = "Type of credentials (service_account or wif)"
  type        = string
  validation {
    condition     = contains(["service_account", "wif"], var.credentials_type)
    error_message = "Credentials type must be either 'service_account' or 'wif'."
  }
}

// Auth variables are specific to credentials_type. See table below
variable "credentials" {
  description = "Service account JSON key or WIF credential configuration file content"
  type        = string
  sensitive   = true
}

variable "project_id" {
  description = "Google Cloud Project ID (required for WIF, optional for service account)"
  type        = string
  default     = ""
}

// (Optional) Relevant only when log_stream_type = "JsonArray"
variable "json_array_envelope_field" {
  description = "Envelope field for json array stream"
  type        = string
  default     = ""
}

// (Optional) Relevant only when log_stream_type = "XML"
variable "xml_root_element" {
  description = "Root element name for XML log streams"
  type        = string
  default     = ""
}

Step 3: Provide values for the defined variables

Add a *.tfvars file that assigns values to the variables you defined in Step 2. Note that to complete this section, you will need the API URL and token outlined in the Prerequisites section.

• Your panther_api_url value should be your root API URL. This is either:

  • A GraphQL API URL without the /public/graphql suffix

  • A REST API URL as-is (REST URLs do not have a suffix after the root URL)

panther_api_token         = "XXXXXXXXXX"
panther_api_url           = "https://your-panther-url/v1"
integration_label         = "my-gcs-logs"
gcs_bucket                = "my-log-bucket"
subscription_id           = "my-panther-subscription"
credentials_type          = "service_account" // service_account or wif
credentials               = "{ ... }" // JSON keyfile or credential config content
project_id                = "my-gcp-project" // Required for WIF, optional for service_account
json_array_envelope_field = "" // Optional, relevant only when log_stream_type = "JsonArray"
xml_root_element          = "" // Optional, relevant only when log_stream_type = "XML"

Authentication method-specific variables

In your variables.tf file, include the values in the Additional variables column below for the authentication method you chose in Step 1.

The credentials_type field must match the type of credentials provided in the credentials field.

Authentication method	credentials_type value	Additional variables
Service account authentication	service_account	credentials (JSON keyfile content)
Workload Identity Federation authentication	wif	credentials (credential configuration file content), project_id

Step 4: Define the Terraform provider

Add the Panther Terraform provider.

terraform {
  required_providers {
    panther = {
      source  = "panther-labs/panther"
      version = "~> 0.2.10"
    }
  }
}

Step 5: Define Panther GCS log source

The following HCL configuration defines the GCS log source in Panther.

provider "panther" {
  token = var.panther_api_token
  url   = var.panther_api_url
}

resource "panther_gcssource" "demo_gcs_source" {
  integration_label = var.integration_label
  log_stream_type   = "JSON" // Options: Auto, JSON, JsonArray, Lines, and XML
  gcs_bucket        = var.gcs_bucket
  subscription_id   = var.subscription_id
  credentials_type  = var.credentials_type
  credentials       = var.credentials
  project_id        = var.project_id // Required for WIF, optional for service_account

  prefix_log_types = [{
    prefix            = "audit-logs/"
    excluded_prefixes = ["audit-logs/exclude/"]
    log_types         = ["GCP.AuditLog"]
  }, {
    prefix            = "cloudtrail/"
    excluded_prefixes = []
    log_types         = ["AWS.CloudTrail"]
  }]

  // (Optional) Configure based on log_stream_type
  log_stream_type_options = {
    json_array_envelope_field = var.json_array_envelope_field // Relevant for "JsonArray"
    xml_root_element          = var.xml_root_element          // Relevant for "XML"
  }
}

Prefix-based log type mapping

Unlike HTTP or S3 sources that use a simple array of log types, GCS sources use prefix_log_types to map different prefixes within the bucket to specific log types:

• prefix: The GCS object prefix to match (e.g., "audit-logs/", "application-logs/")

• excluded_prefixes: Optional array of prefixes to exclude within the main prefix

• log_types: Array of log type schemas to apply to objects matching this prefix

This allows a single GCS bucket to contain multiple types of logs organized by prefix, with different schemas applied to each.

Complete example

For a complete working example with GCP infrastructure setup, see the Panther auxiliary repository.