Managing AWS S3 Log Sources with Terraform

Manage S3 log sources as code in Terraform

Overview

You can define your Panther S3 log source completely in Terraform—meaning you can create the S3 bucket and associated infrastructure in AWS, and onboard it to your Panther instance. Panther is a Terraform provider.

Other methods to create an S3 log source include using the Panther API directly and manual creation in the Panther Console.

How to define your Panther S3 log source in Terraform

The following sections outline how to define your S3 log source in HashiCorp Configuration Language (HCL). You will define both AWS and Panther infrastructure in HCL.

The first log source you create in Panther must be done in the Panther Console. If you use Terraform to set up your first Panther log source, you may run into a "pending confirmation" issue detailed in this Knowledge Base article.

Prerequisite

• Before starting, ensure you have an API URL and token with the Manage Log Sources permission. This is required to complete Step 2.

  • If needed, follow these instructions for creating an API token in the Panther Console.

Step 1: Define variables

• Create a variables.tf file with the following AWS and Panther variables:

variable "aws_account_id" {
  type        = string
  description = "The AWS account ID where the template is being deployed"
}

variable "panther_aws_account_id" {
  type        = string
  description = "The AWS account ID of your Panther instance"
}

variable "panther_aws_region" {
  type = string
  default = "us-east-1"
  description = "The region where the Panther instance is deployed"
}

variable "panther_aws_partition" {
  type        = string
  default     = "aws"
  description = "AWS partition of the account running the Panther backend e.g aws, aws-cn, or aws-us-gov"
}

variable "s3_bucket_name" {
  type        = string
  description = "The S3 Bucket name to onboard"
}

variable "log_source_name" {
  type = string
  description = "The name of the log source to be created in Panther"
}

variable "panther_api_token" {
  type = string
}

variable "panther_api_url" {
  type = string
}

Step 2: Provide values for the defined variables

• Add a *.tfvars file that assigns values to the variables you defined in Step 1. Note that to complete this section, you will need the API URL and token outlined in the Prerequisite section.

  • Your panther_api_url value should be your root API URL. This is either:

    • A GraphQL API URL without the /public/graphql suffix
    • A REST API URL as-is (REST URLs do not have a suffix after the root URL)

aws_account_id         = "XXXXXXXXXX"
panther_aws_account_id = "XXXXXXXXXX"
panther_aws_region     = "us-east-1"
panther_aws_partition  = "aws"
s3_bucket_name         = "test-bucket"
log_source_name        = "test-integration"
panther_api_token      = "XXXXXXXXXX"
panther_api_url        = "https://your-panther-url/v1"

Step 3: Define Terraform providers

• Include both the the AWS and Panther Terraform providers:

terraform {
  required_providers {
    panther = {
      source = "panther-labs/panther"
    }
    aws = {
      source  = "hashicorp/aws"
    }
  }
}

Step 4: Define AWS infrastructure

In AWS, you need to create an S3 bucket and an SNS topic. To ingest logs from an S3 bucket, the bucket must write notifications to an SNS topic on object creation. A subscription on this topic is needed to push object information onto Panther's input queue.

Define S3 bucket

The following HCL configuration defines the S3 bucket and associated IAM role for accessing its contents. This role requires read permissions on the S3 bucket, as it will be assumed by your Panther instance to read incoming logs.

resource "aws_s3_bucket" "log_bucket" {
  bucket = var.s3_bucket_name
}

resource "aws_iam_role" "log_processing_role" {
  name = "PantherLogProcessingRole-${var.s3_bucket_name}"

  # Policy that grants an entity permission to assume the role
  assume_role_policy = jsonencode({
    Version : "2012-10-17",
    Statement : [
      {
        Action : "sts:AssumeRole",
        Effect : "Allow",
        Principal : {
          AWS : "arn:${var.aws_partition}:iam::${var.panther_aws_account_id}:root"
        }
        Condition : {
          Bool : { "aws:SecureTransport" : true }
        }
      }
    ]
  })

  tags = {
    Application = "Panther"
  }
}


# Provides an IAM role inline policy for reading S3 data
resource "aws_iam_role_policy" "read_data_policy" {
  name = "ReadData"
  

  role = aws_iam_role.log_processing_role.id
  policy = jsonencode({
    Version : "2012-10-17",
    Statement : [
      {
        Effect : "Allow",
        Action : [
          "s3:GetBucketLocation",
          "s3:ListBucket",
        ],
        Resource : "arn:${var.aws_partition}:s3:::${aws_s3_bucket.log_bucket.bucket}"
      },
      {
        Effect : "Allow",
        Action : "s3:GetObject",
        Resource : "arn:${var.aws_partition}:s3:::${aws_s3_bucket.log_bucket.bucket}/*"
    }, ]
  })
}

Define SNS topic

The following HCL configuration creates the SNS topic and related policy for enabling S3 bucket notifications. It also creates a subscription to forward messages to Panther's input data notifications queue.

The same SNS topic can be used for multiple S3 buckets integrations.

resource "aws_sns_topic" "panther_notifications_topic" {
  name = "panther-notifications-topic"
}

resource "aws_sns_topic_policy" "default" {
  arn = aws_sns_topic.panther_notifications_topic.arn

  policy = data.aws_iam_policy_document.panther_notifications_topic_policy.json
}

data "aws_iam_policy_document" "panther_notifications_topic_policy" {
  statement {
    sid = "AllowS3EventNotifications"
    actions = [
      "sns:Publish",
    ]
    effect = "Allow"
    principals {
      type = "Service"
      identifiers = ["s3.amazonaws.com"]
    }
    resources = [
      aws_sns_topic.panther_notifications_topic.arn,
    ]
  }
  statement {
    sid = "AllowCloudTrailNotification"
    actions = [
      "sns:Publish",
    ]
    effect = "Allow"
    principals {
      type = "Service"
      identifiers = ["cloudtrail.amazonaws.com"]
    }
    resources = [
      aws_sns_topic.panther_notifications_topic.arn,
    ]
  }
  statement {
    sid = "AllowSubscriptionToPanther"
    actions = [
      "sns:Subscribe",
    ]
    effect = "Allow"
    principals {
      type = "AWS"
      identifiers = ["arn:${var.panther_aws_partition}:iam::${var.panther_aws_account_id}:root"]
    }
    resources = [
      aws_sns_topic.panther_notifications_topic.arn,
    ]
  }
}

resource "aws_s3_bucket_notification" "panther_event_notifications" {
  bucket = aws_s3_bucket.log_bucket.id

  topic {
    topic_arn = aws_sns_topic.panther_notifications_topic.arn
    events        = ["s3:ObjectCreated:*"]
  }
}

resource "aws_sns_topic_subscription" "panther_notifications_subscription" {
  topic_arn = aws_sns_topic.panther_notifications_topic.arn
  protocol  = "sqs"
  endpoint  = "arn:${var.panther_aws_partition}:sqs:${var.panther_aws_region}:${var.panther_aws_account_id}:panther-input-data-notifications-queue"
  raw_message_delivery = false
}

Step 5: Define Panther S3 log source

The following HCL configuration defines the S3 log source in Panther.

Note that panther_managed_bucket_notifications_enabled is set to false. This indicates that all of the infrastructure related to this log source is being managed externally, in this case through Terraform.

provider "panther" {
  token = var.panther_api_token
  url   = var.panther_api_url
}

resource "panther_s3_source" "demo_source" {
  aws_account_id                          = var.aws_account_id
  name                                    = var.log_source_name
  log_processing_role_arn                 = aws_iam_role.log_processing_role.arn
  log_stream_type                         = "JSON"
  panther_managed_bucket_notifications_enabled = false
  bucket_name                             = aws_s3_bucket.log_bucket.bucket
  prefix_log_types = [{
    excluded_prefixes = []
    log_types         = ["AWS.CloudTrail"]
    prefix            = ""
  }]
}