Managing AWS S3 Log Sources with Terraform
Manage S3 log sources as code in Terraform
You can define your S3 log source—meaning you can create the S3 bucket and associated infrastructure in AWS, and onboard it to your Panther instance—all in Terraform. Panther is a Terraform provider.
Other methods to create an S3 log source include using the Panther API directly and manual creation in the Panther Console.
The following sections outline how to define your S3 log source in HashiCorp Configuration Language (HCL). You will define both AWS and Panther infrastructure in HCL.
- Before starting, ensure you have a GraphQL API URL and token with the
Manage Log Sourcespermission. This is required to complete Step 4.
- Define a
*.tfvarsfile with the following AWS and Panther variables.
variable "aws_account_id" {
type = string
description = "The AWS account ID where the template is being deployed"
}
variable "panther_aws_account_id" {
type = string
description = "The AWS account ID of your Panther instance"
}
variable "panther_aws_region" {
type = string
default = "us-east-1"
description = "The region where the Panther instance is deployed"
}
variable "panther_aws_partition" {
type = string
default = "aws"
description = "AWS partition of the account running the Panther backend e.g aws, aws-cn, or aws-us-gov"
}
variable "s3_bucket_name" {
type = string
description = "The S3 Bucket name to onboard"
}
variable "log_source_name" {
type = string
description = "The name of the log source to be created in Panther"
}
variable "panther_api_token" {
type = string
}
variable "panther_api_url" {
type = string
}
terraform {
required_providers {
panther = {
source = "panther-labs/panther"
}
aws = {
source = "hashicorp/aws"
}
}
}
In AWS, you need to create an S3 bucket and an SNS topic. To ingest logs from an S3 bucket, the bucket must write notifications to an SNS topic on object creation. A subscription on this topic is needed to push object information onto Panther's input queue.
.png?alt=media&token=b6f126ac-0dcf-428b-ad44-3aac3e7a1f59)
The following HCL configuration defines the S3 bucket and associated IAM role for accessing its contents. This role requires read permissions on the S3 bucket, as it will be assumed by your Panther instance to read incoming logs.
resource "aws_s3_bucket" "log_bucket" {
bucket = var.s3_bucket_name
}
resource "aws_iam_role" "log_processing_role" {
name = "PantherLogProcessingRole-${var.s3_bucket_name}"
# Policy that grants an entity permission to assume the role.
assume_role_policy = jsonencode({
Version : "2012-10-17",
Statement : [
{
Action : "sts:AssumeRole",
Effect : "Allow",
Principal : {
AWS : "arn:${var.aws_partition}:iam::${var.panther_aws_account_id}:root"
}
Condition : {
Bool : { "aws:SecureTransport" : true }
}
}
]
})
tags = {
Application = "Panther"
}
}
# Provides an IAM role inline policy for reading S3 Data
resource "aws_iam_role_policy" "read_data_policy" {
name = "ReadData"
role = aws_iam_role.log_processing_role.id
policy = jsonencode({
Version : "2012-10-17",
Statement : [
{
Effect : "Allow",
Action : [
"s3:GetBucketLocation",
"s3:ListBucket",
],
Resource : "arn:${var.aws_partition}:s3:::${aws_s3_bucket.log_bucket.bucket}"
},
{
Effect : "Allow",
Action : "s3:GetObject",
Resource : "arn:${var.aws_partition}:s3:::${aws_s3_bucket.log_bucket.bucket}/*"
}, ]
})
}
The following HCL configuration creates the SNS topic and related policy for enabling S3 bucket notifications. It also creates a subscription to forward messages to Panther's input data notifications queue.
The same SNS topic can be used for multiple S3 buckets integrations.
resource "aws_sns_topic" "panther_notifications_topic" {
name = "panther-notifications-topic"
}
resource "aws_sns_topic_policy" "default" {
arn = aws_sns_topic.panther_notifications_topic.arn
policy = data.aws_iam_policy_document.panther_notifications_topic_policy.json
}
data "aws_iam_policy_document" "panther_notifications_topic_policy" {
statement {
sid = "AllowS3EventNotifications"
actions = [
"sns:Publish",
]
effect = "Allow"
principals {
type = "Service"
identifiers = ["s3.amazonaws.com"]
}
resources = [
aws_sns_topic.panther_notifications_topic.arn,
]
}
statement {
sid = "AllowCloudTrailNotification"
actions = [
"sns:Publish",
]
effect = "Allow"
principals {
type = "Service"
identifiers = ["cloudtrail.amazonaws.com"]
}
resources = [
aws_sns_topic.panther_notifications_topic.arn,
]
}
statement {
sid = "AllowSubscriptionToPanther"
actions = [
"sns:Subscribe",
]
effect = "Allow"
principals {
type = "AWS"
identifiers = ["arn:${var.panther_aws_partition}:iam::${var.panther_aws_account_id}:root"]
}
resources = [
aws_sns_topic.panther_notifications_topic.arn,
]
}
}
resource "aws_s3_bucket_notification" "panther_event_notifications" {
bucket = aws_s3_bucket.log_bucket.id
topic {
topic_arn = aws_sns_topic.panther_notifications_topic.arn
events = ["s3:ObjectCreated:*"]
}
}
resource "aws_sns_topic_subscription" "panther_notifications_subscription" {
topic_arn = aws_sns_topic.panther_notifications_topic.arn
protocol = "sqs"
endpoint = "arn:${var.panther_aws_partition}:sqs:${var.panther_aws_region}:${var.panther_aws_account_id}:panther-input-data-notifications-queue"
raw_message_delivery = false
}
The following HCL configuration defines the S3 log source in Panther. Note that to complete this section, you will need the API URL and token outlined in the Prerequisite section.
Note that
panther_managed_bucket_notifications_enabled is set to false. This indicates that all of the infrastructure related to this log source is being managed externally, in this case through Terraform.provider "panther" {
token = var.panther_api_token
url = var.panther_api_url
}
resource "panther_s3_source" "demo_source" {
aws_account_id = var.aws_account_id
name = var.log_source_name
log_processing_role_arn = aws_iam_role.log_processing_role.arn
log_stream_type = "JSON"
panther_managed_bucket_notifications_enabled = false
bucket_name = aws_s3_bucket.log_bucket.bucket
prefix_log_types = [{
excluded_prefixes = []
log_types = ["AWS.CloudTrail"]
prefix = ""
}]
}