S3 Source

Onboarding AWS S3 as a Data Transport log source in the Panther Console

Overview

Panther supports configuring your Amazon S3 bucket as a Data Transport to pull security logs from S3 buckets.

First you will configure the S3 source in your Panther Console, then you will configure your S3 bucket to send notifications when it receives new data.

Data can be sent compressed (or uncompressed). Learn more about compression specifications in Ingesting compressed data in Panther.

See the diagram below to understand how data flows from your application(s) into Panther using S3 (in SaaS):

A diagram shows how data flows from a customer application into Panther, using the S3 bucket Data Transport. The flow is as follows: Application(s), S3 bucket, SNS topic, SQS, Panther application, IAM role (assumed by Panther), KMS key (optional), S3 bucket, Panther application, Parse & normalize, real-time detections, Alerts generated, Long-term retention in Snowflake, and Alert Destinations

How set up an AWS S3 bucket log source in Panther

The instructions below outline how to set up an S3 integration manually, in the Panther Console. It's also possible to manage your S3 log source using the Panther API, or using Terraform.

To set up an S3 log source in Panther, follow the steps below. You can also view the data ingestion video overview for a quick walkthrough of S3 source setup.

Prerequisite

If an Amazon S3 bucket does not already exist, create one by following Amazon's Creating a bucket documentation.

Step 1: Configure the S3 bucket source in Panther

In the left-hand navigation bar of your Panther Console, click Configure > Log Sources.
In the upper right corner, click Create New.
Click the Custom Log Formats tile.
In the AWS S3 Bucket tile on the slide-out panel, click Start.
On the "Configure your source" page, enter values for the following fields:
- Name: Enter a descriptive name for the S3 source.
- AWS Account ID: Enter the 12-digit AWS Account ID where the S3 buckets are located.
- Bucket Name: Enter the ID or name of the S3 bucket to onboard.
- KMS Key ARN (optional): If your data is encrypted using KMS-SSE, provide the ARN of the KMS key.
If you would like to attach schemas for this source and/or configure inclusive or exclusive bucket prefixes, click Configure Prefixes & Schemas (Optional). You can also perform these actions after the source is set up.
1. In the S3 Prefixes & Schemas popup modal, create combinations of S3 prefixes, schemas, and exclusion filters, according the structure of your data storage in S3.
  - To attach one or more schemas to all data in the bucket, leave the S3 Prefix field blank. This will create a wildcard (*) prefix.
2. Click Apply Changes.
Click Setup.

If you add a KMS key to your S3 bucket after creating the S3 log source in Panther, you must recreate the log source in Panther with the KMS key. Editing the original source to add the KMS key will not work.

Step 2: Set up an IAM role

Panther needs an AWS IAM role with permissions to read objects from your S3 bucket. You can choose from the following options:

Using the AWS Console UI
- Follow these steps if this is the first S3 source you are setting up with Panther.
CloudFormation or Terraform File
I want to set up everything on my own

Using the AWS Console UI

Launch a CloudFormation stack using the AWS console:

On the Create IAM Role page, locate the tile labeled Using the AWS Console UI. At the bottom of the tile, click Continue.
Click Launch Console UI.
- You will be redirected to the AWS console in a new browser tab, with the template URL pre-filled.
- The CloudFormation stack will create an AWS IAM role with the minimum required permissions to read objects from your S3 bucket.
- Click the "Outputs" tab of the CloudFormation stack in AWS, and note the Role ARN.
Navigate back to the Panther Console.
Enter the Role ARN.
Click Setup.
Proceed to the Finish the Source Setup section in this documentation.

I want to set everything up on my own

Create the IAM role manually, then fill in the role ARN in Panther. When you set up the IAM role manually, you must also follow the instructions below to configure your S3 buckets to send notifications when new data arrives.

On the Create IAM Role page, click the link that says I want to set everything up on my own.

Create the required IAM role. You may create the required IAM role manually or through your own automation.

The IAM policy, which will be attached to the role, must include the statements defined below:

{
    "Version": "2012-10-17",    
    "Statement": [
        {
            "Action": "s3:GetBucketLocation",
            "Resource": "arn:aws:s3:::<bucket-name>",
            "Effect": "Allow"
        },
        {
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<bucket-name>/<input-file-path>",
            "Effect": "Allow"
        }
    ]
}

If your S3 bucket is configured with server-side encryption using AWS KMS, you must include an additional statement granting the Panther API access to the corresponding KMS key. In this case, the policy will look something like this:

{
    "Version": "2012-10-17",    
    "Statement": [
        {
            "Action": "s3:GetBucketLocation",
            "Resource": "arn:aws:s3:::<bucket-name>",
            "Effect": "Allow"
        },
        {
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<bucket-name>/<input-file-path>",
            "Effect": "Allow"
        },
        {
            "Action": ["kms:Decrypt", "kms:DescribeKey"],
            "Resource": "arn:aws:kms:<region>:<your-account-id>:key/<kms-key-id>",
            "Effect": "Allow"
        }
    ]
}

In addition to the above, if you want to view the contents of your S3 bucket in the Panther Console (such as to utilize the inferring custom schemas from historical data feature), you will need to add the s3:ListBucket action:

{   
     "Version": "2012-10-17",    
    "Statement": [
        {
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::<bucket-name>",
            "Effect": "Allow"
        },
        {
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<bucket-name>/<input-file-path>",
            "Effect": "Allow"
        }
    ]
}

Add a trust policy to your role with the following AssumeRolePolicyDocument statement so that Panther can assume this role:
```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:<aws-partition>:iam::<panther-master-account-id>:root"
        ]
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "Bool": {
          "aws:SecureTransport": true
        }
      }
    }
  ]
}
```
- Populate <AWS-PARTITION> with the partition of the account running the Panther backend (e.g., aws). Note that we do not deploy to aws-cn or aws-us-gov.
- Populate <PANTHER-MASTER-ACCOUNT-ID> with the 12-digit account ID where Panther is deployed. To get your AWS Account ID: Click the gear icon in the upper right side of the Panther Console to access Settings, then the AWS account ID is displayed at the bottom of the page.
In the Panther Console, enter the Role ARN.
Click Setup.
Proceed to Manual IAM role creation: Additional Steps.

Step 3: Finish the source setup

You will be directed to a success screen:

The success screen reads, "Everything looks good! Panther will now automatically pull & process logs from your account"

If any permission errors are detected, they will be displayed and you will be asked to try configuring the IAM role again.
You can optionally enable one or more Detection Packs.
The Trigger an alert when no events are processed setting defaults to YES. We recommend leaving this enabled, as you will be alerted if data stops flowing from the log source after a certain period of time. The timeframe is configurable, with a default of 24 hours.
If you have not done so already, attach one or more schemas to the source.
1. Click Attach or Infer Schemas.
2. Either attach a Panther-managed schema, or follow these instructions to infer a custom schema from historical S3 data.

Viewing ingested logs

After your log source is configured, you can search ingested data using Search or Data Explorer.

Recommended S3 bucket expiration policy

It is recommended to keep the data added to your S3 bucket for at least seven days before expiring it. Under normal circumstances, Panther processes new objects within minutes of them being added to your S3 bucket, however if the Panther ingestion service is experiencing availability issues, it could take longer for new objects to be processed.

Manual IAM role creation: Additional steps

When you set up the IAM role manually, you must also follow the instructions below to configure your S3 buckets to send notifications when new data arrives.

Note: If you already have configured the bucket to send All object create events to an SNS topic, instead follow the Modify an existing SNS topic tab, and subscribe it to Panther's input data queue.

Only one SNS topic (per AWS account) is required, meaning multiple S3 buckets within one AWS account can all use the same SNS topic. If you've already created an SNS topic for a different S3 bucket in the same AWS account, you can skip this step.

First you need to create an SNS Topic and SNS Subscription to notify Panther that new data is ready for processing.

Log into the AWS Console of the account that owns the S3 bucket.
Select the AWS Region where your S3 bucket is located and navigate to the CloudFormation console.
Navigate to the Stacks section. Select Create Stack (with new resources).

Under the "Specify template" section, enter the following Amazon S3 URL:

https://panther-public-cloudformation-templates.s3-us-west-2.amazonaws.com/panther-log-processing-notifications/latest/template.yml

Specify the following stack details:
- Stack name: A name of your choice, e.g. panther-log-processing-notifications-<bucket-label>
- MasterAccountId: The 12 digit AWS Account ID where Panther is deployed
- PantherRegion: The region where Panther is deployed
- SnsTopicName: The name of the SNS topic receiving the notification. The default value is panther-notifications-topic
Click Next, Next, and then Create Stack to complete the process.
- This stack has one output: SnsTopicArn.

Follow the steps below if you wish to use an existing SNS topic for sending S3 bucket notifications. Note that the SNS topic must be in the same region as your S3 bucket.

Log in to the AWS console and navigate to KMS.
Select the KMS key you want to use for encryption.

Edit the policy to ensure it has the appropriate permissions to be used with the SNS topic and S3 bucket notifications.

Example policy:

{
    "Sid": "Allow access for Key User (SNS Service Principal)",
    "Effect": "Allow",
    "Principal": {
        "Service": "sns.amazonaws.com"
    },
    "Action": [
        "kms:GenerateDataKey*",
        "kms:Decrypt"
    ],
    "Resource": "<SNS-TOPIC-ARN>"
},
{
    "Sid": "Allow access for Key User (S3 Service Principal)",
    "Effect": "Allow",
    "Principal": {
        "Service": "s3.amazonaws.com"
    },
    "Action": [
        "kms:GenerateDataKey*",
        "kms:Decrypt"
    ],
    "Resource": "arn:aws:s3:::<bucket-name>"
}

Click the Encryption tab under the SNS topic.
Click Enable, and specify the KMS key you want to use for encryption.

Create a subscription between your SNS topic and Panther's log processing SQS queue.

Navigate to the SNS Console and select the SNS topic currently receiving events.
- Note the ARN of this SNS topic.
Click Edit and scroll down to the Access Policy card.
Add the following statement to the topic's Access Policy:
```
{
  "Sid": "CrossAccountSubscription",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::<PANTHER-MASTER-ACCOUNT-ID>:root"
  },
  "Action": "sns:Subscribe",
  "Resource": "<SNS-TOPIC-ARN>"
}
```
- Populate <PANTHER-MASTER-ACCOUNT-ID> with the 12-digit account ID where Panther is deployed. This AWS account ID can be found in your Panther Console at the bottom of the page after navigating to Settings by clicking the gear icon.
- Populate SNS-TOPIC-ARN with the ARN you noted previously in this documentation.

Create the subscription to the Panther Master account's SQS queue.

From the SNS Console, click Subscriptions.
Click Create subscription.
Fill out the form:
- Topic ARN: Select the SNS topic you would like to use.
- Protocol: Select Amazon SQS.
- Endpoint: arn:aws:sqs:<PantherRegion>:<MasterAccountId>:panther-input-data-notifications-queue
- Enable raw message delivery: Do not check this box. Raw message delivery must be disabled.
Click Create subscription.

If your subscription is in a "Pending" state and does not get confirmed immediately, you must finish setting up this log source in your Panther Console. Panther confirms the SNS subscription only if a Panther log source exists for the AWS account of the SNS topic.

Step 2: Configure event notifications on the S3 bucket

With the SNS topic created, the final step is to enable notifications from the S3 buckets.

Navigate to the AWS S3 Console, select the relevant bucket, and click the Properties tab.
Locate the Event notifications card.
Click Create event notification and use the following settings:
- In the General Configuration section:
  - Event name: PantherEventNotifications
  - Prefix (optional): Limits notifications to objects with keys that start with matching characters
  - Suffix (optional): Limits notifications to objects with keys that end in matching characters
    Avoid creating multiple filters that use overlapping prefixes and suffixes. Otherwise, your configuration will not be considered valid.
- In the Event Types card, check the box next to All object create events.
- In the Destination card:
  - Under Destination, select SNS topic.
  - For SNS topic, select the SNS topic you created or modified in an earlier step.
    If you used the default topic name in the CloudFormation template provided, the SNS topic is named panther-notifications-topic.
    If you are using a custom SNS topic, ensure it has the correct policies set and a subscription to the Panther SQS queue.
Click Save.
- Return to Finish the Source Setup in Panther.

PreviousAWS Sources NextCloudWatch Logs Source

Last updated 7 months ago

Was this helpful?

Overview

How set up an AWS S3 bucket log source in Panther

Prerequisite

Step 1: Configure the S3 bucket source in Panther

Step 2: Set up an IAM role

CloudFormation or Terraform File

Using the AWS Console UI

CloudFormation or Terraform File

I want to set everything up on my own

Step 3: Finish the source setup

Viewing ingested logs

Recommended S3 bucket expiration policy

Manual IAM role creation: Additional steps

Step 1: Create or modify an SNS topic

How to create an SNS topic

How to modify an existing SNS topic

Step 1: Enable KMS encryption for the SNS topic

Step 2: Modify SNS topic Access Policy

Step 3: Create SNS subscription to SQS

Step 2: Configure event notifications on the S3 bucket