S3 Source
Onboarding AWS S3 as a Data Transport log source in the Panther Console
Last updated
Was this helpful?
Onboarding AWS S3 as a Data Transport log source in the Panther Console
Last updated
Was this helpful?
Panther supports configuring your Amazon S3 bucket as a Data Transport to pull security logs from S3 buckets.
First you will configure the S3 source in your Panther Console, then you will configure your S3 bucket to send notifications when it receives new data.
Data can be sent compressed (or uncompressed). Learn more about compression specifications in .
See the diagram below to understand how data flows from your application(s) into Panther using S3 (in ):
In the left-hand navigation bar of your Panther Console, click Configure > Log Sources.
In the upper right corner, click Create New.
Click the Custom Log Formats tile.
On the "Configure your source" page, enter values for the following fields:
Name: Enter a descriptive name for the S3 source.
AWS Account ID: Enter the 12-digit AWS Account ID where the S3 buckets are located.
Bucket Name: Enter the ID or name of the S3 bucket to onboard.
KMS Key ARN (optional): If your data is encrypted using KMS-SSE, provide the ARN of the KMS key.
In the S3 Prefixes & Schemas popup modal, create combinations of S3 prefixes, schemas, and exclusion filters, according the structure of your data storage in S3.
Click Apply Changes.
Click Setup.
If you add a KMS key to your S3 bucket after creating the S3 log source in Panther, you must recreate the log source in Panther with the KMS key. Editing the original source to add the KMS key will not work.
You will be directed to a success screen:
If any permission errors are detected, they will be displayed and you will be asked to try configuring the IAM role again.
The Trigger an alert when no events are processed setting defaults to YES. We recommend leaving this enabled, as you will be alerted if data stops flowing from the log source after a certain period of time. The timeframe is configurable, with a default of 24 hours.
If you have not done so already, attach one or more schemas to the source.
Click Attach or Infer Schemas.
It is recommended to keep the data added to your S3 bucket for at least seven days before expiring it. Under normal circumstances, Panther processes new objects within minutes of them being added to your S3 bucket, however if the Panther ingestion service is experiencing availability issues, it could take longer for new objects to be processed.
The instructions below outline how to set up an S3 integration manually, in the Panther Console. It's also possible to manage your S3 log source , or .
To set up an S3 log source in Panther, follow the steps below. You can also view the for a quick walkthrough of S3 source setup.
If an Amazon S3 bucket does not already exist, create one by following .
In the AWS S3 Bucket tile on the slide-out panel, click Start.
If you would like to attach schemas for this source and/or configure inclusive or exclusive bucket prefixes, click Configure Prefixes & Schemas (Optional). You can also perform these actions after the source is set up.
To attach one or more schemas to all data in the bucket, leave the S3 Prefix field blank. This will create a wildcard (*) prefix.
You can optionally enable one or more .
Either attach a Panther-managed schema, or .
After your log source is configured, you can search ingested data using or .
To read objects from your source, Panther needs an AWS IAM role with certain permissions. To set up this role, you can choose from the following options:
Using the AWS Console UI
If this is the first Data Transport source you are setting up in Panther, select this option.
CloudFormation or Terraform File
I want to set up everything on my own
Launch a CloudFormation stack using the AWS console:
On the Create IAM Role page, on the Using the AWS Console UI tile, click Continue.
You will be redirected to the AWS console in a new browser tab, with the template URL pre-filled.
The CloudFormation stack will create an AWS IAM role with the minimum required permissions to read objects from your source.
Click the "Outputs" tab of the CloudFormation stack in AWS, and note the Role ARN.
Navigate back to the Panther Console, and enter values in the fields:
(Not applicable if setting up an S3 Source) Bucket name – Required: Enter the outputted S3 bucket name.
Role ARN – Required: Enter the outputted IAM role ARN.
Click Setup.
If during log source creation you opted to set up the IAM role manually, you must also follow the instructions below to configure your S3 bucket to send notifications when new data arrives.
Note: If you already have configured the bucket to send All object create events
to an SNS topic, instead follow the "Modify an existing SNS topic" tab, and subscribe it to Panther's input data queue.
First you need to create an SNS Topic and SNS Subscription to notify Panther that new data is ready for processing.
Log into the AWS Console of the account that owns the S3 bucket.
Select the AWS Region where your S3 bucket is located and navigate to the CloudFormation console.
Under the "Specify template" section, enter the following Amazon S3 URL:
Specify the following stack details:
Stack name: A name of your choice, e.g. panther-log-processing-notifications-<bucket-label>
MasterAccountId: The 12 digit AWS Account ID where Panther is deployed
PantherRegion: The region where Panther is deployed
SnsTopicName: The name of the SNS topic receiving the notification. The default value is panther-notifications-topic
Click Next, Next, and then Create Stack to complete the process.
This stack has one output: SnsTopicArn
.
With the SNS topic created, the final step is to enable notifications from the S3 buckets.
Locate the Event notifications card.
Click Create event notification and use the following settings:
In the General Configuration section:
Event name: PantherEventNotifications
Prefix (optional): Limits notifications to objects with keys that start with matching characters
Suffix (optional): Limits notifications to objects with keys that end in matching characters
In the Event Types card, check the box next to All object create events.
In the Destination card:
Under Destination, select SNS topic.
For SNS topic, select the SNS topic you created or modified in an earlier step.
If you used the default topic name in the CloudFormation template provided, the SNS topic is named panther-notifications-topic
.
4. Click Save.
Return to "Step 3: Finish the source setup," above.
Click Launch Console UI.
You can also find the Terraform template at .
In addition to the above, if you want to view the contents of your S3 bucket in the Panther Console (such as to utilize the feature), you will need to add the s3:ListBucket
action: \
Navigate to the Stacks section. Select Create Stack (with new resources).
Edit the policy to ensure it has the to be used with the SNS topic and S3 bucket notifications.
Navigate to the and select the SNS topic currently receiving events.
Navigate to the AWS , select the relevant bucket, and click the Properties tab.
Avoid . Otherwise, your configuration will not be considered valid.
If you are using a custom SNS topic, ensure it has the correct policies set and a subscription to the Panther SQS queue.