> For the complete documentation index, see [llms.txt](https://docs.panther.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.panther.com/~/changes/Dd8nx2iqd1Pp2OzWJaWk/data-onboarding/data-transports/aws/s3.md).

# S3 Source

Onboarding AWS S3 as a Data Transport log source in the Panther Console

## Overview

Follow the steps below to enable secure access for Panther to pull security logs from Amazon S3 bucket(s). First you will configure the S3 source in your Panther Console, then you will configure your S3 bucket to send notifications when it receives new data.

Data can be sent compressed (or uncompressed). Learn more about compression specifications in [Ingesting compressed data in Panther](/~/changes/Dd8nx2iqd1Pp2OzWJaWk/data-onboarding/data-transports.md#ingesting-compressed-data-in-panther).

See the diagram below to understand how data flows from your application(s) into Panther using S3 (in [SaaS](/~/changes/Dd8nx2iqd1Pp2OzWJaWk/system-configuration/panther-deployment-types.md#saas)):

<figure><img src="https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2FodgzjzhWMsP8Y7oJPnXR%2FData_Transport_S3.png?alt=media&amp;token=f42a7622-1f1a-470c-9b25-c74f447c1e81" alt="A diagram shows how data flows from a customer application into Panther, using the S3 bucket Data Transport. The flow is as follows: Application(s), S3 bucket, SNS topic, SQS, Panther application, IAM role (assumed by Panther), KMS key (optional), S3 bucket, Panther application, Parse &#x26; normalize, real-time detections, Alerts generated, Long-term retention in Snowflake, and Alert Destinations"><figcaption></figcaption></figure>

## How to pull logs from AWS S3 buckets into Panther

{% hint style="info" %}
The instructions below outline how to set up an S3 integration manually, in the Panther Console. It's also possible to manage your S3 log source [using the Panther API](/~/changes/Dd8nx2iqd1Pp2OzWJaWk/panther-developer-workflows/api/operations/log-source.md#creating-an-s3-log-source), or [using Terraform](/~/changes/Dd8nx2iqd1Pp2OzWJaWk/panther-developer-workflows/terraform.md).
{% endhint %}

To set up an S3 log source in Panther, follow the steps below. You can also view the [data ingestion video overview](/~/changes/Dd8nx2iqd1Pp2OzWJaWk/data-onboarding.md#video-overview) for a quick walkthrough of S3 source setup.

### Prerequisite

* If an Amazon S3 bucket does not already exist, create one by following [Amazon's Creating a bucket documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html).

### Step 1: Configure the S3 bucket source in Panther

1. In the left-hand navigation bar of your Panther Console, click **Configure** > **Log Sources**.
2. In the upper right corner, click **Create New**.
3. Click the **Custom Log Formats** tile.
4. In the **AWS S3 Bucket** tile on the slide-out panel, click **Start**.\
   ![In the Panther Console, the Custom Log Formats tile has been selected, and the slide-out panel shows a number of Data Transport options. An arrow is drawn to the Start button on the AWS S3 Bucket option.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2F0UdRTKW89LrWEdbwoKFm%2FScreenshot%202023-06-21%20at%209.43.47%20AM.png?alt=media\&token=c1f277c5-b280-4add-b5d1-44ee92d03160)
5. On the **Basic Info** page, enter values for the following fields:
   * **Name**: Enter a descriptive name for the S3 source.
   * **AWS Account ID**: Enter the 12-digit AWS Account ID where the S3 buckets are located.
   * **Bucket Name**: Enter the ID or name of the S3 bucket to onboard.
   * **KMS Key ARN (optional)**: If your data is encrypted using KMS-SSE, provide the ARN of the KMS key.
6. Click **Setup**.

{% hint style="warning" %}
If you add a KMS key to your S3 bucket after creating the S3 log source in Panther, you must recreate the log source in Panther with the KMS key. Editing the original source to add the KMS key will not work.
{% endhint %}

### Step 2: Set up an IAM role

Panther needs an AWS IAM role with permissions to read objects from your S3 bucket. You can choose from the following options:

<figure><img src="https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2FDfBtkYOpGHhrlMCaCNpl%2Fs3source.png?alt=media&amp;token=210f04cd-68e3-42c5-aa13-ac1fad742e52" alt="On the IAM Role Setup page, there are three options: Using the AWS Console UI, CloudFormation or Terraform File, or I want to set everything up on my own"><figcaption></figcaption></figure>

* [**Using the AWS Console UI**](#using-the-aws-console-ui-1)
* #### [**CloudFormation or Terraform File**](#cloudformation-or-terraform-file-2)
* [**I want to set up everything on my own**](#i-want-to-set-everything-up-on-my-own-1)

{% tabs %}
{% tab title="Using the AWS Console UI" %}

#### Using the AWS Console UI

Launch a CloudFormation stack using the AWS console:

1. On the **Create IAM Role** page, locate the tile labeled **Using the AWS Console UI**. At the bottom of the tile, click **Continue**.
2. Click **Launch Console UI**.\
   ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2FOJQ7Wzs2STnAlZaiHtUg%2Fimage.png?alt=media\&token=59843b65-a191-466f-8f27-c89bf6f952b4)
   * You will be redirected to the AWS console in a new browser tab, with the template URL pre-filled.&#x20;
   * The CloudFormation stack will create an AWS IAM role with the minimum required permissions to read objects from your S3 bucket.
   * Click the "Outputs" tab of the CloudFormation stack in AWS, and note the Role ARN.
3. Navigate back to the Panther Console.
4. Enter the **Role ARN**.
5. Click **Setup**.
6. Proceed to the [Finish the Source Setup ](#step-3-finish-the-source-setup)section in this documentation.
   {% endtab %}

{% tab title="CloudFormation or Terraform File" %}

#### **CloudFormation or Terraform File**

Use Panther's provided CloudFormation or Terraform templates to create an IAM role by choosing Select to the right of this option.

1. On the **Create IAM Role** page, locate the tile labeled **CloudFormation or Terraform File**. At the bottom of the tile, click **Continue**.
2. On the "CloudFormation or Terraform Template File" page, choose which type of template you would like to download:
   * If using CloudFormation:
     1. Click **CloudFormation Template.**
     2. Click **Download Template** to download the template to apply it through your own pipeline.
     3. Upload the template file in AWS:
        1. Open your AWS console and navigate to the CloudFormation service.
        2. Click **Create stack**.
        3. Click **Upload a template file** and select the CloudFormation template you downloaded.
   * If using Terraform:
     1. Click **Terraform Template.**
     2. Click **Download Template** to download the template to apply it through your own pipeline.
3. Enter the **Role ARN**.
4. Click **Setup**.
5. Proceed to the [Finish the Source Setup](#step-3-finish-the-source-setup) section in this documentation.
   {% endtab %}

{% tab title="I want to set everything up on my own" %}

#### I want to set everything up on my own

Create the IAM role manually, then fill in the role ARN in Panther. When you set up the IAM role manually, you must also follow the instructions below to configure your S3 buckets to send notifications when new data arrives.

1. On the **Create IAM Role** page, click the link that says **I want to set everything up on my own**.
2. Create the required IAM role. You may create the required IAM role manually or through your own automation.
   * The IAM policy, which will be attached to the role, must include the statements defined below:

     ```json
     {
         "Version": "2012-10-17",    
         "Statement": [
             {
                 "Action": "s3:GetBucketLocation",
                 "Resource": "arn:aws:s3:::<bucket-name>",
                 "Effect": "Allow"
             },
             {
                 "Action": "s3:GetObject",
                 "Resource": "arn:aws:s3:::<bucket-name>/<input-file-path>",
                 "Effect": "Allow"
             }
         ]
     }
     ```
   * If your S3 bucket is configured with server-side encryption using AWS KMS, you must include an additional statement granting the Panther API access to the corresponding KMS key. In this case, the policy will look something like this:

     ```json
     {
         "Version": "2012-10-17",    
         "Statement": [
             {
                 "Action": "s3:GetBucketLocation",
                 "Resource": "arn:aws:s3:::<bucket-name>",
                 "Effect": "Allow"
             },
             {
                 "Action": "s3:GetObject",
                 "Resource": "arn:aws:s3:::<bucket-name>/<input-file-path>",
                 "Effect": "Allow"
             },
             {
                 "Action": ["kms:Decrypt", "kms:DescribeKey"],
                 "Resource": "arn:aws:kms:<region>:<your-account-id>:key/<kms-key-id>",
                 "Effect": "Allow"
             }
         ]
     }
     ```
   * In addition to the above, if you want to view the contents of your S3 bucket in the Panther Console (such as to utilize the [inferring custom schemas from historical data](/~/changes/Dd8nx2iqd1Pp2OzWJaWk/data-onboarding/custom-log-types.md#inferring-custom-schemas-from-historical-s3-data) feature), you will need to add the `s3:ListBucket` action: <br>

     ```json
     {   
          "Version": "2012-10-17",    
         "Statement": [
             {
                 "Action": [
                     "s3:GetBucketLocation",
                     "s3:ListBucket"
                 ],
                 "Resource": "arn:aws:s3:::<bucket-name>",
                 "Effect": "Allow"
             },
             {
                 "Action": "s3:GetObject",
                 "Resource": "arn:aws:s3:::<bucket-name>/<input-file-path>",
                 "Effect": "Allow"
             }
         ]
     }
     ```
3. Add a trust policy to your role with the following `AssumeRolePolicyDocument` statement so that Panther can assume this role:

   ```json
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "AWS": [
             "arn:<aws-partition>:iam::<panther-master-account-id>:root"
           ]
         },
         "Action": "sts:AssumeRole",
         "Condition": {
           "Bool": {
             "aws:SecureTransport": true
           }
         }
       }
     ]
   }
   ```

   * Populate `<AWS-PARTITION>` with the partition of the account running the Panther backend (e.g., `aws`). Note that we do not deploy to `aws-cn` or `aws-us-gov`.
   * Populate `<PANTHER-MASTER-ACCOUNT-ID>` with the 12-digit account ID where Panther is deployed. To get your AWS Account ID: Click the gear icon in the upper right side of the Panther Console to access Settings, then the AWS account ID is displayed at the bottom of the page.
4. In the Panther Console, enter the **Role ARN**.&#x20;
5. Click **Setup.**
6. Proceed to [Manual IAM role creation: Additional Steps](#manual-iam-role-creation-additional-steps).

{% hint style="info" %}
If you choose the option **I want to set everything up on my own** to create an IAM role, you must also [configure the S3 buckets to send notifications](#configure-event-notifications-on-the-bucket) when new data arrives.
{% endhint %}
{% endtab %}
{% endtabs %}

### Step 3: Finish the source setup

After you configure the IAM role in your AWS console, you will see a success screen:

<figure><img src="https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2FG4KJedYPQsxo2nmNiYLe%2FScreenshot%202023-06-22%20at%2010.13.53%20AM.png?alt=media&amp;token=8b4b73bc-e6ce-48a4-a7ea-c2cd455b263c" alt="On the Setup Verification page of the Create AWS S3 source flow, the text reads, &#x22;Everything looks good!&#x22; Below are two buttons, &#x22;Attach or Infer Schemas&#x22; and &#x22;View Log Source&#x22;" width="563"><figcaption></figcaption></figure>

* If any permission errors are detected, they will be displayed and you will be asked to try configuring the IAM role again.

To finish the source setup:

1. Click **Attach or Infer Schemas**.
2. Either attach a Panther-managed schema, or [follow these instructions to infer a custom schema from historical S3 data](/~/changes/Dd8nx2iqd1Pp2OzWJaWk/data-onboarding/custom-log-types.md#inferring-custom-schemas-from-historical-s3-data).

## Viewing Collected Logs

After log sources are configured, your data can be searched in Data Explorer. Learn more in the [Data Explorer documentation](https://docs.runpanther.io/data-analytics/data-explorer).

## Data Backup

Once the S3 bucket has been successfully onboarded to Panther and data is flowing, Panther will backup all raw logs for up to 90 days. After that, the logs will be deleted.&#x20;

The raw logs are used for various reasons, for example: To backup dropped logs that may not have been successfully normalized and classified in Panther's data processing pipeline.

## Manual IAM role creation: Additional steps

When you [set up the IAM role manually](#i-want-to-set-everything-up-on-my-own), you must also follow the instructions below to configure your S3 buckets to send notifications when new data arrives.

### Create SNS Topic

Note: If you already have configured the bucket to send `All object create events` to an SNS topic, proceed to the next section, "Modify an existing SNS topic," and subscribe it to Panther's input data queue.

Only one SNS topic is required per AWS account, so multiple buckets within one AWS account all use the same SNS topic for Panther. If you've already created an SNS topic for a different S3 bucket in the same AWS account, you can skip this step of creating an SNS topic.

First you need to create an SNS Topic and SNS Subscription to notify Panther that new data is ready for processing.

1. Log into the AWS Console of the account that owns the S3 bucket.&#x20;
2. Select the AWS Region where your S3 bucket is located and navigate to the **CloudFormation** console.&#x20;
3. Navigate to the **Stacks** section. Select **Create Stack** (with new resources).\
   ![In the AWS CloudFormation console, there is a "Create Stack" dropdown menu in the upper right. In this image, the menu is expanded and the option "with new resources (standard)" is highlighted.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2FcL15olPssVbmODl1NSFw%2FScreen%20Shot%202022-09-02%20at%2011.02.26%20AM.png?alt=media\&token=f4806d74-a3e0-4389-8ef1-6cf2154f3fd9)
4. Under the "Specify template" section, enter the following Amazon S3 URL:

   ```
   https://panther-public-cloudformation-templates.s3-us-west-2.amazonaws.com/panther-log-processing-notifications/latest/template.yml
   ```

   \
   ![On the Create Stack page in the AWS CloudFormation console, there is a section at the top labeled "Prerequisite - Prepare template." The option "Template is ready" is selected. In the "Specify Template" section, the template source "Amazon S3 URL" is selected and a URL is pasted into the template URL field.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2FU2X57J3GoMqf5gWJVam0%2Fcreate-sns-topic-2.png?alt=media\&token=3ec2242b-8dd2-49ee-8946-e454e4694fb9)<br>
5. Specify the following stack details:
   * **Stack name**: A name of your choice, e.g. `panther-log-processing-notifications-<bucket-label>`
   * **MasterAccountId**: The 12 digit AWS Account ID where Panther is deployed
   * **PantherRegion**: The region where Panther is deployed
   * **SnsTopicName**`:` The name of the SNS topic receiving the notification, by default this is `panther-notifications-topic`
6. Click **Next**, **Next**, and then **Create Stack** to complete the process.&#x20;

Note: This stack has one output named `SnsTopicArn`.

### Modify an existing SNS topic

If you opted to create a new SNS topic in the previous step, skip this step and proceed to the section below, "Configure Event Notifications on the bucket."&#x20;

Follow the steps below if you wish to use an existing topic for sending bucket notifications. Note that the SNS topic must be in the same region as your S3 bucket.

#### Set up encryption&#x20;

We recommend enabling KMS encryption for the SNS topic:

1. Log in to the AWS console and navigate to KMS.
2. Select the KMS key you want to use for encryption.&#x20;
3. Edit the policy to ensure it has the [appropriate permissions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/grant-destinations-permissions-to-s3.html#key-policy-sns-sqs) to be used with the SNS topic and S3 bucket notifications.&#x20;
   * Example policy:

     <pre class="language-json"><code class="lang-json"><strong>{
     </strong>    "Sid": "Allow access for Key User (SNS Service Principal)",
         "Effect": "Allow",
         "Principal": {
             "Service": "sns.amazonaws.com"
         },
         "Action": [
             "kms:GenerateDataKey*",
             "kms:Decrypt"
         ],
         "Resource": "&#x3C;SNS-TOPIC-ARN>"
     },
     {
         "Sid": "Allow access for Key User (S3 Service Principal)",
         "Effect": "Allow",
         "Principal": {
             "Service": "s3.amazonaws.com"
         },
         "Action": [
             "kms:GenerateDataKey*",
             "kms:Decrypt"
         ],
         "Resource": "arn:aws:s3:::&#x3C;bucket-name>"
     }
     </code></pre>
4. Click the **Encryption** tab under the SNS topic.
5. Click **Enable**, and specify the KMS key you want to use for encryption.&#x20;

#### Modify SNS Access Policy

Create a subscription between your SNS topic and Panther's log processing SQS queue.

1. Log into the AWS Console for the account where your S3 bucket exists
2. Navigate to the [SNS Console](https://us-west-2.console.aws.amazon.com/sns/v3/home#/topics) and select the SNS Topic currently receiving events
3. Note the ARN of this SNS topic
4. Click **Edit** and scroll down to the "Access Policy" card
5. Add the following statement to the topic's `Access Policy`:

   ```
   {
     "Sid": "CrossAccountSubscription",
     "Effect": "Allow",
     "Principal": {
       "AWS": "arn:aws:iam::<PANTHER-MASTER-ACCOUNT-ID>:root"
     },
     "Action": "sns:Subscribe",
     "Resource": "<SNS-TOPIC-ARN>"
   }
   ```

   * Populate `<PANTHER-MASTER-ACCOUNT-ID>` with the 12-digit account ID where Panther is deployed. This AWS account ID can be found in your Panther Console at the bottom of the page after navigating to **Settings** by clicking the **gear** ico&#x6E;**.**&#x20;
   * Populate `SNS-TOPIC-ARN` with the ARN you noted previously in this documentation.

### Create SNS Subscription

Create the subscription to the Panther Master account's SQS queue.

From the SNS Console, click **Create subscription**:

1. **Fill out the form:**
   * **Protocol**: `Amazon SQS`
   * **Endpoint**: `arn:aws:sqs:<PantherRegion>:<MasterAccountId>:panther-input-data-notifications-queue`
2. **Do not** check the box for the `Enable raw message delivery` setting; `raw message delivery` must be disabled.
3. Click **Create subscription**.

Note: If your subscription is in a "Pending" state and does not get confirmed immediately, you must finish setting up this log source in your Panther Console. Panther confirms the SNS subscription only if a Panther log source exists for the AWS account of the SNS topic.

### Configure Event Notifications on the bucket

With the SNS Topic created, the final step is to enable notifications from the S3 buckets.

1. Navigate to the AWS [S3 Console](https://s3.console.aws.amazon.com/s3/home), select the relevant bucket, and click the **Properties** tab.
2. Locate the Event Notifications card.&#x20;
3. Click **+ Create event notification** and use the following settings:
   * In the "General Configuration" section:&#x20;
     * **Name**: `PantherEventNotifications`
     * **Suffix**: (optional) limits notifications to objects with keys that end in matching characters
     * **Prefix**: (optional) limits notifications to objects with keys that start with matching characters
   * In the "Event Types" card, check the box next to `All object create events`.
   * In the "Destination" card:

     * Under "Destination," select SNS Topic.
     * For `SNS Topic,` select `panther-notifications-topic` from the drop-down menu.

     ![In the Destination card in the AWS S3 Console, the destination "SNS topic" is selected. Under "Specify SNS topic," the option "Choose from your SNS topics" is selected. The "SNS topic" drop-down menu is set to "panther-notifications-topic." ](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2F2tDJBkeTmU177JTaJFBb%2Fs3-source-setup.png?alt=media\&token=18ae2334-ae08-431f-b058-c68e81b456c3)
4. Click **Save**.
5. Make sure to [Finish the Source Setup in Panther](#step-3-finish-the-source-setup).

Panther will now start processing new files arriving to your bucket.&#x20;


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.panther.com/~/changes/Dd8nx2iqd1Pp2OzWJaWk/data-onboarding/data-transports/aws/s3.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
