# Configuring Databricks for Panther

## Overview

This page describes how to configure [Databricks](https://www.databricks.com/) for use as your Panther data storage backend. As you complete the steps below, you will collect and store various configuration values, then provide them to Panther.

{% hint style="warning" %}
You should complete the process on this page only after arriving at [Step 9](https://docs.panther.com/system-configuration/panther-deployment-types/set-up#step-9-if-using-databricks-configure-databricks-for-panther) on [set-up](https://docs.panther.com/system-configuration/panther-deployment-types/cloud-connected/set-up "mention").
{% endhint %}

This process will:

* Create a Databricks workspace for Panther (along with associated Databricks infrastructure in AWS)
* Create an IAM role in AWS to allow Databricks to read from the Panther S3 staging bucket.
* Create an external storage credential.
* Create an external storage integration so Databricks can read data from S3 for loading.
* Create service principals—one for loading (read/write) and one for querying (read-only).
* Create secrets with KMS keys in AWS to hold OAuth credentials for the service principals.
* Create a catalog in Databricks for Panther tables, with permissions for the service principals.
* Create load, optimize, query, and scheduled query warehouses.

## How to configure Databricks for Panther

### Prerequisites

* You have a Databricks account.
* You have completed the instructions on [set-up](https://docs.panther.com/system-configuration/panther-deployment-types/cloud-connected/set-up "mention") and can log in to the Panther Console.
* You are logged into the AWS console in the AWS account you'd like to use for Panther compute. This is needed because Databricks will create a workspace on your behalf.

{% hint style="warning" %}
This AWS account should not be the AWS account where Panther is hosted.
{% endhint %}

* You have the Databricks and AWS permissions listed in the following pages:
  * [Create a classic workspace](https://docs.databricks.com/aws/en/admin/workspace/create-workspace#requirements)
  * [Create a storage credential and external location for S3 using Catalog Explorer or SQL](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/s3/s3-external-location-manual)
  * [Create a SQL warehouse](https://docs.databricks.com/aws/en/compute/sql-warehouse/create#requirements)

### Step 1: Make a copy of the configuration table

Throughout the configuration process, you'll collect values that you'll send to Panther at the end. To organize theses values, make a copy of the table below.

| Parameter                                 | Value |
| ----------------------------------------- | ----- |
| `databricks_load_role_arn`                |       |
| `databricks_load_secret_kms_key_arn`      |       |
| `databricks_query_secret_kms_key_arn`     |       |
| `databricks_load_secret_arn`              |       |
| `databricks_query_secret_arn`             |       |
| `databricks_catalog`                      |       |
| `databricks_load_warehouse_id`            |       |
| `databricks_optimize_warehouse_id`        |       |
| `databricks_query_warehouse_id`           |       |
| `databricks_scheduled_query_warehouse_id` |       |

### Step 2 (Optional): Create a Databricks workspace

It's recommended to create a dedicated Databricks workspace for Panther, for organizational purposes, but not required. If you'd like to use an existing workspace, skip this step.

{% hint style="info" %}
For additional support while creating a workspace, see the Databricks [Create a workspace with automated configuration](https://docs.databricks.com/aws/en/admin/workspace/create-workspace#create-a-workspace-with-automated-configuration) documentation.
{% endhint %}

1. Log in into the Databricks console.
2. In the left-hand navigation menu, click **Workspaces**.
3. Click **Create workspace**.
4. Fill out the **Create Workspace** modal:
   * **Workspace name**: enter a memorable name.
   * **Region**: select the region that matches your AWS deployment of Panther.
   * **Storage and compute**: select **Use your existing cloud account**.\
     ![An arrow is drawn from a "Workspaces" navigation item to a "Create workspace" button. In the foreground, there is a "Create Workspace" modal.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2FFP3zY1Ol3VYloFfQSP1Y%2FScreenshot%202025-12-11%20at%2012.36.44%E2%80%AFPM.png?alt=media\&token=46e43128-d9bf-4b88-b05c-395223950470)
5. Click **Continue**.
6. Under **Cloud resources**, in the **Cloud credentials** field, select **Add cloud credentials**.\
   ![Under a "Create workspace" header, there are various form fields, like "Workspace name," "Region," and "Cloud credentials."](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fy8IfHW0Sds2v751WaKcT%2FScreenshot%202025-12-11%20at%2012.38.58%E2%80%AFPM.png?alt=media\&token=f58fcbed-1ddb-412b-bdf2-5417655347a0)
7. On the **Add cloud credentials** modal, leave **Add automatically** selected, then click **OK**.
   * Both the **Cloud credentials** and **Cloud storage** fields will be provided an **Add automatically** value. Leave these as-is.
8. Click **Log in to AWS and create workspace**.
9. On the **Review AWS Resources** modal, click **Initiate workspace creation**.
10. On the AWS modal, click **Allow access**.
11. Return to your Databricks browser tab, and wait a few minutes for the new workspace to appear in the **Workspaces** list. When it appears, click **Open** to enter the workspace environment.

### Step 3: Enable variant shredding in your workspace

{% hint style="info" %}
For additional support while enabling variant shredding, see the Databricks [Enable shredding](https://docs.databricks.com/aws/en/delta/variant-shredding#enable-shredding) documentation.
{% endhint %}

1. In your Databricks workspace, in the upper-right corner, click your profile icon, then **Previews**.
2. To the right of **Variant Shredding for Optimized Read Performance on Semi-Structured Data**, click the toggle **On**.\
   ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-6addcda10edcb8e1981426efdb240fdacb23a337%2FScreenshot%202025-10-30%20at%2011.29.48%E2%80%AFAM.png?alt=media)

### Step 4: Create a Panther role for the storage credential

{% hint style="info" %}
For additional support while creating an IAM role, see the Databricks [Step 1: Create an IAM role](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/s3/s3-external-location-manual#step-1-create-an-iam-role) documentation.
{% endhint %}

1. In the AWS account where you created the Databricks workspace infrastructure, create an IAM role named `panther-databricks-s3-reader-role-<region>`, accepting all defaults.
2. In your Panther Console, retrieve the **Processed Data Bucket** value:
   1. Click the gear icon (Settings) > **General**.
   2. Click **Data Lake**.
   3. Under **Databricks Configuration**, copy the **Processed Data Bucket** value.\
      ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-5be9237e052ed673e513d5b50f47b9f854ce561a%2FScreenshot%202025-10-30%20at%2011.11.47%E2%80%AFAM.png?alt=media)
3. Update the role's trust relationship:
   1. In the AWS console, in the **Roles** list, click the newly created role to view its details page.
   2. Click **Trust relationships**.
   3. Click **Edit trust policy**.
   4. Replace the JSON in the code editor with the JSON below:<br>

      <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>The following trust policy sets <code>"sts:ExternalId": "TBD"</code> as a placeholder—you will update this later. You will also later add a self-assumption statement.</p></div>

      ```json
      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": {
                      "AWS": [
                          "arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-14S5ZJVKOTYTL",
                          "arn:aws:iam::<your account for this role>:role/panther-databricks-s3-reader-role-<region>"
                      ]
                  },
                  "Action": "sts:AssumeRole",
                  "Condition": {
                      "StringEquals": {
                          "sts:ExternalId": "TBD"
                      }
                  }
              }
          ]
      }
      ```
   5. Click **Update policy.**
4. Update the role's permissions:
   1. On the role's details page, click **Permissions**.
   2. Click **Add permissions** > **Create inline policy**.
   3. In the **Policy editor** section, click **JSON**.
   4. Replace the JSON in the code editor with the JSON below:<br>

      <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>In the policy below, replace <code>&#x3C;Processed Data Bucket from Panther settings></code> with the <strong>Processed Data Bucket</strong> value you retrieved above.</p></div>

      ```json
      { 	
      "Statement": [ 		
        { 			
         "Action": [ 				
           "s3:ListBucket", 				
            "s3:GetBucketLocation" 			
         ], 			
         "Effect": "Allow", 			
         "Resource": "arn:aws:s3:::<Processed Data Bucket from Panther settings>" 		
       }, 		
       { 			
         "Action": "s3:GetObject", 			
          "Effect": "Allow", 			
          "Resource": "arn:aws:s3:::<Processed Data Bucket from Panther settings>/*" 		
       } 	
       ], 	
       "Version": "2012-10-17" 
      }
      ```
   5. Click **Next**.
   6. Under **Policy details**, enter a **Policy name**.
   7. Click **Create policy**.
5. On the role's details page, copy the **ARN**, and add it as the `databricks_load_role_arn` value in your [configuration table](#step-1-make-a-copy-of-the-configuration-table).
   * Leave the browser window with the role details page open, as you will return to it in Step 5.

### Step 5: Create a storage credential

{% hint style="info" %}
For additional support while creating a storage credential, see the Databricks [Step 2: Give Databricks the IAM role details](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/s3/s3-external-location-manual#step-2-give-databricks-the-iam-role-details) documentation.
{% endhint %}

Create a Databricks [storage credential](https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-storage-credentials#credential) to represent the AWS IAM role you just created:

1. In your Databricks workspace, click **Catalog**, then **External Data**.\
   ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-7c66d9e3150d25358ef4f8a87e5a9ea46a56c582%2FScreenshot%202025-10-30%20at%2012.31.13%E2%80%AFPM.png?alt=media)
2. Click **Credentials**.
3. Click **Create credential**.
4. Fill in the **Create a new credential** form:
   1. **Credential Type**: select **AWS IAM Role**.
   2. **Credential name**: enter `panther-storage-credential`.
   3. **IAM role (ARN)**: enter the ARN of the IAM role you created above (which is `databricks_load_role_arn` in the [configuration table](#step-1-make-a-copy-of-the-configuration-table)).
5. Click **Create**.
   * On the **Credential created** page, copy the **External ID** value, and store it in a secure location, as you will need it in the next step.

### Step 6: Update the IAM role trust relationship policy

{% hint style="info" %}
For additional support while updating the IAM role, see the Databricks [Step 3: Update the IAM role trust relationship policy](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/s3/s3-external-location-manual#step-3-update-the-iam-role-trust-relationship-policy) documentation.
{% endhint %}

1. Return to the AWS console, to the details page for the `panther-databricks-s3-reader-role-<region>` IAM role you created above.
2. Click **Trust relationships**.
3. Click **Edit trust policy**.
4. In the `"sts:ExternalId": "TBD"` line, replace `TBD` with the **External ID** value you copied in Databricks above.
5. Click **Update policy**.

### Step 7: Create an external storage location

{% hint style="info" %}
For additional support while updating the IAM role, see the Databricks [Create an external location for an AWS S3 bucket](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/s3/s3-external-location-manual#-create-an-external-location-for-an-aws-s3-bucket) documentation.
{% endhint %}

1. In your Databricks workspace, click **Catalog**, then **External Data**.\
   ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-7c66d9e3150d25358ef4f8a87e5a9ea46a56c582%2FScreenshot%202025-10-30%20at%2012.31.13%E2%80%AFPM.png?alt=media)
2. Click **Create external location**.
3. Click **Manual**, then **Next**.
4. Fill in the **Create a new external location manually** form:
   * **External location name**: enter `panther-processed-data`.
   * **Storage type**: select **S3.**
   * **URL**: enter the **Processed Data Bucket** value you retrieved from the Settings page in the Panther Console in Step 3.
   * **Storage credential**: select `panther-storage-credential`.
5. Click **Create**.
6. You will be routed to a page with a **Permission Denied** warning box—click **Force create**.\
   ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-14e6e1ccf829f5587ed9e573de2965cb2dfee357%2FScreenshot%202025-10-31%20at%2012.45.58%E2%80%AFPM.png?alt=media)

### Step 8: Create a load service principal in Databricks

1. Access your Databricks workspace settings:
   1. In the upper-right corner, click your initial.
   2. Click **Settings**.\
      ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-d5d21f8e58937167fbd9056eb13de6c408bd6f7b%2Funknown.png?alt=media)
2. In the **Settings** navigation bar, under **Workspace admin**, click **Identity and access**.
3. To the right of **Service principals**, click **Manage**.\
   ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-df9072e7689801a9d7cc031688b04da9f67c88e9%2FScreenshot%202025-10-30%20at%2012.51.56%E2%80%AFPM.png?alt=media)
4. Click **Add service principal**.
5. In the **Add service principal** modal, click **Add new**.
6. In the **Service principal name** field, enter `panther-load`.
7. Click **Add**.
8. In the table, click **panther-load** to view its details page.
9. Click **Secrets**.
10. Click **Generate secret**.
11. Under **Lifetime (days)**, enter `730` (the maximum).
12. Click **Generate**.
13. Copy the **Secret** and **Client ID** values and store them in a secure location, as you'll need them in a later step (as an alternative to copying these values, you can leave this browser tab open).

### Step 9: Create a load secret KMS key in AWS

1. In your AWS console, ensure you are in the correct region. Navigate to [Key Management Service](https://aws.amazon.com/kms/).
2. In the left-hand navigation menu, click **Customer managed keys**.
3. Click **Create Key**.
4. Under **Key type**, select **Symmetric**. Under **Key usage**, select **Encrypt and decrypt**.
5. Click **Next**
6. Enter an **Alias** value, then click **Next**.
7. Under **Key administrators**, optionally select users and/or roles, then click **Next**.
8. On the **Define key usage permissions&#x20;*****- optiona**l* page, under **Other AWS accounts**, click **Add another AWS account**.
   1. In the field that appears, enter the AWS account ID for the account your Panther deployment is in. You can find this value in the Panther Console, in the [general settings footer](https://docs.panther.com/system-configuration/..#general-settings).
   2. Click **Next**.
9. Switch to a browser tab with the Panther Console open, and retrieve the **Delta Controller Role ARN** and **Delta Admin Role ARN** values:
   1. Click the gear icon (Settings) > **General**.
   2. Click **Data Lake**.
   3. Under **Databricks Configuration**, note the **Delta Controller Role ARN** and **Delta Admin Role ARN** values.\
      ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-cf523729c1aa1d9e8d34f225f788e4cd55566b1b%2FScreenshot%202025-10-30%20at%2011.11.47%E2%80%AFAM.png?alt=media)
10. In the AWS console, under **Key policy**, click **Edit**, then replace the JSON in the code editor with the JSON below:

{% hint style="info" %}
In the policy below, replace:

* `<Delta Controller Role ARN from Panther settings>` with the **Delta Controller Role ARN** value you retrieved above
* `<Delta Admin Role ARN from Panther settings>` with the **Delta Admin Role ARN** value you retrieved above
* `<AWS Account ID you are working in>` with the Account ID of the account you are working in
  {% endhint %}

```json
{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "Panther",
			"Effect": "Allow",
			"Principal": {
				"AWS": [
					"<Delta Controller Role ARN from Panther settings>",
					"<Delta Admin Role ARN from Panther settings>"
				]
			},
			"Action": "kms:Decrypt",
			"Resource": "*"
		},
		{
			"Sid": "root",
			"Effect": "Allow",
			"Action": [
				"kms:*"
			],
			"Resource": "*",
			"Principal": {
				"AWS": "arn:aws:iam::<AWS Account ID you are working in>:root"
			}
		}
	]
}
```

11. Click **Next**.
12. On the **Review** page, review the configuration, then click **Finish**.
13. In the Customer managed keys list, click the alias of the key you just created, to view its detail page.
14. Copy the key ARN into the table above for the `databricks_load_secret_kms_key_arn` row.

{% hint style="info" %}
In [Step 12](#step-12-optional-create-a-query-secret-kms-key), you will either reuse this KMS key or create an additional one.
{% endhint %}

### Step 10: Create a load secret in AWS

1. In your AWS console, ensure you are in the correct region. Navigate to [Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html).

{% hint style="warning" %}
You should not be in the AWS account hosting your Panther infrastructure.
{% endhint %}

2. Click **Store a new secret**.
3. Under **Secret type**, select **Other type of secret**.
4. Under **Key/value pairs**, in the **Key/value** tab, enter the following key value pairs:

   <table data-header-hidden><thead><tr><th width="217.18145751953125">Key</th><th>Value</th></tr></thead><tbody><tr><td>Key</td><td>Value</td></tr><tr><td><code>secret</code></td><td>&#x3C;the <strong>Secret</strong> value you generated in Databricks in <a href="#step-8-create-a-load-service-principal-in-databricks">Step 8</a>></td></tr><tr><td><code>client-id</code></td><td>&#x3C;the <strong>Client ID</strong> value you generated in Databricks in <a href="#step-8-create-a-load-service-principal-in-databricks">Step 8</a>></td></tr><tr><td><code>databricks-host</code></td><td>&#x3C;the URL of your Databricks workspace><br><br>While viewing the workspace you created above in your Databricks console, copy the URL of the page. For example, <code>https://dbc-023ca860-3666.cloud.databricks.com</code></td></tr></tbody></table>
5. Under **Encryption key**, select the `databricks_load_secret_kms_key_arn` KMS key you created in the previous step.
6. Click **Next**.
7. In the **Secret name** field, enter `panther-databricks-admin-access`, then click **Next**.
8. Without making any changes on the **Configure rotation&#x20;*****- optional*** page, click **Next**.
9. Review the secret settings, then click **Store**.
10. In the **Secrets** list, click **panther-databricks-admin-access**, to view its details page.
11. In the **Resource permissions** tile, click **Edit permissions**.
12. Under **Resource permissions**, replace the JSON in the code editor with the JSON below:

{% hint style="info" %}
In the policy below:

* Replace `<Delta Controller Role ARN from Panther settings>` with the **Delta Controller Role ARN** value you retrieved above
* Replace `<Delta Admin Role ARN from Panther settings>` with the **Delta Admin Role ARN** value you retrieved above
* If you will reuse the load secret KMS key you created in [Step 9](#step-9-create-a-load-secret-kms-key-in-aws) in [Step 12](#step-12-optional-create-a-query-secret-kms-key), replace `<Databricks Role ARN from Panther settings>` with the **Databricks Role ARN** value you retrieved above
* If you will not reuse the load secret KMS key you created in [Step 9](#step-9-create-a-load-secret-kms-key-in-aws) (i.e., you will create a new query secret KMS key in [Step 12](#step-12-optional-create-a-query-secret-kms-key)), remove `"<Databricks Role ARN from Panther settings>"` completely
  {% endhint %}

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Panther",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
            "<Delta Controller Role ARN from Panther settings>",
            "<Delta Admin Role ARN from Panther settings>",
            "<Databricks Role ARN from Panther settings>"
        ]
      },
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*"
    }
  ]
}
```

13. Click **Save**.
14. Copy the ARN of the newly created secret and add it as the `databricks_load_secret_arn` value in your [configuration table](#step-1-make-a-copy-of-the-configuration-table).
15. In the Databricks console, return to the **External Data** page (click **Catalog** > **External Data**).
16. Under **External Locations**, click the **panther-processed-data** location you created above.
17. Click **Permissions**.
18. Click **Grant**.
19. Under **Principals**, search for and select **panther-load**.
20. Under **Privileges**, check the boxes for **BROWSE** and **READ FILES**.
21. Click **Confirm**.

### Step 11: Create a query service principal in Databricks

1. Access your Databricks workspace settings:
   1. In the upper-right corner, click your initial.
   2. Click **Settings**.\
      ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-d5d21f8e58937167fbd9056eb13de6c408bd6f7b%2Funknown.png?alt=media)
2. In the **Settings** navigation bar, under **Workspace admin**, click **Identity and access**.
3. To the right of **Service principals**, click **Manage**.\
   ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-df9072e7689801a9d7cc031688b04da9f67c88e9%2FScreenshot%202025-10-30%20at%2012.51.56%E2%80%AFPM.png?alt=media)
4. Click **Add service principal**.
5. In the **Add service principal** modal, click **Add new**.
6. In the **Service principal name** field, enter `panther-query`.
7. Click **Add**.
8. In the table, click **panther-query** to view its details page.
9. Click **Secrets**.
10. Click **Generate secret**.
11. Under **Lifetime (days)**, enter `730` (the maximum).
12. Click **Generate**.
13. Copy the **Secret** and **Client ID** values and store them in a secure location, as you'll need them in a later step (as an alternative to copying these values, you can leave this browser tab open).

### Step 12 (Optional): Create a query secret KMS key

In the next step, you'll create an additional secret in AWS. You can either create a new KMS key to associate to this secret, or reuse the KMS key you created in Step 9 (added to your configuration table as `databricks_load_secret_kms_key_arn`).

* If you'd like to reuse the KMS key you created above, copy the value of `databricks_load_secret_kms_key_arn` to `databricks_query_secret_kms_key_arn` in the configuration table above.
* If you'd like to create a new KMS key, repeat [Step 9: Create a load secret KMS key in AWS](#step-9-create-a-load-secret-kms-key-in-aws), then add the ARN for the key as `databricks_query_secret_kms_key_arn` in the configuration table above.

### Step 13: Create a query secret in AWS

1. In your AWS console, ensure you are in the correct region. Navigate to [Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html).

{% hint style="warning" %}
This should NOT be created in the AWS account hosting your Panther infrastructure.
{% endhint %}

2. Click **Store a new secret**.
3. Under **Secret type**, select **Other type of secret**.
4. Under **Key/value pairs**, in the **Key/value** tab, enter the following key value pairs:

   <table data-header-hidden><thead><tr><th width="217.18145751953125">Key</th><th>Value</th></tr></thead><tbody><tr><td>Key</td><td>Value</td></tr><tr><td><code>secret</code></td><td>&#x3C;the <strong>Secret</strong> value you generated in Databricks in <a href="#step-11-create-a-query-service-principal-in-databricks">Step 11</a>></td></tr><tr><td><code>client-id</code></td><td>&#x3C;the <strong>Client ID</strong> value you generated in Databricks in <a href="#step-11-create-a-query-service-principal-in-databricks">Step 11</a>></td></tr><tr><td><code>databricks-host</code></td><td>&#x3C;the URL of your Databricks workspace><br><br>While viewing your Databricks workspace in the Databricks console, copy the URL of the page. For example, <code>https://dbc-023ca860-3666.cloud.databricks.com</code></td></tr></tbody></table>
5. Under **Encryption key**, select the `databricks_query_secret_kms_key_arn` KMS key you created in the previous step (or the `databricks_load_secret_kms_key_arn` KMS key, if you are reusing that one).
6. Click **Next**.
7. In the **Secret name** field, enter `panther-databricks-query-access`, then click **Next**.
8. Without making any changes on the **Configure rotation&#x20;*****- optional*** page, click **Next**.
9. Review the settings, then click **Store**.
10. In the **Secrets** list, click **panther-databricks-query-access**, to view its details page.
11. In the **Resource permissions** tile, click **Edit permissions**.
12. Under **Resource permissions**, replace the JSON in the code editor with the JSON below:

{% hint style="info" %}
In the policy below, replace:

* `<Databricks Role ARN from Panther settings>` with the **Databricks Role ARN** value you retrieved above
  {% endhint %}

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Panther",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
            "<Databricks Role ARN from Panther settings>"
        ]
      },
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*"
    }
  ]
}
```

13. Click **Save**.
14. Copy the ARN of the newly created secret and add it as the `databricks_query_secret_arn` value in your [configuration table](#step-1-make-a-copy-of-the-configuration-table).

### Step 14: Create an S3 bucket and external location

1. In your AWS console, ensure you are in the correct region. Navigate to [S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html).

{% hint style="warning" %}
You should not be in the AWS account hosting your Panther infrastructure.
{% endhint %}

2. Click **Create bucket**.
3. Enter a **Bucket name**.
4. Click **Create bucket**.
5. In the Databricks workspace you created above, click **Catalog**, then **External Data**.\
   ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-7c66d9e3150d25358ef4f8a87e5a9ea46a56c582%2FScreenshot%202025-10-30%20at%2012.31.13%E2%80%AFPM.png?alt=media)
6. Click **Create external location**.
7. Click **AWS Quickstart (Recommended)**, then **Next**.
8. In the **Bucket Name** field, enter the name of the bucket you just created.
9. Under **Personal Access Token**, click **Generate new token**.
   * Copy this value, as you'll need it in the follow steps. Alternatively, you can leave this page open.
10. Click **Launch in Quickstart**.
    * A new browser tab will open in AWS, on a **Quick create stack** screen with the CloudFormation template pre-loaded.
11. In the **Parameters** section, in the **Databricks Personal Access Token** field, enter the **Personal Access Token** you generated above in Databricks.
12. Click **Create stack**.
13. After the stack has completed deploying, return to your Databricks console browser tab. On the **Create external location with Quickstart** screen, click **Ok**.\
    ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-ea25a95983540cd9f8bdbea308848423d4973758%2FScreenshot%202025-11-13%20at%201.51.00%E2%80%AFPM.png?alt=media)
    * Verify that the **External Locations** list contains the one you just created.

### Step 15: Create a Databricks catalog

1. In your Databricks workspace, click **Catalog**.
2. Click **Add data** > **Create a catalog**.\
   ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-db7d214539e16b1700c4b47bcfdf437d380f9732%2FScreenshot%202025-11-05%20at%2012.08.38%E2%80%AFPM.png?alt=media)
3. Fill in the **Create a new catalog** form:
   * **Catalog name**: enter a name for your catalog, e.g., `panther`.

     <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p>It's recommended to name the catalog <code>panther</code>, but not required.</p></div>
   * **Type**: select **Standard**.
   * **Select external location**: choose the external location you created in [Step 14](#step-14-create-an-s3-bucket-and-external-location).

     <div data-gb-custom-block data-tag="hint" data-style="warning" class="hint hint-warning"><p>Do not choose <strong>panther-processed-data</strong>.</p></div>

     \
     ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-860d8df774dc6201495017a87c86574b2126bb46%2FScreenshot%202025-11-14%20at%2010.30.48%E2%80%AFAM.png?alt=media)
4. Click **Create**.
5. On the **Catalog created!** modal, click **View catalog**.
6. Click **Permissions**.
7. Click **Grant**.
8. In the **Grant on panther** modal, fill in the form:
   * **Principals**: type and select `panther-load`.
   * Select the following permissions:
     * **USE CATALOG**
     * **USE SCHEMA**
     * **BROWSE**
     * **SELECT**
     * **MODIFY**
     * **CREATE SCHEMA**
     * **CREATE TABLE**\
       ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-394374ffa67af1e3d59f9bcc845bbb473a62c4f4%2FScreenshot%202025-11-14%20at%2010.48.29%E2%80%AFAM.png?alt=media)
9. Click **Confirm**.
10. Click **Grant**.
11. In the **Grant on panther** modal, fill in the form:
    * **Principals**: type and select `panther-query`.
    * Select the following permissions:
      * **USE CATALOG**
      * **USE SCHEMA**
      * **BROWSE**
      * **SELECT**\
        ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-0ce210c66af22724c02b0b7a58bd413213f8295a%2FScreenshot%202025-11-14%20at%2010.54.34%E2%80%AFAM.png?alt=media)
12. Click **Confirm**.
13. Add the catalog name as the `databricks_catalog` value in your [configuration table](#step-1-make-a-copy-of-the-configuration-table).

### Step 16: Create a panther-load SQL warehouse

{% hint style="info" %}
For additional SQL warehouse creation support, see the Databricks [Create a SQL warehouse](https://docs.databricks.com/aws/en/compute/sql-warehouse/create) documentation.
{% endhint %}

1. In your Databricks workspace, click **Compute**.
2. Click **SQL warehouses**.
3. Click **Create SQL warehouse**.
4. Fill out the **New SQL warehouse** form:
   * **Name**: enter `panther-load`.
   * **Cluster size**: select **2X-Small**.
   * **Scaling**: set the **Max** value to `40` (the maximum allowed).
   * **Type**: select **Pro**.

{% hint style="warning" %}
Do not use **Serverless**.
{% endhint %}

5. Click **Create**.
6. In the **Manage permissions** modal, add the **panther-load** user, then select **Can use** permissions.\
   ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-cf4c503fdbc8ac6265c57d15a44659089505cdb7%2FScreenshot%202025-11-05%20at%201.34.24%E2%80%AFPM.png?alt=media)
7. Click **Add**.
8. Click the **X** in the upper-right corner to close the **Manage permissions** modal.
9. On the **panther-load** warehouse details page, copy the **ID** (next to the name) and add it as the `databricks_load_warehouse_id` value in your [configuration table](#step-1-make-a-copy-of-the-configuration-table).\
   ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-48716b09f2db92922980be4ce1f71cadc99112d5%2FScreenshot%202025-11-05%20at%201.38.43%E2%80%AFPM.png?alt=media)

### Step 17: Create a panther-optimize SQL warehouse

This warehouse runs nightly table maintenance jobs.

{% hint style="info" %}
For additional SQL warehouse creation support, see the Databricks [Create a SQL warehouse](https://docs.databricks.com/aws/en/compute/sql-warehouse/create) documentation.
{% endhint %}

1. In your Databricks workspace, click **Compute**.
2. Click **SQL warehouses**.
3. Click **Create SQL warehouse**.
4. Fill out the **New SQL warehouse** form:
   * **Name**: enter `panther-optimize`.
   * **Cluster size**: select **2X-Small**.
   * **Scaling**: set the **Max** value to `40` (the maximum allowed).
   * **Type**: select **Serverless**.
5. Click **Create**.
6. In the **Manage permissions** modal, add the **panther-load** user, then select **Can use** permissions.\
   ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-cf4c503fdbc8ac6265c57d15a44659089505cdb7%2FScreenshot%202025-11-05%20at%201.34.24%E2%80%AFPM.png?alt=media)
7. Click **Add**.
8. Click the **X** in the upper-right corner to close the **Manage permissions** modal.
9. On the **panther-optimize** warehouse details page, copy the **ID** (next to the name) and add it as the `databricks_optimize_warehouse_id` value in your [configuration table](#step-1-make-a-copy-of-the-configuration-table).\
   ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-c6697074956cf9040ce4a69b363516f8d549e8d9%2FScreenshot%202025-11-05%20at%201.44.47%E2%80%AFPM.png?alt=media)

### Step 18: Create a panther-query SQL warehouse

{% hint style="info" %}
For additional SQL warehouse creation support, see the Databricks [Create a SQL warehouse](https://docs.databricks.com/aws/en/compute/sql-warehouse/create) documentation.
{% endhint %}

1. In your Databricks workspace, click **Compute**.
2. Click **SQL warehouses**.
3. Click **Create SQL warehouse**.
4. Fill out the **New SQL warehouse** form:
   * **Name**: enter `panther-query`.
   * **Cluster size**: select **Medium**.
   * **Scaling**: set the **Max** value to `40` (the maximum allowed).
   * **Type**: select **Serverless** or **Pro**.

{% hint style="info" %}
This SQL warehouse can be **Serverless** or **Pro**, but **Serverless** is recommended. **Pro** warehouses start up slowly.
{% endhint %}

5. Click **Create**.
6. In the **Manage permissions** modal, add the **panther-query** user, then select **Can use** permissions.\
   ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-f1b9a840a4f1fd234c2a3ad47ee78d9ea2ce093b%2FScreenshot%202025-11-05%20at%201.50.14%E2%80%AFPM.png?alt=media)
7. Click **Add**.
8. Click the **X** in the upper-right corner to close the **Manage permissions** modal.
9. On the **panther-query** warehouse details page, copy the **ID** (next to the name) and add it as the `databricks_query_warehouse_id` value in your [configuration table](#step-1-make-a-copy-of-the-configuration-table).\
   ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-d3871aca92acc0f84205e15b79b83ff404fa306e%2FScreenshot%202025-11-05%20at%201.51.10%E2%80%AFPM.png?alt=media)

### Step 19: Create a panther-scheduled-query SQL warehouse

{% hint style="info" %}
For additional SQL warehouse creation support, see the Databricks [Create a SQL warehouse](https://docs.databricks.com/aws/en/compute/sql-warehouse/create) documentation.
{% endhint %}

1. In your Databricks workspace, click **Compute**.
2. Click **SQL warehouses**.
3. Click **Create SQL warehouse**.
4. Fill out the **New SQL warehouse** form:
   * **Name**: enter `panther-scheduled-query`.
   * **Cluster size**: select **3X-Large**.
   * **Scaling**: set the **Max** value to `40` (the maximum allowed).
   * **Type**: select **Serverless** or **Pro**.

{% hint style="info" %}
This SQL warehouse can be **Serverless** or **Pro**, but **Serverless** is recommended. **Pro** warehouses start up slowly.
{% endhint %}

5. Click **Create**.
6. In the **Manage permissions** modal, add the **panther-query** user, then select **Can use** permissions.\
   ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-f1b9a840a4f1fd234c2a3ad47ee78d9ea2ce093b%2FScreenshot%202025-11-05%20at%201.50.14%E2%80%AFPM.png?alt=media)
7. Click **Add**.
8. Click the **X** in the upper-right corner to close the **Manage permissions** modal.
9. On the **panther-scheduled-query** warehouse details page, copy the **ID** (next to the name) and add it as the `databricks_scheduled_query_warehouse_id` value in your [configuration table](#step-1-make-a-copy-of-the-configuration-table).\
   ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-050d443fa678787a928bd8f03b50c8edf6dfa19f%2FScreenshot%202025-11-05%20at%202.00.40%E2%80%AFPM.png?alt=media)

### Step 20: Send configuration values to Panther

* Now that the [configuration table you created in Step 1](#step-1-make-a-copy-of-the-configuration-table) is completely filled-in, share it with the Panther team.

### Step 21: Return to the post-setup recommendations

* Return to the [Post-setup recommendations](https://docs.panther.com/system-configuration/panther-deployment-types/set-up#post-setup-recommendations) on [set-up](https://docs.panther.com/system-configuration/panther-deployment-types/cloud-connected/set-up "mention").


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.panther.com/system-configuration/panther-deployment-types/cloud-connected/databricks.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.