# Custom Enrichments

## Overview

Custom enrichments (also referred to as "Lookup Tables") allow you to store and reference custom enrichment data in Panther. This means you can reference this added context in detections and pass it into alerts. It may be particularly useful to create custom enrichments containing identity/asset information, vulnerability context, or network maps.

There are[ four import method options](#how-to-configure-a-custom-enrichment): a Scheduled Search, a static file, S3 bucket, or Google Cloud Storage (GCS) bucket.

You can associate one or more log types with your custom enrichment—then all incoming logs of those types (that match a enrichment table value) will contain enrichment data. Learn more about the enrichment process in [How incoming logs are enriched](#how-incoming-logs-are-enriched). It's also possible to [dynamically reference enrichment data in Python detections](#option-2-dynamically-using-lookup). Learn how to [view stored enrichment data here](https://docs.panther.com/enrichment/..#viewing-and-managing-enrichments), and how to [view log events with enrichment data here](https://docs.panther.com/enrichment/..#viewing-log-events-with-enrichment-data).

If your data is only needed for a few specific detections and will not be frequently updated, consider using [Global helpers](https://docs.panther.com/writing-detections/globals) instead of an enrichment. Also note that you can use [Panther-managed enrichments](https://docs.panther.com/enrichment/..#panther-managed-lookup-tables) like like IPinfo and Tor Exit Nodes.

To increase the limit on the number of custom enrichments and/or size of your enrichment tables, please contact your Panther support team.

## How incoming logs are enriched

Enrichments in Panther traditionally define both of the following:

* A primary key: A field in the enrichment table data.
  * If the enrichment is defined in the CLI workflow, this is designated by the `PrimaryKey` field in the [YAML configuration file](https://docs.panther.com/enrichment/custom/lookup-table-specification-reference).
* One or more associated log types, each with one or more Selectors: A Selector is an event field whose values are compared to the enrichment table's primary key values to find a match.
  * There are two ways to set log types/Selectors for an enrichment. See [How log types and Selectors are set for an enrichment](#how-log-types-and-selectors-are-set-for-a-lookup-table), below.

When a log is ingested into Panther, if its log type is one that is associated to an enrichment, the values of all of its Selector fields are compared against the enrichment's primary key values. When a match is found between a value in a Selector field and a primary key value, the log is enriched with the matching primary key's associated enrichment data in a `p_enrichment` field. Learn more about `p_enrichment` below, in [`p_enrichment` structure](#p_enrichment-structure).

In the example in the image below, the Selector field (in the events in Incoming Logs) is `ip_address`. The primary key of the enrichment LUT1 is `bad_actor_ip`. In the right-hand Alert Event, the log is enriched with the enrichment data (including `bad_actor_name`) because there was a match between the Selector value (`1.1.1.1`) and a primary key value (`1.1.1.1`).

<figure><img src="https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-10f2fd323cb1b5165c5ed62e15c39e04c711c920%2F6.21.23_Lookup-Tables-Diagram-23.png?alt=media" alt="A diagram showing how Lookup Tables work: On the left, there is a code box labeled &#x22;Incoming logs.&#x22; An arrow branches off the logs and points to a Lookup Table including &#x22;bad_actor_ip&#x22; and &#x22;bad_actor_name.&#x22; On the right, an arrow goes from the Lookup Table to an Alert Event, showing the alert you would receive based on the log example."><figcaption></figcaption></figure>

### How log types and Selectors are set for an enrichment

You can manually set associated log types and Selectors when creating an enrichment ([Option 1](#option-1-manually-choose-log-types-and-selectors)), and/or let them be automatically mapped ([Option 2](#option-2-let-log-types-and-selectors-be-automatically-mapped-by-indicator-fields)).

#### Option 1: Manually choose log types and Selectors

When creating an enrichment, you can choose one or more log types the enrichment should be associated to—and for each log type, one or more Selector fields.

{% hint style="info" %}
Note that even if you manually choose log types and Selectors in this way, the automatic mappings described in [Option 2](#option-2-let-log-types-and-selectors-be-automatically-mapped-by-indicator-fields) will still be applied.
{% endhint %}

{% tabs %}
{% tab title="Panther Console" %}

* When creating an enrichment in the Panther Console, you can set associated log types and Selectors. Learn more in [How to configure a custom enrichment](#how-to-configure-a-lookup-table), below.

<figure><img src="https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-9b670b91bf90f2be483a376cb65bedb05a1eb604%2FScreenshot%202024-08-15%20at%2010.22.48%20AM.png?alt=media" alt="Under an &#x22;Associated Log Types (Optional)&#x22; header is a form with fields for Log Type, Selectors, and Add Log Type. At the bottom is a Continue button."><figcaption></figcaption></figure>
{% endtab %}

{% tab title="CLI workflow" %}

* When creating an enrichment in the CLI workflow, you will create and upload a YAML configuration file. The `AssociatedLogTypes` value will be a list of objects containing `LogType` and `Selectors` fields.
* See the [Custom Enrichment Specification Reference](https://docs.panther.com/enrichment/custom/lookup-table-specification-reference) for a full list of fields.
  {% endtab %}
  {% endtabs %}

#### Option 2: Let log types and selectors be automatically mapped

The primary key of your enrichment can be marked with an [indicator](https://docs.panther.com/search/panther-fields#indicator-fields) or with [CIDR validation](https://docs.panther.com/data-onboarding/custom-log-types/reference#ip-and-cidr-format-validation).

Indicator example:

```yaml
    - name: attack_ids           # this is the primary key
      description: Attack field
      type: array
      element:
        type: string
        indicators:
            - mitre_attack_technique
```

For each indicator value, Panther automatically:

1. Finds all **Active** log types that designate any event field as that same indicator.
2. Associates those log types to the enrichment.
3. For each log type, sets the `p_any` field associated to the indicator as a Selector.

For example, if your enrichment data's schema designates the `attack_ids` primary key as a `mitre_attack_technique` indicator as is shown above, all log types in your Panther instance that also set a `mitre_attack_technique` indicator will be associated to the enrichment, each with a `p_any_mitre_attack_techniques` Selector.

CIDR Validation example:

```yaml
    - name: cidr           # this is the primary key
      description: CIDR block
      type: string
      validate:
        cidr: ipv4
```

In this case, Panther automatically:

* Finds all **Active** log types that designate any event field with `ip` indicator.
* Associates those log types to the enrichment.
* For each log type, sets the `p_any_ip_addresses` field associated to the indicator as a Selector.

This mapping happens each time an enrichment's data is refreshed.

### `p_enrichment` structure

If your log events are injected with enrichment data, a `p_enrichment` field is appended to the event and accessed within a detection using `deep_get()` or `DeepKey`. The `p_enrichment` field will contain:

* One or more enrichment name(s) that matched the incoming log event
* The name of the Selector from the incoming log that matched the enrichment
* The data from the enrichment that matched via the enrichment's primary key (including an injected `p_match` field containing the Selector value that matched)

This is the structure of `p_enrichment` fields:

```yaml
'p_enrichment': {
    <name of enrichment1>: {
        <name of selector>: {
            'p_match': <value of Selector>,
	          <enrichment key>: <enrichment value>,
	          ...
	      }
    }
}
```

Note that `p_enrichment` is not stored with the log event in the data lake. See [Viewing log events with enrichment data](https://docs.panther.com/enrichment/..#viewing-log-events-with-enrichment-data) for more information.

## How to access enrichment data in detections

### Option 1 (if log is enriched): Using `deep_get()`

If your log event was enriched on ingest (as described in [How incoming logs are enriched](#how-incoming-logs-are-enriched)), you can access the data within the `p_enrichment` field ([whose structure is described above](#p_enrichment-structure)) using the `deep_get()` event object function. Learn more about `deep_get()` [on Writing Python Detections](https://docs.panther.com/detections/rules/python#deep_get).

See a full example of this method below, in [Writing a detection using custom enrichment data](#writing-a-detection-using-lookup-table-data).

### Option 2: Dynamically using `lookup()`

It's also possible to dynamically access enrichment data from Python detections using [the `event.lookup()` function](https://docs.panther.com/detections/rules/python#lookup). In this way, you can retrieve data from any enrichment, without it being injected into an incoming event as described in [How incoming logs are enriched](#how-incoming-logs-are-enriched).

## Prerequisites for configuring a custom enrichment

Before configuring an enrichment, be sure you have:

* A schema specifically for your enrichment data
  * This describes the shape of your enrichment data.
* A primary key for your enrichment data
  * This primary key is one of the fields you defined in your enrichment's schema. The value of the primary key is what will be compared with the value of the selector(s) from your incoming logs.
  * See the below [Primary key data types](#primary-key-data-types) section to learn more about primary key requirements.
* (Optional) Selector(s) from your incoming logs
  * The values from these selectors will be used to search for matches in your enrichment data.
* (CLI workflow): An enrichment configuration file
  * See [Custom Enrichment Specification Reference](https://docs.panther.com/enrichment/custom/lookup-table-specification-reference) for a full list of fields.
  * We recommend you [make a fork of the `panther-analysis` repository](https://docs.panther.com/panther-developer-workflows/detections-repo/setup/deprecated/public-fork) and install the [Panther Analysis Tool (PAT)](https://docs.panther.com/panther-developer-workflows/detections-repo/pat).
* If you're using a file upload or S3/GCS bucket:
  * Enrichment data in JSON or CSV format
    * JSON files can format events in various ways, including in lines, arrays, or objects.
* If you're using a Scheduled Search:
  * A SQL query that returns the data you want to use for enrichment
    * Make sure you have access to your data warehouse through [Panther's Data Explorer](https://docs.panther.com/search/data-explorer).

### Primary key data types

Your enrichment table's primary key column must be one of the following data types:

* String
* Number
* Array (of strings or numbers)
  * Using an array lets you associate one row in your enrichment table with multiple string or number primary key values. This prevents you from having to duplicate a certain row of data for multiple primary keys.

<details>

<summary>Example: string array vs. string primary key type</summary>

Perhaps you'd like to store user data in a custom enrichment table so that incoming log events associated with a certain user are enriched with additional personal information. You'd like to match on the user's email address, which means the email field will be the primary key in the enrichment table and the Selector in the log events.

You are deciding whether the primary key column in your enrichment table should be of type string or string array.

First, review the below two events you might expect to receive from your log source:

```json
# Incoming log event one
{
    "actor_email": "janedoeemailone@mail.com",
    "action": "LOGIN"
}

# Incoming log event two
{
    "actor_email": "janedoeemailtwo@mail.com",
    "action": "EXPORT_FILE"
}
```

Note that the two email addresses (`janedoeemailone@mail.com` and `janedoeemailtwo@mail.com`) belong to the same user, `Jane Doe`.

When Panther receives these events, you would like to use an enrichment table to enrich each of them with Jane's full name and role. After enrichment, these events would look like the following:

<pre class="language-json"><code class="lang-json"><strong># Log event one after enrichment
</strong>{
    "actor_email": "janedoeemailone@mail.com",
    "action": "LOGIN",
    "p_enrichment": {
        "&#x3C;lookup_table_name>": {
            "actor_email": {
                "full_name": "Jane Doe",
                "p_match":  "janedoeemailone@mail.com",
                "role": "ADMIN"
            }
        }
    }
}

<strong># Log event two after enrichment
</strong>{
    "actor_email": "janedoeemailtwo@mail.com",
    "action": "EXPORT_FILE",
    "p_enrichment": {
        "&#x3C;lookup_table_name>": {
            "actor_email": {
                "full_name": "Jane Doe",
                "p_match": "janedoeemailtwo@mail.com",
                "role": "ADMIN"
            }
        }
    }
}
</code></pre>

You can accomplish this enrichment by defining a custom enrichment with either:

* (Recommended) A primary key column that is of type array of strings
* A primary key column that is of type string

Using an enrichment table with a primary key column that is of type array of strings, you can include Jane's multiple email addresses in one primary key entry, associated to one row of data. This might look like the following:

Alternatively, you can define an enrichment table with a primary key column that is of type string. However, because the match between the event and enrichment table is made on the user's email address, and a user can have multiple email addresses (as is shown in Jane's case), you must duplicate the enrichment table row for each email. This would look like the following:

While both options yield the same result (i.e., log events are enriched in the same way), defining an enrichment with an array of strings primary key is recommended for its convenience and reduced proneness to maintenance error.

</details>

## How to configure a custom enrichment

After fulfilling the [prerequisites](#prerequisites-for-configuring-a-custom-enrichment), custom enrichments can be created and configured using one of the following methods:

* [Option 1: Import custom enrichment data with a Scheduled Search](#option-1-import-custom-enrichment-data-with-a-scheduled-search)
  * Best for importing data directly from your data warehouse with automatic updates on a recurring schedule.
  * Example: tracking suspicious IPs, user behavior patterns, or AWS account mappings that need to be refreshed automatically based on your current environment.
* [Option 2: ](#option-2-import-lookup-table-data-via-file-upload)[Import custom enrichment data via file upload](https://docs.panther.com/enrichment/custom#option-1-import-lookup-table-data-via-file-upload)
  * Best for data that is relatively static, such as information about AWS accounts or corporate subnets.
  * The maximum size of custom enrichment tables populated via file upload is 6 MB (due to a limit on the payload size of the API request).
  * Example: adding metadata to distinguish developer and production accounts in your AWS CloudTrail logs.
* [Option 3: Sync custom enrichment data from an S3 bucket](#option-3-sync-custom-enrichment-data-from-an-s3-bucket) or [Option 4: Sync custom enrichment data from a Google Cloud Storage (GCS) bucket](#option-4-sync-custom-enrichment-data-from-a-google-cloud-storage-gcs-bucket)
  * Best when you have a large amount of data that updates relatively frequently. Any changes in the S3 or GCS bucket will sync to Panther.
  * The maximum size of custom enrichment tables synced from S3 or GCS bucket is 10 GB.
  * Example: if you wanted to know which groups and permission levels are associated with employees at your company. In this scenario, your company might have an S3 bucket with an up-to-date copy of their Active Directory listing that includes groups and permissions information.

After choosing one of these methods, you can opt to work within the Panther Console or with [PAT](https://docs.panther.com/panther-developer-workflows/detections-repo/pat).

{% hint style="warning" %}
The maximum size for a row in a custom enrichment table is 65535 bytes.
{% endhint %}

### Option 1: Import custom enrichment data with a Scheduled Search

{% hint style="info" %}
Importing custom enrichment data with a Scheduled Search is in open beta starting with Panther version 1.117, and is available to all customers. Please share any bug reports and feature requests with your Panther support team.
{% endhint %}

Custom enrichment data can be automatically populated by the results from a [Scheduled Search](https://docs.panther.com/search/scheduled-searches). This method eliminates the need for external scripts or manual CSV exports, allowing you to maintain enrichment data that updates automatically on a defined schedule.

{% hint style="warning" %}
**Limitations of importing custom enrichment data with a Scheduled Search**

* Custom enrichments can only handle [Scheduled Searches](https://docs.panther.com/search/scheduled-searches) up to 100 MB.
* CIDR matching is not available.
* Scheduled Searches can run at a minimum frequency of every 15 minutes.
* Search timeout is fixed at 15 minutes and cannot be configured.
  {% endhint %}

<details>

<summary>Panther Console</summary>

**Import data with a Scheduled Search through the Panther Console**

1. In the left-hand navigation bar in your Panther Console, click **Configure** > **Enrichments.**
2. In the upper-right corner, click **Create New,** and then select **Custom Enrichment**.
   * Alternatively, you can start from the **Investigate > Data Explorer** page, enter your query, and then click **Create Enrichment via query**.
3. On the **Enrichment Basic Information** page, fill in the fields:
   * **Enrichment Name**: A descriptive name for your custom enrichment.
   * **Enabled?**: Ensure this toggle is set to `YES`. This is required to import your data later in this process.
   * **Description - Optional**: Additional context about the table.
   * **Reference - Optional**: Typically used for a hyperlink to an internal resource.
4. Click **Continue**.
5. On the **Choose Import Method** page, on the **Import Data with a Query** tile, click **Select**.
   * The import method is automatically selected and this page is skipped if you started from the **Data Explorer** page.
6. On the **Associated Log Types (Optional)** page, designate log types/Selectors:

   <div data-gb-custom-block data-tag="hint" data-style="warning" class="hint hint-warning"><p>While this step is technically optional (i.e., you can bypass this page without designating log types/Selectors), you must make these designations in order for any logs to be enriched, as <a href="https://docs.panther.com/enrichment/custom#option-2-let-log-types-and-selectors-be-automatically-mapped">auto-mapping</a> is not supported.</p></div>

   1. Click **Add Log Type**.
   2. Click the **Log Type** dropdown, then select a log type.
   3. Choose one or more **Selectors**, the foreign key fields from the log type you want enriched with your custom enrichment.
      * You also can reference attributes in nested objects using [JSON path syntax](https://goessner.net/articles/JsonPath/). For example, if you wanted to reference a field in a map, you could enter `$.field.subfield`.
   4. Click **Add Log Type** to add another, if needed.
7. Click **Continue**.
8. On the **Set Up Your Query** page, fill in the fields:
   * **SQL Query**: Write your SQL query that will return the data for your custom enrichment. You can test your query in Data Explorer by clicking **Test in Data Explorer**.
     * If your query is pulling data ingested by Panther, make sure to exclude or rename any [standard fields](https://docs.panther.com/search/panther-fields) that start with the `p_` prefix, since these are reserved names.
     * The enrichment schema will be generated based on your SQL query. You can modify the schema fields and type by modifying the query e.g. `SELECT AVG(total_events)::FLOAT AS mean_total_events` will create a field of type `float` and name `mean_total_events`
   * **Schedule Type & Frequency**: Configure the scheduling frequency based on how often your data needs to be refreshed. Choose between:
     * **Period**: Run the query at fixed time intervals (e.g., every 2 hours)
     * **Cron**: Run the query at specific dates and times using [Cron expressions](https://docs.panther.com/search/scheduled-searches#how-to-use-the-scheduled-search-crontab)
9. Click **Continue**. Panther will generate a schema from your SQL query automatically.
10. On the **Table Schema** page, review the automatically generated schema and select your **Primary Key Name** from the dropdown.
    * For any `string` fields, you can optionally add [indicator fields](https://docs.panther.com/search/panther-fields#indicator-fields) by clicking **Add Indicators** next to the field. Select one or more indicator types (e.g., IP Address, Domain Name, SHA256 Hash) from the dropdown. Adding indicators enables [automatic log type and Selector mapping](#option-2-let-log-types-and-selectors-be-automatically-mapped) for your enrichment.
    * To ensure a field is eligible for indicators, cast it as `STRING` in your SQL query.
11. Click **Create Enrichment**. A source setup success page will populate.
12. Optionally, set the **Set an alarm in case this enrichment doesn't receive any data?**, toggle to `YES` to enable an alarm.
    * Fill in the **Number** and **Period** fields to indicate how often Panther should send you this notification.
    * The alert destinations for this alarm are displayed at the bottom of the page. To configure and customize where your notification is sent, see [Panther Destinations](https://docs.panther.com/alerts/destinations).
13. Click **Finish Setup**.

{% hint style="info" %}
Your Scheduled Search will run according to the configured schedule and automatically update your custom enrichment data. You can view Search history on the [Search History page](https://docs.panther.com/search/search-history).
{% endhint %}

</details>

### Option 2: Import custom enrichment data via file upload

You can import data via file upload through the Panther Console or PAT:

<details>

<summary>Panther Console</summary>

**Import custom enrichment data via file upload through the Panther Console**

1. In the left-hand navigation bar in your Panther Console, click **Configure** > **Enrichments**.
2. In the upper-right corner, click **Create New**.
3. On the **Basic Information** page:
   * **Enrichment Name**: A descriptive name for your custom enrichment.
   * **Enabled?**: Ensure this toggle is set to `YES`. This is required to import your data later in this process.
   * **Description - Optional**: Additional context about the table.
   * **Reference - Optional**: Typically used for a hyperlink to an internal resource.\ <img src="https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-305b9013173d5e5ad60d7d98ca2eb6c24eccfcb8%2Fimage%20(1)%20(13).png?alt=media" alt="" data-size="original">
4. Click **Continue**.
5. On the **Associated Log Types (Optional)** page, optionally designate log types/Selectors:
   * Click **Add Log Type**.
   * Click the **Log Type** dropdown, then select a log type.
   * Choose one or more **Selectors**, the foreign key fields from the log type you want enriched with your custom enrichment.
     * You also can reference attributes in nested objects using [JSON path syntax](https://goessner.net/articles/JsonPath/). For example, if you wanted to reference a field in a map, you could enter `$.field.subfield`.
   * Click **Add Log Type** to add another if needed.\
     ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-568dbf4e3ac50d527771a6b7dd5aa7cefc2ceffc%2Flookup-table-log-types.png?alt=media)\
     In the example screen shot above, we selected **AWS.CloudTrail** logs and typed in `accountID` and `recipientAccountID` to represent keys in the CloudTrail logs.
6. Click **Continue**.
7. On the **Table Schema** page, configure the Table Schema:\
   \&#xNAN;*Note: If you have not already created a new schema, please see* [*our documentation on creating schemas*](https://docs.runpanther.io/data-onboarding/custom-log-types/example-csv)*. You can also use your custom enrichment data to infer a schema. Once you have created a schema, you will be able to choose it from the dropdown on the Table Schema page while configuring an Enrichment.*\
   \&#xNAN;*Note: CSV schemas require column headers to work with custom enrichments*.
   * Select a **Schema Name** from the dropdown.
   * Select a **Primary Key Name** from the dropdown. This should be a unique column on the table, such as `accountID`.\
     ![The image shows the Table Schema form from the Panther Console. The Schema Name field is filled in with "Custom.AWSaccountIDs" and the Primary Key Name field is set to "awsacctid."](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-216b4f8d1528f292f1cef160b0daa36722779bc1%2Flookup-table-schema.png?alt=media)
8. Click **Continue**.
9. On the **Choose Import Method** page, on the **Import via File Upload** tile, click **Set Up**.
10. On the **Upload File** page, drag and drop a file or click **Select file** to choose the file of your Enrichment data to import. The file must be in `.csv` or `.json` format.
11. Click **Finish Setup**. A source setup success page will populate.
12. Optionally, next to to ***S*****et an alarm in case this custom enrichment doesn't receive any data?**, toggle the setting to **YES** to enable an alarm.

    * Fill in the **Number** and **Period** fields to indicate how often Panther should send you this notification.
    * The alert destinations for this alarm are displayed at the bottom of the page. To configure and customize where your notification is sent, see documentation on [Panther Destinations](https://docs.panther.com/alerts/destinations).

    ​![The image shows a success page in the Panther Console after creating a Lookup Table. There is an option to set an  alarm in case this Lookup Table doesn't receive any data, and dropdown selectors to choose a time period when you receive the alerts. At the bottom, there is a section showing which destinations will receive the alert.](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2FoyqgS5R8TnNzpDRtl9e9%2Fimage.png?alt=media\&token=5d8c27d1-ac28-460c-a2f5-bd2c503a49f1)

{% hint style="info" %}
Notifications generated for an custom enrichment upload failing are accessible in the **System Errors** tab within the **Alerts & Errors** page in the Panther Console.
{% endhint %}

</details>

<details>

<summary>PAT</summary>

**Import custom enrichment data via file upload through PAT**

{% hint style="info" %}
If your data file is larger than 1MB, it's suggested to instead use the [S3 sync upload method](#option-2-sync-via-s3-source) or [GCS sync upload method](#option-3-sync-via-google-cloud-storage-source).
{% endhint %}

**File and folder setup**

A custom enrichment requires the following files:

* A YAML configuration file for the enrichment table
  * This custom enrichment configuration file must be stored in a folder with a name containing `lookup_tables`*.* This could be a top-level `lookup_tables` directory, or sub-directories with names matching `*lookup_tables*`. You can use the [panther-analysis](https://github.com/panther-labs/panther-analysis) repository as a reference.
* A YAML file defining the schema to use when loading data into the table
  * This data schema file must be stored in a directory outside the `lookup_tables` directory. You might choose to save it with your other custom log schemas, e.g., in a `/schemas` directory at the root of your panther-analysis repository.
* A JSON or CSV file containing data to load into the table (optional, read further).

**Writing the schema and configuration files**

It's usually prudent to begin writing the data schema first, because the table config will reference some of those values.

1. Create the YAML data schema file. This schema defines how to read the files you'll use to upload data to the table. If using a CSV file for data, then the schema should be able to parse CSV.
   * The table schema is formatted the same as a log schema. For more information on writing schemas, read our documentation around [managing Log Schemas.](https://docs.panther.com/data-onboarding/custom-log-types#uploading-log-schemas-with-the-panther-analysis-tool)
2. Create the YAML configuration file. See a full list of required and allowed values on [Custom Enrichment Specification Reference](https://docs.panther.com/enrichment/custom/lookup-table-specification-reference).
   * For a custom enrichment with data stored in a local file, an example configuration would look like:

     ```yaml
     AnalysisType: lookup_table
     LookupName: my_lookup_table # A unique display name
     Schema: Custom.MyTableSchema # The schema defined in the previous step
     Filename: ./my_lookup_table_data.csv # Relative path to data
     Description: >
       A handy description of what information this table contains.
       For example, this table might convert IP addresses to hostnames
     Reference: >
       A URL to some additional documentation around this table
     Enabled: true # Set to false to stop using the table
     LogTypeMap:
       PrimaryKey: ip                # The primary key of the table
       AssociatedLogTypes:           # A list of log types to match this table to
         - LogType: AWS.CloudTrail
           Selectors:
             - "sourceIPAddress"     # A field in CloudTrail logs
             - "p_any_ip_addresses"  # A panther-generated field works too
         - LogType: Okta.SystemLog
           Selectors:
             - "$.client.ipAddress"  # Paths to JSON values are allowed
     ```
3. From the root of the repository, upload the schema file by running [`panther_analysis_tool update-custom-schemas`](https://docs.panther.com/panther-developer-workflows/detections-repo/pat/pat-commands#update-custom-schemas-creating-or-updating-custom-schemas) `--path ./schemas`.
4. From the root of the repository, upload the custom enrichment using\
   [`panther_analysis_tool upload`](https://docs.panther.com/panther-developer-workflows/detections-repo/pat/pat-commands#upload-uploading-packages-to-panther-directly).

**Update custom enrichments via Panther Analysis Tool:**

1. Locate the YAML configuration file for the custom enrichment in question.
2. Open the file, and look for the field `Filename`. You should see a file path which leads to the data file.
3. Update or replace the file indicated in `Filename`. To see allowed values, see [Custom Enrichment Specification Reference](https://docs.panther.com/enrichment/custom/lookup-table-specification-reference).
4. Save the configuration file, then upload your changes with:

   ```yaml
   panther_analysis_tool upload
   ```

   Optionally, you can specify only to upload the custom enrichment:

   ```yaml
   panther_analysis_tool upload --filter AnalysisType=lookup_table
   ```

</details>

### Option 3: Sync custom enrichment data from an S3 bucket

You can set up data sync from an S3 bucket through the Panther Console or PAT:

<details>

<summary>Panther Console</summary>

**Sync custom enrichment data from an S3 bucket through the Panther Console**

1. In the left-hand navigation bar in your Panther Console, click **Configure > Enrichments**.
2. In the upper-right corner, click **Create New**.
3. Click Add **Custom Enrichment** card.
4. On the **Basic Information** page, fill in the fields:
   * **Enrichment Name**: A descriptive name for your custom enrichment.
   * **Enabled?**: Ensure this toggle is set to `YES`. This is required to import your data later in this process.
   * **Description - Optional**: Additional context about the table.
   * **Reference - Optional**: Typically used for a hyperlink to an internal resource.
   * ![The image shows the Lookup Table Basic Information form from the Panther Console. The Lookup Name field is filled in with "Employee Directory." It is enabled. The description says "Directory with groups and permissions a specific user is associated with." There is a blue button labeled "Continue" at the bottom.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-0245a979983a4fede303cdc48f4a6d78d93f1a52%2Flookup-table-basic-s3.png?alt=media)
5. On the **Associated Log Types** page, optionally designate log types/Selectors:
   * Click **Add Log Type**.
   * Click the **Log Type** dropdown, then select a log type.
   * Choose one or more **Selectors**, the foreign key fields form the log type you want enriched with your custom enrichment data.
     * You also can reference attributes in nested objects using [JSON path syntax](https://goessner.net/articles/JsonPath/). For example, if you wanted to reference a field in a map, you could enter `$.field.subfield`.

* Click **Add Log Type** to add another if needed.\
  ![The image shows the Associated Log Types form from the Panther Console. The Log Type field is set to AWS.VPCFlow. Selectors is set to "account."](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-baa85a56b3740e124717f110214bcb2a3965a4fa%2Flut-logtype-s3.png?alt=media)\
  In the example screen shot above, we selected **AWS.VPCFlow** logs and typed in `account` to represent keys in the VPC Flow logs.

6. Click **Continue**.
7. On the **Table Schema** page, configure the Table Schema.\
   \&#xNAN;*Note: If you have not already created a new schema, please see* [*our documentation on creating schemas*](https://docs.runpanther.io/data-onboarding/custom-log-types/example-csv)*. Once you have created a schema, you will be able to select it from the dropdown on the Table Schema page while configuring a custom enrichment.*
   1. Select a **Schema Name** from the dropdown.
   2. Select a **Primary Key Name** from the dropdown. This should be a unique column on the table, such as `accountID`.
8. Click **Continue**.
9. On the **Choose Import Method** page, on the **Sync Data from an S3 Bucket** tile, click **Set Up**.\
   ![The image shows the "Choose Import Method" page in the Panther Console. The options are Import via File Upload and Sync Data from an S3 Bucket. The Setup button is circled next to "Sync Data from an S3 Bucket."](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-1ef1630e6f723e42c0ba5ced3b2ff247a90b227b%2Flut-import-s3.png?alt=media)
10. Set up your S3 source. Note that your data must be in `.csv` or `.json` format.
    * Enter the **Account ID**, the 12-digit AWS Account ID where the S3 bucket is located.
    * Enter the **S3 URI**, the unique path that identifies the specific S3 bucket.
    * Optionally, enter the **KMS Key** if your data is encrypted using KMS-SSE.
    * Enter the **Update Period**, the cadence your S3 source gets updated (defaulted to 1 hour).\
      ![The image shows the "Set up your S3 source" form from the Panther Console. There are fields for Account ID, S3 URI, KMS Key, and selectors to choose the Update Period.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-c2cfc8478d221f30ffc971dd46db52331ce0c35e%2Flut-setup-s3.png?alt=media)
11. Click **Continue**.
12. Set up an IAM role.
    * Please see the next section, Creating an IAM role, for instructions on the three options available to do this.
13. Click **Finish Setup**. A source setup success page will populate.
14. Optionally, next to to *S***et an alarm in case this custom enrichment doesn't receive any data?**, toggle the setting to **YES** to enable an alarm.
    * Fill in the **Number** and **Period** fields to indicate how often Panther should send you this notification.
    * The alert destinations for this alarm are displayed at the bottom of the page. To configure and customize where your notification is sent, see documentation on [Panther Destinations](https://docs.panther.com/alerts/destinations).

​![The image shows a success page in the Panther Console after creating a Lookup Table. There is an option to set an  alarm in case this Lookup Table doesn't receive any data, and dropdown selectors to choose a time period when you receive the alerts. At the bottom, there is a section showing which destinations will receive the alert.](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2FoyqgS5R8TnNzpDRtl9e9%2Fimage.png?alt=media\&token=5d8c27d1-ac28-460c-a2f5-bd2c503a49f1)​

{% hint style="info" %}
Notifications generated for a custom enrichment upload failing are accessible in the **System Errors** tab within the **Alerts & Errors** page in the Panther Console.
{% endhint %}

**Creating an IAM role**

There are three options for creating an IAM Role to use with your Panther custom enrichment using an S3 source:

* [Create an IAM role using AWS Console UI.](#create-an-iam-role-using-aws-console-ui)
* [Create an IAM role using CloudFormation Template File](#create-an-iam-role-using-aws-console-ui).
* [Create an IAM role manually](#create-an-iam-role-manually).

<img src="https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-078d0498ee3d122d2361683f131f70af713240dd%2Fnew-iam-role.png?alt=media" alt="" data-size="original">

**Create an IAM role using AWS Console UI**

1. On the "Set Up an IAM role" page, during the process of creating a custom enrichment with an S3 source, locate the tile labeled "Using the AWS Console UI". On the right side of the tile, click **Select**.
2. Click **Launch Console UI**.\
   ![The image shows the screen after you choose to use AWS UI to set up your role. There is a green button in the center labeled "Launch Coonsole UI." At the bottom, there is a field to enter your Role ARN.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-643baf573c6f0018c463d9792e61d584e75a5faa%2Faws-ui-role.png?alt=media)
   * You will be redirected to the AWS console in a new browser tab, with the template URL pre-filled.
   * The CloudFormation stack will create an AWS IAM role with the minimum required permissions to read objects from your S3 bucket.
   * Click the "Outputs" tab of the CloudFormation stack in AWS, and note the Role ARN.
3. Navigate back to your Panther account.
4. On the "Use AWS UI to set up your role" page, enter the Role ARN.
5. Click **Finish Setup**.

**Create an IAM role using CloudFormation Template File**

1. On the "Set Up an IAM role" page, during the process of creating a custom enrichment with an S3 source, locate the tile labeled "CloudFormation Template File". On the right side of the tile, click **Select**.
2. Click **CloudFormation template**, which downloads the template to apply it through your own pipeline.
3. Upload the template file in AWS:
   1. Open your AWS console and navigate to the CloudFormation product.
   2. Click **Create stack**.
   3. Click **Upload a template file** and select the CloudFormation template you downloaded.
4. On the "CloudFormation Template" page in Panther, enter the Role ARN.
5. Click **Finish Setup**.

**Create an IAM role manually**

1. On the "Set Up an IAM role" page, during the process of creating a custom enrichment with an S3 source, click the link that says **I want to set everything up on my own**.
2. Create the required IAM role. You may create the required IAM role manually or through your own automation. The role must be named using the format `PantherLUTsRole-${Suffix}`(e.g., `PantherLUTsRole-MyLookupTable`).
   * The IAM role policy must include the statements defined below:

     ```
         "Version": "2012-10-17",
         "Statement": [
             {
                 "Action": "s3:GetBucketLocation",
                 "Resource": "arn:aws:s3:::<bucket-name>",
                 "Effect": "Allow"
             },
             {
                 "Action": "s3:GetObject",
                 "Resource": "arn:aws:s3:::<bucket-name>/<input-file-path>",
                 "Effect": "Allow"
             }
         ]
     }
     ```
   * If your S3 bucket is configured with server-side encryption using AWS KMS, you must include an additional statement granting the Panther API access to the corresponding KMS key. In this case, the policy will look something like this:

     ```
         "Version": "2012-10-17",
         "Statement": [
             {
                 "Action": "s3:GetBucketLocation",
                 "Resource": "arn:aws:s3:::<bucket-name>",
                 "Effect": "Allow"
             },
             {
                 "Action": "s3:GetObject",
                 "Resource": "arn:aws:s3:::<bucket-name>/<input-file-path>",
                 "Effect": "Allow"
             },
             {
                 "Action": ["kms:Decrypt", "kms:DescribeKey"],
                 "Resource": "arn:aws:kms:<region>:<your-accound-id>:key/<kms-key-id>",
                 "Effect": "Allow"
             }
         ]
     }
     ```
3. On the "Setting up role manually" page in Panther, enter the Role ARN.
   * This can be found in the "Outputs" tab of the CloudFormation stack in your AWS account.
4. Click **Finish Setup**, and you will be redirected to the Enrichments list page with your new Employee Directory table listed.

</details>

<details>

<summary>PAT</summary>

**Sync custom enrichment data from an S3 bucket through PAT**

**File and folder setup**

A custom enrichment requires the following files:

* A YAML configuration file for the enrichment table
  * This custom enrichment configuration file must be stored in a folder with a name containing `lookup_tables`*.* This could be a top-level `lookup_tables` directory, or sub-directories with names matching `*lookup_tables*`. You can use the [panther-analysis](https://github.com/panther-labs/panther-analysis) repository as a reference.
* A YAML file defining the schema to use when loading data into the table
  * This data schema file must be stored in a directory outside the `lookup_tables` directory (for example, `/schemas` in the root of your panther analysis repo). You might choose to save it with the rest of your custom log schemas.
* A JSON or CSV file containing data to load into the table (optional, read further).

**Writing the schema and configuration files**

It's usually prudent to begin writing the data schema first, because the table configuration will reference some of those values.

1. Create the YAML data schema file. This schema defines how to read the files you'll use to upload data to the table. If using a CSV file for data, then the schema should be able to parse CSV.
   * The table schema is formatted the same as a log schema. For more information on writing schemas, read our documentation around [managing Log Schemas.](https://docs.panther.com/data-onboarding/custom-log-types#uploading-log-schemas-with-the-panther-analysis-tool)
2. Create a YAML file for the table configuration. For a custom enrichment with data stored in a file in S3, an example configuration would look like this:
   * Note that the `Refresh` field contains `RoleARN`, `ObjectPath`, and `PeriodMinutes` fields.
   * See a full list of required and allowed values on [Custom Enrichment Specification Reference](https://docs.panther.com/enrichment/custom/lookup-table-specification-reference).

     ```yaml
     AnalysisType: lookup_table
     LookupName: my_lookup_table # A unique display name
     Schema: Custom.MyTableSchema # The schema defined in the previous step
     Refresh:
       RoleArn: arn:aws:iam::123456789012:role/PantherLUTsRole-my_lookup_table # A role in your organization's AWS account
       ObjectPath: s3://path/to/my_lookup_table_data.csv
       PeriodMinutes: 120 # Sync from S3 every 2 hours
     Description: >
       A handy description of what information this table contains.
       For example, this table might convert IP addresses to hostnames
     Reference: >
       A URL to some additional documentation around this table
     Enabled: true # Set to false to stop using the table
     LogTypeMap:
       PrimaryKey: ip                # The primary key of the table
       AssociatedLogTypes:           # A list of log types to match this table to
         - LogType: AWS.CloudTrail
           Selectors:
             - "sourceIPAddress"     # A field in CloudTrail logs
             - "p_any_ip_addresses"  # A panther-generated field works too
         - LogType: Okta.SystemLog
           Selectors:
             - "$.client.ipAddress"  # Paths to JSON values are allowed
     ```
3. From the root of the repository, upload the schema file by running [`panther_analysis_tool update-custom-schemas`](https://docs.panther.com/panther-developer-workflows/detections-repo/pat/pat-commands#update-custom-schemas-creating-or-updating-custom-schemas) `--path ./schemas`.
4. From the root of the repository, upload the custom enrichment using\
   [`panther_analysis_tool upload`](https://docs.panther.com/panther-developer-workflows/detections-repo/pat/pat-commands#upload-uploading-packages-to-panther-directly).

**Prerequisites**

Before you can configure your custom enrichment to sync with S3, you'll need to have the following ready:

1. The ARN of an IAM role in AWS, which Panther can use to access the S3 bucket. For more information on setting up an IAM role for Panther, see the section on [Creating an IAM Role.](#creating-an-iam-role)
2. The path to the file you intend to store data in. The path should be of the following format: `s3://bucket-name/path_to_file/file.csv`

</details>

### Option 4: Sync custom enrichment data from a Google Cloud Storage (GCS) bucket

You can set up data sync from a GCS bucket through the Panther Console or PAT:

<details>

<summary>Panther Console</summary>

**Sync custom enrichment data from a GCS bucket through the Panther Console**

1. In the left-hand navigation bar in your Panther Console, click **Configure > Enrichments**.
2. In the upper-right corner, click **Create New**.
3. Click the **Custom Enrichment** card.
4. On the **Basic Information** page, fill in the fields:

   * **Enrichment Name**: A descriptive name for your custom enrichment.
   * **Enabled?**: Ensure this toggle is set to `YES`. This is required to import your data later in this process.
   * **Description - Optional**: Additional context about the table.
   * **Reference - Optional**: Typically used for a hyperlink to an internal resource.

   ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-305b9013173d5e5ad60d7d98ca2eb6c24eccfcb8%2Fimage%20\(1\)%20\(13\).png?alt=media)
5. Click **Continue**.
6. On the **Associated Log Types (Optional)** page, optionally designate log types/Selectors:
   * Click **Add Log Type**.
   * Click the **Log Type** dropdown, then select a log type.
   * Choose one or more **Selectors**, the foreign key fields form the log type you want enriched with your custom enrichment data.
     * You also can reference attributes in nested objects using [JSON path syntax](https://goessner.net/articles/JsonPath/). For example, if you wanted to reference a field in a map, you could enter `$.field.subfield`.
   * Click **Add Log Type** to add another if needed.\
     ![The image shows the Associated Log Types form from the Panther Console. The Log Type field is set to AWS.VPCFlow. Selectors is set to "account."](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-baa85a56b3740e124717f110214bcb2a3965a4fa%2Flut-logtype-s3.png?alt=media)\
     In the example screen shot above, we selected **AWS.VPCFlow** logs and typed in `account` to represent keys in the VPC Flow logs.
7. Click **Continue**.
8. On the **Table Schema** page, configure the Table Schema:\
   \&#xNAN;***Note:** If you have not already created a new schema, please see* [*our documentation on creating schemas*](https://docs.runpanther.io/data-onboarding/custom-log-types/example-csv)*. Once you have created a schema, you will be able to select it from the dropdown on the Table Schema page while configuring a custom enrichment.*
   1. Select a **Schema Name** from the dropdown.
   2. Select a **Primary Key Name** from the dropdown. This should be a unique column on the table, such as `accountID`.
9. Click **Continue**.
10. On the **Choose Import Method** page, on the **Sync Data from a Google Cloud Storage Bucket** tile, click **Set Up**.\
    ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-792562953908721ea527434e39a0100dfce7bc1d%2FScreenshot%202025-06-23%20at%2015.59.31.png?alt=media)
11. Set up your Google Cloud Storage source. Note that your data must be in `.csv` or `.json` format.
    * Enter the **Google Cloud Storage URI**, the unique path that identifies the specific object.
    * Enter the **Update Period**, the cadence your S3 source gets updated (defaulted to 1 hour).\
      ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-f6ab127d030dbdf1cf6bf2524d73fe716b6e3e8c%2FScreenshot%202025-06-23%20at%2016.03.39.png?alt=media)
12. Click **Continue**.
13. Set up an identity.
    * Please see the next section, Set up an identity, for instructions on how to do this.
14. Click **Finish Setup**. A source setup success page will populate.
15. Optionally, next to to **Set an alarm in case this custom enrichment doesn't receive any data?**, toggle the setting to **YES** to enable an alarm.

    * Fill in the **Number** and **Period** fields to indicate how often Panther should send you this notification.
    * The alert destinations for this alarm are displayed at the bottom of the page. To configure and customize where your notification is sent, see documentation on [Panther Destinations](https://docs.panther.com/alerts/destinations).

    ![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-6895ea89e9a54421ed762f5a5e756d492189144b%2FScreenshot%202025-06-23%20at%2016.08.09.png?alt=media)​

{% hint style="info" %}
Notifications generated for a custom enrichment upload failing are accessible in the **System Errors** tab within the **Alerts & Errors** page in the Panther Console.
{% endhint %}

**Set up an identity**

The identity method supported is [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation). There are three options for setting up Workflow Identity Federation to use with your Panther custom enrichment using a GCS source:

* Set up Workload Identity Federation using Terraform template file
* Set up Workload Identity Federation manually in GCP console

![](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-d2551e2407e97e12a9cbf4c6e3dbb27df8548086%2FScreenshot%202025-06-23%20at%2016.18.06.png?alt=media)

**Set up Workload Identity Federation using Terraform template file**

1. On the **Setup an Identity** page, during the process of creating a custom enrichment with a GCS source, locate the tile labeled "Terraform Template File". On the right side of the tile, click **Select**.
2. Click **Terraform template**, which downloads the template to apply it through your own pipeline.
3. Fill out the fields in the `panther.tfvars` file with your configuration.
   * Provide values for `panther_workload_identity_pool_id`, `panther_workload_identity_pool_provider_id`**,** and `panther_aws_account_id`.
   * Follow the instructions in the file regarding whether you want to create a bucket along with the rest of the resources or if you already have one.
4. Initialize a working directory containing Terraform configuration files and run `terraform init`.
5. Copy the corresponding **Terraform Command** provided and run it in your CLI.
6. Generate a credential configuration file for the pool by copying the **gcloud Command** provided, replacing the value for the project number, pool ID, and provider ID, and running it in your CLI.
   * You can find the project number, the pool ID and the provider ID in the output of the **Terraform Command**.
7. Under **Provide JSON file**, upload your credential configuration file.
8. Click **Finish Setup**

**Set up Workload Identity Federation manually in GCP console**

1. In your Google Cloud console, determine which bucket Panther will pull logs from.
   * If you have not created a bucket yet, please see [Google's documentation on creating a bucket](https://cloud.google.com/storage/docs/creating-buckets).

{% hint style="warning" %}
[Uniform bucket-level access](https://cloud.google.com/storage/docs/uniform-bucket-level-access) must be enabled on the target bucket in order to grant Workload Identity Federation entities access to cloud storage resources.
{% endhint %}

2. [Enable the IAM API](https://console.cloud.google.com/apis/library/iam.googleapis.com).
3. Configure Workload Identity Federation with AWS by following the [Configure Workload Identity Federation with AWS or Azure](https://cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds) documentation.
   1. As you are [defining an attribute mapping(s) and condition](https://cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds#mappings-and-conditions), take note of the following examples:
      * Example [attribute mappings](https://cloud.google.com/iam/docs/workload-identity-federation#mapping):

        <table><thead><tr><th width="195.8271484375">Google</th><th width="523.1220703125">AWS</th></tr></thead><tbody><tr><td><code>google.subject</code></td><td><code>assertion.arn.extract('arn:aws:sts::{account_id}:')+":"+assertion.arn.extract('assumed-role/{role_and_session}').extract('/{session}')</code></td></tr><tr><td><code>attribute.account</code></td><td><code>assertion.account</code></td></tr></tbody></table>
      * Example [attribute condition](https://cloud.google.com/iam/docs/workload-identity-federation#conditions):\
        `attribute.account=="<PANTHER_AWS_ACCOUNT_ID>"`
   2. When you are [adding a provider to your identity pool](https://cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds#aws), select **AWS**.

{% hint style="warning" %}
The value of the `google.subject` attribute [cannot exceed 127 characters](https://cloud.google.com/iam/docs/workload-identity-federation#mapping). You may use [Common Expression Language (CEL) expressions](https://cloud.google.com/iam/docs/workload-identity-federation#mapping) to transform or combine attributes from the token issued by AWS. The expression suggested in the table above takes this limit into account, and is an attempt at transforming the ARN into a value that uniquely identifies Panther entities. For more information on the AWS attributes, see "Example 2 - Called by user created with AssumeRole" on [this AWS documentation page](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html).
{% endhint %}

4. Assign the required IAM roles to the account.

* The following permissions are required for the project where the Pub/Sub subscription and topic lives:

  <table data-header-hidden><thead><tr><th width="327.374982940047" align="center">Permission required</th><th width="294.15662026309724" align="center">Role</th><th width="208" align="center">Condition</th></tr></thead><tbody><tr><td align="center"><strong>Permissions required</strong></td><td align="center"><strong>Role</strong></td><td align="center"><strong>Scope</strong></td></tr><tr><td align="center"><p><code>storage.objects.get</code></p><p><code>storage.objects.list</code></p></td><td align="center"><code>roles/storage.objectViewer</code></td><td align="center"><em>bucket-name</em></td></tr></tbody></table>

  * **Note:** You can set conditions or IAM policies on permissions for specific resources. This can be done either in the IAM section in GCP (as seen in the example screenshot below) or in the specific resource's page.\
    ![In the Google Cloud console, the "IAM" navigation bar item is circled. In a slide-out panel, sections titled "Add principals" and "Assign roles" are circled.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-fde2bbdd808d22e67b3f2409a908d7a63596593b%2FScreenshot%202025-03-05%20at%2015.20.04.png?alt=media)
  * **Note:** You can create the permissions using the `gcloud` CLI tool, where the `$PRINCIPAL_ID` may be something like:\
    `principalSet://iam.googleapis.com/projects/<THE_ACTUAL_GOOGLE_PROJECT_NUMBER>/locations/global/workloadIdentityPools/<THE_ACTUAL_POOL_ID>/attribute.account/<THE_ACTUAL_PANTHER_AWS_ACCOUNT_ID>`
    * `gcloud projects add-iam-policy-binding $PROJECT_ID --member="$PRINCIPAL_ID" --role="roles/storage.objectViewer"`

5. [Download the credential configuration file](https://cloud.google.com/iam/docs/workload-download-cred-and-grant-access), which will be used in Panther to authenticate to the GCP infrastructure.

* To generate a credential configuration file using the gcloud CLI tool, use the following command format:\
  `gcloud iam workload-identity-pools create-cred-config projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$POOL_ID/providers/$PROVIDER_ID --aws --output-file=config.json`

</details>

<details>

<summary>PAT</summary>

**Sync custom enrichment data from a GCS bucket through PAT**

**File and folder setup**

A custom enrichment requires the following files:

* A YAML configuration file for the enrichment table
  * This custom enrichment configuration file must be stored in a folder with a name containing `lookup_tables`*.* This could be a top-level `lookup_tables` directory, or sub-directories with names matching `*lookup_tables*`. You can use the [panther-analysis](https://github.com/panther-labs/panther-analysis) repository as a reference.
* A YAML file defining the schema to use when loading data into the table
  * This data schema file must be stored in a directory outside the `lookup_tables` directory. You might choose to save it with your other custom log schemas, e.g., in a `/schemas` directory at the root of your panther-analysis repository.
* A JSON or CSV file containing data to load into the table (optional, read further).

**Writing the schema and configuration files**

It's usually prudent to begin writing the data schema first, because the table configuration will reference some of those values.

1. Create the YAML data schema file. This schema defines how to read the files you'll use to upload data to the table. If using a CSV file for data, then the schema should be able to parse CSV.
   * The table schema is formatted the same as a log schema. For more information on writing schemas, read our documentation around [managing Log Schemas.](https://docs.panther.com/data-onboarding/custom-log-types#uploading-log-schemas-with-the-panther-analysis-tool)
2. Create a YAML file for the table configuration. For a custom enrichment with data stored in a file in GCS, see the below example file.
   * Note that the `Refresh` field contains `GCSCredentials`, `StorageProvider`, `ObjectPath`, and `PeriodMinutes` fields.
   * See a full list of required and allowed values on [Custom Enrichment Specification Reference](https://docs.panther.com/enrichment/custom/lookup-table-specification-reference).

     ```yaml
     AnalysisType: lookup_table
     LookupName: my_lookup_table # A unique display name
     Schema: Custom.MyTableSchema # The schema defined in the previous step
     Refresh:
       ObjectPath: gs://path/to/my_lookup_table_data.csv
       GCSCredentials: '{"universe_domain":"googleapis.com","type":"external_account","audience":"//iam.googleapis.com/projects/123456789012/locations/global/workloadIdentityPools/mypool/providers/myprovider","subject_token_type":"urn:ietf:params:aws:token-type:aws4_request","token_url":"https://sts.googleapis.com/v1/token","credential_source":{"environment_id":"aws1","region_url":"http://169.254.169.254/latest/meta-data/placement/availability-zone","url":"http://169.254.169.254/latest/meta-data/iam/security-credentials","regional_cred_verification_url":"https://sts.{region}.amazonaws.com?Action=GetCallerIdentity&Version=2011-06-15"},"token_info_url":"https://sts.googleapis.com/v1/introspect"}'
       StorageProvider: GCS
       PeriodMinutes: 120 # Sync from GCS every 2 hours
     Description: >
       A handy description of what information this table contains.
       For example, this table might convert IP addresses to hostnames
     Reference: >
       A URL to some additional documentation around this table
     Enabled: true # Set to false to stop using the table
     LogTypeMap:
       PrimaryKey: ip                # The primary key of the table
       AssociatedLogTypes:           # A list of log types to match this table to
         - LogType: AWS.CloudTrail
           Selectors:
             - "sourceIPAddress"     # A field in CloudTrail logs
             - "p_any_ip_addresses"  # A panther-generated field works too
         - LogType: Okta.SystemLog
           Selectors:
             - "$.client.ipAddress"  # Paths to JSON values are allowed
     ```
3. From the root of the repository, upload the schema file by running [`panther_analysis_tool update-custom-schemas`](https://docs.panther.com/panther-developer-workflows/detections-repo/pat/pat-commands#update-custom-schemas-creating-or-updating-custom-schemas) `--path ./schemas`.
4. From the root of the repository, upload the custom enrichment using\
   [`panther_analysis_tool upload`](https://docs.panther.com/panther-developer-workflows/detections-repo/pat/pat-commands#upload-uploading-packages-to-panther-directly).

**Prerequisites**

Before you can configure your custom enrichment to sync with a GGS bucket, you'll need to have the following ready:

* The JSON credential configuration for a workload identity pool, which Panther can use to access the Google Cloud Storage object. For more information on setting up Workload Identity Federation for Panther, see the **Set up an identity** section in the **Panther Console** instructions above.
* The path to the file you intend to store data in. The path should be of the following format: `gs://bucket-name/path_to_file/file.csv`

</details>

## Writing a detection using custom enrichment data

After you configure a custom enrichment, you can write detections based on the additional context.

For example, if you configured a custom enrichment to distinguish between developer and production accounts in AWS CloudTrail logs, you might want receive an alert only if the following circumstances are both true:

* A user logged in who did not have MFA enabled.
* The AWS account is a production (not a developer) account.

See how to create a detection using enrichment data below:

{% tabs %}
{% tab title="Python" %}
**Accessing enrichment data the event was automatically enriched with**

In Python, you can use the [`deep_get()` helper function](https://docs.panther.com/detections/rules/python/globals#deep_get) to retrieve the looked up field from `p_enrichment` using the foreign key field in the log. The pattern looks like this:

```python
deep_get(event, 'p_enrichment', <Enrichment name>, <foreign key in log>, <field in Enrichment>)
```

{% hint style="info" %}
The custom enrichment name, foreign key and field name are all optional parameters. If not specified, `deep_get()` will return a hierarchical dictionary with all the enrichment data available. Specifying the parameters will ensure that only the data you care about is returned.
{% endhint %}

The rule would become:

```python
from panther_base_helpers import deep_get
 def rule(event):
   is_production = deep_get(event, 'p_enrichment', 'account_metadata',
'recipientAccountId', 'isProduction') # If the field you're accessing is stored within a list, use deep_walk() instead
   return not event.get('mfaEnabled') and is_production
```

***

**Dynamically accessing enrichment data**

You can also use the [event object's `lookup()` function](https://docs.panther.com/detections/rules/python#lookup) to dynamically access enrichment data in your detection. This may be useful when your event doesn't contain an exact match to a value in the enrichment's primary key column.
{% endtab %}

{% tab title="Simple Detection" %}
In a [Simple Detection](https://docs.panther.com/detections/rules/writing-simple-detections), you can create an [Enrichment match expression](https://docs.panther.com/detections/rules/writing-simple-detections/match-expression#enrichment-match-expressions).

```yaml
Detection:
  - Enrichment:
      Table: account_metadata
      Selector: recipientAccountId
      FieldPath: isProduction
    Condition: Equals
    Value: true
  - KeyPath: mfaEnabled
    Condition: Equals
    Value: false
```

{% endtab %}
{% endtabs %}

The Panther rules engine will take the looked up matches and append that data to the event using the key `p_enrichment` in the following JSON structure:

```json
{ 
    "p_enrichment": {
        <name of enrichment table>: { 
            <key in log that matched>: <matching row looked up>,
            ...
	    <key in log that matched>: <matching row looked up>,
	}    
    }
} 
```

Example:

```json
 {
  "p_enrichment": {
      "account_metadata": {
          "recipientAccountId": {
              "accountID": "90123456", 
              "isProduction": false, 
              "email": "dev.account@example.com",
              "p_match": "90123456"
              }
          }
      }
}

```

If the value of the matching log key is an array (e.g., the value of `p_any_aws_accout_ids`), then the lookup data is an array containing the matching records.

```json
{ 
    "p_enrichment": {
        <name of enrichment table>: { 
            <key in log that matched that is an array>: [
                <matching row looked up>,
                <matching row looked up>,
                <matching row looked up>
            ]
	}
     }
} 
```

Example:

```json
 {
  "p_enrichment": {
      "account_metadata": {
          "p_any_aws_account_ids": [
             {
              "accountID": "90123456", 
              "isProduction": false, 
              "email": "dev.account@example.com",
              "p_match": "90123456"
              },
              {
              "accountID": "12345678", 
              "isProduction": true, 
              "email": "prod.account@example.com",
              "p_match": "12345678"
              }
          ]
      }
  }
}
```

### Testing detections that use enrichment

For rules that use `p_enrichment`, click **Enrich Test Data** in the upper right side of the JSON code editor to populate it with your Enrichment data. This allows you to test a Python function with an event that contains `p_enrichment`.

{% hint style="info" %}
In order for your unit test to enrich properly, your event must specify the following two fields:

* `p_log_type`: This determines which Enrichments to use
* The selector field: This provides a value to match against
  {% endhint %}

<figure><img src="https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-313f3e4ff14db84cfcdb889dea2b17b72d4213c8%2Funit-test-enrichment-example?alt=media" alt="A &#x22;Unit Test&#x22; header is above a code block with JSON. The JSON includes various key/value pairs—for example, &#x22;p_log_type&#x22;: &#x22;My.Log.Type&#x22;"><figcaption></figcaption></figure>
