# Managing Enrichment Providers with the Panther Analysis Tool

## Overview

[Enrichment](https://docs.panther.com/enrichment) sources (also called Lookup Tables) allow you to add more context to your incoming logs. You can manage schemas and mappings for your [custom enrichments](https://docs.panther.com/enrichment/custom) and certain Panther-managed enrichments through the [Panther Analysis Tool (PAT)](https://docs.panther.com/panther-developer-workflows/detections-repo/pat).

This guide will walk you through the following:

* Creating and uploading a custom schema for a custom enrichment source using the [`pantherlog` tool](https://docs.panther.com/panther-developer-workflows/pantherlog).
* Modifying the `Selectors` and `LogTypes` in the enrichment source's YAML configuration file.
  * Note that log types and Selectors can [be set manually](https://docs.panther.com/enrichment/custom#option-1-manually-choose-log-types-and-selectors) or [automatically mapped by indicator fields](https://docs.panther.com/enrichment/custom#option-2-let-log-types-and-selectors-be-automatically-mapped-by-indicator-fields).
* Uploading the enrichment source's YAML configuration file via PAT.
* Testing the enrichment in the Panther Console.

{% hint style="info" %}

* If your team uses CLI workflows, it's recommended to use PAT and CI/CD to manage your enrichment, instead of doing so via Detection Packs in the Console.
* If you choose to manage enrichment tables through PAT after enabling them in the Panther Console, you must first disable the Detection Packs in the Panther Console. Simultaneous use of both the Panther Console and PAT to manage enrichment sources is not supported.
  {% endhint %}

{% hint style="warning" %}
This guide applies to [custom enrichments](https://docs.panther.com/enrichment#custom-enrichments) and [these Panther-managed enrichment sources](https://docs.panther.com/enrichment#additional-enrichment-sources).

["Bring your own API key" log pullers](https://docs.panther.com/enrichment#bring-your-own-api-key-log-pullers) and [Panther log source pullers](https://docs.panther.com/enrichment#panther-log-source-pullers) cannot be enabled in the CLI workflow using PAT.
{% endhint %}

### Custom enrichments vs. Panther-managed enrichments

* [Custom enrichments](https://docs.panther.com/enrichment/custom) are managed by you. You'll need to create and upload a schema, then upload the enrichment table's YAML configuration file.
* Panther-managed enrichment providers are managed by Panther. Their schemas are Panther-defined, and their YAML configuration files (which you can modify to your needs) can be found in the [panther-analysis repo](https://github.com/panther-labs/panther-analysis/blob/master/lookup_tables/) on GitHub.

## How to manage custom and Panther-managed enrichments with PAT

{% tabs %}
{% tab title="Custom enrichments" %}
**Prerequisites**

* A YAML configuration file. You must create the YAML configuration file yourself.
* A data sample (if you need to create a new schema) or an existing YAML schema created in Panther.

**Step 1: Create and upload a schema**

Custom enrichments must be associated with a schema you have created and uploaded to Panther. If you have already created a schema in Panther that you want associate to your custom enrichment, you can skip this step.

1. Create the schema using your sample log data.
   * You can use `pantherlog` to infer a schema from a sample set of data. To generate a schema from a sample JSON log file, use the `infer` command:

     ```bash
     $ ./pantherlog infer sample_logs.jsonl > schema.yml
     ```
   * Remember to review the inferred schema and make any necessary adjustments before uploading it to Panther. For more information about this process, see the [pantherlog documentation](https://docs.panther.com/panther-developer-workflows/pantherlog).
2. Upload the schema.
   * Once you have created the schema, you can upload it to Panther by following the [Uploading log schemas with the Panther Analysis Tool](https://docs.panther.com/data-onboarding/custom-log-types#uploading-log-schemas-with-the-panther-analysis-tool) instructions.

**Step 2: Create the YAML configuration file**

* For custom enrichments, you must create the YAML configuration file from scratch. Reference the [Lookup Table Specification Reference](https://docs.panther.com/enrichment/custom/lookup-table-specification-reference) to see which keys this file must include.

**Step 3: Upload the custom enrichment via PAT**

Once you have created your custom enrichment configuration file, you can upload it to Panther using the PAT [`upload` command](https://docs.panther.com/panther-developer-workflows/detections-repo/pat-commands#upload-uploading-packages-to-panther-directly):

```bash
panther_analysis_tool upload
```

You will need to provide an API token and host with `--api-token` and `--api-host`, respectively, for the upload to occur. Other options include filtering, minimum tests, and more.

{% hint style="warning" %}
Ensure you've uploaded the corresponding schema before uploading the YAML configuration file.
{% endhint %}

**Step 4: Test the custom enrichment**

There are several methods to test if your custom enrichment has been set up correctly.

**Method 1: Enriching test data in the Panther Console or CLI**

{% tabs %}
{% tab title="Console" %}
In the Panther Console's detection editor, click **Enrich Test Data** to verify if your custom enrichment is working correctly. This allows you to input test data and see the output of the enrichment process within your unit test.

{% hint style="warning" %}
For **Enrich Test Data** to work, the unit test must have a `p_log_type` identifying the correct log type. This serves as the basis for Panther's enrichment logic.
{% endhint %}
{% endtab %}

{% tab title="CLI" %}

* Verify your custom enrichment is working properly by using PAT's `enrich-test-data` command. [Learn more about `enrich-test-data` here](https://docs.panther.com/panther-developer-workflows/detections-repo/pat-commands#enrich-test-data-enriching-test-data-with-enrichment-content).
  {% endtab %}
  {% endtabs %}

**Method 2: Checking the `panther_signals` database**

You can verify that your changes have taken effect by checking the `panther_signals.public.correlation_signals` database/table for the `p_enrichment` field. Ensure that the field includes the custom enrichment details you would expect to see.

**Method 3: Using SQL queries**

You can also perform a `LEFT JOIN` between event logs and the enrichment table in SQL. Ensure that the selector is defined in the query. This allows you to verify if the data from your logs is being correctly matched with the data in your custom enrichment.

For example, this query will attempt to match event data to the custom enrichment using a custom selector (which should be the same as the selector you've defined in the YAML configuration file):

```sql
SELECT *
FROM panther_logs.public.<log_type> AS e
LEFT JOIN panther_lookups.public.<lookup_table_name> AS lt
ON e.<field_path> = lt.<field_path>
WHERE e.p_occurs_since('1 day')
```

{% endtab %}

{% tab title="Panther-managed enrichments" %}
**Prerequisite**

* A YAML configuration file. You can use the [Panther-provided configuration files in panther-analysis](https://github.com/panther-labs/panther-analysis/tree/master/lookup_tables).

**Step 1: Modify the YAML configuration file as needed**

If you are enabling a Panther-managed enrichment, you can modify the configuration file that [Panther provides](https://github.com/panther-labs/panther-analysis/tree/master/lookup_tables) to meet your needs.

* When modifying the Panther-provided YAML configuration file for enrichments, you should only ever modify the contents of the `AssociatedLogTypes` key in order to customize the `Selectors`. Alterations of other parameters such as `Refresh` intervals are known to cause issues.

**Example**

{% hint style="warning" %}
Note than even if this log type (`Cloudflare.Firewall`) and Selector (`ClientIP`) weren't set manually in this fashion, `Cloudflare.Firewall` and `p_any_ip_addresses` would be added as a log type and Selector, respectively, due to the [automatic mapping by indicator fields](https://docs.panther.com/enrichment/custom#option-2-let-log-types-and-selectors-be-automatically-mapped-by-indicator-fields).

This is because `ClientIP` is designated as an `ip` indicator field in the `Cloudflare.Firewall` schema and the Tor Lookup Table's primary key, `ip`, is designated as an `ip` indicator in its own data schema, `Tor.ExitNode`.
{% endhint %}

In this example, the `tor_exit_nodes` enrichment is being updated to include a new `LogType` and `Selector`.

* Note that the value of`PrimaryKey` is `ip`.
* The example below shows one `AssociatedLogTypes` included by default.

```yaml
LogTypeMap:
  PrimaryKey: ip
  AssociatedLogTypes:
    - LogType: AlphaSOC.Alert
      Selectors:
        - '$.event.srcIP'
```

Let's add a list item to `AssociatedLogTypes` that adds support for the `ip_address` field in the `Cloudflare.Firewall` schema:

* Note that the `Selectors` can be parent fields or JSON paths for nested fields.

```yaml
LogTypeMap:
  PrimaryKey: ip
  AssociatedLogTypes:
    - LogType: AlphaSOC.Alert
      Selectors:
        - '$.event.srcIP'
    - LogType: Cloudflare.Firewall
      Selectors:
        - "ClientIP"
```

**Step 2: Upload the enrichment via PAT**

Once you have modified your enrichment configuration file, you can upload it to Panther using the PAT [`upload` command](https://docs.panther.com/panther-developer-workflows/detections-repo/pat-commands#upload-uploading-packages-to-panther-directly):

```bash
panther_analysis_tool upload
```

You will need to provide an API token and host with `--api-token` and `--api-host`, respectively, for the upload to occur. Other options include filtering, minimum tests, and more.

**Step 3: Test the enrichment**

There are several methods to test if your enrichment has been set up correctly.

**Method 1: Enriching test data in the Panther Console or CLI**

{% tabs %}
{% tab title="Console" %}
In the Panther Console's detection editor, click **Enrich Test Data** to verify if your enrichment is working correctly. This allows you to input test data and see the output of the enrichment process within your unit test.

{% hint style="warning" %}
For **Enrich Test Data** to work, the unit test must have a `p_log_type` identifying the correct log type. This serves as the basis for Panther's enrichment logic.
{% endhint %}
{% endtab %}

{% tab title="CLI" %}

* Verify your enrichment is working properly by using PAT's `enrich-test-data` command. [Learn more about `enrich-test-data` here](https://docs.panther.com/panther-developer-workflows/detections-repo/pat-commands#enrich-test-data-enriching-test-data-with-enrichment-content).
  {% endtab %}
  {% endtabs %}

**Method 2: Checking the `panther_signals` database**

You can verify that your changes have taken effect by checking the `panther_signals.public.correlation_signals` database/table for the `p_enrichment` field. Ensure that the field includes the enrichment details you would expect to see.

**Method 3: Using SQL queries**

You can also perform a `LEFT JOIN` between event logs and the enrichment table in SQL. Ensure that the selector is defined in the query. This allows you to verify if the data from your logs is being correctly matched with the data in your enrichment table.

For example, this query will attempt to match event data to the enrichment data using a custom selector (which should be the same as the selector you've defined in the enrichment configuration):

```sql
SELECT *
FROM panther_logs.public.<log_type> AS e
LEFT JOIN panther_lookups.public.<lookup_table_name> AS lt
ON e.<field_path> = lt.<field_path>
WHERE e.p_occurs_since('1 day')
```

{% endtab %}

{% tab title="SQL custom enrichments (Beta)" %}
{% hint style="info" %}
SQL custom enrichments are in open beta starting with Panther version 1.120, and is available to all customers. Please share any bug reports and feature requests with your Panther support team.
{% endhint %}

**Step 1: Create the YAML configuration file**

Create a YAML file with a `Query` field. The schema is auto-generated from query results — no manual schema definition is needed. See the [Custom Enrichment Specification Reference](https://docs.panther.com/enrichment/custom/lookup-table-specification-reference) for all available fields.

**Example:**

```yaml
AnalysisType: lookup_table
LookupName: panther_audit_enrichment
Enabled: true
Description: Enriches alerts with Panther audit activity context.
Indicators:
  - Field: actor
    Indicators: [username]
Query: |
  SELECT
    id,
    actionname,
    actiondescription,
    actionresult,
    actor,
    sourceip,
    useragent,
    timestamp
  FROM panther_logs.public.panther_audit
Refresh:
  PeriodMinutes: 60
LogTypeMap:
  PrimaryKey: id
```

**Step 2: Upload via PAT**

`panther_analysis_tool upload`

As with other commands, make sure you provide `--api-token` and `--api-host` for authentication (or fill in the `.panther_settings.yml` file).\
\
To upload enrichments only:

`panther_analysis_tool upload --filter AnalysisType=lookup_table`

{% hint style="info" %}
The schema is auto-generated from your SQL query results. No separate schema upload is needed.
{% endhint %}

**Step 3: Verify**

* Console: Check the enrichment on the Enrichments page to confirm it was created/updated.
* SQL: Query the enrichment table directly to verify data:

`SELECT * FROM panther_lookups.public.name_of_your_enrichment LIMIT 1`
{% endtab %}
{% endtabs %}