# Managing Lookup Tables and Enrichment Providers with the Panther Analysis Tool

## Overview

You can manage Lookup Table and Enrichment Provider schemas and mappings through the [Panther Analysis Tool (PAT)](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/panther-developer-workflows/ci-cd/deployment-workflows/pat).

This guide will walk you through the following:

* Creating and uploading a custom schema for a Lookup Table using the [`pantherlog` tool](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/panther-developer-workflows/pantherlog).
* Modifying the `Selectors` and `LogTypes` in the Lookup Table/Enrichment Provider YAML configuration file.
* Uploading the Lookup Table/Enrichment Provider YAML configuration file via PAT.
* Testing the enrichment in the Panther Console.

{% hint style="info" %}

* If your team uses Developer Workflows, we recommend using PAT and CI/CD to manage your Enrichment, instead of doing so via Detection Packs in the Console
* If you choose to manage Lookup Tables through PAT after enabling them in the Panther Console, you must first disable the Detection Packs in the Panther Console. Simultaneous use of both the Panther Console and PAT to manage Lookup Tables is not supported.
  {% endhint %}

### Lookup Tables vs. Enrichment Providers

In Panther, there is a distinction between custom Lookup Tables and Enrichment Providers.

* [Lookup Tables](#lookup-tables) are user-defined. You'll need to create and upload a schema, then upload the Lookup Table's YAML configuration file.
* [Enrichment Providers](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/enrichment) are managed by Panther. Their schemas are Panther-defined, and their YAML configuration files (which you can modify to your needs) can be found in the [panther-analysis repo](https://github.com/panther-labs/panther-analysis/blob/master/lookup_tables/) in GitHub.

## How to manage Lookup Tables and Enrichment Providers with PAT

{% tabs %}
{% tab title="Lookup Tables" %}

### Prerequisites

* A YAML configuration file. You must create the YAML configuration file yourself. 
* A data sample (if you need to create a new schema) or an existing YAML schema created in Panther.

### Step 1: Create and upload a schema

Lookup Tables must be associated with a schema you have created and uploaded to Panther. If you have already created a schema in Panther that you want associate to your Lookup Table, you can skip this step.

1. Create the schema using your sample log data.
   * You can use `pantherlog` to infer a schema from a sample set of data. To generate a schema from a sample JSON log file, use the `infer` command:

     ```bash
     $ ./pantherlog infer sample_logs.jsonl > schema.yml
     ```
   * Remember to review the inferred schema and make any necessary adjustments before uploading it to Panther. For more information about this process, see the [pantherlog documentation](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/panther-developer-workflows/pantherlog).
2. Upload the schema.
   * Once you have created the schema, you can upload it to Panther by following the [Uploading log schemas with the Panther Analysis Tool](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/data-onboarding/custom-log-types#uploading-log-schemas-with-the-panther-analysis-tool) instructions.

### Step 2: Create the YAML configuration file

* For custom Lookup Tables, you must create the YAML configuration file from scratch. Reference the [Lookup Table Specification Reference](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/enrichment/lookup-tables/lookup-table-specification-reference) to see which keys this file must include.

### Step 3: Upload the Lookup Table via PAT

Once you have created your custom Lookup Table configuration file, you can upload it to Panther using the Panther Analysis Tool's [upload command](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/panther-developer-workflows/ci-cd/deployment-workflows/pat-commands#upload-uploading-packages-to-panther-directly):

```bash
panther_analysis_tool upload
```

You will need to provide an API token and host with `--api-token` and `--api-host`, respectively, for the upload to occur. Other options include filtering, minimum tests, and more.

{% hint style="warning" %}
Ensure you've uploaded the corresponding schema before uploading the YAML configuration file.
{% endhint %}

### Step 4: Test the Lookup Table

There are several methods to test if your Lookup Table has been set up correctly.

#### Method 1: Using the Panther Console's Enrich Test Data button

In the Panther Console's detection editor, click **Enrich Test Data** to verify if your Lookup Table is working correctly. This allows you to input test data and see the output of the enrichment process within your unit test.

{% hint style="warning" %}
For **Enrich Test Data** to work, the unit test must have a `p_log_type` identifying the correct log type. This serves as the basis for Panther's enrichment logic.
{% endhint %}

#### Method 2: Checking the `panther_rule_matches` database

You can verify that your changes have taken effect by checking the `panther_rule_matches` database for the `p_enrichment` field. Ensure that the field includes the Lookup Table details you would expect to see.

#### Method 3: Using SQL queries

You can also perform a `LEFT JOIN` between event logs and the lookup table in SQL. Ensure that the selector is defined in the query. This allows you to verify if the data from your logs is being correctly matched with the data in your Lookup Table.

For example, this query will attempt to match event data to the Lookup Table using a custom selector (which should be the same as the selector you've defined in the YAML configuration file):

```sql
SELECT *
FROM panther_logs.public.<log_type> AS e
LEFT JOIN panther_lookups.public.<lookup_table_name> AS lt
ON e.<field_path> = lt.<field_path>
WHERE e.p_occurs_since('1 day')
```

{% endtab %}

{% tab title="Enrichment Providers" %}

### Prerequisite

* A YAML configuration file. You can use the [Panther-provided configuration files in panther-analysis](https://github.com/panther-labs/panther-analysis/tree/master/lookup_tables).

### Step 1: Modify the YAML configuration file as needed

If you are enabling a Panther-managed Enrichment Provider, you can modify the configuration file that [Panther provides](https://github.com/panther-labs/panther-analysis/tree/master/lookup_tables) to meet your needs.

* When modifying the Panther-provided YAML configuration file for Enrichment Providers, you should only ever modify the contents of the `AssociatedLogTypes` key in order to customize the `Selectors`. Alterations of other parameters such as `Refresh` intervals are known to cause issues.

#### Example

In this example, the `tor_exit_nodes` Enrichment Provider is being updated to include a `Selector` that correlates to a custom schema.

* Note that the value of`PrimaryKey` is `ip`.
* The example below shows one `AssociatedLogTypes` included by default.

```yaml
LogTypeMap:
  PrimaryKey: ip
  AssociatedLogTypes:
    - LogType: AlphaSOC.Alert
      Selectors:
        - '$.event.srcIP'
```

Let's add a list item to `AssociatedLogTypes` that adds support for the `ip_address` field in the `Cloudflare.Firewall` schema:

* Note that the `Selectors` can be parent fields or JSON paths for nested fields.

```yaml
LogTypeMap:
  PrimaryKey: ip
  AssociatedLogTypes:
    - LogType: AlphaSOC.Alert
      Selectors:
        - '$.event.srcIP'
    - LogType: Cloudflare.Firewall
      Selectors:
        - "ip_address"
```

### Step 2: Upload the Enrichment Provider via PAT

Once you have modified your Enrichment Provider configuration file, you can upload it to Panther using the Panther Analysis Tool's [upload command](https://docs.panther.com/~/changes/15ann7vKLltCCAGHtdQr/panther-developer-workflows/ci-cd/deployment-workflows/pat-commands#upload-uploading-packages-to-panther-directly):

```bash
panther_analysis_tool upload
```

You will need to provide an API token and host with `--api-token` and `--api-host`, respectively, for the upload to occur. Other options include filtering, minimum tests, and more.

### Step 3: Test the Enrichment Provider

There are several methods to test if your Enrichment Provider has been set up correctly.

#### Method 1: Using the Panther Console's Enrich Test Data button

In the Panther Console's detection editor, click **Enrich Test Data** to verify if your Enrichment Provider is working correctly. This allows you to input test data and see the output of the enrichment process within your unit test.

{% hint style="warning" %}
For **Enrich Test Data** to work, the unit test must have a `p_log_type` identifying the correct log type. This serves as the basis for Panther's enrichment logic.
{% endhint %}

#### Method 2: Checking the `panther_rule_matches` database

You can verify that your changes have taken effect by checking the `panther_rule_matches` database for the `p_enrichment` field. Ensure that the field includes the LUT or Enrichment Provider details you would expect to see.

#### Method 3: Using SQL queries

You can also perform a `LEFT JOIN` between event logs and the Lookup Table in SQL. Ensure that the selector is defined in the query. This allows you to verify if the data from your logs is being correctly matched with the data in your Enrichment Provider.

For example, this query will attempt to match event data to the Enrichment Provider using a custom selector (which should be the same as the selector you've defined in the Enrichment Provider configuration):

```sql
SELECT *
FROM panther_logs.public.<log_type> AS e
LEFT JOIN panther_lookups.public.<lookup_table_name> AS lt
ON e.<field_path> = lt.<field_path>
WHERE e.p_occurs_since('1 day')
```

{% endtab %}
{% endtabs %}