Custom Lookup Tables
Enrich events with your own stored data
Overview
Custom Lookup Tables (also referred to as simply "Lookup Tables") allow you to store and reference custom enrichment data in Panther. This means you can reference this added context in detections and pass it into alerts. It may be particularly useful to create Lookup Tables containing identity/asset information, vulnerability context, or network maps.
There are three options for Lookup Table data storage: a static file, S3 bucket, or Google Cloud Storage (GCS) bucket.
You can associate one or more log types with your Lookup Table—then all incoming logs of those types (that have a match to a Lookup Table value) will contain enrichment data from your Lookup Table. Learn more about the enrichment process in How incoming logs are enriched. It's also possible to dynamically reference Lookup Table data in Python detections. Learn how to view stored Lookup Table data here, and how to view log events with enrichment data here.
If your lookup data is only needed for a few specific detections and will not be frequently updated, consider using Global helpers instead of a Lookup Table. Also note that there are Panther-managed Lookup Tables, including Enrichment Providers like IPinfo and Tor Exit Nodes, as well as Identity Provider Profiles.
To increase the limit on the number of Lookup Tables or the size of Lookup Tables in your account, please contact your Panther support team.
How incoming logs are enriched
Lookup Tables in Panther traditionally define both of the following:
A primary key: A field in the Lookup Table data.
If the Lookup Table is defined in the CLI workflow, this is designated by the
PrimaryKey
field in the YAML configuration file.
One or more associated log types, each with one or more Selectors: A Selector is an event field whose values are compared to the Lookup Table's primary key values to find a match.
There are two ways to set log types/Selectors for a Lookup Table. See How log types and Selectors are set for a Lookup Table, below.
When a log is ingested into Panther, if its log type is one that is associated to a Lookup Table, the values of all of its Selector fields are compared against the Lookup Table's primary key values. When a match is found between a value in a Selector field and a primary key value, the log is enriched with the matching primary key's associated Lookup Table data in a p_enrichment
field. Learn more about p_enrichment
below, in p_enrichment
structure.
In the example in the image below, the Selector field (in the events in Incoming Logs) is ip_address
. The primary key of the Lookup Table LUT1 is bad_actor_ip
. In the right-hand Alert Event, the log is enriched with the Lookup Table data (including bad_actor_name
) because there was a match between the Selector value (1.1.1.1
) and a primary key value (1.1.1.1
).

How log types and Selectors are set for a Lookup Table
You can manually set associated log types and Selectors when creating a Lookup Table (Option 1), and/or let them be automatically mapped (Option 2).
Option 1: Manually choose log types and Selectors
When creating a Lookup Table, you can choose one or more log types the Lookup Table should be associated to—and for each log type, one or more Selector fields.
When creating a Lookup Table in the Panther Console, you can set associated log types and Selectors. Learn more in How to configure a Lookup Table, below.

Option 2: Let log types and selectors be automatically mapped by indicator fields
This method of associating log types and Selectors to a Lookup Table is not available for Lookup Tables that are populated using file upload. It is only applicable to Lookup Tables that are populated using S3 sync or GCS sync.
If, in the schema for your Lookup Table data, the primary key field is marked as an indicator field (e.g., as an email
and/or username
), for each indicator value, Panther automatically:
Finds all Active log schemas (or log types) that designate any event field as that same indicator.
Associates those log types to the Lookup Table.
For each log type, sets the
p_any
field associated to the indicator as a Selector.
For example, if your Lookup Table data's schema designates an address
field (which has also been set as the primary key) as an ip
indicator, all log types in your Panther instance that also set an ip
indicator will be associated to the Lookup Table, each with a p_any_ip_addresses
Selector.
p_enrichment
structure
p_enrichment
structureIf your log events are injected with enrichment data, a p_enrichment
field is appended to the event and accessed within a detection using deep_get()
or DeepKey
. The p_enrichment
field will contain:
One or more Lookup Table name(s) that matched the incoming log event
The name of the Selector from the incoming log that matched the Lookup Table
The data from the Lookup Table that matched via the Lookup Table's primary key (including an injected
p_match
field containing the Selector value that matched)
This is the structure of p_enrichment
fields:
'p_enrichment': {
<name of lookup table1>: {
<name of selector>: {
'p_match': <value of Selector>,
<lookup key>: <lookup value>,
...
}
}
}
Note that p_enrichment
is not stored with the log event in the data lake. See Viewing log events with enrichment data for more information.
How to access Lookup Table data in detections
Option 1 (if log is enriched): Using deep_get()
deep_get()
If your log event was enriched on ingest (as described in How incoming logs are enriched), you can access the data within the p_enrichment
field (whose structure is described above) using the deep_get()
event object function. Learn more about deep_get()
on Writing Python Detections.
See a full example of this method below, in Writing a detection using Lookup Table data.
Option 2: Dynamically using lookup()
lookup()
It's also possible to dynamically access Lookup Table data from Python detections using the event.lookup()
function. In this way, you can retrieve data from any Lookup Table, without it being injected into an incoming event as described in How incoming logs are enriched.
Prerequisites for configuring a Lookup Table
Before configuring a Lookup Table, be sure you have:
Lookup Table data in JSON or CSV format
JSON files can format events in various ways, including in lines, arrays, or objects.
A schema specifically for your Lookup Table data
This describes the shape of your Lookup Table data.
A primary key for your Lookup Table data
This primary key is one of the fields you defined in your Lookup Table's schema. The value of the primary key is what will be compared with the value of the selector(s) from your incoming logs.
See the below Primary key data types section to learn more about primary key requirements.
(Optional) Selector(s) from your incoming logs
The values from these selectors will be used to search for matches in your Lookup Table data.
(CLI workflow): A Lookup Table configuration file
See Lookup Table Specification Reference for a full list of fields.
We recommend you make a fork of the
panther-analysis
repository and install the Panther Analysis Tool (PAT).
Primary key data types
Your Lookup Table's primary key column must be one of the following data types:
String
Number
Array (of strings or numbers)
Using an array lets you associate one row in your Lookup Table with multiple string or number primary key values. This prevents you from having to duplicate a certain row of data for multiple primary keys.
How to configure a Lookup Table
After fulfilling the prerequisites, Lookup Tables can be created and configured using one of the following methods:
Option 1: Import Lookup Table data via file upload
Best for data that is relatively static, such as information about AWS accounts or corporate subnets.
Example: adding metadata to distinguish developer and production accounts in your AWS CloudTrail logs.
Option 2: Sync Lookup Table data from an S3 bucket or Option 3: Sync Lookup Table data from a Google Cloud Storage (GCS) bucket
Best when you have a large amount of data that updates relatively frequently. Any changes in the S3 or GCS bucket will sync to Panther.
Example: if you wanted to know which groups and permission levels are associated with employees at your company. In this scenario, your company might have an S3 bucket with an up-to-date copy of their Active Directory listing that includes groups and permissions information.
After choosing one of these methods, you can opt to work within the Panther Console or with PAT.
The maximum size for a row in a Lookup Table is 65535 bytes.
Option 1: Import Lookup Table data via file upload
You can import data via file upload through the Panther Console or PAT:
Option 2: Sync Lookup Table data from an S3 bucket
You can set up data sync from an S3 bucket through the Panther Console or PAT:
Option 3: Sync Lookup Table data from a Google Cloud Storage (GCS) bucket
You can set up data sync from a GCS bucket through the Panther Console or PAT:
Writing a detection using Lookup Table data
After you configure a Lookup Table, you can write detections based on the additional context from your Lookup Table.
For example, if you configured a Lookup Table to distinguish between developer and production accounts in AWS CloudTrail logs, you might want receive an alert only if the following circumstances are both true:
A user logged in who did not have MFA enabled.
The AWS account is a production (not a developer) account.
See how to create a detection using Lookup Table data below:
Accessing Lookup Table data the event was automatically enriched with
In Python, you can use the deep_get()
helper function to retrieve the looked up field from p_enrichment
using the foreign key field in the log. The pattern looks like this:
deep_get(event, 'p_enrichment', <Lookup Table name>, <foreign key in log>, <field in Lookup Table>)
The rule would become:
from panther_base_helpers import deep_get
def rule(event):
is_production = deep_get(event, 'p_enrichment', 'account_metadata',
'recipientAccountId', 'isProduction') # If the field you're accessing is stored within a list, use deep_walk() instead
return not event.get('mfaEnabled') and is_production
Dynamically accessing Lookup Table data
You can also use the event object's lookup()
function to dynamically access Lookup Table data in your detection. This may be useful when your event doesn't contain an exact match to a value in the Lookup Table's primary key column.
The Panther rules engine will take the looked up matches and append that data to the event using the key p_enrichment
in the following JSON structure:
{
"p_enrichment": {
<name of lookup table>: {
<key in log that matched>: <matching row looked up>,
...
<key in log that matched>: <matching row looked up>,
}
}
}
Example:
{
"p_enrichment": {
"account_metadata": {
"recipientAccountId": {
"accountID": "90123456",
"isProduction": false,
"email": "[email protected]",
"p_match": "90123456"
}
}
}
}
If the value of the matching log key is an array (e.g., the value of p_any_aws_accout_ids
), then the lookup data is an array containing the matching records.
{
"p_enrichment": {
<name of lookup table>: {
<key in log that matched that is an array>: [
<matching row looked up>,
<matching row looked up>,
<matching row looked up>
]
}
}
}
Example:
{
"p_enrichment": {
"account_metadata": {
"p_any_aws_account_ids": [
{
"accountID": "90123456",
"isProduction": false,
"email": "[email protected]",
"p_match": "90123456"
},
{
"accountID": "12345678",
"isProduction": true,
"email": "[email protected]",
"p_match": "12345678"
}
]
}
}
}
Testing detections that use enrichment
For rules that use p_enrichment
, click Enrich Test Data in the upper right side of the JSON code editor to populate it with your Lookup Table data. This allows you to test a Python function with an event that contains p_enrichment
.

Lookup Table History Tables
Lookup Tables will generate a number of tables in the Data Explorer. There are two main types of tables generated:
The current Lookup Table version:
example
Contains the most up to date Lookup Table data
Should be targeted in any Saved Searches, or anywhere you expect to see the most current data
This table name will never change
In the example above, the table is named
example
The current History Table version:
example_history
Contains a version history of all data uploaded to the current Lookup Table
The table schema is identical to the current Lookup Table (here named
example
) except for two additional fields:p_valid_start
p_valid_end
These fields can be used to view the state of the Lookup Table at any previous point in time
When a new schema is assigned to the Lookup Table, the past versions of the Lookup Table and the History Table are both preserved as well.
These past versions are preserved by the addition of a numeric suffix, _###
and will be present for both the Lookup Table and the History Table. This number will increment by one each time the schema associated with the Lookup Table is replaced, or each time the primary key of the Lookup Table is changed.
Last updated
Was this helpful?