LogoLogo
Knowledge BaseCommunityRelease NotesRequest Demo
  • Overview
  • Quick Start
    • Onboarding Guide
  • Data Sources & Transports
    • Supported Logs
      • 1Password Logs
      • Apache Logs
      • AppOmni Logs
      • Asana Logs
      • Atlassian Logs
      • Auditd Logs
      • Auth0 Logs
      • AWS Logs
        • AWS ALB
        • AWS Aurora
        • AWS CloudFront
        • AWS CloudTrail
        • AWS CloudWatch
        • AWS Config
        • AWS EKS
        • AWS GuardDuty
        • AWS Security Hub
        • Amazon Security Lake
        • AWS S3
        • AWS Transit Gateway
        • AWS VPC
        • AWS WAF
      • Azure Monitor Logs
      • Bitwarden Logs
      • Box Logs
      • Carbon Black Logs
      • Cisco Umbrella Logs
      • Cloudflare Logs
      • CrowdStrike Logs
        • CrowdStrike Falcon Data Replicator
        • CrowdStrike Event Streams
      • Docker Logs
      • Dropbox Logs
      • Duo Security Logs
      • Envoy Logs
      • Fastly Logs
      • Fluentd Logs
      • GCP Logs
      • GitHub Logs
      • GitLab Logs
      • Google Workspace Logs
      • Heroku Logs
      • Jamf Pro Logs
      • Juniper Logs
      • Lacework Logs
        • Lacework Alert Channel Webhook
        • Lacework Export
      • Material Security Logs
      • Microsoft 365 Logs
      • Microsoft Entra ID Audit Logs
      • Microsoft Graph Logs
      • MongoDB Atlas Logs
      • Netskope Logs
      • Nginx Logs
      • Notion Logs
      • Okta Logs
      • OneLogin Logs
      • Orca Security Logs (Beta)
      • Osquery Logs
      • OSSEC Logs
      • Proofpoint Logs
      • Push Security Logs
      • Rapid7 Logs
      • Salesforce Logs
      • SentinelOne Logs
      • Slack Logs
      • Snowflake Audit Logs (Beta)
      • Snyk Logs
      • Sophos Logs
      • Sublime Security Logs
      • Suricata Logs
      • Sysdig Logs
      • Syslog Logs
      • Tailscale Logs
      • Teleport Logs
      • Tenable Vulnerability Management Logs
      • Thinkst Canary Logs
      • Tines Logs
      • Tracebit Logs
      • Windows Event Logs
      • Wiz Logs
      • Zeek Logs
      • Zendesk Logs
      • Zoom Logs
      • Zscaler Logs
        • Zscaler ZIA
        • Zscaler ZPA
    • Custom Logs
      • Log Schema Reference
      • Transformations
      • Script Log Parser (Beta)
      • Fastmatch Log Parser
      • Regex Log Parser
      • CSV Log Parser
    • Data Transports
      • HTTP Source
      • AWS Sources
        • S3 Source
        • CloudWatch Logs Source
        • SQS Source
          • SNS Source
        • EventBridge
      • Google Cloud Sources
        • Cloud Storage (GCS) Source
        • Pub/Sub Source
      • Azure Blob Storage Source
    • Monitoring Log Sources
    • Ingestion Filters
      • Raw Event Filters
      • Normalized Event Filters (Beta)
    • Data Pipeline Tools
      • Chronosphere Onboarding Guide
      • Cribl Onboarding Guide
      • Fluent Bit Onboarding Guide
        • Fluent Bit Configuration Examples
      • Fluentd Onboarding Guide
        • General log forwarding via Fluentd
        • MacOS System Logs to S3 via Fluentd
        • Syslog to S3 via Fluentd
        • Windows Event Logs to S3 via Fluentd (Legacy)
        • GCP Audit to S3 via Fluentd
      • Observo Onboarding Guide
      • Tarsal Onboarding Guide
    • Tech Partner Log Source Integrations
  • Detections
    • Using Panther-managed Detections
      • Detection Packs
    • Rules and Scheduled Rules
      • Writing Python Detections
        • Python Rule Caching
        • Data Models
        • Global Helper Functions
      • Modifying Detections with Inline Filters (Beta)
      • Derived Detections (Beta)
        • Using Derived Detections to Avoid Merge Conflicts
      • Using the Simple Detection Builder
      • Writing Simple Detections
        • Simple Detection Match Expression Reference
        • Simple Detection Error Codes
    • Correlation Rules (Beta)
      • Correlation Rule Reference
    • PyPanther Detections (Beta)
      • Creating PyPanther Detections
      • Registering, Testing, and Uploading PyPanther Detections
      • Managing PyPanther Detections in the Panther Console
      • PyPanther Detections Style Guide
      • pypanther Library Reference
      • Using the pypanther Command Line Tool
    • Signals
    • Policies
    • Testing
      • Data Replay (Beta)
    • Framework Mapping and MITRE ATT&CK® Matrix
  • Cloud Security Scanning
    • Cloud Resource Attributes
      • AWS
        • ACM Certificate
        • CloudFormation Stack
        • CloudWatch Log Group
        • CloudTrail
        • CloudTrail Meta
        • Config Recorder
        • Config Recorder Meta
        • DynamoDB Table
        • EC2 AMI
        • EC2 Instance
        • EC2 Network ACL
        • EC2 SecurityGroup
        • EC2 Volume
        • EC2 VPC
        • ECS Cluster
        • EKS Cluster
        • ELBV2 Application Load Balancer
        • GuardDuty Detector
        • GuardDuty Detector Meta
        • IAM Group
        • IAM Policy
        • IAM Role
        • IAM Root User
        • IAM User
        • KMS Key
        • Lambda Function
        • Password Policy
        • RDS Instance
        • Redshift Cluster
        • Route 53 Domains
        • Route 53 Hosted Zone
        • S3 Bucket
        • WAF Web ACL
  • Alerts & Destinations
    • Alert Destinations
      • Amazon SNS Destination
      • Amazon SQS Destination
      • Asana Destination
      • Blink Ops Destination
      • Custom Webhook Destination
      • Discord Destination
      • GitHub Destination
      • Google Pub/Sub Destination (Beta)
      • Incident.io Destination
      • Jira Cloud Destination
      • Jira Data Center Destination (Beta)
      • Microsoft Teams Destination
      • Mindflow Destination
      • OpsGenie Destination
      • PagerDuty Destination
      • Rapid7 Destination
      • ServiceNow Destination (Custom Webhook)
      • Slack Bot Destination
      • Slack Destination (Webhook)
      • Splunk Destination (Beta)
      • Tines Destination
      • Torq Destination
    • Assigning and Managing Alerts
      • Managing Alerts in Slack
    • Alert Runbooks
      • Panther-managed Policies Runbooks
        • AWS CloudTrail Is Enabled In All Regions
        • AWS CloudTrail Sending To CloudWatch Logs
        • AWS KMS CMK Key Rotation Is Enabled
        • AWS Application Load Balancer Has Web ACL
        • AWS Access Keys Are Used Every 90 Days
        • AWS Access Keys are Rotated Every 90 Days
        • AWS ACM Certificate Is Not Expired
        • AWS Access Keys not Created During Account Creation
        • AWS CloudTrail Has Log Validation Enabled
        • AWS CloudTrail S3 Bucket Has Access Logging Enabled
        • AWS CloudTrail Logs S3 Bucket Not Publicly Accessible
        • AWS Config Is Enabled for Global Resources
        • AWS DynamoDB Table Has Autoscaling Targets Configured
        • AWS DynamoDB Table Has Autoscaling Enabled
        • AWS DynamoDB Table Has Encryption Enabled
        • AWS EC2 AMI Launched on Approved Host
        • AWS EC2 AMI Launched on Approved Instance Type
        • AWS EC2 AMI Launched With Approved Tenancy
        • AWS EC2 Instance Has Detailed Monitoring Enabled
        • AWS EC2 Instance Is EBS Optimized
        • AWS EC2 Instance Running on Approved AMI
        • AWS EC2 Instance Running on Approved Instance Type
        • AWS EC2 Instance Running in Approved VPC
        • AWS EC2 Instance Running On Approved Host
        • AWS EC2 Instance Running With Approved Tenancy
        • AWS EC2 Instance Volumes Are Encrypted
        • AWS EC2 Volume Is Encrypted
        • AWS GuardDuty is Logging to a Master Account
        • AWS GuardDuty Is Enabled
        • AWS IAM Group Has Users
        • AWS IAM Policy Blocklist Is Respected
        • AWS IAM Policy Does Not Grant Full Administrative Privileges
        • AWS IAM Policy Is Not Assigned Directly To User
        • AWS IAM Policy Role Mapping Is Respected
        • AWS IAM User Has MFA Enabled
        • AWS IAM Password Used Every 90 Days
        • AWS Password Policy Enforces Complexity Guidelines
        • AWS Password Policy Enforces Password Age Limit Of 90 Days Or Less
        • AWS Password Policy Prevents Password Reuse
        • AWS RDS Instance Is Not Publicly Accessible
        • AWS RDS Instance Snapshots Are Not Publicly Accessible
        • AWS RDS Instance Has Storage Encrypted
        • AWS RDS Instance Has Backups Enabled
        • AWS RDS Instance Has High Availability Configured
        • AWS Redshift Cluster Allows Version Upgrades
        • AWS Redshift Cluster Has Encryption Enabled
        • AWS Redshift Cluster Has Logging Enabled
        • AWS Redshift Cluster Has Correct Preferred Maintenance Window
        • AWS Redshift Cluster Has Sufficient Snapshot Retention Period
        • AWS Resource Has Minimum Number of Tags
        • AWS Resource Has Required Tags
        • AWS Root Account Has MFA Enabled
        • AWS Root Account Does Not Have Access Keys
        • AWS S3 Bucket Name Has No Periods
        • AWS S3 Bucket Not Publicly Readable
        • AWS S3 Bucket Not Publicly Writeable
        • AWS S3 Bucket Policy Does Not Use Allow With Not Principal
        • AWS S3 Bucket Policy Enforces Secure Access
        • AWS S3 Bucket Policy Restricts Allowed Actions
        • AWS S3 Bucket Policy Restricts Principal
        • AWS S3 Bucket Has Versioning Enabled
        • AWS S3 Bucket Has Encryption Enabled
        • AWS S3 Bucket Lifecycle Configuration Expires Data
        • AWS S3 Bucket Has Logging Enabled
        • AWS S3 Bucket Has MFA Delete Enabled
        • AWS S3 Bucket Has Public Access Block Enabled
        • AWS Security Group Restricts Ingress On Administrative Ports
        • AWS VPC Default Security Group Restricts All Traffic
        • AWS VPC Flow Logging Enabled
        • AWS WAF Has Correct Rule Ordering
        • AWS CloudTrail Logs Encrypted Using KMS CMK
      • Panther-managed Rules Runbooks
        • AWS CloudTrail Modified
        • AWS Config Service Modified
        • AWS Console Login Failed
        • AWS Console Login Without MFA
        • AWS EC2 Gateway Modified
        • AWS EC2 Network ACL Modified
        • AWS EC2 Route Table Modified
        • AWS EC2 SecurityGroup Modified
        • AWS EC2 VPC Modified
        • AWS IAM Policy Modified
        • AWS KMS CMK Loss
        • AWS Root Activity
        • AWS S3 Bucket Policy Modified
        • AWS Unauthorized API Call
    • Tech Partner Alert Destination Integrations
  • Investigations & Search
    • Search
      • Search Filter Operators
    • Data Explorer
      • Data Explorer SQL Search Examples
        • CloudTrail logs queries
        • GitHub Audit logs queries
        • GuardDuty logs queries
        • Nginx and ALB Access logs queries
        • Okta logs queries
        • S3 Access logs queries
        • VPC logs queries
    • Visualization and Dashboards
      • Custom Dashboards (Beta)
      • Panther-Managed Dashboards
    • Standard Fields
    • Saved and Scheduled Searches
      • Templated Searches
        • Behavioral Analytics and Anomaly Detection Template Macros (Beta)
      • Scheduled Search Examples
    • Search History
    • Data Lakes
      • Snowflake
        • Snowflake Configuration for Optimal Search Performance
      • Athena
  • PantherFlow (Beta)
    • PantherFlow Quick Reference
    • PantherFlow Statements
    • PantherFlow Operators
      • Datatable Operator
      • Extend Operator
      • Join Operator
      • Limit Operator
      • Project Operator
      • Range Operator
      • Sort Operator
      • Search Operator
      • Summarize Operator
      • Union Operator
      • Visualize Operator
      • Where Operator
    • PantherFlow Data Types
    • PantherFlow Expressions
    • PantherFlow Functions
      • Aggregation Functions
      • Date/time Functions
      • String Functions
      • Array Functions
      • Math Functions
      • Control Flow Functions
      • Regular Expression Functions
      • Snowflake Functions
      • Data Type Functions
      • Other Functions
    • PantherFlow Example Queries
      • PantherFlow Examples: Threat Hunting Scenarios
      • PantherFlow Examples: SOC Operations
      • PantherFlow Examples: Panther Audit Logs
  • Enrichment
    • Custom Lookup Tables
      • Creating a GreyNoise Lookup Table
      • Lookup Table Examples
        • Using Lookup Tables: 1Password UUIDs
      • Lookup Table Specification Reference
    • Identity Provider Profiles
      • Okta Profiles
      • Google Workspace Profiles
    • Anomali ThreatStream
    • IPinfo
    • Tor Exit Nodes
    • TrailDiscover (Beta)
  • Panther AI (Beta)
  • System Configuration
    • Role-Based Access Control
    • Identity & Access Integrations
      • Azure Active Directory SSO
      • Duo SSO
      • G Suite SSO
      • Okta SSO
        • Okta SCIM
      • OneLogin SSO
      • Generic SSO
    • Panther Audit Logs
      • Querying and Writing Detections for Panther Audit Logs
      • Panther Audit Log Actions
    • Notifications and Errors (Beta)
      • System Errors
    • Panther Deployment Types
      • SaaS
      • Cloud Connected
        • Configuring Snowflake for Cloud Connected
        • Configuring AWS for Cloud Connected
        • Pre-Deployment Tools
      • Legacy Configurations
        • Snowflake Connected (Legacy)
        • Customer-configured Snowflake Integration (Legacy)
        • Self-Hosted Deployments (Legacy)
          • Runtime Environment
  • Panther Developer Workflows
    • Panther Developer Workflows Overview
    • Using panther-analysis
      • Public Fork
      • Private Clone
      • Panther Analysis Tool
        • Install, Configure, and Authenticate with the Panther Analysis Tool
        • Panther Analysis Tool Commands
        • Managing Lookup Tables and Enrichment Providers with the Panther Analysis Tool
      • CI/CD for Panther Content
        • Deployment Workflows Using Panther Analysis Tool
          • Managing Panther Content via CircleCI
          • Managing Panther Content via GitHub Actions
        • Migrating to a CI/CD Workflow
    • Panther API
      • REST API (Beta)
        • Alerts
        • Alert Comments
        • API Tokens
        • Data Models
        • Globals
        • Log Sources
        • Queries
        • Roles
        • Rules
        • Scheduled Rules
        • Simple Rules
        • Policies
        • Users
      • GraphQL API
        • Alerts & Errors
        • Cloud Account Management
        • Data Lake Queries
        • Log Source Management
        • Metrics
        • Schemas
        • Token Rotation
        • User & Role Management
      • API Playground
    • Terraform
      • Managing AWS S3 Log Sources with Terraform
      • Managing HTTP Log Sources with Terraform
    • pantherlog Tool
    • Converting Sigma Rules
  • Resources
    • Help
      • Operations
      • Security and Privacy
        • Security Without AWS External ID
      • Glossary
      • Legal
    • Panther System Architecture
Powered by GitBook
On this page
  • Overview
  • How incoming logs are enriched
  • How log types and Selectors are set for a Lookup Table
  • p_enrichment structure
  • How to access Lookup Table data in detections
  • Option 1 (if log is enriched): Using deep_get()
  • Option 2: Dynamically using lookup()
  • Prerequisites for configuring a Lookup Table
  • Primary key data types
  • How to configure a Lookup Table
  • Option 1: Import via file upload
  • Option 2: Sync via S3 source
  • Writing a detection using Lookup Table data
  • Testing detections that use enrichment
  • Lookup Table History Tables

Was this helpful?

  1. Enrichment

Custom Lookup Tables

Enrich events with your own stored data

PreviousEnrichmentNextCreating a GreyNoise Lookup Table

Last updated 9 days ago

Was this helpful?

Overview

Custom Lookup Tables (also referred to as simply "Lookup Tables") allow you to store and reference custom enrichment data in Panther. This means you can reference this added context in detections and pass it into alerts. It may be particularly useful to create Lookup Tables containing identity/asset information, vulnerability context, or network maps.

You can associate one or more log types with your Lookup Table—then all incoming logs of those types (that have a match to a Lookup Table value) will contain enrichment data from your Lookup Table. Learn more about the enrichment process in . It's also possible to .

Learn how to , and how to .

If your lookup data is only needed for a few specific detections and will not be frequently updated, consider using instead of a Lookup Table. Also note that there are , including Enrichment Providers like IPinfo and Tor Exit Nodes, as well as .

To increase the limit on the number of Lookup Tables or the size of Lookup Tables in your account, please contact your Panther support team.

How incoming logs are enriched

Lookup Tables in Panther traditionally define both of the following:

  • A primary key: A field in the Lookup Table data.

    • If the Lookup Table is defined in the CLI workflow, this is designated by the PrimaryKey field in the .

  • One or more associated log types, each with one or more Selectors: A Selector is an event field whose values are compared to the Lookup Table's primary key values to find a match.

    • There are two ways to set log types/Selectors for a Lookup Table. See , below.

When a log is ingested into Panther, if its log type is one that is associated to a Lookup Table, the values of all of its Selector fields are compared against the Lookup Table's primary key values. When a match is found between a value in a Selector field and a primary key value, the log is enriched with the matching primary key's associated Lookup Table data in a p_enrichment field. Learn more about p_enrichment below, in .

In the example in the image below, the Selector field (in the events in Incoming Logs) is ip_address. The primary key of the Lookup Table LUT1 is bad_actor_ip. In the right-hand Alert Event, the log is enriched with the Lookup Table data (including bad_actor_name) because there was a match between the Selector value (1.1.1.1) and a primary key value (1.1.1.1).

How log types and Selectors are set for a Lookup Table

Option 1: Manually choose log types and Selectors

When creating a Lookup Table, you can choose one or more log types the Lookup Table should be associated to—and for each log type, one or more Selector fields.

  • When creating a Lookup Table in the CLI workflow, you will create and upload a YAML configuration file. The AssociatedLogTypes value will be a list of objects containing LogType and Selectors fields.

Option 2: Let log types and selectors be automatically mapped by indicator fields

  1. Finds all Active log schemas (or log types) that designate any event field as that same indicator.

  2. Associates those log types to the Lookup Table.

  3. For each log type, sets the p_any field associated to the indicator as a Selector.

For example, if your Lookup Table data's schema designates an address field (which has also been set as the primary key) as an ip indicator, all log types in your Panther instance that also set an ip indicator will be associated to the Lookup Table, each with a p_any_ip_addresses Selector.

p_enrichment structure

If your log events are injected with enrichment data, a p_enrichment field is appended to the event and accessed within a detection using deep_get() or DeepKey. The p_enrichment field will contain:

  • One or more Lookup Table name(s) that matched the incoming log event

  • The name of the Selector from the incoming log that matched the Lookup Table

  • The data from the Lookup Table that matched via the Lookup Table's primary key (including an injected p_match field containing the Selector value that matched)

This is the structure of p_enrichment fields:

'p_enrichment': {
    <name of lookup table1>: {
        <name of selector>: {
            'p_match': <value of Selector>,
	    <lookup key>: <lookup value>,
	    ...
	}
    }
}

How to access Lookup Table data in detections

Option 1 (if log is enriched): Using deep_get()

Option 2: Dynamically using lookup()

Prerequisites for configuring a Lookup Table

  • Your Lookup Table data is in JSON or CSV format. (JSON files can format events in various ways, including in lines, arrays, or objects.)

  • A schema specifically for your Lookup Table data.

    • This describes the shape of your Lookup Table data.

  • A primary key for your Lookup Table data.

    • This primary key is one of the fields you defined in your Lookup Table's schema. The value of the primary key is what will be compared with the value of the selector(s) from your incoming logs.

  • (Optional) Selector(s) from your incoming logs.

    • The values from these selectors will be used to search for matches in your Lookup Table data.

  • (CLI workflow): A Lookup Table configuration file.

Primary key data types

Your Lookup Table's primary key column must be one of the following data types:

  • String

  • Number

  • Array (of strings or numbers)

    • Using an array lets you associate one row in your Lookup Table with multiple string or number primary key values. This prevents you from having to duplicate a certain row of data for multiple primary keys.

Example: string array vs. string primary key type

Perhaps you'd like to store user data in a Lookup Table so that incoming log events associated with a certain user are enriched with additional personal information. You'd like to match on the user's email address, which means the email field will be the primary key in the Lookup Table and the selector in the log events.

You are deciding whether the primary key column in your Lookup Table should be of type string or string array. First, review the below two events you might expect to receive from your log source:

# Incoming log event one
{
    "actor_email": "janedoeemailone@mail.com",
    "action": "LOGIN"
}

# Incoming log event two
{
    "actor_email": "janedoeemailtwo@mail.com",
    "action": "EXPORT_FILE"
}

Note that the two email addresses (janedoeemailone@mail.com and janedoeemailtwo@mail.com) belong to the same user, Jane Doe.

When Panther receives these events, you would like to use a Lookup Table to enrich each of them with Jane's full name and role. After enrichment, these events would look like the following:

# Log event one after enrichment
{
    "actor_email": "janedoeemailone@mail.com",
    "action": "LOGIN",
    "p_enrichment": {
        "<lookup_table_name>": {
            "actor_email": {
                "full_name": "Jane Doe",
                "p_match":  "janedoeemailone@mail.com",
                "role": "ADMIN"
            }
        }
    }
}

# Log event two after enrichment
{
    "actor_email": "janedoeemailtwo@mail.com",
    "action": "EXPORT_FILE",
    "p_enrichment": {
        "<lookup_table_name>": {
            "actor_email": {
                "full_name": "Jane Doe",
                "p_match": "janedoeemailtwo@mail.com",
                "role": "ADMIN"
            }
        }
    }
}

You can accomplish this enrichment by defining a Lookup Table with either:

  • (Recommended) A primary key column that is of type array of strings

  • A primary key column that is of type string

Using a Lookup Table with a primary key column that is of type array of strings, you can include Jane's multiple email addresses in one primary key entry, associated to one row of data. This might look like the following:

Alternatively, you can define a Lookup Table with a primary key column that is of type string. However, because the match between the event and Lookup Table is made on the user's email address, and a user can have multiple email addresses (as is shown in Jane's case), you must duplicate the Lookup Table row for each email. This would look like the following:

While both options yield the same result (i.e., log events are enriched in the same way), defining a Lookup Table with an array of strings primary key is recommended for its convenience and reduced proneness to maintenance error.

How to configure a Lookup Table

The maximum size for a row in a Lookup Table is 65535 bytes.

Option 1: Import via file upload

This option is best for data that is relatively static, such as information about AWS accounts or corporate subnets. For example, adding metadata to distinguish developer accounts from production accounts in your AWS CloudTrail logs.

You can import via file upload through the Panther Console or with PAT:

Panther Console import via file upload
  1. In the left-hand navigation bar in your Panther Console, click Configure > Lookup Tables.

  2. In the upper-right corner, click Create New.

  3. On the Basic Information page:

    • Enter a descriptive Lookup Name.

    • Enter a Description (optional) and a Reference (optional). Description is meant for content about the table, while Reference can be used to hyperlink to an internal resource.

  4. Click Continue.

  5. On the Associated Log Types page, optionally designate log types/Selectors:

    • Click Add Log Type.

    • Click the Log Type dropdown, then select a log type.

    • Choose one or more Selectors, the foreign key fields from the log type you want enriched with your Lookup Table.

  6. Click Continue.

    • Select a Schema Name from the dropdown.

  7. Click Continue.

  8. Click Finish Setup. A source setup success page will populate.

  9. Optionally, next to to Set an alarm in case this lookup table doesn't receive any data?, toggle the setting to YES to enable an alarm.

    • Fill in the Number and Period fields to indicate how often Panther should send you this notification.

    Note: Notifications generated for a Lookup Table upload failing are accessible in the System Errors tab within the Alerts & Errors page in the Panther Console.

Once finished, you should be returned to the Lookup Table overview screen. Ensure that your new Lookup Table is listed.

PAT import via file upload

File setup

A Lookup Table requires the following files:

  • A YAML specification file containing the configuration for the table

  • A YAML file defining the schema to use when loading data into the table

  • A JSON or CSV file containing data to load into the table (optional, read further).

Folder setup

Writing the configuration files

It's usually prudent to begin writing the schema config first, because the table config will reference some of those values.

  1. Next, create a YAML file for the table configuration. For a Lookup Table with data stored in a local file, an example configuration would look like:

    AnalysisType: lookup_table
    LookupName: my_lookup_table # A unique display name
    Schema: Custom.MyTableSchema # The schema defined in the previous step
    Filename: ./my_lookup_table_data.csv # Relative path to data
    Description: >
      A handy description of what information this table contains.
      For example, this table might convert IP addresses to hostnames
    Reference: >
      A URL to some additional documentation around this table
    Enabled: true # Set to false to stop using the table
    LogTypeMap:
      PrimaryKey: ip                # The primary key of the table
      AssociatedLogTypes:           # A list of log types to match this table to
        - LogType: AWS.CloudTrail
          Selectors:
            - "sourceIPAddress"     # A field in CloudTrail logs
            - "p_any_ip_addresses"  # A panther-generated field works too
        - LogType: Okta.SystemLog
          Selectors:
            - "$.client.ipAddress"  # Paths to JSON values are allowed
  2. Upload the schema file by running the update schemas command from the repository root: panther_analysis_tool update-custom-schemas --path ./schemas

  3. Finally, from the root of the repo, upload the lookup table: panther_analysis_tool upload

Update Lookup Tables via Panther Analysis Tool:

  1. Locate the YAML configuration file for the Lookup Table in question.

  2. Open the file, and look for the field Filename. You should see a file path which leads to the data file.

  3. Update or replace the file indicated in Filename.

  4. Push your changes to Panther with the following code:

    panther_analysis_tool upload

    Optionally, you can specify only to upload the Lookup Table:

    panther_analysis_tool upload --filter AnalysisType=lookup_table

Option 2: Sync via S3 source

In some cases, you may want to sync from an S3 source to set up a Lookup Table. For example, if you want to know what groups and permission levels are associated with the employees at your company. In this scenario, your company might have an AWS S3 source with an up-to-date copy of their Active Directory listing that includes groups and permissions information.

This option is best for a larger amount of data that updates more frequently from an S3 bucket. Any changes in the S3 bucket will sync to Panther.

You can sync via S3 through the Panther Console or with PAT:

Panther Console sync via S3
  1. In the left-hand navigation bar in your Panther Console, click Configure > Lookup Tables.

  2. In the upper-right corner, click Create New.

  3. On the Basic Information page:

    • Enter a descriptive Lookup Name.

    • Enter a Description (optional) and a Reference (optional).

      • Description is meant for content about the table, while Reference can be used to hyperlink to an internal resource.

    • Make sure the Enabled? toggle is set to Yes.

      • Note: This is required to import your data later in this process.

  4. Click Continue.

  5. On the Associated Log Types page, optionally designate log types/Selectors:

    • Click Add Log Type.

    • Click the Log Type dropdown, then select a log type.

    • Choose one or more Selectors, the foreign key fields form the log type you want enriched with your Lookup Table.

  6. Click Continue.

    1. Select a Schema Name from the dropdown.

    2. Select a Primary Key Name from the dropdown. This should be a unique column on the table, such as accountID.

  7. Click Continue.

  8. Set up your S3 source. Note that your data must be in .csv or .json format.

    • Enter the Account ID, the 12-digit AWS Account ID where the S3 bucket is located.

    • Enter the S3 URI, the unique path that identifies the specific S3 bucket.

    • Optionally, enter the KMS Key if your data is encrypted using KMS-SSE.

  9. Click Continue.

  10. Set up an IAM Role.

  11. Click Finish Setup. A source setup success page will populate.

  12. Optionally, next to to Set an alarm in case this lookup table doesn't receive any data?, toggle the setting to YES to enable an alarm.

    • Fill in the Number and Period fields to indicate how often Panther should send you this notification.

Note: Notifications generated for a Lookup Table upload failing are accessible in the System Errors tab within the Alerts & Errors page in the Panther Console.

Creating an IAM Role

There are three options for creating an IAM Role to use with your Panther Lookup Table using an S3 source:

Create an IAM role using AWS Console UI

  1. On the "Set Up an IAM role" page, during the process of creating a Lookup Table with an S3 source, locate the tile labeled "Using the AWS Console UI". On the right side of the tile, click Select.

    • You will be redirected to the AWS console in a new browser tab, with the template URL pre-filled.

    • The CloudFormation stack will create an AWS IAM role with the minimum required permissions to read objects from your S3 bucket.

    • Click the "Outputs" tab of the CloudFormation stack in AWS, and note the Role ARN.

  2. Navigate back to your Panther account.

  3. On the "Use AWS UI to set up your role" page, enter the Role ARN.

  4. Click Finish Setup.

Create an IAM role using CloudFormation Template File

  1. On the "Set Up an IAM role" page, during the process of creating a Lookup Table with an S3 source, locate the tile labeled "CloudFormation Template File". On the right side of the tile, click Select.

  2. Click CloudFormation template, which downloads the template to apply it through your own pipeline.

  3. Upload the template file in AWS:

    1. Open your AWS console and navigate to the CloudFormation product.

    2. Click Create stack.

    3. Click Upload a template file and select the CloudFormation template you downloaded.

  4. On the "CloudFormation Template" page in Panther, enter the Role ARN.

  5. Click Finish Setup.

Create an IAM role manually

  1. On the "Set Up an IAM role" page, during the process of creating a Lookup Table with an S3 source, click the link that says I want to set everything up on my own.

  2. Create the required IAM role. You may create the required IAM role manually or through your own automation. The role must be named using the format PantherLUTsRole-${Suffix}(e.g., PantherLUTsRole-MyLookupTable).

    • The IAM role policy must include the statements defined below:

          "Version": "2012-10-17",
          "Statement": [
              {
                  "Action": "s3:GetBucketLocation",
                  "Resource": "arn:aws:s3:::<bucket-name>",
                  "Effect": "Allow"
              },
              {
                  "Action": "s3:GetObject",
                  "Resource": "arn:aws:s3:::<bucket-name>/<input-file-path>",
                  "Effect": "Allow"
              }
          ]
      }
    • If your S3 bucket is configured with server-side encryption using AWS KMS, you must include an additional statement granting the Panther API access to the corresponding KMS key. In this case, the policy will look something like this:

          "Version": "2012-10-17",
          "Statement": [
              {
                  "Action": "s3:GetBucketLocation",
                  "Resource": "arn:aws:s3:::<bucket-name>",
                  "Effect": "Allow"
              },
              {
                  "Action": "s3:GetObject",
                  "Resource": "arn:aws:s3:::<bucket-name>/<input-file-path>",
                  "Effect": "Allow"
              },
              {
                  "Action": ["kms:Decrypt", "kms:DescribeKey"],
                  "Resource": "arn:aws:kms:<region>:<your-accound-id>:key/<kms-key-id>",
                  "Effect": "Allow"
              }
          ]
      }
  3. On the "Setting up role manually" page in Panther, enter the Role ARN.

    • This can be found in the "Outputs" tab of the CloudFormation stack in your AWS account.

  4. Click Finish Setup, and you will be redirected to the Lookup Tables list page with your new Employee Directory table listed.

PAT sync via S3

File setup

A Lookup Table requires the following files:

  • A YAML specification file containing the configuration for the table

  • A YAML file defining the schema to use when loading data into the table

  • A JSON or CSV file containing data to load into the table (optional, read further).

Folder setup

Writing the configuration files

It's usually prudent to begin writing the schema config first, because the table config will reference some of those values.

  1. Next, create a YAML file for the table configuration. For a Lookup Table with data stored in a file in S3, an example configuration would look like this:

    AnalysisType: lookup_table
    LookupName: my_lookup_table # A unique display name
    Schema: Custom.MyTableSchema # The schema defined in the previous step
    Refresh:
      RoleArn: arn:aws:iam::123456789012:role/PantherLUTsRole-my_lookup_table # A role in your organization's AWS account
      ObjectPath: s3://path/to/my_lookup_table_data.csv
      PeriodMinutes: 120 # Sync from S3 every 2 hours
    Description: >
      A handy description of what information this table contains.
      For example, this table might convert IP addresses to hostnames
    Reference: >
      A URL to some additional documentation around this table
    Enabled: true # Set to false to stop using the table
    LogTypeMap:
      PrimaryKey: ip                # The primary key of the table
      AssociatedLogTypes:           # A list of log types to match this table to
        - LogType: AWS.CloudTrail
          Selectors:
            - "sourceIPAddress"     # A field in CloudTrail logs
            - "p_any_ip_addresses"  # A panther-generated field works too
        - LogType: Okta.SystemLog
          Selectors:
            - "$.client.ipAddress"  # Paths to JSON values are allowed
  1. Finally, from the root of the repo, upload the lookup table: panther_analysis_tool upload

  2. Upload the schema file by running the update schemas command from the repository root: panther_analysis_tool update-custom-schemas --path ./schemas

Prerequisites

Before you can configure your Lookup Table to sync with S3, you'll need to have the following ready:

  1. The path to the file you intend to store data in. The path should be of the following format: s3://bucket-name/path_to_file/file.csv

Configuring Lookup Table for S3

  1. Navigate to the YAML specification file for this Lookup Table.

  2. In the file, locate (or add) the Refresh field.

  3. Specify the RoleARN, ObjectPath, and PeriodMinutes fields. For specs on the allowed values, see our Lookup Table Config File Specification.

  4. Save the config file, then upload your changes with panther_analysis_tool.​

Writing a detection using Lookup Table data

After you configure a Lookup Table, you can write detections based on the additional context from your Lookup Table.

For example, if you configured a Lookup Table to distinguish between developer and production accounts in AWS CloudTrail logs, you might want receive an alert only if the following circumstances are both true:

  • A user logged in who did not have MFA enabled.

  • The AWS account is a production (not a developer) account.

See how to create a detection using Lookup Table data below:

Accessing Lookup Table data the event was automatically enriched with

deep_get(event, 'p_enrichment', <Lookup Table name>, <foreign key in log>, <field in Lookup Table>)

The Lookup Table name, foreign key and field name are all optional parameters. If not specified, deep_get() will return a hierarchical dictionary with all the enrichment data available. Specifying the parameters will ensure that only the data you care about is returned.

The rule would become:

from panther_base_helpers import deep_get
 def rule(event):
   is_production = deep_get(event, 'p_enrichment', 'account_metadata',
'recipientAccountId', 'isProduction') # If the field you're accessing is stored within a list, use deep_walk() instead
   return not event.get('mfaEnabled') and is_production

Dynamically accessing Lookup Table data

Detection:
  - Enrichment:
      Table: account_metadata
      Selector: recipientAccountId
      FieldPath: isProduction
    Condition: Equals
    Value: true
  - KeyPath: mfaEnabled
    Condition: Equals
    Value: false

The Panther rules engine will take the looked up matches and append that data to the event using the key p_enrichment in the following JSON structure:

{ 
    "p_enrichment": {
        <name of lookup table>: { 
            <key in log that matched>: <matching row looked up>,
            ...
	    <key in log that matched>: <matching row looked up>,
	}    
    }
} 

Example:

 {
  "p_enrichment": {
      "account_metadata": {
          "recipientAccountId": {
              "accountID": "90123456", 
              "isProduction": false, 
              "email": "dev.account@example.com",
              "p_match": "90123456"
              }
          }
      }
}

If the value of the matching log key is an array (e.g., the value of p_any_aws_accout_ids), then the lookup data is an array containing the matching records.

{ 
    "p_enrichment": {
        <name of lookup table>: { 
            <key in log that matched that is an array>: [
                <matching row looked up>,
                <matching row looked up>,
                <matching row looked up>
            ]
	}
     }
} 

Example:

 {
  "p_enrichment": {
      "account_metadata": {
          "p_any_aws_account_ids": [
             {
              "accountID": "90123456", 
              "isProduction": false, 
              "email": "dev.account@example.com",
              "p_match": "90123456"
              },
              {
              "accountID": "12345678", 
              "isProduction": true, 
              "email": "prod.account@example.com",
              "p_match": "12345678"
              }
          ]
      }
  }
}

Testing detections that use enrichment

For rules that use p_enrichment, click Enrich Test Data in the upper right side of the JSON code editor to populate it with your Lookup Table data. This allows you to test a Python function with an event that contains p_enrichment.

In order for your unit test to enrich properly, your event must specify the following two fields:

  • p_log_type: This determines which Lookup Tables to use

  • The selector field: This provides a value to match against

Lookup Table History Tables

Lookup Tables will generate a number of tables in the Data Explorer. There are two main types of tables generated:

  1. The current Lookup Table version: example

    • Contains the most up to date Lookup Table data

    • Should be targeted in any Saved Searches, or anywhere you expect to see the most current data

    • This table name will never change

    • In the example above, the table is named example

  2. The current History Table version: example_history

    • Contains a version history of all data uploaded to the current Lookup Table

    • The table schema is identical to the current Lookup Table (here named example) except for two additional fields:

      • p_valid_start

      • p_valid_end

    • These fields can be used to view the state of the Lookup Table at any previous point in time

When a new schema is assigned to the Lookup Table, the past versions of the Lookup Table and the History Table are both preserved as well.

These past versions are preserved by the addition of a numeric suffix, _### and will be present for both the Lookup Table and the History Table. This number will increment by one each time the schema associated with the Lookup Table is replaced, or each time the primary key of the Lookup Table is changed.

The current Lookup Table and History Table are views that point at the highest numeric suffix table. This means when a new Lookup Table (called example below) is created, you will see 4 tables:

  • example

  • example_history

  • example_001

  • example_history_001

The current-version tables shown here (example and example_history) are views that are pointing at the respective underlying tables (suffixed with _001).

If a new schema is created, then _002 suffixed tables will be created, and the current-version tables will now point at those. The _001 tables will be no longer updated.

You can manually set associated log types and Selectors when creating a Lookup Table (), and/or let them be automatically mapped ().

Note that even if you manually choose log types and Selectors in this way, the automatic mappings described in will still be applied.

When creating a Lookup Table in the Panther Console, you can set associated log types and Selectors. Learn more in , below.

See the for a full list of fields.

This method of associating log types and Selectors to a Lookup Table is not available for Lookup Tables that are populated using . It is only applicable to Lookup Tables that are populated using .

If, in the schema for your Lookup Table data, the primary key field is marked as an (e.g., as an email and/or username), for each indicator value, Panther automatically:

Note that p_enrichment is not stored with the log event in the data lake. See for more information.

If your log event was enriched on ingest (as described in ), you can access the data within the p_enrichment field () using the deep_get() event object function. Learn more about deep_get() .

See a full example of this method below, in .

It's also possible to dynamically access Lookup Table data from Python detections using . In this way, you can retrieve data from any Lookup Table, without it being injected into an incoming event as described in .

See the below section to learn more about primary key requirements.

See for a full list of fields.

We recommend you and install the .

After fulfilling the , Lookup Tables can be created and configured using either of the following methods:

After choosing one of these methods, you can opt to work within the Panther Console or with .

Next to Enabled? toggle the setting to Yes. Note: This is required to import your data later in this process.

You also can reference attributes in nested objects using . For example, if you wanted to reference a field in a map, you could enter $.field.subfield.

Click Add Log Type to add another if needed. In the example screen shot above, we selected AWS.CloudTrail logs and typed in accountID and recipientAccountID to represent keys in the CloudTrail logs.

Configure the Table Schema. Note: If you have not already created a new schema, please see . You can also use your Lookup Table data to infer a schema. Once you have created a schema, you will be able to choose it from the dropdown on the Table Schema page while configuring a Lookup Table. Note: CSV schemas require column headers to work with Lookup Tables.

Select a Primary Key Name from the dropdown. This should be a unique column on the table, such as accountID.

Drag and drop a file or click Select File to choose the file of your Lookup Table data to import. The file must be in .csv or .json format.

The alert destinations for this alarm are displayed at the bottom of the page. To configure and customize where your notification is sent, see documentation on .

​ ​

If your data file is larger than 1MB, it's suggested to instead use the .

All files related to your Lookup Tables must be stored in a folder with a name containing lookup_tables. This could be a top-level lookup_tables directory, or sub-directories with names matching *lookup_tables*. You can use the repo as a reference.

Create a YAML file for the schema, and save it with the rest of your custom schemas, outside the lookup_tables directory (for example, /schemas in the root of your panther analysis repo). This schema defines how to read the files you'll use to upload data to the table. If using a CSV file for data, then the schema should be able to parse CSV. The table schema is formatted the same as a log schema. For more information on writing schemas, read our documentation around

You also can reference attributes in nested objects using . For example, if you wanted to reference a field in a map, you could enter $.field.subfield.

Click Add Log Type to add another if needed. In the example screen shot above, we selected AWS.VPCFlow logs and typed in account to represent keys in the VPC Flow logs.

Configure the Table Schema. Note: If you have not already created a new schema, please see . Once you have created a schema, you will be able to select it from the dropdown on the Table Schema page while configuring a Lookup Table.

On the "Choose Import Method" page, click Set Up next to "Sync Data from an S3 Bucket."

Enter the Update Period, the cadence your S3 source gets updated (defaulted to 1 hour).

Please see the next section, , for instructions on the three options available to do this.

The alert destinations for this alarm are displayed at the bottom of the page. To configure and customize where your notification is sent, see documentation on .

​​

.

.

Click Launch Console UI.

All files related to your Lookup Tables must be stored in a folder with a name containing lookup_tables. This could be a top-level lookup_tables directory, or sub-directories with names matching *lookup_tables*. You can use the repo as a reference.

Create a YAML file for the schema, and save it in the lookup table directory (for example, lookup_tables/my_table/my_table_schema.yml). This schema defines how to read the files you'll use to upload data to the table. If using a CSV file for data, then the schema should be able to parse CSV. The table schema is formatted the same as a log schema. For more information on writing schemas, read our documentation around

The ARN of an IAM role in AWS, which Panther can use to access the S3 bucket. For more information on setting up an IAM role for Panther, see the section on

In Python, you can use the to retrieve the looked up field from p_enrichment using the foreign key field in the log. The pattern looks like this:

You can also use the to dynamically access Lookup Table data in your detection. This may be useful when your event doesn't contain an exact match to a value in the Lookup Table's primary key column.

In a , you can create an .

Lookup Table Specification Reference
Lookup Table Specification Reference
make a fork of the panther-analysis repository
Panther Analysis Tool (PAT)
PAT
JSON path syntax
our documentation on creating schemas
Panther Destinations
Panther Analysis
managing Log Schemas.
JSON path syntax
our documentation on creating schemas
Panther Destinations
Panther Analysis
managing Log Schemas.
Option 1
Option 2
Option 2
How to configure a Lookup Table
file upload
S3 sync
Writing a detection using Lookup Table data
Primary key data types
prerequisites
Import via file upload
Sync via S3 source
S3 sync upload method
Creating an IAM Role
Create an IAM role using AWS Console UI.
Create an IAM role using CloudFormation Template File
Create an IAM role manually
Creating an IAM Role.
Viewing log events with enrichment data
deep_get() helper function
How incoming logs are enriched
whose structure is described above
on Writing Python Detections
How incoming logs are enriched
the event.lookup() function
event object's lookup() function
indicator field
Global helpers
Identity Provider Profiles
YAML configuration file
How incoming logs are enriched
dynamically reference Lookup Table data in Python detections
How log types and Selectors are set for a Lookup Table
p_enrichment structure
view stored Lookup Table data here
view log events with enrichment data here
Panther-managed Lookup Tables
Simple Detection
Enrichment match expression
A diagram showing how Lookup Tables work: On the left, there is a code box labeled "Incoming logs." An arrow branches off the logs and points to a Lookup Table including "bad_actor_ip" and "bad_actor_name." On the right, an arrow goes from the Lookup Table to an Alert Event, showing the alert you would receive based on the log example.
Under an "Associated Log Types (Optional)" header is a form with fields for Log Type, Selectors, and Add Log Type. At the bottom is a Continue button.
A "Unit Test" header is above a code block with JSON. The JSON includes various key/value pairs—for example, "p_log_type": "My.Log.Type"