LogoLogo
Knowledge BaseCommunityRelease NotesRequest Demo
  • Overview
  • Quick Start
    • Onboarding Guide
  • Data Sources & Transports
    • Supported Logs
      • 1Password Logs
      • Apache Logs
      • AppOmni Logs
      • Asana Logs
      • Atlassian Logs
      • Auditd Logs
      • Auth0 Logs
      • AWS Logs
        • AWS ALB
        • AWS Aurora
        • AWS CloudFront
        • AWS CloudTrail
        • AWS CloudWatch
        • AWS Config
        • AWS EKS
        • AWS GuardDuty
        • AWS Security Hub
        • Amazon Security Lake
        • AWS S3
        • AWS Transit Gateway
        • AWS VPC
        • AWS WAF
      • Azure Monitor Logs
      • Bitwarden Logs
      • Box Logs
      • Carbon Black Logs
      • Cisco Umbrella Logs
      • Cloudflare Logs
      • CrowdStrike Logs
        • CrowdStrike Falcon Data Replicator
        • CrowdStrike Event Streams
      • Docker Logs
      • Dropbox Logs
      • Duo Security Logs
      • Envoy Logs
      • Fastly Logs
      • Fluentd Logs
      • GCP Logs
      • GitHub Logs
      • GitLab Logs
      • Google Workspace Logs
      • Heroku Logs
      • Jamf Pro Logs
      • Juniper Logs
      • Lacework Logs
        • Lacework Alert Channel Webhook
        • Lacework Export
      • Material Security Logs
      • Microsoft 365 Logs
      • Microsoft Entra ID Audit Logs
      • Microsoft Graph Logs
      • MongoDB Atlas Logs
      • Netskope Logs
      • Nginx Logs
      • Notion Logs
      • Okta Logs
      • OneLogin Logs
      • Orca Security Logs (Beta)
      • Osquery Logs
      • OSSEC Logs
      • Proofpoint Logs
      • Push Security Logs
      • Rapid7 Logs
      • Salesforce Logs
      • SentinelOne Logs
      • Slack Logs
      • Snowflake Audit Logs (Beta)
      • Snyk Logs
      • Sophos Logs
      • Sublime Security Logs
      • Suricata Logs
      • Sysdig Logs
      • Syslog Logs
      • Tailscale Logs
      • Teleport Logs
      • Tenable Vulnerability Management Logs
      • Thinkst Canary Logs
      • Tines Logs
      • Tracebit Logs
      • Windows Event Logs
      • Wiz Logs
      • Zeek Logs
      • Zendesk Logs
      • Zoom Logs
      • Zscaler Logs
        • Zscaler ZIA
        • Zscaler ZPA
    • Custom Logs
      • Log Schema Reference
      • Transformations
      • Script Log Parser (Beta)
      • Fastmatch Log Parser
      • Regex Log Parser
      • CSV Log Parser
    • Data Transports
      • HTTP Source
      • AWS Sources
        • S3 Source
        • CloudWatch Logs Source
        • SQS Source
          • SNS Source
        • EventBridge
      • Google Cloud Sources
        • Cloud Storage (GCS) Source
        • Pub/Sub Source
      • Azure Blob Storage Source
    • Monitoring Log Sources
    • Ingestion Filters
      • Raw Event Filters
      • Normalized Event Filters (Beta)
    • Data Pipeline Tools
      • Chronosphere Onboarding Guide
      • Cribl Onboarding Guide
      • Fluent Bit Onboarding Guide
        • Fluent Bit Configuration Examples
      • Fluentd Onboarding Guide
        • General log forwarding via Fluentd
        • MacOS System Logs to S3 via Fluentd
        • Syslog to S3 via Fluentd
        • Windows Event Logs to S3 via Fluentd (Legacy)
        • GCP Audit to S3 via Fluentd
      • Observo Onboarding Guide
      • Tarsal Onboarding Guide
    • Tech Partner Log Source Integrations
  • Detections
    • Using Panther-managed Detections
      • Detection Packs
    • Rules and Scheduled Rules
      • Writing Python Detections
        • Python Rule Caching
        • Data Models
        • Global Helper Functions
      • Modifying Detections with Inline Filters (Beta)
      • Derived Detections (Beta)
        • Using Derived Detections to Avoid Merge Conflicts
      • Using the Simple Detection Builder
      • Writing Simple Detections
        • Simple Detection Match Expression Reference
        • Simple Detection Error Codes
    • Correlation Rules (Beta)
      • Correlation Rule Reference
    • PyPanther Detections (Beta)
      • Creating PyPanther Detections
      • Registering, Testing, and Uploading PyPanther Detections
      • Managing PyPanther Detections in the Panther Console
      • PyPanther Detections Style Guide
      • pypanther Library Reference
      • Using the pypanther Command Line Tool
    • Signals
    • Policies
    • Testing
      • Data Replay (Beta)
    • Framework Mapping and MITRE ATT&CK® Matrix
  • Cloud Security Scanning
    • Cloud Resource Attributes
      • AWS
        • ACM Certificate
        • CloudFormation Stack
        • CloudWatch Log Group
        • CloudTrail
        • CloudTrail Meta
        • Config Recorder
        • Config Recorder Meta
        • DynamoDB Table
        • EC2 AMI
        • EC2 Instance
        • EC2 Network ACL
        • EC2 SecurityGroup
        • EC2 Volume
        • EC2 VPC
        • ECS Cluster
        • EKS Cluster
        • ELBV2 Application Load Balancer
        • GuardDuty Detector
        • GuardDuty Detector Meta
        • IAM Group
        • IAM Policy
        • IAM Role
        • IAM Root User
        • IAM User
        • KMS Key
        • Lambda Function
        • Password Policy
        • RDS Instance
        • Redshift Cluster
        • Route 53 Domains
        • Route 53 Hosted Zone
        • S3 Bucket
        • WAF Web ACL
  • Alerts & Destinations
    • Alert Destinations
      • Amazon SNS Destination
      • Amazon SQS Destination
      • Asana Destination
      • Blink Ops Destination
      • Custom Webhook Destination
      • Discord Destination
      • GitHub Destination
      • Google Pub/Sub Destination (Beta)
      • Incident.io Destination
      • Jira Cloud Destination
      • Jira Data Center Destination (Beta)
      • Microsoft Teams Destination
      • Mindflow Destination
      • OpsGenie Destination
      • PagerDuty Destination
      • Rapid7 Destination
      • ServiceNow Destination (Custom Webhook)
      • Slack Bot Destination
      • Slack Destination (Webhook)
      • Splunk Destination (Beta)
      • Tines Destination
      • Torq Destination
    • Assigning and Managing Alerts
      • Managing Alerts in Slack
    • Alert Runbooks
      • Panther-managed Policies Runbooks
        • AWS CloudTrail Is Enabled In All Regions
        • AWS CloudTrail Sending To CloudWatch Logs
        • AWS KMS CMK Key Rotation Is Enabled
        • AWS Application Load Balancer Has Web ACL
        • AWS Access Keys Are Used Every 90 Days
        • AWS Access Keys are Rotated Every 90 Days
        • AWS ACM Certificate Is Not Expired
        • AWS Access Keys not Created During Account Creation
        • AWS CloudTrail Has Log Validation Enabled
        • AWS CloudTrail S3 Bucket Has Access Logging Enabled
        • AWS CloudTrail Logs S3 Bucket Not Publicly Accessible
        • AWS Config Is Enabled for Global Resources
        • AWS DynamoDB Table Has Autoscaling Targets Configured
        • AWS DynamoDB Table Has Autoscaling Enabled
        • AWS DynamoDB Table Has Encryption Enabled
        • AWS EC2 AMI Launched on Approved Host
        • AWS EC2 AMI Launched on Approved Instance Type
        • AWS EC2 AMI Launched With Approved Tenancy
        • AWS EC2 Instance Has Detailed Monitoring Enabled
        • AWS EC2 Instance Is EBS Optimized
        • AWS EC2 Instance Running on Approved AMI
        • AWS EC2 Instance Running on Approved Instance Type
        • AWS EC2 Instance Running in Approved VPC
        • AWS EC2 Instance Running On Approved Host
        • AWS EC2 Instance Running With Approved Tenancy
        • AWS EC2 Instance Volumes Are Encrypted
        • AWS EC2 Volume Is Encrypted
        • AWS GuardDuty is Logging to a Master Account
        • AWS GuardDuty Is Enabled
        • AWS IAM Group Has Users
        • AWS IAM Policy Blocklist Is Respected
        • AWS IAM Policy Does Not Grant Full Administrative Privileges
        • AWS IAM Policy Is Not Assigned Directly To User
        • AWS IAM Policy Role Mapping Is Respected
        • AWS IAM User Has MFA Enabled
        • AWS IAM Password Used Every 90 Days
        • AWS Password Policy Enforces Complexity Guidelines
        • AWS Password Policy Enforces Password Age Limit Of 90 Days Or Less
        • AWS Password Policy Prevents Password Reuse
        • AWS RDS Instance Is Not Publicly Accessible
        • AWS RDS Instance Snapshots Are Not Publicly Accessible
        • AWS RDS Instance Has Storage Encrypted
        • AWS RDS Instance Has Backups Enabled
        • AWS RDS Instance Has High Availability Configured
        • AWS Redshift Cluster Allows Version Upgrades
        • AWS Redshift Cluster Has Encryption Enabled
        • AWS Redshift Cluster Has Logging Enabled
        • AWS Redshift Cluster Has Correct Preferred Maintenance Window
        • AWS Redshift Cluster Has Sufficient Snapshot Retention Period
        • AWS Resource Has Minimum Number of Tags
        • AWS Resource Has Required Tags
        • AWS Root Account Has MFA Enabled
        • AWS Root Account Does Not Have Access Keys
        • AWS S3 Bucket Name Has No Periods
        • AWS S3 Bucket Not Publicly Readable
        • AWS S3 Bucket Not Publicly Writeable
        • AWS S3 Bucket Policy Does Not Use Allow With Not Principal
        • AWS S3 Bucket Policy Enforces Secure Access
        • AWS S3 Bucket Policy Restricts Allowed Actions
        • AWS S3 Bucket Policy Restricts Principal
        • AWS S3 Bucket Has Versioning Enabled
        • AWS S3 Bucket Has Encryption Enabled
        • AWS S3 Bucket Lifecycle Configuration Expires Data
        • AWS S3 Bucket Has Logging Enabled
        • AWS S3 Bucket Has MFA Delete Enabled
        • AWS S3 Bucket Has Public Access Block Enabled
        • AWS Security Group Restricts Ingress On Administrative Ports
        • AWS VPC Default Security Group Restricts All Traffic
        • AWS VPC Flow Logging Enabled
        • AWS WAF Has Correct Rule Ordering
        • AWS CloudTrail Logs Encrypted Using KMS CMK
      • Panther-managed Rules Runbooks
        • AWS CloudTrail Modified
        • AWS Config Service Modified
        • AWS Console Login Failed
        • AWS Console Login Without MFA
        • AWS EC2 Gateway Modified
        • AWS EC2 Network ACL Modified
        • AWS EC2 Route Table Modified
        • AWS EC2 SecurityGroup Modified
        • AWS EC2 VPC Modified
        • AWS IAM Policy Modified
        • AWS KMS CMK Loss
        • AWS Root Activity
        • AWS S3 Bucket Policy Modified
        • AWS Unauthorized API Call
    • Tech Partner Alert Destination Integrations
  • Investigations & Search
    • Search
      • Search Filter Operators
    • Data Explorer
      • Data Explorer SQL Search Examples
        • CloudTrail logs queries
        • GitHub Audit logs queries
        • GuardDuty logs queries
        • Nginx and ALB Access logs queries
        • Okta logs queries
        • S3 Access logs queries
        • VPC logs queries
    • Visualization and Dashboards
      • Custom Dashboards (Beta)
      • Panther-Managed Dashboards
    • Standard Fields
    • Saved and Scheduled Searches
      • Templated Searches
        • Behavioral Analytics and Anomaly Detection Template Macros (Beta)
      • Scheduled Search Examples
    • Search History
    • Data Lakes
      • Snowflake
        • Snowflake Configuration for Optimal Search Performance
      • Athena
  • PantherFlow (Beta)
    • PantherFlow Quick Reference
    • PantherFlow Statements
    • PantherFlow Operators
      • Datatable Operator
      • Extend Operator
      • Join Operator
      • Limit Operator
      • Project Operator
      • Range Operator
      • Sort Operator
      • Search Operator
      • Summarize Operator
      • Union Operator
      • Visualize Operator
      • Where Operator
    • PantherFlow Data Types
    • PantherFlow Expressions
    • PantherFlow Functions
      • Aggregation Functions
      • Date/time Functions
      • String Functions
      • Array Functions
      • Math Functions
      • Control Flow Functions
      • Regular Expression Functions
      • Snowflake Functions
      • Data Type Functions
      • Other Functions
    • PantherFlow Example Queries
      • PantherFlow Examples: Threat Hunting Scenarios
      • PantherFlow Examples: SOC Operations
      • PantherFlow Examples: Panther Audit Logs
  • Enrichment
    • Custom Lookup Tables
      • Creating a GreyNoise Lookup Table
      • Lookup Table Examples
        • Using Lookup Tables: 1Password UUIDs
      • Lookup Table Specification Reference
    • Identity Provider Profiles
      • Okta Profiles
      • Google Workspace Profiles
    • Anomali ThreatStream
    • IPinfo
    • Tor Exit Nodes
    • TrailDiscover (Beta)
  • Panther AI (Beta)
    • Managing Panther AI Response History
  • System Configuration
    • Role-Based Access Control
    • Identity & Access Integrations
      • Azure Active Directory SSO
      • Duo SSO
      • G Suite SSO
      • Okta SSO
        • Okta SCIM
      • OneLogin SSO
      • Generic SSO
    • Panther Audit Logs
      • Querying and Writing Detections for Panther Audit Logs
      • Panther Audit Log Actions
    • Notifications and Errors (Beta)
      • System Errors
    • Panther Deployment Types
      • SaaS
      • Cloud Connected
        • Setting Up a Cloud Connected Panther Instance
      • Legacy Configurations
        • Snowflake Connected (Legacy)
        • Customer-configured Snowflake Integration (Legacy)
        • Self-Hosted Deployments (Legacy)
          • Runtime Environment
  • Panther Developer Workflows
    • Panther Developer Workflows Overview
    • Using panther-analysis
      • Public Fork
      • Private Clone
      • Panther Analysis Tool
        • Install, Configure, and Authenticate with the Panther Analysis Tool
        • Panther Analysis Tool Commands
        • Managing Lookup Tables and Enrichment Providers with the Panther Analysis Tool
      • CI/CD for Panther Content
        • Deployment Workflows Using Panther Analysis Tool
          • Managing Panther Content via CircleCI
          • Managing Panther Content via GitHub Actions
        • Migrating to a CI/CD Workflow
    • Panther API
      • REST API (Beta)
        • Alerts
        • Alert Comments
        • API Tokens
        • Data Models
        • Globals
        • Log Sources
        • Queries
        • Roles
        • Rules
        • Scheduled Rules
        • Simple Rules
        • Policies
        • Users
      • GraphQL API
        • Alerts & Errors
        • Cloud Account Management
        • Data Lake Queries
        • Log Source Management
        • Metrics
        • Schemas
        • Token Rotation
        • User & Role Management
      • API Playground
    • Terraform
      • Managing AWS S3 Log Sources with Terraform
      • Managing HTTP Log Sources with Terraform
    • pantherlog Tool
    • Converting Sigma Rules
  • Resources
    • Help
      • Operations
      • Security and Privacy
        • Security Without AWS External ID
      • Glossary
      • Legal
    • Panther System Architecture
Powered by GitBook
On this page
  • Overview
  • Custom Lookup Tables vs. Enrichment Providers
  • How to manage Custom Lookup Tables and Enrichment Providers with PAT

Was this helpful?

  1. Panther Developer Workflows
  2. Using panther-analysis
  3. Panther Analysis Tool

Managing Lookup Tables and Enrichment Providers with the Panther Analysis Tool

Manage Custom Lookup Tables and Enrichment Providers using PAT

PreviousPanther Analysis Tool CommandsNextCI/CD for Panther Content

Last updated 1 year ago

Was this helpful?

Overview

You can manage and (also known as Panther-managed Lookup Table) schemas and mappings through the .

This guide will walk you through the following:

  • Creating and uploading a custom schema for a Custom Lookup Table using the .

  • Modifying the Selectors and LogTypes in the Lookup Table/Enrichment Provider YAML configuration file.

    • Note that log types and Selectors can or .

  • Uploading the Lookup Table/Enrichment Provider YAML configuration file via PAT.

  • Testing the enrichment in the Panther Console.

  • If your team uses CLI workflows, it's recommended to use PAT and CI/CD to manage your Enrichment, instead of doing so via Detection Packs in the Console.

  • If you choose to manage Lookup Tables through PAT after enabling them in the Panther Console, you must first disable the Detection Packs in the Panther Console. Simultaneous use of both the Panther Console and PAT to manage Lookup Tables is not supported.

This guide applies to all except for , which cannot be enabled in the CLI workflow using PAT.

Custom Lookup Tables vs. Enrichment Providers

In Panther, there is a distinction between Custom Lookup Tables and Enrichment Providers (also known as Panther-managed Lookup Tables):

  • are user-managed. You'll need to create and upload a schema, then upload the Lookup Table's YAML configuration file.

  • are Panther-managed Lookup Tables. Their schemas are Panther-defined, and their YAML configuration files (which you can modify to your needs) can be found in the in GitHub.

How to manage Custom Lookup Tables and Enrichment Providers with PAT

Prerequisites

  • A YAML configuration file. You must create the YAML configuration file yourself.

  • A data sample (if you need to create a new schema) or an existing YAML schema created in Panther.

Step 1: Create and upload a schema

Custom Lookup Tables must be associated with a schema you have created and uploaded to Panther. If you have already created a schema in Panther that you want associate to your Lookup Table, you can skip this step.

  1. Create the schema using your sample log data.

    • You can use pantherlog to infer a schema from a sample set of data. To generate a schema from a sample JSON log file, use the infer command:

      $ ./pantherlog infer sample_logs.jsonl > schema.yml
  2. Upload the schema.

Step 2: Create the YAML configuration file

Step 3: Upload the Lookup Table via PAT

panther_analysis_tool upload

You will need to provide an API token and host with --api-token and --api-host, respectively, for the upload to occur. Other options include filtering, minimum tests, and more.

Ensure you've uploaded the corresponding schema before uploading the YAML configuration file.

Step 4: Test the Lookup Table

There are several methods to test if your Lookup Table has been set up correctly.

Method 1: Enriching test data in the Panther Console or CLI

In the Panther Console's detection editor, click Enrich Test Data to verify if your Lookup Table is working correctly. This allows you to input test data and see the output of the enrichment process within your unit test.

For Enrich Test Data to work, the unit test must have a p_log_type identifying the correct log type. This serves as the basis for Panther's enrichment logic.

Method 2: Checking the panther_rule_matches database

You can verify that your changes have taken effect by checking the panther_rule_matches database for the p_enrichment field. Ensure that the field includes the Lookup Table details you would expect to see.

Method 3: Using SQL queries

You can also perform a LEFT JOIN between event logs and the lookup table in SQL. Ensure that the selector is defined in the query. This allows you to verify if the data from your logs is being correctly matched with the data in your Lookup Table.

For example, this query will attempt to match event data to the Lookup Table using a custom selector (which should be the same as the selector you've defined in the YAML configuration file):

SELECT *
FROM panther_logs.public.<log_type> AS e
LEFT JOIN panther_lookups.public.<lookup_table_name> AS lt
ON e.<field_path> = lt.<field_path>
WHERE e.p_occurs_since('1 day')

Prerequisite

Step 1: Modify the YAML configuration file as needed

  • When modifying the Panther-provided YAML configuration file for Enrichment Providers, you should only ever modify the contents of the AssociatedLogTypes key in order to customize the Selectors. Alterations of other parameters such as Refresh intervals are known to cause issues.

Example

This is because ClientIP is designated as an ip indicator field in the Cloudflare.Firewall schema and the Tor Lookup Table's primary key, ip, is designated as an ip indicator in its own data schema, Tor.ExitNode.

In this example, the tor_exit_nodes Enrichment Provider is being updated to include a new LogType and Selector.

  • Note that the value ofPrimaryKey is ip.

  • The example below shows one AssociatedLogTypes included by default.

LogTypeMap:
  PrimaryKey: ip
  AssociatedLogTypes:
    - LogType: AlphaSOC.Alert
      Selectors:
        - '$.event.srcIP'

Let's add a list item to AssociatedLogTypes that adds support for the ip_address field in the Cloudflare.Firewall schema:

  • Note that the Selectors can be parent fields or JSON paths for nested fields.

LogTypeMap:
  PrimaryKey: ip
  AssociatedLogTypes:
    - LogType: AlphaSOC.Alert
      Selectors:
        - '$.event.srcIP'
    - LogType: Cloudflare.Firewall
      Selectors:
        - "ClientIP"

Step 2: Upload the Enrichment Provider via PAT

panther_analysis_tool upload

You will need to provide an API token and host with --api-token and --api-host, respectively, for the upload to occur. Other options include filtering, minimum tests, and more.

Step 3: Test the Enrichment Provider

There are several methods to test if your Enrichment Provider has been set up correctly.

Method 1: Enriching test data in the Panther Console or CLI

In the Panther Console's detection editor, click Enrich Test Data to verify if your Enrichment Provider is working correctly. This allows you to input test data and see the output of the enrichment process within your unit test.

For Enrich Test Data to work, the unit test must have a p_log_type identifying the correct log type. This serves as the basis for Panther's enrichment logic.

Method 2: Checking the panther_rule_matches database

You can verify that your changes have taken effect by checking the panther_rule_matches database for the p_enrichment field. Ensure that the field includes the LUT or Enrichment Provider details you would expect to see.

Method 3: Using SQL queries

You can also perform a LEFT JOIN between event logs and the Lookup Table in SQL. Ensure that the selector is defined in the query. This allows you to verify if the data from your logs is being correctly matched with the data in your Enrichment Provider.

For example, this query will attempt to match event data to the Enrichment Provider using a custom selector (which should be the same as the selector you've defined in the Enrichment Provider configuration):

SELECT *
FROM panther_logs.public.<log_type> AS e
LEFT JOIN panther_lookups.public.<lookup_table_name> AS lt
ON e.<field_path> = lt.<field_path>
WHERE e.p_occurs_since('1 day')

Remember to review the inferred schema and make any necessary adjustments before uploading it to Panther. For more information about this process, see the .

Once you have created the schema, you can upload it to Panther by following the instructions.

For custom Lookup Tables, you must create the YAML configuration file from scratch. Reference the to see which keys this file must include.

Once you have created your custom Lookup Table configuration file, you can upload it to Panther using the Panther Analysis Tool's :

Verify your Lookup Table is working properly by using PAT's enrich-test-data command. .

A YAML configuration file. You can use the .

If you are enabling an Enrichment Provider (i.e. a Panther-managed Lookup Table), you can modify the configuration file that to meet your needs.

Note than even if this log type (Cloudflare.Firewall) and Selector (ClientIP) weren't set manually in this fashion, Cloudflare.Firewall and p_any_ip_addresses would be added as a log type and Selector, respectively, due to the .

Once you have modified your Enrichment Provider configuration file, you can upload it to Panther using the Panther Analysis Tool's :

Verify your Enrichment Provider is working properly by using PAT's enrich-test-data command. .

pantherlog documentation
Lookup Table Specification Reference
Panther-provided configuration files in panther-analysis
Panther provides
Custom Lookup Table
Panther Analysis Tool (PAT)
pantherlog tool
Enrichment Providers
Anomali ThreatStream
Custom Lookup Tables
Enrichment Providers
panther-analysis repo
Enrichment Provider
be set manually
automatically mapped by indicator fields
automatic mapping by indicator fields
upload command
Learn more about enrich-test-data here
upload command
Learn more about enrich-test-data here
Uploading log schemas with the Panther Analysis Tool