Correlation Rules (Beta)

Correlation rules establish correlations across logs, identify anomalies, and model complex attack behavior, then generate alerts

Overview

Correlation rules are in open beta starting with Panther version 1.107, and are available to all customers (except those with Connected Snowflake). Please share any bug reports and feature requests with your Panther support team.

Using correlation rules in Panther, you can track multiple actions across log types. In a correlation rule, you specify a group or specific sequence of signals that must occur in a certain window of time in order to be considered a match—then generating a signal and optionally, an alert.

You can also include the absence of a signal in your correlation rule criteria. Because matches on correlation rules are determined by signals, not by rule matches or alerts, it's possible to include rules, scheduled rules, and correlation rules that have alerting disabled.

Correlation rules may be particularly useful if you want to generate an alert when, for example:

  • A certain Okta user logs in successfully after at least one hundred unsuccessful login attempts, then logs in to AWS as a root user (see full example below)

  • Advanced security settings were disabled for a GitHub repository, which then was not archived (see full example below)

Learn how to create correlation rules below, and more about the YAML keys that make up correlation rules on Correlation Rule Reference.

How correlation rules work

Correlation rules are written in YAML and reference previously created rules, scheduled rules, and/or correlation rules. Each correlation rule runs on a schedule, and defines a "lookback window," i.e., the amount of time in the past the rule should look to find signals (or absences of signals).

You can apply additional criteria to your correlation rule, such as:

  • Requiring a minimum number of signals to be found for a certain rule, or a maximum

  • Requiring certain event values to match from one rule to another (e.g., requiring signals for all individual rules to contain the same IP address)

  • Requiring subsequent steps in a sequence to have occurred within a certain amount of time

It is not currently possible to write tests for a correlation rule. The tests for rules and/or scheduled rules referenced in a correlation rule do not affect the correlation rule.

What happens when there is a match on a correlation rule

Matches on correlation rules generate signals. When a correlation rule has alerting enabled, rule matches are generated, which can create alerts according to the correlation rule's deduplication configuration. Learn more about the difference between signals, rule matches, and alerts here.

When an alert is generated for a correlation rule, the individual rules, scheduled rules, and correlation rules referenced in the correlation rule may or may not also generate their own alerts. This depends on:

  • Whether each individual rule, scheduled rule, and correlation rule has alerting enabled

  • The Threshold value on individual rules or scheduled rules, and the MinMatchCount value on the correlation rule. An edge case in which the correlation rule could generate an alert but not the rules that make it up is if the event threshold for an individual rule (set with the Threshold key) is higher than the MinMatchCount on the correlation rule, and the number of actual rule matches is somewhere in the middle.

Group vs. sequence

There are two types of correlation rules: groups and sequences. Both group and sequence correlation rules define a collection of rules for which signals must be found (or not found, by defining Absence: true).

Group correlation rules will generate an alert if all rules have produced signals (or absences), no matter the order in which signals were found. Sequence rules, however, define a particular order in which signals (or absences) must be found in order to generate an alert.

Deduplication of events

The deduplication period in correlation rules is the value of the LookbackWindowMinutes field. This means overlapping correlation rules within the same LookbackWindowMinutes time frame will only contain the unique events that caused that correlation rule to match.

Deduplication set on individual rules and scheduled rules referenced in a correlation rule (with dedup(), DedupPeriodMinutes, Threshold, or set in the Console) is not applicable to the correlation rule.

Correlation rule errors

If you have used incorrect syntax to construct a correlation rule, you will receive a Simple Detections error code.

If execution of your correlation rule fails, you will receive a detection error.

If your correlation rule times out (likely due to a too-large LookbackWindowMinutes value), you will receive a system error.

Group correlation rules

A group correlation rule defines a collection of rules for which signals (or a lack of signals) must occur in a given lookback window. The signals can occur in any order.

If you would like the collection of events to occur in a specific order, use a sequence correlation rule instead.

MatchCriteria

In a group correlation rule, the MatchCriteria key defines fields, per rule, scheduled rule, and correlation rule, that must have matching values in order for the correlation rule to pass.

For rules associated to multiple log types, scheduled rules, or correlation rules, only p_ fields can be matched on. (For rules associated to only one log type, any field may be matched on.)

If match criteria is not defined, because there is no requirement for certain event field values to match, the correlation rule is less specific.

Learn more about MatchCriteria on Correlation Rule Reference.

Group examples

The examples below refer to these JSON events:

Example events

These sample events contain a subset of fields, for readability.

[
  {
    "p_rule_id": "Standard.BruteForceByIP",
    "event_type": "failed_login",
    "p_event_time": "2023-12-08 10:27:03.496000000",
    "p_alert_context": {
      "ip": "136.24.229.58",
      "geolocation": "San Francisco, California in United States"
    }
  },
  {
    "p_rule_id": "Standard.BruteForceByIP",
    "event_type": "failed_login",
    "p_event_time": "2023-12-08 10:27:03.623000000",
    "p_alert_context": {
      "ip": "136.24.229.58",
      "geolocation": "San Francisco, California in United States"
    }
  },
  {
    "p_rule_id": "Standard.BruteForceByIP",
    "event_type": "failed_login",
    "p_event_time": "2023-12-08 10:27:03.861000000",
    "p_alert_context": {
      "ip": "136.24.229.58",
      "geolocation": "San Francisco, California in United States"
    }
  },
  {
    "p_rule_id": "Okta.Login.Success",
    "event_type": "successful_login",
    "p_event_time": "2023-12-08 10:27:54.317000000",
    "p_alert_context": {
      "ip": "136.24.229.58",
      "geolocation": "San Francisco, California in United States"
    }
  },
  {
    "p_rule_id": "AWS.Console.RootLogin",
    "eventType": "AwsApiCall",
    "p_event_time": "2023-12-08 10:30:13.274000000",
    "p_alert_context": {
      "sourceIPAddress": "136.24.229.58"
    }
  }
]

In this example, MatchCriteria specifies that the IP field in all four rules must contain the same value.

Detection:
  - Group:
      - ID: &failed_login Failed Login
        RuleID: Standard.BruteForceByIP
        MinMatchCount: 7
      - ID: &successful_login Successful Login
        RuleID: Okta.Login.Success
      - ID: &root_access Root Access
        RuleID: AWS.Console.RootLogin
      - ID: &missing_crowdstrike Missing Crowdstrike 
        RuleID: Crowdstrike.Detection.passthrough
        Absence: true
    MatchCriteria:
      ip:
        - GroupID: *failed_login
          Match: p_alert_context.ip
        - GroupID: *successful_login
          Match: p_alert_context.ip
        - GroupID: *root_access
          Match: p_alert_context.sourceIPAddress
        - GroupID: *missing_crowdstrike
          Match: p_any_ip_addresses
    EventEvaluationOrder: Chronological
    LookbackWindowMinutes: 60
    Schedule:
      RateMinutes: 30
      TimeoutMinutes: 7

Sequence correlation rules

A sequence correlation rule defines a collection of rules for which signals (or a lack of signals) must occur in a specific order within a given lookback window.

The order of the sequence is defined by the order of rules defined within the Sequence key.

If you would like the correlation rule to match merely if all rules have signals (or absences), without requiring a specific order, use a group correlation rule instead.

Transitions

Within a sequence, you can optionally define transitions. Transitions define additional criteria for how one step in a sequence can traverse to the next, including how much time can occur between steps, as well as which event fields must have matching values. Using Transitions with WithinTimeFrameMinutes and/or Match increases the specificity of your correlation rule.

If transitions are defined, there must be one fewer transition than the number of rules included in Sequence. Additionally, the items within Transitions must be in the same order as the Sequence list.

Currently, there can only be one type of field matched on per correlation rule (e.g., all IP address fields or all email address fields). For rules associated to multiple log types, scheduled rules, or correlation rules, only p_ fields can be matched on. (For rules associated to only one log type, any field may be matched on.)

Learn more about transitions on Correlation Rule Reference.

Sequence examples

The examples below refer to these JSON events:

Example events

These sample events contain a subset of fields, for readability.

[
  {
    "p_rule_id": "Standard.BruteForceByIP",
    "event_type": "failed_login",
    "p_event_time": "2023-12-08 10:27:03.496000000",
    "p_alert_context": {
      "ip": "136.24.229.58",
      "geolocation": "San Francisco, California in United States"
    }
  },
  {
    "p_rule_id": "Standard.BruteForceByIP",
    "event_type": "failed_login",
    "p_event_time": "2023-12-08 10:27:03.623000000",
    "p_alert_context": {
      "ip": "136.24.229.58",
      "geolocation": "San Francisco, California in United States"
    }
  },
  {
    "p_rule_id": "Standard.BruteForceByIP",
    "event_type": "failed_login",
    "p_event_time": "2023-12-08 10:27:03.861000000",
    "p_alert_context": {
      "ip": "136.24.229.58",
      "geolocation": "San Francisco, California in United States"
    }
  },
  {
    "p_rule_id": "Okta.Login.Success",
    "event_type": "successful_login",
    "p_event_time": "2023-12-08 10:27:54.317000000",
    "p_alert_context": {
      "ip": "136.24.229.58",
      "geolocation": "San Francisco, California in United States"
    }
  },
  {
    "p_rule_id": "AWS.Console.RootLogin",
    "eventType": "AwsApiCall",
    "p_event_time": "2023-12-08 10:30:13.274000000",
    "p_alert_context": {
      "sourceIPAddress": "136.24.229.58"
    }
  }
]

Using Transitions with Match within Sequence is useful if you'd like to define event fields whose values must match.

Detection:
  - Sequence:
      - ID: Failed Login
        RuleID: Standard.BruteForceByIP
        MinMatchCount: 7
      - ID: Successful Login
        RuleID: Okta.Login.Success
        MinMatchCount: 1
      - ID: Root Login
        RuleID: AWS.Console.RootLogin
        MinMatchCount: 1
      - ID: Missing Crowdstrike 
        RuleID: Crowdstrike.Detection.passthrough
        Absence: true
    Transitions:
      - ID: Brute Force Login Success
        From: Failed Login
        To: Successful Login
        WithinTimeFrameMinutes: 10
        Match:
          - On: client.ipAddress
      - ID: Gained Root Access
        From: Successful Login
        To: Root Login
        Match:
          - From: client.ipAddress
            To: p_alert_context.sourceIPAddress
      - ID: Absence of Crowdstrike
        From: Root Login
        To: Missing Crowdstrike
        Match:
          - From: p_alert_context.sourceIPAddress
            To: p_any_ip_addresses
    LookbackWindowMinutes: 60
    EventEvaluationOrder: Chronological
    Schedule:
      RateMinutes: 30
      TimeoutMinutes: 3

Limitations of Correlation detections

  • It is not yet possible to add tests for a correlation rule.

  • If a sequence uses transitions:

    • The number of transitions allowed is one fewer than the number of rules included in the sequence collection.

    • The order of items in the Transitions list must correspond to the order of rules in the Sequence field.

  • If you use event field matching:

    • The value of the previous Match.On/Match.From must match the next Match.On/Match.To value.

    • For rules associated to more than one log type, scheduled rules, or correlation rules, values for Match.On/Match.From/Match.To or MatchCriteria.Match must be one of the p_ fields listed on Standard Fields.

    • Only one type of field may be matched on throughout the correlation rule. For example, only IP addresses can be matched on, or only email addresses.

How to create a correlation rule

You can write correlation rules in the Panther Console or locally. For explanations of the YAML keys used to construct a correlation rule, see Correlation Rule Reference.

Using the flow chart visualizer in the Console

You can use the flow chart visualizer in the Panther Console while working with Correlation Rules. This renders the correlation rules visually while you construct them in YAML, and the UI provides immediate validation feedback on your rules.

Creating a correlation rule in YAML in the Console

To create a correlation rule in the Panther Console, you can either select detections from the list view to generate YAML, or construct the YAML yourself.

  1. In the left-hand navigation bar of your Panther Console, click Build > Detections.

  2. In the list of detections, click the checkbox on the left-hand side of each detection you'd like to include in your correlation rule.

    • The order in which you click the detections will be the order of the generated sequence correlation rule.

  3. Click Correlate.

  4. On the detection creation page, finish configuring your correlation rule:

    • Name: Enter a descriptive name for the correlation rule.

    • ID (optional): Click the pen icon and enter a unique ID for your correlation rule.

    • In the upper-right corner, the Enabled toggle will be set to ON by default. If you'd like to disable the rule, flip the toggle to OFF.

    • In the Detection section:

    • In the Create Alert section, set the Create Alert ON/OFF toggle. This indicates whether an alert should be created when there are matches, or only a signal. If you set this toggle to ON:

      • Severity: Select a severity level for the alerts triggered by this detection.

      • In the Optional Fields section, optionally provide values for the following fields:

        • Description: Enter additional context about the rule.

        • Runbook: Enter the procedures and operations relating to this rule.

        • Reference: Enter an external link to more information relating to this rule.

        • Destination Overrides: Choose destinations to receive alerts for this detection, regardless of severity. Note that destinations can also be set dynamically, in the rule function. See Routing Order Precedence to learn more about routing precedence.

        • Summary Attributes: Enter the attributes you want to showcase in the alerts that are triggered by this detection.

        • Tags: Enter custom tags to help you understand the rule at a glance (e.g., HIPAA.)

        • In the Framework Mapping section:

          1. Click Add New to enter a report.

          2. Provide values for the following fields:

            • Report Key: Enter a key relevant to your report.

            • Report Values: Enter values for that report.

  5. In the upper-right corner, click Deploy.

Creating a correlation rule in YAML in the CLI workflow

Expand instructions

If you're writing Correlations detections locally (instead of in the Panther Console), we recommend managing your local detection files in a version control system like GitHub or GitLab.

Folder setup

If you group your correlation rules into folders, each folder name must contain rules in order for them to be found during upload (using either PAT or the bulk uploader in the Console).

File setup

Each correlation rule consists of:

  • A YAML specification file (a file with a .yml extension) containing the detection logic, as well as metadata attributes of the detection.

More information about correlation detection YAML syntax can be found on Correlation Rule Reference, including a full list of required and optional fields.

  • Create a YAML file (e.g. my_new_correlation_rule.yml) using the template below (including a top-level Detection key):

    AnalysisType: correlation_rule
    DisplayName: Example Correlation Rule to Check the Format of the Spec
    Enabled: true
    RuleID: Correlation.Type.Behavior.MoreContext
    Severity: High
    Reports:
      ReportName (like CIS, MITRE ATT&CK):
        - The specific report section relevant to this correlation rule
    Tags:
      - Tags
      - Go
      - Here
    Description: >
      This correlation rule exists to validate the CLI workflows of the Panther CLI
    Runbook: >
      First, find out who wrote this the spec format, then notify them with feedback.
    Reference: https://www.a-clickable-link-to-more-info.com
    Detection:
      - Sequence:
          - ID: Failed Login
            RuleID: Okta.Login.Fail
            MinMatchCount: 7
          - ID: Successful Login
            RuleID: Okta.Login.Success
            MinMatchCount: 1
        LookbackWindowMinutes: 15
        Schedule:
          RateMinutes: 5
          TimeoutMinutes: 3

After this rule is uploaded to Panther, it will be viewable in the Console.

Correlation rule full examples

Discovering exfiltrated GitHub credentials

This Discovering.Exfiltrated.Credentials correlation rule checks every 10 minutes to see if there has been a signal for the AWS.CloudTrail.IaaS rule (defined in the second tab below) not followed by a signal for the GitHub.CICD rule (defined in the third tab below) in the last 10 minutes.

AnalysisType: correlation_rule
RuleID: 'Discovering.Exfiltrated.Credentials'
DisplayName: 'Discovering.Exfiltrated.Credentials'
Enabled: true
Severity: High
Description: >
  There was at least one IaaS activity match not followed 
  by a CI/CD activity within 10 minutes. 
Detection:
  - Sequence:
      - ID:  &iaas_activity IaaS Activity
        RuleID: AWS.CloudTrail.IaaS
      - ID: &cicd_activityCICD CICD Activity
        RuleID: Github.CICD
        Absence: true
    Transitions: 
      - From: *iaas_activity
        To: *cicd_activity
        WithinTimeFrameMinutes: 10
    Schedule:
      RateMinutes: 10
      TimeoutMinutes: 3T

Brute force Okta login to AWS root login

This Brute.Force.Login correlation rule checks every 30 minutes to see if there has been a signal for the Standard.BruteForceByIP rule followed by a signal for the Okta.Login.Success rule (defined in the second tab below) followed by a signal for the AWS.Console.RootLogin rule, with additional time frame and event IP value matching requirements.

AnalysisType: correlation_rule
RuleID: 'Brute.Force.Login'
DisplayName: 'Brute.Force.Login'
Enabled: true
Severity: High
Description >
  At least 100 failed logins followed by a successful login
  that occurred within 10 minutes of the last failed login
  and then a root login from the AWS console. 
Detection:
  - Sequence:
      - ID: &failed Failed Login
        RuleID: Standard.BruteForceByIP
        MinMatchCount: 100
      - ID: &success Successful Login
        RuleID: Okta.Login.Success
      - ID: &root Root Login
        RuleID: AWS.Console.RootLogin
    Transitions:
      - ID: Brute Force Login Success
        From: *failed
        To: *success
        WithinTimeFrameMinutes: 10
        Match:
          - From: p_alert_context.ip
            To: client.ipAddress
      - ID: Gained Root Access
        From: *success
        To: *root
        Match:
          - From: client.ipAddress
            To: p_alert_context.sourceIPAddress
    LookbackWindowMinutes: 60
    EventEvaluationOrder: Chronological
    Schedule:
      RateMinutes: 30
      TimeoutMinutes: 3

GitHub repository security policy disabled without subsequent archival

This Github.Repo.Security.Policy.Disabled.Without.Archival correlation rule checks every 10 minutes to see if there has been a signal for the GitHub.Advanced.Security.Change rule not followed by a signal for the GitHub.Repo.Archived rule (defined in the second tab below) where there is also a matching value for the p_alert_context.repo event field within the last 10 minutes.

AnalysisType: correlation_rule
RuleID: 'Github.Repo.Security.Policy.Disabled.Without.Archival'
DisplayName: 'Github Repo Security Policy Disabled Without Archival'
Enabled: true
Tags:
  - Github
  - Repo Archived
Severity: High
Detection:
  - Sequence:
      - ID: &github_advanced_security_change GitHub Advanced Security Change
        RuleID: GitHub.Advanced.Security.Change
      - ID: &github_repo_archived Github Repo Archived
        RuleID: Github.Repo.Archived
        Absence: true
    Transitions:
      - ID: TR1
        From: *github_advanced_security_change
        To: *github_repo_archived
        WithinTimeFrameMinutes: 10
        Match:
          - On: p_alert_context.repo
    Schedule:
      RateMinutes: 10
      TimeoutMinutes: 3

Last updated