Correlation Rules (Beta)

Correlation rules establish correlations across logs, identify anomalies, and model complex attack behavior, then generate alerts

Overview

Correlation rules are in open beta starting with Panther version 1.108, and are available to all customers. Please share any bug reports and feature requests with your Panther support team.

Using correlation rules in Panther, you can track multiple actions across log types. In a correlation rule, you specify a group or specific sequence of signals that must occur in a certain window of time in order to be considered a match—then generating a signal and optionally, an alert.

You can also include the absence of a signal in your correlation rule criteria. Because matches on correlation rules are determined by signals, not by rule matches or alerts, it's possible to include rules, scheduled rules, and correlation rules that have alerting disabled.

Correlation rules may be particularly useful if you want to generate an alert when, for example:

• A certain Okta user logs in successfully after at least one hundred unsuccessful login attempts, then logs in to AWS as a root user (see full example below)

• Advanced security settings were disabled for a GitHub repository, which then was not archived (see full example below)

Learn how to create custom correlation rules below, and more about the YAML keys that make up correlation rules on Correlation Rule Reference. You can also leverage Panther-managed correlation rules.

Using correlation rules may increase your Snowflake compute costs. To learn how to make your correlation rules more cost efficient, see the guidelines in Making correlation rules more efficient, below.

How correlation rules work

Correlation rules are written in YAML and reference previously created rules, scheduled rules, and/or correlation rules. Each correlation rule runs on a schedule, and defines a "lookback window," i.e., the amount of time in the past the rule should look to find signals (or absences of signals).

You can apply additional criteria to your correlation rule, such as:

• Requiring a minimum number of signals to be found for a certain rule, or a maximum

• Requiring certain event values to match from one rule to another (e.g., requiring signals for all individual rules to contain the same IP address)

• Requiring subsequent steps in a sequence to have occurred within a certain amount of time

What happens when there is a match on a correlation rule

Matches on correlation rules generate signals. When a correlation rule has alerting enabled, rule matches are generated, which can create alerts according to the correlation rule's deduplication configuration. Learn more about the difference between signals, rule matches, and alerts here.

When an alert is generated for a correlation rule, the individual rules, scheduled rules, and correlation rules referenced in the correlation rule may or may not also generate their own alerts. This depends on:

• Whether each individual rule, scheduled rule, and correlation rule has alerting enabled

• The Threshold value on individual rules or scheduled rules, and the MinMatchCount value on the correlation rule. An edge case in which the correlation rule could generate an alert but not the rules that make it up is if the event threshold for an individual rule (set with the Threshold key) is higher than the MinMatchCount on the correlation rule, and the number of actual rule matches is somewhere in the middle.

Group vs. sequence

There are two types of correlation rules: groups and sequences. Both group and sequence correlation rules define a collection of rules for which signals must be found (or not found, by setting Absence: true).

Group correlation rules will generate an alert if all rules have produced signals (or absences), no matter the order in which signals were found. Sequence rules, however, define a particular order in which signals (or absences) must be found in order to generate an alert.

Setting the schedule and lookback window

Each correlation rule defines both of the following:

• A schedule: Defined by the Schedule field (which uses either RateMinutes or CronExpression), the schedule indicates how frequently the correlation rule should run.
• A lookback window: Defined by the LookbackWindowMinutes field, the lookback window specifies the number of minutes in the past the correlation rule should look to find signals (or absences of signals) for the rules, scheduled rules, or correlation rules included in its group or sequence.

When setting the values of Schedule and LookbackWindowMinutes, it's recommended to take a few factors into account:

• How important it is for you to be alerted in a timely manner for matches on this correlation rule

  • For example, it might suffice to run lower priority correlation rules only every 24 hours; alternatively, you might want to run higher priority correlation rules every 15 minutes.

• How far apart in time the first and last signal the correlation rule is looking for can be to be considered part of the same occurrence that should generate a match

  • See Ensuring signals aren't split between lookback windows
• The maximum latency expected from the data source(s) evaluated by the rules associated to the correlation rule

  • See Accounting for log source latency in LookbackWindowMinutes

Schedule and LookbackWindowMinutes values can have an impact on Snowflake compute costs. See Making correlation rules more efficient for more information.

Ensuring signals aren't split between lookback windows

It's recommended to set your lookback window such that you won't miss occurrences of the thing you're searching for because the required signals were split across the lookback windows of multiple correlation rule runs.

Example of the situation to avoid

To illustrate this situation we're trying to avoid, take this simplified correlation rule configuration:

 # Bad example; do not replicate
 - Group:
    - RuleID: First.Rule
    - RuleID: Second.Rule
    - RuleID: Third.Rule
   Schedule:
     RateMinutes: 60
   LookbackWindowMinutes: 90

In this scenario, every hour, the correlation rule looks back over the previous 90 minutes to find signals for the three rules included in the Group. Let's say each of these rules matched/generated a signal at the times below:

• Signal for First.Rule generated at 12:58 PM

• Signal for Second.Rule generated at 1:05 PM

• Signal for Third.Rule generated at 1:40 PM

The correlation rule is scheduled to run at 30 minutes past the hour. It runs at the following times:

• 1:30 PM (looking back to 12:00 PM): Sees only the first and second signals (12:58 and 1:05 PM), and therefore does not generate a match.

• 2:30 PM (looking back to 1:00 PM): Sees only the second and third signals (1:05 and 1:40 PM), and again does not generate a match.

It's likely that the author of this correlation rule would have expected that if signals for all three rules were generated within 42 minutes (between 12:58 and 1:40 PM), the correlation rule would have matched. However, because all three of the required signals weren't found within the lookback window of a single run of the correlation rule, a match was not generated.

To ensure the signals required for your correlation rule to match are not split across different runs (i.e., they are included in the same lookback window), first determine how far apart in time the first and last signal can be in order to be considered part of the same occurrence that should be alerted on. For convenience, let's call this value "maximum signal timespan minutes."

(This should not be confused with WithinTimeFrameMinutes, which is the time frame within which two steps in a sequence must occur in order to pass. "Maximum signal timespan minutes" and WithinTimeFrameMinutes can be equal if the correlation rule is a sequence and specifies only two steps.)

It's recommended to configure your LookbackWindowMinutes value so that every possible window of length "maximum signal timespan minutes" is covered within at least one run of the correlation rule. It's generally possible to do this with the following formula:

• If RateMinutes <= "maximum signal timespan minutes":

  • LookbackWindowMinutes = 2 x "maximum signal timespan minutes" + log ingest latency

• If RateMinutes > "maximum signal timespan minutes":

  • LookbackWindowMinutes = "maximum signal timespan minutes" + RateMinutes + log ingest latency

To understand the inclusion of log ingest latency, see Accounting for log source latency in LookbackWindowMinutes

Accounting for log source latency in LookbackWindowMinutes

Log ingest latency is the maximum latency expected from the data sources evaluated by the rules associated to the correlation rule.

Because signals are fetched within the lookback window based on the time their associated event occurred (p_event_time), not the time the event was ingested into Panther (p_parse_time), it's necessary to take ingest latency into account when determining LookbackWindowMinutes to ensure you are processing all "new" data since the last time the correlation rule ran.

For example, if you have configured your correlation rule to run every hour (e.g., by setting RateMinutes to 60) and the source of the logs processed by the rules associated to the correlation rule states that it may delay log forwarding to Panther by up to three hours, you might set LookbackWindowMinutes to 60 + 3*60, or 240. To further illustrate this, consider that because of the three-hour ingestion delay, data received by Panther at 9:01am can have a p_event_time of as early as 6:01am. If the correlation rule runs every hour, on the hour, at 10:00am, it would need to look back to at least 6:01am.

Deduplication of events

The deduplication period in correlation rules is the value of the LookbackWindowMinutes field. This means overlapping correlation rules within the same LookbackWindowMinutes time frame will only contain the unique events that caused that correlation rule to match.

Deduplication set on individual rules and scheduled rules referenced in a correlation rule (with dedup(), DedupPeriodMinutes, Threshold, or set in the Console) is not applicable to the correlation rule.

Correlation rule errors

While working with correlation rules, you may receive one of the following errors:

• Simple Detections error code: You have used incorrect syntax to construct a correlation rule
• Detection error: Execution of your correlation rule has failed
• System error: Your correlation rule has timed out (likely due to a too-large LookbackWindowMinutes value)

Group correlation rules

A group correlation rule defines a collection of rules for which signals (or a lack of signals) must occur in a given lookback window. The signals can occur in any order.

If you would like the collection of events to occur in a specific order, use a sequence correlation rule instead.

MatchCriteria

In a group correlation rule, the MatchCriteria key defines fields, per rule, scheduled rule, and correlation rule, that must have matching values in order for the correlation rule to pass.

For rules associated to multiple log types, scheduled rules, or correlation rules, only p_ fields can be matched on. (For rules associated to only one log type, any field may be matched on.)

If match criteria is not defined, because there is no requirement for certain event field values to match, the correlation rule is less specific.

Learn more about MatchCriteria on Correlation Rule Reference.

MinMatchCount

In a group correlation rule, MinMatchCount is an optional field that specifies the minimum number of individual rules, scheduled rules, or correlation rules (defined in Group) that must match in order for this correlation rule to match.

For example, if you list five rules within Group and add MinMatchCount: 2, the correlation rule will match if any two of the five rules generate a signal.

MinMatchCount is also an available field on individual rules defined in Group. See both fields in use together in the Group with MinMatchCount example, below.

Group examples

The examples below refer to these JSON events:

Example events

These sample events contain a subset of fields, for readability.

[
  {
    "p_rule_id": "Standard.BruteForceByIP",
    "event_type": "failed_login",
    "p_event_time": "2023-12-08 10:27:03.496000000",
    "p_alert_context": {
      "ip": "136.24.229.58",
      "geolocation": "San Francisco, California in United States"
    }
  },
  {
    "p_rule_id": "Standard.BruteForceByIP",
    "event_type": "failed_login",
    "p_event_time": "2023-12-08 10:27:03.623000000",
    "p_alert_context": {
      "ip": "136.24.229.58",
      "geolocation": "San Francisco, California in United States"
    }
  },
  {
    "p_rule_id": "Standard.BruteForceByIP",
    "event_type": "failed_login",
    "p_event_time": "2023-12-08 10:27:03.861000000",
    "p_alert_context": {
      "ip": "136.24.229.58",
      "geolocation": "San Francisco, California in United States"
    }
  },
  {
    "p_rule_id": "Okta.Login.Success",
    "event_type": "successful_login",
    "p_event_time": "2023-12-08 10:27:54.317000000",
    "p_alert_context": {
      "ip": "136.24.229.58",
      "geolocation": "San Francisco, California in United States"
    }
  },
  {
    "p_rule_id": "AWS.Console.RootLogin",
    "eventType": "AwsApiCall",
    "p_event_time": "2023-12-08 10:30:13.274000000",
    "p_alert_context": {
      "sourceIPAddress": "136.24.229.58"
    }
  }
]

Group with MatchCriteria

In this example, MatchCriteria specifies that the IP field in all four rules must contain the same value.

Detection:
  - Group:
      - ID: &failed_login Failed Login
        RuleID: Standard.BruteForceByIP
        MinMatchCount: 7
      - ID: &successful_login Successful Login
        RuleID: Okta.Login.Success
      - ID: &root_access Root Access
        RuleID: AWS.Console.RootLogin
      - ID: &missing_crowdstrike Missing Crowdstrike 
        RuleID: Crowdstrike.Detection.passthrough
        Absence: true
    MatchCriteria:
      ip:
        - GroupID: *failed_login
          Match: p_alert_context.ip
        - GroupID: *successful_login
          Match: p_alert_context.ip
        - GroupID: *root_access
          Match: p_alert_context.sourceIPAddress
        - GroupID: *missing_crowdstrike
          Match: p_any_ip_addresses
    EventEvaluationOrder: Chronological
    LookbackWindowMinutes: 60
    Schedule:
      RateMinutes: 30
      TimeoutMinutes: 7

Group without MatchCriteria

In this example, if signals are found for the first three rules but not the fourth one, the correlation rule will pass.

Detection:
  - Group:
      - RuleID: Standard.BruteForceByIP
        MinMatchCount: 7
      - RuleID: Okta.Login.Success
      - RuleID: AWS.Console.RootLogin
      - RuleID: Crowdstrike.Detection.passthrough
        Absence: true
    LookbackWindowMinutes: 60
    Schedule:
      RateMinutes: 30
      TimeoutMinutes: 3

Group with MinMatchCount

In this example, the MinMatchCount value under Detection specifies that if signals are found for any two of the three rules defined in Group, the correlation rule will match. Note there is also a MinMatchCount defined on a rule in Group, which is used in combination with the outer-level MinMatchCount. See the list below for examples of when this correlation rule would pass.

Detection:
  - Group:
      - RuleID: Standard.BruteForceByIP
        MinMatchCount: 7
      - RuleID: Okta.Login.Success
      - RuleID: AWS.Console.RootLogin
    MinMatchCount: 2
    LookbackWindowMinutes: 60
    Schedule:
      RateMinutes: 30
      TimeoutMinutes: 3

The above correlation rule would pass in any of the following scenarios:

• When there is at least one signal from Okta.Login.Success and at least one signal from AWS.Console.RootLogin

• When there is at least seven signals from Standard.BruteForceByIP and at least one signal from Okta.Login.Success

• When there is at least seven signals from Standard.BruteForceByIP and at least one signal from AWS.Console.RootLogin

• When there are signals from all three rules (with at least seven signals from Standard.BruteForceByIP)

Sequence correlation rules

A sequence correlation rule defines a collection of rules for which signals (or a lack of signals) must occur in a specific order within a given lookback window.

The order of the sequence is defined by the order of rules defined within the Sequence key.

If you would like the correlation rule to match merely if all rules have signals (or absences), without requiring a specific order, use a group correlation rule instead.

Transitions

Within a sequence, you can optionally define transitions. Transitions define additional criteria for how one step in a sequence can traverse to the next, including how much time can occur between steps, as well as which event fields must have matching values. Using Transitions with WithinTimeFrameMinutes and/or Match increases the specificity of your correlation rule.

If transitions are defined, there must be one fewer transition than the number of rules included in Sequence. Additionally, the items within Transitions must be in the same order as the Sequence list.

Currently, there can only be one type of field matched on per correlation rule (e.g., all IP address fields or all email address fields). For rules associated to multiple log types, scheduled rules, or correlation rules, only p_ fields can be matched on. (For rules associated to only one log type, any field may be matched on.)

Learn more about transitions on Correlation Rule Reference.

Sequence examples

The examples below refer to these JSON events:

Example events

These sample events contain a subset of fields, for readability.

[
  {
    "p_rule_id": "Standard.BruteForceByIP",
    "event_type": "failed_login",
    "p_event_time": "2023-12-08 10:27:03.496000000",
    "p_alert_context": {
      "ip": "136.24.229.58",
      "geolocation": "San Francisco, California in United States"
    }
  },
  {
    "p_rule_id": "Standard.BruteForceByIP",
    "event_type": "failed_login",
    "p_event_time": "2023-12-08 10:27:03.623000000",
    "p_alert_context": {
      "ip": "136.24.229.58",
      "geolocation": "San Francisco, California in United States"
    }
  },
  {
    "p_rule_id": "Standard.BruteForceByIP",
    "event_type": "failed_login",
    "p_event_time": "2023-12-08 10:27:03.861000000",
    "p_alert_context": {
      "ip": "136.24.229.58",
      "geolocation": "San Francisco, California in United States"
    }
  },
  {
    "p_rule_id": "Okta.Login.Success",
    "event_type": "successful_login",
    "p_event_time": "2023-12-08 10:27:54.317000000",
    "p_alert_context": {
      "ip": "136.24.229.58",
      "geolocation": "San Francisco, California in United States"
    }
  },
  {
    "p_rule_id": "AWS.Console.RootLogin",
    "eventType": "AwsApiCall",
    "p_event_time": "2023-12-08 10:30:13.274000000",
    "p_alert_context": {
      "sourceIPAddress": "136.24.229.58"
    }
  }
]

Sequence w/ Transitions and Match

Using Transitions with Match within Sequence is useful if you'd like to define event fields whose values must match.

Detection:
  - Sequence:
      - ID: Failed Login
        RuleID: Standard.BruteForceByIP
        MinMatchCount: 7
      - ID: Successful Login
        RuleID: Okta.Login.Success
        MinMatchCount: 1
      - ID: Root Login
        RuleID: AWS.Console.RootLogin
        MinMatchCount: 1
      - ID: Missing Crowdstrike 
        RuleID: Crowdstrike.Detection.passthrough
        Absence: true
    Transitions:
      - ID: Brute Force Login Success
        From: Failed Login
        To: Successful Login
        WithinTimeFrameMinutes: 10
        Match:
          - On: client.ipAddress
      - ID: Gained Root Access
        From: Successful Login
        To: Root Login
        Match:
          - From: client.ipAddress
            To: p_alert_context.sourceIPAddress
      - ID: Absence of Crowdstrike
        From: Root Login
        To: Missing Crowdstrike
        Match:
          - From: p_alert_context.sourceIPAddress
            To: p_any_ip_addresses
    LookbackWindowMinutes: 60
    EventEvaluationOrder: Chronological
    Schedule:
      RateMinutes: 30
      TimeoutMinutes: 3

Sequence w/ Transitions w/o Match

Using Transitions within Sequence without Match is useful if you'd just like to specify a time frame within which two steps in a sequence must occur, using WithinTimeFrameMinutes.

Detection:
  - Sequence:
      - ID: Failed Login
        RuleID: Standard.BruteForceByIP
        MinMatchCount: 7
      - ID: Successful Login
        RuleID: Okta.Login.Success
      - ID: Root Login
        RuleID: AWS.Console.RootLogin
      - ID: Missing Crowdstrike 
        RuleID: Crowdstrike.Detection.passthrough
        Absence: true
    Transitions:
      - ID: Brute Force Login Success
        From: Failed Login
        To: Successful Login
        WithinTimeFrameMinutes: 10 # Define timeframe
      - ID: Gained Root Access
        From: Successful Login
        To: Root Login
      - ID: Absence of Crowdstrike
        From: Root Login
        To: Missing Crowdstrike
    LookbackWindowMinutes: 60
    Schedule:
      RateMinutes: 30
      TimeoutMinutes: 3

Sequence w/o Transitions

You can omit Transitions from within Sequence if you don't want to specify event fields to match on or a time frame within which steps must occur.

Detection:
  - Sequence:
      - RuleID: Standard.BruteForceByIP
        MinMatchCount: 7
      - RuleID: Okta.Login.Success
      - RuleID: AWS.Console.RootLogin
      - RuleID: Crowdstrike.Detection.passthrough
        Absence: true
    LookbackWindowMinutes: 60
    Schedule:
      RateMinutes: 30
      TimeoutMinutes: 3

Testing correlation rules

You can add unit tests to a correlation rule to evaluate whether, given certain conditions, a match on the correlation rule would be generated (potentially creating an alert—see What happens when there is a match on a correlation rule).

Correlation rule tests exist to test the correlation logic only. To test the rule logic of the individual rules making up the correlation rule, use the unit tests on the individual rules themselves.

Unit tests are defined on a correlation rule within the Unit Tests tab (in the Panther Console) or the top-level Tests field (in the CLI workflow), and are structured similarly to unit tests for rules or policies.

Each unit test on a correlation rule must include a Name, ExpectedResult, and RuleOutputs field. Learn more about the YAML structure for unit tests, including how to construct a RuleOutputs value here, on Correlation Rule Reference.

After writing tests for correlation rules, you can run them using the Panther Analysis Tool test command. Running pat test for correlation rules requires an API token—see Authenticating with an API token for more information.

Unit test examples

Using the following correlation rule:

- Sequence:
    - ID: OktaLoginFailure
      RuleID: Okta.Login.Failure
      MinMatchCount: 10
    - ID: OktaLoginSuccess
      RuleID: Okta.Login.Success
      MinMatchCount: 1
    - ID: RootLogin
      RuleID: AWS.Console.RootLogin
      MinMatchCount: 1
  
  Transitions:
    - ID: Okta Brute Force Login
      From: OktaLoginFailure
      To: OktaLoginSuccess
      WithinTimeFrameMinutes: 10
      Match:
        - On: client.ipAddress
    - ID: AWS Root Login
      From: OktaLoginSuccess
      To: RootLogin
      Match:
        - From: client.ipAddress
          To: srcIpAddress7
  
  Schedule:
    RateMinutes: 15
    TimeoutMinutes: 2
  
  LookbackWindowMinutes: 60

We can write the following tests:

If the below tests were included in the rule's YAML file (as is required when managing your detections in the CLI workflow), they would be positioned under a Tests key.

Test w/absolute timestamps

If there are ten login attempts followed by a successful login, then a root login, we expect the correlation rule to return true. This test uses absolute timestamps—learn more about how to use absolute timestamps here, on Correlation Rule Reference.

Name: Successful Login with Timestamps
ExpectedResult: true
RuleOutputs:
  - ID: OktaLoginFailure
    Matches:
      client.ipAddress:
      # Since the original rule has a MinMatchCount of 10, we need at least 10 timestamps
        123.123.123.123: ["2006-01-02T15:04:05Z", "2006-01-02T15:04:06Z","2006-01-02T15:04:07Z", "2006-01-02T15:04:08Z", "2006-01-02T15:04:09Z", "2006-01-02T15:04:10Z", "2006-01-02T15:04:11Z", "2006-01-02T15:04:12Z", "2006-01-02T15:04:13Z", "2006-01-02T15:04:14Z"]
  - ID: OktaLoginSuccess
    Matches:
      client.ipAddress:
        123.123.123.123: ["2006-01-02T15:04:15Z"]
  - ID: RootLogin
    Matches:
      srcIpAddress7:
        123.123.123.123: ["2006-01-02T15:04:16Z"]

Test w/relative timestamps

If there are ten login attempts followed by a successful login, then a root login, we expect the correlation rule to return true. This test uses relative timestamps—learn more about how to use relative timestamps here, on Correlation Rule Reference.

Name: Successful Login with Relative Minutes
ExpectedResult: true
RuleOutputs:
  - ID: OktaLoginFailure
    Matches:
      client.ipAddress:
      # Since the original rule has a MinMatchCount of 10, we need at least 10 timestamps
        123.123.123.123: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
  - ID: OktaLoginSuccess
    Matches:
      client.ipAddress:
        123.123.123.123: [11]
  - ID: RootLogin
    Matches:
      srcIpAddress7:
        123.123.123.123: [12]

Test w/ExpectedResult: false

This tests expects the rule to evaluate to false because there are only nine login attempts followed by a successful login, not 10.

Name: Nine Login Failures Before Success
ExpectedResult: false
RuleOutputs:
  - ID: OktaLoginFailure
    Matches:
      client.ipAddress:
        123.123.123.123: [1, 2, 3, 4, 5, 6, 7, 8, 9]
  - ID: OktaLoginSuccess
    Matches:
      client.ipAddress:
        123.123.123.123: [10]
  - ID: RootLogin
    Matches:
      srcIpAddress7:
        123.123.123.123: [11]

Limitations of correlation rules

• If a sequence uses transitions:

  • The number of transitions allowed is one fewer than the number of rules included in the sequence collection.

  • The order of items in the Transitions list must correspond to the order of rules in the Sequence field.

• If you use event field matching:

  • The value of the previous Match.On/Match.From must match the next Match.On/Match.To value.

  • For rules associated to more than one log type, scheduled rules, or correlation rules, values for Match.On/Match.From/Match.To or MatchCriteria.Match must be one of the p_ fields listed on Standard Fields.
  • Only one type of field may be matched on throughout the correlation rule. For example, only IP addresses can be matched on, or only email addresses.

How to create a correlation rule

You can write correlation rules in the Panther Console or locally. For explanations of the YAML keys used to construct a correlation rule, see Correlation Rule Reference.

In addition to creating custom correlation rules, you can also leverage Panther-managed correlation rules, available in the correlation_rules directory of the panther-analysis repository.

Using the flow chart visualizer in the Console

You can use the flow chart visualizer in the Panther Console while working with Correlation Rules. This renders the correlation rules visually while you construct them in YAML, and the UI provides immediate validation feedback on your rules.

Creating a correlation rule in YAML in the Console

To create a correlation rule in the Panther Console, you can either select detections from the list view to generate YAML, or construct the YAML yourself.

Select rules and generate YAML

1. In the left-hand navigation bar of your Panther Console, click Detections.

2. In the list of detections, click the checkbox on the left-hand side of each detection you'd like to include in your correlation rule.

   • The order in which you click the detections will be the order of the generated sequence correlation rule.
3. Click Correlate.
4. On the detection creation page, finish configuring your correlation rule:

   • Name: Enter a descriptive name for the correlation rule.

   • ID (optional): Click the pen icon and enter a unique ID for your correlation rule.

   • In the upper-right corner, the Enabled toggle will be set to ON by default. If you'd like to disable the rule, flip the toggle to OFF.

   • Under the YAML Editor tab:

     • If desired, make modifications to the generated correlation rule. The rule, by default:

       • Is a Sequence correlation rule.

         • You can change this to Group by clicking Organize Rules as Group. Learn more in Switching between sequence and group in the Console.
       • Sets MinMatchCount to 1 for all rules and scheduled rules included in the correlation rule.
       • Within Transitions, sets Match.On for each transition to p_any_usernames.

         • You can update the Match.On values, or instead use Match.To and Match.From.
       • Sets Schedule.RateMinutes to 15 and Schedule.TimeoutMinutes to 2.
       • Sets LookbackWindowMinutes to 60.
     • More information about correlation detection YAML syntax can be found on Correlation Rule Reference, including a full list of required and optional fields.
   • Under the Alert Settings tab:

     • Within the Basics tab, configure the following fields:

       • Create Alert: This ON/OFF toggle indicates whether an alert should be created when there are matches, or only a signal.
       • (Only applicable if Create Alert is set to ON) Severity: Select a severity level for the alerts triggered by this detection.
       • (Only applicable if Create Alert is set to ON) Destination Overrides: Optionally choose destinations to receive alerts for this detection, regardless of severity. Note that destinations can also be set dynamically, in the rule function. See Routing Order Precedence to learn more about routing precedence.

       
     • (Only applicable if Create Alert is set to ON) Within the Context sub-tab, optionally provide values for the following fields:

       • Description: Enter additional context about the rule.

       • Runbook: Enter the procedures and operations relating to this rule.

         • To see examples of runbooks for built-in rules, see Alert Runbooks.
       • Reference: Enter an external link to more information relating to this rule.

       • Summary Attributes: Enter the attributes you want to showcase in the alerts that are triggered by this detection.

         • To use a nested field as a summary attribute, use the Snowflake dot notation in the Summary Attribute field to traverse a path in a JSON object:

           <column>:<level1_element>.<level2_element>.<level3_element>

           The alert summary will then be generated for the referenced object in the alert. Learn more about traversing semi-structured data in Snowflake here.
         • For more information on Alert Summaries, see Assigning and Managing Alerts.
       • Tags: Enter custom tags to help you understand the rule at a glance (e.g., HIPAA.)

       • In the Framework Mapping section:

         1. Click Add New to enter a report.

         2. Provide values for the following fields:

            • Report Key: Enter a key relevant to your report.

            • Report Values: Enter values for that report.

   • Under the Unit Tests tab, optionally add tests:

     1. Click + Add new unit test.

        • Boilerplate code for a test for this correlation rule will be populated.

     2. Make any necessary adjustments to the populated text, and fill in the remainder of the test, including the contents of Matches.

     3. Below the code editor, click Run Test to evaluate the test.
5. In the upper-right corner, click Deploy.

Write YAML from scratch

1. In the left-hand navigation bar of your Panther Console, click Build > Detections.

2. Click Create New.

3. In the Select Detection Type modal, choose Correlation Rule.

4. On the detection creation page, fill in the following fields:

   • Name: Enter a descriptive name for the correlation rule.

   • ID (optional): Click the pen icon and enter a unique ID for your correlation rule.

   • In the upper-right corner, the Enabled toggle will be set to ON by default. If you'd like to disable the rule, flip the toggle to OFF.

   • Under the YAML Editor tab, define your correlation rule.

     • By default, the correlation rule is organized as a Sequence. You can change this to Group by clicking Organize Rules as Group. Learn more in Switching between sequence and group in the Console.
     • More information about correlation detection YAML syntax can be found on Correlation Rule Reference, including a full list of required and optional fields.
   • Under the Alert Settings tab:

     • Within the Basics tab, configure the following fields:

       • Create Alert: This ON/OFF toggle indicates whether an alert should be created when there are matches, or only a signal.
       • (Only applicable if Create Alert is set to ON) Severity: Select a severity level for the alerts triggered by this detection.
       • (Only applicable if Create Alert is set to ON) Destination Overrides: Optionally choose destinations to receive alerts for this detection, regardless of severity. Note that destinations can also be set dynamically, in the rule function. See Routing Order Precedence to learn more about routing precedence.

       
     • (Only applicable if Create Alert is set to ON) Within the Context sub-tab, optionally provide values for the following fields:

       • Description: Enter additional context about the rule.

       • Runbook: Enter the procedures and operations relating to this rule.

         • To see examples of runbooks for built-in rules, see Alert Runbooks.
       • Reference: Enter an external link to more information relating to this rule.

       • Summary Attributes: Enter the attributes you want to showcase in the alerts that are triggered by this detection.

         • To use a nested field as a summary attribute, use the Snowflake dot notation in the Summary Attribute field to traverse a path in a JSON object:

           <column>:<level1_element>.<level2_element>.<level3_element>

           The alert summary will then be generated for the referenced object in the alert. Learn more about traversing semi-structured data in Snowflake here.
         • For more information on Alert Summaries, see Assigning and Managing Alerts.
       • Tags: Enter custom tags to help you understand the rule at a glance (e.g., HIPAA.)

       • In the Framework Mapping section:

         1. Click Add New to enter a report.

         2. Provide values for the following fields:

            • Report Key: Enter a key relevant to your report.

            • Report Values: Enter values for that report.

   • Under the Unit Tests tab, optionally add tests:

     1. Click + Add new unit test.

        • Boilerplate code for a test for this correlation rule will be populated.

     2. Make any necessary adjustments to the populated text, and fill in the remainder of the test, including the contents of Matches.

     3. Below the code editor, click Run Test to evaluate the test.
5. In the upper-right corner, click Deploy.

Switching between sequence and group in the Console

You can switch your correlation rule from a sequence to a group, and vice versa, using the Organize Rules as Group/Sequence button at the top right of the YAML Editor panel.

If your correlation rule is already a sequence and you click Organize Rules as Group:

• The Sequence key will be changed to Group.

• The Transitions section will be removed.

• A MatchCriteria section will be added.

  • If you have previously edited a MatchCriteria section, that version will be added. Otherwise, a default MatchCriteria section will be provided.

If your correlation rule is already a group and you click Organize Rules as Sequence:

• The Group key will be changed to Sequence.

• The MatchCriteria section will be removed.

• A Transitions section will be added.

  • If you have previously edited a Transitions section, that version will be added. Otherwise, a default Transitions section will be provided.

Creating a correlation rule in YAML in the CLI workflow

Expand instructions

If you're writing Correlations detections locally (instead of in the Panther Console), we recommend managing your local detection files in a version control system like GitHub or GitLab.

Folder setup

If you group your correlation rules into folders, each folder name must contain rules in order for them to be found during upload (using either PAT or the bulk uploader in the Console).

File setup

Each correlation rule consists of:

• A YAML specification file (a file with a .yml extension) containing the detection logic, as well as metadata attributes of the detection.

More information about correlation detection YAML syntax can be found on Correlation Rule Reference, including a full list of required and optional fields.

• Create a YAML file (e.g. my_new_correlation_rule.yml) using the template below (including a top-level Detection key):

  AnalysisType: correlation_rule
  DisplayName: Example Correlation Rule to Check the Format of the Spec
  Enabled: true
  RuleID: Correlation.Type.Behavior.MoreContext
  Severity: High
  Reports:
    ReportName (like CIS, MITRE ATT&CK):
      - The specific report section relevant to this correlation rule
  Tags:
    - Tags
    - Go
    - Here
  Description: >
    This correlation rule exists to validate the CLI workflows of the Panther CLI
  Runbook: >
    First, find out who wrote this the spec format, then notify them with feedback.
  Reference: https://www.a-clickable-link-to-more-info.com
  Detection:
    - Sequence:
        - ID: Failed Login
          RuleID: Okta.Login.Fail
          MinMatchCount: 7
        - ID: Successful Login
          RuleID: Okta.Login.Success
          MinMatchCount: 1
      LookbackWindowMinutes: 15
      Schedule:
        RateMinutes: 5
        TimeoutMinutes: 3
  Tests:
    - Name: Rule should return false if no matches
      ExpectedResult: false
      RuleOutputs:
        - ID: Failed Login
          Matches: {} # Empty mapping implies no matches

After this rule is uploaded to Panther, it will be viewable in the Console.

Correlation rule full examples

Discovering exfiltrated GitHub credentials

This Discovering.Exfiltrated.Credentials correlation rule checks every 10 minutes to see if there has been a signal for the AWS.CloudTrail.IaaS rule (defined in the second tab below) not followed by a signal for the GitHub.CICD rule (defined in the third tab below) in the last 10 minutes.

Discovering Exfiltrated Credentials

AnalysisType: correlation_rule
RuleID: 'Discovering.Exfiltrated.Credentials'
DisplayName: 'Discovering.Exfiltrated.Credentials'
Enabled: true
Severity: High
Description: >
  There was at least one IaaS activity match not followed 
  by a CI/CD activity within 10 minutes. 
Detection:
  - Sequence:
      - ID: IaaS Activity
        RuleID: AWS.CloudTrail.IaaS
      - ID: CICD Activity
        RuleID: Github.CICD
        Absence: true
    Transitions: 
      - From: IaaS Activity
        To: CICD Activity
        WithinTimeFrameMinutes: 10
    Schedule:
      RateMinutes: 10
      TimeoutMinutes: 3
Tests:
  - Name: IaaS Activity without CICD Activity
    ExpectedResult: true
    RuleOutputs:
      - ID: IaaS Activity
        Matches:
          username: 
            my_username: [1]
  - Name: IaaS Activity with CICD Activity
    ExpectedResult: false
    RuleOutputs:
      - ID: IaaS Activity
        Matches:
          username: 
            my_username: [1]
      - ID: CICD Activity
        Matches:
          username: 
            my_username: [2]

AWS.CloudTrail.IaaS

AnalysisType: rule
RuleID: 'AWS.CloudTrail.IaaS'
DisplayName: 'AWS CloudTrail IaaS'
Enabled: true
LogTypes:
  - AWS.CloudTrail
Severity: Info
CreateAlert: false
Detection:
  - KeyPath: userIdentity.arn
    Condition: IsIn
    Values:
      - DeploymentUpdateGitHubRole
  - KeyPath: eventName
    Condition: IsIn
    Values:
      - StartSession
      - ListResources
      - UpdateResource
      - DescribeResource
      - WriteLog

GitHub.CICD

AnalysisType: rule
RuleID: 'GitHub.CICD'
DisplayName: 'GitHub CICD'
Enabled: true
LogTypes:
  - GitHub.Audit
Severity: Info
CreateAlert: false
Detection:
  - KeyPath: repository
    Condition: Equals
    Value: panther-labs/example-repo
  - KeyPath: action
    Condition: Equals
    Value: workflows.created_workflow_run
  - KeyPath: name
    Condition: Equals
    Value: CI

Brute force Okta login to AWS root login

This Brute.Force.Login correlation rule checks every 30 minutes to see if there has been a signal for the Standard.BruteForceByIP rule followed by a signal for the Okta.Login.Success rule (defined in the second tab below) followed by a signal for the AWS.Console.RootLogin rule, with additional time frame and event IP value matching requirements.

Brute Force Login

AnalysisType: correlation_rule
RuleID: 'Brute.Force.Login'
DisplayName: 'Brute.Force.Login'
Enabled: true
Severity: High
Description: >
  At least 100 failed logins followed by a successful login
  that occurred within 10 minutes of the last failed login
  and then a root login from the AWS console. 
Detection:
  - Sequence:
      - ID: Failed Login
        RuleID: Standard.BruteForceByIP
        MinMatchCount: 100
      - ID: Successful Login
        RuleID: Okta.Login.Success
        MinMatchCount: 1
      - ID: Root Login
        RuleID: AWS.Console.RootLogin
        MinMatchCount: 1
    Transitions:
      - ID: Brute Force Login Success
        From: Failed Login
        To: Successful Login
        WithinTimeFrameMinutes: 10
        Match:
          - From: p_alert_context.ip
            To: client.ipAddress
      - ID: Gained Root Access
        From: Successful Login
        To: Root Login
        Match:
          - From: client.ipAddress
            To: p_alert_context.sourceIPAddress
    LookbackWindowMinutes: 60
    EventEvaluationOrder: Chronological
    Schedule:
      RateMinutes: 30
      TimeoutMinutes: 3
Tests:
# See additional tests for this correlation rule above, in "Unit test examples"
  - Name: Successful Login with Relative Minutes
    ExpectedResult: true
    RuleOutputs:
      - ID: Failed Login
        Matches:
          client.ipAddress:
          # Since the original rule has a MinMatchCount of 10, we need at least 10 timestamps
            123.123.123.123: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
      - ID: Successful Login
        Matches:
          client.ipAddress:
            123.123.123.123: [11]
      - ID: Root Login
        Matches:
          srcIpAddress7:
            123.123.123.123: [12]

Okta.Login.Success

AnalysisType: rule
RuleID: 'Okta.Login.Success'
DisplayName: 'Okta Login Success'
Enabled: true
LogTypes:
  - Okta.SystemLog
Severity: Info
CreateAlert: false
Detection:
  - DeepKey:
      - outcome
      - result
    Condition: Equals
    Value: SUCCESS
  - Key: eventType
    Condition: Equals
    Value: user.session.start

GitHub repository security policy disabled without subsequent archival

This Github.Repo.Security.Policy.Disabled.Without.Archival correlation rule checks every 10 minutes to see if there has been a signal for the GitHub.Advanced.Security.Change rule not followed by a signal for the GitHub.Repo.Archived rule (defined in the second tab below) where there is also a matching value for the p_alert_context.repo event field within the last 10 minutes.

Github.Repo.Security.Policy.Disabled.Without.Archival

AnalysisType: correlation_rule
RuleID: 'Github.Repo.Security.Policy.Disabled.Without.Archival'
DisplayName: 'Github Repo Security Policy Disabled Without Archival'
Enabled: true
Tags:
  - Github
  - Repo Archived
Severity: High
Detection:
  - Sequence:
      - ID: GitHub Advanced Security Change
        RuleID: GitHub.Advanced.Security.Change
      - ID: Github Repo Archived
        RuleID: Github.Repo.Archived
        Absence: true
    Transitions:
      - ID: TR1
        From: GitHub Advanced Security Change
        To: Github Repo Archived
        WithinTimeFrameMinutes: 10
        Match:
          - On: p_alert_context.repo
    Schedule:
      RateMinutes: 10
      TimeoutMinutes: 3
Tests:
  - Name: Github Repo Security Policy Disabled Without Archival
    ExpectedResult: true
    RuleOutputs:
      - ID: GitHub Advanced Security Change
        Matches:
          p_alert_context.repo:
            my_production_repo: [1]
  - Name: Github Repo Security Policy Disabled With Archival
    ExpectedResult: false
    RuleOutputs:
      - ID: GitHub Advanced Security Change
        Matches:
          p_alert_context.repo:
            my_production_repo: [1]
      - ID: Github Repo Archived
        Matches:
          p_alert_context.repo:
            my_production_repo: [2]

GitHub.Repo.Archived

AnalysisType: rule
RuleID: 'GitHub.Repo.Archived'
DisplayName: 'GitHub Repo Archived'
Enabled: true
LogTypes:
  - GitHub.Audit
Severity: Info
CreateAlert: false
AlertContext:
  - KeyName: repo
    KeyValue:
      KeyPath: repo
Detection:
  - KeyPath: action
    Condition: Equals
    Value: repo.archived

Making correlation rules more efficient

Correlation rules use complex pattern recognitions, which means they have the potential to be computationally expensive. To reduce Snowflake costs associated with correlation rules, keep the following guidelines in mind.

Run the correlation rule as infrequently as possible

How often a correlation rule runs—determined by its Schedule value—can have a large impact on its cost. It's therefore recommended to ensure your correlation rule is not running more often than it needs to (while still meeting your detection needs—see Setting Schedule, above, for considerations when configuring this field).

A general guideline when thinking about the relationship between a correlation rule's run frequency and cost is: Each time you double the interval on which the correlation rule is run (e.g. by doubling RateMinutes), you halve the cost it generates.

Set LookbackWindowMinutes as low as possible

The amount of data a correlation rule runs over is largely defined by its LookbackWindowMinutes value, and this data quantity is a leading factor in the correlation rule's processing time and resulting cost. It's recommended to reduce the amount of data your correlation rule processes by setting its LookbackWindowMinutes value as low as possible (while still meeting your detection needs—see Setting LookbackWindowMinutes, above, for considerations when configuring this field).

As an example, say you'd like to identify when two rules each generate a signal within 10 minutes of one another (using WithinTimeFrameMinutes). While setting LookbackWindowMinutes to exactly 10 is not advisable, you can safely use a value of as low as 15.

Choose match fields with the lowest cardinality possible

Match field cardinality is positively correlated with correlation rule cost. If your correlation rule uses event value matching, when choosing which fields to match on, it's recommended to use fields that have lower cardinality.

Cardinality of a match field can be influenced by a few factors, including:

• The number of possible values the field can have—the more possible values, the higher the cardinality.

  • For example, if field_a can have one of three possible values (e.g., "yellow", "red", or "blue"), but field_b will only ever have one of two values (e.g., "purple" or "green"), field_b has lower cardinality than field_a.

• The field's data type—typically, fields with a non-scalar data type (i.e., that are an array or object) have higher cardinality than fields with a scalar data type (i.e., string, boolean, or number).

  • For example, if your log schema designates both an email and username field as a username indicator, meaning your p_any_usernames field will include them both in an array (e.g., p_any_usernames: ["Bob Smith", "[e-mail]"]), that p_any_usernames field will have a higher cardinality than the email field, which has a string type with a single value.