# Rules and Scheduled Rules ## Overview Rules and scheduled rules are segments of logic through which log data is run to detect suspicious activity and generate [signals](https://docs.panther.com/detections/signals) (and optionally [alerts](https://docs.panther.com/alerts)). Rules analyze real-time events, while scheduled rules analyze events queried from your data lake. These detection types differ from [policies](https://docs.panther.com/detections/policies), which apply to cloud resource configurations. Panther provides a number of [Panther-managed rules and scheduled rules](https://docs.panther.com/detections/panther-managed) in Python, which are already written and continuously updated. Custom rules can be created in the Console using the [Simple Detection builder](https://docs.panther.com/detections/rules/simple-detection-builder) or by writing [Python](https://docs.panther.com/detections/rules/python); in the CLI workflow, rules can be written [as Simple Detections (in YAML)](https://docs.panther.com/detections/rules/writing-simple-detections) or in [Python](https://docs.panther.com/detections/rules/python). Scheduled rules can be written in [Python](https://docs.panther.com/detections/rules/python) (in the Console or CLI workflow). {% hint style="info" %} Use the [AI Detection Builder](https://docs.panther.com/detections/rules/ai-builder) to generate detection code from natural language. {% endhint %} If you'd like to create rules or scheduled rules for actions that aren't worthy—at least on their own—of generating an alert, you can [configure them to not generate alerts](https://docs.panther.com/signals#how-to-create-a-rule-that-only-produces-signals) (only generating [signals](https://docs.panther.com/detections/signals)). You may then want to include those rules in a [correlation rule](https://docs.panther.com/detections/correlation-rules). Rules can use [derivation](https://docs.panther.com/detections/rules/derived), which allows you to create one or more Derived Detections from a single Base Detection. You can also [convert Sigma rules into Panther rules](https://docs.panther.com/panther-developer-workflows/converting-sigma-rules). Common examples of rules include analyzing logs for: * Authentication from unknown or unexpected locations * Sensitive API calls, such as administrative changes to SaaS services * Network traffic access to sensitive data sources, such as databases or virtual machines ### Rules vs. scheduled rules Both rules and scheduled rules define logic through which log events are run—but rules analyze real-time events, while scheduled rules analyze events queried from your data lake. * **Rules** * Rules are the default mechanism of analyzing data sent to Panther. Rules work by accepting a defined set of log types such as one or more of the [Supported Logs sources](https://docs.panther.com/data-onboarding/supported-logs), or your own [custom data](https://docs.panther.com/data-onboarding/custom-log-types). Rules have the benefit of low-latency detection and alerting. * Use cases: High-signal logs that do not require joins with other data. * Rules may also be known as streaming rules, real-time rules, and low-latency rules. * **Scheduled rules** * Scheduled rules work by accepting individual rows output from an associated [Scheduled Search](https://docs.panther.com/search/scheduled-searches). * Use cases: Searching windows of time further in the past, running statistical analysis over data, or joining separate data streams. ### Using Python vs. Simple Detections YAML If you are writing rules locally, in the CLI workflow, you can create them as Python or Simple Detections. (Scheduled rules can only be written in Python.) There are advantages to both methods of rule creation, outlined below. * [Write rules in Python](https://docs.panther.com/detections/rules/python) when: * The complexity of the logic is very high, or the detection requires any of the features currently listed as [limitations of Simple Detections YAML](https://docs.panther.com/detections/writing-simple-detections#limitations-of-simple-detections-yaml). * [Write rules as Simple Detections](https://docs.panther.com/detections/rules/writing-simple-detections) when: * You'd like to promote collaboration between team members with differing levels of technical skill. When Simple Detections written in the CLI workflow are uploaded to Panther, they're represented in the Console in the [Simple Detection builder](https://docs.panther.com/detections/rules/simple-detection-builder). You can then edit (or create new) detections in the Console using the Simple Detection builder. For examples of the same detection logic written in both Python and YAML, see [Rule examples, below](#rule-examples). ## How rules and scheduled rules work Rules and scheduled rules both analyze one event at a time. They use event thresholds and de-duplication to create event grouping within windows of time. At a minimum, each rule and scheduled rule must define rule logic. For [Python detections](https://docs.panther.com/detections/rules/python), this means they each must contain a `rule()` function; for [Simple Detections written in the CLI workflow](https://docs.panther.com/detections/rules/writing-simple-detections), each must contain a `Detection` key with at least one [match expression](https://docs.panther.com/detections/rules/writing-simple-detections/match-expression). Python rules can also define [dynamic alert functions](https://docs.panther.com/detections/python#alert-functions-in-python-detections), and Simple Detections can use [dynamic alert keys](https://docs.panther.com/detections/writing-simple-detections#dynamic-alert-keys-in-simple-detections). Both can use [Inline Filters](https://docs.panther.com/detections/rules/inline-filters), which run before detection logic. {% hint style="warning" %} Rules cannot run longer than 15 seconds. If this happens, a [rule error](#rule-errors-and-scheduled-rule-errors) is generated and processing is terminated. (This constraint does not apply to scheduled rules.) {% endhint %} Matches on rules and scheduled rules generate [signals](https://docs.panther.com/detections/signals). When rules and scheduled rules have alerting enabled, rule matches are generated, which can create alerts, according to event threshold and deduplication configurations. [Learn more about the difference between signals, rule matches, and alerts here](https://docs.panther.com/detections/..#signals-vs.-rule-matches-vs.-alerts). ### Title of associated alerts When a detection has alerting enabled, the title of a resulting alert is determined by that detection's configuration. When a separate [deduplication string](#deduplication-of-alerts) is not explicitly set, the alert title is also used to deduplicate alerts. #### How the alert title is set The order of precedence for setting the alert title is as follows: {% tabs %} {% tab title="Python" %} **Alert title precedence in Python:** 1. The output of the dynamic [`title()`](https://docs.panther.com/detections/python#title) function is used. 2. If `title()` is not defined, the value of the detection's display name is used. This is defined: * In the Console: in the **Name** field * In the CLI workflow: in the `DisplayName` field in the YAML file 3. If there is no display name defined, the detection's ID is used. This is defined: * In the Console: in the **ID** field * In the CLI workflow: in the`RuleID` or `PolicyID` field in the YAML file {% endtab %} {% tab title="Simple Detections" %} **Alert title precedence in Simple Detections YAML:** 1. The dynamic alert title is used. This is defined: * In the Console: in the **Optional Fields** > **Default title is** > **Change to** field * In the developer workflow: in the [`AlertTitle`](https://docs.panther.com/detections/writing-simple-detections#alerttitle) field 2. If a dynamic alert title is not defined, the value of the detection's display name is used. This is defined: * In the Console: in the **Name** field * In the developer workflow: in the `DisplayName` field 3. If there is no display name defined, the detection's ID is used. This is defined: * In the Console: in the **ID** field * In the developer workflow: in the`RuleID` or `PolicyID` field {% endtab %} {% endtabs %} ### Deduplication of alerts Events triggering the same detection within its deduplication period that also share a deduplication string will be grouped together in a single alert. Each rule and scheduled rule has a default event threshold of `1` and deduplication period of `1h.` This means all events returning `True` from the `rule` function (with the same deduplication string) will be grouped into a single alert within the hour after first being generated. A rule or scheduled rule with an event threshold of `5` and deduplication period of `15m` would not trigger an alert until five or more events (with the same deduplication string) passed into the `rule` function returned `True` within a 15-minute time period. If your deduplication threshold is greater than one (meaning multiple events must match before an alert is generated), the *first* matching event will be used as the event parameter to [alert functions in Python detections](https://docs.panther.com/detections/python#alert-functions-in-python-detections) and [alert keys in Simple Detections](https://docs.panther.com/detections/writing-simple-detections#dynamic-alert-keys-in-simple-detections). #### Unique value thresholding For Python rules, you can enable unique value threshold detection by defining a `unique()` function. This allows your rule to alert based on the count of distinct values rather than total event count. For example, you could alert when 5+ unique IP addresses attempt to log in, rather than after 5 total login attempts. When a `unique()` function is present: * The threshold applies to the estimated count of unique values returned by `unique()`. * Events are still deduplicated normally based on the deduplication string. * The unique count resets at the end of each deduplication period. See [Unique value threshold detection](https://docs.panther.com/detections/python#unique-value-threshold-detection) for detailed examples and usage. #### Deduplication period When setting the deduplication period, the following constraints apply: * When working in the detection editor in the Panther Console, there is a minimum of **15min** and a maximum of **24h**. * When uploading a detection via the bulk uploader or the [Panther Analysis Tool (PAT)](https://docs.panther.com/panther-developer-workflows/detections-repo/pat), for the `DedupPeriodMinutes` field, the minimum value is `5` and there is no maximum value. The deduplication period is not affected by changing the status of an alert. This means, for example, events will continue to be grouped into the same alert for the length of the deduplication period *even* if an alert's status is changed to `Resolved`. #### How the deduplication string is set The order of precedence for setting the deduplication string is as follows: {% tabs %} {% tab title="Python" %} **Deduplication string precedence in Python:** 1. The output of the dynamic [`dedup()`](https://docs.panther.com/detections/python#dedup) function is used. 2. If `dedup()` is not defined, the alert title is used. The alert title is defined by [the order of precedence defined here](#alert-title-precedence-in-python). {% endtab %} {% tab title="Simple Detections" %} **Deduplication string precedence in Simple Detections YAML:** 1. The value of the [`GroupBy`](https://docs.panther.com/detections/writing-simple-detections#groupby) key is used. 2. If `GroupBy` is not present, the alert title is used. The alert title is defined by [the order of precedence defined here](#alert-title-precedence-in-simple-detections-yaml). {% endtab %} {% endtabs %} Which method you use to set the deduplication string depends on how granularly you want to group alerts: * If you want alerts generated by a certain detection to be grouped separately from alerts triggered by the same detection where one or more event fields have different values, you can use [`title()`](https://docs.panther.com/detections/python#title) or [`AlertTitle`](https://docs.panther.com/detections/writing-simple-detections#alerttitle) (for Python and Simple Detections, respectively) to dynamically set the alert title and therefore, the deduplication string. * If you want to group alerts generated by a certain detection distinctly from alerts generated by the same detection where one or more event field values differ *and* you want to group the alerts based on one or more event fields that weren't used in `title()` or `AlertTitle` (or if you didn't set `title()` or `AlertTitle` at all), you can use [`dedup()`](https://docs.panther.com/detections/python#dedup) or [`GroupBy`](https://docs.panther.com/detections/writing-simple-detections#groupby) (for Python and Simple Detections, respectively) to set the deduplication string. ## How to write rules and scheduled rules You can write rules in the Panther Console (using the [Simple Detection builder](https://docs.panther.com/detections/rules/simple-detection-builder) or [Python](https://docs.panther.com/detections/rules/python)) or locally (in [Simple Detections YAML](https://docs.panther.com/detections/rules/writing-simple-detections) or [Python](https://docs.panther.com/detections/rules/python)); scheduled rules can be written in [Python](https://docs.panther.com/detections/rules/python) (in the Panther Console or locally). {% hint style="info" %} Use the [AI Detection Builder](https://docs.panther.com/detections/rules/ai-builder) to create and modify rules using natural language. Panther AI can generate detection code, add test cases, and explain detection logic directly in the rule editor. {% endhint %} Writing detections locally means creating Python and/or YAML files that define a Panther detection on your own machine. After writing detections locally, you upload the files to your Panther instance, typically via [PAT](https://docs.panther.com/panther-developer-workflows/detections-repo/pat). Note that anything printed to `stdout` or `stderr` by your Python code will be visible in CloudWatch. For SaaS/CPaaS customers, Panther employees can see these CloudWatch logs during routine application monitoring. {% hint style="info" %} Before writing a new rule, check to see if there's an existing [Panther-managed rule](https://docs.panther.com/detections/panther-managed) that meets your needs. If you (or Panther) has already created a rule similar to the one you want to build, consider creating a [Derived Detection](https://docs.panther.com/detections/rules/derived). {% endhint %} ### How to write rules These instructions outline how to set up real-time rules. To configure a scheduled rule, see [How to write scheduled rules](#how-to-write-scheduled-rules). Follow the below instructions to learn how to write rules: * [In Python (in either the Console, or the CLI workflow)](https://docs.panther.com/detections/python#how-to-create-a-rule-in-python) * [In the Simple Detection builder in the Console](https://docs.panther.com/detections/simple-detection-builder#how-to-create-a-rule-in-the-simple-detection-builder) * [As a Simple Detection in the CLI workflow](https://docs.panther.com/detections/writing-simple-detections#how-to-create-a-simple-detection-rule-in-yaml) ### How to write scheduled rules Scheduled rules are associated with one or more [Scheduled Searches](https://docs.panther.com/search/scheduled-searches). If you have not yet created a scheduled search, follow the [How to create a Saved and Scheduled Search](https://docs.panther.com/search/scheduled-searches#how-to-create-a-scheduled-search) instructions first, then return to these instructions to create the scheduled rule. If the Scheduled Search returns multiple rows, each row is processed by the rule logic as a separate event. The number of alerts triggered depends on the deduplication settings you've configured on the scheduled rule. {% hint style="info" %} It's recommended to do as much data processing as is possible in SQL (i.e., in the [Scheduled Search](https://docs.panther.com/search/scheduled-searches)) in order to take advantage of database optimizations and improve rule performance. {% endhint %} Follow the below instructions to learn how to write scheduled rules: * [In Python (in either the Console, or the CLI workflow)](https://docs.panther.com/detections/python#how-to-create-a-scheduled-rule-in-python) ## Rule errors and scheduled rule errors Rule errors and scheduled rule errors are types of [detection errors](https://docs.panther.com/alerts) generated when: * A detection's code raises an exception. * Python processing of a rule or Scheduled Rule runs longer than 15 seconds. If no [alert destination](https://docs.panther.com/alerts/destinations) is configured to receive rule errors (or scheduled rule errors), the alert for a rule error (or scheduled rule error) will be routed to the same destination as the one that would be used for that rule or scheduled rule's alert. See [Routing order precedence on Alert Destinations](https://docs.panther.com/alerts/destinations#routing-order-precedence) for more information. In the event of a [Scheduled Search](https://docs.panther.com/search/scheduled-searches) timeout, a [System Error](https://docs.panther.com/system-configuration/notifications/system-errors) is generated. ## Rule examples See templates for rules and scheduled rules in the [panther-analysis GitHub repository](https://github.com/panther-labs/panther-analysis/tree/master/templates). For in-depth information on how to write detections, see [Writing Python Detections](https://docs.panther.com/detections/rules/python) and [Writing Simple Detections](https://docs.panther.com/detections/rules/writing-simple-detections). ### Auth0 multi-factor authentication policy disabled See how one detection is represented in both Python and as a Simple Detection: {% tabs %} {% tab title="Python" %} **Auth0 multi-factor authentication policy disabled detection in Python** View this detection in full [in the panther-analysis repository in GitHub](https://github.com/panther-labs/panther-analysis/blob/13e49e6589b1160928ec85678884da0e72e986f3/rules/auth0_rules/auth0_mfa_policy_disabled.py). ```python from panther_auth0_helpers import auth0_alert_context, is_auth0_config_event from panther_base_helpers import deep_get def rule(event): data_description = deep_get(event, "data", "description", default="") request_path = deep_get( event, "data", "details", "request", "path", default="" ) request_body = deep_get(event, "data", "details", "request", "body", default=[-1]) return all( [ data_description == "Set the Multi-factor Authentication policies", request_path == "/api/v2/guardian/policies", request_body == [], is_auth0_config_event(event), ] ) ``` {% endtab %} {% tab title="Simple Detection" %} **Auth0 multi-factor authentication policy disabled detection in Simple Detection YAML** ```yaml Detection: - DeepKey: - data - description Condition: Equals Value: Set the Multi-factor Authentication policies - DeepKey: - data - details - request - path Condition: Equals Value: /api/v2/guardian/policies - DeepKey: - data - details - request - body Condition: IsNullOrEmpty - DeepKey: - data - details - request - channel Condition: Equals Value: 'https://manage.auth0.com/' ``` {% endtab %} {% endtabs %} ### Detect and alert when an admin panel is accessed on a web server As an example, let's write a rule to send an alert when an admin panel is accessed on a web server. Take the following NGINX log: ```javascript { "httpReferer": "https://domain1.com/?p=1", "httpUserAgent": "Chrome/80.0.3987.132 Safari/537.36", "remoteAddr": "180.76.15.143", "request": "GET /admin-panel/ HTTP/1.1", "status": 200, "time": "2019-02-06 00:00:38 +0000 UTC" } ``` We want to create a detection that: * Looks for 200 (OK) web requests to any URL containing "admin-panel" * Generates an an alert title that says there has been successful admin panel logins from a specific IP address * Deduplicates events based on IP address {% tabs %} {% tab title="Python" %} ```python def rule(event): return event.get('status') == 200 and 'admin-panel' in event.get('request') def title(event): return f"Successful admin panel login detected from {event.get('remoteAddr')}" def dedup(event): return event.get('remoteAddr') ``` {% endtab %} {% tab title="Simple Detection" %} ```yaml Detection: - KeyPath: status Condition: Equals Value: 200 - KeyPath: request Condition: Contains Value: 'admin-panel' AlertTitle: "Successful admin panel login detected from {remoteAddr}" GroupBy: - KeyPath: remoteAddr ``` {% endtab %} {% endtabs %} Then, the following would occur: 1. An alert would be generated and sent to the set of associated [destinations](https://docs.panther.com/alerts/destinations), which by default are based on the rule severity. 2. The alert's title would be: `Successful admin panel login detected from 180.76.15.143`. 3. Similar events with the same deduplication string of `180.76.15.143` would be appended to the same alert. * A unique alert will be generated for each unique deduplication string, which in this case, is the IP of the requestor. 4. The recipient of the alert could then log into Panther to view all alert metadata and a summary of the events, as well as execute [searches](https://docs.panther.com/search) across all events to perform additional analysis. ## Reference ### Alert severity We recommend following these guidelines to define alert severity levels:
SeverityExploitabilityDescriptionExamples
SeverityExploitabilityDescriptionExamples
InfoNoneNo risk, simply informationalGaining operational awareness
LowDifficultLittle to no risk if exploitedNon-sensitive information leaking such as system time and OS versions
MediumDifficultModerate risk if exploitedExpired credentials, missing protection against accidental data loss, encryption settings, best practice settings for audit tools
HighModerateVery damaging if exploitedLarge gaps in visibility, directly vulnerable infrastructure, misconfigurations directly related to data exposure
CriticalEasyCauses extreme damage if exploitedPublic data/systems available, leaked access keys