Alerts & Destinations

Panther detections trigger alerts on suspicious behavior

Overview

Alerts are generated when your rules, scheduled rules, or policies detect suspicious behavior. When an alert is triggered, it is routed to the appropriate Alert Destination(s) using the Alert routing scenarios.

Panther can generate three types of alerts:

Alerts: This classification includes rule matches, policy matches, and scheduled rule matches from enabled detections.
Detection errors: These are generated due to incorrect code or permissions issues. When this occurs, a rule returns an error and the rule does not complete its run successfully. This includes rule errors and scheduled rule errors.
System errors: Panther's System Errors alert users when a part of the Panther platform is not functioning correctly. This includes log source inactivity, log classification failures, log source permission failures, alert delivery failures, and cloud account scanning failures.
- To learn more about these errors, see System Errors.

You can also interact with alerts using the Panther GraphQL API.

Customizing alerts

You can customize the content of the alerts you receive for detection matches by using the alert functions in Python detections or alert keys in YAML detections.

These functions allow you to, for example, include the matching event (using alert_context() or AlertContext) or add event values to the alert title (using title() or AlertTitle).

Limiting alerts

The alert limiter functionality is intended to safeguard from "alert storms" arising from (likely) misconfigured detections.

If a single detection creates 1,000 alerts within one hour, its CreateAlert field (or Create Alert toggle in the Console) is set to False (or OFF in the Console), which stops the detection from creating additional alerts. (The detection will continue to generate signals on matches.) When this happens, you will receive a System Error notification and alert notifying you of the change.

You can set the CreateAlert/Create Alert value back to True/ON when you are ready—perhaps after some detection tuning.

If you would like to set this alert limit lower than 1,000, please reach out to your Panther Support team.

Alerts with multiple events

If a detection has set a deduplication period and deduplication string, all events matching the detection that share the same deduplication string will be appended to the first alert created, for the length of the deduplication period. This can result in an alert with more than one event associated to it.

Any alert information generated dynamically from a detection (by the alert functions in Python detections or dynamic alert keys in Simple Detections) is generated by the output of the first matching event. The output of the alert functions/keys for additional events attached to the alert does not alter the alert.

For example, if a detection uses severity() to dynamically set the alert severity based on an event property and the first matching event dictates a LOW severity, the alert severity will be set to LOW indefinitely—even if an event later associated to the alert, run through the same severity() function, would dictate a HIGH severity.

Working with alerts

Receiving an alert

To receive alerts outside of the Panther Console, set up an alert destination and ensure it's configured to receive alerts based on the Alert routing scenarios.

When deciding how to receive an alert, consider the following options:

Natively supported destinations: Panther supports a number of alert destinations natively, like Slack, Jira, and Amazon SNS. See the Alert Destinations documentation for a complete list of supported destinations.
Custom Webhook: If you'd like to route alerts to a destination that is not natively supported but that has an API, you can use the Custom Webhook option to send alert notifications—see the Custom Webhook documentation for configuration instructions.
Panther's API: If you'd like to receive Panther alerts at a destination that is not natively supported and that does not have an API, you can receive alerts by polling Panther's API for alerts on a schedule. See the available API operations for fetching and manipulating alerts on Alerts & Errors.

Investigating an alert

When you receive an alert to your configured destination, it’s time to investigate. Common investigation workflows include:

Reviewing data from the Panther Console overview dashboard.
Using Search to investigate indicators of compromise (IoCs).
Using Data Explorer to search robustly using SQL.

See the Investigations & Search documentation for more information on Panther’s data analysis tools.

Triaging and managing alerts

For more information on triaging, assigning, and managing alerts, see the following documentation pages:

Assigning and Managing Alerts
- Triage, use alert summaries, assign, un-assign, view alert history, and add comments to alerts.
- Quickly tune detections directly from alerts.
- You can also manage alerts in Slack when using the Slack Bot alert destination.
Alert Runbooks
- For many Panther-managed detections, find recommended steps to remediate the issue that triggered the alert.

Viewing alerts

Viewing the alerts list in the Panther Console

Log in to your Panther Console.
- The landing page is the overview dashboard, where you can view alert metrics and a list of the alerts assigned to you. Continue on to the next steps to view a list of all alerts.
Click Alerts in the left sidebar.
- By default, this page lists alerts from most recent to oldest and displays only Open and Triaged alerts.
Filter the list as needed to narrow your results.
Click the tabs at the top of the list to filter by the alert type: Alerts, Detection Errors, or System Errors.

Viewing alert details in the Panther Console

While viewing the list of alerts as described above, click an alert title to view the alert details page:

The alert details page includes:

Basic information.
- This includes the detection that triggered the alert, the associated log types, the assignee, the alert status, the MITRE tactic, and the alert runbook.
A list of event matches.
- For each event, you can view event time, event source, the associated p_log_type and p_source_label, and IP information.
External conversations.
- If there is additional context in a Slack Boomerang, Jira ticket, or Asana task where an alert was delivered, you can click the links in this section to view that information.
- You can also add comments to an alert in this section.
Alert history.
- This includes a history of all status changes and comments.
- If the alert failed to deliver to one of the configured alert destinations, you will see an "Alert delivery failed" error above the history.