# System Errors

## Overview

Panther's System Errors alert you when a part of the Panther platform is not functioning as expected. This includes the following:

* [Log Source Health Notifications](#log-source-health-notifications)
  * Log sources turning unhealthy as a result of a failed health check
  * Logs dropping off entirely from a log source
* [Alert limit notifications](https://docs.panther.com/alerts#limiting-alerts)
* [Alert Delivery Failure](#alert-delivery-failure)
  * Alerts failing to deliver to the Alert Destination
* [Log Classification Errors](#log-classification-alerts)
  * Logs failing to classify
* [S3 GetObject Errors](#s3-getobject-error-notifications)
  * Panther failing to fetch S3 objects
* [Cloud Security Scanning Failure](#cloud-security-scanning-failure)
  * Panther failing to scan a cloud resource because of an "access denied" error
* Query errors in [Scheduled Searches](https://docs.panther.com/search/scheduled-searches) not connected to a Scheduled Rule
  * Query errors in [Scheduled Searches](https://docs.panther.com/search/scheduled-searches) connected to a Scheduled Rule are surfaced as [rule errors](https://docs.panther.com/detections/rules#rule-errors-and-scheduled-rule-errors)
* [Timed out Scheduled Searches](https://docs.panther.com/search/scheduled-searches#use-limits-in-scheduled-searches)

These types of alerts are classified as `System Errors` in Panther. `System Errors` will always have a `CRITICAL` severity level—and be sent to alert destinations [configured to receive `System Errors`](#configuring-an-alert-destination-for-system-health-errors), even if they are not configured to receive alerts with a `CRITICAL` severity. They are automatically generated, with the exception of log drop-off alarms which you can configure manually per log source. `System Error` alerts are visible in your Panther Console within **Alerts & Errors > System Errors**.

{% hint style="info" %}
It's strongly recommended to configure an [alert destination](https://docs.panther.com/alerts/destinations) to receive the `System Error` alert type.
{% endhint %}

System Errors are also a type of in-app notification in your Panther Console. Learn more about notifications on [Notifications and Errors](https://docs.panther.com/system-configuration/notifications).

## How to configure System Error alarms

To ensure that you receive alerts for all types of System Errors:

* Configure an alert destination that is receiving the `System Error` alert type.
* Configure Log Drop-off alarms for log sources that will trigger an alert when data is no longer being received.
  * Note that you do not need to enable alerts for Log Classification errors, Alert Delivery failure, S3 GetObject errors, and Cloud Security Scanning failure.

### Configuring an Alert Destination for System Errors

By default, Panther will send `System Errors` alerts to the **Alerts** page in your Panther Console. It is also strongly recommended to configure one of your alert destinations to receive them.

{% hint style="info" %}
Alert destinations configured to receive `System Errors` will receive them even if the destination is not configured to receive alerts with a `CRITICAL` severity.
{% endhint %}

To ensure these alerts are sent to a custom Alert Destination, follow the steps below:

1. Log in to your Panther Console.
2. On the left sidebar navigation, click **Configure** > **Alert Destinations**
3. Choose an existing Alert Destination or [add a new Alert Destination](https://docs.panther.com/alerts/destinations).
4. On the configuration page for the Alert Destination, add `System Errors` to the **Alert Types** section:

<figure><img src="https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-482c5eae16a7277e8dd1cbcd334541057bdaf2b2%2Fimage%20(1).png?alt=media" alt="On the configuration screen for a Slack destination in the Panther Console, &#x22;System Errors&#x22; is one of the selected Alert Types." width="563"><figcaption></figcaption></figure>

### Configuring log drop-off alarms for log sources

Panther allows you to set up event threshold alarms for individual log sources, which will trigger an alert if data is not received over a specific time interval.

For example, if you configure the threshold to 15 minutes, then you will receive an alert if no events are processed in 15 minutes.

This can be useful for log sources that have been incorrectly linked to Panther or are experiencing issues outside of Panther.

{% hint style="warning" %}
When the threshold is crossed, a single alert is generated. No further alerts will be issued unless the threshold condition is reset (i.e., data is received) and subsequently triggered again.
{% endhint %}

You can add an alarm to a new or an existing log source:

{% tabs %}
{% tab title="Add alarm to new log source" %}
**Setting up an alarm for a new log source**

1. In the left-hand navigation bar of your Panther Console, click **Configure** > **Log Sources**.
2. In the upper-right corner, click **Create New**.
3. Complete each step of the onboarding workflow.
   * See [Data Sources and Transports](https://docs.panther.com/data-onboarding) for specific setup instructions by source.
4. On the success page at the end of the onboarding workflow, the **Trigger an alert when no events are processed** defaults to **YES**. Leave this enabled.
   * Enter your desired time period by filling in the **Number** and **Period** fields next to **How long should Panther wait before it sends you an alert that no events have been processed?**.

<figure><img src="https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-820da60240fd5ed816d9aafdab93e9a020ce2d03%2Fimage.png?alt=media" alt="The &#x22;Trigger an alert when no events are processed&#x22; toggle is set to YES. The &#x22;How long should Panther wait before it sends you an alert that no events have been processed&#x22; setting is set to 1 Day" width="480"><figcaption></figcaption></figure>
{% endtab %}

{% tab title="Add alarm to existing source" %}
**Setting up an alarm for an existing log source**

1. In the left-hand navigation bar of your Panther Console, click **Configure** > **Log Sources**.
2. Select the log source to configure an alarm for.
3. On the log source's details page, in the **Overview** tab, click the pencil icon to the right of the value in the **Drop-off Alarm** field.\
   ![A log source named "test cloudtrail" is shown, and in the Basic Info section there is a blurred-out Source ID and AWS Account ID. There's also a Drop-off Alarm with a value of Not Set.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-caf04b73ea608668db44a0f1c9caa9b36052b148%2FScreenshot%202023-05-09%20at%203.58.13%20PM.png?alt=media)
4. In the pop-up window, toggle the **Trigger an alert when no events are processed** setting to **ON**.
5. Next to **How long should Panther wait before it sends you an alert that no events have been processed?** set a **Number** and **Period**.\
   ![In a "Configure Drop-off Alarm" pop-up window, the "Trigger an alert when no events are processed" is toggled to "ON." Below, it asks "How long should Panther wait before it sends you an alert that no events have been processed?" and in the Number selector is "1" and in the Period selector is "Day(s)." At the bottom is a blue "Apply Changes" button.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-b0e8f55a2878eb0432b432de1fdb1eb07c3f593e%2FScreenshot%202023-05-09%20at%204.02.05%20PM.png?alt=media)
6. Click **Apply Changes**.
   {% endtab %}
   {% endtabs %}

## Types of System Errors

### Log Source Health alerts

Panther performs health checks on log sources to ensure that Panther is correctly linked to the source, has the right credentials, and is receiving data from the source consistently.

#### Log drop-off alerts

Panther allows you to set up event threshold alarms for individual log sources, which will trigger an alert if data is not received over a specific time interval. For instructions on enabling these alerts, see the section above:[ Configuring log drop-off alarms for log sources](#configuring-log-drop-off-alarms-for-log-sources).

It is not possible to set up a log drop-off alarm for [Panther audit logs](https://docs.panther.com/data-onboarding/supported-logs/panther-audit-logs), when enabled as a log source.

### Log Classification alerts

Panther generates a Log Classification alert when incoming logs fail to parse correctly according to the schema(s) attached to their log source. When this happens:

* Logs that failed to classify are sent to the data lake and are searchable in a table called `classification_failures` in the `panther_monitor` database.
* An alert is generated immediately after the first log fails to classify. The alert will display all log lines that are failing to classify.

An alert's details page in the Panther Console highlights the log lines that fail to parse correctly, to help you determine which lines in the log type's respective schemas need to be corrected or added.

The alert includes a link to the respective log source's Log Source Ops page where you can view the rate at which events are failing to classify within the Health tab.

![The Log Source operations page, which includes a graph. The graph shows the rate at which events are misclassified.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-7232e87070e8778dd78ae968c05aa48fcb8ef4ed%2Fimage.png?alt=media)

#### Remediating Classification Failures

When one of your log sources receives a Classification Failure, you can remediate it by taking the following steps:

1. If the log source has more than one schema attached, identify which schema failed.
   * You can find this information either on the **Health** tab of the log source's detail page, or directly from Data Explorer, in a table called `classification_failures` in the `panther_monitor` database.
2. Understand why the schema failed to parse the event(s). Common causes for Classification Failures include:
   * A field with `required:true` didn't exist on some of the incoming data
   * A field has `type:string` but the actual data received is an `object`
   * A timestamp field has the wrong format definition
   * The event was of a LogType that was not configured in the source
   * Event was malformed in some other way
3. [Update the schema](https://docs.panther.com/data-onboarding/custom-log-types#editing-a-custom-schema) as needed.
4. Resolve the Classification Failure alert by clicking **Mark as Resolved**.

   <figure><img src="https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-704a71213cf7ef0ffcee6918e87f4a1dca80c205%2FScreenshot%202025-07-11%20at%2010.38.06%E2%80%AFAM.png?alt=media" alt="Under an &#x22;Overview&#x22; tab, a button labeled &#x22;Mark as Resolved&#x22; is circled."><figcaption></figcaption></figure>
5. On the **Reprocess events?** pop-up modal that appears, handle the events that failed to classify clicking either:
   * (Beta) **Reprocess Events**: events that failed classification will be processed again.\
     ![Under a "Reprocess events?" header, a "Reprocess Events" button is circled.](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-33a07abdc21c5c00eb049331c76134559e04643b%2FScreenshot%202025-07-11%20at%2008.13.59.png?alt=media)<br>

     <div data-gb-custom-block data-tag="hint" data-style="warning" class="hint hint-warning"><p>Event reclassification is only possible for events that failed classification within the last 15 days.</p><p>If the Classification Failure alert you are resolving contains both events received within the last 15 days and events older than 15 days, only the former will be reprocessed—the latter will be ignored.</p><p>Reprocessed events count toward your ingestion quota at the time they are successfully ingested.</p></div>

     * If reprocessing completes successfully, you will receive a [System Notification](https://docs.panther.com/system-configuration/notifications/..#system-notifications):\
       ![A rectangle with an "i" icon on the left side says, "Panther completed reclassifying logs for source \[Test Source\]"](https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-e8b5c20c3b5c8c17fe1949cbed1b3b38ce6989d2%2FScreenshot%202025-07-11%20at%2008.17.58.png?alt=media)
     * If classification fails again, you will receive a new Classification Failure alert.
   * **Skip Reprocessing**: events that failed classification will not be ingested into Panther.

### S3 GetObject Error Notifications

S3 GetObject error alerts generate when Panther fails to fetch S3 objects. When this happens, the following actions take place by default:

* Panther stores the S3 objects in the data lake which can be queried through the Data Explorer in a table titled `panther_monitor.data_audit`.
* An alert is generated if Panther fails to fetch any S3 object in the last 24 hours. The alert displays the specific S3 objects that are failing.

{% hint style="info" %}
This type of error is generated when Panther attempts to ingest a log event that exceeds the [data ingestion size limit (15 MB)](https://docs.panther.com/data-onboarding#data-ingestion-size-limit). Similar errors are generated for CloudWatch, Azure Blob Storage, and Google Cloud Storage (GCS).
{% endhint %}

### Alert Delivery Failure

Alert Delivery Failure alerts are generated when Panther fails to deliver an alert to a destination.

If the initial attempt to deliver an alert fails, Panther automatically attempts to re-deliver it. After breaching a certain threshold of alert delivery failures, a system health alert is generated and sent to any alert destinations configured to receive `System Error` alerts.

### Cloud Security Scanning Failure

Cloud Security Scanning Failure alerts are generated when Panther fails to scan a cloud resource because of an "access denied" error.

This occurs when permissions are not configured properly to allow scanning to occur. This is most commonly caused by one of the following scenarios:

* Our scanning role (`PantherAuditRole`) is not configured with sufficient permissions.
  * This is an extremely rare case as the permissions of this role rarely change. This can be resolved by updating the `PantherAuditRole` to the latest version.
* An AWS organizations Service Control Policy (SCP) is preventing our scanning role from carrying out scans.
  * Commonly this occurs with SCP's with restrictions for certain regions or services. This can be resolved by either modifying the SCP to add an exception for our scanning role, or by modifying the Cloud Security integration to exclude certain regions or resource types.
* An AWS resource base policy is preventing our scanning role from carrying out scans.
  * In AWS, permissions are bidirectional. The `PantherAuditRole` may be granted permission to access a resource, but the resource itself may not grant permission to be accessed by our role. This can be resolved by either modifying the resource based policy to add an exception for our scanning role, or by modifying the Cloud Security integration to exclude certain resources or resource types.

The alert will indicate which resource scanning failed on, and the AWS error that caused the scanning to fail:

<figure><img src="https://4011785613-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LgdiSWdyJcXPahGi9Rs-2910905616%2Fuploads%2Fgit-blob-aeabf5161b824fcf1da79067a1a8b4cb6e5d6214%2Fscanning-errors.png?alt=media" alt="The image shows an alert in the Panther Console titled &#x22;Source [panther-account] has scanning errors.&#x22; The &#x22;Events&#x22; tab is open, and it includes metadata for the alert."><figcaption></figcaption></figure>

You can use this information to pinpoint the exact permissions issue. In the example above, we can see `no resource-based policy allows the kms:ListResourcetags action`. This indicates to us that the issue is related to a resource-based policy.