# Monitoring Log Sources ## Overview Once a log source is onboarded in Panther, you can monitor data processing metrics and the log source's health within the [log sources list](#using-the-log-sources-list) or on the [log source's operations page](#using-a-log-sources-operations-page) in the Panther Console. You can also attach new schemas and view raw data associated with the log source. At **Configure > Log Sources** - the Log Sources Overview page - you can view ingestion stats for all log sources combined. ## How to monitor overall log source ingestion The log ingestion monitoring summary on the Log Sources Overview page shows data from the current month: Your total number of log sources, unhealthy log sources, a mini-trend chart of the events flow, the amount and the volume of the processed events, the volume of filtered events, and the ingestion quota progression. If you are nearing your data ingestion capacity limit, you will see a notification banner above the summary. To view a list of only the unhealthy log sources, click the number under `Unhealthy` in the dashboard. To see more detailed charts with configurable timeframes, click the **Analytics Dashboard** tab:
On the Log Sources page, there is a tab labeled Analytics Dashboard.
View the following additional charts within the **Analytics Dashboard** tab: * Events over time by log type * Filtered out events * Events processed (volume) * Data stored per source * Click a log source on the chart to see the associated schemas for that source and that source's contribution to the ingestion quota. * Data stored per log type * Average data latency by log type
The "Log Sources" dashboard shows various data, graphs, and charts about log ingestion. There are "Events Over Time By Log Type," "Sources," "Ingestion Quota," "Data Stored Per Source," among others.
To adjust the timeframe of the charts, use the date picker in the upper right corner. ### Viewing filtered event volume To view metrics on events filtered out using [ingestion filters](https://docs.panther.com/data-onboarding/ingestion-filters): 1. In the left-hand navigation bar of your Panther Console, click **Configure** > **Log Sources**. 2. Click the **Analytics Dashboard** tab. 3. In the **Data Ingestion Dashboard**, click the **Filtered Out Events** tab. * See the **Filtered Events** data points.
## How to monitor individual log sources ### Using the log sources list The table of log sources found at **Configure** > **Log Sources** contains monitoring and health information for each log source. Scroll to the right to see additional columns, such as **Volume**, **Log filters**, **Data filtered**, and **Classification errors**.
A "23 Log Sources" heading is above a table containing headers such as, "Source Name," "Status," "Log Types," and more.
### Using a log source's operations page 1. In the left-hand navigation bar of your Panther Console, click **Configure** > **Log Sources**. 2. Click on any log source in the list to view that log source's operations page. ### Viewing the log source Overview The **Overview** tab displays actionable log source metrics and health information. The **Basic Info** section displays frequently used, click-to-copy metadata, as well as the ingestion pipeline status. #### **Source Status** All statuses display the most recent timestamp for when that event occurred. * **Source Created:** Log source configuration is complete and Panther should be able to ingest log data from the source. * **Last Data Received:** Log data is available for normalization. * **Last Data Ingested:** Log data has been processed and normalized.
On a log sources's page, in the Overview tab, there is a "Source Status" section. It displays timestamps for Source Created, Last Data Received, and Last Data Ingested.
#### **Overview stats** All data visualizations reflect the time period selected in the date picker. Events are included based on their ingestion time, or `p_parse_time` values. * **Vol. of data processed**: The amount of uncompressed log data that has been successfully processed and normalized for the selected time period. * **% of total processed data:** The amount of uncompressed log data ingested by this log source vs. the amount of all log data ingested for the selected time period. * **# of events processed:** The number of successfully processed and normalized events for the selected time period. * **Data Processed by Log Type** (chart)**:** The amount of data ingested by the log type (the type of data). * **Events** (chart)**:** The number of events processed. * Note: There is a [corresponding table for each log type](https://docs.panther.com/data-analytics/backend#available-databases).
The Overview tab of a log source page shows various metrics, e.g., Vol. of data processed and % of total processed data, as well as graphs like Data Processed by Log Type
### Viewing the log source schemas At the top of the operations page, click the **Schemas** tab to see all schemas that are parsing and normalizing the data for this source. If you are looking at a custom log source that uses a Data Transport, you will be able to add or remove schemas here.
#### View in Search In the list of schemas, on the right side of a schema's tile, click **View Data**. You will be redirected to [Search](https://docs.panther.com/search/search-tool) with pre-filled selections that you can run to view data associated with the schema for that source. #### View log source raw data {% hint style="info" %} This feature is only available to log sources onboarded with the S3 transport method. {% endhint %} {% hint style="danger" %} The permission `View Log Source Raw Data` is required. By default, only users with the Admin role have this permission. {% endhint %} When onboarding an S3 log source with or without log types, you get direct access to the log source's raw data that Panther receives. To access the log source's raw data: 1. Log in to the Panther Console. 2. In the left sidebar menu, click **Configure > Log Sources**. 3. Click the **Schemas** tab then the **Edit** button
In the Schema Configuration, you can view events for a specific time range. Optionally, you can limit results by applying filters for the S3 key prefix and a search string to match each event.
The schema configuration page shows raw data at the bottom. The filters above the data are set to search for the keyword "GET" with the prefix "AWSLogs."
### Viewing the log source health At the top of the operations page, click the **Health** tab to see all system health alerts related to the health of the log source you are viewing. This page displays errors related to data classification, log drop-off, S3 Get.Object, and permissions. You can learn more about these error types in the documentation: [System Health Notifications](https://docs.panther.com/system-configuration/notifications/system-errors).
### Viewing the log source configuration In the upper right side of the operations page, click **Configuration** to see the log source's configuration details. * Click **Delete** to delete the log source and associated configurations. * Click **Edit** to update the existing configuration.