Data Models
Data Models provide a way to configure a set of unified fields across all log types
Overview
Data Models provide a way to configure a set of unified fields across all log types.
Suppose you want to check for a particular source ip address in all events that log network traffic. These LogTypes might not only span different categories (DNS, Zeek, Apache, etc.), but also different vendors. Without a common logging standard, each of these LogTypes may represent the source ip by a different name, such as ipAddress
, srcIP
, or ipaddr
. The more LogTypes you want to monitor, the more complex and cumbersome this simple check becomes:
If instead we define a Data Model for each of these LogTypes, we can translate the unified data model field name to the LogType field name and our logic simplifies to:
Data Models for detections are different from the Panther Unified Data Model fields (also known as Core Fields). To learn more, see Core Fields vs. Data Models in Python detections.
Built-in Data Models
By default, Panther comes with built-in data models for several log types, such as AWS.S3ServerAccess
, AWS.VPCFlow
, and Okta.SystemLog
. All currently supported data models can be found here.
How to add Data Models
New Data Models are added in the Panther Console, via the Panther Analysis Tool (PAT), or in the Panther API.
Each log type can only have one enabled data model specified. If you want to change or update an existing data model, disable
the existing one, and create a new, enabled one.
To create a new Data Model in the Panther Console:
In the upper right corner, click Create New.
In the upper right corner, click Save.
You can now access this Data Model in your rule logic with the event.udm()
method.
Using Data Models
Using Data Models in rules
Use your Data Model in a rule via any of the following methods:
Add the LogType under the Rule specification
LogType
fieldAdd the LogType to all the Rule's
Test
cases, in thep_log_type
fieldLeverage the
event.udm()
method in the Rule's python logic:
See examples of Data Models in Panther's Github repository.
Leveraging existing Data Models
Rules can be updated to use unified data model field names by leveraging the event.udm()
method. For example:
Update the rule specification to include the pertinent LogTypes:
Using Data Models with Enrichment
Panther provides a built-in method on the event object called event.udm_path
. It returns the original path that was used for the Data Model.
AWS.VPCFlow logs example
Using event.udm_path('destination_ip')
will return 'dstAddr'
, since this is the path defined in the Data Model for that log type.
The following example uses event.udm_path
:
This test case was used:
The test case returns an alert that includes Alert Context with the datakey
and datavalue
:
DataModel Specification Reference
A complete list of DataModel specification fields:
Field Name | Required | Description | Expected Value |
| Yes | Indicates whether this specification is defining a rule, policy, data model, or global |
|
| Yes | The unique identifier of the data model | String |
| No | What name to display in the UI and alerts. The | String |
| Yes | Whether this data model is enabled | Boolean |
| No | The path (with file extension) to the python DataModel body | String |
| Yes | What log type this policy will apply to | Singleton List of strings
Note: Although |
| Yes | Mapping from source field name or method to unified data model field name | List of Maps |
DataModel Mappings
Mappings translate LogType fields to unified data model fields. Each mapping entry must define a unified data model field name (Name
) and either a Path (Path
) or a method (Method
). The Path
can be a simple field name or a JSON Path. The method must be implemented in the file listed in the data model specification Filename
field.
For more information about jsonpath-ng, see pypi.org's documentation here.
Unified Data Model Field Reference
The initial set of supported unified data model fields are described below.
Unified Data Model Field Name | Description |
| ID or username of the user whose action triggered the event. |
| Admin role ID or name assigned to a user in the event. |
| Destination IP for the traffic |
| Destination port for the traffic |
| Custom description for the type of event. Out of the box support for event types can be found in the global, |
| Numeric http status code for the traffic |
| Source IP for the traffic |
| Source port for the traffic |
| User agent associated with the client in the event. |
| ID or username of the user that was acted upon to trigger the event. |
Last updated