Data Models

Data Models provide a way to configure a set of unified fields across all log types

Overview

Use Data Models to configure a set of unified fields across all log types, by creating mappings between event fields for various log types and unified Data Model names. You can leverage Panther-managed Data Models, and create custom ones.

Data Models for detections are different from the Panther Unified Data Model fields (also known as Core Fields). To learn more, see Core Fields vs. Data Models in Python detections.

Data Models use case

Suppose you have a detection that checks for a particular source IP address in network traffic logs, and you'd like to use it for multiple log types. These log types might not only span different categories (e.g., DNS, Zeek, Apache), but also different vendors. Without a common logging standard, each of these log types may represent the source IP using a different field name, such as ipAddress, srcIP, or ipaddr. The more log types you'd like to monitor, the more complex and cumbersome the logic of this check becomes. For example, it might look something like:

(event.get('ipAddress') == '127.0.0.1' or 
event.get('srcIP') == '127.0.0.1' or 
event.get('ipaddr') == '127.0.0.1')

If we instead define a Data Model for each of these log types, we can translate the event's field name to the Data Model name, meaning the detection can simply reference the Data Model version. The above logic then simplifies to:

event.udm('source_ip') == '127.0.0.1'

Panther-managed Data Models

By default, Panther comes with built-in Data Models for several log types, such as AWS.S3ServerAccess, AWS.VPCFlow, and Okta.SystemLog. All currently supported data models can be found in the panther-analysis repository, here.

The names of the supported Data Model mappings are listed in the Panther-managed Data Model mapping names table, below.

How to create custom Data Models

Custom Data Models can be created in a few ways: in the Panther Console, using the Panther Analysis Tool (PAT), or in the Panther API. See the tabs below for creation instructions for each method.

Your custom Data Model mappings can use the names referenced in Panther-managed Data Models, or your own custom names. Each mapping Name can map to an event field (with Path or Field Path) or a method you define (with Field Method or Method). If you map to a method, you must define the method either in a separate Python file (if working in the CLI workflow), which is referenced in the YAML file using Filename, or in the Python Module field in the Console.

Each log type can only have one enabled Data Model specified (however, a single Data Model can contain multiple mappings). If you want to change or update an existing Data Model, disable the existing one, then create a new, enabled one.

To create a new Data Model in the Panther Console:

In the left-hand navigation bar of your Panther Console, click Detections.
Click the Data Models tab. .
In the upper-right corner, click Create New.
Under Settings, fill in the form fields.
- Display Name: Enter a user-friendly display name for this Data Model.
- ID: Enter a unique ID for this Data Model.
- Log Type: Select a log type this Data Model should apply to. Only one log type per Data Model is permitted.
- Enabled: Select wether you'd like this Data Model enabled or disabled.
Under Data Model Mappings, create Name/Field Path or Name/Field Method pairs.
If you used the Field Method field, define the method(s) in the Python Module (optional) section.
In the upper right corner, click Save.
- You can now reference this Data Model in your rules. Learn more in Using Data Models in rules.

How to create a Data Model in the CLI workflow

Folder setup

All files related to your custom Data Models must be stored in a folder with a name containing data_models (this could be a top-level data_models directory, or sub-directories with names matching *data_models*).

File setup

Create your Data Model specification YAML file (e.g., data_models/aws_cloudtrail_datamodel.yml):
```
AnalysisType: datamodel
LogTypes: 
  - AWS.CloudTrail
DataModelID: AWS.CloudTrail
Filename: aws_cloudtrail_data_model.py
Enabled: true
Mappings:
  - Name: actor_user
    Path: $.userIdentity.userName
  - Name: event_type
    Method: get_event_type
  - Name: source_ip
    Path: sourceIPAddress
  - Name: user_agent
    Path: userAgent
```
- Set AnalysisType to datamodel.
- For LogTypes, provide the name of one of your log types. Despite this field taking a list, only one log type per Data Model is supported.
- Provide a value for the DataModelID field.
- Within Mappings, create Name / Path or Name / Method pairs.
  - Learn more about Mappings syntax below, in DataModel Mappings.
- See Data Model Specification Reference below for a complete list of required and optional fields.

If you included one or more Method fields within Mappings, create an associated Python file (data_models/aws_cloudtrail_datamodel.py), and define any referenced methods.

In this case, you must also add the Filename field to the Data Model YAML file. If no Method fields are present, no Python file/Filename field is required.

from panther_base_helpers import deep_get
def get_event_type(event):
    if event.get('eventName') == 'ConsoleLogin' and deep_get(event, 'userIdentity', 'type') == 'IAMUser':
        if event.get('responseElements', {}).get('ConsoleLogin') == 'Failure':
            return "failed_login"
        if event.get('responseElements', {}).get('ConsoleLogin') == 'Success':
            return "successful_login"
    return None

Upload your Data Model to your Panther instance using the PAT upload command.
- You can now reference this Data Model in your rules. Learn more in Using Data Models in rules.

How to create a Data Model using the Panther API

See the POST operation on Data Models.

Evaluating whether a field exists in `Path`

Within a Path value, you can include logic that checks whether a certain event field exists. If it does, the mapping will be applied; if it doesn't, the mapping doesn't take effect. For example, take the following Path values from the Panther-managed gsuite_data_model.yml:

  - Name: assigned_admin_role
    Path: $.events[*].parameters[?(@.name == 'ROLE_NAME')].value

Using Data Models

Referencing Data Models in a rule

To reference a Data Model field in a rule:

In a rule's YAML file, ensure LogTypes field contains the LogType of the Data Model you'd like applied:

AnalysisType: rule
DedupPeriodMinutes: 60
DisplayName: DataModel Example Rule
Enabled: true
Filename: my_new_rule.py
RuleID: DataModel.Example.Rule
Severity: High
LogTypes:
  # Add LogTypes where this rule is applicable
  # and a Data Model exists for that LogType
  - AWS.CloudTrail
Tags:
  - Tags
Description: >
  This rule exists to validate the CLI workflows of the Panther CLI
Runbook: >
  First, find out who wrote this the spec format, then notify them with feedback.
Tests:
  - Name: test rule
    ExpectedResult: true
    # Add the LogType to the test specification in the 'p_log_type' field
    Log: {
      "p_log_type": "AWS.CloudTrail"
    }

Add the LogType to all the Rule's Test cases, in the p_log_type field.

Leverage the event.udm() method in the rule's Python logic:

def rule(event):    
    # filter events on unified data model field
    return event.udm('event_type') == 'failed_login'


def title(event):
    # use unified data model field in title
    return '{}: User [{}] from IP [{}] has exceeded the failed logins threshold'.format(
        event.get('p_log_type'), event.udm('actor_user'),
        event.udm('source_ip'))

See examples of Data Models in Panther's Github repository.

Using Data Models with Enrichment

Panther provides a built-in method on the event object called event.udm_path. It returns the original path that was used for the Data Model.

AWS.VPCFlow logs example

Using event.udm_path('destination_ip') will return 'dstAddr', since this is the path defined in the Data Model for that log type. The following example uses event.udm_path:

from panther_base_helpers import deep_get

def rule(event):
    return True

def title(event):
    return event.udm_path('destination_ip')

def alert_context(event):
    enriched_data = deep_get(event, 'p_enrichment', 'lookup_table_name', event.udm_path('destination_ip'))
    return {'enriched_data':enriched_data}

This test case was used:

{   
  "p_log_type": "AWS.VPCFlow",
   "dstAddr": "1.1.1.1",
   "p_enrichment": {
      "lookup_table_name": {
        "dstAddr": {
          "datakey": "datavalue" }}}}

The test case returns an alert that includes Alert Context with the datakey and datavalue:

Testing Data Models

To test a Data Model, write unit tests for a detection that references a Data Model mapping using event.udm() in its rule() logic.

DataModel specification reference

A complete list of DataModel specification fields:

DataModel `Mappings`

Mappings translate LogType fields to unified Data Model fields. Each mapping entry must define a Name and either a Path or a method Method. The Path can be a simple field name or a JSON Path. The method must be implemented in the file listed in the data model specification Filename field.

Mappings:
  - Name: source_ip
    Path: srcIp
  - Name: user
    Path: $.events[*].parameters[?(@.name == 'USER_EMAIL')].value
  - Name: event_type
    Method: get_event_type

The example above depicts logic within the user mapping's Path value to check if the USER_EMAIL event field exists. Learn more in Evaluating whether a field exists in Path.

For more information about jsonpath-ng, see pypi.org's documentation here.

Panther-managed Data Model mapping names

The Panther-supported Data Model mapping names are described below. When creating your own Data Model mappings, you may use the names below, in addition to custom ones.

PreviousPython Rule Caching NextGlobal Helper Functions

Last updated 3 months ago