Standard Fields

Panther's log analysis applies normalization fields to all log records

Panther's log analysis applies normalization fields (IPs, domains, etc) to all log records. These fields provide standard names for attributes across all data sources enabling fast and easy data correlation.
For example, each data source has a time that an event occurred, but each data source will likely not name the attribute the same, nor is it guaranteed that the associated time has a timezone consistent with other data sources.
The Panther attribute p_event_time is mapped to each data source's corresponding event time and normalized to UTC. This way you can query over multiple data sources joining and ordering by p_event_time to properly align and correlate the data despite the disparate schemas of each data source.
All appended standard fields begin with p_.
Required Fields
The fields below are appended to all log records:
Field Name
Type
Description
p_log_type
string
The type of log.
p_row_id
string
Unique id (UUID) for the row.
p_event_time
timestamp
The associated event time for the log type is copied here and normalized to UTC.
Format: YYYY-MM-DD HH:MM:SS.fff
p_parse_time
timestamp
The current time when the event was parsed, normalized to UTC.
Format: YYYY-MM-DD HH:MM:SS.fff
p_schema_version
integer
The version of the schema used for this row.
p_source_id
string
The Panther generated internal id for the source integration.
p_source_label
string
The user supplied label for the source integration (may change if edited).
p_source_file
object 
Available for S3 sources only, this field contains metadata of the file that this event originated from, including the bucket name and object key.
If an event does not have a timestamp, then p_event_time will be set to p_parse_time, which is the time the event was parsed.
The p_source_id and p_source_label fields indicate where the data originated. For example, you might have multiple CloudTrail sources registered with Panther, each with a unique name (e.g., "Dev Accounts", "Production Accounts", "HR Accounts", etc.). These fields allow you to separate data based on the source, which is beneficial when configuring detections in Panther.
In addition, the fields below are appended to log records of all tables in the panther_rule_matches database:
Field Name in panther_rule_matches
Type
Description
p_alert_id
string
Id of alert related to row.
p_alert_creation_time
timestamp
Time of alert creation related to row.
p_alert_context
object
A JSON object returned from the rule's alert_context() function.
p_alert_severity
string
The severity level of the rule at the time of the alert. This could be different from the default severity as it can be dynamically set.
p_alert_update_time
timestamp
Time of last alert update related to row.
p_rule_id
string
The id of the rule that generated the alert.
p_rule_error
string
The error message if there was an error running the rule.
p_rule_reports
map[string]array[string]
List of user defined rule reporting tags related to row.
p_rule_severity
string
The default severity of the rule.
p_rule_tags
array[string]
List of user defined rule tags related to row.
Core Fields
Panther Core Fields are in open beta starting with Panther version 1.86, and are available to all customers. Please share any bug reports and feature requests with your Panther support team.
Core Fields make up the Panther Unified Data Model (UDM). They normalize data from various sources into a consistent structure while maintaining its context. This makes Core Fields useful for searching and writing detections across log types.
The Panther UDM fields help define user and machine attributes. The user performing the action in the log (i.e. the actor) is represented as user, while machines are represented as either source or destination.
Learn how to map fields in your custom log source schemas to Core Fields below, in Mapping Core Fields in custom log schemas. In certain cases, you may want to map one event field to both an Indicator (p_any) field, and a Core Field—learn more in Core Fields vs. Indicator Fields. Core Fields also differ from Data Models for detections; their differences are described below, in Core Fields vs. Data Models in Python detections.
Certain Panther-managed log sources have UDM field mappings configured. Support for additional sources is planned.
Events ingested prior to the Panther UDM being enabled in your Panther instance will not contain Core Fields.
See the full list of Panther Core Fields (also known as UDM fields) below:
Field Name
Display Name
Description
Example
destination.address
Destination Address
Some event destination addresses are defined ambiguously. The event will sometimes list an IP, a domain, or a unix socket. You should always store the raw address in the .address field.
Then it should be duplicated to .mac, .ip, or .domain, depending on which one it is. In the case of multiple, preference is for domain over IP, over mac.
foo.acme.com, 1.1.1.1, 1.1.1.1:55001
destination.arns
Destination ARNs
ARNs associated with the destination resource
[”arn:aws:iam::560769337183:role/PantherLogProcessingRole-panther-account-us-west-2”]
destination.bytes
Destination Bytes
Bytes sent from the destination to the source
192
destination.domain
Destination Domain
The domain name of the destination system.
This value may be a host name, a fully qualified domain name, or another host naming format. The value may derive from the original event or be added from enrichment.
foo.acme.com
destination.ip
Destination IP
IP address of the destination (IPv4 or IPv6)
1.1.1.1
destination.mac
Destination MAC
Mac address of the destination.
The notation format from RFC 7042 is suggested: Each octet (that is, 8-bit byte) is represented by two [uppercase] hexadecimal digits giving the value of the octet as an unsigned integer. Successive octets are separated by a hyphen.
00-00-5E-00-53-23
destination.port
Destination Port
Port of the destination
80
source.address
Source Address
Some event source addresses are defined ambiguously. The event will sometimes list an IP, a domain, or a unix socket. You should always store the raw address in the .address field.
Then it should be duplicated to .mac, .ip, or .domain, depending on which one it is. In the case of multiple, preference is for domain over IP, over mac.
foo.acme.com, 1.1.1.1, 1.1.1.1:55001
source.arns
Source ARNs
ARNs associated with the source resource
[”arn:aws:iam::560769337183:role/PantherLogProcessingRole-panther-account-us-west-2”]
source.bytes
Source Bytes
Bytes sent from the source to the destination
192
source.domain
Source Domain
The domain name of the source system.
This value may be a host name, a fully qualified domain name, or another host naming format. The value may derive from the original event or be added from enrichment.
foo.acme.com
source.ip
Source IP
IP address of the source (IPv4 or IPv6)
1.1.1.1
source.mac
Source MAC
Mac address of the source.
The notation format from RFC 7042 is suggested: Each octet (that is, 8-bit byte) is represented by two [uppercase] hexadecimal digits giving the value of the octet as an unsigned integer. Successive octets are separated by a hyphen.
00-00-5E-00-53-23
source.port
Source Port
Port of the source
80
user.arns
User ARNs
ARNs associated with the user
[”arn:aws:iam::560769337183:role/PantherLogProcessingRole-panther-account-us-west-2”]
user.email
User Email
User email address
[email protected]
user.full_name
User Full Name
User’s full name, if available
Bob Sanders
user.name
User Display Name
Short name or login of the user
b.sanders
user.provider_id
User Providers ID
The ID given by another provider for the user, if it exists
abc123
Core Fields vs. Indicator Fields
It may make sense to classify certain event fields as both Core (UDM) and Indicator (p_any) fields. For example, you might map one event field to the destination.ip UDM field, another event field to the source.ip UDM field, and include the ip indicator on both fields, so that the value of each one may be included in p_any_ip_addresses.
In general, when a field can be classified as both a UDM and p_any field, the UDM mapping maintains relationship information, while the p_any field records that the value was in the event at all (but does not indicate whether it came from the side performing the action or having the action performed on it).
When a field can be classified as both a UDM and p_any field, it is recommended to create both mappings. This will allow you to, in cross-log detections and searches, either include logic that asks if the value was present in the log at all (using the p_any field), or if the value came from a certain side of the relationship (the UDM field).
Core Fields vs. Data Models in Python detections
In addition to Core Fields, Panther supports Data Models in detections, which allow you to define common aliases for event fields across log types, which can then be referenced in Python detections. Below are key differences between Core Fields and Data Models for detections:
When to use each one:
Use Data Models for detections if your objective is only to write detections that can use one alias to reference differently named fields in various log types.
Use Core Fields if, in addition to the above ability in detections, you would also like the convenience of being able to use a single field to search your data lake for values in those differently named event fields across detection types.
How each one is defined:
Core Fields are defined in a log source schema. 
​Data Models for detections, while specific to a log type, are defined separately from the schema: in the Data Models tab in the Panther Console, or in a datamodel file in the CLI workflow.
How each one transforms an incoming log:
When a Core Field is mapped, for each incoming event of that log type, the Core Field/value pair will be added within the event’s p_udm object.
Creating an alias in Data Models for detections does not alter the event structure.
How you would access each one in a Python detection:
To access a Core Field in a Python detection, you would use event.deep_get("p_udm", ...). 
To access a Data Model for detections field, you would use event.udm(...).
If you are using both Core Fields and Data Models for detections, it is possible for a naming conflict to arise if p_udm. is at the beginning of your Data Models for detections alias name. In these cases, the Core Field takes precedence.
For example, say you mapped an event field to the source.ip Core Field and defined a Data Models for detections alias called p_udm.source.ip. In your Python detection, when calling event.udm(p_udm.source.ip), the value for the event field mapped to the Core Field (not the Data Model for detections alias) will be used.
Mapping Core Fields in Custom Log Schemas
You can map fields in custom log schemas to Core Fields.
To map fields in a custom log schema to a Core Field: 
1.In the left-hand navigation bar of your Panther Console, click Configure > Schemas.
2.Locate the custom schema you'd like to map Core Fields within, and upper right corner of its tile, click the three docs icon > Edit.
​
3.In the code editor, under the existing schema definition, add a udm key.
4.The udm field takes a list of name and paths pairs. 
The name key takes one of the values in the "Field Name" column of the Core Fields table above. The value is denoted with JSON dot notation.
The paths key takes a list of path keys, whose value is the path to the event key whose value you'd like to set for the UDM field indicated in name. The paths list will be evaluated chronologically and the first non-null path value will be set to the UDM field. The value is denoted with JSON dot notation.
Example:
schema: MySchema
fields:
 - name: actor
 type: object
 fields:
 - name: email
 type: string
 - name: name
 type: string
 - name: eventType
 type: string
 - name: eventName
 type: string
​
# Below is the new section. All the names here leave off the p_udm prefix.
# For example user.email here will correspond to p_udm.user.email
udm:
- name: user.email
 paths:
 - path: actor.email
 - path: actor.name
- name: user.name
 paths:
 - path: actor.name
Indicator Fields
A common security question is, “Was some indicator ever observed in any of our logs?” Panther's Search tool enables you to find the answer by searching across data from all of your various log sources.
As log events are ingested, the indicators field in their corresponding schema identifies which fields should have their values extracted into p_any_ fields, which are appended to and stored with the event. The table below shows which p_any_ field(s) data is extracted into, by indicator.
When constructing a custom schema, you can use the values in the Indicator Name column in the table below in your schema's indicators field. Each of the rows (except for hostname, net_addr, and url) corresponds to a "Panther Fields" option in Search. You may want to map certain event fields to Core (UDM) Fields, in addition to Indicator Fields. Learn more in Core Fields vs. Indicator Fields.
Note that field name/value pairs outside of the fields in the table below can be searched with Search's key/value filter expression functionality—though because those fields have not been mapped to corresponding ones (with different syntax) in different log sources, only matches from log sources containing the exact field name searched will be returned.
Indicator Name
Extracted into fields
Description
actor_id
p_any_actor_ids
Append value to p_any_actor_ids.
aws_account_id
p_any_aws_account_ids
If the value is a valid AWS account id then append to p_any_aws_account_ids. 
To use this indicator in custom schemas, ensure the field type is string (not bigint).
aws_arn
p_any_aws_arns,
p_any_aws_instance_ids,
p_any_aws_account_ids,
p_any_emails
If value is a valid AWS ARN then append to p_any_aws_arns.
If the ARN contains an AWS account id, extract and append to p_any_aws_account_ids.
If the ARN contains an EC2 instance id, extract and append to p_any_aws_instance_ids.
If the ARN references an AWS STS Assume Role and contains and email address, then extract email address into p_any_emails.
aws_instance_id
p_any_aws_instance_ids
If the value is a valid AWS instance id then append to p_any_aws_instance_ids.
aws_tag
p_any_aws_tags
Append value into p_any_aws_tags.
domain
p_any_domain_names
Append value to p_any_domain_names.
email
p_any_emails
If value is a valid email address then append value into p_any_emails.
hostname
p_any_domain_names, p_any_ip_addresses
Append value to p_any_domain_names.
If value is a valid ipv4 or ipv6 address then append to p_any_ip_addresses.
ip
p_any_ip_addresses
If value is a valid ipv4 or ipv6 address then append to p_any_ip_addresses.
mac
p_any_mac_addresses
If a value is a valid IEEE 802 MAC-48, EUI-48, EUI-64, or a 20-octet IP over InfiniBand link-layer address then append to p_any_mac_addresses.
md5
p_any_md5_hashes
If the value is a valid md5 then append value into p_any_md5_hashes.
net_addr
p_any_domain_names, p_any_ip_addresses
Extracts from values of the form <host>:<port>.
Append host portion to p_any_domain_names.
If host portion is a valid ipv4 or ipv6 address then append to p_any_ip_addresses.
serial_number
p_any_serial_numbers
Append value to p_any_serial_numbers.
This feature is in closed beta starting with Panther version 1.69. To share any bug reports or feature requests, please contact your Panther support team.
sha1
p_any_sha1_hashes
If the value is a valid sha1 then append to p_any_sha1_hashes.
sha256
p_any_sha256_hashes
If the value is a valid sha256 then append to p_any_sha256_hashes.
trace_id
p_any_trace_ids
Append value to p_any_trace_ids.
Tag fields such as session ids and document ids that are used to associated elements with other logs in order to trace the full activity of a sequence of related events.
url
p_any_domain_names, p_any_ip_addresses
Parse url, extract the host portion after "http://" or "https://".
Append host portion to p_any_domain_names.
If host portion is a valid ipv4 or ipv6 address then append to p_any_ip_addresses.
username
p_any_usernames
Append value into p_any_usernames.
​
Enrichment Fields
The Panther rules engine will take the looked-up matches from Lookup Tables and append that data to the event using the key p_enrichment in the following JSON structure:
{ 
 'p_enrichment': {
 <name of lookup table>: { 
 <key in log that matched>: <matching row looked up>,
 ...
 <key in log that matched>: <matching row looked up>,
 } 
 }
} 
Enrichment Field Name
Type
Description of Enrichment Field
p_enrichment
object
Dictionary of lookup results where matching rows were found.
p_match
string
p_match is injected into the data of each matching row within p_enrichment. Its value is the value that matched in the event.
The "all_logs" view
Panther manages a view over all data sources with standard fields.
This allows you to answer questions such as, "Was there any activity from some-bad-ip, and if so, where?"
The query below will show how many records, by log type, are associated with the IP address 95.123.145.92:
Snowflake
Athena
SELECT
 p_log_type, count(1) AS row_count
FROM panther_views.public.all_logs
WHERE p_occurs_between('2020-1-30', '2020-1-31')
 AND array_contains('95.123.145.92'::variant, p_any_ip_addresses)
GROUP BY p_log_type
SELECT
 p_log_type, count(1) AS row_count
FROM panther_views.all_logs
WHERE p_occurs_between('2020-1-30', '2020-1-31') 
 AND contains(p_any_ip_addresses, '95.123.145.92')
GROUP BY p_log_type
From these results, you can pivot to the specific logs where activity is indicated.
Standard Fields in detections
The Panther standard fields can be used in detections. 
For example, the Python rule below triggers when any GuardDuty alert is on a resource tagged as Critical:
def rule(event):
 if 'p_any_aws_tags' in event:
 for tag in event['p_any_aws_tags']:
 if 'critical' in tag:
 return True
 return False 

Field Name	Display Name	Description	Example
`destination.address`	Destination Address	Some event destination addresses are defined ambiguously. The event will sometimes list an IP, a domain, or a unix socket. You should always store the raw address in the .address field. Then it should be duplicated to `.mac`, `.ip`, or `.domain`, depending on which one it is. In the case of multiple, preference is for domain over IP, over mac.	foo.acme.com, 1.1.1.1, 1.1.1.1:55001
`destination.arns`	Destination ARNs	ARNs associated with the destination resource	[”arn:aws:iam::560769337183:role/PantherLogProcessingRole-panther-account-us-west-2”]
`destination.bytes`	Destination Bytes	Bytes sent from the destination to the source	192
`destination.domain`	Destination Domain	The domain name of the destination system. This value may be a host name, a fully qualified domain name, or another host naming format. The value may derive from the original event or be added from enrichment.	foo.acme.com
`destination.ip`	Destination IP	IP address of the destination (IPv4 or IPv6)	1.1.1.1
`destination.mac`	Destination MAC	Mac address of the destination. The notation format from RFC 7042 is suggested: Each octet (that is, 8-bit byte) is represented by two [uppercase] hexadecimal digits giving the value of the octet as an unsigned integer. Successive octets are separated by a hyphen.	00-00-5E-00-53-23
`destination.port`	Destination Port	Port of the destination	80
`source.address`	Source Address	Some event source addresses are defined ambiguously. The event will sometimes list an IP, a domain, or a unix socket. You should always store the raw address in the .address field. Then it should be duplicated to `.mac`, `.ip`, or `.domain`, depending on which one it is. In the case of multiple, preference is for domain over IP, over mac.	foo.acme.com, 1.1.1.1, 1.1.1.1:55001
`source.arns`	Source ARNs	ARNs associated with the source resource	[”arn:aws:iam::560769337183:role/PantherLogProcessingRole-panther-account-us-west-2”]
`source.bytes`	Source Bytes	Bytes sent from the source to the destination	192
`source.domain`	Source Domain	The domain name of the source system. This value may be a host name, a fully qualified domain name, or another host naming format. The value may derive from the original event or be added from enrichment.	foo.acme.com
`source.ip`	Source IP	IP address of the source (IPv4 or IPv6)	1.1.1.1
`source.mac`	Source MAC	Mac address of the source. The notation format from RFC 7042 is suggested: Each octet (that is, 8-bit byte) is represented by two [uppercase] hexadecimal digits giving the value of the octet as an unsigned integer. Successive octets are separated by a hyphen.	00-00-5E-00-53-23
`source.port`	Source Port	Port of the source	80
`user.arns`	User ARNs	ARNs associated with the user	[”arn:aws:iam::560769337183:role/PantherLogProcessingRole-panther-account-us-west-2”]
`user.email`	User Email	User email address	[email protected]
`user.full_name`	User Full Name	User’s full name, if available	Bob Sanders
`user.name`	User Display Name	Short name or login of the user	b.sanders
`user.provider_id`	User Providers ID	The ID given by another provider for the user, if it exists	abc123

Indicator Name	Extracted into fields	Description
actor_id	p_any_actor_ids	Append value to p_any_actor_ids.
aws_account_id	p_any_aws_account_ids	If the value is a valid AWS account id then append to p_any_aws_account_ids. To use this indicator in custom schemas, ensure the field type is `string` (not `bigint`).
aws_arn	p_any_aws_arns, p_any_aws_instance_ids, p_any_aws_account_ids, p_any_emails	If value is a valid AWS ARN then append to p_any_aws_arns. If the ARN contains an AWS account id, extract and append to p_any_aws_account_ids. If the ARN contains an EC2 instance id, extract and append to p_any_aws_instance_ids. If the ARN references an AWS STS Assume Role and contains and email address, then extract email address into p_any_emails.
aws_instance_id	p_any_aws_instance_ids	If the value is a valid AWS instance id then append to p_any_aws_instance_ids.
aws_tag	p_any_aws_tags	Append value into p_any_aws_tags.
domain	p_any_domain_names	Append value to p_any_domain_names.
email	p_any_emails	If value is a valid email address then append value into p_any_emails.
hostname	p_any_domain_names, p_any_ip_addresses	Append value to p_any_domain_names. If value is a valid ipv4 or ipv6 address then append to p_any_ip_addresses.
ip	p_any_ip_addresses	If value is a valid ipv4 or ipv6 address then append to p_any_ip_addresses.
mac	p_any_mac_addresses	If a value is a valid IEEE 802 MAC-48, EUI-48, EUI-64, or a 20-octet IP over InfiniBand link-layer address then append to p_any_mac_addresses.
md5	p_any_md5_hashes	If the value is a valid md5 then append value into p_any_md5_hashes.
net_addr	p_any_domain_names, p_any_ip_addresses	Extracts from values of the form <host>:<port>. Append host portion to p_any_domain_names. If host portion is a valid ipv4 or ipv6 address then append to p_any_ip_addresses.
serial_number	p_any_serial_numbers	Append value to p_any_serial_numbers. This feature is in closed beta starting with Panther version 1.69. To share any bug reports or feature requests, please contact your Panther support team.
sha1	p_any_sha1_hashes	If the value is a valid sha1 then append to p_any_sha1_hashes.
sha256	p_any_sha256_hashes	If the value is a valid sha256 then append to p_any_sha256_hashes.
trace_id	p_any_trace_ids	Append value to p_any_trace_ids. Tag fields such as session ids and document ids that are used to associated elements with other logs in order to trace the full activity of a sequence of related events.
url	p_any_domain_names, p_any_ip_addresses	Parse url, extract the host portion after "http://" or "https://". Append host portion to p_any_domain_names. If host portion is a valid ipv4 or ipv6 address then append to p_any_ip_addresses.
username	p_any_usernames	Append value into p_any_usernames.

Enrichment Field Name	Type	Description of Enrichment Field
`p_enrichment`	object	Dictionary of lookup results where matching rows were found.
`p_match`	string	`p_match` is injected into the data of each matching row within `p_enrichment`. Its value is the value that matched in the event.

Panther Console Dashboard

Saved and Scheduled Searches

Last modified 37m ago

Field Name	Type	Description
`p_log_type`	`string`	The type of log.
`p_row_id`	`string`	Unique id (UUID) for the row.
`p_event_time`	`timestamp`	The associated event time for the log type is copied here and normalized to UTC. Format: `YYYY-MM-DD HH:MM:SS.fff`
`p_parse_time`	`timestamp`	The current time when the event was parsed, normalized to UTC. Format: `YYYY-MM-DD HH:MM:SS.fff`
`p_schema_version`	`integer`	The version of the schema used for this row.
`p_source_id`	`string`	The Panther generated internal id for the source integration.
`p_source_label`	`string`	The user supplied label for the source integration (may change if edited).
`p_source_file`	`object`	Available for S3 sources only, this field contains metadata of the file that this event originated from, including the bucket name and object key.

Field Name in panther_rule_matches	Type	Description
`p_alert_id`	`string`	Id of alert related to row.
`p_alert_creation_time`	`timestamp`	Time of alert creation related to row.
`p_alert_context`	object	A JSON object returned from the rule's alert_context() function.
`p_alert_severity`	`string`	The severity level of the rule at the time of the alert. This could be different from the default severity as it can be dynamically set.
`p_alert_update_time`	`timestamp`	Time of last alert update related to row.
`p_rule_id`	`string`	The id of the rule that generated the alert.
`p_rule_error`	`string`	The error message if there was an error running the rule.
`p_rule_reports`	`map[string]array[string]`	List of user defined rule reporting tags related to row.
`p_rule_severity`	`string`	The default severity of the rule.
`p_rule_tags`	`array[string]`	List of user defined rule tags related to row.