Log Schema Reference
In this guide, you will find common fields used to build YAML-based schemas when onboarding Custom Log Types and Lookup Table schemas.
Required fields throughout this page are in bold.
Each log schema contains the following fields:
- The fields in each Log Event.
- Define a parser that will convert non-JSON logs to JSON.
Additionally, schemas defined using a CI/CD workflow can contain the following fields:
schema
(string
)- The name of the schema
description
(string
)- A short description that will appear in the UI
referenceURL
(string
)- A link to an external document which specifies the log structure. Often, this is a link to a 3rd party's documentation.
fieldDiscoveryEnabled
(boolean
)
See the Custom Logs page for information on how to manage schemas through a CI/CD pipeline using Panther Analysis Tool (PAT).
The example below contains the CI/CD fields mentioned above.
schema: Custom.MySchema
description: (Optional) A handy description so I know what the schema is for.
referenceURL: (Optional) A link to some documentation on the logs this schema is for.
fieldDiscoveryEnabled: true
parser:
csv:
delimiter: ','
hasHeader: true
fields:
- name: action
type: string
required: true
- name: time
type: timestamp
timeFormats:
- unix
A ParserSpec specifies a parser to use to convert non-JSON input to JSON. Only one of the following fields can be specified:
fastmatch
(FastmatchParser{}
): Usefastmatch
parserregex
(RegexParser{}
): Useregex
parsercsv
(CSVParser{}
): Usecsv
parser- Note: The
columns
field is required when there are multiple CSV schemas in the same log source.
See the fields for
fastmatch
, regex
, and csv
in the tabs below.fastmatch
regex
csv
match
([]string
): One or more patterns to match log lines against. This field cannot be empty.emptyValues
([]string
): Values to consider asnull
.expandFields
(map[string]string
): Additional fields to be injected by expanding text templates.trimSpace
(bool
): Trim space surrounding each value.
match
([]string
): A pattern to match log lines against (can be split it into parts for documentation purposes). This field cannot be empty.patternDefinitions
(map[string]string
): Additional named patterns to use in match pattern.emptyValues
([]string
): Values to consider asnull
.expandFields
(map[string]string
): Additional fields to be injected by expanding text templates.trimSpace
(bool
): Trim space surrounding each value.
delimiter
(string
): A character to use as field delimiter.hasHeader
(bool
): Use first row to derive column names (unlesscolumns
is set also in which case the header is just skipped).columns
([]string
,required(without hasHeader)
,non-empty
): Names for each column in the CSV file. If not set, the first row is used as a header.emptyValues
([]string
): Values to consider asnull
.trimSpace
(bool
): Trim space surrounding each value.expandFields
(map[string]string
): Additional fields to be injected by expanding text templates.
A FieldSchema defines a field and its value. The field is defined by:
name
(string
)- The name of the field.
required
(boolean
)- If the field is required or not.
description
(string
)- Some text documenting the field.
- If present, the field's value will be copied from the referenced
object
.
- If present, the field's name will be changed.
- If present, the field's value will be the combination of the values of two or more other fields.
- If present, the field's value will be extracted from another string field by splitting it based on a separator.
- If present, the field's value will be masked.
A
ValueSchema
defines a value and how it should be processed. Each ValueSchema
has a type
field that can be of the following values:Type Values | Description |
string | A string value |
int | A 32-bit integer number in the range -2147483648 , 2147483647 |
smallint | A 16-bit integer number in the range -32768 , 32767 |
bigint | A 64-bit integer number in the range -9223372036854775808 , 9223372036854775807 |
float | A 64-bit floating point number |
boolean | A boolean value true / false |
timestamp | A timestamp value |
array | A JSON array where each element is of the same type |
object | A JSON object of known keys |
json | Any valid JSON value (JSON object, array, number, string, boolean) |
The fields of a
ValueSchema
depend on the value of the type
.Type | Field | Value | Description |
object | fields (required) | An array of FieldSpec objects describing the fields of the object. | |
array | element (required) | A ValueSchema describing the elements of an array. | |
timestamp | timeFormats (required) | String | |
timestamp | isEventTime | Boolean | A flag to tell Panther to use this timestamp as the Log Event Timestamp. |
timestamp | isExpiration | Boolean | (For lookup tables only) A flag to tell Panther to ignore all events after this timestamp |
string | indicators | []String | |
string | validate |