# Log Schema Reference

In this guide, you will find common fields used to build YAML-based schemas when onboarding [Custom Log Types](https://docs.panther.com/data-onboarding/custom-log-types) and [Lookup Table](https://docs.panther.com/enrichment/lookup-tables) schemas. 

{% hint style="info" %}
Required fields throughout this page are in **bold**.
{% endhint %}

## LogSchema fields

Each log schema contains the following fields:

* **`fields`** ([`[]FieldSchema`](#fieldschema))  
  * The fields in each *Log Event*.
* `parser` ([`ParserSpec`](#parserspec))  
  * A parser that can convert non-JSON logs to JSON and/or perform custom transformations

### CI/CD schema fields

Additionally, schemas defined using a CI/CD workflow can contain the following fields:

* **`schema`** (`string`)
  * The name of the schema
* `description` (`string`)
  * A short description that will appear in the UI
* `referenceURL` (`string`)
  * A link to an external document which specifies the log structure. Often, this is a link to a 3rd party's documentation.
* `fieldDiscoveryEnabled` (`boolean`)
  * Indicates whether [field discovery](https://docs.panther.com/~/changes/Cd1BxbxeaFl8dlynhNpt/data-onboarding/custom-log-types/..#enabling-field-discovery) will be enabled for this schema or not. 

{% hint style="info" %}
See the Custom Logs page for [information on how to manage schemas through a CI/CD pipeline](https://docs.panther.com/~/changes/Cd1BxbxeaFl8dlynhNpt/data-onboarding/custom-log-types/..#uploading-log-schemas-with-the-panther-analysis-tool)[ ](https://docs.panther.com/~/changes/Cd1BxbxeaFl8dlynhNpt/data-onboarding/custom-log-types/..#uploading-log-schemas-with-the-panther-analysis-tool)using Panther Analysis Tool (PAT).
{% endhint %}

### Example

The example below contains the CI/CD fields mentioned above.

```yaml
schema: Custom.MySchema
description: (Optional) A handy description so I know what the schema is for.
referenceURL: (Optional) A link to some documentation on the logs this schema is for.
fieldDiscoveryEnabled: true
parser:
  csv:
    delimiter: ','
    hasHeader: true
fields:
- name: action
  type: string
  required: true
- name: time
  type: timestamp
  timeFormats:
    - unix
```

### ParserSpec

A ParserSpec specifies a parser to use to convert non-JSON input to JSON. Only one of the following fields can be specified:

* `fastmatch` (`FastmatchParser{}`): Use `fastmatch` parser
  * Learn more on [Fastmatch Log Parser](https://docs.panther.com/~/changes/Cd1BxbxeaFl8dlynhNpt/data-onboarding/custom-log-types/fastmatch-parser).
* `regex` (`RegexParser{}`): Use `regex` parser
  * Learn more on [Regex Log Parser](https://docs.panther.com/~/changes/Cd1BxbxeaFl8dlynhNpt/data-onboarding/custom-log-types/regex-parser).
* `csv` (`CSVParser{}`): Use `csv` parser 
  * Note: The `columns` field is required when there are multiple CSV schemas in the same log source.
  * Learn more on [CSV Log Parser](https://docs.panther.com/~/changes/Cd1BxbxeaFl8dlynhNpt/data-onboarding/custom-log-types/csv-parser).
* `script`: Use `script` parser
  * Learn more on [Script Log Parser](https://docs.panther.com/~/changes/Cd1BxbxeaFl8dlynhNpt/data-onboarding/custom-log-types/script-parser).

See the fields for `fastmatch`, `regex`, and `csv` in the tabs below.

{% tabs %}
{% tab title="fastmatch" %}

#### Parser `fastmatch` fields

* **`match`** (`[]string`): One or more patterns to match log lines against. This field cannot be empty.
* `emptyValues` (`[]string`): Values to consider as `null`.
* `expandFields` (`map[string]string`): Additional fields to be injected by expanding text templates.
* `trimSpace` (`bool`): Trim space surrounding each value.
  {% endtab %}

{% tab title="regex" %}

#### Parser `regex` fields <a href="#parser-regex-fields" id="parser-regex-fields"></a>

* **`match`** (`[]string`): A pattern to match log lines against (can be split it into parts for documentation purposes). This field cannot be empty.
* `patternDefinitions` (`map[string]string`): Additional named patterns to use in match pattern.
* `emptyValues` (`[]string`): Values to consider as `null`.
* `expandFields` (`map[string]string`): Additional fields to be injected by expanding text templates.
* `trimSpace` (`bool`): Trim space surrounding each value.
  {% endtab %}

{% tab title="csv" %}

#### Parser `csv` fields <a href="#parser-csv-fields" id="parser-csv-fields"></a>

* **`delimiter`** (`string`): A character to use as field delimiter.
* `hasHeader` (`bool`): Use first row to derive column names (unless `columns` is set also in which case the header is just skipped).
* `columns` (`[]string`, `required(without hasHeader)`, `non-empty`): Names for each column in the CSV file. If not set, the first row is used as a header.
* `emptyValues` (`[]string`): Values to consider as `null`.
* `trimSpace` (`bool`): Trim space surrounding each value.
* `expandFields` (`map[string]string`): Additional fields to be injected by expanding text templates.
  {% endtab %}

{% tab title="script" %}

#### Parser `script` fields <a href="#parser-csv-fields" id="parser-csv-fields"></a>

* **`function`** (`string`): The [Starlark](https://bazel.build/rules/language) function to run per event
  {% endtab %}
  {% endtabs %}

### FieldSchema

A *FieldSchema* defines a field and its value. The field is defined by:

* **`name`** (`string`)&#x20;
  * The name of the field.
* `required` (`boolean`) &#x20;
  * If the field is required or not.
* `description` (`string`) &#x20;
  * Some text documenting the field.
* [`copy`](https://docs.panther.com/~/changes/Cd1BxbxeaFl8dlynhNpt/data-onboarding/transformations#copy) (`object`)
  * If present, the field's value will be copied from the referenced `object`.
* [`rename`](https://docs.panther.com/~/changes/Cd1BxbxeaFl8dlynhNpt/data-onboarding/transformations#rename) (`object`)
  * If present, the field's name will be changed.
* [`concat`](https://docs.panther.com/~/changes/Cd1BxbxeaFl8dlynhNpt/data-onboarding/transformations#concat) (`object`)
  * If present, the field's value will be the combination of the values of two or more other fields.
* [`split`](https://docs.panther.com/~/changes/Cd1BxbxeaFl8dlynhNpt/data-onboarding/transformations#split) (`object`)
  * If present, the field's value will be extracted from another string field by splitting it based on a separator.&#x20;
* [`mask`](https://docs.panther.com/~/changes/Cd1BxbxeaFl8dlynhNpt/data-onboarding/transformations#mask) (`object`)
  * If present, the field's value will be masked.&#x20;

Its value is defined using the fields of a [`ValueSchema`](#valueschema).

### ValueSchema

A `ValueSchema` defines a value and how it should be processed. Each `ValueSchema` has a `type` field that can be of the following values:

<table data-header-hidden><thead><tr><th width="268">Value Type</th><th>Description</th></tr></thead><tbody><tr><td>Type Values</td><td>Description</td></tr><tr><td><code>string</code></td><td>A string value</td></tr><tr><td><code>int</code></td><td>A 32-bit integer number in the range <code>-2147483648</code>, <code>2147483647</code></td></tr><tr><td><code>smallint</code></td><td>A 16-bit integer number in the range <code>-32768</code>, <code>32767</code></td></tr><tr><td><code>bigint</code></td><td>A 64-bit integer number in the range <code>-9223372036854775808</code>, <code>9223372036854775807</code></td></tr><tr><td><code>float</code></td><td>A 64-bit floating point number</td></tr><tr><td><code>boolean</code></td><td>A boolean value <code>true</code> / <code>false</code></td></tr><tr><td><code>timestamp</code></td><td>A timestamp value</td></tr><tr><td><code>array</code></td><td>A JSON array where each element is of the same type</td></tr><tr><td><code>object</code></td><td>A JSON object of <em>known</em> keys</td></tr><tr><td><code>json</code></td><td>Any valid JSON value (JSON object, array, number, string, boolean)</td></tr></tbody></table>

The fields of a `ValueSchema` depend on the value of the `type`.

<table data-header-hidden><thead><tr><th width="171">Type</th><th width="192">Field</th><th width="162">Value</th><th>Description</th></tr></thead><tbody><tr><td>Type</td><td>Field</td><td>Value</td><td>Description</td></tr><tr><td><code>object</code></td><td><strong><code>fields</code></strong>  (required)</td><td><a href="#fieldschema"><code>[]FieldSpec</code></a></td><td>An array of <code>FieldSpec</code> objects describing the fields of the object.</td></tr><tr><td><code>array</code></td><td><strong><code>element</code></strong> (required)</td><td><a href="#valueschema"><code>ValueSchema</code></a></td><td>A <code>ValueSchema</code> describing the elements of an array.</td></tr><tr><td><code>timestamp</code></td><td><strong><code>timeFormats</code></strong> (required)</td><td><code>[]String</code></td><td>An array specifying the formats to use for parsing the timestamp (see <a href="#timestamps">Timestamps</a>)</td></tr><tr><td><code>timestamp</code></td><td><code>isEventTime</code></td><td><code>Boolean</code></td><td>A flag to tell Panther to use this timestamp as the <em>Log Event Timestamp</em>.</td></tr><tr><td><code>string</code></td><td><code>indicators</code></td><td><code>[]String</code></td><td>Tells Panther to extract indicators from this value (see <a href="#indicators">Indicators</a>)</td></tr><tr><td><code>string</code></td><td><code>validate</code></td><td><a href="#validation"><code>Validation</code></a></td><td>Validation rules for the string value</td></tr></tbody></table>

### Timestamps

{% hint style="warning" %}
The `timeFormats` field was introduced in version 1.46 to support multiple timestamp formats in custom log schemas. While `timeFormat` is still supported for existing log sources, we recommend using timeFormats for all new schemas.
{% endhint %}

Timestamps are defined by setting the `type` field to `timestamp` and specifying the timestamp format using the `timeFormats` field. Timestamp formats can be any of the built-in timestamp formats:

<table><thead><tr><th width="154.00000000000003">timeFormat</th><th width="276">Example</th><th>Description</th></tr></thead><tbody><tr><td><code>rfc3339</code></td><td><code>2022-04-04T17:09:17Z</code></td><td>The most common timestamp format.</td></tr><tr><td><code>unix_auto</code></td><td><code>1649097448</code> (seconds)<br><br><code>1649097491531</code> (milliseconds)<br><br><code>1649097442000000</code> (microseconds)<br><br><code>1649097442000000000</code> (nanoseconds)</td><td>Timestamp expressed in time passed since UNIX epoch time. It can handle seconds, milliseconds, microseconds, and nanoseconds.</td></tr><tr><td><code>unix</code></td><td><code>1649097448</code></td><td>Timestamp expressed in seconds since UNIX epoch time. It can handle fractions of seconds as a decimal part.</td></tr><tr><td><code>unix_ms</code></td><td><code>1649097491531</code></td><td>Timestamp expressed in milliseconds since UNIX epoch time.</td></tr><tr><td><code>unix_us</code></td><td><code>1649097442000000</code></td><td>Timestamp expressed in microseconds since UNIX epoch time.</td></tr><tr><td><code>unix_ns</code></td><td><code>1649097442000000000</code></td><td>Timestamp expressed in nanoseconds since UNIX epoch time. Scientific float notation is supported.</td></tr></tbody></table>

#### Defining a custom format

You can also define a custom format by using [strftime](https://strftime.org) notation. For example:

```yaml
# The field is a timestmap using a custom timestamp format like "2020-09-14 14:29:21"
- name: ts
  type: timestamp
  timeFormats:
    - "%Y-%m-%d %H:%M:%S" # note the quotes required for proper YAML syntax
```

Panther's strftime format supports using `%N` code to parse nanoseconds. For example:&#x20;

`%H:%M:%S.%N` can be used to parse `11:12:13.123456789`

#### Using multiple time formats

When multiple time formats are defined, each of them will be tried sequentially until successful parsing is achieved:

```yaml
- name: ts
  type: timestamp
  timeFormats:
    - rfc3339
    - unix
```

Timestamp values can be marked with `isEventTime: true` to tell Panther that it should use this timestamp as the `p_event_time` field. It is possible to set `isEventTime` on multiple fields. This covers the cases where some logs have optional or mutually exclusive fields holding event time information. Since there can only be a single `p_event_time` for every *Log Event*, the priority is defined using the order of fields in the schema.

**Schema test cases** that are used with the `pantherlog test` command must define the time field value in the`result` payload formatted as `YYYY-MM-DD HH:MM:SS.fffffffff.` For backwards compatibility reasons, single time format configurations will retain the same format. &#x20;

Example:

```yaml
- name: singleFormatTimestamp
  type: timestamp
  timeFormats:
    - unix
```

```yaml
input: >
  {
    "singleFormatTimestamp": "1666613239"
  }
result: >
  {
    "singleFormatTimestamp": "1666613239"
  }
```

When multiple time formats are defined:

```yaml
- name: multipleFormatTimestamp
  type: timestamp
  timeFormats:
    - unix
    - rfc3339
```

```yaml
input: >
  {
    "multipleFormatTimestamp": "1666613239"
  }
result: >
  {
    "multipleFormatTimestamp": "2022-10-24 12:07:19.459326000"
  }
```

### Indicators

Values of `string` type can be used as indicators. To mark a field as an indicator, set the `indicators` field to an array of indicator scanner names (more than one may be used). This will instruct Panther to parse the string and store any indicator values it finds to the relevant `p_any_` field. For a list of values that are valid to use in the `indicators` field, see [Standard Fields](https://docs.panther.com/~/changes/Cd1BxbxeaFl8dlynhNpt/search/panther-fields#indicator-fields).&#x20;

For example:

```yaml
# Will scan the value as IP address and store it to `p_any_ip_addresses`
- name: remote_ip
  type: string
  indicators: [ ip ]

# Will scan the value as a domain name and/or IP address.
# Will store the result in `p_any_domain_names` and/or `p_any_ip_addresses`
- name: target_url
  type: string
  indicators: [ url ]
```

### Validation - Allow/Deny lists

Values of `string` type can be further restricted by declaring a list of values to `allow` or `deny`. This allows to have different log types that have common overlapping fields but differ on values of those fields.

```yaml
# Will only allow 'login' and 'logout' event types to match this log type
- name: event_type
  type: string
  validate:
    allow: [ "login", "logout"]
    
# Will match any event type other than 'login' and 'logout'
- name: event_type
  type: string
  validate:
    deny: [ "login", "logout"]
```

### Validation by string type

Values of `string` type can be restricted to match well-known formats. Currently, Panther supports the `ip` and `cidr` formats to require that a string value be a valid IP address or CIDR range. Note that the `ip` and `cidr` validation types can be combined with `allow` or `deny` rules but it is somewhat redundant, for example, if you allow two IP addresses, then adding an `ip` validation will simply ensure that your validation will not include false positives if the IP addresses in your list are not valid.

```yaml
# Will allow valid ipv4 IP addresses e.g. 100.100.100.100
- name: address
  type: string
  validate:
    ip: "ipv4"
    
# Will allow valid ipv6 CIDR ranges 
# e.g. 2001:0db8:85a3:0000:0000:0000:0000:0000/64
- name: address
  type: string
  validate:
    cidr: "ipv6"
    
# Will allow any valid ipv4 or ipv6 address
- name: address
  type: string
  validate:
    ip: "any"    
```

## Using JSON schema in an IDE

If your code editor or integrated development environment (IDE) supports [JSON Schema](https://json-schema.org/), you can configure it to use [this schema file](https://panther-community-us-east-1.s3.amazonaws.com/latest/logschema/schema.json) for Panther schemas and [this schema-tests file](https://panther-community-us-east-1.s3.amazonaws.com/latest/logschema/schema-tests.json) for schema tests. Doing so will allow you to receive suggestions and error messages while developing Panther schemas and their tests.

### JetBrains custom JSON schemas

See the [JetBrains documentation](https://www.jetbrains.com/help/phpstorm/json.html#ws_json_schema_add_custom) for instructions on how to configure JetBrains IDEs to use custom JSON Schemas.

### VSCode custom JSON schemas

See the [VSCode documentation](https://code.visualstudio.com/Docs/languages/json#_json-schemas-and-settings) for instructions on how to configure VSCode to use JSON Schemas.

## Stream type

While performing certain actions in the Panther Console, such as [configuring an S3 bucket for Data Transport ](https://docs.panther.com/~/changes/Cd1BxbxeaFl8dlynhNpt/data-onboarding/data-transports/aws/s3)or [inferring a custom schema from raw logs](https://docs.panther.com/~/changes/Cd1BxbxeaFl8dlynhNpt/data-onboarding/custom-log-types/..#infer-a-schema-from-raw-data), you need to select a log stream type.&#x20;

View example log events for each type below.

<table><thead><tr><th width="144">Stream type</th><th width="205">Description</th><th>Example log event(s)</th></tr></thead><tbody><tr><td>Auto</td><td>Panther will automatically detect the appropriate stream type.</td><td>n/a</td></tr><tr><td>Lines</td><td>Events are separated by a new line character.</td><td><pre class="language-json"><code class="lang-json">"10.0.0.1","user-1@example.com","France"
"10.0.0.2","user-2@example.com","France"
"10.0.0.3","user-3@example.com","France"
</code></pre></td></tr><tr><td>JSON</td><td>Events are in JSON format.</td><td><pre class="language-json"><code class="lang-json">{ 
"ip": "10.0.0.1", 
"un": "user-1@example.com", 
"country": "France" 
}
or
{ "ip": "10.0.0.1", "un": "user-1@example.com", "country": "France" }{ "ip": "10.0.0.2", "un": "user-2@example.com", "country": "France" }{ "ip": "10.0.0.3", "un": "user-3@example.com", "country": "France" }
or
{ "ip": "10.0.0.1", "un": "user-1@example.com", "country": "France" }
{ "ip": "10.0.0.2", "un": "user-2@example.com", "country": "France" }
{ "ip": "10.0.0.3", "un": "user-3@example.com", "country": "France" }
</code></pre></td></tr><tr><td>JSON Array</td><td>Events are inside an array of JSON objects.</td><td><pre class="language-json"><code class="lang-json">[
	{ "ip": "10.0.0.1", "username": "user-1@example.com", "country": "France" },
	{ "ip": "10.0.0.2", "username": "user-2@example.com", "country": "France" },
	{ "ip": "10.0.0.3", "username": "user-3@example.com", "country": "France" }
]
</code></pre></td></tr><tr><td>CloudWatch Logs</td><td>Events came from CloudWatch Logs. </td><td><pre class="language-json"><code class="lang-json">{
  "owner": "111111111111",
  "logGroup": "services/foo/logs",
  "logStream": "111111111111_CloudTrail/logs_us-east-1",
  "messageType": "DATA_MESSAGE",
  "logEvents": [
      {
          "id": "31953106606966983378809025079804211143289615424298221568",
          "timestamp": 1432826855000,
          "message": "{\"ip\": \"10.0.0.1\", \"user\": \"user-1@example.com\", \"country\": \"France\"}"
      },
      {
          "id": "31953106606966983378809025079804211143289615424298221569",
          "timestamp": 1432826855000,
          "message": "{\"ip\": \"10.0.0.2\", \"user\": \"user-2@example.com\", \"country\": \"France\"}"
      },
      {
          "id": "31953106606966983378809025079804211143289615424298221570",
          "timestamp": 1432826855000,
          "message": "{\"ip\": \"10.0.0.3\", \"user\": \"user-3@example.com\", \"country\": \"France\"}"
      }
  ]
}
</code></pre></td></tr></tbody></table>