# pantherlog Tool
## Overview
You can use `pantherlog`, a CLI tool, to work with Custom Logs. It parses logs using Panther-managed or [custom schemas](https://docs.panther.com/data-onboarding/custom-log-types), and uses sample logs to infer custom schemas.
For information on working with custom logs in the Panther Console instead, see the [Custom Logs](https://docs.panther.com/data-onboarding/custom-log-types) documentation.
### pantherlog limitations
* See the [infer limitations](#infer-limitations) and [test limitations](#test-limitations) sections, below.
## Download
Download the latest version at the following links:
## `list-schemas`: List Panther-managed schemas
You can use pantherlog's `list-schemas` command to list Panther's managed schemas.
```
./pantherlog list-schemas
```
## `export-schemas`: Export Panther-managed schemas
You can use pantherlog's `export-schemas` command to export Panther-managed schemas into a local directory, or print them in the terminal.
### Export schemas to local directory
```
./pantherlog export-schemas --path directory-name
```
* If `directory-name` does not exist, it will be created.
* Note that `-p` may be used in place of `--path`.
### Print schemas
To print schemas to `stdout` instead of exporting to a local directory, use a dash.
```
./pantherlog export-schemas -p -
```
### Export select schemas
You can filter the schemas to be exported by using the `-s` option with the names of the schemas you'd like to export, separated by commas.
```
./pantherlog export-schemas --path ./managed-schemas -s 'AWS.ALB,Slack.AuditLogs'
```
## `infer`: Generate a schema from JSON log samples
You can use pantherlog to generate a schema file out of sample files in [new-line delimited JSON](https://jsonlines.org/) format. The tool will scan the provided logs and print the inferred schema to `stdout`.
For example, to infer the schema of logs `sample_logs.jsonl` and output to `schema.yml`, use:
```
$ ./pantherlog infer --name SchemaName sample_logs.jsonl > schema.yml
```
Note that YAML keys and values are case sensitive. The tool will attempt to infer multiple timestamp formats. Learn more about schema inference on [Custom Logs](https://docs.panther.com/data-onboarding/custom-log-types#automatically-infer-the-schema-in-panther).
{% embed url="" %}
The workflow of inferring a schema from sample logs
{% endembed %}
### `infer` limitations
The `infer` command will not mark any timestamp field as `isEventTime:true`. Make sure to select the appropriate `timestamp` field and mark it as `isEventTime:true`. For more information regarding `isEventTime:true`, see the [Timestamps section on Log Schema Reference](https://docs.panther.com/data-onboarding/custom-log-types/reference#timestamps).
The `infer` command is able to infer only the following types of [indicators](https://docs.panther.com/search/panther-fields#indicator-fields): `ip`, `aws_arn`, `url`, `email`, `hash` digests (`MD5`, `SHA1` and `SHA2`), and `mac`. Make sure to review the fields and add more indicators as appropriate.
It's strongly recommended to review the schema generated by the `infer` command, and edit it appropriately before deploying to your production environment.
## `parse`: Validate a schema
You can use the tool to validate a schema file and use it to parse log files. Note that the events in the log files need to be separated by new line. Processed logs are written to `stdout` and errors to `stderr`.
For example, to parse logs in `sample_logs.jsonl` with the log schema in `schema.yml`, use:
```
$ ./pantherlog parse --path schema.yml --schemas Schema.Name sample_logs.jsonl
```
The tool can also accept input via `stdin` so it can be used in a pipeline:
```
$ cat sample_logs.jsonl | ./pantherlog parse --path schema.yml
```
## `test`: Run tests for a schema
You can use pantherlog to run unit tests for your custom schema. To run tests defined in a `schema_tests.yml` file for a custom schema defined in `schema.yml`, you would run:
```
$ ./pantherlog test schema.yml schema_tests.yml
```
The first argument is a file or directory containing schema YAML files. The rest of the arguments are test files to run. If you don't specify any test files arguments, and the first argument is a directory, the tool will look for tests in YAML files with a `_tests.yml` or `_tests.yaml` suffix.
### Creating a schema test file
In your test file, include an `input` key containing the event to parse, and a `result` key containing the expected result. The pantherlog `test` command checks that the schema can parse the event without error, and that the normalized event matches your expected result.
{% hint style="info" %}
The `result` event should include any [Panther Standard Fields](https://docs.panther.com/search/panther-fields) that are expected to be injected into the event during parsing, such as `p_log_type`.
{% endhint %}
Take note of [these instructions on working with `timeFormats` in schema tests](https://docs.panther.com/data-onboarding/custom-log-types/reference#working-with-timeformats-in-schema-tests).
{% hint style="info" %}
For an example of writing multiple tests for one schema, see this article in Panther's Knowledge Base: [How can I write multiple pantherlog tests for a schema?](https://help.panther.com/Data_Sources/Custom_Logs/How_can_I_write_multiple_pantherlog_tests_for_a_schema%3F)
{% endhint %}
Example:
{% tabs %}
{% tab title="schema\_tests.yml" %}
```json
# Make sure to use camelCase when naming the schema or log type
name: Custom Log Test Name
logType: Custom.SampleLog.V1
input: |
{
"method": "GET",
"path": "/-/metrics",
"format": "html",
"controller": "MetricsController",
"action": "index",
"status": 200,
"params": [],
"remote_ip": "1.1.1.1",
"user_id": null,
"username": null,
"ua": null,
"queue_duration_s": null,
"correlation_id": "c01ce2c1-d9e3-4e69-bfa3-b27e50af0268",
"cpu_s": 0.05,
"db_duration_s": 0,
"view_duration_s": 0.00039,
"duration_s": 0.0459,
"tag": "test",
"time": "2019-11-14T13:12:46.156Z"
}
result: |
{
"action": "index",
"controller": "MetricsController",
"correlation_id": "c01ce2c1-d9e3-4e69-bfa3-b27e50af0268",
"cpu_s": 0.05,
"db_duration_s": 0,
"duration_s": 0.0459,
"format": "html",
"method": "GET",
"path": "/-/metrics",
"remote_ip": "1.1.1.1",
"status": 200,
"tag": "test",
"time": "2019-11-14T13:12:46.156Z",
"view_duration_s": 0.00039,
"p_log_type": "Custom.SampleLog.V1",
"p_row_id": "acde48001122a480ca9eda991001",
"p_event_time": "2019-11-14T13:12:46.156Z",
"p_parse_time": "2022-04-04T16:12:41.059224Z",
"p_any_ip_addresses": [
"1.1.1.1"
]
}
```
{% endtab %}
{% tab title="schema.yml" %}
```yaml
version: 0
schema: Custom.SampleLog.V1
fields:
- name: action
required: true
type: string
- name: controller
required: true
type: string
- name: correlation_id
required: true
type: string
- name: cpu_s
required: true
type: float
- name: db_duration_s
required: true
type: bigint
- name: duration_s
required: true
type: float
- name: format
required: true
type: string
- name: method
required: true
type: string
- name: path
required: true
type: string
- name: remote_ip
required: true
type: string
indicators:
- ip
- name: status
required: true
type: bigint
- name: tag
required: false
type: string
- name: time
required: true
type: timestamp
timeFormats:
- rfc3339
isEventTime: true
- name: view_duration_s
required: true
type: float
```
{% endtab %}
{% endtabs %}
### `test` limitations
* The `test` command only supports JSON events.
## Uploading schemas via PAT
For information on uploading schemas via Panther Analysis Tool (PAT), see [Custom Logs: Uploading log schemas with the Panther Analysis Tool](https://docs.panther.com/data-onboarding/custom-log-types#uploading-log-schemas-with-the-panther-analysis-tool).