# pantherlog Tool
## Overview
You can use `pantherlog`, a CLI tool, to work with Custom Logs. It parses logs using Panther-managed or [custom schemas](https://docs.panther.com/data-onboarding/custom-log-types), and uses sample logs to infer custom schemas.
For information on working with custom logs in the Panther Console instead, see the [Custom Logs](https://docs.panther.com/data-onboarding/custom-log-types) documentation.
### pantherlog limitations
* See the [infer limitations](#infer-limitations) and [test limitations](#test-limitations) sections, below.
## Download
Download the latest version at the following links:
## `list-schemas`: List Panther-managed schemas
You can use pantherlog's `list-schemas` command to list Panther's managed schemas.
```
./pantherlog list-schemas
```
## `export-schemas`: Export Panther-managed schemas
You can use pantherlog's `export-schemas` command to export Panther-managed schemas into a local directory, or print them in the terminal.
### Export schemas to local directory
```
./pantherlog export-schemas --path directory-name
```
* If `directory-name` does not exist, it will be created.
* Note that `-p` may be used in place of `--path`.
### Print schemas
To print schemas to `stdout` instead of exporting to a local directory, use a dash.
```
./pantherlog export-schemas -p -
```
### Export select schemas
You can filter the schemas to be exported by using the `-s` option with the names of the schemas you'd like to export, separated by commas.
```
./pantherlog export-schemas --path ./managed-schemas -s 'AWS.ALB,Slack.AuditLogs'
```
## `infer`: Generate a schema from JSON log samples
You can use pantherlog to generate a schema file out of sample files in [new-line delimited JSON](https://jsonlines.org/) format. The tool will scan the provided logs and print the inferred schema to `stdout`.
For example, to infer the schema of logs `sample_logs.jsonl` and output to `schema.yml`, use:
```
$ ./pantherlog infer --name SchemaName sample_logs.jsonl > schema.yml
```
Note that YAML keys and values are case sensitive. The tool will attempt to infer multiple timestamp formats. Learn more about schema inference on [Custom Logs](https://docs.panther.com/data-onboarding/custom-log-types#automatically-infer-the-schema-in-panther).
{% embed url="" %}
The workflow of inferring a schema from sample logs
{% endembed %}
### `infer` limitations
The `infer` command will not mark any timestamp field as `isEventTime:true`. Make sure to select the appropriate `timestamp` field and mark it as `isEventTime:true`. For more information regarding `isEventTime:true`, see the [Timestamps section on Log Schema Reference](https://docs.panther.com/data-onboarding/custom-log-types/reference#timestamps).
The `infer` command is able to infer only the following types of [indicators](https://docs.panther.com/search/panther-fields#indicator-fields): `ip`, `aws_arn`, `url`, `email`, `hash` digests (`MD5`, `SHA1` and `SHA2`), and `mac`. Make sure to review the fields and add more indicators as appropriate.
It's strongly recommended to review the schema generated by the `infer` command, and edit it appropriately before deploying to your production environment.
## `parse`: Validate a schema
You can use the tool to validate a schema file and use it to parse log files. Note that the events in the log files need to be separated by new line. Processed logs are written to `stdout` and errors to `stderr`.
For example, to parse logs in `sample_logs.jsonl` with the log schema in `schema.yml`, use:
```
$ ./pantherlog parse --path schema.yml --schemas Schema.Name sample_logs.jsonl
```
The tool can also accept input via `stdin` so it can be used in a pipeline:
```
$ cat sample_logs.jsonl | ./pantherlog parse --path schema.yml
```
## `test`: Run tests for a schema
You can use pantherlog to run unit tests for your custom schema. To run tests defined in a `schema_tests.yml` file for a custom schema defined in `schema.yml`, you would run:
```
$ ./pantherlog test schema.yml schema_tests.yml
```
The first argument is a file or directory containing schema YAML files. The rest of the arguments are test files to run. If you don't specify any test files arguments, and the first argument is a directory, the tool will look for tests in YAML files with a `_tests.yml` or `_tests.yaml` suffix.
### Creating a schema test file
In your test file, include an `input` key containing the event to parse, and a `result` key containing the expected result. The pantherlog `test` command checks that the schema can parse the event without error, and that the normalized event matches your expected result.
{% hint style="info" %}
The `result` event should include any [Panther Standard Fields](https://docs.panther.com/search/panther-fields) that are expected to be injected into the event during parsing, such as `p_log_type`.
{% endhint %}
Take note of [these instructions on working with `timeFormats` in schema tests](https://docs.panther.com/data-onboarding/custom-log-types/reference#working-with-timeformats-in-schema-tests).
{% hint style="info" %}
For an example of writing multiple tests for one schema, see this article in Panther's Knowledge Base: [How can I write multiple pantherlog tests for a schema?](https://help.panther.com/Data_Sources/Custom_Logs/How_can_I_write_multiple_pantherlog_tests_for_a_schema%3F)
{% endhint %}
Example:
{% tabs %}
{% tab title="schema\_tests.yml" %}
```json
# Make sure to use camelCase when naming the schema or log type
name: Custom Log Test Name
logType: Custom.SampleLog.V1
input: |
{
"method": "GET",
"path": "/-/metrics",
"format": "html",
"controller": "MetricsController",
"action": "index",
"status": 200,
"params": [],
"remote_ip": "1.1.1.1",
"user_id": null,
"username": null,
"ua": null,
"queue_duration_s": null,
"correlation_id": "c01ce2c1-d9e3-4e69-bfa3-b27e50af0268",
"cpu_s": 0.05,
"db_duration_s": 0,
"view_duration_s": 0.00039,
"duration_s": 0.0459,
"tag": "test",
"time": "2019-11-14T13:12:46.156Z"
}
result: |
{
"action": "index",
"controller": "MetricsController",
"correlation_id": "c01ce2c1-d9e3-4e69-bfa3-b27e50af0268",
"cpu_s": 0.05,
"db_duration_s": 0,
"duration_s": 0.0459,
"format": "html",
"method": "GET",
"path": "/-/metrics",
"remote_ip": "1.1.1.1",
"status": 200,
"tag": "test",
"time": "2019-11-14T13:12:46.156Z",
"view_duration_s": 0.00039,
"p_log_type": "Custom.SampleLog.V1",
"p_row_id": "acde48001122a480ca9eda991001",
"p_event_time": "2019-11-14T13:12:46.156Z",
"p_parse_time": "2022-04-04T16:12:41.059224Z",
"p_any_ip_addresses": [
"1.1.1.1"
]
}
```
{% endtab %}
{% tab title="schema.yml" %}
```yaml
version: 0
schema: Custom.SampleLog.V1
fields:
- name: action
required: true
type: string
- name: controller
required: true
type: string
- name: correlation_id
required: true
type: string
- name: cpu_s
required: true
type: float
- name: db_duration_s
required: true
type: bigint
- name: duration_s
required: true
type: float
- name: format
required: true
type: string
- name: method
required: true
type: string
- name: path
required: true
type: string
- name: remote_ip
required: true
type: string
indicators:
- ip
- name: status
required: true
type: bigint
- name: tag
required: false
type: string
- name: time
required: true
type: timestamp
timeFormats:
- rfc3339
isEventTime: true
- name: view_duration_s
required: true
type: float
```
{% endtab %}
{% endtabs %}
### `test` limitations
* The `test` command only supports JSON events.
## Uploading schemas via PAT
For information on uploading schemas via Panther Analysis Tool (PAT), see [Custom Logs: Uploading log schemas with the Panther Analysis Tool](https://docs.panther.com/data-onboarding/custom-log-types#uploading-log-schemas-with-the-panther-analysis-tool).
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://docs.panther.com/panther-developer-workflows/pantherlog.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.