CLI Reference¶
This page provides documentation for our command line tools.
cr-kyoushi-dataset¶
Run Cyber Range Kyoushi Dataset.
Usage:
cr-kyoushi-dataset [OPTIONS] COMMAND [ARGS]...
Options:
| Name | Type | Description | Default |
|---|---|---|---|
--dataset, -d |
directory | The dataset to process | ./ |
--logstash, -l |
file | The logstash binary to use for parsing | /usr/share/logstash/bin/logstash |
--elasticsearch, -e |
text | The connection string for the elasticsearch database | http://127.0.0.1:9200 |
--help |
boolean | Show this message and exit. | False |
cr-kyoushi-dataset label¶
Apply the labeling rules to the dataset
RULE_DIRS The directories from which to load the label rules (defaults to
Rules are automatically loaded from all *.json, *.yaml, *.yml files in the given rule dirs.
Usage:
cr-kyoushi-dataset label [OPTIONS] [RULE_DIRS]...
Options:
| Name | Type | Description | Default |
|---|---|---|---|
--dataset-config |
file | The dataset configuration file (defaults to |
./dataset.yaml |
--label-object |
text | The field to store the labels in | kyoushi_labels |
--label / --no-label |
boolean | If the labeling rules should be applied or not | True |
--write / --no-write |
boolean | If the label files should be written or not | True |
--write-skip-files |
text | Optionally a comma separated list of log files to not write labels for.(if this is not set label files will be written for all files with labeled log lines) | required |
--write-exclude-index, -e |
text | Comma separated list of indices to explicitly exclude when writing label files | required |
--help |
boolean | Show this message and exit. | False |
cr-kyoushi-dataset prepare¶
Usage:
cr-kyoushi-dataset prepare [OPTIONS]
Options:
| Name | Type | Description | Default |
|---|---|---|---|
--gather-dir, -g |
directory | The logs and facts gather source directory. This directory will be copied to the dataset directory. | required |
--process-dir, -p |
directory | The processing source directory (containing the process pipelines, templates and rules. | required |
--name |
text | The name to use for the dataset (will be prompted if not supplied) | required |
--start |
datetime (%Y-%m-%d | %Y-%m-%dT%H:%M:%S | %Y-%m-%d %H:%M:%S | %Y-%m-%dT%H:%M:%S.%f | %Y-%m-%d %H:%M:%S.%f | %Y-%m-%dT%H:%M:%S%z | %Y-%m-%dT%H:%M:%SZ | %Y-%m-%dT%H:%M:%S.%f%z | %Y-%m-%dT%H:%M:%S.%fZ) |
The the datasets observation start time (will be prompted if not supplied) | required |
--end |
datetime (%Y-%m-%d | %Y-%m-%dT%H:%M:%S | %Y-%m-%d %H:%M:%S | %Y-%m-%dT%H:%M:%S.%f | %Y-%m-%d %H:%M:%S.%f | %Y-%m-%dT%H:%M:%S%z | %Y-%m-%dT%H:%M:%SZ | %Y-%m-%dT%H:%M:%S.%f%z | %Y-%m-%dT%H:%M:%S.%fZ) |
The the datasets observation end time (will be prompted if not supplied) | required |
--yes, -y |
boolean | Affirm all confirmation prompts (use for non-interactive mode) | False |
--help |
boolean | Show this message and exit. | False |
cr-kyoushi-dataset process¶
Process the dataset and prepare it for labeling.
Usage:
cr-kyoushi-dataset process [OPTIONS]
Options:
| Name | Type | Description | Default |
|---|---|---|---|
--config, -c |
file | The processing configuration file (defaults to |
./processing/process.yaml |
--dataset-config |
file | The dataset configuration file (defaults to |
./dataset.yaml |
--skip-pre |
boolean | Skip the pre processing phase | False |
--skip-parse |
boolean | Skip the parsing phase | False |
--skip-post |
boolean | Skip the post processing phase | False |
--help |
boolean | Show this message and exit. | False |
cr-kyoushi-dataset sample¶
Usage:
cr-kyoushi-dataset sample [OPTIONS] [SIZE]
Options:
| Name | Type | Description | Default |
|---|---|---|---|
--dataset-config |
file | The dataset configuration file (defaults to |
./dataset.yaml |
--label-object |
text | The field to store the labels in | kyoushi_labels |
--label, -l |
text | The label to get sample log lines for (if this is not set then unlabeled log lines will be sampled) | required |
--from-timestamp |
datetime (%Y-%m-%d | %Y-%m-%dT%H:%M:%S | %Y-%m-%d %H:%M:%S | %Y-%m-%dT%H:%M:%S.%f | %Y-%m-%d %H:%M:%S.%f | %Y-%m-%dT%H:%M:%S%z | %Y-%m-%dT%H:%M:%SZ | %Y-%m-%dT%H:%M:%S.%f%z | %Y-%m-%dT%H:%M:%S.%fZ) |
Optional minium timestamp for log rows to consider | required |
--until-timestamp |
datetime (%Y-%m-%d | %Y-%m-%dT%H:%M:%S | %Y-%m-%d %H:%M:%S | %Y-%m-%dT%H:%M:%S.%f | %Y-%m-%d %H:%M:%S.%f | %Y-%m-%dT%H:%M:%S%z | %Y-%m-%dT%H:%M:%SZ | %Y-%m-%dT%H:%M:%S.%f%z | %Y-%m-%dT%H:%M:%S.%fZ) |
Optional maximum timestamp for log rows to consider | required |
--files, -f |
text | Optionally a comma separated list of files to get sample log lines from (if this is not set all files matching the label option will be drawn from). | required |
--related, -r |
text | Optionally a comma separated list of elasticsearch indices for which to include the log line, that is closest (based on the timestamp) to the selected sample, as meta information. Given indices are prefixed with the dataset name. | required |
--default-label |
text | The label to assign to unlabeled log row (e.g., when --label is not used) | normal |
--index, -i |
text | Comma separated list of indices to consider for sampling | required |
--exclude-index, -e |
text | Comma separated list of indices to explicitly exclude from the sampling | required |
--seed, -s |
text | The random seed to use for the sampling query | required |
--seed-field |
text | The field to use for the elasticsearch random score | _seq_no |
--list |
boolean | Only list the available labels with their log line counts as JSON array | False |
--help |
boolean | Show this message and exit. | False |
cr-kyoushi-dataset version¶
Get the library version.
Usage:
cr-kyoushi-dataset version [OPTIONS]
Options:
| Name | Type | Description | Default |
|---|---|---|---|
--help |
boolean | Show this message and exit. | False |