Parser module

This module defines a python interface for using Logstash as a dataset parser.

LogstashParser

Utility class for controling Logstash

__init__(self, dataset_config, parser_config, logstash) special

Parameters:

Name	Type	Description	Default
dataset_config	DatasetConfig	The dataset configuration	required
parser_config	LogstashParserConfig	The logstash configuration (e.g., CLI options etc.)	required
logstash	Path	The path to the logstash executable	required

Source code in dataset/parser.py

def __init__(
    self,
    dataset_config: DatasetConfig,
    parser_config: LogstashParserConfig,
    logstash: Path,
):
    """
    Args:
        dataset_config: The dataset configuration
        parser_config: The logstash configuration (e.g., CLI options etc.)
        logstash: The path to the logstash executable
    """
    self.dataset_config: DatasetConfig = dataset_config
    self.parser_config: LogstashParserConfig = parser_config
    self.logstash = logstash
    self._child: Optional[subprocess.Popen] = None
    self._sigint_handler: Union[
        Callable[[signal.Signals, FrameType], None], int, signal.Handlers, None
    ] = None

parse(self)

Execute the parsing process by running logstash as sub process.

Source code in dataset/parser.py

def parse(self):
    """Execute the parsing process by running logstash as sub process."""
    args = [
        str(self.logstash.absolute()),
        "--path.settings",
        str(self.parser_config.settings_dir.absolute()),
    ]

    if self.parser_config.log_level is not None:
        args.extend(["--log.level", self.parser_config.log_level])

    self.proc = subprocess.Popen(args)

    self.proc.wait()