Concepts

Passive Monitoring

NetEye’s passive monitoring abilities count on Tornado, a Complex Event Processor that receives reports of events from data sources such as monitoring, email, and telegram, matches them against pre-configured rules, and executes the actions associated with those rules, which can include sending notifications, logging to files, and annotating events in a time series graphing system.

Tornado is a high performance, scalable application, and is intended to handle millions of events each second on standard server hardware. Its overall architecture is depicted in Tornado architecture.

Fig. 151 Tornado architecture

Tornado Architecture

Tornado is structured as a library, with three example binaries included that show how it can be used. The three main components of the Tornado architecture are:

• The Tornado Collector(s), or just Collector(s)

• The Tornado Engine, or Engine

• The Tornado Executor(s), or Executor(s)*

The term Tornado refers to the whole project or to a deployed system that includes all three components.

Along with the main components, the following concepts are fundamental to the Tornado architecture:

• An Action: An operation performed by Tornado, usually on an external system. For example, writing to Elastic Search or setting a state in a monitoring system.

• A Tornado (or Internal) Event: The Tornado-specific Event format.

• A Datasource: A system that sends External Events to Tornado, or a system to which Tornado subscribes to receive External Events.

• An External Event: An input received from a datasource, whose format depends on its source. An example of input are events collected from rsyslog.

• A Rule: A group of conditions that an Internal Event must match to trigger a set of Actions.

See also

Events and Actions are described in more detail in the Tornado common API documentation.

Architecturally, Tornado is organized as a processing pipeline, where input events move from Collectors to the Engine, to Executors, without branching or returning.

When the system receives an External Event, it first arrives at a Collector where it is converted into a Tornado Event. Then it is forwarded to the Tornado Engine where it is matched against user-defined, composable Rules. Finally, generated Actions are dispatched to the Executors.

The Tornado pipeline:

Datasources (e.g. rsyslog)
  |
  | External Events
  |
  \-> Tornado Collectors
        |
        | Tornado (or Internal) Events
        |
        \-> Tornado Engine (matches based on Rules)
              |
              | Actions
              |
              \-> Tornado Executors (execute the Actions)

Collectors

The purpose of a Collector is to receive and convert external events into the internal Tornado Event structure, and forward them to the Tornado Engine.

Collectors are Datasource-specific. For each datasource, there must be at least one Collector that knows how to manipulate the datasource’s Events and generate Tornado Events.

Out of the box, Tornado provides a number of Collectors for handling inputs from snmptrapd, rsyslog, JSON from Nats channels and generic Webhooks.

Because all Collectors are defined with a simple format, Collectors for new event types can easily be added or extended from existing types for:

• Monitoring events

• Email messages

• Telegram

• DNS

• Cloud monitoring (AWS, Azure, Cisco/Meraki, etc.)

• Netflow

• Elastic Stack

• SMS

• Operating system and authorization events

More information on the Collectors can be found in Section Tornado Collectors.

Engine

The Engine is the second step of the pipeline. It receives and processes the events produced by the Collectors. The outcome of this step is fully defined by a processing tree composed of Filters and Rule Sets.

A Filter is a processing node that defines an access condition on the children nodes.

A Rule Set is a node that contains an ordered set of Rules, where each Rule determines:

• The conditions a Tornado Event has to respect to match it

• The actions to be executed in case of a match

The processing tree is parsed at startup from a configuration folder where the node definitions are stored in JSON format.

When an event matches one or more Rules, the Engine produces a set of Actions and forwards them to one or more Executors.

Executors

The Executors are the last element in the Tornado pipeline. They receive the Actions produced from the Engine and trigger the associated executable instructions.

An Action can be any command, process or operation. For example it can include:

• Forwarding the events to a monitoring system

• Logging events locally (e.g., as processed, discarded or matched) or remotely

• Archiving events using software such as the Elastic Stack

• Invoking a custom shell script

A single Executor usually takes care of a single Action type.

More information and example code snippets can be found in Section Tornado Executors.

Processing Tree

Within the NetEye interface you can view the Tornado rule configuration graphically instead of in a command shell. The Configuration Viewer (available after clicking Tornado in the left-hand side menu) shows the processing tree in a top-down format, allowing you to verify that the structure of the rules you have created is as you intended.

Definitions

While a more complete description of all Tornado elements is available in the official Tornado engine documentation, they are detailed below.

• Processing Tree: The entire set of filters and rules that creates the hierarchical structure where events are filtered and then matched against one or more rules.

• Filter: A node in the processing tree that contains (1) a filter definition and (2) a set of child nodes, each of which corresponds to a condition in the filter definition. You can use a Filter to create independent pipelines for different types of events, reducing the amount of time needed to process them.

• Implicit Filter: If a filter is not present in a node, a default filter is created which forwards an event to ALL child nodes, rather than a particular one that matched a filter condition.

• Ruleset: A leaf node that contains multiple rules to be matched against the event one by one once the filters in the parent nodes have let an event through.

• Rule: A single rule within a Ruleset, which can be matched against an event.

Details about the configuration and set up of the various Tornado elements is available in Section Processing Tree Configuration.

Tornado Rule Editor

The tornado Rule Editor allows to manage the single rules that are in a ruleset. On top of the window, all the parents of the current ruleset are shown, to allow you to quickly check on which leaf of the processing tree the rules shown are located. Like in the processing tree editor, a JSON validator assists you in checking that the syntax of the rules is correct.

In the main area, all the defined rules are shown, together with a number of information about them: name, action, and status (enabled or not).

Like in the Processing Tree Editor, available options differ depending whether the Edit mode is active or not:

When the Edit mode is not active, it is possible to click on the Open test window button on the top right-hand side of the window to check which events the current rule selection would match.

With active Edit mode, the Open test window button is disabled, but new rules can be added, modified, or deleted; each rule can also can be moved along the list with a simple drag and drop.