DataTracks is a streaming engine, which simplifies the setup up and management of data stream processing, whether the data is structured or unstructured. It is aimed at developers and data engineers who need a reliable and scalable solution for handling diverse data types and streaming workloads.
- Easy Setup: Quickly set up data streams with minimal configuration.
- Flexibility: Supports both structured and unstructured data in various data formats.
- Prototype System: Currently in the prototype stage, providing a foundational system for further development.
DataTracks is a prototype engine, which provides simple creation and management of complex data streaming workloads for various degrees of structured and unstructured data.
A simple plan which restructures an input MQTT stream, aggregates it with SQLite data and sends it forward might look like this:
1--2{sql|SELECT {time: $1.timestamp, id: $lite($1.name), name: $1.name} FROM $1}--3
In
MQTT{...}:1
Out
TPC{...}:3
Transform
$lite:SQLite{"query":"SELECT \"id\" FROM \"company\" WHERE \"name\" = $",...}
DataTracks uses an abstraction model based on train and track logic.
In its model a data point is represented by a Wagon, which gets assigned timestamp on initialization at the
client (wagon number).
Multiple Wagons can be connected to one another as a Train.
Trains travel a track Plan, which defines how different components are connected.
Each component is represented by a Stations.
Stations can receive and send out Trains. If a Train passes through a Station
it has tho conform to the structural requirement of the station (Who?).
Trains might are collected at a Station and grouped into time windows according to their train number (
Where?).
Further Trains might be held back until a defined condition is reached (When?).
Due to counterparties there might be the a late Train arriving later at the station (How?).
{sql|SELECT $1 FROM $1}
...}[3s]
...}(f?) //nullable
...}(s') //optional
...}({name: s'}) // document with key name of optional string type
...}@windowEnd
...}@element
...}@windowNext
...}@windowNext@element // trigger on next window and on element
{sql|SELECT $1 FROM $1}(f?)[3s]@watermark // all
SELECT *
FROM $0
WHERE id == "TEST"
GROUP BY "salary"
WINDOW INTERVAL (5, SECONDS)
EMIT ELEMENT, WINDOW (END)
MARK LASTgit submodule update --recursive --remote