Skip to main content
Apache StreamPipes

The open-source platform
for industrial data.

StreamPipes connects to machines and industrial systems, works on the data as it arrives, keeps the history, and turns it into charts and dashboards. It runs on your own servers under the Apache license.

Get startedStar on GitHub

Runs with Docker Compose or Kubernetes.

Adapter library with industrial and software connectors
Connect: pick an adapter for OPC UA, S7, Modbus, MQTT, Kafka and others.Learn more
Connectivity

Works with the systems you already have.

All 147 integrations
OPC UASiemens S7ModbusMQTTApache KafkaApache PulsarNATSInfluxDBApache IoTDBPostgreSQLSlackROSOI4Milvus
ApacheTop-level projectA project of the Apache Software FoundationOpen sourceApache License 2.0Source on GitHub36+AdaptersOPC UA, S7, Modbus, MQTT, Kafka and more111+Processors and sinksFiltering, enrichment, detection, routing, storage
Platform

From the first connection
to the dashboard.

Connecting a machine, processing its data, storing it and charting it usually means four tools. In StreamPipes they are modules of one platform, and all of them are open source.

StreamPipes adapter library
Connect
Adapters connect machines, brokers and other software. You see the incoming events before you save anything, and you fix field names, types and units once.
Explore connectivity
StreamPipes visual pipeline editor
Pipelines
Filter, enrich and route live data in a visual editor. Elements are checked against the stream's schema, so a pipeline that saves is a pipeline that runs.
Explore pipelines
StreamPipes chart editor
Charts and dashboards
Query a dataset, pick a chart type, and put several charts together into a dashboard for the team.
Explore dashboards
Datasets
Stored history for the streams you choose to keep. Inspect it, export it, import CSV files, and set retention rules.
Explore datasets
Assets and sites
Describe your plants, lines and machines, and attach the streams, datasets, pipelines and dashboards that belong to each.
Explore assets
Use cases

What people
build with it.

Scenarios we see often, with the adapters, processors and sinks they use and the guide that goes with each.

Pipeline routing CNC process data to trend detection, a Microsoft Teams notification and dataset storage
Condition monitoring with OPC UA
Connect machines over OPC UA, keep an eye on the values that matter, and get a message when a trend or a threshold needs attention.
Read the use case
Adapter library with PLC4X, Kafka and other connectors
PLC data to Kafka and InfluxDB
Read Siemens S7 and Modbus controllers, clean up units and field names, and forward the events to the systems IT already runs.
Read the use case
Adapter settings with a live event preview
MQTT and a unified namespace
Subscribe to the topics your broker already carries, turn raw payloads into typed streams with context, and publish the results back into the namespace.
Read the use case
How it works

What happens
to an event.

An adapter turns raw signals into a typed stream. Pipelines work on that stream as events arrive. Datasets keep what you want to look at later, and charts and dashboards show it.

  1. 01Machines and systemsOPC UA, S7, MQTT, Kafka
  2. 02ConnectAdapters, schemas, units
  3. 03PipelinesProcessors and sinks
  4. 04DatasetsHistory and retention
  5. 05Charts and dashboardsOperators and teams
01Machines and systemsOPC UA, S7, MQTT, Kafka02ConnectAdapters, schemas, units03PipelinesProcessors and sinks04DatasetsHistory and retention05Charts and dashboardsOperators and teams
Architecture

Several services,
one platform.

A core, a message broker, and as many extension services as you need. Run all of it on one host with Docker Compose, or put the extension services into the plants and the core into the data center.

StreamPipes architecture: interfaces on top, the core and time-series database in the middle, a messaging layer, and extension services with adapters, processors, sinks and Functions connected to machines
Architecture overview
  1. 01

    Interfaces

    The web UI for daily work, plus Java and Python clients and a REST API for anything you want to automate.

  2. 02

    Core

    Keeps track of adapters, pipelines, datasets, assets and users, watches what is running, and serves the UI.

  3. 03

    Messaging layer

    NATS or Kafka moves events between adapters, pipelines and sinks. Nothing talks point to point, so each part can run where it makes sense.

  4. 04

    Extension services

    Adapters, processors, sinks and Functions are packaged as separate services. Write your own with the SDK and install them while the platform is running.

  5. 05

    Time-series storage

    Datasets go into a time-series database. Retention rules decide how long they stay, and you can export them as CSV, JSON or Excel.

Architecture in detail
Deployment

Extension services in the plant, the core in your data center.

Each plant runs a small extension service next to its PLCs and machines. The service connects outbound to the broker, so you do not open any ports into the OT network. The core, the storage and the heavier processing stay in IT.

  1. OTPlant networkPLCs, OPC UA servers and sensors feed an edge extension service with adapters and Functions
  2. FWFirewallEdge services connect outbound to the broker; no inbound ports on the OT side
  3. ITData center or cloudMessage broker, core with UI and APIs, time-series storage, processors and sinks
OTPlant networksPlant APLCSiemens S7, ModbusOPC UA serverMachines, linesSensors, brokersMQTT, files, HTTPExtension service at the edgeAdapters and Functions, deployed with Docker next to the machinesPlant BPLCSiemens S7, ModbusOPC UA serverMachines, linesSensors, brokersMQTT, files, HTTPExtension service at the edgeAdapters and Functions, deployed with Docker next to the machinesFirewallITData center or cloudMessage brokerNATS or Kafkaoutbound only, TLSCoreUI, management, monitoring, APIsTime-series storageDatasets, retention, exportExtension services in ITProcessors and sinksClients and third-party systemsPython, Java, REST, databases, brokers
  1. 01

    Run it where the data is

    Put extension services into the plant network, next to the equipment, and keep the core in the data center or in the cloud. The broker connects the two.

  2. 02

    Start small, grow in place

    One Docker Compose host is enough to try things out. The same services scale out on Kubernetes later, and your pipelines stay as they are.

  3. 03

    The data stays yours

    Everything runs on infrastructure you control. There is no usage metering and no proprietary format, and you can export a dataset whenever you want to.

Deployment optionsExtension services
Enterprise readiness

The things
IT will ask about.

Single sign-on, roles and permissions, an audit log, retention rules and encrypted connections are part of the platform. There is no separate enterprise edition.

Single sign-on
Sign in through your identity provider with OpenID Connect or OAuth2. Several providers can be configured side by side, and roles can be taken from token claims.
Configure OpenID Connect
Roles, groups and permissions
Built-in roles, custom roles made from individual privileges, user groups, and permissions on every resource. Service accounts and API keys for anything automated.
User management
Audit log
Logins and changes to adapters, datasets, charts, dashboards, exports and security settings are recorded. Filter the log, download it, and give it its own retention rule.
Audit log
Data retention and archival
Each dataset gets a retention policy that deletes old data, archives it to a folder or S3, or does both, on a daily, weekly or monthly schedule.
Dataset retention
Secured connections
HTTPS for the UI, encrypted secrets at rest, and certificate management for OPC UA, MQTT and Kafka with a warning before a certificate expires.
Certificates and transport
Operations
Metrics and logs for every adapter and pipeline, a Prometheus endpoint, a Helm chart that includes Prometheus and Grafana, and migrations that run on upgrade.
Deploy and operate

Also in the box: a terms-of-use screen after login, switches for self-registration and password recovery, and a configurable app name. All of it under the Apache license.

Developers

Use it from
code, too.

The visual tools are the start. Python and Java clients, Functions for event-driven code and an SDK for your own extensions cover the rest.

Your data in a notebook.

Load a dataset into pandas, subscribe to a live stream, or script the platform from Python.

Python client docs
explore_dataset.pyPython
# With an authenticated StreamPipes client
# and an existing dataset named "flow-rate":

dataset = client.dataLakeMeasureApi.get(
    identifier="flow-rate"
)

frame = dataset.to_pandas()
print(frame.describe())
Get started

Up and running
in a few minutes.

The command on the right downloads the Compose files for release 0.98.0 and starts StreamPipes with Docker. From there, the quickstart takes you from a first data source to a dashboard.

Open the quickstartAll downloads
docker composerelease 0.98.0
mkdir -p streampipes-0.98.0 && cd streampipes-0.98.0 \
  && curl -fsSLO https://raw.githubusercontent.com/apache/streampipes/0.98.0/installer/compose/docker-compose.yml \
  && curl -fsSLO https://raw.githubusercontent.com/apache/streampipes/0.98.0/installer/compose/.env \
  && docker compose up -d

GitHub Discussions

Questions, ideas, and showing what you built. The maintainers read it.

Mailing lists

The user list for questions, the dev list for decisions. This is how Apache projects work.

Support and training

The community helps on GitHub. If you need guaranteed response times, companies around the project offer support contracts.

Contribute

Code, docs, tests, ideas. Start with a discussion or pick an issue labelled good first issue.