{ "version": "https://jsonfeed.org/version/1", "title": "Apache StreamPipes blog", "home_page_url": "https://streampipes.apache.org/blog/", "description": "Release notes, how-tos and news from the Apache StreamPipes project.", "items": [ { "id": "https://streampipes.apache.org/blog/2026/10/30/assets-as-the-backbone/", "content_html": "

If you have used StreamPipes for a while, you will have noticed that the number of things in it grows quickly. One machine easily ends up with an adapter, a persisted dataset, two or three pipelines, a handful of charts and a dashboard. Multiply that by a line, and then by a plant, and the lists in the UI stop being useful. Everything is there, but you have to remember what it is called.

\n

Assets are our answer to that. They let you describe the plant the way people on the floor think about it, as plants, lines, machines and components, and attach all of those technical resources to the object they belong to. Over the past year most of the UI has been rebuilt around this idea, and in this post I want to explain what that means in practice.

\n

The idea in one sentence​

\n

An asset is a real-world thing: a plant, a line, a machine, a station, a subsystem. Assets can be nested, so a plant contains lines and a line contains machines. Each asset can be linked to the StreamPipes resources that belong to it. Once those links exist, you can start from the machine and find everything around it, instead of starting from a list of adapters and working out which one is which.

\n

Sites add a second dimension. A site is shared location context, such as a factory, a building or a hall, with an optional area. You define a site once and assign it to as many assets as you like. An asset can also carry an exact position, which is what puts it on the map.

\n

Start on the home screen​

\n

The home screen is the first place where this shows. Instead of a list of modules, it opens with your assets. In table mode every row is one asset with its type, location and a row of icons for the linked resources, and each icon takes you straight to that adapter, pipeline, chart or dashboard. Switch to map mode and the same assets appear as markers on a map, which is the view we see most often on large screens in control rooms.

\n

\"Home

\n

One page per machine​

\n

Open an asset and you get a detail page: identifier, type, labels, the site and area it belongs to, a small map if a position is set, and the full list of linked resources. The demo machine below has 29 of them, three adapters, their datasets, the pipelines that persist and process the data, and the charts and dashboards built on top. This is the page we point people to when they ask \"where is the data for this machine\".

\n

\"Asset

\n

Asset context everywhere else​

\n

The part that took the longest, and that I think matters most, is that assets are not confined to their own module. Every resource list in StreamPipes, adapters, pipelines, datasets, charts and dashboards, shows the asset context of each row and can be filtered by it. The filter button in the top right lets you narrow a list by asset, by asset type, by site and by label, and the filter is kept while you edit resources.

\n

So an operator responsible for the Hamburg plant can set the filter and see only the adapters, pipelines and dashboards for that plant. A maintenance team interested in everything tagged critical can do the same with a label.

\n

\"Asset

\n

New resources pick up the context automatically where it makes sense. When a pipeline persists a stream into a dataset, that dataset is assigned to the same asset as the pipeline.

\n

Modelling your own plant​

\n

The asset editor shows the hierarchy on the left and the details of the selected node on the right. A few things we learned while building and using it:

\n\n

\"Asset

\n

Linking resources is the last step, and the one most often forgotten. The editor lets you add links one at a time or open a dialog to attach several at once. Our rule of thumb is to model the hierarchy first, assign the site second and link resources third. The structure should exist before the technical objects are hung onto it.

\n

\"Site

\n

Permissions follow the model​

\n

Assets have their own permissions, so a shared installation can give one team edit rights on their line and read access on the rest. Combined with the filters, this is how we run installations where several groups share one StreamPipes instance without seeing each other's clutter.

\n

Where this is heading​

\n

Most of what is shown here is in the development version of StreamPipes and will ship with the next release; parts of it are already in 0.98.0. The direction is clear: assets are becoming the way you navigate the platform, and new features are designed with that context in mind from the start. If you run StreamPipes today, it is worth taking an afternoon to model your plant. The benefit grows with every resource you add afterwards.

\n

The assets and sites documentation covers the editor step by step. If you have ideas for what asset context should do next, the mailing lists and GitHub Discussions are the right places.

", "url": "https://streampipes.apache.org/blog/2026/10/30/assets-as-the-backbone/", "title": "Assets are the backbone of StreamPipes", "summary": "Why StreamPipes organises adapters, datasets, pipelines, charts and dashboards around machines, lines and plants, and how to model your own assets.", "date_modified": "2026-10-30T00:00:00.000Z", "author": { "name": "Dominik Riemer", "url": "https://github.com/dominikriemer" }, "tags": [ "tutorial" ] }, { "id": "https://streampipes.apache.org/blog/2026/07/14/streampipes-vs-node-red/", "content_html": "

A question we get a lot: \"We already use Node-RED, why would we need StreamPipes?\" It is a fair question. Both tools have a browser-based editor where you connect boxes into a flow, and if that is all you have seen of either, they look like the same thing. They are not, and this post tries to explain where the difference is, so you can pick the right one for the job, or use both.

\n

What Node-RED is built for​

\n

Node-RED is a general-purpose, flow-based programming tool. A flow runs inside a single Node.js process, and the node library covers almost everything: HTTP, MQTT, serial ports, databases, cloud services, and thousands of community nodes. It is excellent for wiring devices and services together, for prototyping, for small automations at the edge, and for home and lab setups. A single developer can build something useful in an afternoon.

\n

What Node-RED does not try to be is a data platform. There is no built-in time-series storage, no idea of a dataset with a retention rule, no user and role management beyond basic authentication, and no catalog of typed streams. Dashboards are an add-on. Once you run many flows across several sites, you end up building the operations side yourself.

\n

What Apache StreamPipes is built for​

\n

StreamPipes is an industrial IoT data platform. Its unit of work is a typed data stream: an adapter connects a source such as OPC UA, Siemens S7, Modbus or MQTT, describes the fields, units and metadata once, and every pipeline, chart and dashboard builds on that description. Pipelines are built from processors and sinks that are validated against the stream schema, run as separate services, and are monitored by the core. Datasets keep history with retention rules and export. Assets and sites give streams their place in the plant. Users, roles, groups, permissions, single sign-on and an audit log are part of the platform.

\n

That is the setting StreamPipes is made for: several people, several machines, data that has to be kept and governed, and one platform that operators, engineers and developers all use.

\n

Where the difference is biggest: getting data out of machines​

\n

This is the point most comparisons gloss over, so it deserves its own section. Both tools can talk to a PLC. The experience of doing so is very different.

\n

In Node-RED, industrial connectivity comes from community nodes. The OPC UA nodes, for example, expect you to know the node ids you want to read, to type them into a node configuration or inject them from a function node, and to deal with security settings, reconnects and data types in the flow yourself. That works well for a handful of tags on a server you know. It becomes hard to maintain once there is a server with thousands of nodes, several machines from different vendors, certificates to manage, and colleagues who have to understand the flow later.

\n

StreamPipes treats machine connectivity as a product feature, not as a library. The OPC UA adapter opens the server's node tree in a browser, lets you pick the variables you want, and shows live values before you commit. You choose pull or subscription mode per adapter, pick the security mode and policy, and either upload a client certificate or let StreamPipes generate one. Certificates are managed in one place for all adapters, and X.509 user authentication is supported. Complex types in extension objects are decoded into flat fields. A second adapter subscribes to OPC UA alarms and conditions and turns them into a stream of events. All of that is configured in a wizard, with the schema, units and metadata of each field refined in the next step, and saved as an adapter that colleagues can inspect, copy and restart.

\n

The same applies beyond OPC UA. StreamPipes ships adapters for Siemens S7, Modbus TCP, RTU and ASCII, Beckhoff TwinCAT ADS, Allen-Bradley and Logix CIP, EtherNet/IP, CANopen and generic CAN, IEC 60870-5-104, Mitsubishi SLMP, Schneider UMAS, KNXnet/IP, OI4 and the Siemens WinCC alarm archive, next to the usual MQTT, Kafka, NATS, HTTP, file and database sources. Every adapter follows the same pattern: a guided configuration, a live preview of the incoming events, a schema step where you fix field names, types and measurement units and can run a short transformation script against a sample, and then monitoring of throughput, errors and logs once it runs. When a machine changes, you edit the adapter and the change is validated against the pipelines that depend on it.

\n

That is what we mean by professional-grade machine data integration. It is not that Node-RED cannot read a PLC. It is that in StreamPipes the hard parts are done once, in the platform, and every machine after the first one benefits.

\n

Where each one wins​

\n
QuestionNode-REDApache StreamPipes
Wire a device to a service in an afternoonYes, with the broadest node library availableYes, for anything with an adapter; the SDK covers the rest
Connect industrial machinesCommunity nodes, node ids and security handled in the flowGuided adapters with OPC UA node browser, certificate management, PLC protocols, live preview and schema editor
Keep and query months of machine historyNot built inDatasets with retention, export and charts
Typed streams with units and metadataPer flow, by conventionCore concept, shared by all features
Dashboards for operatorsVia the dashboard add-onBuilt in, with assets and permissions
Users, roles, SSO, audit logBasicBuilt in
Run processing next to the machines, manage centrallyMultiple independent instancesExtension services in OT, one core in IT
Custom logicJavaScript function nodesJava or Python Functions, SDK for reusable elements
Operating modelOne process per instanceCore, broker and extension services on Docker or Kubernetes
\n

Using both​

\n

The two get along fine. Node-RED can act as a bridge for a device StreamPipes has no adapter for, publishing to MQTT, where a StreamPipes adapter picks it up. In the other direction, StreamPipes can publish derived values to a broker that Node-RED flows react to. If you have Node-RED flows that work, keep them.

\n

Summary​

\n

Use Node-RED for integrations and automations where the flow is the whole product and one process is enough. Use StreamPipes when the sources are machines and the data has to become something a team shares: connected through adapters that handle the protocols, certificates and schemas properly, kept with history, shown on dashboards and governed with access control. The integrations catalog lists every adapter, and the quickstart shows the StreamPipes side in about ten minutes.

", "url": "https://streampipes.apache.org/blog/2026/07/14/streampipes-vs-node-red/", "title": "When to use Apache StreamPipes, and when Node-RED is the better fit", "summary": "Node-RED and StreamPipes both have a flow editor, and that is roughly where the similarity ends. Where each one is strong, why StreamPipes is the better fit for professional machine data integration, and when to use which.", "date_modified": "2026-07-14T00:00:00.000Z", "author": { "name": "Dominik Riemer", "url": "https://github.com/dominikriemer" }, "tags": [ "comparison" ] }, { "id": "https://streampipes.apache.org/blog/2026/03/21/_streampipes-with-windows-and-podman/", "content_html": "

When it comes to ease of use, Docker Desktop is the goto on windows platform, however for business use there are licensing involved.

\n

So why not use Podman Desktop, it’s Docker compatible?\nWell yes, almost, would be the answer.
\nFor Podman Desktop to handle Apache StreamPipes we need to do a few tricks.

\n

This guide walks you through installing and running Apache StreamPipes using Podman inside a WSL environment and will try to explain to you the how’s and why’s.

\n

Prerequisits​

\nEnable WSL on your machine.\n
\n Install Podman Desktop.\n

After installing Podman Desktop you need to run the setup to enable Podman core.

\n\"Setup\n

Alter this you can setup a virtual machine in which to host Apache Streampipes on.

\n\"Setup\n

Now we have to embark on a voyage into the (dark) Linux terminal country, so that we can later enjoy the use of a familiar Windows UI in Podman Desktop.

\n
\n

Connect to Your Podman WSL Machine​

\n

Open PowerShell to get a list of available machines.

\n
wsl -l
\n\"Available\n
\n
\n

Or you can see the Podman machine in the tray menu

\n\"Available\n
\n

In PowerShell connect to the Podman machine.

\n
wsl -d podman-machine-default
\n

Install Required Tools​

\n

Podman uses Fedora Linux and the dnf package manager. Install the following packages to make your life easier.

\n\n
sudo dnf install podman-compose.noarch
\n\n
sudo dnf install unzip.x86_64
\n\n
sudo dnf install nano.x86_64
\n\n
sudo dnf install mc.x86_64
\n

Get Apache StreamPipes​

\n

The current release can be downloaded here.

\n

For ease save it in a subfolder on your C-drive, e.g. c:\\Download
\nLets make a folder in WSL to hold our files, and go into it.

\n
mkdir StreamPipes
cd StreamPipes
\n

We can now directly copy the downloaded *.zip file to our WSL, note the version number on the zip file.

\n
cp /mnt/c/Download/apache-streampipes-0.98.0-source-release.zip /home/user/StreamPipes/
\n

If the terminal is still a bit outside you'r comfort zone Midnight Commander can help you.

\n
mc
\n\"Use\n

It is possible to reach your WSL from Windows filemanager, but unzipping files will give issues due to long filenames.\nTherefore the approch of handling the files from WSL is preferable.

\n\"Fail\"\n

Unpack and Navigate to the Compose Directory​

\n

Unzip the package by using the unzip command and navigate to the extracted folder and go to 'installer/compose' directory.

\n
unzip apache-streampipes-0.98.0-source-release.zip

\n
cd apache-streampipes-0.98.0/installer/compose
\n

Update docker-compose.yml​

\n

Modify the docker-compose-yml with nano

\n
nano docker-compose.yml
\n

And add the following.

\n
- ./nginx_config/default.conf:/etc/nginx/conf.d/default.conf:ro
\n

Looking like this.

\n\"Edit\n

Remove this line since it is not relevant for this setup.

\n
environment:
      - SP_HTTP_SERVER_ADAPTER_ENDPOINT=extensions-all-iiot:8090
\n

And yes you can copy paste from windows into nano.\nUse CRTL + x to quit, and answer ‘y’ to save the file.\nHere is how the docker-compose.yml should look loke

\n
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the \"License\"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an \"AS IS\" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# global logging
x-logging:
  &default-logging
  options:
    max-size: '12m'
    max-file: '5'
  driver: json-file

services:

  # StreamPipes services
  backend:
    image: \"${SP_DOCKER_REGISTRY}/backend:${SP_VERSION}\"
    depends_on:
      - couchdb
    environment:
      - SP_PRIORITIZED_PROTOCOL=nats
    volumes:
      - backend:/root/.streampipes
    logging: *default-logging
    restart: unless-stopped
    networks:
      spnet:

  ui:
    image: \"${SP_DOCKER_REGISTRY}/ui:${SP_VERSION}\"
    ports:
      - \"80:8088\"
    depends_on:
      - couchdb
      - backend
    volumes:
      - nginx:/etc/nginx/
      - ./nginx_config/default.conf:/etc/nginx/conf.d/default.conf:ro
    logging: *default-logging
    restart: unless-stopped
    networks:
      spnet:

  couchdb:
    image: couchdb:3.3.1
    environment:
      - COUCHDB_USER=admin
      - COUCHDB_PASSWORD=admin
    volumes:
      - couchdb:/opt/couchdb/data
    logging: *default-logging
    restart: unless-stopped
    networks:
      spnet:

  influxdb:
    image: influxdb:2.6
    environment:
      - INFLUXDB_DATA_ENGINE=tsm1
      - INFLUXDB_REPORTING_DISABLED=false
      - INFLUXDB_ADMIN_ENABLED=true
      - DOCKER_INFLUXDB_INIT_USERNAME=admin
      - DOCKER_INFLUXDB_INIT_PASSWORD=sp-admin
      - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=sp-admin
      - DOCKER_INFLUXDB_INIT_ORG=sp
      - DOCKER_INFLUXDB_INIT_BUCKET=sp
      - DOCKER_INFLUXDB_INIT_MODE=${SP_INFLUX_INIT_MODE}
    volumes:
      - influxdb:/var/lib/influxdb
      - influxdb2:/var/lib/influxdb2
    logging: *default-logging
    restart: unless-stopped
    networks:
      spnet:

  nats:
    image: nats
    restart: unless-stopped
    networks:
      spnet:

  # Lite version pipeline elements
  extensions-all-iiot:
    image: \"${SP_DOCKER_REGISTRY}/extensions-all-iiot:${SP_VERSION}\"
    logging: *default-logging
    restart: unless-stopped
    networks:
      spnet:

volumes:
  backend:
  couchdb:
  influxdb:
  influxdb2:
  nginx:


networks:
  spnet:
    driver: bridge


\n

Add Custom Nginx Configuration​

\n

Create a 'nginx_config' directory

\n
mkdir nginx_config
\n

In that directory you add a 'default.conf' with the help of Nano.

\n
cd nginx_config
nano default.conf
\n

The following nginx.conf is almost identical to the default Docker-based setup, with the only change being that the resolve parameter is removed as it doesn't work reliably with Podman's networking.

\n
server {
  listen       8088 default_server;
  root /usr/share/nginx/html;
 
  # Enable gzip compression
  gzip on;
  gzip_static on;
  gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
  gzip_proxied  any;
  gzip_vary on;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
 
  # StreamPipes Backend Proxy

  location /streampipes-backend {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-NginX-Proxy true;
    proxy_pass http://backend:8030;
    proxy_ssl_session_reuse off;
    proxy_set_header Host $http_host;
    proxy_redirect off;
  }
}
\n

Now we are almost there, and you are getting good at this terminal thing, so is there a use for Podman Desktop, you ask yourself?

\n

Start Up StreamPipes Using Podman Compose​

\n

Do

\n
sudo podman-compose -f docker-compose.yml up -d
\n

to import and run all containers.

\n\"Done\"\n
\n

Once complete, Apache StreamPipes will be available in Podman Desktop and accessible at http://localhost:80.

\n\"Running\"\n

Congratulations, now we can start to use the strongpoints of Podman which enable us in a fast and terminal free way to make backups of our projects or scale them further with Kubernetes.

\n

Happy StreamPiping on Windows with Podman Desktop

", "url": "https://streampipes.apache.org/blog/2026/03/21/_streampipes-with-windows-and-podman/", "title": "StreamPipes on Windows with Podman", "summary": "When it comes to ease of use, Docker Desktop is the goto on windows platform, however for business use there are licensing involved.", "date_modified": "2026-03-21T00:00:00.000Z", "author": { "name": "Morten Torp", "url": "https://www.linkedin.com/in/morten-torp-7b731526/" }, "tags": [ "tutorial" ] }, { "id": "https://streampipes.apache.org/blog/2026/02/23/_streampipes-functions/", "content_html": "

With the introduction of StreamPipes Functions, Apache StreamPipes provides a lightweight way to attach fixed runtime logic to existing streams — without creating a full pipeline element.

\n

Functions are ideal when:

\n\n

Previously, functions could maintain in-memory state during runtime, but any restart of the extensions service resulted in a complete state reset.

\n

We now provide a new functionality in developer preview that introduces typed state persistence for Functions, allowing state to survive service restarts.

\n

This allows functions to survive restarts and resume exactly where they stopped.

\n

Why Stateful Functions?​

\n

Many real-world use cases require minimal but persistent runtime state:

\n\n

Previously, developers had to implement external persistence manually.
\nNow, state management is built directly into the Function lifecycle.

\n
\n

The New State Store API​

\n

The FunctionContext now provides access to a typed state store:

\n
StateStore<T> getStateStore(Class<T> stateClass)
\n

This API allows you to:

\n\n

State handling is fully optional and backward compatible.\nIf you don’t request a state store, your function behaves exactly as before.

\n
\n

Lifecycle Integration​

\n

Functions still follow the same lifecycle:

\n\n

The only difference is that you can now initialize a StateStore during startup.

\n
\n

Example: Threshold Monitoring Function​

\n

Let’s implement a simple function that:

\n\n

1. Define the State Object​

\n

State is defined as a typed POJO.

\n
public class ProcessingState {

    private int schemaVersion = 1;
    private long processedEventCount = 0;
    private long lastProcessedEpochMs = 0;
    private long alertThreshold = 1000;
    
   public ProcessingState() {
   }

    // getters/setters omitted for brevity
}
\n
\n

2. Implement the Stateful Function​

\n
public class ThresholdMonitoringFunction implements StreamPipesFunction {

    private StateStore<ProcessingState> stateStore;
    private ProcessingState state;

    @Override
    public String getFunctionId() {
        return \"threshold-monitoring-function\";
    }

    @Override
    public List<String> requiredStreamIds() {
        return List.of(\"your-stream-resource-id\");
    }

    @Override
    public void onServiceStarted(FunctionContext context) {

        // Request typed state store
        this.stateStore = context.getStateStore(ProcessingState.class);

        // Load existing state or use default
        this.state = stateStore.load(new ProcessingState());
    }

    @Override
    public void onEvent(Event event, String streamId) {

        state.setProcessedEventCount(
            state.getProcessedEventCount() + 1
        );

        state.setLastProcessedEpochMs(
            System.currentTimeMillis()
        );

        if (state.getProcessedEventCount() 
                >= state.getAlertThreshold()) {

            // Handle alert once when threshold is reached
        }
    }

    @Override
    public void onServiceStopped() {

        // Persist state before shutdown
        stateStore.persist(state);
    }
}
\n
\n

State Persistence Lifecycle​

\n

The persistence mechanism follows a simple and predictable pattern:

\n

Startup​

\n
state = stateStore.load(new ProcessingState());
\n\n

Runtime​

\n\n

Shutdown​

\n
stateStore.persist(state);
\n\n

After a restart, the function continues with the restored values.

\n

No external database setup.\nNo manual serialization logic.\nNo additional infrastructure required.

\n
\n

Backward Compatibility​

\n

Stateful behavior is opt-in.

\n

Existing functions:

\n\n
\n

Conclusion & Feedback​

\n

Functions and state persistence are currently available in developer preview, and the API may evolve.

\n

We are open to feedback from developers, if you try out stateful functions, let us know your thoughts and suggestions.

", "url": "https://streampipes.apache.org/blog/2026/02/23/_streampipes-functions/", "title": "Persistent State for StreamPipes Functions", "summary": "With the introduction of StreamPipes Functions, Apache StreamPipes provides a lightweight way to attach fixed runtime logic to existing streams — without creating a full pipeline element.", "date_modified": "2026-02-23T00:00:00.000Z", "author": { "name": "Philipp Zehnder", "url": "https://github.com/tenthe" }, "tags": [ "tutorial" ] }, { "id": "https://streampipes.apache.org/blog/2026/01/20/streampipes-vs-telegraf-influxdb-grafana/", "content_html": "

A lot of teams start with the same three tools: Telegraf collects metrics, InfluxDB stores them, Grafana charts them. We used them ourselves before StreamPipes existed, and they are still a good choice for what they do. This post is about what the stack covers, what you tend to build around it over time, and where StreamPipes fits in, including the setup where StreamPipes writes into the same InfluxDB.

\n

What the stack does well​

\n

Telegraf has input plugins for OPC UA, Modbus, MQTT, SNMP and hundreds of other sources, and output plugins for InfluxDB and many other databases. InfluxDB is a time-series database with retention policies and a query language built for time-based analysis. Grafana is the standard for charting time-series data, with alerting and a large ecosystem of data sources and panels. If the task is \"collect these metrics and chart them\", the three tools do it with little effort, and all of them are open source.

\n

What you build around it​

\n

The stack has no shared model of what a stream is. Field names, units and types are conventions in Telegraf configuration files, repeated per source. There is no event preview while configuring a source, no catalog of streams, and no place where a line or machine is modelled so that its data can be found from the asset. Processing is limited to Telegraf's processor plugins and database-side queries; reacting to a live event, for example detecting a trend and notifying a team, means another component such as Kapacitor, Grafana alert rules or custom code. Users and permissions exist in Grafana, not across the pipeline. Extending the stack means writing Telegraf plugins in Go and deploying new agents.

\n

None of this is a flaw. The stack is a set of focused tools, and teams with enough engineering time assemble them into a platform. The point is that the assembly is yours to do and to maintain.

\n

What StreamPipes adds​

\n

StreamPipes is that platform, already assembled. Adapters connect sources with a guided setup that shows incoming events and lets you set names, units and metadata once. The resulting typed streams drive pipelines that filter, enrich, detect and route in real time, with processors and sinks installed at runtime and validated against the schema. Datasets keep history with retention and export. Charts and dashboards are built on datasets, and assets and sites make them discoverable per machine. Roles, groups, permissions, single sign-on and an audit log cover the whole platform. Python and Java clients and an SDK extend it.

\n

The storage underneath is a time-series database, and one of the included sinks writes to InfluxDB. Grafana can keep charting the data it already has, while StreamPipes handles connectivity, processing, context and history.

\n

Side by side​

\n
TopicTelegraf, InfluxDB, GrafanaApache StreamPipes
Connect a sourceTelegraf input plugin, configured in filesAdapter with guided setup and event preview
Schema, units, metadataConventions per configurationTyped streams, set once, shared by all features
Real-time processingProcessor plugins, external alertingPipelines with processors and sinks, Functions
HistoryInfluxDB retention policiesDatasets with retention, archival and export
VisualizationGrafanaBuilt-in charts and dashboards; Grafana via InfluxDB sink
Asset contextNot modelledAssets and sites
Access control and auditGrafana users and teamsPlatform-wide roles, groups, SSO, audit log
ExtensibilityGo pluginsSDK for adapters, processors and sinks; Java and Python Functions
\n

Choosing​

\n

Keep the stack if what you need is metrics and charts, your team is happy assembling tools, and Grafana is already the house standard. Use StreamPipes when the sources are industrial, when several people need a shared and governed view of the data, and when real-time processing and history should come out of one platform. The two meet at InfluxDB: StreamPipes can feed it, and Grafana can read it. The quickstart takes about ten minutes.

", "url": "https://streampipes.apache.org/blog/2026/01/20/streampipes-vs-telegraf-influxdb-grafana/", "title": "Apache StreamPipes compared with Telegraf, InfluxDB and Grafana", "summary": "Telegraf, InfluxDB and Grafana are a well-worn path for collecting and charting machine data. Where that path ends, and how StreamPipes relates to it.", "date_modified": "2026-01-20T00:00:00.000Z", "author": { "name": "Dominik Riemer", "url": "https://github.com/dominikriemer" }, "tags": [ "comparison" ] }, { "id": "https://streampipes.apache.org/blog/2025/12/15/_release-098/", "content_html": "

Apache StreamPipes 0.98.0 is a major step forward in the evolution of the platform, delivering powerful dashboard\nenhancements, extended OPC UA capabilities, improved security and authentication, and a large set of UX and\narchitectural refinements.

\n

This release focuses on consolidating visualization and dashboard functionality, improving operational transparency, and\nmodernizing the underlying architecture. Highlights include dashboard cloning and kiosk mode, new aggregation\nfunctions, enhanced OPC UA authentication and certificate handling, file logging support, and a modernized\nUI.

\n

In total, this release closes around 100 issues across enhancements, bug fixes, and maintenance tasks.

\n

Read on to discover what’s new in StreamPipes 0.98.0!

\n

Industrial Connectivity & OPC-UA​

\n

Redesign of the adapter creation page​

\n

StreamPipes Connect is the central interface to connect a wide variety of industrial protocols with Apache StreamPipes.\nIn this release, a new overview page simplifies selection of appropriate adapters, taking less space and allowing users\nto get a better overview over available connectors.

\n\"Excel\n

OPC UA X.509 Authentication & Certificate Management​

\n

Support for OPC-UA security is greatly enhanced in 0.98.0.\nWe added a new authentication option that supports X.509 user authentication. Users can now directly upload their\nX.509 certificates which are securely stored within StreamPipes.

\n

For encrypted communication, we also added enterprise features to handle OPC-UA certificates.\nUntil now, certificates had to be managed through environment variables. In the new release, server certificates can be\ncentrally managed from the user interface, simplifying secure deployments.\nIn case a non-trusted certificate is provided by the OPC-UA server, the certificate is stored in the StreamPipes core\nfor manual approval. In the settings menu, admins can explicitly trust or reject a given certificate.\nThe UI view also contains extended certificate information, making it easy to get an overview of all OPC-UA certificates\nthat are currently used.

\n

Version 0.98.0 also supports more OPC-UA data types such as better support of ExtensionObjects, allowing more\ncomplex OPC-UA data structures to be processed.

\n

These improvements strengthen StreamPipes as a secure and enterprise-ready IIoT integration platform.

\n\"OPC-UA\n\"Certificate\n

Other improvements​

\n

Minor improvements for other adapters include:

\n\n

Dashboards & Data Explorer​

\n

Harmonized Data Explorer & Dashboard Experience​

\n

The consolidation of visualization tools is finished in 0.98.0.\nThe Chart and Dashboard views have been further harmonized, ensuring consistent configuration and behavior.\nThe deprecated dashboard has been removed and the new dashboard tool is fully in place. Users can now add any chart\ndirectly to the live dashboard.

\n

This work lays the foundation for faster innovation and a more consistent visualization experience going forward.

\n

Dashboard Cloning & Kiosk Mode​

\n

Dashboards have received further several long-awaited improvements:

\n\n

These enhancements significantly simplify dashboard reuse and deployment in operational environments.

\n

New Visualizations & Chart Improvements​

\n

Several new visualization and charting capabilities were added:\nThe status heatmap visualization was enhanced with a dedicated color-mapping component, in the chart appearance\nsettings, additional chart-specific configurations were added, including improved heatmap configuration with\ncustomizable min/max values.\nTo improve usability, icons and descriptions were added to the chart type dropdown, making chart selection more\nintuitive.

\n

New Aggregation Functions​

\n

The chart view now supports additional aggregation functions:

\n\n

These functions enable more advanced statistical analysis directly within pipelines and the Chart view.

\n

Excel export​

\n

Besides CSV and JSON, datasets can now be directly exported to a Microsoft Excel format (xlsx). In addition, Excel\ntemplates can be uploaded that can be used to add any company-specific header data to an Excel export.

\n\"Excel\n

Security & Authentication​

\n

Since we aim for enterprise-grade security, we added additional advanced configuration options to integrate Apache\nStreamPipes with 3rd-party authentication providers.

\n

OAuth & External Authentication Improvements​

\n

Security and identity management received several upgrades.

\n

External role assignment is now supported in the OAuth configurations. Besides default roles, users can now be\nassigned their roles directly from an external authentication service.\nTo further increase flexibility, StreamPipes-internal roles and groups can be assigned an alternate ID, which can be\nused to map external role names to StreamPipes roles and groups.

\n

OAuth and OpenID configurations are now loaded at startup time, so that invalid settings are detected earlier upon\nstartup.

\n

User Management Enhancements​

\n

User transparency was improved with:

\n\n

UX & UI Improvements​

\n

Modernized UI with Material 3​

\n

The StreamPipes UI has been upgraded to Material Design 3, delivering a cleaner, more modern look and feel.\nAdditional UI improvements include:

\n\n

StreamPipes becomes multilingual​

\n

Release 0.98.0 starts to add internationalization features to Apache StreamPipes.\nWhile this will be a greater effort spanning multiple releases, this new release lays the groundwork for a completely\nlocalized Apache StreamPipes experience.

\n

As a first feature, we've localized most parts of the UI to the german language. More languages will follow in the\nfuture - if you are willing to help us translate StreamPipes to more languages, let us know!

\n\"Localized\n

Actions menu​

\n

To ensure a consistent design and a clean, streamlined look and feel, all data tables across StreamPipes now use a\nunified actions menu. Instead of displaying multiple action buttons inline, available actions are grouped under a\nthree-dot menu. Simply hover over the icon to reveal the actions for a given resource, making tables easier to scan\nwhile keeping all functionality readily accessible.

\n\"New\n

Developer experience​

\n

Migration to the New Data Processor API​

\n

The migration to the new Data Processor API continues, improving consistency and performance across pipeline\nelements.\nAs part of this effort:

\n\n

Developers are encouraged to migrate to the new APIs to ensure long-term compatibility.

\n

Custom theming​

\n

With the upgrade of the design system, we changed the way how the StreamPipes UI can be customized with company-specific\ncolor settings.\nSettings that can be used for custom theming have been isolated in two files (_custom-variables.scss and _theme-colors.scss).\nWe encourage developers to investigate the modified prebuild script and the two scss files to generate individual\ntheming files.

\n

Breaking Change​

\n

Default Messaging System Switched to NATS 💣​

\n

The default docker-compose setup in the installer directory now uses NATS instead of Apache Kafka for internal\nmessaging.

\n

⚠️ Important:
\nThis change only affects users who are updating from a previous installation.
\nIf your existing StreamPipes instance uses Kafka, continue using the provided docker-compose.kafka.yml file.

\n

Deprecations ⚠️​

\n

The following components and features are deprecated and scheduled for removal in future versions:

\n\n

Users are encouraged to migrate early to avoid disruption in upcoming releases.

\n

Other improvements​

\n

As usual, StreamPipes 0.98.0 includes a variety of minor improvements, bug fixes and dependency updates. Users are\nencouraged to update as soon as possible.

\n

Conclusion​

\n

Apache StreamPipes 0.98.0 delivers major improvements in dashboarding, OPC UA integration, security, and user\nexperience, while continuing the architectural cleanup and modernization started in previous releases.

\n

We would like to thank all contributors and community members who made this release possible.
\nAs always, we welcome your feedback—join the discussion on GitHub\nor the mailing list, and help shape the future of StreamPipes!

", "url": "https://streampipes.apache.org/blog/2025/12/15/_release-098/", "title": "Apache StreamPipes release 0.98.0", "summary": "Apache StreamPipes 0.98.0 is a major step forward in the evolution of the platform, delivering powerful dashboard", "date_modified": "2025-12-15T00:00:00.000Z", "author": { "name": "Dominik Riemer", "url": "https://github.com/dominikriemer" }, "tags": [ "release" ] }, { "id": "https://streampipes.apache.org/blog/2025/09/16/streampipes-vs-thingsboard/", "content_html": "

ThingsBoard comes up in almost every evaluation we hear about, and it deserves to. It is a mature open-source IoT platform with dashboards, rule processing and a plugin model, which on paper sounds like StreamPipes. The difference is the question each tool starts from. ThingsBoard asks: which devices do I have, and what are they reporting? StreamPipes asks: what data is coming from my machines, and what should happen with it? Most of the differences below follow from that.

\n

ThingsBoard in short​

\n

ThingsBoard is a device management and telemetry platform. Devices connect over MQTT, CoAP, HTTP or LwM2M, are provisioned and grouped, and send telemetry and attributes. Rule chains react to incoming data with filters, enrichment and actions. Dashboards are built from widgets bound to device telemetry, and a multi-tenant model with customers and users is part of the core. The community edition is open source; some features such as white-labeling, advanced rule nodes and integrations are reserved for the professional edition.

\n

It fits well when you own a fleet of devices that speak IP protocols, when provisioning and device state are what you manage, and when you need dashboards separated by tenant for different customers.

\n

StreamPipes in short​

\n

StreamPipes is an industrial data platform under the Apache Software Foundation. It connects to industrial systems with adapters for OPC UA, Siemens S7, Modbus, PLC4X-based protocols, MQTT, Kafka and more, including an OPC UA node browser and support for OPC UA alarms and events. Every source becomes a typed data stream. Pipelines process streams in real time with validated processors and sinks. Datasets keep history with retention and export. Charts and dashboards are built on datasets, and assets and sites organize everything by plant, line and machine. Users, roles, groups, permissions, single sign-on and an audit log are included, and everything is Apache licensed with no separate edition.

\n

It fits well when the sources are machines and controllers rather than devices you provision yourself, and when the goal is a data foundation that operators, engineers and developers share.

\n

Side by side​

\n
TopicThingsBoardApache StreamPipes
Primary objectDevice with telemetry and attributesTyped data stream from an adapter
Industrial protocolsGateway with OPC UA, Modbus and othersNative adapters incl. OPC UA browser, S7, Modbus, PLC4X family
ProcessingRule chainsPipelines with validated processors and sinks, plus Functions
HistoryTelemetry storage per deviceDatasets with retention, archival and export
Context modelDevices, assets, customers, tenantsAssets and sites linked to streams, datasets, dashboards
Multi-tenancyYes, core conceptNo; one instance per organization
LicensingCommunity edition open source, professional edition commercialFully Apache licensed
ExtensibilityCustom rule nodes, widgets, integrationsSDK for adapters, processors, sinks; Java and Python Functions; UI modules
\n

Choosing​

\n

If you manage many devices and need views separated by customer, ThingsBoard is built for exactly that, and the professional edition adds what larger deployments need. If your data comes from machines and controllers, if you want processing, history and dashboards on typed streams, and if you want all of it under one open-source license with no feature tiers, start with StreamPipes.

\n

The two can also run side by side: a ThingsBoard gateway can forward telemetry to MQTT, and StreamPipes subscribes to it from there. The quickstart is the fastest way to see the StreamPipes side.

", "url": "https://streampipes.apache.org/blog/2025/09/16/streampipes-vs-thingsboard/", "title": "Apache StreamPipes and ThingsBoard: device platform or data platform?", "summary": "ThingsBoard starts from the device, StreamPipes starts from the data. That difference explains most of what each one does well.", "date_modified": "2025-09-16T00:00:00.000Z", "author": { "name": "Dominik Riemer", "url": "https://github.com/dominikriemer" }, "tags": [ "comparison" ] }, { "id": "https://streampipes.apache.org/blog/2025/06/11/oi4-adapter/", "content_html": "

Integrating industrial sensors into IIoT architectures often poses significant challenges—especially for IT teams unfamiliar with traditional operational technologies (OT).\nThe complexity of fieldbus protocols, bespoke integrations, and vendor lock-in can create significant barriers to digital transformation on the shop floor.

\n

This blog post introduces an easy, scalable approach to bridging this IT/OT divide using IO-Link, a standardized protocol for industrial sensors, and Apache StreamPipes.\nWe'll walk you through a complete setup—from connecting IO-Link sensors to visualizing actionable insights—with a focus on a vibration sensor and an IO-Link Master.

\n

Whether you're building condition monitoring systems, predictive maintenance pipelines, or just starting with industrial data analytics, this guide will show how StreamPipes makes it simple to move from raw sensor data to intelligent decisions.

\n\"Image\n\n

IO-Link is an increasingly popular choice in modern industrial environments—and for good reason.\nIt enables seamless sensor integration without the headaches of proprietary protocols or specialized wiring.

\n\n

IO-Link sensors support a wide range of measurements including vibration, temperature, flow, and pressure.\nThanks to IODD (IO Device Description) files, these sensors are interoperable across manufacturers, making configuration fast and reliable.\nWith simple three-wire cabling and support for up to eight sensors via a single IO-Link master, IO-Link dramatically reduces the complexity of sensor installations.

\n

What is IO-Link?​

\n

IO-Link is a globally standardized communication protocol (IEC 61131-9) used to connect industrial sensors and actuators.\nUnlike traditional analog or proprietary fieldbus protocols, IO-Link communicates digitally and supports detailed sensor diagnostics, parameterization, and status monitoring.\nIt also integrates easily into IT environments, helping to close the gap between OT data generation and IT-driven analytics.

\n\n

To simplify integration with modern IoT platforms, the Open Industry 4.0 Alliance (OI4) provides a standardized interface specification for IO-Link over MQTT.\nThis includes well-defined JSON structures, topic hierarchies, and security mechanisms such as TLS encryption and authentication.\nOI4-compliant devices ensure that sensor data can be consumed and interpreted reliably—no matter the vendor.

\n

Use Case: Multi-Sensor Condition Monitoring​

\n

Let’s consider a typical industrial use case: monitoring the condition of electric motors using vibration, temperature, and pressure sensors.\nThese sensors are connected to a Hilscher sensorEDGE FIELD gateway, which publishes all sensor values over MQTT using the OI4 format.

\n

Real-Time Insights with StreamPipes​

\n

In Apache StreamPipes, this data can be ingested using the OI4 adapter and routed through various processing pipelines.\nFor instance, you can configure alerts for high vibration (e.g., aPeak > 8.0 m/s²) and high temperature (> 80°C) to detect early signs of mechanical stress or overheating.\nAdditionally, a rolling average calculation on pressure data helps identify leaks or irregularities.

\n

This setup not only enables instant notifications via email but also stores the processed data in a timeseries database for long-term trend analysis—reducing the need for manual inspections and improving asset reliability.

\n

Expanding Beyond Vibration​

\n

IO-Link sensors can be used across many other domains in industrial environments. Here are a few examples:

\n
Sensor TypeUse CaseStreamPipes Processor(s)
FlowDetect pipe blockages in real-timeThreshold Alert, Rate of Change
DistanceMonitor silo fill levelsMoving Average, Anomaly Detection
PressurePredict pump failure from pressure spikesPattern Matching
\n

Integration Overview: From Hardware to StreamPipes​

\n

Setting up the hardware and data flow is straightforward—even for teams without OT experience.

\n

After connecting the VVB001 vibration sensor to an IO-Link Master, it is automatically detected and begins sending data.\nThis gateway then publishes sensor readings to a local or remote MQTT broker in a structured JSON format, following the OI4 specification.

\n

Apache StreamPipes, with its OI4 adapter, subscribes to these MQTT topics, automatically infers the data schema, and prepares the events for downstream processing—completely eliminating the need for manual parsing or custom integration scripts.

\n\"Architectural\n

StreamPipes Configuration Guide​

\n

Setting up the data stream in StreamPipes is quick and intuitive. Here's how it works:

\n

Adding the OI4 Adapter​

\n

Start by opening the Connect section in the StreamPipes UI. From the list of available adapters, choose the OI4 MQTT Adapter.\nThis adapter is purpose-built for consuming OI4-compliant MQTT streams, so it comes with built-in support for schema detection and mapping.

\n\"OI4\n

Configuring the Adapter​

\n

In the configuration screen, specify the connection to your MQTT broker. For this use case, the broker runs at tcp://192.168.188.73:1883, with unauthenticated access. Select the desired sensor type (VVB001) and confirm that all available sensors should be included.

\n

You’ll also need to enter the App ID that identifies the publishing application—in this case, hilscher.com/.../netfield-app-opc-ua-io-link-adapter-1. Once configured, click Next, and StreamPipes will automatically detect the event schema based on the live payloads from the broker.

\n\"Configuration\n

Deep Dive: VVB001 Vibration Sensor Data​

\n

The VVB001 sensor is an IO-Link-enabled device capable of measuring several key vibration and temperature metrics.\nThese metrics are essential for detecting imbalances, misalignments, and early-stage mechanical wear in rotating equipment.

\n

Key Metrics Explained​

\n\n

Sample JSON Payload​

\n
{
  \"Temperature\": 37.9,
  \"sensorId\": \"000008740649\",
  \"Crest\": 4,
  \"vRms\": 0,
  \"aRms\": 0,
  \"aPeak\": 0.2,
  \"timestamp\": 1600599689805
}
\n

Preprocessing in StreamPipes​

\n

Before this data can be used for analytics or alerting, it must be scaled and cleaned. For example:

\n\n

The StreamPipes connect functionality van handle this effortlessly.\nYou can edit the event schema to apply scaling and remove unused fields.

\n\"Configuration\n

Visual Data Exploration​

\n

Once the data is flowing, you can use the Data Explorer in StreamPipes to monitor sensor metrics in real-time. Line charts help you visualize trends, spikes, and anomalies in vibration and temperature levels.

\n

Whether you’re fine-tuning processing logic or monitoring asset health, StreamPipes visualizations make it easy to detect anomalies and trends in real time.

\n\"Configuration\n

Alerting with Pipelines​

\n

With pipelines, you can define rule-based logic to trigger alerts based on sensor values. For example, if the Crest Factor exceeds a defined threshold, you can send an email notification to maintenance staff.

\n

Pipelines are extensible—you can integrate third party tools, store data in databases, or implement predictive analytics using machine learning models.

\n\"Configuration\n

Conclusion​

\n

Apache StreamPipes and IO-Link together offer a highly accessible yet powerful IIoT solution.\nFrom a simple dashboard setup to complex, multi-sensor condition monitoring pipelines, this integration enables IT teams to drive real-time insights from machine data—without needing a background in industrial automation.

", "url": "https://streampipes.apache.org/blog/2025/06/11/oi4-adapter/", "title": "From Sensor to Insight - Rapid IIoT Integration for IO-Link Devices with Apache StreamPipes", "summary": "Integrating industrial sensors into IIoT architectures often poses significant challenges—especially for IT teams unfamiliar with traditional operational technologies (OT).", "date_modified": "2025-06-11T00:00:00.000Z", "author": { "name": "Philipp Zehnder", "url": "https://github.com/tenthe" }, "tags": [ "tutorial" ] }, { "id": "https://streampipes.apache.org/blog/2025/05/20/opc-ua-certificates/", "content_html": "

With the release of Apache StreamPipes 0.97.0, the OPC UA adapter now supports encrypted communication using\ncertificates, enhancing security for Industrial IoT applications. This guide walks you through generating certificates,\nconfiguring them in StreamPipes, and setting up your OPC UA server for secure communication.

\n

While an unencrypted security policy is the way most users try when they start testing StreamPipes for their first time,\nproduction-grade scenarios typically require encrypted communication.

\n

In case you are already using Apache StreamPipes, you might have noticed that the OPC UA adapter now supports encrypted\nsecurity policies:

\n\"OPC-UA\n

In this guide, I'll show how to set up a secure connection between your OPC UA server and Apache StreamPipes. This guide\nis intended to be a step-by-step-guide starting from certificate generation to the configuration of the OPC UA adapter\nin StreamPipes.

\n

I'll show a working example on a local installation using Eclipse Milo as the OPC UA server. The same steps can be\napplied to production systems by integrating certificates into the volume of the extension service. There are other\nways to configure OPC-UA secure connections, but this guide will focus on a complete approach using self-signed\ncertificates for beginners.

\n

Prerequisites

\n\n

Generate client certificates

\n

When creating a new OPC-UA adapter in Apache StreamPipes, the extension service where the adapter is running in serves\nas the client.\nFirst, we need to create a client certificate for the client. In this example, we'll create a self-signed certificate.

\n
[req]
default_bits       = 2048
prompt             = no
default_md         = sha256
req_extensions     = req_ext
x509_extensions    = v3_ca
distinguished_name = dn

[dn]
CN = stream-pipes-client

[req_ext]
subjectAltName = @alt_names
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment, keyCertSign
extendedKeyUsage = serverAuth, clientAuth

[v3_ca]
subjectKeyIdentifier = hash
basicConstraints = CA:false
subjectAltName = @alt_names
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment, keyCertSign
extendedKeyUsage = serverAuth, clientAuth

[alt_names]
URI.1 = urn:org:apache:streampipes:opcua:client
\n

When using a self-signed certificate, it is requried to to provide the CA:false flag under basicConstraints.\nIn addition, note the subject alternative name that later needs to be matched by the application URI of the OPC-UA\nclient.

\n

The next step is to generate the certificate and import it into a PKCS12 keystore. The keystore is used to store the key\npair we are generating in the first step.

\n
openssl req -x509 -nodes -days 365 \\
  -newkey rsa:2048 \\
  -keyout client.key \\
  -out client.crt \\
  -config cert.conf \\
  && openssl pkcs12 -export \\
  -in client.crt \\
  -inkey client.key \\
  -out client.p12 \\
  -name streampipes-client
\n

When I was asked for a password, I chose streampipes. You can choose any password you like, but make sure to remember\nit, as\nwe will need it in the next step.

\n

Check that the keystore lists the correct key by executing:

\n
keytool -list -v -storetype PKCS12 -keystore client.p12
\n

You'll see the key you've just generated.

\n

Configure StreamPipes to support encrypted connections

\n

To enable the OPC UA adapter to use the generated client certificate, we need to provide a few environment variables to\nStreamPipes.

\n

Here is a list of all environment variables that are available:

\n\n

We could have made the previous configuration easier by reusing some of the defaults. For learning purposes, we used\nsome other names for the alias and keystore filename.

\n

In my setup, I've started a local StreamPipes instance. When using Docker, you can add the environment variables to the\nextensions service (e.g., extensions-all-iiot). In my local instance, I just add the emnvironment variables to my IDE\nsettings.

\n

I add the following variables:

\n
SP_OPCUA_SECURITY_DIR=/home/user/streampipes-security/opcua
SP_OPCUA_KEYSTORE_FILE=client.p12
SP_OPCUA_KEYSTORE_PASSWORD=streampipes
SP_OPCUA_KEYSTORE_ALIAS=streampipes-client
\n

Connecting to an OPC-UA server

\n

For this demo, I'm using the Eclipse Milo OPC UA server. You can download it from here.\nOnce the server is running, you can connect to it using the following URL:

\n
opc.tcp://localhost:62541/milo
\n

Now it's time to switch to StreamPipes!

\n

Go to Connect -> New adapter -> OPC UA and enter the following configuration:

\n
Pull interval 1000
Incomplete Events Ignore
Security Mode Sign & Encrypt
Security Policy Basic256Sha256
URL opc.tcp://localhost:62541/milo
\n

Defocus the URL input field and wait for the OPC-UA server to connect. After a few seconds, you should see an error message appearing:

\n\"OPC-UA\n

This is expected, as we haven't added the client certificate to the trusted certificates of the OPC-UA server yet.\nSo we have to let Milo know that the client certificate can be trusted. In my local setup, I need to move the client certificate from the rejected to the trusted/certs folder as seen below:

\n\"OPC-UA\n

So let's try again by clicking Reload in the Available Nodes section of the adapter configuration page.\nWe'll get another error saying that the server certificate is not trusted.\nThis is again expected, as we haven't added the server certificate to the trusted certificates of the OPC-UA client yet.

\n

To do that, we can have a look at the folder structure under SP_OPC_UA_SECURITY_DIR:

\n\"OPC-UA\n

Several new folders have been created by the Apache StreamPipes OPC-UA client.\nThe trusted folder contains the trusted server certificates, while the rejected folder contains the rejected server certificates.\nWe'll move the server certificate from the rejected folder to the trusted/certs folder.

\n

No we'll try again - and this time, we should see the available nodes in the adapter configuration page:

\n\"OPC-UA\n

You'll see the beautiful tree view where you can navigate through all OPC-UA nodes.

\n

Now you can select the nodes you want to use in your adapter.

\n

That's it! You've successfully configured the OPC-UA adapter to use encrypted communication with self-signed certificates.

", "url": "https://streampipes.apache.org/blog/2025/05/20/opc-ua-certificates/", "title": "Secure OPC-UA Integration with Apache StreamPipes: A Comprehensive Guide", "summary": "With the release of Apache StreamPipes 0.97.0, the OPC UA adapter now supports encrypted communication using", "date_modified": "2025-05-20T00:00:00.000Z", "author": { "name": "Dominik Riemer", "url": "https://github.com/dominikriemer" }, "tags": [ "tutorial" ] }, { "id": "https://streampipes.apache.org/blog/2025/04/30/usage-based-maintanence/", "content_html": "

In many manufacturing settings, maintenance is still scheduled at fixed time intervals — often based on past experience rather than actual machine usage.\nThis approach can lead to unnecessary servicing or, worse, unexpected breakdowns.\nA common limitation is that many machines, especially older ones, offer only minimal sensor data, making advanced maintenance strategies seem out of reach.

\n

However, with Apache StreamPipes, even limited data — such as a simple boolean signal indicating activity — can be leveraged to implement usage-based maintenance.\nIn this blog post, we demonstrate how this is achieved using a real-world example: a packaging machine that processes goods at high throughput.\nYou'll learn how to count machine operations, store event data, visualize usage over time, and trigger notifications when maintenance is due — all with just a few clicks in StreamPipes.

\n

The Scenario: Packaging Machine Monitoring​

\n

Let’s consider a packaging machine that wraps goods into individual packages.\nThe wear and tear on this machine is directly related to the number of items processed.\nHowever, due to the machine’s age, only minimal data is available.

\n

Currently:

\n\n

With Apache StreamPipes, we’ll build a pipeline that:

\n\n

The Data: A Simple Boolean Signal​

\n

The machine provides a boolean flag — active — which is true when a packaging operation occurs.\nThis flag is streamed using an Apache StreamPipes S7 adapter. The stream also includes a few additional process parameters:

\n
Timestampactive...
1719897720887false...
1719897721952true...
1719897722952true...
.........
\n

Part 1: Absolute Counter with Notification​

\n

We’ll begin by building a pipeline that counts how often the machine packages a product and sends an email alert when a predefined threshold is reached.

\n

Pipeline Overview​

\n\"Counting\n

A simple StreamPipes pipeline for monitoring the parts produced at the machine

\n

Step-by-Step Guide​

\n
    \n
  1. \n

    Add the Data Source

    \n\n
    2. \n
  2. \n

    Insert the Boolean Counter Processor

    \n\n
    \n

    The processor emits a new event each time the machine packages an item, incrementing a counter value.

    \n
    \n
    3. \n
  3. \n

    Persist the Count

    \n\n
    4. \n
  4. \n

    Add a Threshold Alert

    \n\n
    5. \n
  5. \n

    Visualize in Dashboard

    \n\n
    6. \n
\n\"Displaying\n

The live dashboard with an indicator chart showing the current number of goods produced

\n

This setup gives you a live counter of how many products were packaged and notifies the maintenance team when attention is needed.

\n

Part 2: Aggregated Counting Over Time​

\n

While a cumulative counter is useful, often it's more insightful to understand how usage varies over time — for example, how many items are processed each day.

\n

A Time-Aggregated Approach​

\n

Instead of a total counter, we’ll now generate a single event for each packaging operation and then use the Data Explorer to visualize activity per hour or per day.

\n

Alternative Pipeline Using Signal Edge Filter​

\n\"Count\n

Pipeline to recognize when a part is being processed

\n

Step-by-Step:

\n
    \n
  1. \n

    Replace the Boolean Counter with a Signal Edge Filter processor.

    \n\n
    2. \n
  2. \n

    Store the filtered stream in the Data Lake sink.

    \n
    3. \n
\n

This gives one clean event per packaging operation — ideal for aggregation.

\n

Visualization in the Data Explorer​

\n

Now that we have one event per packaging action, we can analyze the data over time in the Data Explorer.

\n
    \n
  1. Open the Data Explorer
    2. \n
  2. Table View Configuration:\n\n
    3. \n
\n\"Count\n
    \n
  1. Bar Chart Visualization:\n\n
    2. \n
\n\"Count\n

You now have a clear, time-based view of machine usage. This helps spot trends, detect unusual downtimes, and better plan maintenance.

\n

Conclusion​

\n

In this post, we’ve shown how even limited sensor data — like a simple boolean signal — can be used to build powerful maintenance logic using Apache StreamPipes. With just a few clicks, you can:

\n\n

By replacing static maintenance intervals with actual usage insights, you can reduce unnecessary work and improve efficiency — all without needing to retrofit expensive new sensors.

", "url": "https://streampipes.apache.org/blog/2025/04/30/usage-based-maintanence/", "title": "Usage-Based Maintenance with Apache StreamPipes", "summary": "In many manufacturing settings, maintenance is still scheduled at fixed time intervals — often based on past experience rather than actual machine usage.", "date_modified": "2025-04-30T00:00:00.000Z", "author": { "name": "Philipp Zehnder", "url": "https://github.com/tenthe" }, "tags": [ "tutorial" ] }, { "id": "https://streampipes.apache.org/blog/2025/04/28/prompt-processor/", "content_html": "\"Prompt\n

Need a quick sentiment check, a rolling summary, or a safety rule that’s just a bit too fuzzy for SQL?
\nWith the Multi-Model Prompt Processor you can drop the full might of OpenAI, Anthropic, or your own Llama into any\nApache StreamPipes pipeline—no extra code required.

\n

Below you’ll find:

\n\n
\n

What does the processor do?​

\n
OptionWhat it controls
Model provider / nameChoose any OpenAI or Anthropic model, or point to a Llama on Ollama.
Temperature0 = repeatable answers, 1 = creative answers.
History strategyStateless, Windowed (N turns), or Full history.
Window sizeHow many past turns to keep in Windowed mode.
Multiple input fieldsCombine several event attributes into one user message.
\n

The processor sends your prompt + current event data to the model, grabs the reply, and writes it to a new field\nllmResponse; ideal when the answer must flow straight into the next processor.

\n
\n

Five ways to use it​

\n
#ScenarioPrompt idea (few-shot style)History
1Data-quality gate – flag numbers > 15Return WARNING if value > 15 else OK. Example → 12 → OK, 18 → WARNING.Stateless
2Rolling minimum of a sensorPass previousMin + current. “Return the smallest of the two numbers only.”Windowed (1)
3Five-tweet sentimentConcatenate last 5 tweets. “Return POSITIVE / NEGATIVE / NEUTRAL only.”Windowed (5)
4Chat summariser for support ticketsSystem prompt: “Write one-line summary of the whole conversation.”Full
5Dynamic threshold helperFeed last 10 load values. “Return max(values)*1.1 rounded 0 decimals.”Windowed (10)
\n

(Replace the $placeholders$ with the actual stream attributes in the mapping dialog.)

\n
\n

Tutorial – from stream to JSON warning​

\n

Below is a short clip showing the processor in action in a temperature pipeline.
\nThe prompt forces the model to output a two-field JSON—perfect for machine parsing.

\n\"Animated\n

Model: claude-3-5-sonnet-20241022
\nPrompt Used:

\n
You are an assistant that ALWAYS replies with a single JSON object
containing two string fields: \"status\" and \"message\".

Rules
1. If the temperature is strictly greater than 45 °C:
     • \"status\" must be \"WARNING\"
     • \"message\" must explain that the temperature is above the safe limit
2. Otherwise:
     • \"status\" must be \"OK\"
     • \"message\" must note that the temperature is within the safe range
3. Do not output anything except the JSON object.

### Examples

Temperature: 50
{
  \"status\": \"WARNING\",
  \"message\": \"Temperature exceeds the 45 °C threshold\"
}

Temperature: 30
{
  \"status\": \"OK\",
  \"message\": \"Temperature is within the safe range\"
}

Temperature: 45
{
  \"status\": \"OK\",
  \"message\": \"Temperature is within the safe range\"
}

Temperature: 60
{
  \"status\": \"WARNING\",
  \"message\": \"Temperature exceeds the 45 °C threshold\"
}
\n

The model now returns either:

\n
{
  \"status\": \"WARNING\",
  \"message\": \"Temperature exceeds the 45 °C threshold\"
}
\n

or

\n
{
  \"status\": \"OK\",
  \"message\": \"Temperature is within the safe range\"
}
\n

Nothing else—exactly what we want for downstream processors.

\n
\n

Prompt writing tips for event streams​

\n
    \n
  1. State the output format first: “Reply with OK or WARNING only.”
    2. \n
  2. Give one or two concrete examples – LLMs copy patterns.
    3. \n
  3. End with a clear cue: “Answer:” or “Output:”.
    4. \n
  4. Keep temperature ≤ 0.3 when strict structure matters.
    5. \n
  5. Short replies reduce context size in Windowed / Full modes.
    6. \n
\n
\n

Supported models​

\n
Anthropic models
claude-3-7-sonnet-20250219
claude-3-5-sonnet-20241022
claude-3-5-haiku-20241022
claude-3-5-sonnet-20240620
claude-3-opus-20240229
claude-3-sonnet-20240229
claude-3-haiku-20240307
claude-2.1
claude-2.0
\n
OpenAI models
gpt-3.5-turbo              gpt-3.5-turbo-1106     gpt-3.5-turbo-0125
gpt-3.5-turbo-16k          gpt-4                  gpt-4-0613
gpt-4-turbo-preview        gpt-4-1106-preview     gpt-4-0125-preview
gpt-4-turbo                gpt-4-turbo-2024-04-09 gpt-4-32k
gpt-4-32k-0613             gpt-4o                 gpt-4o-2024-05-13
gpt-4o-2024-08-06          gpt-4o-2024-11-20      gpt-4o-mini
gpt-4o-mini-2024-07-18     o1                     o1-mini
o1-2024-12-17              o1-mini-2024-09-12     o1-preview
o1-preview-2024-09-12      o3-mini-2025-01-31
\n
Llama suggestions (Ollama)

llama3, llama3:8b, llama2, mistral-7b-instruct, or any custom model you have pulled.

\n
\n

Final thoughts​

\n

Whether you need a lightweight rule engine, live summaries, or just a smarter filter, the Multi-Model Prompt Processor\ndrops straight into your pipeline and starts working within minutes.

\n

Give it a try, and let us know what you build!

", "url": "https://streampipes.apache.org/blog/2025/04/28/prompt-processor/", "title": "Bringing LLM Power to Every Pipeline – The Multi-Model Prompt Processor", "summary": "Need a quick sentiment check, a rolling summary, or a safety rule that’s just a bit too fuzzy for SQL?", "date_modified": "2025-04-28T00:00:00.000Z", "author": { "name": "Grainier Perera", "url": "https://github.com/grainier" }, "tags": [ "tutorial" ] }, { "id": "https://streampipes.apache.org/blog/2025/04/09/javascript-processor/", "content_html": "

Have a use case where none of the existing processors fit?
\nDon’t worry—we’ve got just the thing for you.

\n

The JavaScript Evaluator Processor in Apache StreamPipes allows you to define custom event logic with just a few lines of JavaScript. Whether you want to modify values, filter events, or even add stateful logic, this processor empowers you to do just that—without writing a full processing element in Java.

\n

Let’s take a closer look at how it works, what the configuration options are, and how you can use it in three different ways.

\n

What is the JavaScript Evaluator?​

\n

Sometimes you need to apply custom logic to your data stream. Maybe it's a quick transformation or a complex filter that isn’t supported out of the box. The JavaScript Evaluator Processor is designed exactly for these cases.

\n

With a simple JavaScript function, you can:

\n\n

All you need is a little coding and a bit of creativity.

\n

Configuring the JavaScript Processor​

\n

When adding the JavaScript Evaluator to your pipeline, you’ll notice two key configuration sections:

\n\"Javascript\n
    \n
  1. The JavaScript Function
    \nThis function is triggered once for each incoming event.
    2. \n
\n\"Javascript\n
    \n
  1. The Output Schema
    \nThis defines the structure of your transformed events. It’s important to ensure that the output schema matches what your function returns—otherwise, downstream processors might not be able to validate the events correctly.
    2. \n
\n\"Javascript\n

Let’s explore some examples that show different ways to apply custom logic to your event streams using JavaScript.

\n

Example 1: Stateless Event Transformation​

\n

The simplest use case is when you want to modify values in an event.

\n

Use case: You want to halve the temperature value of each incoming event.

\n
function process(event) {
    event.temperature = event.temperature / 2;
    return event;
}
\n

That’s it! Each event passes through this function, gets modified, and is sent on its way. No extra setup needed.

\n

Example 2: Custom Event Filtering​

\n

You don’t always want to forward every event. The JavaScript Evaluator lets you return null (or just omit return) to drop an event.

\n

Use case: Only forward events where the temperature is between 50 and 100.

\n
function process(event) {
    if (event.temperature > 50 && event.temperature < 100) {
        return event;
    }
}
\n

With this setup, only \"valid\" events get through. All others are silently ignored. Perfect for quick, custom filtering logic.

\n

Example 3: Stateful Logic​

\n

Need to track information across multiple events? You can wrap your function in an Immediately Invoked Function Expression (IIFE) to create persistent variables.

\n

Use case: Count how many events have passed through the processor and append that number to each event.

\n
(() => {
    let count = 0;

    return function process(event) {
        count += 1;
        event.count = count;
        return event;
    };
})()
\n

This small trick makes it easy to introduce state without needing any external dependencies or full-blown processor development. You can track counts, store intermediate values, or even implement basic alerting.\nJust make sure that any new properties you add to the event, like count in the example, are also included in the output event schema during processor configuration, otherwise the pipeline may fail validation.

\n

Wrap-Up​

\n

The JavaScript Evaluator is one of the most flexible processors in StreamPipes. Whether you're a beginner or a power user, it lets you:

\n

✅ Write custom transformations
\n✅ Implement simple filters
\n✅ Track state between events
\n✅ Prototype complex logic—without deploying a new microservice

\n

So, next time you think \"I wish there was a processor that could...\", just try the JavaScript Evaluator. Chances are, it can.

\n

Got a cool use case using this processor? Let us know or share it with the community—we’d love to see what you’re building with Apache StreamPipes!

", "url": "https://streampipes.apache.org/blog/2025/04/09/javascript-processor/", "title": "How to Use the JavaScript Evaluator Processor", "summary": "Have a use case where none of the existing processors fit?", "date_modified": "2025-04-09T00:00:00.000Z", "author": { "name": "Philipp Zehnder", "url": "https://github.com/tenthe" }, "tags": [ "tutorial" ] }, { "id": "https://streampipes.apache.org/blog/2025/03/25/file-stream-adapter/", "content_html": "

Introduction​

\n

Are you excited to explore Apache StreamPipes but find yourself without any sensors to play with?\nOr maybe you're looking to validate StreamPipes for your use case and kick off a Proof of Concept (PoC) but the hardware is not yet available?\nNo worries—the File Stream Adapter is here to save the day!

\n

With the File Stream Adapter, you can upload a file containing sensor data in various formats (JSON, CSV, XML) and replay this data, giving you a hands-on way to dive into StreamPipes.\nIt's not meant for production use, but it's perfect for experimenting with StreamPipes using your own data.

\n

This blog post will take you on a exciting journey through configuring the File Stream Adapter and using it to simulate sensor data streams.

\n
info

The adapter supports any type of data as long as it contains a valid timestamp in the form of a Unix timestamp in milliseconds.

\n

Adapter Configuration​

\n

Let's break down the configuration options for the File Stream Adapter:

\n\"Adapter\n

Configuration Options Explained​

\n
    \n
  1. File: Choose an existing file or upload a new one.
    2. \n
  2. Overwrite file time: If checked, the current time at which the event is triggered is used for the timestamp of the event, regardless of the timestamp value in the file.
    3. \n
  3. Replay Once: Decide if the file should be replayed once or continuously until stopped.\n\n
    4. \n
  4. Replay Speed: Set the frequency of events.\n\n
    5. \n
  5. Format: Choose the format to parse the events.\n\n
    6. \n
\n

Use Case: Create a Sensor Simulator​

\n

Let's create a simulation data stream using your own sensor data. First, store the sample CSV data in a local file.

\n
timestamp,density,mass_flow,sensorId,sensor_fault_flags,temperature,volume_flow
1719897720887,41.82,6.306,flowrate01,false,41.9,3.365
1719897721952,45.09,9.659,flowrate01,false,43.2,4.103
1719897722958,42.84,9.532,flowrate01,true,42.6,3.110
1719897723964,49.58,1.929,flowrate01,true,40.3,3.4684
1719897724970,41.83,4.173,flowrate01,false,40.7,0.246
1719897725975,48.11,10.90,flowrate01,false,46.4,5.275
\n

To set up the adapter, follow these steps:

\n
    \n
  1. Open StreamPipes Connect.
    2. \n
  2. Click on + New Adapter.
    3. \n
  3. Select the File Stream Adapter.
    4. \n
  4. Enter the following configuration:\n\n
    5. \n
  5. Click Next once you've added the correct configuration.
    6. \n
  6. Configure the event schema by editing the timestamp field and selecting Mark as timestamp. Save your changes.
    7. \n
  7. You can add additional meta information to the event schema, but it's not required for this tutorial. Click Next when done.
    8. \n
  8. Enter an Adapter Name.
    9. \n
  9. Check the Persist events box to store the data in the StreamPipes data store.
    10. \n
  10. Click Start Adapter.
    11. \n
\n

Once configured, you should see the following preview:

\n\"Adapter\n

Use Case: Persist CSV File to Data Lake​

\n

Now, let's store the entire content of the file in the data lake. Once stored, you can analyze it with the Data Explorer. Here’s how to configure the adapter:

\n\n
info

Ensure the Persist events checkbox is selected to save the data to the data lake during the Start Adapter configuration.

\n

Conclusion​

\n

In this tutorial, we've explored how to use the File Stream Adapter in Apache StreamPipes to create a simulation data stream and persist CSV data into a data lake.\nThis adapter lets you experiment with StreamPipes' capabilities and explore its functionality using your own data.\nHappy streaming!

", "url": "https://streampipes.apache.org/blog/2025/03/25/file-stream-adapter/", "title": "Explore StreamPipes with Data Simulators!", "summary": "Introduction", "date_modified": "2025-03-25T00:00:00.000Z", "author": { "name": "Philipp Zehnder", "url": "https://github.com/tenthe" }, "tags": [ "tutorial" ] }, { "id": "https://streampipes.apache.org/blog/2025/02/15/release-097/", "content_html": "

Introduction​

\n

Apache StreamPipes 0.97.0 brings exciting new features, enhanced security, and significant UX improvements to streamline your IIoT data processing workflows.\nThis release introduces Adapters & Pipelines as Code, enabling users to define and deploy StreamPipes resources programmatically using YAML or JSON.\nSecurity enhancements, such as OPC UA certificate support and Kafka SSL/SASL authentication, improve connectivity and data integrity.\nAdditionally, we have made various UX improvements, including an enhanced Data Explorer and asset-based filtering.

\n

In total, the new release closes 159 issues including new features, bug fixes and improvements.

\n

Read on to discover what’s new in StreamPipes 0.97.0!

\n

Adapters & Pipeline Elements​

\n

Adapter & Pipeline as Code​

\n

StreamPipes now supports managing adapters and pipelines programmatically using compact Json or YAML definitions.\nThese features are ideal for automating deployments, integrating with CI/CD pipelines, and applying infrastructure-as-code practices.

\n

Adapters as Code​

\n

With Adapters as Code, you can define and deploy adapters via a compact description, streamlining the creation of multiple similar adapters and simplifying automation.\nFor example, here's a sample YAML snippet to define an OPC-UA adapter:

\n
name: My OPC Adapter
id: testadapter
description: Test
appId: org.apache.streampipes.connect.iiot.adapters.opcua
configuration:
  - opc_host_or_url: OPC_URL
    opc_server_url: opc.tcp://localhost:62541/milo
  - adapter_type: PULL_MODE
    pulling_interval: 1000
  - access_mode: UNAUTHENTICATED
  - available_nodes:
    - \"ns=2;s=Leakage Test Station/temperature\"
    - \"ns=2;s=Leakage Test Station/pressure\"

schema:
  temperature:
    propertyScope: measurement
    label: Temp
    description: Temperature value
    semanticType: http://schema.org/temperature

enrich:
  timestamp: timestamp

transform:
  rename:
    pressure: pressure3

createOptions:
  persist: true
  start: true
\n

This approach considerably simplifies the programmatic creation of adapters Learn more about Adapters as Code

\n

Pipelines as Code​

\n

Pipelines as Code lets you define end-to-end data flows—from data sources through processors to sinks.\nThis function is ideal for the standardization and automation of pipeline deployments.\nFor example, here’s a sample YAML snippet for a simple pipeline:

\n
id: my-pipeline
name: \"\"
description: \"\"
pipelineElements:  # Define pipeline components here
  - type: stream     # Data source
    ref: <reference> # Unique reference ID
    id: <data-stream-id> # ID of the stream

  - type: processor  # Data transformation
    ref: <reference> # Unique reference ID
    id: <processor-id> # ID of the processor
    connectedTo:     # Previous pipeline element reference(s)
      - <reference>
    configuration:   # Processor-specific configurations
      - <configuration-option>

  - type: sink       # Data sink (output)
    ref: <reference> # Unique reference ID
    id: <sink-id>    # ID of the sink
    connectedTo:     # Previous pipeline element reference(s)
      - <reference>
    configuration:   # Sink-specific configurations
      - <configuration-option>

createOptions:
  start: <true|false>  # Whether to start the pipeline immediately
\n

The code preview of a pipeline has also been added to the Start Pipeline dialog, which has been further improved to make the execution status of pipelines more clear.

\n\"Pipeline\n

This method provides a programmatic interface to deploy and manage pipelines quickly and easily.\nLearn more about Pipelines as Code

\n

OPC UA Certificates​

\n

The OPC UA adapter now supports both signed/encrypted and unencrypted communication.\nCertificate provisioning is handled externally via environment variables, following a dedicated security directory layout. Refer to the OPC UA adapter documentation for further details.

\n

https://github.com/apache/streampipes/pull/3354

\n\"OPC\n

Kafka Avro & Certificates​

\n

The Kafka adapter and sink now support SSL and SASL for enhanced security.\nAdditionally, users can provide custom producer/consumer configurations.\nA new Avro parser has been added along with an input field for specifying the consumer group in the Kafka adapter.

\n\"Kafka\n

New Features & Deprecations​

\n

Asset Browser​

\n

StreamPipes now allows users to model real-world assets and associate them with StreamPipes resources such as adapters, pipelines, and dashboards.\nThese asset mappings enable intuitive filtering across different modules, making it easier to navigate and manage related resources.

\n

The screenshot below illustrates an example where a StreamPipes instance is configured with two machines and various stations.\nClicking on a machine or station dynamically filters the associated resources in the list on the right.\nThis asset-based filtering is available across all main modules of StreamPipes.

\n\"Asset\n

The asset overview panel has been redesigned to provide a more intuitive and user-friendly experience. It is possible to assign labels, locations, and ISA95 types to assets.\nFor each asset, resources such as pipelines, adapters and dashboards can be assigned. The asset overview contains an intuitive overview of all linked resources.

\n\"Asset\n

To manage locations & sites, we extended the configuration menu with additional options to add a custom map layer provider and to configure the map view.

\n\"Asset\n

UX improvements​

\n

File Relocation​

\n

The file upload configuration has been moved from the top-level navigation to the settings menu, improving the overall structure and clarity of the main navigation.

\n

This change does not affect file uploads required by adapters or pipeline elements—these configurations remain unchanged.\nFurther, users can still upload files as before via the files configuration section.

\n\"File\n

Adapter Data Preview​

\n

Viewing live data streams in the adapter configuration has been improved. By clicking on \"details\", a full-screen view of the data preview is displayed.\nChanging data values are highlighted, so that it is ease to track changes. The Code panel shows the configuration in JSON or YAML format.

\n\"Adapter\n

Data Explorer Configuration Improvements​

\n

Configuring data views in the Data Explorer is now more intuitive and user-friendly.\nWe have enhanced validation, making it easier to select data sources and specify queries while reducing potential misconfigurations.

\n

Key improvements include:

\n\n\"Improve\n

Here is a screenshot of the new traffic light visualization:

\n\"Traffic\n

Pipeline Details​

\n

The pipeline details view has been completely remodeled. It now shows the pipeline in full-screen mode, making it easier to view and edit complex pipelines.\nA sidebar contains further information on the pipeline, such as logs, pipeline elements, and execution status.\nIn addition, execution metrics are now intuitively accessible right in the pipeline details view.

\n\"Pipeline\n

Deprecation Notice: Dashboard​

\n

The current live dashboard feature in StreamPipes is being deprecated and will be removed in a future release.\nAs a replacement, the Data Explorer dashboards has been enhanced with auto-refresh functionality, ensuring a seamless transition for users who rely on real-time data visualization.

\n

The harmonization of data explorer and live dashboard functionalities will provide a more consistent user experience and will ease the addition of new charts in future version, since all visualizations now rely on a single visualization library.

\n

We encourage all users to migrate their existing dashboards to the Data Explorer to continue leveraging powerful visualization and analysis capabilities.

\n

If you think any feature is missing in the data explorer, please let us know!

\n\"Dashboard\n

Conclusion​

\n

We appreciate the ongoing support and contributions from our community in making StreamPipes better with each release.\nVersion 0.97.0 brings improvements in usability, security, and automation, helping users to analyze their IIoT data sources.

\n

As always, we welcome your feedback and suggestions to continue refining and enhancing StreamPipes.\nFeel free to share your thoughts or get involved through GitHub or mailing list.

", "url": "https://streampipes.apache.org/blog/2025/02/15/release-097/", "title": "Apache StreamPipes release 0.97.0", "summary": "Introduction", "date_modified": "2025-02-15T00:00:00.000Z", "author": { "name": "Philipp Zehnder", "url": "https://github.com/tenthe" }, "tags": [ "release" ] }, { "id": "https://streampipes.apache.org/blog/2024/06/13/release-095/", "content_html": "

We're excited to share the release of StreamPipes 0.95.0, bringing with it a range of bug fixes and new features.\nIn total, the newest release closes over 200 issues.\nThis release doesn't include many large feature drops, we have worked a lot on making small things smoother and improving StreamPipes under the hood.\nAs we continue to work on the 1.0 release of Apache StreamPipes probably later this year, this release features many improvements to the end-user experience and bug fixes.

\n

Let's delve into the key updates introduced in this release.

\n

The current release can be downloaded here.

\n

Adapters & Pipeline Elements​

\n

Improve deletion workflow of adapters​

\n

Within StreamPipes 0.93.0 we introduced the possibility to edit existing adapters.\nThis release improves adapter management by conveniently allowing to delete existing adapters even if they are used in a pipeline.\nWhen deleting a new adapter, a dialog appears which allows users to delete all pipelines which are using the data stream produced by the adapter.

\n

https://github.com/apache/streampipes/issues/1936

\n

Enhance data lake schema management​

\n

We've added a new option to the data lake sink which provides users more flexibility when writing data to the internal time-series storage.\nUsers can choose between the Update Schema and Extend existing schema options.\nBy choosing Extend schema, it is possible to also see \"old\" data in the data explorer after the schema has changed.

\n\n

https://github.com/apache/streampipes/issues/2252

\n

New Adapter: Open Industry 4.0 devices​

\n

A new adapter in 0.95.0 is support for the Open Industry 4.0 specification over MQTT.\nThe adapter can be used to connect to a variety of devices which support the data format defined in the Development Guidelines for Open Edge Computing.

\n

For instance, the adapter makes it easy to use IO-Link sensors together with an OI4-compatible IO-Link master.

\n

https://github.com/apache/streampipes/pull/2511

\n

New Adapter(s): All PLC4X protocols supported as adapters​

\n

Since our \"sister project\" Apache PLC4X has introduced a new metadata API for gathering available drivers, StreamPipes 0.95.0 now supports a much larger collection of PLC connectors.\nThe list of newly supported protocols include:

\n\n

The already existing S7 adapter is now also available with additional configuration options.

\n\n

Configurations on protocol and transport level are dynamically resolved from the PLC4X API. This ensures that users now have much greater flexibility when connecting to industrial protocols.

\n

The screenshot below shows the advanced options panel for Modbus TCP:

\n\n

The new adapters should be considered a beta feature and we still recommend to use the existing Modbus and S7 adapters until the new adapters have proven their stability.\nMore information on the usage of these new adapters and supported features can also be found on the PLC4X website.

\n

https://github.com/apache/streampipes/issues/2632

\n

New Processor: Static Metadata enrichment​

\n

We added a new processor to add static metadata (such as sensor ID or location IDs) to any data stream within a pipeline.

\n

https://github.com/apache/streampipes/issues/2350

\n

New Processor: Datetime from String​

\n

The new Datetime from String processor allows to convert a datetime string to a timestamp value and lets users conveniently choose the desired target time zone.

\n\n

https://github.com/apache/streampipes/issues/1865

\n

New Sink: MS Teams​

\n

StreamPipes 0.95.0 comes with a new data sink for Microsoft Teams, which allows to publish notifications from a pipeline to a teams channel using Webhooks.

\n\n

https://github.com/apache/streampipes/pull/2248

\n

Remove deprecated adapters​

\n

We removed several outdated adapters that were deprecated in release 0.93.0:

\n\n

https://github.com/apache/streampipes/issues/2128

\n

UX improvements​

\n

Display file types in different colors​

\n

Different file types in the Files overview are now better highlighted by a dedicated color.

\n

https://github.com/apache/streampipes/issues/2107

\n

Download button for asset & files​

\n

In the asset and file overview, resources can be now directly downloaded by using a new download button.

\n

https://github.com/apache/streampipes/issues/2192\nhttps://github.com/apache/streampipes/issues/2074

\n

StreamPipes Python​

\n

As already mentioned in our blog post, we added an example to use an existing ONNX model as part of a StreamPipes function.

\n

https://github.com/apache/streampipes/pull/2687

\n

Administration & Development​

\n

Remove Consul entirely​

\n

Following the replacement of Consul with an internal service registration and discovery mechanism, StreamPipes 0.95.0 also removes Consul from all installation files.

\n

https://github.com/apache/streampipes/issues/2133

\n

New Apache StreamPipes Go-Client​

\n

Apache StreamPipes now includes a new client library for Golang.\nOur first version of the Go client interacts with the StreamPipes API and provides various ways to gather data from StreamPipes.

\n

Here is an example on the usage of the Go client:

\n


package main

import (
\t\"github.com/apache/streampipes/streampipes-client-go/streampipes\"
\t\"github.com/apache/streampipes/streampipes-client-go/streampipes/config\"
\t\"log\"
)

/*
\tHere are some examples of using go client, including outputting the data returned by streams.
\tOnly supports outputting model data
*/

func main() {

\tclientConfig := config.StreamPipesClientConfig{
\t\tUrl: \"http://localhost:8030\",
\t\tCredential: config.StreamPipesApiKeyCredentials{
\t\t\tUserName: \"<Your-User-Name>\",
\t\t\tApiKey:   \"<Your-API-Key>\",
\t\t},
\t}

\tstreamPipesClient, err := streampipes.NewStreamPipesClient(clientConfig)
\tif err != nil {
\t\tlog.Fatal(err)
\t}

\tdataSeries, err := streamPipesClient.DataLakeMeasures().GetSingleDataSeries(\"measureName\")
\tif err != nil {
\t\tlog.Fatal(err)
\t}
\tdataSeries.Print()

\t/*
\t\t\toutput format:

\t\t\tThere are 2 pieces of DataSerie in the Dataseries
\t\t\tThe 1 DataSeries
\t\t\ttime                   msg                   test
\t\t\t2024-02-23T13:37:09.052Z   go-client_test   2f4556
\t\t\t2024-02-23T13:37:26.044Z   go-client_test   2f4556
\t\t\t2024-02-23T13:37:29.007Z   go-client_test   2f4556
\t\t\tThe 2 DataSeries
\t\t    time                   msg                   test
\t\t\t2024-02-23T13:38:06.052Z   go-client_test   2f4556
\t\t\t2024-02-23T13:38:35.044Z   go-client_test   2f4556
\t\t\t2024-02-23T13:38:38.007Z   go-client_test   2f4556

\t*/

}
\n

https://github.com/apache/streampipes/pull/2437

\n

First release, are working on more examples and better docs - stay tuned

\n

Final Remarks​

\n

We highly recommend updating to StreamPipes 0.95.0 to take advantage of these new features, bug fixes, and improvements.\nFor more detailed information, please refer to\nthe release notes.

\n

We appreciate your continued support and valuable feedback.\nStreamPipes is continually evolving, and we are dedicated to providing you with a powerful and reliable platform for\nyour IIoT streaming data needs.

", "url": "https://streampipes.apache.org/blog/2024/06/13/release-095/", "title": "Apache StreamPipes release 0.95.0", "summary": "We're excited to share the release of StreamPipes 0.95.0, bringing with it a range of bug fixes and new features.", "date_modified": "2024-06-13T00:00:00.000Z", "author": { "name": "Dominik Riemer", "url": "https://github.com/dominikriemer" }, "tags": [ "release" ] }, { "id": "https://streampipes.apache.org/blog/2024/03/27/anomaly-detection-with-python-functions/", "content_html": "

Apache StreamPipes saves the day when it comes to connecting to data sources in the IIoT world. Want to do more with\nyour IIoT data than just analyze it in a dashboard? If so, this blog post is for you! We'll show you how to extract\nhistorical data from StreamPipes, use it to train a machine learning model, bring the model back to StreamPipes using\nONNX, and apply the model to live data.

\n
\"anomaly-detection\"
\n

Motivation​

\n

With this blogpost we want to illustrate how one can easily extract historical IIoT data collected with StreamPipes,\ntrain a machine learning model on this data and bringing the model back to StreamPipes with the interoperability\nstandard ONNX to make inference in live data.

\n

A very common use case in the area of IIoT is the detection of anomalies, so we want to tackle this challenge in this\narticle as well. We will use data generated by\nthe Machine Data Simulator\nadapter. More specifically, we will focus on the flowrate data, which consists of various sensor values coming from a\nwater pipe system. Our goal is to keep an eye on the parameter volume_flow, which represents the current volume flow\nin\ncubic meters/second. For this parameter, we want to detect anomalies that could indicate problems such as leaks,\nblockages, etc.

\n

To get the concerned data, we simply need to create an instance of the machine data simulator and persist the data in\nthe data lake:

\n

\"tutorial-preparation\"

\n

Set Up & Prepare Python Client​

\n

As a prerequisite, we need to install the StreamPipes Python client and all other dependencies,

\n
pip install git+https://github.com/apache/streampipes.git#subdirectory=streampipes-client-python
pip install scikit-learn==1.4.0 skl2onnx==1.16.0 onnxruntime==1.17.1
\n

The next step is to configure and initialize an instance of the client.

\n
import os
from streampipes.client import StreamPipesClient
from streampipes.client.config import StreamPipesClientConfig
from streampipes.client.credential_provider import StreamPipesApiKeyCredentials

os.environ[\"BROKER-HOST\"] = \"localhost\"
os.environ[\"KAFKA-PORT\"] = \"9094\"  # When using Kafka as message broker

config = StreamPipesClientConfig(
    credential_provider=StreamPipesApiKeyCredentials(
        username=\"admin@streampipes.apache.org\",
        api_key=\"TOKEN\",
    ),
    host_address=\"localhost\",
    https_disabled=True,
    port=80
)

client = StreamPipesClient(client_config=config)
\n

In case you have never worked with the Python client before and have problems to get started,\nplease have a look at\nour tutorial.

\n

If you already have an ONNX model and are only interested in applying it with StreamPipes on a data stream, you can skip\nthe following section.

\n

Model Training with Historic Data​

\n

As said above, the aim of our model is to detect anomalies of the volume_flow parameter. For this task, we will use\nIsolation Forests. Please note that the focus of the tutorial is not\non training the model, so please be patient even though the training is very simplified and lacks important preparation\nsteps such as standardization.

\n

As a first step, lets query the flowrate data from the StreamPipes data lake and extract the values of volume_flow\nas a feature:

\n
flowrate_df = client.dataLakeMeasureApi.get(\"flow-rate\").to_pandas()
X = flowrate_df[\"volume_flow\"].values.reshape(-1, 1).astype(\"float32\")
\n

As a next step, we can already train our model with the historic data:

\n
from sklearn.ensemble import IsolationForest

model = IsolationForest(contamination=0.01)
model.fit(X)
\n

The contamination parameter models the proportion of outliers in the data. See\nthe scikit-learn\ndocumentation for more information.

\n

Here you can see how this simple model performs:

\n\n

\n

This doesn't look too bad, right? Let's continue by converting our model to the ONNX representation.

\n
from onnxconverter_common import FloatTensorType
from skl2onnx import to_onnx

model_onnx = to_onnx(
    model,
    initial_types=[('input', FloatTensorType([None, X.shape[1]]))],
    target_opset={'ai.onnx.ml': 3, 'ai.onnx': 15, '': 15}
)

with open(\"isolation_forest.onnx\", \"wb\") as f:
    f.write(model_onnx.SerializeToString())
\n

Model Inference with Live Data​

\n

Utilizing a pre-trained model within StreamPipes becomes seamless with the ONNX interoperability standard, enabling\neffortless application of your existing model on live data streams.

\n

Interacting with live data from StreamPipes is facilitated through StreamPipes functions. Below, we'll create a Python\nStreamPipes function that leverages an ONNX model to generate predictions for each incoming event, making the results\naccessible as a data stream within StreamPipes for subsequent steps.

\n

So let's create an ONNXFunction that is capable of applying a model in ONNX representation to a StreamPipes data\nstream.\nIf you'd like to read more details about how functions are defined, refer\nto our tutorial.

\n
import numpy as np
import onnxruntime as rt

from streampipes.functions.broker.broker_handler import get_broker_description
from streampipes.functions.streampipes_function import StreamPipesFunction
from streampipes.functions.utils.data_stream_generator import create_data_stream, RuntimeType
from streampipes.functions.utils.function_context import FunctionContext
from streampipes.model.resource import FunctionDefinition, DataStream

from typing import Dict, Any, List


class ONNXFunction(StreamPipesFunction):

    def __init__(self, feature_names: list[str], input_stream: DataStream):
        output_stream = create_data_stream(
            name=\"flowrate-prediction\",
            attributes={
                \"is_anomaly\": RuntimeType.BOOLEAN.value
            },
            broker=get_broker_description(input_stream)
        )

        function_definition = FunctionDefinition(
            consumed_streams=[input_stream.element_id]
        ).add_output_data_stream(output_stream)

        self.feature_names = feature_names
        self.input_name = None
        self.output_name = None
        self.session = None

        super().__init__(function_definition=function_definition)
    
    ...
\n

First, we need to take care about the data stream that is required to send the predictions from our function to\nStreamPipes. Thus, we create a dedicated output data stream which we need to provide with the attributes our event will\nconsist of (a timestamp attribute is always added automatically). This output data stream needs to be registered at the\nfunction definition which is to be passed to the parent class. Lastly, we need to define some instance variables that\nare mainly required for the ONNX runtime.

\n

Next, we need to ensure that ONNX runtime session is created on start up. Thus, we need to invoke an InferenceSession\nand retrieving the corresponding configuration parameters:

\n
class ONNXFunction(StreamPipesFunction):
  
    ...
    
    def onServiceStarted(self, context: FunctionContext) -> None:
        self.session = rt.InferenceSession(
            path_or_bytes=\"isolation_forest.onnx\",
            providers=rt.get_available_providers(),
        )
        self.input_name = self.session.get_inputs()[0].name
        self.output_name = self.session.get_outputs()[0].name
        
    ...

\n

Lastly, we need to implement the inference logic that is applied to every event.\nIf you have brought up your own model, you need to adapt line 10-13:

\n
class ONNXFunction(StreamPipesFunction):

    ...

    def onEvent(self, event: Dict[str, Any], streamId: str) -> None:
        feature_vector = []
        for feature in self.feature_names:
            feature_vector.append(event[feature])

        prediction = self.session.run(
            [self.output_name],
            {self.input_name: np.expand_dims(np.array(feature_vector), axis=0).astype(\"float32\")}
        )[0]

        output = {
            \"is_anomaly\": int(prediction[0]) == -1
        }

        self.add_output(
            stream_id=self.function_definition.get_output_stream_ids()[0],
            event=output
        )

    def onServiceStopped(self) -> None:
        pass
\n

Having the function code in place, we can start the function with the following:

\n
from streampipes.functions.registration import Registration
from streampipes.functions.function_handler import FunctionHandler

stream = [
    stream
    for stream
    in client.dataStreamApi.all()
    if stream.name == \"flow-rate\"
][0]

function = ONNXFunction(
    feature_names=[\"volume_flow\"],
    input_stream=stream
)

registration = Registration()
registration.register(function)
function_handler = FunctionHandler(registration, client)
function_handler.initializeFunctions()
\n

We can now access the live values of the prediction in the StreamPipes UI, e.g., in the pipeline editor.

\n\n

\n

From here on you can further work with the prediction events in StreamPipes, e.g., by sending notifications\nto MS Teams.

", "url": "https://streampipes.apache.org/blog/2024/03/27/anomaly-detection-with-python-functions/", "title": "Anomaly Detection with StreamPipes Functions in Python and ONNX", "summary": "Apache StreamPipes saves the day when it comes to connecting to data sources in the IIoT world. Want to do more with", "date_modified": "2024-03-27T00:00:00.000Z", "author": { "name": "Tim Bossenmaier", "url": "https://github.com/bossenti" }, "tags": [ "tutorial" ] }, { "id": "https://streampipes.apache.org/blog/2023/11/28/release-093/", "content_html": "

9 minutes to read

\n

We're excited to share the release of StreamPipes 0.93.0, bringing with it a range of bug fixes and new features.\nThis release places a strong emphasis on refining the technical foundations of\nStreamPipes, featuring significant changes that set the stage for our major 1.0 release.\nNevertheless, we haven't overlooked the end-user experience and have implemented several enhancements.\nLet's delve into the key updates introduced in this release.

\n

The current release can be downloaded here.

\n

Adapters & Pipeline Elements​

\n

Editing existing adapters​

\n

StreamPipes has streamlined the process of modifying existing adapter instances, eliminating the inconvenience users\npreviously faced. In prior versions, altering an adapter required a cumbersome sequence of stopping, deleting, and then\nrecreating the adapter. However, with the introduction of StreamPipes 0.93.0, this arduous process is now a thing of the\npast.

\n

Users can now seamlessly edit running adapters, and the changes will be applied automatically, obviating the need for\nthe aforementioned manual steps. To guide you through this newfound capability, refer to the below animation,\nwhich demonstrates how to modify an existing adapter.

\n\"animation\n
\n
\n

While the automatic migration seamlessly handles the majority of cases, there are instances where it may not be\nfeasible. One such scenario arises when an event property, used by a pipeline element, is removed. In such cases,\nStreamPipes proactively alerts users before initiating the adapter update. Following the alert, the system takes\nprecautionary measures by halting the adapter and marking the associated pipeline for review.

\n\"animation\n

OPC-UA allows detailed & asynchronous browsing​

\n

We have fully reworked the browsing interface of our OPC-UA adapter.

\n\"demonstration\n
\n
\n

Now, all you need to do is specify the URL where the OPC UA server is accessible, eliminating the need to concern\nyourself with namespaces or node ids. Once the URL is provided, the adapter effortlessly presents all top-level nodes\nwithin the user interface. You can then navigate through the node tree interactively by clicking on the desired node.

\n

For every data node, accessing node information is a breeze – simply click on the eye icon. If you wish to subscribe to\na particular node, a straightforward click on the plus icon is all it takes.

\n

All the nodes you select become integral to the event structure of this adapter, accessible by clicking on Next. The\nensuing view not only displays the OPC UA status for each node but also allows for the removal of a node with ease,\nespecially if its status is not Good.

\n\"Screenshot\n

Adapter for S7 PLCs supports array-based data​

\n

Until now, our S7 PLC adapter lacked the capability to read values structured in arrays. We're pleased to announce that\nthis limitation has been addressed through the introduction of an Advanced register input mode.

\n\"Screenshot\n
\n
\n

This new mode empowers you to configure the registers to be queried in a manner specific to the requirements of our\nunderlying dependency, PLC4X. For detailed insights and guidance on the configuration\nprocess, refer to the PLC4X S7 documentation.

\n

When dealing with array definitions, the configuration would resemble the following:

\n
# propertyName=%{Memory-Area}{start-address}:{Data-Type}[{array-size}]
temperature=%DB3.DB0.0[BOOL]
\n

New Adapter: ifm IOLink​

\n

The IO-Link Master (AL1350) from ifm comes equipped with an IoT interface\ndesigned for transmitting sensor data via MQTT. With the introduction of this adapter, users gain the capability to\neffortlessly incorporate IO-Link sensor data into StreamPipes. This experimental integration empowers users to\neffortlessly connect to data from the ifm sensor VVB001.

\n

For more details refer to\nthe adapter documentation.

\n

For the integration of additional sensors we are thinking about supporting IODD.\nYour feedback on these considerations is invaluable to us. If you find the integration of IODD\nfor supporting more sensors within StreamPipes appealing or have insights to share, we welcome your input. Please let us\nknow your thoughts on this potential feature enhancement.

\n

New Processor: Numeric Rounding​

\n

Introducing our latest data processor, Numeric Rounding , designed to perform rounding operations on numeric values\nwithin an event stream. This processor enables users to select one or more numeric values from their event data,\nspecifying the desired rounding precision and the rounding mode.

\n

For more details refer to\nthe processor documentation.

\n

UX improvements​

\n

Ordered measurement units​

\n

In the adapter configuration step, when setting up a transformation rule for an event value, the available units of\nmeasurement are now conveniently organized in alphabetical order for improved accessibility and ease of selection.

\n

Extended information for existing adapters​

\n

When accessing the Connect module, a comprehensive list of all existing adapters is presented. Within this view, each\nadapter now includes details about the timestamp of the last event creation and the total count of elements published\nsince startup. This enhancement provides a quick and efficient means to evaluate the health of each adapter.

\n\"Screenshot\n

Interactive tutorial is back​

\n

Upon initiating a new StreamPipes instance, a dialog prompts users to embark on an interactive tutorial showcasing the\nplatform's key features. We're pleased to announce that this tutorial is now operational again. Enjoy exploring and\ndiscovering the most important functionalities in StreamPipes. Happy learning!

\n\"Screenshot\n

Deployment​

\n

Kubernetes Deployment​

\n

Add probes to the Helm chart​

\n

Integrating readiness, liveness, and startup probes into StreamPipes' Kubernetes containers enhances the system's health\nmonitoring. These probes automatically detect and handle anomalies, significantly improving application availability and\nstability. The readiness probe ensures preparedness, the liveness probe validates ongoing functionality, and the startup\nprobe assesses initialization status, collectively fortifying StreamPipes against potential disruptions.

\n

Rework the values file​

\n

We've refined and simplified the values file for our Helm chart, optimizing its structure. Furthermore, we've invested\nin comprehensive documentation for all available options. Check out\nthe README file of our Helm chart for detailed\ninformation on configuration and deployment.

\n

Add Apache Pulsar to Helm file​

\n

In our last release, we've expanded StreamPipes' capabilities to include Apache Pulsar as a supported messaging layer.\nAlongside this enhancement, our Helm chart has now been updated to seamlessly deploy StreamPipes on Kubernetes with\nPulsar\nas the underlying messaging infrastructure.

\n

Monitoring Integration (Grafana & Prometheus)​

\n

We've now integrated monitoring support for StreamPipes through Prometheus and Grafana, making it seamlessly available\nin both our Kubernetes and Docker deployments. If you're using Kubernetes, we recommend checking out\nour Helm chart for\nstraightforward setup instructions. For Docker users, stay tuned for an upcoming blog post where we'll provide a\nstep-by-step guide on configuring Prometheus and Grafana to monitor StreamPipes effectively.

\n

New Deployment mode for first time users​

\n

To simplify onboarding with Apache StreamPipes and demonstrate its capabilities, we've added a quickstart deployment\noption. This docker-compose variant provides users with a pre-configured StreamPipes instance and rich examples. Explore\nit in our user guide.

\n

StreamPipes Python​

\n

Our official Python support for Apache StreamPipes got some small improvements as well.\nNotable among other improvements are especially the following two:

\n\n

Administration & Development​

\n

Email Templates are configurable​

\n

StreamPipes already supported the sending of notifications, such as those for password recovery and account\nregistration, when properly configured. However, in previous versions, the email content was predefined and couldn't be\nadjusted. In this latest release, administrators now have the flexibility to modify the email templates, offering\ngreater customization options for notifications.

\n

Remove legacy resources​

\n

In this release, we've eliminated legacy resources that were initially employed for demonstration purposes. As these\nresources are no longer actively supported, we have decided to completely remove them. The affected components include\nthe vehicle simulator and watertank simulator.\nThis removal has led to the discontinuation of the StreamPipes package streampipes-sources.

\n

Furthermore, we are pleased to announce the forthcoming removal of Consul as an additional dependency required to run\nStreamPipes in the upcoming release. Within this release, we have already implemented the fundamental steps to eliminate\nthe need for Consul. This marks a significant stride towards making StreamPipes a more lightweight platform. It's\ncrucial to highlight that this modification is purely a technical enhancement and will have no impact on the end user\nexperience.

\n

Enable migrations for pipeline elements​

\n

We have established a crucial technical foundation to simplify the update process for upcoming versions of StreamPipes.\nIt is now possible to define migrations for adapters, data processors, and data sinks. For a more in-depth understanding\nof how migrations function and how to define them, please refer to\nour documentation.

\n

New Homepage​

\n

We've undergone a comprehensive redesign of our website, streampipes.apache.org. Take\na moment to explore the new look,\nand we welcome your feedback on your experience. Let us know what you think!

\n

Security​

\n

We've addressed a vulnerability in the default password generation for new users during the self-registration process.\nPreviously, a cryptographically weak pseudo-random number generator (PRNG) was utilized, but we've now rectified this by\nimplementing Apache Commons RNG. We extend our sincere thanks to Alessandro Albani from the Digital Security Division\nVar\nGroup for bringing this to our attention.

\n

Final Remarks​

\n

We highly recommend updating to StreamPipes 0.93.0 to take advantage of these new features, bug fixes, and improvements.\nFor more detailed information, please refer to\nthe release notes.

\n

We appreciate your continued support and valuable feedback.\nStreamPipes is continually evolving, and we are dedicated to providing you with a powerful and reliable platform for\nyour IIoT streaming data needs.

", "url": "https://streampipes.apache.org/blog/2023/11/28/release-093/", "title": "Apache StreamPipes release 0.93.0", "summary": "9 minutes to read", "date_modified": "2023-11-28T00:00:00.000Z", "author": { "name": "Tim Bossenmaier", "url": "https://github.com/bossenti" }, "tags": [ "release" ] }, { "id": "https://streampipes.apache.org/blog/2023/06/16/release-092/", "content_html": "

8 minutes to read

\n

We are pleased to announce the release of StreamPipes 0.92.0, which includes several bug fixes, performance improvements, and new features.\nThis version focuses on enhancing the technical aspects of StreamPipes.\nLet's explore the key updates in this release:

\n

The current release can be downloaded here.

\n

Improved Python Support:​

\n

In response to valuable user feedback, we have improved Python support in StreamPipes.\nUsers can now seamlessly integrate Python-based scripts, libraries, and modules into their functions.\nThis enhancement enables leveraging the power of machine learning and advanced analytics algorithms for IIoT data sources.\nPlease refer to the Python documentation for further details.

\n\n

To expand the functionality of StreamPipes, we have introduced several new processing elements.\nAmong them is the Swinging Door Trending (SDT) Filter Processor.\nThis processor provides a linear trend compression algorithm.\nEssentially, it replaces a series of continuous points (timestamp, value) with a straight line determined by the start and end points.

\n\"demonstration\n

Bug Fixes and Improvements:​

\n

We have addressed various bugs and issues reported by our users, resulting in a more stable and reliable StreamPipes experience.\nWe have also made several performance and usability improvements to enhance overall system efficiency.

\n

Deprecation of Set Adapters​

\n

In order to streamline the StreamPipes ecosystem, we have deprecated the functionality of Set Adapters.\nGoing forward, our focus will solely be on streaming event sources.\nHowever, we understand the importance of working with files of sensor data, especially during development.\nTo accommodate this need, we have extended the File Stream Adapter, which now offers the following functionalities:

\n
    \n
  1. Create a live simulation of sensor data based on the file.
    2. \n
  2. Replay the data in the file once, for example, to store it in the Data Lake.
    3. \n
\n

IMPORTANT: The set adapters will be removed in the next release.

\n

From this release onwards, we encourage you to utilize the File Stream Adapter instead of the File Set Adapter.\nTo simulate a file set using the File Stream Adapter, you can configure it with Replay Once: Yes and Replay Speed: Fastest (Ignore original time).\nSee the example below:

\n\"demonstration\n

Final Remarks​

\n

We highly recommend updating to StreamPipes 0.92.0 to take advantage of these new features, bug fixes, and improvements.\nFor more detailed information, please refer to the release notes.

\n

We appreciate your continued support and valuable feedback.\nStreamPipes is continually evolving, and we are dedicated to providing you with a powerful and reliable platform for your IIoT streaming data needs.

", "url": "https://streampipes.apache.org/blog/2023/06/16/release-092/", "title": "Apache StreamPipes release 0.92.0", "summary": "8 minutes to read", "date_modified": "2023-06-16T00:00:00.000Z", "author": { "name": "Philipp Zehnder", "url": "https://github.com/tenthe" }, "tags": [ "release" ] }, { "id": "https://streampipes.apache.org/blog/2023/04/12/release-091/", "content_html": "

14 minutes to read

\n
\n

The Apache StreamPipes community is delighted to announce the release of Apache StreamPipes version 0.91.0!\nThe biggest highlight of this release is the birth of our official StreamPipes Python library.\nApart from that, we have made a lot of improvements under the hood, especially in terms of improving the development experience (e.g., by introducing and enforcing code style rules).\nNevertheless, our new version includes more than 90 improvements and bug fixes in total.

\n

The current release can be downloaded here.

\n

StreamPipes Python​

\n

Apache StreamPipes aims to enable non-technical users to connect and analyze IoT data streams.\nTo this end, it provides an easy-to-use and convenient user interface that allows one to connect to an IoT data source and create some visual\ngraphs within a few minutes.
\nAlthough this is the main use case of Apache StreamPipes, it can also provide great value for people who are eager to work on data analysis or data science with IoT data, but don't want to get in touch with all the hassle associated with extracting data from devices in a suitable format.\nIn this scenario, StreamPipes helps you connect to your data source and extract the data for you.\nYou then can make the data available outside StreamPipes by writing it into an external source, such as a database, Kafka, etc.\nWhile this requires another component, you can also extract your data directly from StreamPipes programmatically using the StreamPipes API.\nFor convenience, we also provide you with a StreamPipes client both available for Java and Python.\nSpecifically with StreamPipes Python, we want to address the amazing data analytics and data science community in Python and benefit from the great universe of Python libraries out there.

\n

Getting started​

\n

Besides our official version, which you can access here,\nwe provide Apache StreamPipes as a convenient release via PyPI.\nTherefore, you can easily install it via the following command

\n
pip install streampipes
\n

All documentation related to StreamPipes Python can be found here.

\n

But now enough words are written, let's just start with a few examples.\nAs a first step, we want to establish a connection to a running StreamPipes instance.

\n
from streampipes.client import StreamPipesClient
from streampipes.client.config import StreamPipesClientConfig
from streampipes.client.credential_provider import StreamPipesApiKeyCredentials

config = StreamPipesClientConfig(
    credential_provider = StreamPipesApiKeyCredentials(
        username = \"test@streampipes.apache.org\",
        api_key = \"DEMO-KEY\",
        ),
    host_address = \"localhost\",
    https_disabled = True,
    port = 80
)

client = StreamPipesClient(client_config=config)
\n

client.describe() gives you an overview about what can be discovered via the Python client:

\n
Hi there!
You are connected to a StreamPipes instance running at http://localhost:80.
The following StreamPipes resources are available with this client:
1x DataLakeMeasures
1x DataStreams
\n

The required API key can be retrieved from StreamPipes via the UI:

\n\"demonstration\n

Query Data from StreamPipes data lake​

\n

As a first step, we can see which data sets are available in the data lake:

\n
client.dataLakeMeasureApi.all().to_pandas()
\n

To query data for a specific data lake measure, the following command takes you there:

\n
client.dataLakeMeasureApi.get(identifier=\"flow-rate\").to_pandas()
\n

There are some parameters that allow you to query data more specifically, visit the docs for more detailed information.

\n

If you are curious which features are available for StreamPipes Python, have a look at our tutorial section.

\n

We plan to further extend the Python library in the next releases - we welcome your ideas on any features you'd like to see!

\n

Create Adapter without starting​

\n

Until now, when creating an adapter in StreamPipes, the adapter was started immediately after completion.\nFrom now on you can choose via the adapter overview in the last dialog whether the adapter should be started immediately or later (this is then possible in the adapter overview).

\n\"demonstration\n

Redirect after Login​

\n

This is another small user interface improvement: when a user opens a StreamPipes link even though they are not logged in, e.g.,\nhttp://streampipes.demo/#/connect, StreamPipes displays the login dialog and now redirects the user directly to the target view\n(in this case the connection overview).

\n

Support Output Streams for Standalone Functions​

\n

Within the last release, 0.90.0, we introduced StreamPipes Functions as a lightweight alternative to processing elements.
\nStreamPipes Functions do now support having output streams, which means that they allow not only to consume but to send data to StreamPipes as well.

\n

Below you can see a simple example function that demonstrates how events can be pushed back to StreamPipes again.

\n
public class FunctionPublishExample extends StreamPipesFunction {

  private static final Logger LOG = LoggerFactory.getLogger(FunctionPublishExample.class);

  private static final String FUNCTION_ID = \"org.apache.streampipes.example.function.publish\";
  private static final String STREAM_APP_ID = \"example-output-stream-1\";

  private SpOutputCollector outputCollector;

  @Override
  public List<String> requiredStreamIds() {
    return List.of(\"urn:streampipes.apache.org:eventstream:EtMUkN\");
  }

  @Override
  public void onServiceStarted(FunctionContext context) {
    LOG.info(\"Service started\");
    this.outputCollector = context.getOutputCollectors().get(STREAM_APP_ID);
  }

  @Override
  public void onEvent(Event event, String streamId) {
    LOG.info(\"on event\");
    var ev = new Event();
    ev.addField(\"timestamp\", System.currentTimeMillis());
    ev.addField(\"example-property\", \"abc\");
    this.outputCollector.collect(ev);
  }

  @Override
  public void onServiceStopped() {
    LOG.info(\"service stopped\");
  }

  @Override
  public FunctionConfig getFunctionConfig() {

    return FunctionConfigBuilder
        .create(FunctionId.from(FUNCTION_ID, 1))
        .withOutputStream(DataStreamBuilder.create(STREAM_APP_ID, \"My Function Stream\", \"\")
            .property(EpProperties.timestampProperty(\"timestamp\"))
            .property(EpProperties.stringEp(
                Labels.from(\"my-example-property\", \"test\", \"test\"),
                \"example-property\",
                SO.TEXT,
                PropertyScope.MEASUREMENT_PROPERTY))
            .format(Formats.jsonFormat())
            .protocol(Protocols.kafka(\"localhost\", 9094, STREAM_APP_ID))
            .build())
        .build();
  }
}
\n

This functionality is also already supported in StreamPipes Python! 🚀

\n

This means that you can write a StreamPipes function in Python that allows you to consume streams of data from StreamPipes,\nperform any data manipulation in Python, and send the data back to StreamPipes, where it can be further processed.

\n
\n

To get a comprehensive overview of all changes released within Apache StreamPipes 0.91.0\nplease visit our release notes.

", "url": "https://streampipes.apache.org/blog/2023/04/12/release-091/", "title": "Apache StreamPipes release 0.91.0", "summary": "14 minutes to read", "date_modified": "2023-04-12T00:00:00.000Z", "author": { "name": "Tim Bossenmaier", "url": "https://github.com/bossenti" }, "tags": [ "release" ] }, { "id": "https://streampipes.apache.org/blog/2023/01/17/release-090/", "content_html": "

5 minutes to read

\n
\n

The Apache StreamPipes community is pleased to announce the immediate availability of Apache StreamPipes version 0.90.0!\nIn total, this release was focused on stabilization and closes 57 issues with improvements and bug fixes as well as some new features such as adapters and data sinks.

\n

The current release can be downloaded here.

\n

Improvements​

\n

The focus of this release cycle was on general usability improvements and bug fixes. We continued to harmonize the\nStreamPipes web interface, so that all views follow a similar navigation. Many improvements were added to the data explorer:

\n\n

In addition, we improved features to monitor pipelines and extensions services. The pipeline monitoring component has been restructured and now works independent from the underlying message broker.\nIn addition, we extended the SDK so that runtime errors can be forwarded to the StreamPipes UI.\nFurther improvements to the monitoring stack will be added in the next versions.

\n

New Adapters and Data Sinks​

\n

This version improves support for existing message brokers in form of adapters and sinks:

\n\n

GitHub Community Features​

\n

Apache StreamPipes is a community effort - everyone is welcome not only to contribute code, but also to provide feedback and other input which helps improving the product.\nTo improve community interaction, we switched our issue management from Jira to GitHub issues - making it much easier to create issues and to discuss future features.\nIn addition, we've enabled Github Discussions - feel free to participate in the existing discussions and ask questions if you have problems!

\n\n

And if you prefer the mailing list, subscribe to dev@streampipes.apache.org or users@streampipes.apache.org.\nTech-related and general project discussions as well as release votes will continue to happen on the dev list.

\n\"Github\n

Preview of Standalone Functions​

\n

An exciting new feature is available in preview mode: StreamPipes Functions are a new way to interact with connected data streams in StreamPipes.\nAs an open system, there are already several ways to extend the functionality of StreamPipes: New adapter can be added to connect to non-supported systems.\nIn addition, the SDK supports building custom data processors and sinks, so that StreamPipes can be easily extended at runtime in case new analytics operators are needed.

\n

But providing extensions in form of pipeline elements are not always the best choice: E.g., in cases where non-generic algorithms should be executed on a specific data stream,\nproviding a reusable component for pipeline development is unsuitable.

\n

This is where Functions come into play! StreamPipes Functions allow to define standalone components, which consume one or more connected data streams from StreamPipes and can be used to add custom processing logic.\nE.g., it becomes easy to apply an ML model on continuous data without the need to build a specific data processing pipeline.

\n

Functions can be registered in an extensions service similar to data processors and sinks by extending a single class:

\n
public class MyFunction extends StreamPipesFunction {
  @Override
  public FunctionId getFunctionId() {
    return FunctionId.from(\"my-app-id\", 1);
  }

  @Override
  public List<String> requiredStreamIds() {
    return List.of(\"my-stream-id\");
  }

  @Override
  public void onServiceStarted(FunctionContext context) {
    // called on service startup
  }

  @Override
  public void onEvent(Event event, String streamId) {
    // called for each incoming event
  }

  @Override
  public void onServiceStopped() {
    // called on service shutdown
  }
}
\n

This concept, which will also come for more programming languages besides Java offer new ways for developers to interact with available data in StreamPipes and build custom extensions for IIoT analytics.

\n

Functions are available in preview mode and their API should therefore be considered unstable.

\n

Future Roadmap​

\n

After this stabilization release, the next version will most probably bring a major new feature - Python support!\nThe community is currently working hard to bring the StreamPipes client and functions to the Python world. Stay tuned!

\n
\n

To get a comprehensive overview of all changes released within Apache StreamPipes 0.90.0\nplease visit our release notes.

", "url": "https://streampipes.apache.org/blog/2023/01/17/release-090/", "title": "Apache StreamPipes release 0.90.0", "summary": "5 minutes to read", "date_modified": "2023-01-17T00:00:00.000Z", "author": { "name": "Dominik Riemer", "url": "https://github.com/dominikriemer" }, "tags": [ "release" ] } ] }