Skip to main content

StreamPipes release 0.61.0

· 4 min read

5 minutes to read


We are proud to release StreamPipes version 0.61.0, which brings many improvements and some exciting new features that simplify pipeline development! The new version, which is already available on Docker Hub and can be downloaded using our installer brings over 20 improvements. Under the hood, a reworked event model simplifies the development of new pipeline elements using the built-in SDK.

Our main mission is to make IoT data management & analytics accessible to everyone. This release focuses on better integrating StreamPipes with other systems and data sources and contains UX improvements to ease the development of pipelines.

StreamPipes Connect

StreamPipes Connect, which aims to easily connect arbitrary data sources with StreamPipes, has been first introduced as a public beta in version 0.60.0. Version 0.61.0 brings many new features to Connect and also implements many bug fixes that now make Connect the recommended way to connect data sources.

StreamPipes Connect relies on an extensible architecture to connect either generic protocols or specific adapters.

In more detail, StreamPipes Connect supports the following protocols, with a completely code-free integration process:

  • Data Streams

    • HTTP/REST
    • MQTT
    • Apache Kafka
    • File
    • ROS (preview)
    • GDELT
  • Data Sets

    • HTTP/REST
    • HDFS
    • File
StreamPipes Connect

The next version will also include an adapter for OPC UA, which is currently being tested in a real setting.

For protocols that are not yet supported by StreamPipes Connect, a new Maven archetype for easily connecting data sources using the SDK is provided.

Improved Pipeline Modeling

On the UI side, we try to make the graphical modeling of pipelines as simple as possible. Besides some minor improvements (e.g., case-insensitive search for pipeline elements), pipelines are now constantly validated and users get feedback on steps that are still missing (e.g., missing pipeline elements). As many other features we are currently implementing, this feature directly comes from a user request - so if you have any suggestions for improvement, let us know!

Pipeline Validation

Reworked Event Model

One of the most important features of v0.61.0 is not directly visible to end users, but drastically eases the development of new pipeline elements with the SDK. From this release on, events at runtime are represented in form of a new Event Object instead of the former Map representation. This feature allows us for much better support of complex data processors which have more than one input data stream and comes with many convenience methods to better extrat fields from an event for processing.

For instance, prior to v0.61.0 you had to extract fields from an incoming events as follows:

@Override
public void onEvent(Map<String, Object> in, String s, SpOutputCollector out) {
Integer numberValue = Integer.parseInt(String.valueOf(in.get(MAPPING_PROPERTY_ID));
}

The new event model is much simpler and less error-prone:

 public void onEvent(Event in, SpOutputCollector out) throws SpRuntimeException {
Integer numberValue = in
.getFieldBySelector(MAPPING_PROPERTY_SELECTOR)
.getAsPrimitive()
.getAsInteger();
}

The new event model is even more useful in cases where your data streams have a complex nested structure or contain list fields. If you connect two data streams containing the same fields to a processor, field names are automatically renamed in the background to avoid naming conflicts. A complete guide can be found in the developer guide.

Documentation

The developer guide has been extended: Besides reworked tutorials on developing new data processors and data sources, the new documentation includes instructions on adding stream requirements, the new event model, and in-depth guides on defining static properties and output strategies.

Other features and improvements

  • SSL support: The UI can now be configured to support SSL, see the documentation on required configurations in your docker-compose file.
  • A new map visualization to visualize spatial data streams
  • The customize dialog is now hidden in case no static properties are present
  • Adapter descriptions can be exported and imported
  • The UI does no longer load any external sources
  • Updated Maven archetypes

See the release notes for more details.

Feedback

We are absolutely open to your suggestions for further improvements! Let us know (by mail, slack or twitter) and we'll consider your feature request in the next release!