Introduction​

In many manufacturing settings, maintenance is still scheduled at fixed time intervals — often based on past experience rather than actual machine usage. This approach can lead to unnecessary servicing or, worse, unexpected breakdowns. A common limitation is that many machines, especially older ones, offer only minimal sensor data, making advanced maintenance strategies seem out of reach.

However, with Apache StreamPipes, even limited data — such as a simple boolean signal indicating activity — can be leveraged to implement usage-based maintenance. In this blog post, we demonstrate how this is achieved using a real-world example: a packaging machine that processes goods at high throughput. You'll learn how to count machine operations, store event data, visualize usage over time, and trigger notifications when maintenance is due — all with just a few clicks in StreamPipes.

The Scenario: Packaging Machine Monitoring​

Let’s consider a packaging machine that wraps goods into individual packages. The wear and tear on this machine is directly related to the number of items processed. However, due to the machine’s age, only minimal data is available.

Currently:

• Maintenance occurs at fixed intervals.
• These intervals were determined on the basis of the technicians' experience.
• There is suspicion that maintenance is done too frequently, leading to unnecessary costs.

With Apache StreamPipes, we’ll build a pipeline that:

• Counts packaging operations in real time.
• Stores data for analysis and visualization.
• Sends notifications once a maintenance threshold is reached.

The Data: A Simple Boolean Signal​

The machine provides a boolean flag — active — which is true when a packaging operation occurs. This flag is streamed using an Apache StreamPipes S7 adapter. The stream also includes a few additional process parameters:

Part 1: Absolute Counter with Notification​

We’ll begin by building a pipeline that counts how often the machine packages a product and sends an email alert when a predefined threshold is reached.

Pipeline Overview​

A simple StreamPipes pipeline for monitoring the parts produced at the machine

Step-by-Step Guide​

1. Add the Data Source

   • Drag the data stream of the packaging machine into the pipeline editor.

2. Insert the Boolean Counter Processor

   • Monitors a boolean field and counts transitions (e.g., false → true).
   • Configuration:
     • Boolean Field: active
     • Flank Parameter: FALSE → TRUE

   The processor emits a new event each time the machine packages an item, incrementing a counter value.

3. Persist the Count

   • Add the Data Lake sink to store these events for visualization.

4. Add a Threshold Alert

   • Insert the Numerical Filter processor.
     • Field: counter
     • Filter Operation: >
     • Threshold: 100000
   • Connect this to the Email Notification sink.
     • Configure recipient, subject, and set the Silent Period to avoid spamming the team.

5. Visualize in Dashboard

   • Start the pipeline.
   • In the dashboard, add an Indicator Widget displaying the counter value.

The live dashboard with an indicator chart showing the current number of goods produced

This setup gives you a live counter of how many products were packaged and notifies the maintenance team when attention is needed.

Part 2: Aggregated Counting Over Time​

While a cumulative counter is useful, often it's more insightful to understand how usage varies over time — for example, how many items are processed each day.

A Time-Aggregated Approach​

Instead of a total counter, we’ll now generate a single event for each packaging operation and then use the Data Explorer to visualize activity per hour or per day.

Alternative Pipeline Using Signal Edge Filter​

Pipeline to recognize when a part is being processed

Step-by-Step:

1. Replace the Boolean Counter with a Signal Edge Filter processor.

   • Filters events based on signal transitions.
   • Configuration:
     • Boolean Field: active
     • Signal Edge: FALSE → TRUE
     • Delay: 0
     • Output Event Selection: First

2. Store the filtered stream in the Data Lake sink.

This gives one clean event per packaging operation — ideal for aggregation.

Visualization in the Data Explorer​

Now that we have one event per packaging action, we can analyze the data over time in the Data Explorer.

1. Open the Data Explorer
2. Table View Configuration:
   • Data Source: Select the stream used in the pipeline.
   • Aggregation: Group by Hour or Day.
   • Field Aggregation: Use count on any field to count the number of events.

3. Bar Chart Visualization:
   • Switch to a Time-Series Chart.
   • Select the counted field.
   • Choose bar from the chart type dropdown.

You now have a clear, time-based view of machine usage. This helps spot trends, detect unusual downtimes, and better plan maintenance.

Conclusion​

In this post, we’ve shown how even limited sensor data — like a simple boolean signal — can be used to build powerful maintenance logic using Apache StreamPipes. With just a few clicks, you can:

• Count operations and trigger maintenance alerts
• Store and visualize historical data
• Understand machine usage over time with time-based aggregations

By replacing static maintenance intervals with actual usage insights, you can reduce unnecessary work and improve efficiency — all without needing to retrofit expensive new sensors.

Need a quick sentiment check, a rolling summary, or a safety rule that’s just a bit too fuzzy for SQL?
With the Multi-Model Prompt Processor you can drop the full might of OpenAI, Anthropic, or your own Llama into any Apache StreamPipes pipeline—no extra code required.

Below you’ll find:

• A quick tour of the options
• Five every day use-cases
• A hands-on GIF tutorial
• Tips on writing prompts that return only what you need
• A full list of the models that work out of the box

What does the processor do?​

The processor sends your prompt + current event data to the model, grabs the reply, and writes it to a new field llmResponse; ideal when the answer must flow straight into the next processor.

Five ways to use it​

(Replace the $placeholders$ with the actual stream attributes in the mapping dialog.)

Tutorial – from stream to JSON warning​

Below is a short clip showing the processor in action in a temperature pipeline.
The prompt forces the model to output a two-field JSON—perfect for machine parsing.

Model: claude-3-5-sonnet-20241022
Prompt Used:

You are an assistant that ALWAYS replies with a single JSON object
containing two string fields: "status" and "message".

Rules
1. If the temperature is strictly greater than 45 °C:
     • "status" must be "WARNING"
     • "message" must explain that the temperature is above the safe limit
2. Otherwise:
     • "status" must be "OK"
     • "message" must note that the temperature is within the safe range
3. Do not output anything except the JSON object.

### Examples

Temperature: 50
{
  "status": "WARNING",
  "message": "Temperature exceeds the 45 °C threshold"
}

Temperature: 30
{
  "status": "OK",
  "message": "Temperature is within the safe range"
}

Temperature: 45
{
  "status": "OK",
  "message": "Temperature is within the safe range"
}

Temperature: 60
{
  "status": "WARNING",
  "message": "Temperature exceeds the 45 °C threshold"
}

The model now returns either:

{
  "status": "WARNING",
  "message": "Temperature exceeds the 45 °C threshold"
}

or

{
  "status": "OK",
  "message": "Temperature is within the safe range"
}

Nothing else—exactly what we want for downstream processors.

Prompt writing tips for event streams​

1. State the output format first: “Reply with OK or WARNING only.”
2. Give one or two concrete examples – LLMs copy patterns.
3. End with a clear cue: “Answer:” or “Output:”.
4. Keep temperature ≤ 0.3 when strict structure matters.
5. Short replies reduce context size in Windowed / Full modes.

Supported models​

claude-3-7-sonnet-20250219
claude-3-5-sonnet-20241022
claude-3-5-haiku-20241022
claude-3-5-sonnet-20240620
claude-3-opus-20240229
claude-3-sonnet-20240229
claude-3-haiku-20240307
claude-2.1
claude-2.0

gpt-3.5-turbo              gpt-3.5-turbo-1106     gpt-3.5-turbo-0125
gpt-3.5-turbo-16k          gpt-4                  gpt-4-0613
gpt-4-turbo-preview        gpt-4-1106-preview     gpt-4-0125-preview
gpt-4-turbo                gpt-4-turbo-2024-04-09 gpt-4-32k
gpt-4-32k-0613             gpt-4o                 gpt-4o-2024-05-13
gpt-4o-2024-08-06          gpt-4o-2024-11-20      gpt-4o-mini
gpt-4o-mini-2024-07-18     o1                     o1-mini
o1-2024-12-17              o1-mini-2024-09-12     o1-preview
o1-preview-2024-09-12      o3-mini-2025-01-31

Final thoughts​

Whether you need a lightweight rule engine, live summaries, or just a smarter filter, the Multi-Model Prompt Processor drops straight into your pipeline and starts working within minutes.

Give it a try, and let us know what you build!

Have a use case where none of the existing processors fit?
Don’t worry—we’ve got just the thing for you.

The JavaScript Evaluator Processor in Apache StreamPipes allows you to define custom event logic with just a few lines of JavaScript. Whether you want to modify values, filter events, or even add stateful logic, this processor empowers you to do just that—without writing a full processing element in Java.

Let’s take a closer look at how it works, what the configuration options are, and how you can use it in three different ways.

What is the JavaScript Evaluator?​

Sometimes you need to apply custom logic to your data stream. Maybe it's a quick transformation or a complex filter that isn’t supported out of the box. The JavaScript Evaluator Processor is designed exactly for these cases.

With a simple JavaScript function, you can:

• Modify existing fields
• Add new fields
• Filter out unwanted events
• Maintain state across events

All you need is a little coding and a bit of creativity.

Configuring the JavaScript Processor​

When adding the JavaScript Evaluator to your pipeline, you’ll notice two key configuration sections:

1. The JavaScript Function
   This function is triggered once for each incoming event.

2. The Output Schema
   This defines the structure of your transformed events. It’s important to ensure that the output schema matches what your function returns—otherwise, downstream processors might not be able to validate the events correctly.

Let’s explore some examples that show different ways to apply custom logic to your event streams using JavaScript.

Example 1: Stateless Event Transformation​

The simplest use case is when you want to modify values in an event.

Use case: You want to halve the temperature value of each incoming event.

function process(event) {
    event.temperature = event.temperature / 2;
    return event;
}

That’s it! Each event passes through this function, gets modified, and is sent on its way. No extra setup needed.

Example 2: Custom Event Filtering​

You don’t always want to forward every event. The JavaScript Evaluator lets you return null (or just omit return) to drop an event.

Use case: Only forward events where the temperature is between 50 and 100.

function process(event) {
    if (event.temperature > 50 && event.temperature < 100) {
        return event;
    }
}

With this setup, only "valid" events get through. All others are silently ignored. Perfect for quick, custom filtering logic.

Example 3: Stateful Logic​

Need to track information across multiple events? You can wrap your function in an Immediately Invoked Function Expression (IIFE) to create persistent variables.

Use case: Count how many events have passed through the processor and append that number to each event.

(() => {
    let count = 0;

    return function process(event) {
        count += 1;
        event.count = count;
        return event;
    };
})()

This small trick makes it easy to introduce state without needing any external dependencies or full-blown processor development. You can track counts, store intermediate values, or even implement basic alerting. Just make sure that any new properties you add to the event, like count in the example, are also included in the output event schema during processor configuration, otherwise the pipeline may fail validation.

Wrap-Up​

The JavaScript Evaluator is one of the most flexible processors in StreamPipes. Whether you're a beginner or a power user, it lets you:

✅ Write custom transformations
✅ Implement simple filters
✅ Track state between events
✅ Prototype complex logic—without deploying a new microservice

So, next time you think "I wish there was a processor that could...", just try the JavaScript Evaluator. Chances are, it can.

Got a cool use case using this processor? Let us know or share it with the community—we’d love to see what you’re building with Apache StreamPipes!

Introduction​

Are you excited to explore Apache StreamPipes but find yourself without any sensors to play with? Or maybe you're looking to validate StreamPipes for your use case and kick off a Proof of Concept (PoC) but the hardware is not yet available? No worries—the File Stream Adapter is here to save the day!

With the File Stream Adapter, you can upload a file containing sensor data in various formats (JSON, CSV, XML) and replay this data, giving you a hands-on way to dive into StreamPipes. It's not meant for production use, but it's perfect for experimenting with StreamPipes using your own data.

This blog post will take you on a exciting journey through configuring the File Stream Adapter and using it to simulate sensor data streams.

Adapter Configuration​

Let's break down the configuration options for the File Stream Adapter:

Configuration Options Explained​

1. File: Choose an existing file or upload a new one.
2. Overwrite file time: If checked, the current time at which the event is triggered is used for the timestamp of the event, regardless of the timestamp value in the file.
3. Replay Once: Decide if the file should be replayed once or continuously until stopped.
   • Options: yes (file is replayed once), no (file is replayed continuously)
4. Replay Speed: Set the frequency of events.
   • Options: Keep original time, Fastest (Ignore original time), Speed Up Factor
5. Format: Choose the format to parse the events.
   • Options: JSON, CSV, XML, Image (experimental feature)

Use Case: Create a Sensor Simulator​

Let's create a simulation data stream using your own sensor data. First, store the sample CSV data in a local file.

timestamp,density,mass_flow,sensorId,sensor_fault_flags,temperature,volume_flow
1719897720887,41.82,6.306,flowrate01,false,41.9,3.365
1719897721952,45.09,9.659,flowrate01,false,43.2,4.103
1719897722958,42.84,9.532,flowrate01,true,42.6,3.110
1719897723964,49.58,1.929,flowrate01,true,40.3,3.4684
1719897724970,41.83,4.173,flowrate01,false,40.7,0.246
1719897725975,48.11,10.90,flowrate01,false,46.4,5.275

To set up the adapter, follow these steps:

1. Open StreamPipes Connect.
2. Click on + New Adapter.
3. Select the File Stream Adapter.
4. Enter the following configuration:
   • File: Upload your sample data file.
   • Overwrite file time: Check this box to use the current timestamp instead of the original time of the event in the file.
   • Replay Once: Choose no to create an ongoing data stream.
   • Replay Speed: Select Keep original time to maintain the original event frequency.
   • Format: Pick CSV, set the delimiter as , and check the Header box.
5. Click Next once you've added the correct configuration.
6. Configure the event schema by editing the timestamp field and selecting Mark as timestamp. Save your changes.
7. You can add additional meta information to the event schema, but it's not required for this tutorial. Click Next when done.
8. Enter an Adapter Name.
9. Check the Persist events box to store the data in the StreamPipes data store.
10. Click Start Adapter.

Once configured, you should see the following preview:

Use Case: Persist CSV File to Data Lake​

Now, let's store the entire content of the file in the data lake. Once stored, you can analyze it with the Data Explorer. Here’s how to configure the adapter:

• Overwrite file time: Leave this unchecked to retain the original timestamps.
• Replay Once: Choose yes to read the file only once.
• Replay Speed: Select Fastest (Ignore original time) to write data as fast as possible.

Conclusion​

In this tutorial, we've explored how to use the File Stream Adapter in Apache StreamPipes to create a simulation data stream and persist CSV data into a data lake. This adapter lets you experiment with StreamPipes' capabilities and explore its functionality using your own data. Happy streaming!

Introduction​

Apache StreamPipes 0.97.0 brings exciting new features, enhanced security, and significant UX improvements to streamline your IIoT data processing workflows. This release introduces Adapters & Pipelines as Code, enabling users to define and deploy StreamPipes resources programmatically using YAML or JSON. Security enhancements, such as OPC UA certificate support and Kafka SSL/SASL authentication, improve connectivity and data integrity. Additionally, we have made various UX improvements, including an enhanced Data Explorer and asset-based filtering.

In total, the new release closes 159 issues including new features, bug fixes and improvements.

Read on to discover what’s new in StreamPipes 0.97.0!

Adapters & Pipeline Elements​

Adapter & Pipeline as Code​

StreamPipes now supports managing adapters and pipelines programmatically using compact Json or YAML definitions. These features are ideal for automating deployments, integrating with CI/CD pipelines, and applying infrastructure-as-code practices.

Adapters as Code​

With Adapters as Code, you can define and deploy adapters via a compact description, streamlining the creation of multiple similar adapters and simplifying automation. For example, here's a sample YAML snippet to define an OPC-UA adapter:

name: My OPC Adapter
id: testadapter
description: Test
appId: org.apache.streampipes.connect.iiot.adapters.opcua
configuration:
  - opc_host_or_url: OPC_URL
    opc_server_url: opc.tcp://localhost:62541/milo
  - adapter_type: PULL_MODE
    pulling_interval: 1000
  - access_mode: UNAUTHENTICATED
  - available_nodes:
    - "ns=2;s=Leakage Test Station/temperature"
    - "ns=2;s=Leakage Test Station/pressure"

schema:
  temperature:
    propertyScope: measurement
    label: Temp
    description: Temperature value
    semanticType: http://schema.org/temperature

enrich:
  timestamp: timestamp

transform:
  rename:
    pressure: pressure3

createOptions:
  persist: true
  start: true

This approach considerably simplifies the programmatic creation of adapters Learn more about Adapters as Code

Pipelines as Code​

Pipelines as Code lets you define end-to-end data flows—from data sources through processors to sinks. This function is ideal for the standardization and automation of pipeline deployments. For example, here’s a sample YAML snippet for a simple pipeline:

id: my-pipeline
name: ""
description: ""
pipelineElements:  # Define pipeline components here
  - type: stream     # Data source
    ref: <reference> # Unique reference ID
    id: <data-stream-id> # ID of the stream

  - type: processor  # Data transformation
    ref: <reference> # Unique reference ID
    id: <processor-id> # ID of the processor
    connectedTo:     # Previous pipeline element reference(s)
      - <reference>
    configuration:   # Processor-specific configurations
      - <configuration-option>

  - type: sink       # Data sink (output)
    ref: <reference> # Unique reference ID
    id: <sink-id>    # ID of the sink
    connectedTo:     # Previous pipeline element reference(s)
      - <reference>
    configuration:   # Sink-specific configurations
      - <configuration-option>

createOptions:
  start: <true|false>  # Whether to start the pipeline immediately

The code preview of a pipeline has also been added to the Start Pipeline dialog, which has been further improved to make the execution status of pipelines more clear.

This method provides a programmatic interface to deploy and manage pipelines quickly and easily. Learn more about Pipelines as Code

OPC UA Certificates​

The OPC UA adapter now supports both signed/encrypted and unencrypted communication. Certificate provisioning is handled externally via environment variables, following a dedicated security directory layout. Refer to the OPC UA adapter documentation for further details.

https://github.com/apache/streampipes/pull/3354

Kafka Avro & Certificates​

The Kafka adapter and sink now support SSL and SASL for enhanced security. Additionally, users can provide custom producer/consumer configurations. A new Avro parser has been added along with an input field for specifying the consumer group in the Kafka adapter.

New Features & Deprecations​

Asset Browser​

StreamPipes now allows users to model real-world assets and associate them with StreamPipes resources such as adapters, pipelines, and dashboards. These asset mappings enable intuitive filtering across different modules, making it easier to navigate and manage related resources.

The screenshot below illustrates an example where a StreamPipes instance is configured with two machines and various stations. Clicking on a machine or station dynamically filters the associated resources in the list on the right. This asset-based filtering is available across all main modules of StreamPipes.

The asset overview panel has been redesigned to provide a more intuitive and user-friendly experience. It is possible to assign labels, locations, and ISA95 types to assets. For each asset, resources such as pipelines, adapters and dashboards can be assigned. The asset overview contains an intuitive overview of all linked resources.

To manage locations & sites, we extended the configuration menu with additional options to add a custom map layer provider and to configure the map view.

UX improvements​

File Relocation​

The file upload configuration has been moved from the top-level navigation to the settings menu, improving the overall structure and clarity of the main navigation.

This change does not affect file uploads required by adapters or pipeline elements—these configurations remain unchanged. Further, users can still upload files as before via the files configuration section.

Adapter Data Preview​

Viewing live data streams in the adapter configuration has been improved. By clicking on "details", a full-screen view of the data preview is displayed. Changing data values are highlighted, so that it is ease to track changes. The Code panel shows the configuration in JSON or YAML format.

Data Explorer Configuration Improvements​

Configuring data views in the Data Explorer is now more intuitive and user-friendly. We have enhanced validation, making it easier to select data sources and specify queries while reducing potential misconfigurations.

Key improvements include:

• Default Table Visualization: New data views now default to a table view, minimizing setup effort.
• Improved Time Range Picker: More flexibility in selecting time ranges, including support for different time ranges per widget in Data Lake dashboards.
• Auto-Refresh for Dashboards: Live updates are now possible with the new auto-refresh functionality.
• Better Error Messages: Enhanced feedback when configurations are incorrect to improve usability.
• Custom color coding for pie chart: In the pie chart, custom colors can be mapped to specific data values.
• Traffic light visualization: A new traffic light visualization has been added to the Data Explorer.

Here is a screenshot of the new traffic light visualization:

Pipeline Details​

The pipeline details view has been completely remodeled. It now shows the pipeline in full-screen mode, making it easier to view and edit complex pipelines. A sidebar contains further information on the pipeline, such as logs, pipeline elements, and execution status. In addition, execution metrics are now intuitively accessible right in the pipeline details view.

Deprecation Notice: Dashboard​

The current live dashboard feature in StreamPipes is being deprecated and will be removed in a future release. As a replacement, the Data Explorer dashboards has been enhanced with auto-refresh functionality, ensuring a seamless transition for users who rely on real-time data visualization.

The harmonization of data explorer and live dashboard functionalities will provide a more consistent user experience and will ease the addition of new charts in future version, since all visualizations now rely on a single visualization library.

We encourage all users to migrate their existing dashboards to the Data Explorer to continue leveraging powerful visualization and analysis capabilities.

If you think any feature is missing in the data explorer, please let us know!

Conclusion​

We appreciate the ongoing support and contributions from our community in making StreamPipes better with each release. Version 0.97.0 brings improvements in usability, security, and automation, helping users to analyze their IIoT data sources.

As always, we welcome your feedback and suggestions to continue refining and enhancing StreamPipes. Feel free to share your thoughts or get involved through GitHub or mailing list.

We're excited to share the release of StreamPipes 0.95.0, bringing with it a range of bug fixes and new features. In total, the newest release closes over 200 issues. This release doesn't include many large feature drops, we have worked a lot on making small things smoother and improving StreamPipes under the hood. As we continue to work on the 1.0 release of Apache StreamPipes probably later this year, this release features many improvements to the end-user experience and bug fixes.

Let's delve into the key updates introduced in this release.

Apache StreamPipes saves the day when it comes to connecting to data sources in the IIoT world. Want to do more with your IIoT data than just analyze it in a dashboard? If so, this blog post is for you! We'll show you how to extract historical data from StreamPipes, use it to train a machine learning model, bring the model back to StreamPipes using ONNX, and apply the model to live data.

We're excited to share the release of StreamPipes 0.93.0, bringing with it a range of bug fixes and new features. This release places a strong emphasis on refining the technical foundations of StreamPipes, featuring significant changes that set the stage for our major 1.0 release. Nevertheless, we haven't overlooked the end-user experience and have implemented several enhancements. Let's delve into the key updates introduced in this release.

We are pleased to announce the release of StreamPipes 0.92.0, which includes several bug fixes, performance improvements, and new features. This version focuses on enhancing the technical aspects of StreamPipes. Let's explore the key updates in this release:

The Apache StreamPipes community is delighted to announce the release of Apache StreamPipes version 0.91.0! The biggest highlight of this release is the birth of our official StreamPipes Python library. Apart from that, we have made a lot of improvements under the hood, especially in terms of improving the development experience (e.g., by introducing and enforcing code style rules). Nevertheless, our new version includes more than 90 improvements and bug fixes in total.