Have a use case where none of the existing processors fit?
Don’t worry—we’ve got just the thing for you.

The JavaScript Evaluator Processor in Apache StreamPipes allows you to define custom event logic with just a few lines of JavaScript. Whether you want to modify values, filter events, or even add stateful logic, this processor empowers you to do just that—without writing a full processing element in Java.

Let’s take a closer look at how it works, what the configuration options are, and how you can use it in three different ways.

What is the JavaScript Evaluator?​

Sometimes you need to apply custom logic to your data stream. Maybe it's a quick transformation or a complex filter that isn’t supported out of the box. The JavaScript Evaluator Processor is designed exactly for these cases.

With a simple JavaScript function, you can:

• Modify existing fields
• Add new fields
• Filter out unwanted events
• Maintain state across events

All you need is a little coding and a bit of creativity.

Configuring the JavaScript Processor​

When adding the JavaScript Evaluator to your pipeline, you’ll notice two key configuration sections:

1. The JavaScript Function
   This function is triggered once for each incoming event.

2. The Output Schema
   This defines the structure of your transformed events. It’s important to ensure that the output schema matches what your function returns—otherwise, downstream processors might not be able to validate the events correctly.

Let’s explore some examples that show different ways to apply custom logic to your event streams using JavaScript.

Example 1: Stateless Event Transformation​

The simplest use case is when you want to modify values in an event.

Use case: You want to halve the temperature value of each incoming event.

function process(event) {
    event.temperature = event.temperature / 2;
    return event;
}

That’s it! Each event passes through this function, gets modified, and is sent on its way. No extra setup needed.

Example 2: Custom Event Filtering​

You don’t always want to forward every event. The JavaScript Evaluator lets you return null (or just omit return) to drop an event.

Use case: Only forward events where the temperature is between 50 and 100.

function process(event) {
    if (event.temperature > 50 && event.temperature < 100) {
        return event;
    }
}

With this setup, only "valid" events get through. All others are silently ignored. Perfect for quick, custom filtering logic.

Example 3: Stateful Logic​

Need to track information across multiple events? You can wrap your function in an Immediately Invoked Function Expression (IIFE) to create persistent variables.

Use case: Count how many events have passed through the processor and append that number to each event.

(() => {
    let count = 0;

    return function process(event) {
        count += 1;
        event.count = count;
        return event;
    };
})()

This small trick makes it easy to introduce state without needing any external dependencies or full-blown processor development. You can track counts, store intermediate values, or even implement basic alerting. Just make sure that any new properties you add to the event, like count in the example, are also included in the output event schema during processor configuration, otherwise the pipeline may fail validation.

Wrap-Up​

The JavaScript Evaluator is one of the most flexible processors in StreamPipes. Whether you're a beginner or a power user, it lets you:

✅ Write custom transformations
✅ Implement simple filters
✅ Track state between events
✅ Prototype complex logic—without deploying a new microservice

So, next time you think "I wish there was a processor that could...", just try the JavaScript Evaluator. Chances are, it can.

Got a cool use case using this processor? Let us know or share it with the community—we’d love to see what you’re building with Apache StreamPipes!

Introduction​

Are you excited to explore Apache StreamPipes but find yourself without any sensors to play with? Or maybe you're looking to validate StreamPipes for your use case and kick off a Proof of Concept (PoC) but the hardware is not yet available? No worries—the File Stream Adapter is here to save the day!

With the File Stream Adapter, you can upload a file containing sensor data in various formats (JSON, CSV, XML) and replay this data, giving you a hands-on way to dive into StreamPipes. It's not meant for production use, but it's perfect for experimenting with StreamPipes using your own data.

This blog post will take you on a exciting journey through configuring the File Stream Adapter and using it to simulate sensor data streams.

Adapter Configuration​

Let's break down the configuration options for the File Stream Adapter:

Configuration Options Explained​

1. File: Choose an existing file or upload a new one.
2. Overwrite file time: If checked, the current time at which the event is triggered is used for the timestamp of the event, regardless of the timestamp value in the file.
3. Replay Once: Decide if the file should be replayed once or continuously until stopped.
   • Options: yes (file is replayed once), no (file is replayed continuously)
4. Replay Speed: Set the frequency of events.
   • Options: Keep original time, Fastest (Ignore original time), Speed Up Factor
5. Format: Choose the format to parse the events.
   • Options: JSON, CSV, XML, Image (experimental feature)

Use Case: Create a Sensor Simulator​

Let's create a simulation data stream using your own sensor data. First, store the sample CSV data in a local file.

timestamp,density,mass_flow,sensorId,sensor_fault_flags,temperature,volume_flow
1719897720887,41.82,6.306,flowrate01,false,41.9,3.365
1719897721952,45.09,9.659,flowrate01,false,43.2,4.103
1719897722958,42.84,9.532,flowrate01,true,42.6,3.110
1719897723964,49.58,1.929,flowrate01,true,40.3,3.4684
1719897724970,41.83,4.173,flowrate01,false,40.7,0.246
1719897725975,48.11,10.90,flowrate01,false,46.4,5.275

To set up the adapter, follow these steps:

1. Open StreamPipes Connect.
2. Click on + New Adapter.
3. Select the File Stream Adapter.
4. Enter the following configuration:
   • File: Upload your sample data file.
   • Overwrite file time: Check this box to use the current timestamp instead of the original time of the event in the file.
   • Replay Once: Choose no to create an ongoing data stream.
   • Replay Speed: Select Keep original time to maintain the original event frequency.
   • Format: Pick CSV, set the delimiter as , and check the Header box.
5. Click Next once you've added the correct configuration.
6. Configure the event schema by editing the timestamp field and selecting Mark as timestamp. Save your changes.
7. You can add additional meta information to the event schema, but it's not required for this tutorial. Click Next when done.
8. Enter an Adapter Name.
9. Check the Persist events box to store the data in the StreamPipes data store.
10. Click Start Adapter.

Once configured, you should see the following preview:

Use Case: Persist CSV File to Data Lake​

Now, let's store the entire content of the file in the data lake. Once stored, you can analyze it with the Data Explorer. Here’s how to configure the adapter:

• Overwrite file time: Leave this unchecked to retain the original timestamps.
• Replay Once: Choose yes to read the file only once.
• Replay Speed: Select Fastest (Ignore original time) to write data as fast as possible.

Conclusion​

In this tutorial, we've explored how to use the File Stream Adapter in Apache StreamPipes to create a simulation data stream and persist CSV data into a data lake. This adapter lets you experiment with StreamPipes' capabilities and explore its functionality using your own data. Happy streaming!

Introduction​

Apache StreamPipes 0.97.0 brings exciting new features, enhanced security, and significant UX improvements to streamline your IIoT data processing workflows. This release introduces Adapters & Pipelines as Code, enabling users to define and deploy StreamPipes resources programmatically using YAML or JSON. Security enhancements, such as OPC UA certificate support and Kafka SSL/SASL authentication, improve connectivity and data integrity. Additionally, we have made various UX improvements, including an enhanced Data Explorer and asset-based filtering.

In total, the new release closes 159 issues including new features, bug fixes and improvements.

Read on to discover what’s new in StreamPipes 0.97.0!

Adapters & Pipeline Elements​

Adapter & Pipeline as Code​

StreamPipes now supports managing adapters and pipelines programmatically using compact Json or YAML definitions. These features are ideal for automating deployments, integrating with CI/CD pipelines, and applying infrastructure-as-code practices.

Adapters as Code​

With Adapters as Code, you can define and deploy adapters via a compact description, streamlining the creation of multiple similar adapters and simplifying automation. For example, here's a sample YAML snippet to define an OPC-UA adapter:

name: My OPC Adapter
id: testadapter
description: Test
appId: org.apache.streampipes.connect.iiot.adapters.opcua
configuration:
  - opc_host_or_url: OPC_URL
    opc_server_url: opc.tcp://localhost:62541/milo
  - adapter_type: PULL_MODE
    pulling_interval: 1000
  - access_mode: UNAUTHENTICATED
  - available_nodes:
    - "ns=2;s=Leakage Test Station/temperature"
    - "ns=2;s=Leakage Test Station/pressure"

schema:
  temperature:
    propertyScope: measurement
    label: Temp
    description: Temperature value
    semanticType: http://schema.org/temperature

enrich:
  timestamp: timestamp

transform:
  rename:
    pressure: pressure3

createOptions:
  persist: true
  start: true

This approach considerably simplifies the programmatic creation of adapters Learn more about Adapters as Code

Pipelines as Code​

Pipelines as Code lets you define end-to-end data flows—from data sources through processors to sinks. This function is ideal for the standardization and automation of pipeline deployments. For example, here’s a sample YAML snippet for a simple pipeline:

id: my-pipeline
name: ""
description: ""
pipelineElements:  # Define pipeline components here
  - type: stream     # Data source
    ref: <reference> # Unique reference ID
    id: <data-stream-id> # ID of the stream

  - type: processor  # Data transformation
    ref: <reference> # Unique reference ID
    id: <processor-id> # ID of the processor
    connectedTo:     # Previous pipeline element reference(s)
      - <reference>
    configuration:   # Processor-specific configurations
      - <configuration-option>

  - type: sink       # Data sink (output)
    ref: <reference> # Unique reference ID
    id: <sink-id>    # ID of the sink
    connectedTo:     # Previous pipeline element reference(s)
      - <reference>
    configuration:   # Sink-specific configurations
      - <configuration-option>

createOptions:
  start: <true|false>  # Whether to start the pipeline immediately

The code preview of a pipeline has also been added to the Start Pipeline dialog, which has been further improved to make the execution status of pipelines more clear.

This method provides a programmatic interface to deploy and manage pipelines quickly and easily. Learn more about Pipelines as Code

OPC UA Certificates​

The OPC UA adapter now supports both signed/encrypted and unencrypted communication. Certificate provisioning is handled externally via environment variables, following a dedicated security directory layout. Refer to the OPC UA adapter documentation for further details.

https://github.com/apache/streampipes/pull/3354

Kafka Avro & Certificates​

The Kafka adapter and sink now support SSL and SASL for enhanced security. Additionally, users can provide custom producer/consumer configurations. A new Avro parser has been added along with an input field for specifying the consumer group in the Kafka adapter.

New Features & Deprecations​

Asset Browser​

StreamPipes now allows users to model real-world assets and associate them with StreamPipes resources such as adapters, pipelines, and dashboards. These asset mappings enable intuitive filtering across different modules, making it easier to navigate and manage related resources.

The screenshot below illustrates an example where a StreamPipes instance is configured with two machines and various stations. Clicking on a machine or station dynamically filters the associated resources in the list on the right. This asset-based filtering is available across all main modules of StreamPipes.

The asset overview panel has been redesigned to provide a more intuitive and user-friendly experience. It is possible to assign labels, locations, and ISA95 types to assets. For each asset, resources such as pipelines, adapters and dashboards can be assigned. The asset overview contains an intuitive overview of all linked resources.

To manage locations & sites, we extended the configuration menu with additional options to add a custom map layer provider and to configure the map view.

UX improvements​

File Relocation​

The file upload configuration has been moved from the top-level navigation to the settings menu, improving the overall structure and clarity of the main navigation.

This change does not affect file uploads required by adapters or pipeline elements—these configurations remain unchanged. Further, users can still upload files as before via the files configuration section.

Adapter Data Preview​

Viewing live data streams in the adapter configuration has been improved. By clicking on "details", a full-screen view of the data preview is displayed. Changing data values are highlighted, so that it is ease to track changes. The Code panel shows the configuration in JSON or YAML format.

Data Explorer Configuration Improvements​

Configuring data views in the Data Explorer is now more intuitive and user-friendly. We have enhanced validation, making it easier to select data sources and specify queries while reducing potential misconfigurations.

Key improvements include:

• Default Table Visualization: New data views now default to a table view, minimizing setup effort.
• Improved Time Range Picker: More flexibility in selecting time ranges, including support for different time ranges per widget in Data Lake dashboards.
• Auto-Refresh for Dashboards: Live updates are now possible with the new auto-refresh functionality.
• Better Error Messages: Enhanced feedback when configurations are incorrect to improve usability.
• Custom color coding for pie chart: In the pie chart, custom colors can be mapped to specific data values.
• Traffic light visualization: A new traffic light visualization has been added to the Data Explorer.

Here is a screenshot of the new traffic light visualization:

Pipeline Details​

The pipeline details view has been completely remodeled. It now shows the pipeline in full-screen mode, making it easier to view and edit complex pipelines. A sidebar contains further information on the pipeline, such as logs, pipeline elements, and execution status. In addition, execution metrics are now intuitively accessible right in the pipeline details view.

Deprecation Notice: Dashboard​

The current live dashboard feature in StreamPipes is being deprecated and will be removed in a future release. As a replacement, the Data Explorer dashboards has been enhanced with auto-refresh functionality, ensuring a seamless transition for users who rely on real-time data visualization.

The harmonization of data explorer and live dashboard functionalities will provide a more consistent user experience and will ease the addition of new charts in future version, since all visualizations now rely on a single visualization library.

We encourage all users to migrate their existing dashboards to the Data Explorer to continue leveraging powerful visualization and analysis capabilities.

If you think any feature is missing in the data explorer, please let us know!

Conclusion​

We appreciate the ongoing support and contributions from our community in making StreamPipes better with each release. Version 0.97.0 brings improvements in usability, security, and automation, helping users to analyze their IIoT data sources.

As always, we welcome your feedback and suggestions to continue refining and enhancing StreamPipes. Feel free to share your thoughts or get involved through GitHub or mailing list.

We're excited to share the release of StreamPipes 0.95.0, bringing with it a range of bug fixes and new features. In total, the newest release closes over 200 issues. This release doesn't include many large feature drops, we have worked a lot on making small things smoother and improving StreamPipes under the hood. As we continue to work on the 1.0 release of Apache StreamPipes probably later this year, this release features many improvements to the end-user experience and bug fixes.

Let's delve into the key updates introduced in this release.

Apache StreamPipes saves the day when it comes to connecting to data sources in the IIoT world. Want to do more with your IIoT data than just analyze it in a dashboard? If so, this blog post is for you! We'll show you how to extract historical data from StreamPipes, use it to train a machine learning model, bring the model back to StreamPipes using ONNX, and apply the model to live data.

We're excited to share the release of StreamPipes 0.93.0, bringing with it a range of bug fixes and new features. This release places a strong emphasis on refining the technical foundations of StreamPipes, featuring significant changes that set the stage for our major 1.0 release. Nevertheless, we haven't overlooked the end-user experience and have implemented several enhancements. Let's delve into the key updates introduced in this release.

We are pleased to announce the release of StreamPipes 0.92.0, which includes several bug fixes, performance improvements, and new features. This version focuses on enhancing the technical aspects of StreamPipes. Let's explore the key updates in this release:

The Apache StreamPipes community is delighted to announce the release of Apache StreamPipes version 0.91.0! The biggest highlight of this release is the birth of our official StreamPipes Python library. Apart from that, we have made a lot of improvements under the hood, especially in terms of improving the development experience (e.g., by introducing and enforcing code style rules). Nevertheless, our new version includes more than 90 improvements and bug fixes in total.

The Apache StreamPipes community is pleased to announce the immediate availability of Apache StreamPipes version 0.90.0! In total, this release was focused on stabilization and closes 57 issues with improvements and bug fixes as well as some new features such as adapters and data sinks.

The Apache StreamPipes (incubating) community is delighted to announce the release of Apache StreamPipes version 0.70.0! Key highlights of this release include many enhancements to the data explorer for fast exploration of IoT data and a brand new asset management that allows importing and exporting of StreamPipes resources. In addition, our new version includes more than 50 improvements and bug fixes.