Kafka Publisher

---

Description

The Kafka Publisher sink enables StreamPipes to publish events to Apache Kafka topics. It provides:

• Real-time event publishing to Kafka topics
• Support for various security configurations
• Support for multiple brokers
• Automatic topic creation if not exists
• Configurable message handling
• JSON message serialization

---

Required Input

This sink accepts any incoming event type and serializes it to JSON format before publishing to Kafka.

---

Configuration

Broker Settings

• Kafka Brokers: A comma-separated list of Kafka brokers as host:port (e.g., broker1:9092,broker2:9092).

Security Settings

• Security Protocol: Choose the security protocol for broker communication:

  • PLAINTEXT: No authentication and plaintext communication
  • SSL: Using SSL with no authentication
  • SASL/PLAINTEXT: SASL authentication without encryption
  • SASL/SSL: SASL authentication with SSL encryption

• Authentication (when using SASL):

  • Security Mechanism: Choose the SASL mechanism:
    • PLAIN: Simple username/password authentication
    • SCRAM-SHA-256: SCRAM authentication with SHA-256
    • SCRAM-SHA-512: SCRAM authentication with SHA-512
  • Username: SASL authentication username
  • Password: SASL authentication password

Topic Settings

• Topic: The Kafka topic where events will be published. If the topic doesn't exist, it will be created automatically with default settings.

Message Key

• No key: Records are published without a key
• Static value: A fixed text is attached to every record
• Event field: The value of a selected event field is used, primitive fields only
• Expression: Text combined with field placeholders in the form #fieldName#

Advanced Settings

• Additional Configurations: Add custom Kafka producer configurations in key=value format. Each configuration should be on a new line. For example:

  buffer.memory=33554432
  batch.size=16384
  linger.ms=20

---

Features

• Message Handling:

  • Automatic JSON serialization of events
  • Configurable message size limits
  • Batch processing support
  • Automatic topic creation

• Security:

  • SSL/TLS encryption support
  • SASL authentication with multiple mechanisms
  • Configurable security protocols

---

Use Cases

• Data Distribution: Publish processed events to Kafka for other systems to consume
• Event Streaming: Stream events to Kafka for real-time processing
• Data Integration: Integrate StreamPipes with Kafka-based data pipelines

---

Important Notes

• The sink uses the Kafka Producer API to publish messages
• Events are automatically serialized to JSON format
• Topics are created automatically if they don't exist
• For production use, it's recommended to configure appropriate security settings
• The sink supports all standard Kafka producer configurations through the additional settings