For installation instructions check out the getting started guide.

Added

• Field app_name added to the MongoDB components. (@mihaitodor)
• New openai_chat_completion processor. (@rockwotj)
• New openai_embeddings processor. (@rockwotj)
• New openai_image_generation processor. (@rockwotj)
• New openai_speech processor. (@rockwotj)
• New openai_transcription processor. (@rockwotj)
• New openai_translation processor. (@rockwotj)
• New ollama_chat processor. (@rockwotj)
• New ollama_embeddings processor. (@rockwotj)

Changed

• The gcp_pubsub output now rejects messages with metadata values which contain invalid UTF-8-encoded runes. (@AndreasBergmeier6176)
• The .goreleaser.yml configuration has been set back to version 1. (@Jeffail)
• The number of release build artifacts for the community and cloud flavours have been reduced due to Github Action Runner disk space limitations.

The full change log can be found here.

For installation instructions check out the getting started guide.

4.31.0 - 2024-07-19

Added

• The splunk input and splunk_hec output now support custom tls configuration. (@mihaitodor)
• Field timestamp added to the kafka and kafka_franz outputs. (@mihaitodor)
• (Benthos) Field max_retries added to the retry processor. (@mihaitodor)
• (Benthos) Metadata fields retry_count and backoff_duration added to the retry processor. (@mihaitodor)
• (Benthos) Parameter escape_html added to the format_json() Bloblang method. (@mihaitodor)
• (Benthos) New array bloblang method. (@gramian)
• (Benthos) Algorithm fnv32 added to the hash bloblang method. (@CallMeMhz)
• New experimental redpanda_data_transform. (@rockwotj)
• New -community suffixed build included in release artifacts, containing only FOSS functionality. (@Jeffail)
• New -cloud suffixed build included in release artifacts, containing components enabled in Redpanda Cloud. (@Jeffail)
• Field status_topic added to the global redpanda config block. (@Jeffail)
• New pinecone output. (@rockwotj)
• (Benthos) The /ready endpoint in regular operation now provides a detailed summary of all inputs and outputs, including connection errors where applicable. (@Jeffail)

Changed

• (Benthos) All cli subcommands that previously relied on root-level flags (streams, lint, test, echo) now explicitly define those flags such that they appear in help-text and can be specified after the subcommand itself. This means previous commands such as connect -r ./foo.yaml streams ./bar.yaml can now be more intuitively written as connect streams -r ./foo.yaml ./bar.yaml and so on. The old style will still work in order to preserve backwards compatibility, but the help-text for these root-level flags has been hidden. (@Jeffail)

For installation instructions check out the getting started guide.

Added

• (Benthos) Field omit_empty added to the lines scanner. (@mihaitodor)
• (Benthos) New scheme gcm added to the encrypt_aes and decrypy_aes Bloblang methods. (@abergmeier)
• (Benthos) New Bloblang method pow. (@mfamador)
• (Benthos) New sin, cos, tan and pi bloblang methods. (@mfamador)
• (Benthos) Field proxy_url added to the websocket input and output. (@mihaitodor)
• New experimental splunk input. (@mihaitodor)

Fixed

• The sql_insert and sql_raw components no longer fail when inserting large binary blobs into Oracle BLOB columns. (@mihaitodor)
• (Benthos) The websocket input and output now obey the HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables. (@mihaitodor)
• AWS Lambda serverless build artifacts have been added back to official releases.

Changed

• The splunk_hec output is now implemented as a native Go component. (@mihaitodor)

The full change log can be found here.

For installation instructions check out the getting started guide.

Added

• (Benthos) Field omit_empty added to the lines scanner. (@mihaitodor)
• (Benthos) New scheme gcm added to the encrypt_aes and decrypy_aes Bloblang methods. (@abergmeier)
• (Benthos) New Bloblang method pow. (@mfamador)
• (Benthos) New sin, cos, tan and pi bloblang methods. (@mfamador)
• (Benthos) Field proxy_url added to the websocket input and output. (@mihaitodor)
• New experimental splunk input. (@mihaitodor)

Fixed

• The sql_insert and sql_raw components no longer fail when inserting large binary blobs into Oracle BLOB columns. (@mihaitodor)
• (Benthos) The websocket input and output now obey the HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables. (@mihaitodor)

Changed

• The splunk_hec output is now implemented as a native Go component. (@mihaitodor)

The full change log can be found here.

For installation instructions check out the getting started guide.

Added

• Go API: New packages public/bundle/free and public/bundle/enterprise with explicit licensing for bundles of component imports.
• Field auth.oauth2.scope added to the pulsar input and output. (@srenatus)
• Field subscription_initial_position added to the pulsar input. (@srenatus)

Fixed

• The pulsar input and output should no longer ignore auth.oauth2 fields. (@srenatus)
• Creating builds using make no longer prints warnings when the repository does not contain a tag. (@mkysel)
• Messages resulting from the redis processor are no longer invalid when using hash commands. (@mkysel)
• The nats_jetstream input no longer fails to initialise when a stream is specified and a subject is not. (@maxarndt)

The full change log can be found here.

For installation instructions check out the getting started guide.

Changed

• The repository has been moved to redpanda-data/connect and no longer contains the core Benthos engine, which is now broken out into redpanda-data/benthos.

The full change log can be found here.

For installation instructions check out the getting started guide.

Added

• New nats_kv cache type.
• The nats_jetstream input now supports last_per_subject and new deliver fallbacks.
• Field error_patterns added to the drop_on output.
• New redis_scan input type.
• Field auto_replay_nacks added to all inputs that traditionally automatically retry nacked messages as a toggle for this behaviour.
• New retry processor.
• New noop cache.
• Field targets_input added to the azure_blob_storage input.
• New reject_errored output.
• New nats_request_reply processor.
• New json_documents scanner.

Fixed

• The unarchive processor no longer yields linting errors when the format csv:x is specified. This is a regression introduced in v4.25.0.
• The sftp input will no longer consume files when the watcher cache returns an error. Instead, it will reattempt the file upon the next poll.
• The aws_sqs input no longer logs error level logs for visibility timeout refreshing errors.
• The nats_kv processor now allows nats wildcards for the keys operation.
• The nats_kv processor keys operation now returns a single message with an array of found keys instead of a batch of messages.
• The nats_kv processor history operation now returns a single message with an array of objects containing the record fields instead of a batch of messages.
• Field timeout added to the nats_kv processor to specify the maximum period to wait on an operation before aborting and returning an error.
• Bloblang comparison operators (>, <, <=, >=) now match the precision of the compared integers when applicable.
• The parse_form_url_encoded Bloblang method no longer produces results with an unknown data type for repeated query parameters.
• The echo CLI command no longer fails to sanitise configs when encountering an empty password field.

Changed

• The log events from all inputs and outputs when they first connect have been made more consistent and no longer contain any information regarding the nature of their connections.
• Splitting message batches with a split processor (or custom plugins) no longer results in downstream error handling loops around nacks. This was previously implemented as a feature to ensure unbounded expanded and split batches don't flood downstream services in the event of a minority of errors. However, introducing more clever origin tracking of errored messages has eliminated the need for this undocumented behaviour.

The full change log can be found here.

For installation instructions check out the getting started guide.

Added

• Field credit added to the amqp_1 input to specify the maximum number of unacknowledged messages the sender can transmit.
• Bloblang now supports root-level if statements.
• New experimental sql cache.
• Fields batch_size, sort and limit added to the mongodb input.
• Field idemponent_write added to the kafka output.

Changed

• The default value of the amqp_1.credit input has changed from 1 to 64.
• The mongodb processor and output now support extended JSON in canonical form for document, filter and hint mappings.
• The open_telemetry_collector tracer has had the url field of gRPC and HTTP collectors deprecated in favour of address, which more accurately describes the intended format of endpoints. The old style will continue to work, but eventually will have its default value removed and an explicit value will be required.

Fixed

• Resource config imports containing % characters were being incorrectly parsed during unit test execution. This was a regression introduced in v4.25.0.
• Dynamic input and output config updates containing % characters were being incorrectly parsed. This was a regression introduced in v4.25.0.

The full change log can be found here.

For installation instructions check out the getting started guide.

Fixed

• Fixed a regression in v4.25.0 where template based components were not parsing correctly from configs.

The full change log can be found here.

For installation instructions check out the getting started guide.

Added

• Field address_cache added to the socket_server input.
• Field read_header added to the amqp_1 input.
• All inputs with a codec field now support a new field scanner to replace it. Scanners are more powerful as they are configured in a structured way similar to other component types rather than via a single string field, for more information check out the scanners page.
• New diff and patch Bloblang methods.
• New processors processor.
• Field read_header added to the amqp_1 input.
• A debug endpoint /debug/pprof/allocs has been added for profiling allocations.
• New cockroachdb_changefeed input.
• The open_telemetry_collector tracer now supports sampling.
• The aws_kinesis input and output now support specifying ARNs as the stream target.
• New azure_cosmosdb input, processor and output.
• All sql_* components now support the gocosmos driver.
• New opensearch output.

Fixed

• The javascript processor now handles module imports correctly.
• Bloblang if statements now provide explicit errors when query expressions resolve to non-boolean values.
• Some metadata fields from the amqp_1 input were always empty due to type mismatch, this should no longer be the case.
• The zip Bloblang method no longer fails when executed without arguments.
• The amqp_0_9 output no longer prints bogus exchange name when connecting to the server.
• The generate input no longer adds an extra second to interval: '@every x' syntax.
• The nats_jetstream input no longer fails to locate mirrored streams.
• Fixed a rare panic in batching mechanisms with a specified period, where data arrives in low volumes and is sporadic.
• Executing config unit tests should no longer fail due to output resources failing to connect.

Changed

• The parse_parquet Bloblang function, parquet_decode, parquet_encode processors and the parquet input have all been upgraded to the latest version of the underlying Parquet library. Since this underlying library is experimental it is likely that behaviour changes will result. One significant change is that encoding numerical values that are larger than the column type (float64 into FLOAT, int64 into INT32, etc) will no longer be automatically converted.
• The parse_log processor field codec is now deprecated.
• WARNING: Many components have had their underlying implementations moved onto newer internal APIs for defining and extracting their configuration fields. It's recommended that upgrades to this version are performed cautiously.
• WARNING: All AWS components have been upgraded to the latest client libraries. Although lots of testing has been done, these libraries have the potential to differ in discrete ways in terms of how credentials are evaluated, cross-account connections are performed, and so on. It's recommended that upgrades to this version are performed cautiously.

The full change log can be found here.