Questions tagged [apache-kafka]
Apache Kafka is a distributed streaming platform that is used for building real-time data pipelines and streaming applications.
apache-kafka
33,263
questions
1
vote
1
answer
17
views
How to start consuming messages from specific offset using Confluent Kafka Python consumer
Using the Confluent Kafka Python consumer, how do I start consuming from a specific offset? At first I was going use the seek() method, but documentation makes it seem like you can not start reading ...
- 9,646
0
votes
1
answer
14
views
How Can I Reduce Delay When Sending File Contents to Kafka Producer Using Script?
I want to write a bash script to constantly add new files to a folder and use cat to send the file contents to a Kafka producer with:
cat $FILEPATH |$KAFKA_HOME/bin/kafka-console-producer.sh --...
- 1
0
votes
0
answers
9
views
Guarantee consumer order message into kafka with one topic and multiple producers
I'm looking to scale my Kafka producers by having two or three instances of the same producer sending events to Kafka. However, due to network failures or other issues, some events might be produced ...
- 89
0
votes
0
answers
11
views
Kafka. Partition leader election
Does kafka automatically perform unclean leader election for partitions?
I have topic
./kafka-topics.sh --bootstrap-server XXX --describe --topic my-topic
Topic: my-topic TopicId: ...
0
votes
0
answers
8
views
conflit class loader between quarkus and springframework
I'm actually working in project to test quarkus kafka and in the same time I tried CXFclient in quarkus also.
When I run my unit test, I get the following error:
Runtime java.lang.RuntimeException: io....
0
votes
0
answers
19
views
How to create a custom deserialiser for Apache Flink
I am building a Flink Project using Python that streams events from a Kafka topic. The payload in the topics has been encoded and the SimpleStringSchema is unable to decode them. I have create a ...
0
votes
0
answers
12
views
Confluent-Kafka: no broker available for coordinator query: intervaled in state query-coord
I am using python-confluent-kafka to create a producer.
The Kafka cluster is on MSK 3.7.x KRaft with IAM enabled and TLS enabled, both within the cluster and between clients and brokers.
Any ideas on ...
- 305
0
votes
0
answers
8
views
Kafka message order while processing the message
In Kafka there are 3 topics, but the messages are like order ready, ship and pickup.. There are huge requests coming for these messages. These messages are related to each other, so how do you make ...
0
votes
0
answers
14
views
Strimzi Kafka Connect Cluster - Resource planning
I'm planing resources which will be dedicated to run Kafka Connect Clusters in Strimzi Distribution.
My workloads are pretty small, there will be probably only 1 task per connector needed.
I would go ...
- 134
0
votes
0
answers
7
views
Failed to find any class that implements Connector and which name matches io.debezium.connector.postgresql.PostgresConnector
getting below error while trying to connect debezium with kafka
.\bin\windows\connect-standalone.bat config/connect-standalone.properties config/postgres.properties
INFO Kafka Connect started (org....
0
votes
1
answer
16
views
Kafka-streams filter messages using headers
We're trying to use kafka streams for our project to read data from one topic and write to another, and we have a use case to use KafkaHeaders as a mechanism to filter our certain records.
Example, in ...
- 2,503
1
vote
0
answers
13
views
Cannot connect with kafka-ui to kafka
I start kafka locally with
$ docker run -p 9092:9092 apache/kafka:3.7.1
$ docker exec -it gifted_wright /bin/bash
e8bebba64b30:/$ cd /opt/kafka/bin
e8bebba64b30:/opt/kafka/bin$ sh kafka-topics.sh --...
- 33
-1
votes
1
answer
32
views
"Microservices Using Database Per Service" pattern in practice
I am working on a project where I would like to use this pattern. I have read for days, but I am still not sure how to implement this pattern properly and keep the multiple DBs in sync.
Let's suppose ...
- 21.9k
0
votes
0
answers
8
views
Filebeat output using SAS token to azure eventhub
I want to send my filebeat logs to eventhubs using SAS token, earlier we wanted to use Kafka output but we came to know Kafka doesn't support SAS tokens, so now is there some way we can send data ...
- 13
1
vote
0
answers
17
views
Kafka UI message count
I am using kafka UI as a visual interface for viewing topics. I wanted to check if my service produced correct number of messages to the topic. When i used live mode in given topic i was able to ...
- 81