Rebalances can lead to wrong lag being reported

[nachogiljaldo]

Describe the bug

When commits are not immediate and/or some events can have a relatively high processing time and topics substain a low traffic, it can lead to lag being reported even if events were processed and committed.

Kafka Version

3.6.x

To Reproduce

This test reprocuces the behavior:
https://github.com/nachogiljaldo/kafka-go/blob/do_not_commit_offset_of_not_owned_partitions/reader_test.go#L1184

The situation is:

• consumer A has partitions 1 and 2
• consumer A receives msg 1 for partition 2
• consumer B appears and gets assigned partition 1
• consumer B receives msg 1 and 2 and commits them
• consumer A finishes consuming msg 1 and commits it --> offset goes back to 1, but consumer B does not know it, so it keeps on asking for offset 2

Expected Behavior

There are 2 things I would expect:

• committing a message for a partition that is not assigned to the current consumer fails
• if that happens, we read all events that now became uncommited but that doesn't happen either

Observed Behavior

We do not get the missing message which leads to a fake lag when the traffic has little traffic.