What are the options for committing offsets?

[joedyndale]

How are offsets committed? Only using auto-commit, or can we commit after each message or message batch? This is crucial information to have, since auto-commit does not provide at-least-once guarantees.

[joedyndale]

Found the information here. Would be nice with a link to that in the README or the Wiki Q&A seeing as this is very important information :) Very glad to see it's handled in the manner guaranteeing at-least-once delivery :)

[IainAdamsLabs]

@joedyndale - could you help me? I am looking at that document but unable to find the information on committing offsets. Can you explicitly commit an offset?

[joedyndale]

@IainAdamsLabs I don't think you can explicitly commit, so you don't have full control. However, from the following section (emphasis mine):

The function is executed by two type of executors: SingleItem and MultiItem. A single item executor calls the received items in a loop, one by one in parallel if multiple partitions exist. The multi item executor calls the function a single time, items will be in order, batch can contain items from distinct partitions. Once all items have been processed the checkpoint is saved.

I understand it to mean the caller only commits messages passed to the function after the function returns successfully. So if there's an unhandled exception the messages will be retried automatically (possibly leading to duplicates).