Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] In pull mode, provide PushConsumer with precise control of consumption rate #8302

Open
cserwen opened this issue Jun 17, 2024 · 0 comments · May be fixed by #8306
Open

[Feature] In pull mode, provide PushConsumer with precise control of consumption rate #8302

cserwen opened this issue Jun 17, 2024 · 0 comments · May be fixed by #8306

Comments

@cserwen
Copy link
Member

cserwen commented Jun 17, 2024

Is Your Feature Request Related to a Problem?

If we set clientRebalance as false, we can noly control consumption rate by following configurations:

consumeThreadMin
consumeThreadMax

In some scenarios, even a single thread may exceed the consumption rate that the business can accept. And it is difficult to calculate how many threads should be set based on the consumption rate

Generally, developers use RateLimiter to control the rate in the consumer listener, but this can easily lead to consume-timeouts. It is best to provide this function within the SDK.

Describe the Solution You'd Like

Developers can set following configurations to control consumption rate:

enableConsumeRateLimit=true
// 10 qps/s for single consumer
consumptionRate=10

Describe Alternatives You've Considered

Use RateLimiter to control rate in the messageListener

RateLimiter rateLimiter = RateLimiter.create(10);
consumer.registerMessageListener(new MessageListenerConcurrently() {
    @Override
    public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
        rateLimiter.acquire();
        System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
        return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
    }
});

Additional Context

20240617-112036

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant