-
Notifications
You must be signed in to change notification settings - Fork 293
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Limit alarm handler walltime to 15minutes #1622
Conversation
83fa87a
to
2d5ce68
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like this is tested upstream, nice.
How risky is this change? Would it make sense to put it behind an autogate to roll it out independent of a release?
Otherwise LGTM
1932198
to
8d3d366
Compare
Updated PR with new expected behaviour where alarms will be deleted after 1 run + 6 retries.
|
8d3d366
to
71ea6db
Compare
Updated PR with timeout as |
2a7afc0
to
dd52149
Compare
Updated PR:
|
dd52149
to
4e03280
Compare
4e03280
to
ddbde5c
Compare
6d13fe7
to
2f914a8
Compare
2f914a8
to
3f6584b
Compare
This PR adds a wall time timeout for alarm handlers.
When the timeout for an alarm it hit, we cancel deferred alarm deletion and abort the currently running handler.
The alarm timeouts are counted against limits, so they get deleted once the maximum retry limit is reached.