-
-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
readiness probe #6403
Comments
Although Caddy calls https://github.com/caddyserver/caddy/blob/master/modules/caddyhttp/reverseproxy/admin.go Do neither of these help? |
implementing a custom hook for plugins to notify their readiness in the admin API might work, but it means that I need to, from my understanding
this seems like a nightmare, and I would have trouble doing it for caddyhttp since that's not my module. maybe there is a way that I am not seeing here.
this is exactly part of the problem. some apps start and I can't check them for readiness, hence me wanting to wait for every Start() for each module to finish running before sending any traffic to caddy. the only way I can really know if caddy is ready is if all Start() hooks have executed without error, which there is no endpoint to figure this info out that I can see. I basically want to use the same hook as systemd (done config loading), or the post startup tcp callback (after load), as the condition for marking caddy as ready for external programs. the least intrusive hack I have thought of so far is to run a sidecar, or to bundle a second process in my docker images, that receives the post startup tcp message from caddy and then starts serving readiness on a different port that we can health check. but this feels really bad. in our specific use case, we have a caddy app that is serving postgres over tcp. we originally used the admin api http endpoints to determine if we had started up but we realized doing that was wrong. we then added tcp probe to check on the postgres port, and it's mostly good, but there is time between the start of listening and actually ready to serve, and it seems things are trying to connect before ready has completely finished running, resulting in lost/failed connections. we could change our plugin a little bit to reduce this, but we need information from the listening socket post bind to finalize our handler, so it's a bit difficult. we could bind to a second tcp port at the end of start, and probe there, but that seems wrong overall I felt like wanting to wait for all Start() hooks to run before sending traffic to caddy is a pretty standard need (especially in the case of zero downtime redeploy), and so maybe it shouldn't be as difficult as either writing a set of custom modules, or writing your own health check endpoints for each app and configuring all of them. |
It's rather valuable when running on an orchestrator to be able to query an http endpoint to know when an app is both 'live' and 'ready'.
People have had this problem in the past, and have used workarounds such as: #3441 (comment)
however, I don't think that this workaround is perfect, because if I'm mounting multiple servers, I don't know if every
caddy.App
has finished itsStart()
, and so as a result, health could mistakenly be reported as true. I understand that I could health check my individual servers, and then configure one check per server - but now we are talking about more complicated hacks.along with this - i believe it's possible for modules to still be mutating after Start() has been called, depending on how the module is implemented.
caddy currently already has mechanism to notify an arbitrary system when it is ready - so i don't believe this sort of ask is out of scope for the project
caddy/cmd/commandfuncs.go
Lines 239 to 256 in fab6375
does it make sense mount an endpoint somewhere here https://github.com/caddyserver/caddy/blob/master/caddy.go#L559 like
GET /health/readiness
to the admin api so that one could use it to determine when to start sending traffic to an app?The text was updated successfully, but these errors were encountered: