> if you would rather not restart your services unless absolutely necessary (ie. you're handling long-lived TCP connections that cannot be handed off gracefully).
If you have no way to hand off your TCP connections gracefully, that is probably not a great position to be in because computers are unreliable and sometimes you need security updates. One approach I tried experimentally for a TCP server (but unfortunately did not have the opportunity to try in production) was to have a small server acting as the frontend that handled sessions and called into the real implementation which was essentially stateless. This small server could theoretically support self updating without dropping connections, if need be. (The main reason I was doing this was so that the rest of the stack could be dynamically-scaling based on Kubernetes, while this part remained relatively static.) I think anytime you have forced statefulness, it’s worth isolating as much as possible so it doesn’t constrain the design of the whole system.
(Even better, of course, is to just have a protocol where clients can gracefully handle draining and reconnect without side effects.)
> ACL changes, quota changes, etc
At a certain level I think these are runtime data and not configs. Aforementioned service also dealt with ACLs and configs and we were serving it through another service that persisted to a database layer. One cool thing about doing it as a service is that you can manage and scale it like any other service.
>I think that's the gist of the issue - sometimes it's difficult to tell which one is which :).
Absolutely... in fact, there's definitely stuff that I've had start as configuration and gradually turn into runtime data, and there are things I would call configuration that need to be dynamic at runtime too... I'm never 100% satisfied with the split.
If you have no way to hand off your TCP connections gracefully, that is probably not a great position to be in because computers are unreliable and sometimes you need security updates. One approach I tried experimentally for a TCP server (but unfortunately did not have the opportunity to try in production) was to have a small server acting as the frontend that handled sessions and called into the real implementation which was essentially stateless. This small server could theoretically support self updating without dropping connections, if need be. (The main reason I was doing this was so that the rest of the stack could be dynamically-scaling based on Kubernetes, while this part remained relatively static.) I think anytime you have forced statefulness, it’s worth isolating as much as possible so it doesn’t constrain the design of the whole system.
(Even better, of course, is to just have a protocol where clients can gracefully handle draining and reconnect without side effects.)
> ACL changes, quota changes, etc
At a certain level I think these are runtime data and not configs. Aforementioned service also dealt with ACLs and configs and we were serving it through another service that persisted to a database layer. One cool thing about doing it as a service is that you can manage and scale it like any other service.