Ensure that when scaling down, the Service does not invalidate route in Router if other replicase are still running #93

New issue

Closed

opened 2025-11-02 07:36:13 +00:00 by stanislav.hejny · 1 comment

stanislav.hejny commented

2025-11-02 07:36:13 +00:00

Member

what: REMOVE_ROUTE event is a feature to allow proper client notification if the service exists because it run out its allocated time. This is meant to support limited service, like it was envisioned in CleverData service, where a specific stack would be started based on the job description, and the stack would exist only for limited time, after which the route to it should be removed.

RCA: the Route Database does not contain tag identifying instance associated with the route. Thus REMOVE_ROUTE event invalidates route for all replicas, not only for the one that exited.
Solution: add SWARM_TASK_ID (a global instance ID) as tag on the route in Route Database, and upon REMOVE_ROUTE event invalidate only the route for this particular replica.

why: When testing scaleability of a service, after successful test that verified the replica count being reduced, the further interaction with the service is not possible, all requests fail with HTTP 503 - Service not available error, despite the fact that at least one replica of the service is still up and running. It appears that REMOVE_ROUTE event that the Service emits on shutdown, invalidates route even if there are still active replicas. what: REMOVE_ROUTE event is a feature to allow proper client notification if the service exists because it run out its allocated time. This is meant to support limited service, like it was envisioned in CleverData service, where a specific stack would be started based on the job description, and the stack would exist only for limited time, after which the route to it should be removed. RCA: the Route Database does not contain tag identifying instance associated with the route. Thus REMOVE_ROUTE event invalidates route for all replicas, not only for the one that exited. Solution: add SWARM_TASK_ID (a global instance ID) as tag on the route in Route Database, and upon REMOVE_ROUTE event invalidate only the route for this particular replica.

Rows
Columns