fix(server): pin locally-built swarm services to the build node - #4829
Open
ilysorc wants to merge 1 commit into
Open
fix(server): pin locally-built swarm services to the build node#4829ilysorc wants to merge 1 commit into
ilysorc wants to merge 1 commit into
Conversation
A locally built application image (git/dockerfile/buildpacks, no registry
configured) only exists on the node that built it. The swarm service is
created with no placement constraint, so on a multi-node swarm the
scheduler can place a task on another node that lacks the image. That task
is rejected with "No such image" while the previous task keeps running, so
the deployment reports success ("done") but the app is never updated.
Pin such services to the build node by adding a `node.id==<build node>`
placement constraint (resolved from the deploy connection's
`docker info` Swarm.NodeID), mirroring how bind mounts are already pinned.
Only applied for locally-built images and only when the user has not
configured their own placement; images pushed to a registry and external
(docker) images are unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & Why
While testing Dokploy on a multi-node Docker Swarm, I noticed that applications built locally (Git, Dockerfile, or Buildpacks without a registry) don't always update correctly after a redeploy.
The deployment finishes successfully and is marked as
done, but the application may continue serving the previous image.Root cause
The problem only appears when the image is built locally and never pushed to a registry.
In that case, the newly built image exists only on the build node. However, if the service has no placement constraints (
generateConfigContainerreturnsPlacement.Constraints: []when there are no mounts or user-defined placement rules), Swarm is free to schedule the replacement task on any node.During a redeploy,
ForceUpdatecorrectly creates a new task, but that task may land on another node that doesn't have the newly built image. Swarm rejects it withNo such image, while the existing task continues running. Since the old task never stops, the deployment still reports success even though nothing has actually changed.Reproduction (2-node swarm: manager + worker)
Inspecting the service shows that
Spec.TaskTemplate.Placementis simply{}, so Swarm is free to place the replacement task on the worker.Accessing the origin directly (bypassing any CDN) confirms that the previous build is still being served. The only reliable workaround is recreating the service (Stop → Deploy), which causes the task to be scheduled back onto the build node.
The fix
The solution is to automatically pin locally built application services to the build node by adding a
node.id==<build-node>placement constraint.The node ID is obtained from
docker info(Swarm.NodeID) using the samegetRemoteDocker(serverId)connection that builds the image and creates the service.This follows the same idea already used for bind mounts, which are pinned to the manager node, but instead pins the service to the actual build node. That makes it work correctly even when deployments are initiated from a worker.
The constraint is added only when:
sourceType !== "docker"),registry,buildRegistry, androllbackRegistryare all unset), andplacementSwarm.Registry-backed images, external Docker images, and database services are unaffected. If the user explicitly defines placement rules, those always take precedence.
Verification
The change was verified on a real two-node Swarm cluster.
With the placement constraint in place:
:latestimage.Additional validation:
packages/serverpassestsc --noEmitwith no errors.Added unit tests in
mechanizeDockerContainer.test.tscovering:biome checkpasses cleanly.Notes
docker.info().Swarm.NodeIDfrom the existing deployment connection. If the node ID cannot be resolved, deployment continues exactly as before without adding a placement constraint.