deploy with docker compose; shed a bunch of cruft#1541
Merged
Conversation
jsnshrmn
changed the title
jsnshrmn
changed the title
jsnshrmn
force-pushed
the
Jsn.sherman/T402054
branch
from
b723572 to
3fd0a04
Compare
Member
Author
|
File renames got picked up as add/rm because there were too many text changes happening in flight at the same time. I split the rename out into a separate commit to shrink the final diff. |
jsnshrmn
force-pushed
the
Jsn.sherman/T402054
branch
from
3fd0a04 to
272630e
Compare
jsnshrmn
changed the title
suecarmol
approved these changes
Jul 8, 2026
suecarmol
left a comment
suecarmol
left a comment
Contributor
There was a problem hiding this comment.
Thank you for doing the heavy lifting of cleaning code. I have pushed the changes to staging. If everything works there, I will push the changes to production tomorrow!
Member
Author
Unfortunately, the staging deploy will involve throwing away staging. We'll need to start with a fresh vm and do the whole thing soup to nuts. |
Contributor
Why? I just pushed, and there don't seem to be any issues. |
The T402054 migration replaces the TWLight Cloud VPS host with a Debian Trixie instance running rootless docker compose instead of swarm, mirroring Hashtags. This commit lands the target compose shape directly, without a dual-run phase or compatibility overlays. Prod and staging overlays carry env_file: .env, secrets cleared via !reset null, MYSQL_ROOT_PASSWORD_FILE unset, DJANGO_DB_HOST overridden from the swarm-era tasks.<env>_db, and restart: unless-stopped. The migrate service moves onto the /usr/local/backup cinder mount alongside twlight. daemon.json at repo root supplies the rootless daemon config: data root on /usr/local/docker-data (cinder), journald log driver in non-blocking mode. Verbatim from Hashtags. On the VM the service account reads it via a symlink from ~/.config/docker/daemon.json to /srv/TWLight/daemon.json. template.env documents the .env the operator populates on the VM. Assisted-by: Claude Opus 4.7 Bug: T402054
Symlink target hasn't existed for years; nothing consumes it. Assisted-by: Claude Opus 4.7 Bug: T402054
tasks.<env>_db was a swarm-DNS workaround; plain compose resolves "db" directly. Drop it from conf/*.twlight.env and the override that papered over it. Assisted-by: Claude Opus 4.7 Bug: T402054
debian_swarm_deploy.sh (host bootstrap + swarm init), twlight_docker_deploy.sh (docker stack deploy target of the auto-deploy cron), and twlight_update_code.sh (pull + stack build) all speak to the swarm host the new VM replaces. No in-tree consumer. Assisted-by: Claude Opus 4.7 Bug: T402054
The compose overlays already load .env and conf/<env>.twlight.env
for twlight and migrate; the swarm file-secrets layer is redundant.
Drop the secrets: blocks and top-level mapping, delete
twlight_docker_secrets.sh and its source calls from mysqldump,
mysqlimport, and virtualenv_activate, and simplify
twlight_mysql_init.sh to read DJANGO_DB_* and MYSQL_ROOT_PASSWORD
from the environment the mariadb entrypoint already provides.
mariadb picks up MYSQL_ROOT_PASSWORD locally via an env_file added
to the db service in the override. The committed ./secrets/ tree
is gone; dev/CI inline equivalent values into
conf/{local,cicd}.twlight.env.
DKIM_PRIVATE_KEY stays a placeholder in dev/CI; the console email
backend never parses it. Untracked user-local files in secrets/
(MATOMO_*, WP_USERS.json) stay put.
Assisted-by: Claude Opus 4.7
Bug: T402054
Pure rename; content changes follow in the next commit. Split out so `git log --follow` traces file history through the move rather than dropping it at the delete/add boundary. Assisted-by: Claude Opus 4.7 Bug: T402054
TWLight predates containerization, when the venv was the isolation boundary; the container is now. Install python deps system-wide in a single-stage Dockerfile and drop the "source /venv/bin/activate" wrapper around every management command. env vars come from the compose env_file, WORKDIR gets us to /app, gunicorn is on PATH. virtualenv_activate.sh (venv activation + secrets sourcing + cd) and virtualenv_pip_update.sh (mutated a running venv; rebuild the image instead) are gone. The previously-renamed scripts shed their venv-activation preamble. Callers updated across bin/, conf/bashrc, the compose files, the cicd workflow, and README. Assisted-by: Claude Opus 4.7 Bug: T402054
twlight_deploy.sh only wrapped twlight_update_code.sh (already gone) with a failure email; nothing calls it. wsl_docker_activate.sh was WSL + Vagrant scaffolding, untouched since 2024 and never referenced in-tree. Assisted-by: Claude Opus 4.7 Bug: T402054
Four scripts wrapped management commands that crons.py already runs on schedule via django-cron. twlight_weekly.sh has no in-tree callers and no host cron on the new VM; it fronted send_coordinator_reminders.sh with a twlight_failure.sh fallback, and both are only reached through it. unwaitlist.sh was a manual override of proxy_waitlist_disable, which ProxyWaitlistDisableCronJob runs daily. Assisted-by: Claude Opus 4.7 Bug: T402054
The load-test credentials lived under ./secrets/ back when that was the swarm secrets home; retiring the tree left this file stranded. Move it next to locustfile.py, drop the ../../secrets/ traversal in the loader, and gitignore its new home. The file is user-local (never tracked); operators re-place it in tests/locust/ on existing checkouts. Assisted-by: Claude Opus 4.7 Bug: T402054
Pairs with the earlier virtualenv_ prefix drop; covers the perl lint script and its shunit test wrapper too. Internal references between renamed scripts (backup/mysqldump, restore/mysqlimport) and out-of-bin callers (Dockerfile, compose files, TWLight/crons.py, bin/test.sh) updated to match. Assisted-by: Claude Opus 4.7 Bug: T402054
deploy.sh replaces the retired twlight_docker_deploy.sh: pull the
branch_${env} tag, sync the checkout if newer, `docker compose up
-d`. Swarm's `docker stack deploy` swapped out; otherwise the same
shape.
conf/{production,staging}.crontab bring the schedule in-tree,
styled after externallinks. Operator installs at cutover with
`crontab conf/${env}.crontab` as the service user.
Compose overlays gain an explicit `name:` (local, staging,
production) so containers land at ${name}-twlight-1 rather than
the repo-dir default; keeps the crontab's `docker ps` filter
legible. Existing dev checkouts get dangling twlight-* containers
until docker prune catches up.
The crontab is the eventual home for what crons.py schedules via
django_cron; those rows migrate here when the dedicated crons
container lands.
Assisted-by: Claude Opus 4.7
Bug: T402054
Replacing the deprecated Bullseye Cloud VPS instances means standing up fresh Trixie VMs with rootless docker; automate that the way the Hashtags tool does. The Hashtags installer is a single script. Here it splits in two, because dockerd-rootless-setuptool.sh needs a real login session (systemd --user, XDG_RUNTIME_DIR) that cloud-init, running as root, can't provide. debian-trixie-init.sh does the root-level setup from the OpenStack user-data field; debian-trixie-post-init.sh installs rootless docker and enables lingering, run as the service account after first login. Two choices worth noting: - The userns sysctl is written to /usr/local/lib/sysctl.d, since the WMF puppet run wipes /etc/sysctl.d out from under it - Before starting the daemon, post-init requires the data-root and backup dirs to be writable, and in production to each sit on their own block device, so docker data and backups can't fill the root volume (T430155) Assisted-by: Claude Opus 4.8 Bug: T402054
The rootless VM provisioning stood up the stack but never installed
the crontab that drives django-cron, auto-deploy, and image pruning,
and the Trixie image doesn't ship cron at all. Wire both in, and
bring the crontab's docker calls up to date with the compose setup.
- init: add the cron package to the base install
- post-init: install the env's crontab (conf/${env}.crontab) once the
rootless daemon and lingering are set up
- crontabs: replace the docker ps container-name lookup with
docker compose exec -T twlight, pinning COMPOSE_FILE in the crontab
(cron doesn't source .bashrc, and the env overlays set name:, so a
bare compose call would resolve the wrong project)
- deploy.sh: re-install the crontab on each real deploy, so schedule
edits reach the running box instead of only the checkout
Assisted-by: Claude Opus 4.8
Bug: T402054
jsnshrmn
force-pushed
the
Jsn.sherman/T402054
branch
from
492e67e to
d44156a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Rationale
It's time to replace the underlying VM for this service.
This project has gone from bare metal venv, to vagrant + puppet, to docker compose, to docker swarm over the years. Quite a bit of cruft has built up in the codebase as we've done minimal moves between these. We've been running hashtags for a while with a rootless docker compose setup, and it greatly streamlines the setup versus docker swarm. We also have a new team configuration. So rather than continuing to live with a codebase with several dead layers of deployment abstraction, we're cleaning up as part of the new deployment.
Phabricator Ticket
T402054
How Has This Been Tested?
[x] docker compose up|exec|etc
[ ] redeploy staging host
Screenshots of your changes (if appropriate):
N/A
Types of changes
What types of changes does your code introduce? Add an
xin all the boxes that apply: