Automate Container Builds to NERSC Registry via GitHub Actions (Dev & Prod CI/CD)

[Copilot]

Description

Implements automated multi-arch container builds to NERSC registry with dev/prod separation enforced via image tags and K8s namespaces.

Workflows (4 automated pipelines):

• build-backend-dev.yml - Triggers on main push, tags :dev + :sha-<commit>
• build-frontend-dev.yml - Triggers on main push, tags :dev + :sha-<commit> (for NERSC-hosted dev frontend)
• build-backend-prod.yml - Triggers on releases/tags, semantic versioning (:vX.Y.Z, :vX.Y, :vX, :latest)
• build-frontend-prod.yml - Triggers on releases/tags, includes build-time VITE_API_BASE_URL injection

Architecture:

main push → :dev tag → NERSC Spin dev namespace (backend + frontend)
release   → :vX.Y.Z tag → NERSC Spin prod namespace (backend + frontend)

Both dev frontend and backend are hosted on NERSC Spin with automated builds from main. A Vercel-hosted dev frontend remains available for rapid prototyping.

Registry: registry.nersc.gov/e3sm/simboard/{backend,frontend}

All workflows use Docker Buildx for linux/amd64,linux/arm64 with layer caching.

All deployment instructions reference the Rancher UI for managing NERSC Spin services.

Documentation (streamlined under docs/cicd/):

• docs/cicd/README.md - Quick start guide with setup steps and common operations
• docs/cicd/DEPLOYMENT.md - Complete reference with workflows, Rancher UI deployment steps, and troubleshooting
• docs/README.md - Main documentation index

Simplification: Consolidated 6 documentation files (1694 lines) into 2 focused guides — 63% reduction — while preserving all essential information and eliminating redundancy.

Checklist

• Code follows project style guidelines
• Self-reviewed code
• No new warnings
• Tests added or updated (if needed)
• All tests pass (locally and CI/CD)
• Documentation/comments updated (if needed)
• Breaking change noted (if applicable)

Deployment Notes (if any)

Required before workflows activate:

1. Configure GitHub Secrets (repo admin): NERSC_REGISTRY_USERNAME, NERSC_REGISTRY_PASSWORD
2. Verify NERSC account has push access to e3sm/simboard/ namespace
3. Test with manual dispatch: Actions → Build Backend (Dev) or Build Frontend (Dev) → Run workflow

NERSC Spin deployment updates (via Rancher UI):

• Dev backend: Set image to registry.nersc.gov/e3sm/simboard/backend:dev with Pull Policy Always
• Dev frontend: Set image to registry.nersc.gov/e3sm/simboard/frontend:dev with Pull Policy Always
• Prod: Reference immutable versions (:vX.Y.Z) with Pull Policy IfNotPresent
• Redeploy workloads via Workloads → Deployments → ⋮ → Redeploy in Rancher

Documentation:

• Quick start: docs/cicd/README.md
• Complete reference: docs/cicd/DEPLOYMENT.md

Original prompt

---

This section details on the original issue you should resolve

<issue_title>[DevOps]: Automate Container Builds to NERSC Registry via GitHub Actions (Dev & Prod CI/CD)</issue_title>
<issue_description>## Task Details

Automate container image builds and publication to the NERSC container registry using GitHub Actions CI/CD, with a clear separation between development and production builds.

The development frontend is hosted on Vercel for rapid iteration. Both development and production backends, as well as the production frontend, are hosted on NERSC Spin.

Environment separation is enforced via image tags, Kubernetes namespaces, and deployment configuration, not by physical infrastructure.

Environment Overview

• Development
  • Frontend: Vercel
  • Backend: NERSC Spin (dev namespace)
  • Image source: builds from the main branch
• Production
  • Frontend: NERSC Spin (prod namespace)
  • Backend: NERSC Spin (prod namespace)
  • Image source: versioned GitHub Releases / tags

This setup ensures production deployments are immutable, traceable, and explicitly promoted, while main remains a continuously deployable integration branch for development and testing.

Key Design Decisions

• ✅ main branch → development backend images
• ✅ GitHub versioned releases (e.g. v0.3.0) → production frontend and backend images
• ✅ Dev and prod backends may run on the same NERSC Spin cluster, separated by namespaces and image tags
• ✅ Dev frontend is intentionally hosted on Vercel to optimize for rapid UX iteration

This follows standard DevOps best practices and supports the Vercel (dev frontend) + Spin (backend and prod frontend) split.

Steps to Complete

1. Define CI/CD Strategy

   • Confirm branching and release model
     • main → development backend images
     • GitHub tags/releases → production frontend and backend images
   • Define container image naming and tagging conventions
     (e.g. :dev, :sha-<commit>, :vX.Y.Z)
   • Identify which services are built via this pipeline (backend, prod frontend)

2. GitHub Actions Workflow

   • Create reusable GitHub Actions workflows for:
     • backend container builds
     • production frontend container builds
   • Configure workflow triggers:
     • push to main
     • GitHub Release creation or push of version tags
   • Add secure authentication to the NERSC container registry using GitHub Secrets
   • Push built images to the NERSC registry

3. Development Build Path (Backend)

   • On main, build the backend container image
   • Tag image with:
     • dev
     • commit SHA
   • Push development backend image to the registry
   • Deploy image to the dev backend on NERSC Spin
   • Verify compatibility with the Vercel-hosted development frontend

4. Production Build Path (Frontend + Backend)

   • On GitHub Release or version tag:
     • Build backend container image
     • Build production frontend container image
     • Tag images with semantic version (e.g. v0.3.0)
     • Push immutable production images to the registry
   • Reference versioned images explicitly in the production Spin deployments

5. Documentation

   • Document branch / tag → environment mapping
   • Document backend and frontend release and promotion process
   • Document how to cut and deploy a production release on Spin
   • Clarify that dev frontend runs on Vercel while prod frontend runs on Spin

6. Validation

   • Verify dev backend image builds on main
   • Verify prod images build only on tagged releases
   • Verify backend and frontend images are accessible from NERSC Spin
   • Verify dev and prod deployments reference the correct image tags
   • Confirm no accidental promotion of untagged builds to production

Additional Notes

• This approach intentionally avoids deploying main directly to production.
• Production releases are explicit, auditable, and reproducible.
• Hosting the dev frontend on Vercel optimizes developer velocity while preserving backend parity.
• Aligns with institutional expectations for controlled deployments on NERSC Spin.
  </issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes [DevOps]: Automate Container Builds to NERSC Registry via GitHub Actions (Dev & Prod CI/CD) #80

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[tomvothecoder]

@copilot I also have a development frontend hosted at NERSC. Replicate the Docker dev build process but for the frontend.

[Copilot]

@copilot I also have a development frontend hosted at NERSC. Replicate the Docker dev build process but for the frontend.

Added development frontend workflow in commit bf346ae.

The new .github/workflows/build-frontend-dev.yml workflow:

• Triggers on push to main (when frontend/** changes)
• Builds multi-arch image (linux/amd64, linux/arm64)
• Tags with :dev and :sha-<commit>
• Uses https://simboard-dev-api.e3sm.org as the default API URL
• Supports manual dispatch with custom API URL override

All documentation updated to include the dev frontend workflow:

• docs/DEPLOYMENT.md - Added frontend dev deployment section
• docs/CI_CD_REFERENCE.md - Added workflow details
• docs/QUICKSTART_CICD.md - Added frontend dev build steps
• docs/CI_CD_IMPLEMENTATION_SUMMARY.md - Updated architecture
• README.md - Updated automated builds list

The workflow follows the same pattern as the backend dev workflow for consistency.

[tomvothecoder]

@copilot Organize CI/CD docs under an appropriately named sub-directory.

[Copilot]

@copilot Organize CI/CD docs under an appropriately named sub-directory.

Organized CI/CD documentation under docs/cicd/ subdirectory in commit 425de82.

Changes:

• Created docs/cicd/ directory
• Moved and renamed documentation files with cleaner names:
  • QUICKSTART_CICD.md → cicd/QUICKSTART.md
  • DEPLOYMENT.md → cicd/DEPLOYMENT.md
  • GITHUB_SECRETS.md → cicd/GITHUB_SECRETS.md
  • CI_CD_REFERENCE.md → cicd/REFERENCE.md
  • CI_CD_IMPLEMENTATION_SUMMARY.md → cicd/IMPLEMENTATION_SUMMARY.md
• Added docs/cicd/README.md with quick access links
• Updated all internal cross-references in documentation
• Updated main README.md and docs/README.md with new paths

The documentation is now better organized with a clear structure under the cicd/ subdirectory.

[tomvothecoder]

@copilot Simplify the cicd docs, there's too many pages.

[Copilot]

@copilot Simplify the cicd docs, there's too many pages.

Simplified CI/CD documentation in commit 0aa8c91.

Consolidation:

• Reduced from 6 files (1694 lines) to 2 files (619 lines)
• 63% fewer lines, 67% fewer files

New structure:

• docs/cicd/README.md - Quick start guide with setup steps, overview, and common operations
• docs/cicd/DEPLOYMENT.md - Complete reference with all workflows, Kubernetes examples, and troubleshooting

Merged content:

• QUICKSTART.md → Integrated into README.md quick start section
• GITHUB_SECRETS.md → Consolidated into DEPLOYMENT.md secrets section
• REFERENCE.md → Workflow details merged into DEPLOYMENT.md
• IMPLEMENTATION_SUMMARY.md → Architecture details integrated throughout

All essential information preserved, redundancy eliminated, and navigation simplified.

[tomvothecoder]

@copilot Update DEPLOYMENT.md to remove kubectl references. I don't have access to those commands. I am using the Rancher UI service provided by NERSC Spin to manage services: https://rancher2.spin.nersc.gov/dashboard/home.

[Copilot]

@copilot Update DEPLOYMENT.md to remove kubectl references. I don't have access to those commands. I am using the Rancher UI service provided by NERSC Spin to manage services: https:/...

Replaced all kubectl references with Rancher UI instructions in commit 15e8e5d.

Changes in both docs/cicd/DEPLOYMENT.md and docs/cicd/README.md:

• Dev deployment: Redeploy via Workloads → Deployments → ⋮ → Redeploy in Rancher
• Prod deployment: Edit image tag via ⋮ → Edit Config in Rancher
• Rollback: Use ⋮ → Rollback in Rancher
• Verification: Check pod status and logs through the Rancher UI
• Troubleshooting: Updated "Dev Image Not Updating" to reference Rancher redeploy
• Removed the entire "Kubernetes Configuration" section (raw YAML manifests)

[Copilot]

Pull request overview

This PR implements automated CI/CD pipelines for building and publishing multi-architecture Docker images to the NERSC container registry, with clear separation between development and production environments enforced via image tags and Kubernetes namespaces.

Changes:

• Added 4 GitHub Actions workflows: dev and prod builds for both backend and frontend, triggered automatically on main branch pushes (dev) or GitHub releases (prod)
• Consolidated 6 documentation files into 2 streamlined guides (docs/cicd/README.md and docs/cicd/DEPLOYMENT.md), achieving 63% reduction while maintaining completeness
• Updated main README.md with automated deployment information and manual build examples using proper tags

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
.github/workflows/build-backend-dev.yml	Dev backend workflow: builds on main push, tags with :dev and :sha-, pushes to NERSC registry
.github/workflows/build-backend-prod.yml	Prod backend workflow: builds on release/tag, applies semantic versioning (:vX.Y.Z, :vX.Y, :vX, :latest)
.github/workflows/build-frontend-dev.yml	Dev frontend workflow: builds on main push with configurable VITE_API_BASE_URL, tags with :dev and :sha-
.github/workflows/build-frontend-prod.yml	Prod frontend workflow: builds on release/tag with production API URL, applies semantic versioning
docs/cicd/README.md	Quick start guide covering setup, workflows, release process, and deployment via Rancher UI
docs/cicd/DEPLOYMENT.md	Complete reference with detailed workflow descriptions, tagging strategy, deployment procedures, and troubleshooting
docs/README.md	New documentation index providing navigation to CI/CD guides
README.md	Updated deployment section with automated build information and manual build examples

Comments suppressed due to low confidence (3)

.github/workflows/build-backend-prod.yml:10

• The workflow_dispatch trigger for production builds may not generate semantic version tags when manually triggered from a non-tag ref. The docker/metadata-action with type=semver patterns requires a tag context to work properly. Consider either: (1) removing workflow_dispatch from production workflows since production should only build from explicit releases/tags, or (2) adding a fallback tag strategy for manual builds (e.g., type=raw with a manual- prefix).

  workflow_dispatch: {}

.github/workflows/build-frontend-prod.yml:15

• The workflow_dispatch trigger for production builds may not generate semantic version tags when manually triggered from a non-tag ref. The docker/metadata-action with type=semver patterns requires a tag context to work properly. Consider either: (1) removing workflow_dispatch from production workflows since production should only build from explicit releases/tags, or (2) adding a fallback tag strategy for manual builds.

  workflow_dispatch:
    inputs:
      vite_api_base_url:
        description: 'VITE_API_BASE_URL for the build'
        required: false
        default: 'https://simboard-api.e3sm.org'

README.md:298

• For consistency with the new CI/CD documentation, consider updating the Rancher Dashboard URL to use the more general path: https://rancher2.spin.nersc.gov/dashboard/home instead of the cluster-specific path. The current cluster-specific URL may become outdated or inaccessible if cluster configurations change.

- **Rancher Dashboard:** <https://rancher2.spin.nersc.gov/dashboard/c/c-fwj56/explorer/apps.deployment>