Use cases for BlueAPI Deployment PVC

[DominicOram]

As a support engineer wanting to manage my deployments I would like to do the following (in order of importance):

• Roll back to a known "good" PVC
• Have BlueAPI make a new PVC pulling a clean main (or a release tag) of my repos, the current PVC should be kept around
• Remotely see if my current PVC has uncommited changes
• Read the code in an old PVC without changing live
• Delete an old PVC
• See when a PVC was made/last used/last edited
• Delete based on some criteria e.g. all that haven't been used for X time

Acceptance Criteria

• All the above are documented on how to do them
• If they are hard or require more than 2 or 3 steps there are issues to make them easier

[fajinyuan]

Hi @DominicOram Thanks for the request. Could you please clarify in this ticket a bit more - what is this PVC for? is it for

• storage for user scripts?
• storage for visit experiment data?
• storage for workspace of application cache?
• or anything else?
  we may need to develop policy for each of these storage type if not already available.
  I am not yet a Cloud software developer, and not sure PVC can or cannot be rolled back or forth independent of pod.

[coretl]

@hyperrealist @gilesknap please can you think about whether we have the same requirements for FastCS / epics-containers dev PVCs

[DominicOram]

Could you please clarify in this ticket a bit more - what is this PVC for?

This is for the PVC that BlueAPI currently uses for deployments i.e. what mounts village-bluesky plan repositories and the python environment that runs them (so dodal + ophyd-async etc.). So I guess from your list "storage for workspace of application cache?" is the closest match.

storage for user scripts?
storage for visit experiment data?

I'm not sure what the D-II architecture is for storing these? At the moment I have assumed experiment data is stored in /dls/ixx/data as previously and I have sidestepped the question of user scripts. If DAQ core have a plan to also store these via PVC then I would probably have some similar requirements I could feedback but I suggest we cross the bridge when we come to it.

[fajinyuan]

You are correct. I was just told by the team the detector data and nexus files are currently not accessed via PVC, it still uses Posix file system directly. I am not aware of this decision, nor for what reasons this had been decision as such.
There still no discussion on how to handle storage for user scripts so far.
So what you asked here is for deployment source codes, thanks for confirming this.

[dan-fernandes]

Of the criteria, the most difficult will be (in no order):

1. Seeing when a PVC was last edited - I think this will be tricky in the general case, but more possible if scoped to the currently mounted PVC. We already have code (not shipped) to tag PVCs with when they were last used. Hopefully this largely fulfils the use case.
2. Auto-deletion - Again, we have an un-shipped feature to auto-delete on pre-defined criteria. We're unable to ship this because we don't have a good solution to alerting staff prior to deletion.

I think the rest are achievable. However the human interaction with them is all quite messy:

• Mounting a specific PVC / creating a new one which pulls from main will be done by changing a field in the Values.yaml file. It can also be overridden in ArgoCD (not recommended).
• Remotely viewing whether a PVC has uncommitted changes can done via the /python_environment endpoint, eg. this response body having edited P48:

{
  "installed_packages": [
    {
      "name": "dls-dodal",
      "version": "https://github.com/DiamondLightSource/dodal.git @main",
      "location": "/app/.venv/lib/python3.11/site-packages &&/workspace/dodal/src &&",
      "is_dirty": false,
      "source": "scratch"
    },
    {
      "name": "htss-rig-bluesky",
      "version": "https://github.com/DiamondLightSource/htss-rig-bluesky.git @foobar",
      "location": "/app/.venv/lib/python3.11/site-packages &&/workspace/htss-rig-bluesky/src &&",
      "is_dirty": true,
      "source": "scratch"
    }
  ],
  "scratch_enabled": true
}

• A PVC's last use is stamped into its Kubernetes annotations. This enabled PVC auto-deletion, but is not very human friendly. You can get the timestamp with a Kubectl command, or by looking at the beamlines' Kubernetes dashbaord or ArgoCD:

Feedback on whether this needs to be rationalised into a more singular experience would be great. Nonetheless they're technically possible.

[Alexj9837]

A link to the PR with the PVC auto deletion + time stamping logic #1514