Fix scheduler bugs: heap invariant corruption and lost stop event at dispatch boundary

[Marenz]

This PR fixes two scheduler bugs when updating an existing dispatch.

1. Removing a queued event could corrupt the scheduler heap because the queue was not re-heapified afterward, which could lead to incorrect ordering of future events.
2. An update exactly at the stop boundary could remove the pending stop event without notifying the actor, causing the stop transition to be missed.

This PR restores the heap after removals, tracks which queued event was removed, and sends the missing stop notification when a boundary update unschedules the pending stop event.

Regression tests and release notes were added.

[llucax]

I have a few comments about structure mainly. Also a few extra minor comments:

1. No empty line between the last line and signed-off-by in the commit message:

   Fix: call heapify() after pop(idx) to restore the heap invariant.
   Signed-off-by: Mathias L. Baumann <mathias.baumann@frequenz.com>
   

2. The release notes update is mixed in the last commit, I guess it makes more sense to put it in its own commit.

3. The PR description also IMHO focus on the wrong stuff, it didn't help a lot understanding what was not working. Asking AI to create a better description that explains what was exactly the bug, and how it was fixed by the PR, it came up with:

   This PR fixes two scheduler bugs when updating an existing dispatch.

   1. Removing a queued event could corrupt the scheduler heap because the queue was not re-heapified afterward, which could lead to incorrect ordering of future events.
   2. An update exactly at the stop boundary could remove the pending stop event without notifying the actor, causing the stop transition to be missed.

   This PR restores the heap after removals, tracks which queued event was removed, and sends the missing stop notification when a boundary update unschedules the pending stop event.

   Regression tests and release notes were added.

PS: TIL: re-sifted == re-heapified

[llucax]

One more question. Could the boundary issue happen for updates happen at the start boundary instead of the stop boundary?

[Marenz]

Okay, I think I addressed all feedback now

[Marenz]

One more question. Could the boundary issue happen for updates happen at the start boundary instead of the stop boundary?

The lost-event bug only occurs at the stop boundary, not the start boundary.

At the start boundary, the update path already has a fallback notification path through _update_changed_running_state(). The key point is that started is one of the runtime_state_attributes, and started is derived from the dispatch window and the current time, not stored independently.

That means:

• just before the start boundary, old_dispatch.started == False
• once the boundary is crossed, dispatch.started == True

So even if _remove_scheduled() consumed the queued start event first, _update_changed_running_state() would still see the started transition and emit the running-state update.

At the stop boundary, the problematic case is different:

• _remove_scheduled() can consume the queued stop event
• but the concurrent update may not independently trigger _update_changed_running_state()
  In that case, nothing replaces the consumed stop event, so the stop notification is lost.

[llucax]

Awesome!

[llucax]

Wow, no merge queue for this repo?

[llucax]

On a second thought, I'm doubtful the merge queue really adds any value to us, I guess the try-merge and remove, go to the next in the queue is good for monorepos, that might have many many concurrent merges, but for us, that we hardly have any (maybe only when dependabot kicks in), it is starting to feel it just makes things slower and more expensive (testing twice), without much extra gain. But I guess this is something to discuss elsewhere...

[Marenz]

Shit. I targeted the wrong upstream branch. We are on v1.x.x

[Marenz]

And that's also why there is no merge queue.

[llucax]

Maybe remove the v0.x.x branch if we are not making more releases from it?

[Marenz]

Maybe remove the v0.x.x branch if we are not making more releases from it?

already done :P