Initial filewriter options analysis - #110
Conversation
There was a problem hiding this comment.
@mattclarke - sorry to tag you directly on a PR like this, it would great if you had time to look through this ess.md file and tell us where there are areas we're:
- Understating or overstating the amount of work required, or where we'd run strongly against architectural assumptions
- Missing mechanisms in the ESS filewriter which you think would be a better fit than what has been described here
- Overestimating or underestimating the extent of the existing support
Or anything else you feel we are misrepresenting about the ESS filewriter.
Tom-Willemsen
changed the title
| - **Unsupported**; it is unclear how this would be implemented. | ||
| - The ESS filewriter *does* use HDF5 SWMR support, so a separate/cooperating process may be able to take a 'snapshot' of an in-progress file, fix-up the metadata, and then emit that as an intermediate file. | ||
| - We could emit a new runStart/runStop pair, which would be picked up by another filewriter in the job-pool. This is an option because the ESS filewriter is inherently pooled and parallelizable by adding more writer processes. However, this would cause an expensive re-read of all of the data in the run from Kafka, and may cause our intermediate files to then be emitted so slowly that they are not useful to scientists. | ||
| - We could add full 'snapshotting' support to the filewriter, via a dedicated Kafka message, but this may be a rather invasive change. |
There was a problem hiding this comment.
this is a genuine question rather than a comment. What is the logic used to prompt the creation of an autosave file? is it a set number of frames, a time interval or something else?
There was a problem hiding this comment.
An autosave can be triggered after a number of frames, events, uAh, or a number of "dashboard polls" currently.
Triggering the autosaves could be built directly into the filewriter, or some external process would have to look at the counters from kafka_dae_diagnostics and decide to send this new dedicated "autosave now" Kafka message.
|
|
||
| This filewriter option proposes a brand-new development, specific to ISIS. | ||
|
|
||
| This would be implemented in Rust, as per other performance-sensitive components of the datastreaming pipeline. |
There was a problem hiding this comment.
i feel like we wrote this in an ADR somewhere, but now i can't find it....
| * Add support for the small number of missing datasets (for example total_counts). | ||
| - [Neutron histogram data](https://github.com/isisComputingGroup/datastreaming/issues/84) | ||
| * **Unsupported** in the filewriter itself | ||
| * The SuperMuSR pipeline generates histograms as a separate step from event-mode Nexus file writing. This is instead done by [MNeuEventLib](https://github.com/ISISMuon/MNeuEventLib) for SuperMuSR. |
There was a problem hiding this comment.
is this something we can use/adapt regardless of the choice we make for a filewriter?
There was a problem hiding this comment.
When I spoke to that team, they did say it "should work for Neutrons" - though I haven't done a detailed assessment.
I think the direction I'm leaning is that we probably need to support histogramming in the FW directly though - as some future instruments may have prohibitively large event files, and if their only reason to exist is to later be histogrammed anyway, then it seems much more efficient to histogram incrementally in the FW as events come in from Kafka.
Having histogramming in the FW directly also makes cases like "event + histogram data in the same file" easier to achieve.
There was a problem hiding this comment.
That project is now being more widely accepted into the MNeuData umbrella, so may be a suitable candidate for this. Anthony L or Alex R are the best people to ask about it.
3 participants
Initial analysis of three filewriter-implementation options: ESS, SuperMuSR, greenfield.
No opinion is formed yet on which option is most suitable; these documents just aim to describe what each option would look like.