Investigate cpu usage in training

[SarahAlidoost]

From #27, job on small subset used only ~37% of the available CPU resources, the task may not scale efficiently to 256 cores, room for optimization and investigation how the model uses cpus.

[SarahAlidoost]

as suggested here, we can investigate whether to use Ray library.

[rogerkuou]

New experiments in a bigger subset dataset:

JobID|JobName|Partition|AllocCPUS|Elapsed|TotalCPU|MaxRSS|State|ExitCode
24438134|eso4clima|compute|256|02:00:09|6-04:57:47||COMPLETED|0:0
24438134.batch|batch||256|02:00:09|6-04:57:47|21279432K|COMPLETED|0:0
24438134.extern|extern||256|02:00:09|00:00.001|3752K|COMPLETED|0:0

This results in roughly (6×86400+4×3600+57×60+47 ) / (2×3600×256+9)=30% . Indeed we can do further investigation.