Caching s3 datasets

[avterekhov]

Is it possible to have a cache of the Hub dataset stored on s3?

If I create a dataset on s3 and access it, my access time is about 100 longer than if I create a local dataset. Changing storage_cache parameter does not seem to have any impact.

More details on the experiments:

1. I create a dataset on s3 composed of 20 large images (100MP each) with the chunk size of 1x64x64x3. I flush the dataset.
2. I create a loop which accesses random 1x512x512x3 portions of that images and compute the average access time. I expect the average access time to decrease over time, however that is not what I observe.
3. I repeat the experiment of (2), but limiting the range of random portions I access to a 1000x1000 area. I expect to see the average access time to drop over time, however I do no see this (more 100,000 access cycles).
4. I repeat (3) setting storage_cache to a large number (64GB), however I see no reduction in the access time.

[dhiganthrao]

Greetings, @avterekhov, and welcome to Hub! Apologies for the delayed reply.

We've now released Hub 2.0, a faster, easy-to-use, and more performant version of Hub! You can access the docs here, and access some small tutorials to get started with 2.0 here! You can also let me know if you run into any issues, I'll be happy to help!
If you haven't, please consider joining our Community Slack to receive further updates.