Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide example use-case for datamon as a volume mount #200

Open
jakedsouza opened this issue Jun 11, 2019 · 1 comment
Open

Provide example use-case for datamon as a volume mount #200

jakedsouza opened this issue Jun 11, 2019 · 1 comment
Labels
k8s Issues around k8s integration new-use-case New feature or request usability Issues to improve use experience.

Comments

@jakedsouza
Copy link

Couple of useful examples could be

  • Mount as a shared k8s volume
  • Shared volume in a simple argo workflow
@jakedsouza jakedsouza added new-use-case New feature or request k8s Issues around k8s integration usability Issues to improve use experience. labels Jun 11, 2019
@ransomw1c
Copy link
Contributor

want to make clear that the main use case of datamon as a volume mount is not the appearance of datamon as a FUSE filesystem mounted via a k8s Shared Volume or whatever.

the volume mount is useful because it enables streaming downloads.

with the volume mount, i can open() and seek() files before read()ing (all pseudo syscalls here) and only download they exact bytes (give or take 1-2k) my data-science simulation or whatever requires.

we do not have a similar story for incremental uploads like the volume mount provides for streaming downloads. #255 is part of the way there, imo.


also, a "dirty secret" of the streaming downloads that i only noticed within the last several months is that google cloud charges for operations on buckets not just storage space of the buckets themselves. so if you want a disadvantage of the feature (i.e. streaming downloads – the feature is not volume mount. that's the reason i've disabled all the FUSE stuff in my fork), the disadvantage is cost, and the cost is due to the billing structure of the data operations. i haven't done the math.

i think streaming is super useful. in fact, i think we need the "accelerator pods" setup in #255 to have something like streaming, except for uploads. i just don't think streaming is going to be useful for any personal use-case that i foresee.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
k8s Issues around k8s integration new-use-case New feature or request usability Issues to improve use experience.
Development

No branches or pull requests

2 participants