-
Notifications
You must be signed in to change notification settings - Fork 24
-
Hey @kylebarron.
I am the main author of LitData: Lightning-AI/litData and we have lot of GeoSpatial users.
I am quite curious to see how objstore will behave with PyTorch DataLoader.
We have tried to build a rust backend to pull the data (our own or tried hf_transfer from Hugging Face).
It was faster in raw benchmark outside of PyTorch DataLoader but it became 10-30% slower otherwise. The CPU usage was pretty chaotic using a rust backend and constantly at 100% with boto3.
I would be pretty keen to connect and see if there is room for collaboration. I still believe that we could make a rush backend works well.
Best regards,
Thomas Chaton.
Beta Was this translation helpful? Give feedback.
All reactions
Replies: 1 comment 1 reply
-
I haven't used PyTorch and I'm not familiar with its DataLoader. Is it async-compatible? The greatest performance improvements from obstore come when using it with high concurrency in async Python code.
Beta Was this translation helpful? Give feedback.
All reactions
-
Obstore is now being used as part of LitData - Lightning-AI/litData#652
Beta Was this translation helpful? Give feedback.