425 questions
- Bountied 0
- Unanswered
- Frequent
- Score
- Trending
- Week
- Month
- Unanswered (my tags)
Score of 0
0 answers
50 views
torch dataloader next-method when using multiple workers
I have a Dataset that is based on IterableDataSet, looking like that
class MyDataSet(torch.utils.data.IterableDataset):
def __init__(self):
# doing init stuff here
def __iter__(self):
...
- reputation score 1597
Score of 0
1 answer
177 views
Best method to create generator for TensorFlow with list of array inputs
I am using TensorFlow/Keras to create a deep learning model. The network is built as follows:
inps = []
features = []
for i in range(number_windows):
inp = Input(shape=(window_length,), name=f&...
- reputation score 941
Score of 3
1 answer
225 views
TensorFlow data loader from generator error "Dataset had more than one element"
I am trying to implement a TensorFlow dataset from a Python generator because I am having problems with my model consuming memory, inevitably resulting in a OOM crash (see my question on that here). ...
- reputation score 941
Score of 0
0 answers
121 views
Why is my numpy-based custom data loader extremely slow and unstable when iterating over large tick data
I'm currently working on a model similar to DeepLOB, using high-frequency tick-level financial data. Due to the massive volume and the need to structure the data into time series format, it's ...
- reputation score 11
Score of 0
0 answers
383 views
In HotChocolate 15, how to use projection and data loader for many to many relationship?
I'm using Hotchocolate v15, EF Core. For 1:m relationship, I'm using dataloader and projection to just fetch selected fields for child items. It work as expected, for example: load products by branch.
...
- reputation score 55
Score of 1
1 answer
60 views
GraphQL DataLoader fails with 400 when batching many IDs
I'm running a GraphQL server using postgraphile and dataloader to batch and load data from a backend microservice. When the number of IDs passed to the loader grows large (~100+), the request fails ...
- reputation score 2002
Score of 0
1 answer
334 views
The FAISS indexing and the dataset indexing don't match
I'm trying to compute the recall after performing a HNSW search in FAISS. By recall, I mean the following metric:
Recall = TP / (TP + FN)
Where I consider an image as a True Positive (TP) if it ...
- reputation score 23
Score of 1
0 answers
159 views
How to Build a More Efficient DataLoader to Load Large Image Datasets?
I am trying to train a deep learning model on a very large image dataset. The model input requires a pair of images (A and B). Because my image sizes are quite large, I have resized each of them to a ...
- reputation score 199
Score of 0
1 answer
126 views
Why is my DataLoader process using up to 2.6GB of virtual memory, and is there any way to reduce it?
Why is my DataLoader process using up to 2.6GB of virtual memory, and is there any way to reduce it?
Each DataLoader process takes up 2.6GB of virtual memory, and 4 processes take up 10.4GB.
from ...
- reputation score 168
Score of 1
0 answers
31 views
Asynchronous parallel data loading with torch in R
I want train cnns on a big dataset via transfer learning using torch in R. Since my dataset is to big to be loaded all at once, I have to load each sample from the SSD in the dataloader. But loading ...
- reputation score 11
Score of 0
2 answers
288 views
TypeError: 'DataLoader' object is not subscriptable in SuperGradients Trainer
I've created DataLoader objects for my training and validation datasets, but when I try to pass them to the trainer.train() method, I get the following error:
Log summary:
TypeError: 'DataLoader' ...
- reputation score 741
Score of 1
2 answers
310 views
Setting random seed in Torch dataloader
I'm trying to get the torch dataloader to load the data under a specific sequence determined by the random seed 1. Here's my code:
import random
import torch.utils.data.dataset as Dataset
import torch....
- reputation score 109
Score of 0
1 answer
673 views
How to run torch dataloader in a sub-process of multiprocessing.Pool?
I want to inference model in multiprocessing, instead of use torch.distributed, how can I use multiprocessing.Pool?
I have to use num_workers=0 in subprocess to avoid error like "daemonic ...
- reputation score 31
Score of 1
1 answer
77 views
How to use tf.data.interleave() with tf.py_function
I am trying to build TF data pipeline with tf.data API. I have ~100k of npz files to load and each npz has key of ["input"] and ["output"]. Some preprocessing is needed before ...
- reputation score 11
Score of 0
1 answer
963 views
Is there a good way to BatchMapping or use Data Loaders in Spring GraphQL utilising non-exposed fields?
Consider the below minimal GQL schema:
type query {
appointments: [Appointment!]!
}
type Appointment {
id: ID!
job: Job!
}
type Company {
id: ID!
job: Job!
}
type Job {
id: ...
- reputation score 123