# Cannot stream custom dataset

**URL:** <https://discuss.huggingface.co/t/cannot-stream-custom-dataset/58038>\
**Category:** 🤗Datasets\
**Created:** [October 10, 2023, 1:38pm UTC](https://discuss.huggingface.co/t/cannot-stream-custom-dataset/58038 "2023-10-10T13:38:43Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![philipphager](https://avatars.discourse-cdn.com/v4/letter/p/97f17d/32.png) [@philipphager](https://discuss.huggingface.co/u/philipphager)\
**Post date:** [October 10, 2023, 1:38pm UTC](https://discuss.huggingface.co/t/cannot-stream-custom-dataset/58038/1 "2023-10-10T13:38:43Z")

</div>

Hey all,

I’m stuck writing a custom dataset for loading .safetensor files (I need to store a lot of vectors in fp16, and parquet does not support that). So, I would like to have the following:

I would like to download my entire custom dataset from huggingface and then use it as an IterableDataset to avoid processing millions of rows. I can get this to work with local files, and its super fast:

```auto
from datasets import IterableDataset
from safetensors.torch import load_file

def generate_sample(files):
    for file in files:
        dataset = load_file(file)
        n_rows = len(dataset["query_ids"])

        for i in range(n_rows):
            yield {
                "query_id": dataset["query_ids"][i],
                "features": dataset["features"][i],
                "label": dataset["labels"][i],
            }

if __name__ == " __main__":
    files = [f"part-0_split-{i}.safetensors" for i in range(10)]
    dataset = IterableDataset.from_generator(generate_sample, gen_kwargs={"files": files})
    loader = DataLoader(dataset, batch_size=16, num_workers=4)

    for batch in loader:
        pass

```

However, I would like to automatically load the dataset from my repository. It seems like the only way to make this work is to stream the dataset, however, this is for some weird reason super slow. And if I call load\_data and then immediately call .to\_iterable\_dataset(), it tries to preprocess/generate my entire training data, which is exactly what I want to avoid…

```auto
# This is very slow
dataset = load_data("philipphager/baidu-ultr-590k", split="train").to_iterable_dataset()

```

Can somebody help me out? My custom dataset builder is here: [https://huggingface.co/datasets/philipphager/baidu-ultr-590k/blob/main/baidu-ultr-590k.py](https://huggingface.co/datasets/philipphager/baidu-ultr-590k/blob/main/baidu-ultr-590k.py)

Also, when I call streaming, my \_generate\_examples does not get local file paths anymore, but URLs of files that should have been donwloaded before by the DownloadManager. This does not happen when not streaming…

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [October 11, 2023, 3:54pm UTC](https://discuss.huggingface.co/t/cannot-stream-custom-dataset/58038/2 "2023-10-11T15:54:25Z")

</div>

The script yields 1GB vectors, so streaming is expected to be slow. Yielding smaller vector slices (if possible) will improve the performance.

> Also, when I call streaming, my \_generate\_examples does not get local file paths anymore, but URLs of files that should have been donwloaded before by the DownloadManager. This does not happen when not streaming…

`dl_manager.download_and_extract` returns a (`fsspec`) URL to avoid downloading the data locally in the streaming mode.