# Question about streaming

**URL:** <https://discuss.huggingface.co/t/question-about-streaming/36951>\
**Category:** 🤗Datasets\
**Created:** [April 18, 2023, 8:18am UTC](https://discuss.huggingface.co/t/question-about-streaming/36951 "2023-04-18T08:18:43Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![zyh3826](https://avatars.discourse-cdn.com/v4/letter/z/49beb7/32.png) [@zyh3826](https://discuss.huggingface.co/u/zyh3826)\
**Post date:** [April 18, 2023, 8:18am UTC](https://discuss.huggingface.co/t/question-about-streaming/36951/1 "2023-04-18T08:18:43Z")

</div>

I have a huge audio dataset, I process this dataset using code like this:

```python
dataset = load_dataset(load_script_path, configuration, cache_dir=self.cache_dir, split=split)
dataset = dataset.cast_column('audio', Audio(sampling_rate=self.sample_rate))
dataset = dataset.map(self.prepare_dataset, remove_columns=dataset.column_names, num_proc=self.num_proc)
dataset.save_to_disk(save_path)

```

My question is when I use `save_to_disk` after `map`, is there a possible way to load this saved dataset streaming, because the time use of `map` is huge, thanks a lot.

---

<div class="post-metadata">

**Author:** ![zyh3826](https://avatars.discourse-cdn.com/v4/letter/z/49beb7/32.png) [@zyh3826](https://discuss.huggingface.co/u/zyh3826)\
**Post date:** [April 18, 2023, 8:24am UTC](https://discuss.huggingface.co/t/question-about-streaming/36951/2 "2023-04-18T08:24:39Z")

</div>

I also notice that `save_to_disk` split the dataset into many shards, so I think there maybe a way to load it streaming, but I don’t know how

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [April 19, 2023, 9:26am UTC](https://discuss.huggingface.co/t/question-about-streaming/36951/3 "2023-04-19T09:26:03Z")

</div>

You can use `load_from_disk` to load a saved dataset, but it doesn’t have the `streaming` argument (yet ?). But if you want to get an iterable dataset you can do

```python
idataset = load_from_disk(...).to_iterable_dataset()

```

---

<div class="post-metadata">

**Author:** ![zyh3826](https://avatars.discourse-cdn.com/v4/letter/z/49beb7/32.png) [@zyh3826](https://discuss.huggingface.co/u/zyh3826)\
**Post date:** [April 25, 2023, 8:50am UTC](https://discuss.huggingface.co/t/question-about-streaming/36951/4 "2023-04-25T08:50:22Z")

</div>

yes, I do it that way, thanks