Datasets documentation

了解你的数据集

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.2.0).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

了解你的数据集

数据集对象有两种类型:常规的 Dataset 和 ✨ IterableDataset ✨。Dataset 提供对行的快速随机访问,并支持内存映射,因此即使加载大型数据集,也只会占用相对少量的设备内存。但对于那些大到无法完全放入磁盘或内存中的超大数据集,IterableDataset 允许你在数据集完全下载之前就开始访问和使用它!

本教程将向您展示如何加载并访问 Dataset 和 IterableDataset。

数据集

加载数据集拆分时,你将获得一个 Dataset 对象。你可以对 Dataset 对象执行许多操作,因此学习如何操作和交互其中存储的数据非常重要。

本教程使用 rotten_tomatoes 数据集,但您可以自由加载任何想要的数据集并跟随操作!

>>> from datasets import load_dataset

>>> dataset = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")

索引

Dataset 包含数据列,每一列可以是不同类型的数据。index(即轴标签)用于访问数据集中的样本。例如,按行索引会返回数据集中一个样本的字典:

# Get the first row in the dataset
>>> dataset[0]
{'label': 1,
 'text': 'the rock is destined to be the 21st century\'s new " conan " and that he\'s going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}

使用 - 运算符从数据集末尾开始:

# Get the last row in the dataset
>>> dataset[-1]
{'label': 0,
 'text': 'things really get weird , though not particularly scary : the movie is all portent and no content .'}

按列名索引会返回该列中所有值的列表:

>>> dataset["text"]
['the rock is destined to be the 21st century\'s new " conan " and that he\'s going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .',
 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson\'s expanded vision of j . r . r . tolkien\'s middle-earth .',
 'effective but too-tepid biopic',
 ...,
 'things really get weird , though not particularly scary : the movie is all portent and no content .']

您可以结合行名和列名索引来返回特定位置的值:

>>> dataset[0]["text"]
'the rock is destined to be the 21st century\'s new " conan " and that he\'s going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'

索引顺序无关紧要。首先按列名进行索引会返回一个 Column 对象,你可以像往常一样使用行索引对其进行索引:

>>> import time

>>> start_time = time.time()
>>> text = dataset[0]["text"]
>>> end_time = time.time()
>>> print(f"Elapsed time: {end_time - start_time:.4f} seconds")
Elapsed time: 0.0031 seconds

>>> start_time = time.time()
>>> text = dataset["text"][0]
>>> end_time = time.time()
>>> print(f"Elapsed time: {end_time - start_time:.4f} seconds")
Elapsed time: 0.0042 seconds

切片

切片操作会返回数据集的一个切片(即子集),这在需要同时查看多行数据时非常有用。要对数据集进行切片,请使用 : 运算符来指定位置范围。

# Get the first three rows
>>> dataset[:3]
{'label': [1, 1, 1],
 'text': ['the rock is destined to be the 21st century\'s new " conan " and that he\'s going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .',
  'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson\'s expanded vision of j . r . r . tolkien\'s middle-earth .',
  'effective but too-tepid biopic']}

# Get rows between three and six
>>> dataset[3:6]
{'label': [1, 1, 1],
 'text': ['if you sometimes like to go to the movies to have fun , wasabi is a good place to start .',
  "emerges as something rare , an issue movie that's so honest and keenly observed that it doesn't feel like one .",
  'the film provides some great insight into the neurotic mindset of all comics -- even those who have reached the absolute top of the game .']}

IterableDataset

当您在 load_dataset() 中将 streaming 参数设置为 True 时,会加载一个 IterableDataset:

>>> from datasets import load_dataset

>>> iterable_dataset = load_dataset("ethz/food101", split="train", streaming=True)
>>> for example in iterable_dataset:
...     print(example)
...     break
{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=384x512 at 0x7F0681F5C520>, 'label': 6}

您还可以从现有的Dataset创建一个IterableDataset,但由于数据集是从本地文件流式传输的,因此比流式模式更快:

>>> from datasets import load_dataset

>>> dataset = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train")
>>> iterable_dataset = dataset.to_iterable_dataset()

IterableDataset 会逐个示例地渐进式遍历数据集,因此您无需等待整个数据集下载完成即可开始使用。可以想象,这对于需要立即使用的大型数据集来说非常实用!

索引

IterableDataset 的行为与常规的 Dataset 不同。在 IterableDataset 中,您无法随机访问示例。相反,您应该遍历其元素,例如通过调用 next(iter()) 或使用 for 循环来从 IterableDataset 中返回下一个项目:

>>> next(iter(iterable_dataset))
{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=384x512 at 0x7F0681F59B50>,
 'label': 6}

>>> for example in iterable_dataset:
...     print(example)
...     break
{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=384x512 at 0x7F7479DE82B0>, 'label': 6}

但 IterableDataset 支持列索引,返回该列值的可迭代对象:

>>> next(iter(iterable_dataset["label"]))
6

创建子集

您可以使用 IterableDataset.take() 返回包含特定数量示例的数据集子集:

# Get first three examples
>>> list(iterable_dataset.take(3))
[{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=384x512 at 0x7F7479DEE9D0>,
  'label': 6},
 {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=512x512 at 0x7F7479DE8190>,
  'label': 6},
 {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=512x383 at 0x7F7479DE8310>,
  'label': 6}]

但与切片不同,IterableDataset.take()会创建一个新的IterableDataset。

后续步骤

想了解这两类数据集之间的更多差异?请参阅概念指南Dataset 与 IterableDataset 之间的差异。

要更深入地了解这些数据集类型,请查看 Process 指南,学习如何预处理 Dataset,或查看 Stream 指南,学习如何预处理 IterableDataset。

Update on GitHub