HuggingFace datasets

A guidance of usage for Hugging Face datasets.

Install

Start by installing 🤗 Datasets:

pip install datasets

🤗 Datasets also support audio and image data formats:

pip install datasets[audio]
pip install datasets[vision]

Load Dataset

Import load_dataset:

from datasets import load_dataset

Load remote dataset:

dataset = load_dataset("glue", "mrpc", split="train")

Load local CSV files:

dataset1 = load_dataset('csv', data_files='data.csv')
dataset2 = load_dataset('csv', data_files=['train.csv', 'test.csv'])
dataset3 = load_dataset('csv', data_files='/path/to/directory/*.csv')

Load local JSON files:

dataset = load_dataset('json', data_files='data.json')

Load local TXT files:

dataset = load_dataset('text', data_files='data.txt')

Load from disk

The dataset downloaded to the local machine via the 🤗 datasets library can be loaded using the load_from_disk() function. This function allows you to load datasets that have been previously cached or downloaded locally, without the need to fetch them again from the 🤗 database.

from datasets import load_from_disk

local_dataset = load_from_disk("./local_imdb_dataset")

print(local_dataset)