||
DataLoader是Pytorch用来加载数据的一个类,其实就是一个迭代器,而迭代的数据从哪来?就需要用到DataSet了。
DataSet就是用来封装数据的类,主要用来对数据进行相关的自定义操作(比如图片的裁剪、标签的定义等),通过__getitem__
函数返回所需要的数据。
一般来说,需要重新定义一个新的类来继承DataSet类,然后再通过DataLoader来加载器数据。
继承DataSet类一般需要重写其中的__getitem__
函数,该函数用于返回第index个数据。其中常常也会重写__len__
函数,用于返回整个数据集的大小。
举例:
class MyData(Dataset):
def __init__(self,imag_path):
self.imag_path = imag_path
self.imag_path_list = os.listdir(imag_path)
def __getitem__(self, item):
imag_name = self.imag_path_list[item]
imag_item_path = os.path.join(self.imag_path,imag_name)
imag = Image.open(imag_item_path)
label = imag_name
return imag,label # 返回的第item项的图片以及对应的标签
def __len__(self):
return len(self.imag_path_list)
DataLoader一般通过torch.utils.data.DataLoader直接调用即可。DataLoader就是对DataSet中的数据进行迭代,通过__getiem__函数来获取DataSet对应数据集中的第item项数据,然后组合成batch,给程序进行训练。
关于DataLoader类参数详解见:http://blog.sciencenet.cn/blog-3428464-1253283.html
torch.utils.data.DataLoader
中的参数:dataset (Dataset) – dataset from which to load the data.
batch_size (int, optional) – how many samples per batch to load (default: 1).
shuffle (bool, optional) – set to True
to have the data reshuffled at every epoch (default: False).
sampler (Sampler, optional) – defines the strategy to draw samples from the dataset. If specified, shuffle
must be False.
batch_sampler (Sampler, optional) – like sampler, but returns a batch of indices at a time. Mutually exclusive with batch_size, shuffle, sampler, and drop_last.
num_workers (int, optional) – how many subprocesses to use for data loading. 0 means that the data will be loaded in the main process. (default: 0)
collate_fn (callable*, *optional) – merges a list of samples to form a mini-batch.
pin_memory (bool, optional) – If True
, the data loader will copy tensors into CUDA pinned memory before returning them.
drop_last (bool, optional) – set to True
to drop the last incomplete batch, if the dataset size is not divisible by the batch size. If False
and the size of dataset is not divisible by the batch size, then the last batch will be smaller. (default: False)
timeout (numeric, optional) – if positive, the timeout value for collecting a batch from workers. Should always be non-negative. (default: 0)
worker_init_fn (callable, optional) – If not None, this will be called on each worker subprocess with the worker id (an int in [0, num_workers - 1]
) as input, after seeding and before data loading. (default: None)
见博客:https://zhuanlan.zhihu.com/p/76893455
点滴分享,福泽你我!Add oil!
Archiver|手机版|科学网 ( 京ICP备07017567号-12 )
GMT+8, 2024-9-1 07:12
Powered by ScienceNet.cn
Copyright © 2007- 中国科学报社