ding.utils.data.base_dataloader¶
ding.utils.data.base_dataloader
¶
Full Source Code
../ding/utils/data/base_dataloader.py
1from typing import Optional, Callable, List, Any, Iterable 2import torch 3 4 5def example_get_data_fn() -> Any: 6 """ 7 Overview: 8 Get data from file or other middleware 9 .. note:: 10 staticmethod or static function, all the operation is on CPU 11 """ 12 # 1. read data from file or other middleware 13 # 2. data post-processing(e.g.: normalization, to tensor) 14 # 3. return data 15 pass 16 17 18class IDataLoader: 19 """ 20 Overview: 21 Base class of data loader 22 Interfaces: 23 ``__init__``, ``__next__``, ``__iter__``, ``_get_data``, ``close`` 24 """ 25 26 def __next__(self, batch_size: Optional[int] = None) -> torch.Tensor: 27 """ 28 Overview: 29 Get one batch data 30 Arguments: 31 - batch_size (:obj:`Optional[int]`): sometimes, batch_size is specified by each iteration, \ 32 if batch_size is None, use default batch_size value 33 """ 34 # get one batch train data 35 if batch_size is None: 36 batch_size = self._batch_size 37 data = self._get_data(batch_size) 38 return self._collate_fn(data) 39 40 def __iter__(self) -> Iterable: 41 """ 42 Overview: 43 Get data iterator 44 """ 45 46 return self 47 48 def _get_data(self, batch_size: Optional[int] = None) -> List[torch.Tensor]: 49 """ 50 Overview: 51 Get one batch data 52 Arguments: 53 - batch_size (:obj:`Optional[int]`): sometimes, batch_size is specified by each iteration, \ 54 if batch_size is None, use default batch_size value 55 """ 56 57 raise NotImplementedError 58 59 def close(self) -> None: 60 """ 61 Overview: 62 Close data loader 63 """ 64 65 # release resource 66 pass