ding.data¶
ding.data
¶
Buffer
¶
Bases: ABC
Buffer is an abstraction of device storage, third-party services or data structures, For example, memory queue, sum-tree, redis, or di-store.
push(data, meta=None)
abstractmethod
¶
Overview
Push data and it's meta information in buffer.
Arguments:
- data (:obj:Any): The data which will be pushed into buffer.
- meta (:obj:dict): Meta information, e.g. priority, count, staleness.
Returns:
- buffered_data (:obj:BufferedData): The pushed data.
sample(size=None, indices=None, replace=False, sample_range=None, ignore_insufficient=False, groupby=None, unroll_len=None)
abstractmethod
¶
Overview
Sample data with length size.
Arguments:
- size (:obj:Optional[int]): The number of the data that will be sampled.
- indices (:obj:Optional[List[str]]): Sample with multiple indices.
- replace (:obj:bool): If use replace is true, you may receive duplicated data from the buffer.
- sample_range (:obj:slice): Sample range slice.
- ignore_insufficient (:obj:bool): If ignore_insufficient is true, sampling more than buffer size
with no repetition will not cause an exception.
- groupby (:obj:Optional[str]): Groupby key in meta, i.e. groupby="episode"
- unroll_len (:obj:Optional[int]): Number of consecutive frames within a group.
Returns:
- sample_data (:obj:Union[List[BufferedData], List[List[BufferedData]]]):
A list of data with length size, may be nested if groupby is set.
update(index, data=None, meta=None)
abstractmethod
¶
Overview
Update data and meta by index
Arguments:
- index (:obj:str): Index of data.
- data (:obj:any): Pure data.
- meta (:obj:dict): Meta information.
Returns:
- success (:obj:bool): Success or not, if data with the index not exist in buffer, return false.
delete(index)
abstractmethod
¶
Overview
Delete one data sample by index
Arguments:
- index (:obj:str): Index
save_data(file_name)
abstractmethod
¶
Overview
Save buffer data into a file.
Arguments:
- file_name (:obj:str): file name of buffer data
load_data(file_name)
abstractmethod
¶
Overview
Load buffer data from a file.
Arguments:
- file_name (:obj:str): file name of buffer data
get(idx)
abstractmethod
¶
Overview
Get item by subscript index
Arguments:
- idx (:obj:int): Subscript index
Returns:
- buffered_data (:obj:BufferedData): Item from buffer
use(func)
¶
Overview
Use algorithm middleware to modify the behavior of the buffer.
Every middleware should be a callable function, it will receive three argument parts, including:
1. The buffer instance, you can use this instance to visit every thing of the buffer, including the storage.
2. The functions called by the user, there are three methods named push , sample and clear , so you can use these function name to decide which action to choose.
3. The remaining arguments passed by the user to the original function, will be passed in *args .
Each middleware handler should return two parts of the value, including:
1. The first value is done (True or False), if done==True, the middleware chain will stop immediately, no more middleware will be executed during this execution
2. The remaining values, will be passed to the next middleware or the default function in the buffer.
Arguments:
- func (:obj:Callable): The middleware handler
Returns:
- buffer (:obj:Buffer): The instance self
view()
¶
Overview
A view is a new instance of buffer, with a deepcopy of every property except the storage. The storage is shared among all the buffer instances.
Returns:
- buffer (:obj:Buffer): The instance self
DequeBuffer
¶
Bases: Buffer
Overview
A buffer implementation based on the deque structure.
__init__(size, sliced=False)
¶
Overview
The initialization method of DequeBuffer.
Arguments:
- size (:obj:int): The maximum number of objects that the buffer can hold.
- sliced (:obj:bool): The flag whether slice data by unroll_len when sample by group
push(data, meta=None)
¶
Overview
The method that input the objects and the related meta information into the buffer.
Arguments:
- data (:obj:Any): The input object which can be in any format.
- meta (:obj:Optional[dict]): A dict that helps describe data, such as category, label, priority, etc. Default to None.
sample(size=None, indices=None, replace=False, sample_range=None, ignore_insufficient=False, groupby=None, unroll_len=None)
¶
Overview
The method that randomly sample data from the buffer or retrieve certain data by indices.
Arguments:
- size (:obj:Optional[int]): The number of objects to be obtained from the buffer.
If indices is not specified, the size is required to randomly sample the corresponding number of objects from the buffer.
- indices (:obj:Optional[List[str]]): Only used when you want to retrieve data by indices.
Default to None.
- replace (:obj:bool): As the sampling process is carried out one by one, this parameter determines whether the previous samples will be put back into the buffer for subsequent sampling. Default to False, it means that duplicate samples will not appear in one sample call.
- sample_range (:obj:Optional[slice]): The indices range to sample data. Default to None, it means no restrictions on the range of indices for the sampling process.
- ignore_insufficient (:obj:bool): whether throw ValueError if the sampled size is smaller than the required size. Default to False.
- groupby (:obj:Optional[str]): If this parameter is activated, the method will return a target size of object groups.
- unroll_len (:obj:Optional[int]): The unroll length of a trajectory, used only when the groupby is activated.
Returns:
- sampled_data (Union[List[BufferedData], List[List[BufferedData]]]): The sampling result.
update(index, data=None, meta=None)
¶
Overview
the method that update data and the related meta information with a certain index.
Arguments:
- data (:obj:Any): The data which is supposed to replace the old one. If you set it to None, nothing will happen to the old record.
- meta (:obj:Optional[dict]): The new dict which is supposed to merge with the old one.
delete(indices)
¶
Overview
The method that delete the data and related meta information by specific indices.
Arguments: - indices (Union[str, Iterable[str]]): Where the data to be cleared in the buffer.
count()
¶
Overview
The method that returns the current length of the buffer.
get(idx)
¶
Overview
The method that returns the BufferedData object by subscript idx (int).
get_by_index(index)
¶
Overview
The method that returns the BufferedData object given a specific index (str).
clear()
¶
Overview
The method that clear all data, indices, and the meta information in the buffer.
Full Source Code
../ding/data/__init__.py