PrototypeMultiProcessingReadingService

class torchdata.dataloader2.PrototypeMultiProcessingReadingService(num_workers: int = 0, multiprocessing_context=None, prefetch_worker: int = 10, prefetch_mainloop: int = 10)

PrototypeMultiProcessingReadingService that spawns multiple subprocesses to iterate the DataPipe graph. This ReadingService is still under prototype stage and will replace MultiProcessingReadingService eventually.

Parameters:

• num_workers (int, optional) – How many subprocesses to use for data loading. 0 will be replaced by InProcessReadingService in the future.

• multiprocessing_context (str, optional) – Multiprocessing starting method. If method is None then the default context is returned. Otherwise method should be ‘fork’, ‘spawn’.

finalize() → None

PrototypeMultiProcessingReadingService invalidate states & properly exits all subprocesses.

initialize(datapipe: Union[IterDataPipe, MapDataPipe]) → Union[IterDataPipe, MapDataPipe]

PrototypeMultiProcessingReadingService finds information about sharding, separates graph by multiple pieces and reconnects it using queues. creates subprocesses.

initialize_iteration() → None

ReadingService spins up service for an epoch. Called at the beginning of every time getting DataLoader2 iterator.