FileOpener

class torchdata.datapipes.iter.FileOpener(datapipe: Iterable[str], mode: str = 'r', encoding: Optional[str] = None, length: int = -1)

Given pathnames, opens files and yield pathname and file stream in a tuple (functional name: open_files).

Parameters:

• datapipe – Iterable datapipe that provides pathnames

• mode – An optional string that specifies the mode in which the file is opened by open(). It defaults to b which means open for reading in binary mode. Another option is to use t for text mode

• encoding – An optional string that specifies the encoding of the underlying file. It defaults to None to match the default encoding of open.

• length – Nominal length of the datapipe

Note

The opened file handles will be closed by Python’s GC periodically. Users can choose to close them explicitly.

Example

>>> from torchdata.datapipes.iter import FileLister, FileOpener, StreamReader
>>> dp = FileLister(root=".").filter(lambda fname: fname.endswith('.txt'))
>>> dp = FileOpener(dp)
>>> dp = StreamReader(dp)
>>> list(dp)
[('./abc.txt', 'abc')]