Class RawLocalFileSystem

All Implemented Interfaces:
Closeable, AutoCloseable, Configurable, BulkDeleteSource, org.apache.hadoop.fs.PathCapabilities, org.apache.hadoop.security.token.DelegationTokenIssuer

@Public @Stable public class RawLocalFileSystem extends FileSystem
Implement the FileSystem API for the raw local filesystem.
  • Constructor Details

    • RawLocalFileSystem

      public RawLocalFileSystem()
  • Method Details

    • useStatIfAvailable

      @VisibleForTesting public static void useStatIfAvailable()
    • pathToFile

      public File pathToFile(Path path)
      Convert a path to a File.
      Parameters:
      path - the path.
      Returns:
      file.
    • getUri

      public URI getUri()
      Description copied from class: FileSystem
      Returns a URI which identifies this FileSystem.
      Specified by:
      getUri in class FileSystem
      Returns:
      the URI of this filesystem.
    • initialize

      public void initialize(URI uri, Configuration conf) throws IOException
      Description copied from class: FileSystem
      Initialize a FileSystem. Called after the new FileSystem instance is constructed, and before it is ready for use. FileSystem implementations overriding this method MUST forward it to their superclass, though the order in which it is done, and whether to alter the configuration before the invocation are options of the subclass.
      Overrides:
      initialize in class FileSystem
      Parameters:
      uri - a URI whose authority section names the host, port, etc. for this FileSystem
      conf - the configuration
      Throws:
      IOException - on any failure to initialize this instance.
    • open

      public FSDataInputStream open(Path f, int bufferSize) throws IOException
      Description copied from class: FileSystem
      Opens an FSDataInputStream at the indicated Path.
      Specified by:
      open in class FileSystem
      Parameters:
      f - the file name to open
      bufferSize - the size of the buffer to be used.
      Returns:
      input stream.
      Throws:
      IOException - IO failure
    • open

      public FSDataInputStream open(PathHandle fd, int bufferSize) throws IOException
      Description copied from class: FileSystem
      Open an FSDataInputStream matching the PathHandle instance. The implementation may encode metadata in PathHandle to address the resource directly and verify that the resource referenced satisfies constraints specified at its construciton.
      Overrides:
      open in class FileSystem
      Parameters:
      fd - PathHandle object returned by the FS authority.
      bufferSize - the size of the buffer to use
      Returns:
      input stream.
      Throws:
      InvalidPathHandleException - If PathHandle constraints are not satisfied
      IOException - IO failure
    • append

      public FSDataOutputStream append(Path f, int bufferSize, Progressable progress) throws IOException
      Description copied from class: FileSystem
      Append to an existing file (optional operation).
      Specified by:
      append in class FileSystem
      Parameters:
      f - the existing file to be appended.
      bufferSize - the size of the buffer to be used.
      progress - for reporting progress if it is not null.
      Returns:
      output stream.
      Throws:
      IOException - IO failure
    • create

      public FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
      Description copied from class: FileSystem
      Create an FSDataOutputStream at the indicated Path with write-progress reporting.
      Overrides:
      create in class FileSystem
      Parameters:
      f - the file name to open
      overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.
      bufferSize - the size of the buffer to be used.
      replication - required block replication for the file.
      blockSize - the size of the buffer to be used.
      progress - to report progress.
      Returns:
      output stream.
      Throws:
      IOException - IO failure
    • createOutputStream

      protected OutputStream createOutputStream(Path f, boolean append) throws IOException
      Throws:
      IOException
    • createOutputStreamWithMode

      protected OutputStream createOutputStreamWithMode(Path f, boolean append, FsPermission permission) throws IOException
      Throws:
      IOException
    • createNonRecursive

      public FSDataOutputStream createNonRecursive(Path f, FsPermission permission, EnumSet<CreateFlag> flags, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
      Description copied from class: FileSystem
      Opens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as create(), except fails if parent directory doesn't already exist.
      Overrides:
      createNonRecursive in class FileSystem
      Parameters:
      f - the file name to open
      permission - file permission
      flags - CreateFlags to use for this stream.
      bufferSize - the size of the buffer to be used.
      replication - required block replication for the file.
      blockSize - block size
      progress - the progress reporter
      Returns:
      output stream.
      Throws:
      IOException - IO failure
      See Also:
    • create

      public FSDataOutputStream create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
      Description copied from class: FileSystem
      Create an FSDataOutputStream at the indicated Path with write-progress reporting.
      Specified by:
      create in class FileSystem
      Parameters:
      f - the file name to open
      permission - file permission
      overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.
      bufferSize - the size of the buffer to be used.
      replication - required block replication for the file.
      blockSize - block size
      progress - the progress reporter
      Returns:
      output stream.
      Throws:
      IOException - IO failure
      See Also:
    • createNonRecursive

      public FSDataOutputStream createNonRecursive(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
      Description copied from class: FileSystem
      Opens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as create(), except fails if parent directory doesn't already exist.
      Overrides:
      createNonRecursive in class FileSystem
      Parameters:
      f - the file name to open
      permission - file permission
      overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.
      bufferSize - the size of the buffer to be used.
      replication - required block replication for the file.
      blockSize - block size
      progress - the progress reporter
      Returns:
      output stream.
      Throws:
      IOException - IO failure
      See Also:
    • concat

      public void concat(Path trg, Path[] psrcs) throws IOException
      Description copied from class: FileSystem
      Concat existing files together.
      Overrides:
      concat in class FileSystem
      Parameters:
      trg - the path to the target destination.
      psrcs - the paths to the sources to use for the concatenation.
      Throws:
      IOException - IO failure
    • rename

      public boolean rename(Path src, Path dst) throws IOException
      Description copied from class: FileSystem
      Renames Path src to Path dst.
      Specified by:
      rename in class FileSystem
      Parameters:
      src - path to be renamed
      dst - new path after rename
      Returns:
      true if rename is successful
      Throws:
      IOException - on failure
    • handleEmptyDstDirectoryOnWindows

      @VisibleForTesting public final boolean handleEmptyDstDirectoryOnWindows(Path src, File srcFile, Path dst, File dstFile) throws IOException
      Throws:
      IOException
    • truncate

      public boolean truncate(Path f, long newLength) throws IOException
      Description copied from class: FileSystem
      Truncate the file in the indicated path to the indicated size.
      • Fails if path is a directory.
      • Fails if path does not exist.
      • Fails if path is not closed.
      • Fails if new size is greater than current size.
      Overrides:
      truncate in class FileSystem
      Parameters:
      f - The path to the file to be truncated
      newLength - The size the file is to be truncated to
      Returns:
      true if the file has been truncated to the desired newLength and is immediately available to be reused for write operations such as append, or false if a background process of adjusting the length of the last block has been started, and clients should wait for it to complete before proceeding with further file updates.
      Throws:
      IOException - IO failure
    • delete

      public boolean delete(Path p, boolean recursive) throws IOException
      Delete the given path to a file or directory.
      Specified by:
      delete in class FileSystem
      Parameters:
      p - the path to delete
      recursive - to delete sub-directories
      Returns:
      true if the file or directory and all its contents were deleted
      Throws:
      IOException - if p is non-empty and recursive is false
    • listStatus

      public FileStatus[] listStatus(Path f) throws IOException
      List the statuses of the files/directories in the given path if the path is a directory.

      Does not guarantee to return the List of files/directories status in a sorted order.

      Will not return null. Expect IOException upon access error. (Note: Returned list is not sorted in any given order, due to reliance on Java's File.list() API.)

      Specified by:
      listStatus in class FileSystem
      Parameters:
      f - given path
      Returns:
      the statuses of the files/directories in the given patch
      Throws:
      FileNotFoundException - when the path does not exist
      IOException - see specific implementation
    • exists

      public boolean exists(Path f) throws IOException
      Description copied from class: FileSystem
      Check if a path exists. It is highly discouraged to call this method back to back with other FileSystem.getFileStatus(Path) calls, as this will involve multiple redundant RPC calls in HDFS.
      Overrides:
      exists in class FileSystem
      Parameters:
      f - source path
      Returns:
      true if the path exists
      Throws:
      IOException - IO failure
    • mkOneDir

      protected boolean mkOneDir(File p2f) throws IOException
      Throws:
      IOException
    • mkOneDirWithMode

      protected boolean mkOneDirWithMode(Path p, File p2f, FsPermission permission) throws IOException
      Throws:
      IOException
    • mkdirs

      public boolean mkdirs(Path f) throws IOException
      Creates the specified directory hierarchy. Does not treat existence as an error.
      Overrides:
      mkdirs in class FileSystem
      Parameters:
      f - path
      Returns:
      true if the directory was created
      Throws:
      IOException - IO failure
    • mkdirs

      public boolean mkdirs(Path f, FsPermission permission) throws IOException
      Description copied from class: FileSystem
      Make the given file and all non-existent parents into directories. Has roughly the semantics of Unix @{code mkdir -p}. Existence of the directory hierarchy is not an error.
      Specified by:
      mkdirs in class FileSystem
      Parameters:
      f - path to create
      permission - to apply to f
      Returns:
      if mkdir success true, not false.
      Throws:
      IOException - IO failure
    • getHomeDirectory

      public Path getHomeDirectory()
      Description copied from class: FileSystem
      Return the current user's home directory in this FileSystem. The default implementation returns "/user/$USER/".
      Overrides:
      getHomeDirectory in class FileSystem
      Returns:
      the path.
    • setWorkingDirectory

      public void setWorkingDirectory(Path newDir)
      Set the working directory to the given directory.
      Specified by:
      setWorkingDirectory in class FileSystem
      Parameters:
      newDir - Path of new working directory
    • getWorkingDirectory

      public Path getWorkingDirectory()
      Description copied from class: FileSystem
      Get the current working directory for the given FileSystem
      Specified by:
      getWorkingDirectory in class FileSystem
      Returns:
      the directory pathname
    • getInitialWorkingDirectory

      protected Path getInitialWorkingDirectory()
      Description copied from class: FileSystem
      Note: with the new FileContext class, getWorkingDirectory() will be removed. The working directory is implemented in FileContext. Some FileSystems like LocalFileSystem have an initial workingDir that we use as the starting workingDir. For other file systems like HDFS there is no built in notion of an initial workingDir.
      Overrides:
      getInitialWorkingDirectory in class FileSystem
      Returns:
      if there is built in notion of workingDir then it is returned; else a null is returned.
    • getStatus

      public FsStatus getStatus(Path p) throws IOException
      Description copied from class: FileSystem
      Returns a status object describing the use and capacity of the filesystem. If the filesystem has multiple partitions, the use and capacity of the partition pointed to by the specified path is reflected.
      Overrides:
      getStatus in class FileSystem
      Parameters:
      p - Path for which status should be obtained. null means the default partition.
      Returns:
      a FsStatus object
      Throws:
      IOException - see specific implementation
    • moveFromLocalFile

      public void moveFromLocalFile(Path src, Path dst) throws IOException
      Description copied from class: FileSystem
      The src file is on the local disk. Add it to the filesystem at the given dst name, removing the source afterwards.
      Overrides:
      moveFromLocalFile in class FileSystem
      Parameters:
      src - local path
      dst - path
      Throws:
      IOException - IO failure
    • startLocalOutput

      public Path startLocalOutput(Path fsOutputFile, Path tmpLocalFile) throws IOException
      Description copied from class: FileSystem
      Returns a local file that the user can write output to. The caller provides both the eventual target name in this FileSystem and the local working file path. If this FileSystem is local, we write directly into the target. If the FileSystem is not local, we write into the tmp local area.
      Overrides:
      startLocalOutput in class FileSystem
      Parameters:
      fsOutputFile - path of output file
      tmpLocalFile - path of local tmp file
      Returns:
      the path.
      Throws:
      IOException - IO failure
    • completeLocalOutput

      public void completeLocalOutput(Path fsWorkingFile, Path tmpLocalFile) throws IOException
      Description copied from class: FileSystem
      Called when we're all done writing to the target. A local FS will do nothing, because we've written to exactly the right place. A remote FS will copy the contents of tmpLocalFile to the correct target at fsOutputFile.
      Overrides:
      completeLocalOutput in class FileSystem
      Parameters:
      fsWorkingFile - path of output file
      tmpLocalFile - path to local tmp file
      Throws:
      IOException - IO failure
    • close

      public void close() throws IOException
      Description copied from class: FileSystem
      Close this FileSystem instance. Will release any held locks, delete all files queued for deletion through calls to FileSystem.deleteOnExit(Path), and remove this FS instance from the cache, if cached. After this operation, the outcome of any method call on this FileSystem instance, or any input/output stream created by it is undefined.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Overrides:
      close in class FileSystem
      Throws:
      IOException - IO failure
    • toString

      public String toString()
      Overrides:
      toString in class Object
    • getFileStatus

      public FileStatus getFileStatus(Path f) throws IOException
      Description copied from class: FileSystem
      Return a file status object that represents the path.
      Specified by:
      getFileStatus in class FileSystem
      Parameters:
      f - The path we want information from
      Returns:
      a FileStatus object
      Throws:
      FileNotFoundException - when the path does not exist
      IOException - see specific implementation
    • setOwner

      public void setOwner(Path p, String username, String groupname) throws IOException
      Use the command chown to set owner.
      Overrides:
      setOwner in class FileSystem
      Parameters:
      p - The path
      username - If it is null, the original username remains unchanged.
      groupname - If it is null, the original groupname remains unchanged.
      Throws:
      IOException - IO failure
    • setPermission

      public void setPermission(Path p, FsPermission permission) throws IOException
      Use the command chmod to set permission.
      Overrides:
      setPermission in class FileSystem
      Parameters:
      p - The path
      permission - permission
      Throws:
      IOException - IO failure
    • setTimes

      public void setTimes(Path p, long mtime, long atime) throws IOException
      Sets the Path's last modified time and last access time to the given valid times.
      Overrides:
      setTimes in class FileSystem
      Parameters:
      mtime - the modification time to set (only if no less than zero).
      atime - the access time to set (only if no less than zero).
      p - The path
      Throws:
      IOException - if setting the times fails.
    • createPathHandle

      protected PathHandle createPathHandle(FileStatus stat, org.apache.hadoop.fs.Options.HandleOpt... opts)
      Hook to implement support for PathHandle operations.
      Overrides:
      createPathHandle in class FileSystem
      Parameters:
      stat - Referent in the target FileSystem
      opts - Constraints that determine the validity of the PathHandle reference.
      Returns:
      path handle.
    • supportsSymlinks

      public boolean supportsSymlinks()
      Description copied from class: FileSystem
      Overrides:
      supportsSymlinks in class FileSystem
      Returns:
      if support symlinkls true, not false.
    • createSymlink

      public void createSymlink(Path target, Path link, boolean createParent) throws IOException
      Description copied from class: FileSystem
      Overrides:
      createSymlink in class FileSystem
      Parameters:
      target - target path.
      link - link.
      createParent - create parent.
      Throws:
      AccessControlException - if access is denied.
      FileAlreadyExistsException - when the path does not exist.
      FileNotFoundException - when the path does not exist.
      ParentNotDirectoryException - if the parent path of dest is not a directory.
      UnsupportedFileSystemException - if there was no known implementation for the scheme.
      IOException - raised on errors performing I/O.
    • getFileLinkStatus

      public FileStatus getFileLinkStatus(Path f) throws IOException
      Return a FileStatus representing the given path. If the path refers to a symlink return a FileStatus representing the link rather than the object the link refers to.
      Overrides:
      getFileLinkStatus in class FileSystem
      Parameters:
      f - the path.
      Returns:
      file status
      Throws:
      AccessControlException - if access is denied.
      FileNotFoundException - when the path does not exist.
      IOException - raised on errors performing I/O.
      UnsupportedFileSystemException - if there was no known implementation for the scheme.
    • getLinkTarget

      public Path getLinkTarget(Path f) throws IOException
      Description copied from class: FileSystem
      Overrides:
      getLinkTarget in class FileSystem
      Parameters:
      f - the path.
      Returns:
      the path.
      Throws:
      IOException - IO failure.
    • hasPathCapability

      public boolean hasPathCapability(Path path, String capability) throws IOException
      Description copied from class: FileSystem
      The base FileSystem implementation generally has no knowledge of the capabilities of actual implementations. Unless it has a way to explicitly determine the capabilities, this method returns false. Probe for a specific capability under the given path. If the function returns true, this instance is explicitly declaring that the capability is available. If the function returns false, it can mean one of:
      • The capability is not known.
      • The capability is known but it is not supported.
      • The capability is known but the filesystem does not know if it is supported under the supplied path.
      The core guarantee which a caller can rely on is: if the predicate returns true, then the specific operation/behavior can be expected to be supported. However a specific call may be rejected for permission reasons, the actual file/directory not being present, or some other failure during the attempted execution of the operation.

      Implementors: PathCapabilitiesSupport can be used to help implement this method.

      Specified by:
      hasPathCapability in interface org.apache.hadoop.fs.PathCapabilities
      Overrides:
      hasPathCapability in class FileSystem
      Parameters:
      path - path to query the capability of.
      capability - non-null, non-empty string to query the path for support.
      Returns:
      true if the capability is supported under that part of the FS.
      Throws:
      IOException - this should not be raised, except on problems resolving paths or relaying the call.