Package org.apache.hadoop.fs
Class RawLocalFileSystem
java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.hadoop.fs.FileSystem
org.apache.hadoop.fs.RawLocalFileSystem
- All Implemented Interfaces:
Closeable,AutoCloseable,Configurable,BulkDeleteSource,org.apache.hadoop.fs.PathCapabilities,org.apache.hadoop.security.token.DelegationTokenIssuer
Implement the FileSystem API for the raw local filesystem.
-
Nested Class Summary
Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem
org.apache.hadoop.fs.FileSystem.DirectoryEntries, org.apache.hadoop.fs.FileSystem.DirListingIterator<T extends FileStatus>, org.apache.hadoop.fs.FileSystem.Statistics -
Field Summary
Fields inherited from class org.apache.hadoop.fs.FileSystem
DEFAULT_FS, FS_DEFAULT_NAME_KEY, LOG, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIXFields inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer
TOKEN_LOG -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionappend(Path f, int bufferSize, Progressable progress) Append to an existing file (optional operation).voidclose()Close this FileSystem instance.voidcompleteLocalOutput(Path fsWorkingFile, Path tmpLocalFile) Called when we're all done writing to the target.voidConcat existing files together.create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) Create an FSDataOutputStream at the indicated Path with write-progress reporting.create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) Create an FSDataOutputStream at the indicated Path with write-progress reporting.createNonRecursive(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) Opens an FSDataOutputStream at the indicated Path with write-progress reporting.createNonRecursive(Path f, FsPermission permission, EnumSet<CreateFlag> flags, int bufferSize, short replication, long blockSize, Progressable progress) Opens an FSDataOutputStream at the indicated Path with write-progress reporting.protected OutputStreamcreateOutputStream(Path f, boolean append) protected OutputStreamcreateOutputStreamWithMode(Path f, boolean append, FsPermission permission) protected PathHandlecreatePathHandle(FileStatus stat, org.apache.hadoop.fs.Options.HandleOpt... opts) Hook to implement support forPathHandleoperations.voidcreateSymlink(Path target, Path link, boolean createParent) booleanDelete the given path to a file or directory.booleanCheck if a path exists.Return a FileStatus representing the given path.Return a file status object that represents the path.Return the current user's home directory in this FileSystem.protected PathNote: with the new FileContext class, getWorkingDirectory() will be removed.Returns a status object describing the use and capacity of the filesystem.getUri()Returns a URI which identifies this FileSystem.Get the current working directory for the given FileSystemfinal booleanhandleEmptyDstDirectoryOnWindows(Path src, File srcFile, Path dst, File dstFile) booleanhasPathCapability(Path path, String capability) The base FileSystem implementation generally has no knowledge of the capabilities of actual implementations.voidinitialize(URI uri, Configuration conf) Initialize a FileSystem.listStatus(Path f) List the statuses of the files/directories in the given path if the path is a directory.booleanCreates the specified directory hierarchy.booleanmkdirs(Path f, FsPermission permission) Make the given file and all non-existent parents into directories.protected booleanprotected booleanmkOneDirWithMode(Path p, File p2f, FsPermission permission) voidmoveFromLocalFile(Path src, Path dst) The src file is on the local disk.open(PathHandle fd, int bufferSize) Open an FSDataInputStream matching the PathHandle instance.Opens an FSDataInputStream at the indicated Path.pathToFile(Path path) Convert a path to a File.booleanRenames Path src to Path dst.voidUse the command chown to set owner.voidsetPermission(Path p, FsPermission permission) Use the command chmod to set permission.voidSets thePath's last modified time and last access time to the given valid times.voidsetWorkingDirectory(Path newDir) Set the working directory to the given directory.startLocalOutput(Path fsOutputFile, Path tmpLocalFile) Returns a local file that the user can write output to.booleantoString()booleanTruncate the file in the indicated path to the indicated size.static voidMethods inherited from class org.apache.hadoop.fs.FileSystem
access, append, append, append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, checkPath, clearStatistics, closeAll, closeAllForUGI, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, createBulkDelete, createDataInputStreamBuilder, createDataInputStreamBuilder, createDataOutputStreamBuilder, createFile, createMultipartUploader, createNewFile, createNonRecursive, createSnapshot, createSnapshot, delete, deleteOnExit, deleteSnapshot, enableSymlinks, fixRelativePart, get, get, get, getAclStatus, getAdditionalTokenIssuers, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalServiceName, getCanonicalUri, getChildFileSystems, getContentSummary, getDefaultBlockSize, getDefaultBlockSize, getDefaultPort, getDefaultReplication, getDefaultReplication, getDefaultUri, getDelegationToken, getEnclosingRoot, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getLength, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getReplication, getScheme, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStoragePolicy, getStorageStatistics, getTrashRoot, getTrashRoots, getUsed, getUsed, getXAttr, getXAttrs, getXAttrs, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusBatch, listStatusIterator, listXAttrs, makeQualified, mkdirs, modifyAclEntries, moveFromLocalFile, moveToLocalFile, msync, newInstance, newInstance, newInstance, newInstanceLocal, open, open, openFile, openFile, openFileWithOptions, openFileWithOptions, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, satisfyStoragePolicy, setAcl, setDefaultUri, setDefaultUri, setQuota, setQuotaByStorageType, setReplication, setStoragePolicy, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, unsetStoragePolicyMethods inherited from class org.apache.hadoop.conf.Configured
getConf, setConfMethods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, waitMethods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer
addDelegationTokens
-
Constructor Details
-
RawLocalFileSystem
public RawLocalFileSystem()
-
-
Method Details
-
useStatIfAvailable
@VisibleForTesting public static void useStatIfAvailable() -
pathToFile
Convert a path to a File.- Parameters:
path- the path.- Returns:
- file.
-
getUri
Description copied from class:FileSystemReturns a URI which identifies this FileSystem.- Specified by:
getUriin classFileSystem- Returns:
- the URI of this filesystem.
-
initialize
Description copied from class:FileSystemInitialize a FileSystem. Called after the new FileSystem instance is constructed, and before it is ready for use. FileSystem implementations overriding this method MUST forward it to their superclass, though the order in which it is done, and whether to alter the configuration before the invocation are options of the subclass.- Overrides:
initializein classFileSystem- Parameters:
uri- a URI whose authority section names the host, port, etc. for this FileSystemconf- the configuration- Throws:
IOException- on any failure to initialize this instance.
-
open
Description copied from class:FileSystemOpens an FSDataInputStream at the indicated Path.- Specified by:
openin classFileSystem- Parameters:
f- the file name to openbufferSize- the size of the buffer to be used.- Returns:
- input stream.
- Throws:
IOException- IO failure
-
open
Description copied from class:FileSystemOpen an FSDataInputStream matching the PathHandle instance. The implementation may encode metadata in PathHandle to address the resource directly and verify that the resource referenced satisfies constraints specified at its construciton.- Overrides:
openin classFileSystem- Parameters:
fd- PathHandle object returned by the FS authority.bufferSize- the size of the buffer to use- Returns:
- input stream.
- Throws:
InvalidPathHandleException- IfPathHandleconstraints are not satisfiedIOException- IO failure
-
append
Description copied from class:FileSystemAppend to an existing file (optional operation).- Specified by:
appendin classFileSystem- Parameters:
f- the existing file to be appended.bufferSize- the size of the buffer to be used.progress- for reporting progress if it is not null.- Returns:
- output stream.
- Throws:
IOException- IO failure
-
create
public FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException Description copied from class:FileSystemCreate an FSDataOutputStream at the indicated Path with write-progress reporting.- Overrides:
createin classFileSystem- Parameters:
f- the file name to openoverwrite- if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.bufferSize- the size of the buffer to be used.replication- required block replication for the file.blockSize- the size of the buffer to be used.progress- to report progress.- Returns:
- output stream.
- Throws:
IOException- IO failure
-
createOutputStream
- Throws:
IOException
-
createOutputStreamWithMode
protected OutputStream createOutputStreamWithMode(Path f, boolean append, FsPermission permission) throws IOException - Throws:
IOException
-
createNonRecursive
public FSDataOutputStream createNonRecursive(Path f, FsPermission permission, EnumSet<CreateFlag> flags, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException Description copied from class:FileSystemOpens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as create(), except fails if parent directory doesn't already exist.- Overrides:
createNonRecursivein classFileSystem- Parameters:
f- the file name to openpermission- file permissionflags-CreateFlags to use for this stream.bufferSize- the size of the buffer to be used.replication- required block replication for the file.blockSize- block sizeprogress- the progress reporter- Returns:
- output stream.
- Throws:
IOException- IO failure- See Also:
-
create
public FSDataOutputStream create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException Description copied from class:FileSystemCreate an FSDataOutputStream at the indicated Path with write-progress reporting.- Specified by:
createin classFileSystem- Parameters:
f- the file name to openpermission- file permissionoverwrite- if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.bufferSize- the size of the buffer to be used.replication- required block replication for the file.blockSize- block sizeprogress- the progress reporter- Returns:
- output stream.
- Throws:
IOException- IO failure- See Also:
-
createNonRecursive
public FSDataOutputStream createNonRecursive(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException Description copied from class:FileSystemOpens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as create(), except fails if parent directory doesn't already exist.- Overrides:
createNonRecursivein classFileSystem- Parameters:
f- the file name to openpermission- file permissionoverwrite- if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.bufferSize- the size of the buffer to be used.replication- required block replication for the file.blockSize- block sizeprogress- the progress reporter- Returns:
- output stream.
- Throws:
IOException- IO failure- See Also:
-
concat
Description copied from class:FileSystemConcat existing files together.- Overrides:
concatin classFileSystem- Parameters:
trg- the path to the target destination.psrcs- the paths to the sources to use for the concatenation.- Throws:
IOException- IO failure
-
rename
Description copied from class:FileSystemRenames Path src to Path dst.- Specified by:
renamein classFileSystem- Parameters:
src- path to be renameddst- new path after rename- Returns:
- true if rename is successful
- Throws:
IOException- on failure
-
handleEmptyDstDirectoryOnWindows
@VisibleForTesting public final boolean handleEmptyDstDirectoryOnWindows(Path src, File srcFile, Path dst, File dstFile) throws IOException - Throws:
IOException
-
truncate
Description copied from class:FileSystemTruncate the file in the indicated path to the indicated size.- Fails if path is a directory.
- Fails if path does not exist.
- Fails if path is not closed.
- Fails if new size is greater than current size.
- Overrides:
truncatein classFileSystem- Parameters:
f- The path to the file to be truncatednewLength- The size the file is to be truncated to- Returns:
trueif the file has been truncated to the desirednewLengthand is immediately available to be reused for write operations such asappend, orfalseif a background process of adjusting the length of the last block has been started, and clients should wait for it to complete before proceeding with further file updates.- Throws:
IOException- IO failure
-
delete
Delete the given path to a file or directory.- Specified by:
deletein classFileSystem- Parameters:
p- the path to deleterecursive- to delete sub-directories- Returns:
- true if the file or directory and all its contents were deleted
- Throws:
IOException- if p is non-empty and recursive is false
-
listStatus
List the statuses of the files/directories in the given path if the path is a directory.Does not guarantee to return the List of files/directories status in a sorted order.
Will not return null. Expect IOException upon access error. (Note: Returned list is not sorted in any given order, due to reliance on Java's
File.list()API.)- Specified by:
listStatusin classFileSystem- Parameters:
f- given path- Returns:
- the statuses of the files/directories in the given patch
- Throws:
FileNotFoundException- when the path does not existIOException- see specific implementation
-
exists
Description copied from class:FileSystemCheck if a path exists. It is highly discouraged to call this method back to back with otherFileSystem.getFileStatus(Path)calls, as this will involve multiple redundant RPC calls in HDFS.- Overrides:
existsin classFileSystem- Parameters:
f- source path- Returns:
- true if the path exists
- Throws:
IOException- IO failure
-
mkOneDir
- Throws:
IOException
-
mkOneDirWithMode
- Throws:
IOException
-
mkdirs
Creates the specified directory hierarchy. Does not treat existence as an error.- Overrides:
mkdirsin classFileSystem- Parameters:
f- path- Returns:
- true if the directory was created
- Throws:
IOException- IO failure
-
mkdirs
Description copied from class:FileSystemMake the given file and all non-existent parents into directories. Has roughly the semantics of Unix @{code mkdir -p}. Existence of the directory hierarchy is not an error.- Specified by:
mkdirsin classFileSystem- Parameters:
f- path to createpermission- to apply to f- Returns:
- if mkdir success true, not false.
- Throws:
IOException- IO failure
-
getHomeDirectory
Description copied from class:FileSystemReturn the current user's home directory in this FileSystem. The default implementation returns"/user/$USER/".- Overrides:
getHomeDirectoryin classFileSystem- Returns:
- the path.
-
setWorkingDirectory
Set the working directory to the given directory.- Specified by:
setWorkingDirectoryin classFileSystem- Parameters:
newDir- Path of new working directory
-
getWorkingDirectory
Description copied from class:FileSystemGet the current working directory for the given FileSystem- Specified by:
getWorkingDirectoryin classFileSystem- Returns:
- the directory pathname
-
getInitialWorkingDirectory
Description copied from class:FileSystemNote: with the new FileContext class, getWorkingDirectory() will be removed. The working directory is implemented in FileContext. Some FileSystems like LocalFileSystem have an initial workingDir that we use as the starting workingDir. For other file systems like HDFS there is no built in notion of an initial workingDir.- Overrides:
getInitialWorkingDirectoryin classFileSystem- Returns:
- if there is built in notion of workingDir then it is returned; else a null is returned.
-
getStatus
Description copied from class:FileSystemReturns a status object describing the use and capacity of the filesystem. If the filesystem has multiple partitions, the use and capacity of the partition pointed to by the specified path is reflected.- Overrides:
getStatusin classFileSystem- Parameters:
p- Path for which status should be obtained. null means the default partition.- Returns:
- a FsStatus object
- Throws:
IOException- see specific implementation
-
moveFromLocalFile
Description copied from class:FileSystemThe src file is on the local disk. Add it to the filesystem at the given dst name, removing the source afterwards.- Overrides:
moveFromLocalFilein classFileSystem- Parameters:
src- local pathdst- path- Throws:
IOException- IO failure
-
startLocalOutput
Description copied from class:FileSystemReturns a local file that the user can write output to. The caller provides both the eventual target name in this FileSystem and the local working file path. If this FileSystem is local, we write directly into the target. If the FileSystem is not local, we write into the tmp local area.- Overrides:
startLocalOutputin classFileSystem- Parameters:
fsOutputFile- path of output filetmpLocalFile- path of local tmp file- Returns:
- the path.
- Throws:
IOException- IO failure
-
completeLocalOutput
Description copied from class:FileSystemCalled when we're all done writing to the target. A local FS will do nothing, because we've written to exactly the right place. A remote FS will copy the contents of tmpLocalFile to the correct target at fsOutputFile.- Overrides:
completeLocalOutputin classFileSystem- Parameters:
fsWorkingFile- path of output filetmpLocalFile- path to local tmp file- Throws:
IOException- IO failure
-
close
Description copied from class:FileSystemClose this FileSystem instance. Will release any held locks, delete all files queued for deletion through calls toFileSystem.deleteOnExit(Path), and remove this FS instance from the cache, if cached. After this operation, the outcome of any method call on this FileSystem instance, or any input/output stream created by it is undefined.- Specified by:
closein interfaceAutoCloseable- Specified by:
closein interfaceCloseable- Overrides:
closein classFileSystem- Throws:
IOException- IO failure
-
toString
-
getFileStatus
Description copied from class:FileSystemReturn a file status object that represents the path.- Specified by:
getFileStatusin classFileSystem- Parameters:
f- The path we want information from- Returns:
- a FileStatus object
- Throws:
FileNotFoundException- when the path does not existIOException- see specific implementation
-
setOwner
Use the command chown to set owner.- Overrides:
setOwnerin classFileSystem- Parameters:
p- The pathusername- If it is null, the original username remains unchanged.groupname- If it is null, the original groupname remains unchanged.- Throws:
IOException- IO failure
-
setPermission
Use the command chmod to set permission.- Overrides:
setPermissionin classFileSystem- Parameters:
p- The pathpermission- permission- Throws:
IOException- IO failure
-
setTimes
Sets thePath's last modified time and last access time to the given valid times.- Overrides:
setTimesin classFileSystem- Parameters:
mtime- the modification time to set (only if no less than zero).atime- the access time to set (only if no less than zero).p- The path- Throws:
IOException- if setting the times fails.
-
createPathHandle
protected PathHandle createPathHandle(FileStatus stat, org.apache.hadoop.fs.Options.HandleOpt... opts) Hook to implement support forPathHandleoperations.- Overrides:
createPathHandlein classFileSystem- Parameters:
stat- Referent in the target FileSystemopts- Constraints that determine the validity of thePathHandlereference.- Returns:
- path handle.
-
supportsSymlinks
public boolean supportsSymlinks()Description copied from class:FileSystem- Overrides:
supportsSymlinksin classFileSystem- Returns:
- if support symlinkls true, not false.
-
createSymlink
Description copied from class:FileSystem- Overrides:
createSymlinkin classFileSystem- Parameters:
target- target path.link- link.createParent- create parent.- Throws:
AccessControlException- if access is denied.FileAlreadyExistsException- when the path does not exist.FileNotFoundException- when the path does not exist.ParentNotDirectoryException- if the parent path of dest is not a directory.UnsupportedFileSystemException- if there was no known implementation for the scheme.IOException- raised on errors performing I/O.
-
getFileLinkStatus
Return a FileStatus representing the given path. If the path refers to a symlink return a FileStatus representing the link rather than the object the link refers to.- Overrides:
getFileLinkStatusin classFileSystem- Parameters:
f- the path.- Returns:
- file status
- Throws:
AccessControlException- if access is denied.FileNotFoundException- when the path does not exist.IOException- raised on errors performing I/O.UnsupportedFileSystemException- if there was no known implementation for the scheme.
-
getLinkTarget
Description copied from class:FileSystem- Overrides:
getLinkTargetin classFileSystem- Parameters:
f- the path.- Returns:
- the path.
- Throws:
IOException- IO failure.
-
hasPathCapability
Description copied from class:FileSystemThe base FileSystem implementation generally has no knowledge of the capabilities of actual implementations. Unless it has a way to explicitly determine the capabilities, this method returns false. Probe for a specific capability under the given path. If the function returnstrue, this instance is explicitly declaring that the capability is available. If the function returnsfalse, it can mean one of:- The capability is not known.
- The capability is known but it is not supported.
- The capability is known but the filesystem does not know if it is supported under the supplied path.
Implementors:
PathCapabilitiesSupportcan be used to help implement this method.- Specified by:
hasPathCapabilityin interfaceorg.apache.hadoop.fs.PathCapabilities- Overrides:
hasPathCapabilityin classFileSystem- Parameters:
path- path to query the capability of.capability- non-null, non-empty string to query the path for support.- Returns:
- true if the capability is supported under that part of the FS.
- Throws:
IOException- this should not be raised, except on problems resolving paths or relaying the call.
-