org.apache.hadoop.fs.azurebfs

Class AzureBlobFileSystemStore

java.lang.Object

org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.fs.azurebfs.services.ListingSupport

@InterfaceAudience.Public
 @InterfaceStability.Evolving
public class AzureBlobFileSystemStore
extends Object
implements Closeable, org.apache.hadoop.fs.azurebfs.services.ListingSupport

Provides the bridging logic between Hadoop's abstract filesystem and Azure Storage.

Constructor Summary

Constructors

Constructor and Description
AzureBlobFileSystemStore(org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.AzureBlobFileSystemStoreBuilder abfsStoreBuilder) FileSystem Store for AzureBlobFileSystem for Abfs operations.

Method Summary

Modifier and Type	Method and Description
void	access(Path path, FsAction mode, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
void	breakLease(Path path, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext) Break any current lease on an ABFS file.
void	close()
void	createDirectory(Path path, FsPermission permission, FsPermission umask, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
OutputStream	createFile(Path path, org.apache.hadoop.fs.FileSystem.Statistics statistics, boolean overwrite, FsPermission permission, FsPermission umask, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
void	createFilesystem(org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
void	delete(Path path, boolean recursive, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
void	deleteFilesystem(org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
static String	extractEtagHeader(org.apache.hadoop.fs.azurebfs.services.AbfsHttpOperation result) Get the etag header from a response, stripping any quotations.
org.apache.hadoop.fs.azurebfs.AbfsConfiguration	getAbfsConfiguration()
AclStatus	getAclStatus(Path path, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
org.apache.hadoop.fs.azurebfs.services.AbfsClient	getClient()
org.apache.hadoop.fs.azurebfs.services.AbfsClient	getClient(org.apache.hadoop.fs.azurebfs.constants.AbfsServiceType serviceType)
org.apache.hadoop.fs.azurebfs.services.AbfsClientHandler	getClientHandler()
FileStatus	getFileStatus(Path path, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
Hashtable<String,String>	getFilesystemProperties(org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
boolean	getIsNamespaceEnabled(org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext) Resolves namespace information of the filesystem from the state of isNamespaceEnabled.
Hashtable<String,String>	getPathStatus(Path path, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
String	getPrimaryGroup()
String	getRelativePath(Path path)
String	getUser()
boolean	isAppendBlobKey(String key) Checks if the given key in Azure Storage should be stored as a page blob instead of block blob.
boolean	isAtomicRenameKey(String key)
boolean	isInfiniteLeaseKey(String key)
String	listStatus(Path path, String startFrom, List<FileStatus> fileStatuses, boolean fetchAll, String continuation, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
FileStatus[]	listStatus(Path path, String startFrom, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
FileStatus[]	listStatus(Path path, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
void	modifyAclEntries(Path path, List<AclEntry> aclSpec, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
org.apache.hadoop.fs.azurebfs.services.AbfsInputStream	openFileForRead(Path path, org.apache.hadoop.fs.FileSystem.Statistics statistics, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
org.apache.hadoop.fs.azurebfs.services.AbfsInputStream	openFileForRead(Path path, Optional<org.apache.hadoop.fs.impl.OpenFileParameters> parameters, org.apache.hadoop.fs.FileSystem.Statistics statistics, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
OutputStream	openFileForWrite(Path path, org.apache.hadoop.fs.FileSystem.Statistics statistics, boolean overwrite, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
void	removeAcl(Path path, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
void	removeAclEntries(Path path, List<AclEntry> aclSpec, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
void	removeDefaultAcl(Path path, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
boolean	rename(Path source, Path destination, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext, String sourceEtag) Rename a file or directory.
void	setAcl(Path path, List<AclEntry> aclSpec, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
void	setFilesystemProperties(Hashtable<String,String> properties, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
void	setOwner(Path path, String owner, String group, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
void	setPathProperties(Path path, Hashtable<String,String> properties, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
void	setPermission(Path path, FsPermission permission, org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AzureBlobFileSystemStore

public AzureBlobFileSystemStore(org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.AzureBlobFileSystemStoreBuilder abfsStoreBuilder)
                         throws IOException

FileSystem Store for AzureBlobFileSystem for Abfs operations. Built using the AzureBlobFileSystemStore.AzureBlobFileSystemStoreBuilder with parameters required.

Parameters:

abfsStoreBuilder - Builder for AzureBlobFileSystemStore.

Throws:

IOException - Throw IOE in case of failure during constructing.

Method Detail

isAppendBlobKey

public boolean isAppendBlobKey(String key)

Checks if the given key in Azure Storage should be stored as a page blob instead of block blob.

Parameters:

key - The key to check.

Returns:

True if the key should be stored as a page blob, false otherwise.

getUser

public String getUser()

Returns:

local user name.

getPrimaryGroup

public String getPrimaryGroup()

Returns:

primary group that user belongs to.

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

getIsNamespaceEnabled

public boolean getIsNamespaceEnabled(org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                              throws AzureBlobFileSystemException

Resolves namespace information of the filesystem from the state of isNamespaceEnabled. if the state is UNKNOWN, it will be determined by making a GET_ACL request to the root of the filesystem. GET_ACL call is synchronized to ensure a single call is made to determine the namespace information in case multiple threads are calling this method at the same time. The resolution of namespace information would be stored back as state of isNamespaceEnabled.

Parameters:

tracingContext - tracing context

Returns:

true if namespace is enabled, false otherwise.

Throws:

AzureBlobFileSystemException - server errors.

getAbfsConfiguration

public org.apache.hadoop.fs.azurebfs.AbfsConfiguration getAbfsConfiguration()

getFilesystemProperties

public Hashtable<String,String> getFilesystemProperties(org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                                                 throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

setFilesystemProperties

public void setFilesystemProperties(Hashtable<String,String> properties,
                                    org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                             throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

getPathStatus

public Hashtable<String,String> getPathStatus(Path path,
                                              org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                                       throws IOException

Throws:

IOException

setPathProperties

public void setPathProperties(Path path,
                              Hashtable<String,String> properties,
                              org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                       throws IOException

Throws:

IOException

createFilesystem

public void createFilesystem(org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                      throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

deleteFilesystem

public void deleteFilesystem(org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                      throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

createFile

public OutputStream createFile(Path path,
                               org.apache.hadoop.fs.FileSystem.Statistics statistics,
                               boolean overwrite,
                               FsPermission permission,
                               FsPermission umask,
                               org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                        throws IOException

Throws:

IOException

createDirectory

public void createDirectory(Path path,
                            FsPermission permission,
                            FsPermission umask,
                            org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                     throws IOException

Throws:

IOException

openFileForRead

public org.apache.hadoop.fs.azurebfs.services.AbfsInputStream openFileForRead(Path path,
                                                                              org.apache.hadoop.fs.FileSystem.Statistics statistics,
                                                                              org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                                                                       throws IOException

Throws:

IOException

openFileForRead

public org.apache.hadoop.fs.azurebfs.services.AbfsInputStream openFileForRead(Path path,
                                                                              Optional<org.apache.hadoop.fs.impl.OpenFileParameters> parameters,
                                                                              org.apache.hadoop.fs.FileSystem.Statistics statistics,
                                                                              org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                                                                       throws IOException

Throws:

IOException

openFileForWrite

public OutputStream openFileForWrite(Path path,
                                     org.apache.hadoop.fs.FileSystem.Statistics statistics,
                                     boolean overwrite,
                                     org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                              throws IOException

Throws:

IOException

breakLease

public void breakLease(Path path,
                       org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                throws AzureBlobFileSystemException

Break any current lease on an ABFS file.

Parameters:

path - file name

tracingContext - TracingContext instance to track correlation IDs

Throws:

AzureBlobFileSystemException - on any exception while breaking the lease

rename

public boolean rename(Path source,
                      Path destination,
                      org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext,
                      String sourceEtag)
               throws IOException

Rename a file or directory. If a source etag is passed in, the operation will attempt to recover from a missing source file by probing the destination for existence and comparing etags.

Parameters:

source - path to source file

destination - destination of rename.

tracingContext - trace context

sourceEtag - etag of source file. may be null or empty

Returns:

true if recovery was needed and succeeded.

Throws:

AzureBlobFileSystemException - failure, excluding any recovery from overload failures.

IOException

delete

public void delete(Path path,
                   boolean recursive,
                   org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
            throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

getFileStatus

public FileStatus getFileStatus(Path path,
                                org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                         throws IOException

Throws:

IOException

listStatus

public FileStatus[] listStatus(Path path,
                               org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                        throws IOException

Specified by:

listStatus in interface org.apache.hadoop.fs.azurebfs.services.ListingSupport

Parameters:

path - The list path.

tracingContext - Tracks identifiers for request header

Returns:

the entries in the path.

Throws:

IOException - in case of error

listStatus

@InterfaceStability.Unstable
public FileStatus[] listStatus(Path path,
                                                            String startFrom,
                                                            org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                                                     throws IOException

Specified by:

listStatus in interface org.apache.hadoop.fs.azurebfs.services.ListingSupport

Parameters:

path - Path the list path.

startFrom - the entry name that list results should start with. For example, if folder "/folder" contains four files: "afile", "bfile", "hfile", "ifile". Then listStatus(Path("/folder"), "hfile") will return "/folder/hfile" and "folder/ifile" Notice that if startFrom is a non-existent entry name, then the list response contains all entries after this non-existent entry in lexical order: listStatus(Path("/folder"), "cfile") will return "/folder/hfile" and "/folder/ifile".

tracingContext - Tracks identifiers for request header

Returns:

the entries in the path start from "startFrom" in lexical order.

Throws:

IOException - in case of error

listStatus

public String listStatus(Path path,
                         String startFrom,
                         List<FileStatus> fileStatuses,
                         boolean fetchAll,
                         String continuation,
                         org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                  throws IOException

Specified by:

listStatus in interface org.apache.hadoop.fs.azurebfs.services.ListingSupport

Parameters:

path - The list path

startFrom - The entry name that list results should start with. For example, if folder "/folder" contains four files: "afile", "bfile", "hfile", "ifile". Then listStatus(Path("/folder"), "hfile") will return "/folder/hfile" and "folder/ifile" Notice that if startFrom is a non-existent entry name, then the list response contains all entries after this non-existent entry in lexical order: listStatus (Path("/folder"), "cfile") will return "/folder/hfile" and "/folder/ifile".

fileStatuses - This list has to be filled with the FileStatus objects

fetchAll - flag to indicate if the above list needs to be filled with just one page os results or the entire result.

continuation - Contiuation token. null means start rom the begining.

tracingContext - TracingContext instance to track identifiers

Returns:

Continuation tokem

Throws:

IOException - in case of error

setOwner

public void setOwner(Path path,
                     String owner,
                     String group,
                     org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
              throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

setPermission

public void setPermission(Path path,
                          FsPermission permission,
                          org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                   throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

modifyAclEntries

public void modifyAclEntries(Path path,
                             List<AclEntry> aclSpec,
                             org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                      throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

removeAclEntries

public void removeAclEntries(Path path,
                             List<AclEntry> aclSpec,
                             org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                      throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

removeDefaultAcl

public void removeDefaultAcl(Path path,
                             org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                      throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

removeAcl

public void removeAcl(Path path,
                      org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
               throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

setAcl

public void setAcl(Path path,
                   List<AclEntry> aclSpec,
                   org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
            throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

getAclStatus

public AclStatus getAclStatus(Path path,
                              org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
                       throws IOException

Throws:

IOException

access

public void access(Path path,
                   FsAction mode,
                   org.apache.hadoop.fs.azurebfs.utils.TracingContext tracingContext)
            throws AzureBlobFileSystemException

Throws:

AzureBlobFileSystemException

isAtomicRenameKey

public boolean isAtomicRenameKey(String key)

isInfiniteLeaseKey

public boolean isInfiniteLeaseKey(String key)

getRelativePath

public String getRelativePath(Path path)

getClient

@VisibleForTesting
public org.apache.hadoop.fs.azurebfs.services.AbfsClient getClient()

getClient

@VisibleForTesting
public org.apache.hadoop.fs.azurebfs.services.AbfsClient getClient(org.apache.hadoop.fs.azurebfs.constants.AbfsServiceType serviceType)

getClientHandler

@VisibleForTesting
public org.apache.hadoop.fs.azurebfs.services.AbfsClientHandler getClientHandler()

extractEtagHeader

public static String extractEtagHeader(org.apache.hadoop.fs.azurebfs.services.AbfsHttpOperation result)

Get the etag header from a response, stripping any quotations. see: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag

Parameters:

result - response to process.

Returns:

the quote-unwrapped etag.