Package org.apache.hadoop.fs

Interface BulkDelete

All Superinterfaces:
AutoCloseable, Closeable, org.apache.hadoop.fs.statistics.IOStatisticsSource
@Public @Unstable public interface BulkDelete extends org.apache.hadoop.fs.statistics.IOStatisticsSource, Closeable
API for bulk deletion of objects/files, but not directories. After use, call close() to release any resources and to guarantee store IOStatistics are updated.

Callers MUST have no expectation that parent directories will exist after the operation completes; if an object store needs to explicitly look for and create directory markers, that step will be omitted.

Be aware that on some stores (AWS S3) each object listed in a bulk delete counts against the write IOPS limit; large page sizes are counterproductive here, as are attempts at parallel submissions across multiple threads.

See Also:

  • Method Summary

    Modifier and Type
    Method
    Description
    Path
    basePath()
    Base path of a bulk delete operation.
    List<Map.Entry<Path,String>>
    bulkDelete(Collection<Path> paths)
    Delete a list of files/objects.
    int
    pageSize()
    The maximum number of objects/files to delete in a single request.

    Methods inherited from interface java.io.Closeable

    close

    Methods inherited from interface org.apache.hadoop.fs.statistics.IOStatisticsSource

    getIOStatistics

  • Method Details

    • pageSize

      int pageSize()
      The maximum number of objects/files to delete in a single request.
      Returns:
      a number greater than zero.

    • basePath

      Path basePath()
      Base path of a bulk delete operation. All paths submitted in bulkDelete(Collection) must be under this path.
      Returns:
      base path of a bulk delete operation.

    • bulkDelete

      List<Map.Entry<Path,String>> bulkDelete(Collection<Path> paths) throws IOException, IllegalArgumentException
      Delete a list of files/objects.
      • Files must be under the path provided in basePath().
      • The size of the list must be equal to or less than the page size declared in pageSize().
      • Directories are not supported; the outcome of attempting to delete directories is undefined (ignored; undetected, listed as failures...).
      • The operation is not atomic.
      • The operation is treated as idempotent: network failures may trigger resubmission of the request -any new objects created under a path in the list may then be deleted.
      • There is no guarantee that any parent directories exist after this call.
      Parameters:
      paths - list of paths which must be absolute and under the base path. provided in basePath().
      Returns:
      a list of paths which failed to delete, with the exception message.
      Throws:
      IOException - IO problems including networking, authentication and more.
      IllegalArgumentException - if a path argument is invalid.