Class RemoteParForUtils


  • public class RemoteParForUtils
    extends Object
    Common functionalities for parfor workers in MR jobs. Used by worker wrappers in mappers (base RemoteParFor) and reducers (fused data partitioning and parfor)
    • Constructor Detail

      • RemoteParForUtils

        public RemoteParForUtils()
    • Method Detail

      • incrementParForMRCounters

        public static void incrementParForMRCounters​(org.apache.hadoop.mapred.Reporter reporter,
                                                     long deltaTasks,
                                                     long deltaIterations)
      • cleanupWorkingDirectories

        public static void cleanupWorkingDirectories()
        Cleanup all temporary files created by this SystemDS process.
      • cleanupWorkingDirectoriesOnShutdown

        public static void cleanupWorkingDirectoriesOnShutdown()
        Cleanup all temporary files created by this SystemDS process, on shutdown via exit or interrupt.
      • setupBufferPool

        public static void setupBufferPool​(long workerID,
                                           boolean isLocal)
                                    throws IOException
        Init and register-cleanup of buffer pool
        Parameters:
        workerID - worker id
        isLocal - in local spark mode (single JVM)
        Throws:
        IOException - exception