Package org.apache.sysds.utils.stats

Class InfrastructureAnalyzer

java.lang.Object

org.apache.sysds.utils.stats.InfrastructureAnalyzer

public class InfrastructureAnalyzer
extends Object

Central place for analyzing and obtaining static infrastructure properties such as memory and number of logical processors.

Field Summary

Fields

Modifier and Type	Field	Description
static long	DEFAULT_JVM_SIZE
static String	MR_FRAMEWORK_NAME
static String	MR_JOBTRACKER_ADDRESS

Constructor Summary

Constructors

Constructor	Description
InfrastructureAnalyzer()

Method Summary

Modifier and Type	Method	Description
static long	extractMaxMemoryOpt(String javaOpts)
static long	getBlockSize(org.apache.hadoop.fs.FileSystem fs)
static int	getCkMaxCP()	Gets the maximum local parallelism constraint.
static int	getCkMaxMR()	Gets the maximum remote parallelism constraint
static double	getClusterUtilization(boolean mapOnly)	Gets the fraction of running map/reduce tasks to existing map/reduce task slots.
static long	getCmMax()	Gets the maximum memory constraint [in bytes].
static long	getHDFSBlockSize()	Gets the HDFS blocksize of the used cluster in bytes.
static long	getLocalMaxMemory()	Gets the maximum memory [in bytes] of the current JVM.
static double	getLocalMaxMemoryFraction()
static int	getLocalParallelism()	Gets the number of logical processors of the current node, including hyper-threading if enabled.
static int	getRemoteParallelMapTasks()	Gets the number of remote parallel map slots.
static int	getRemoteParallelNodes()	Gets the number of cluster nodes (number of tasktrackers).
static int	getRemoteParallelReduceTasks()	Gets the total number of remote parallel reduce slots.
static boolean	isLocalMode()
static boolean	isLocalMode(org.apache.hadoop.mapred.JobConf job)
static void	setLocalMaxMemory(long localMem)
static void	setLocalPar(int localPar)
static void	setMaxMemoryOpt(org.apache.hadoop.mapred.JobConf job, String key, long bytes)
static void	setRemoteParallelMapTasks(int pmap)
static void	setRemoteParallelReduceTasks(int preduce)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_JVM_SIZE

public static final long DEFAULT_JVM_SIZE

See Also:

Constant Field Values

MR_FRAMEWORK_NAME

public static final String MR_FRAMEWORK_NAME

See Also:

Constant Field Values

MR_JOBTRACKER_ADDRESS

public static final String MR_JOBTRACKER_ADDRESS

See Also:

Constant Field Values

Constructor Detail

InfrastructureAnalyzer

public InfrastructureAnalyzer()

Method Detail

getLocalParallelism

public static int getLocalParallelism()

Gets the number of logical processors of the current node, including hyper-threading if enabled.

Returns:

number of local processors of the current node

getRemoteParallelNodes

public static int getRemoteParallelNodes()

Gets the number of cluster nodes (number of tasktrackers). If multiple tasktracker are started per node, each tasktracker is viewed as individual node.

Returns:

number of cluster nodes

getRemoteParallelMapTasks

public static int getRemoteParallelMapTasks()

Gets the number of remote parallel map slots.

Returns:

number of remote parallel map tasks

setRemoteParallelMapTasks

public static void setRemoteParallelMapTasks(int pmap)

getRemoteParallelReduceTasks

public static int getRemoteParallelReduceTasks()

Gets the total number of remote parallel reduce slots.

Returns:

number of remote parallel reduce tasks

setRemoteParallelReduceTasks

public static void setRemoteParallelReduceTasks(int preduce)

getLocalMaxMemory

public static long getLocalMaxMemory()

Gets the maximum memory [in bytes] of the current JVM.

Returns:

maximum memory of the current JVM

setLocalMaxMemory

public static void setLocalMaxMemory(long localMem)

setLocalPar

public static void setLocalPar(int localPar)

getLocalMaxMemoryFraction

public static double getLocalMaxMemoryFraction()

isLocalMode

public static boolean isLocalMode()

isLocalMode

public static boolean isLocalMode(org.apache.hadoop.mapred.JobConf job)

getCkMaxCP

public static int getCkMaxCP()

Gets the maximum local parallelism constraint.

Returns:

maximum local parallelism constraint

getCkMaxMR

public static int getCkMaxMR()

Gets the maximum remote parallelism constraint

Returns:

maximum remote parallelism constraint

getCmMax

public static long getCmMax()

Gets the maximum memory constraint [in bytes].

Returns:

maximum memory constraint

getBlockSize

public static long getBlockSize(org.apache.hadoop.fs.FileSystem fs)

getHDFSBlockSize

public static long getHDFSBlockSize()

Gets the HDFS blocksize of the used cluster in bytes.

Returns:

HDFS block size

extractMaxMemoryOpt

public static long extractMaxMemoryOpt(String javaOpts)

setMaxMemoryOpt

public static void setMaxMemoryOpt(org.apache.hadoop.mapred.JobConf job,
                                   String key,
                                   long bytes)

getClusterUtilization

public static double getClusterUtilization(boolean mapOnly)
                                    throws IOException

Gets the fraction of running map/reduce tasks to existing map/reduce task slots. NOTE: on YARN the number of slots is a spurious indicator because containers are purely scheduled based on memory.

Parameters:

mapOnly - if true, only look at map tasks

Returns:

cluster utilization (current / capacity)

Throws:

IOException - if IOException occurs