Package org.apache.sysds.resource.enumeration

Class InterestBasedEnumerator

java.lang.Object

org.apache.sysds.resource.enumeration.Enumerator

org.apache.sysds.resource.enumeration.InterestBasedEnumerator

public class InterestBasedEnumerator
extends Enumerator

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.sysds.resource.enumeration.Enumerator

Enumerator.Builder, Enumerator.EnumerationStrategy, Enumerator.OptimizationStrategy

Field Summary

Fields

Modifier and Type	Field	Description
static double	BROADCAST_MEMORY_FACTOR
static double	MEMORY_DELTA_FRACTION
static long	MINIMUM_RELEVANT_MEM_ESTIMATE
static boolean	USE_MEMORY_DELTA

Fields inherited from class org.apache.sysds.resource.enumeration.Enumerator

COST_DELTA_FRACTION, DEFAULT_MAX_EXECUTORS, DEFAULT_MIN_EXECUTORS, MAX_LEVEL_PARALLELISM

Constructor Summary

Constructors

Constructor	Description
InterestBasedEnumerator(Enumerator.Builder builder, boolean fitDriverMemory, boolean fitBroadcastMemory, boolean checkSingleNodeExecution, boolean fitCheckpointMemory)

Method Summary

Modifier and Type	Method	Description
ArrayList<Integer>	estimateRangeExecutors(long executorMemory, int executorCores)	Estimates the minimum and maximum number of executors based on given VM instance characteristics and on the enumeration strategy
boolean	evaluateSingleNodeExecution(long driverMemory)
static TreeSet<Long>	getMemoryEstimates(Program currentProgram, boolean outputOnly, double memoryFactor)	Extracts the memory estimates which original size is larger than MINIMUM_RELEVANT_MEM_ESTIMATE
void	preprocessing()	Called once to enumerate the search space for VM instances for driver or executor nodes.

Methods inherited from class org.apache.sysds.resource.enumeration.Enumerator

getDriverSpace, getExecutorSpace, getInstances, getSolutionPool, loadInstanceTableFile, postprocessing, processing, setDriverSpace, setExecutorSpace, setInstanceTable, setSolutionPool

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MINIMUM_RELEVANT_MEM_ESTIMATE

public static final long MINIMUM_RELEVANT_MEM_ESTIMATE

See Also:

Constant Field Values

USE_MEMORY_DELTA

public static final boolean USE_MEMORY_DELTA

See Also:

Constant Field Values

MEMORY_DELTA_FRACTION

public static final double MEMORY_DELTA_FRACTION

See Also:

Constant Field Values

BROADCAST_MEMORY_FACTOR

public static final double BROADCAST_MEMORY_FACTOR

See Also:

Constant Field Values

Constructor Detail

InterestBasedEnumerator

public InterestBasedEnumerator(Enumerator.Builder builder,
                               boolean fitDriverMemory,
                               boolean fitBroadcastMemory,
                               boolean checkSingleNodeExecution,
                               boolean fitCheckpointMemory)

Method Detail

preprocessing

public void preprocessing()

Description copied from class: Enumerator

Called once to enumerate the search space for VM instances for driver or executor nodes. These instances are being represented as

Specified by:

preprocessing in class Enumerator

evaluateSingleNodeExecution

public boolean evaluateSingleNodeExecution(long driverMemory)

Specified by:

evaluateSingleNodeExecution in class Enumerator

estimateRangeExecutors

public ArrayList<Integer> estimateRangeExecutors(long executorMemory,
                                                 int executorCores)

Description copied from class: Enumerator

Estimates the minimum and maximum number of executors based on given VM instance characteristics and on the enumeration strategy

Specified by:

estimateRangeExecutors in class Enumerator

Parameters:

executorMemory - memory of currently considered executor instance

executorCores - CPU of cores of currently considered executor instance

Returns:

- [min, max]

getMemoryEstimates

public static TreeSet<Long> getMemoryEstimates(Program currentProgram,
                                               boolean outputOnly,
                                               double memoryFactor)

Extracts the memory estimates which original size is larger than MINIMUM_RELEVANT_MEM_ESTIMATE

Parameters:

currentProgram - program for extracting the memory estimates from

outputOnly - true - output estimate only; false - sum of input, intermediate and output estimates

memoryFactor - factor for reverse scaling the estimates to avoid scaling the search space parameters representing the nodes' memory budget

Returns:

memory estimates in ascending order ensured by the TreeSet data structure