Package org.apache.sysds.resource.enumeration

Class Enumerator

java.lang.Object

org.apache.sysds.resource.enumeration.Enumerator

Direct Known Subclasses:

GridBasedEnumerator, InterestBasedEnumerator

public abstract class Enumerator
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	Enumerator.Builder
static class	Enumerator.EnumerationStrategy
static class	Enumerator.OptimizationStrategy

Field Summary

Fields

Modifier and Type	Field	Description
static double	COST_DELTA_FRACTION	Time/Monetary delta for considering optimal solutions as fraction
static int	DEFAULT_MAX_EXECUTORS	A reasonable upper bound for the possible number of executors is required to set limits for the search space and to avoid evaluating cluster configurations that most probably would have too high distribution overhead
static int	DEFAULT_MIN_EXECUTORS
static int	MAX_LEVEL_PARALLELISM

Constructor Summary

Constructors

Constructor	Description
Enumerator(Enumerator.Builder builder)

Method Summary

Modifier and Type	Method	Description
abstract ArrayList<Integer>	estimateRangeExecutors(long executorMemory, int executorCores)	Estimates the minimum and maximum number of executors based on given VM instance characteristics and on the enumeration strategy
abstract boolean	evaluateSingleNodeExecution(long driverMemory)
EnumerationUtils.InstanceSearchSpace	getDriverSpace()	Meant to be used for testing purposes
EnumerationUtils.InstanceSearchSpace	getExecutorSpace()	Meant to be used for testing purposes
HashMap<String,CloudInstance>	getInstances()	Meant to be used for testing purposes
ArrayList<EnumerationUtils.SolutionPoint>	getSolutionPool()	Meant to be used for testing purposes
void	loadInstanceTableFile(String path)	Loads the info table for the available VM instances and filters out the instances that are not contained in the set of allowed instance types and sizes.
EnumerationUtils.SolutionPoint	postprocessing()	Deciding in the overall best solution out of the filled pool of potential solutions after processing.
abstract void	preprocessing()	Called once to enumerate the search space for VM instances for driver or executor nodes.
void	processing()	Called once after preprocessing to fill the pool with optimal solutions by parsing the enumerated search space.
void	setDriverSpace(EnumerationUtils.InstanceSearchSpace inputSpace)	Meant to be used for testing purposes
void	setExecutorSpace(EnumerationUtils.InstanceSearchSpace inputSpace)	Meant to be used for testing purposes
void	setInstanceTable(HashMap<String,CloudInstance> inputInstances)	Setting the available VM instances manually.
void	setSolutionPool(ArrayList<EnumerationUtils.SolutionPoint> solutionPool)	Meant to be used for testing purposes

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_MIN_EXECUTORS

public static final int DEFAULT_MIN_EXECUTORS

See Also:

Constant Field Values

DEFAULT_MAX_EXECUTORS

public static final int DEFAULT_MAX_EXECUTORS

A reasonable upper bound for the possible number of executors is required to set limits for the search space and to avoid evaluating cluster configurations that most probably would have too high distribution overhead

See Also:

Constant Field Values

MAX_LEVEL_PARALLELISM

public static final int MAX_LEVEL_PARALLELISM

See Also:

Constant Field Values

COST_DELTA_FRACTION

public static final double COST_DELTA_FRACTION

Time/Monetary delta for considering optimal solutions as fraction

See Also:

Constant Field Values

Constructor Detail

Enumerator

public Enumerator(Enumerator.Builder builder)

Method Detail

getInstances

public HashMap<String,CloudInstance> getInstances()

Meant to be used for testing purposes

Returns:

?

getDriverSpace

public EnumerationUtils.InstanceSearchSpace getDriverSpace()

Meant to be used for testing purposes

Returns:

?

setDriverSpace

public void setDriverSpace(EnumerationUtils.InstanceSearchSpace inputSpace)

Meant to be used for testing purposes

Parameters:

inputSpace - ?

getExecutorSpace

public EnumerationUtils.InstanceSearchSpace getExecutorSpace()

Meant to be used for testing purposes

Returns:

?

setExecutorSpace

public void setExecutorSpace(EnumerationUtils.InstanceSearchSpace inputSpace)

Meant to be used for testing purposes

Parameters:

inputSpace - ?

getSolutionPool

public ArrayList<EnumerationUtils.SolutionPoint> getSolutionPool()

Meant to be used for testing purposes

Returns:

?

setSolutionPool

public void setSolutionPool(ArrayList<EnumerationUtils.SolutionPoint> solutionPool)

Meant to be used for testing purposes

Parameters:

solutionPool - ?

setInstanceTable

public void setInstanceTable(HashMap<String,CloudInstance> inputInstances)

Setting the available VM instances manually. Meant to be used for testing purposes.

Parameters:

inputInstances - initialized map of instances

loadInstanceTableFile

public void loadInstanceTableFile(String path)
                           throws IOException

Loads the info table for the available VM instances and filters out the instances that are not contained in the set of allowed instance types and sizes.

Parameters:

path - csv file with instances' info

Throws:

IOException - in case the loading part fails at reading the csv file

preprocessing

public abstract void preprocessing()

Called once to enumerate the search space for VM instances for driver or executor nodes. These instances are being represented as

processing

public void processing()

Called once after preprocessing to fill the pool with optimal solutions by parsing the enumerated search space. Within its execution the number of potential executor nodes is being estimated (enumerated) dynamically for each parsed executor instance.

postprocessing

public EnumerationUtils.SolutionPoint postprocessing()

Deciding in the overall best solution out of the filled pool of potential solutions after processing.

Returns:

single optimal cluster configuration

evaluateSingleNodeExecution

public abstract boolean evaluateSingleNodeExecution(long driverMemory)

estimateRangeExecutors

public abstract ArrayList<Integer> estimateRangeExecutors(long executorMemory,
                                                          int executorCores)

Estimates the minimum and maximum number of executors based on given VM instance characteristics and on the enumeration strategy

Parameters:

executorMemory - memory of currently considered executor instance

executorCores - CPU of cores of currently considered executor instance

Returns:

- [min, max]