Package org.apache.sysds.hops

Class AggBinaryOp

java.lang.Object

org.apache.sysds.hops.Hop

org.apache.sysds.hops.MultiThreadedHop

org.apache.sysds.hops.AggBinaryOp

All Implemented Interfaces:

ParseInfo

public class AggBinaryOp
extends MultiThreadedHop

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	AggBinaryOp.MMultMethod
static class	AggBinaryOp.SparkAggType

Field Summary

Fields

Modifier and Type	Field	Description
static AggBinaryOp.MMultMethod	FORCED_MMULT_METHOD
static double	MAPMULT_MEM_MULTIPLIER

Fields inherited from class org.apache.sysds.hops.Hop

_beginColumn, _beginLine, _endColumn, _endLine, _filename, _text, CPThreshold

Constructor Summary

Constructors

Constructor	Description
AggBinaryOp(String l, Types.DataType dt, Types.ValueType vt, Types.OpOp2 innOp, Types.AggOp outOp, Hop in1, Hop in2)

Method Summary

Modifier and Type	Method	Description
boolean	allowsAllExecTypes()
void	checkArity()	Check whether this Hop has a correct number of inputs.
MapMultChain.ChainType	checkMapMultChain()	MapMultChain: Determine if XtwXv/XtXv pattern applies for this aggbinary and if yes which type.
MMTSJ.MMTSJType	checkTransposeSelf()	TSMM: Determine if XtX pattern applies for this aggbinary and if yes which type.
Object	clone()
boolean	compare(Hop that)
void	computeMemEstimate(MemoTable memo)	Computes the estimate of memory required to store the input/output of this hop in memory.
Lop	constructLops()	NOTE: overestimated mem in case of transpose-identity matmult, but 3/2 at worst and existing mem estimate advantageous in terms of consistency hops/lops, and some special cases internally materialize the transpose for better cache locality
static double	getMapmmMemEstimate(long m1_rows, long m1_cols, long m1_blen, long m1_nnz, long m2_rows, long m2_cols, long m2_blen, long m2_nnz, int cachedInputIndex, boolean pmm)	Estimates the memory footprint of MapMult operation depending on which input is put into distributed cache.
AggBinaryOp.MMultMethod	getMMultMethod()
String	getOpString()
boolean	hasLeftPMInput()
boolean	isGPUEnabled()	In memory-based optimizer mode (see OptimizerUtils.isMemoryBasedOptLevel()), the exectype is determined by checking this method as well as memory budget of this Hop.
boolean	isMatrixMultiply()
boolean	isMultiThreadedOpType()
void	refreshSizeInformation()	Update the output size information for this hop.
void	setHasLeftPMInput(boolean flag)

Methods inherited from class org.apache.sysds.hops.MultiThreadedHop

getMaxNumThreads, setMaxNumThreads

Methods inherited from class org.apache.sysds.hops.Hop

activatePrefetch, addAllInputs, addInput, checkAndSetForcedPlatform, checkAndSetInvalidCPDimsAndSize, clearMemEstimate, colsKnown, compressedSize, computeBoundsInformation, computeBoundsInformation, computeBoundsInformation, computeSizeInformation, computeSizeInformation, computeSizeInformation, constructAndSetLopsDataFlowProperties, createOffsetLop, deactivatePrefetch, dimsKnown, dimsKnown, dimsKnownAny, getBeginColumn, getBeginLine, getBlocksize, getCompressedSize, getDataCharacteristics, getDataType, getDim, getDim1, getDim2, getEndColumn, getEndLine, getExecType, getFederatedOutput, getFilename, getForcedExecType, getHopID, getInput, getInput, getInputMemEstimate, getInputMemEstimate, getInputOutputSize, getIntermediateMemEstimate, getLength, getLops, getMemEstimate, getName, getNnz, getOutputMemEstimate, getOutputMemEstimate, getParent, getSparsity, getSpBroadcastSize, getText, getUpdateType, getValueType, hasCompressedInput, hasFederatedOutput, hasLocalOutput, hasMatrixInputWithDifferentBlocksizes, hasValidCPDimsAndSize, isCompressedOutput, isFederated, isFederatedDataOp, isMatrix, isMemEstimated, isOutputEmptyBlocks, isRequiredDecompression, isScalar, isTransposeSafe, isVisited, prefetchActivated, printErrorLocation, refreshColsParameterInformation, refreshColsParameterInformation, refreshMemEstimates, refreshRowsParameterInformation, refreshRowsParameterInformation, requiresCheckpoint, requiresCompression, requiresLineageCaching, requiresReblock, requiresRecompile, resetExecType, resetRecompilationFlag, resetRecompilationFlag, resetVisitStatus, resetVisitStatus, resetVisitStatus, resetVisitStatusForced, rowsKnown, setBeginColumn, setBeginLine, setBlocksize, setCompressedOutput, setCompressedSize, setDataType, setDim, setDim1, setDim2, setEndColumn, setEndLine, setExecType, setFederatedOutput, setFilename, setForcedExecType, setLops, setMemEstimate, setName, setNnz, setOutputEmptyBlocks, setParseInfo, setRequiresCheckpoint, setRequiresCompression, setRequiresCompression, setRequiresDeCompression, setRequiresLineageCaching, setRequiresReblock, setRequiresRecompile, setText, setUpdateType, setValueType, setVisited, setVisited, someInputFederated, toString, updateLopFedOut, updateLopFedOut

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

MAPMULT_MEM_MULTIPLIER

public static final double MAPMULT_MEM_MULTIPLIER

See Also:

Constant Field Values

FORCED_MMULT_METHOD

public static AggBinaryOp.MMultMethod FORCED_MMULT_METHOD

Constructor Detail

AggBinaryOp

public AggBinaryOp(String l,
                   Types.DataType dt,
                   Types.ValueType vt,
                   Types.OpOp2 innOp,
                   Types.AggOp outOp,
                   Hop in1,
                   Hop in2)

Method Detail

checkArity

public void checkArity()

Description copied from class: Hop

Check whether this Hop has a correct number of inputs. (Some Hops can have a variable number of inputs, such as DataOp, DataGenOp, ParameterizedBuiltinOp, ReorgOp, TernaryOp, QuaternaryOp, MultipleOp, DnnOp, and SpoofFusedOp.) Parameterized Hops (such as DataOp) can check that the number of parameters matches the number of inputs.

Specified by:

checkArity in class Hop

setHasLeftPMInput

public void setHasLeftPMInput(boolean flag)

hasLeftPMInput

public boolean hasLeftPMInput()

getMMultMethod

public AggBinaryOp.MMultMethod getMMultMethod()

isGPUEnabled

public boolean isGPUEnabled()

Description copied from class: Hop

In memory-based optimizer mode (see OptimizerUtils.isMemoryBasedOptLevel()), the exectype is determined by checking this method as well as memory budget of this Hop. Please see findExecTypeByMemEstimate for more detail. This method is necessary because not all operator are supported efficiently on GPU (for example: operations on frames and scalar as well as operations such as table).

Specified by:

isGPUEnabled in class Hop

Returns:

true if the Hop is eligible for GPU Exectype.

constructLops

public Lop constructLops()

NOTE: overestimated mem in case of transpose-identity matmult, but 3/2 at worst and existing mem estimate advantageous in terms of consistency hops/lops, and some special cases internally materialize the transpose for better cache locality

Specified by:

constructLops in class Hop

getOpString

public String getOpString()

Specified by:

getOpString in class Hop

computeMemEstimate

public void computeMemEstimate(MemoTable memo)

Description copied from class: Hop

Computes the estimate of memory required to store the input/output of this hop in memory. This is the default implementation (orchestration of hop-specific implementation) that should suffice for most hops. If a hop requires more control, this method should be overwritten with awareness of (1) output estimates, and (2) propagation of worst-case matrix characteristics (dimensions, sparsity). TODO remove memo table and, on constructor refresh, inference in refresh, single compute mem, maybe general computeMemEstimate, flags to indicate if estimate or not.

Overrides:

computeMemEstimate in class Hop

Parameters:

memo - memory table

isMatrixMultiply

public boolean isMatrixMultiply()

isMultiThreadedOpType

public boolean isMultiThreadedOpType()

Specified by:

isMultiThreadedOpType in class MultiThreadedHop

allowsAllExecTypes

public boolean allowsAllExecTypes()

Specified by:

allowsAllExecTypes in class Hop

checkTransposeSelf

public MMTSJ.MMTSJType checkTransposeSelf()

TSMM: Determine if XtX pattern applies for this aggbinary and if yes which type.

Returns:

MMTSJType

checkMapMultChain

public MapMultChain.ChainType checkMapMultChain()

MapMultChain: Determine if XtwXv/XtXv pattern applies for this aggbinary and if yes which type.

Returns:

ChainType

getMapmmMemEstimate

public static double getMapmmMemEstimate(long m1_rows,
                                         long m1_cols,
                                         long m1_blen,
                                         long m1_nnz,
                                         long m2_rows,
                                         long m2_cols,
                                         long m2_blen,
                                         long m2_nnz,
                                         int cachedInputIndex,
                                         boolean pmm)

Estimates the memory footprint of MapMult operation depending on which input is put into distributed cache. This function is called by optFindMMultMethod() to decide the execution strategy, as well as by piggybacking to decide the number of Map-side instructions to put into a single GMR job.

Parameters:

m1_rows - m1 rows

m1_cols - m1 cols

m1_blen - m1 rows/cols per block

m1_nnz - m1 num non-zeros

m2_rows - m2 rows

m2_cols - m2 cols

m2_blen - m2 rows/cols per block

m2_nnz - m2 num non-zeros

cachedInputIndex - true if cached input index

pmm - true if permutation matrix multiply

Returns:

map mm memory estimate

refreshSizeInformation

public void refreshSizeInformation()

Description copied from class: Hop

Update the output size information for this hop.

Specified by:

refreshSizeInformation in class Hop

clone

public Object clone()
             throws CloneNotSupportedException

Specified by:

clone in class Hop

Throws:

CloneNotSupportedException

compare

public boolean compare(Hop that)

Specified by:

compare in class Hop