Package org.apache.sysds.lops

Class PartialAggregate

java.lang.Object

org.apache.sysds.lops.Lop

org.apache.sysds.lops.PartialAggregate

public class PartialAggregate
extends Lop

Lop to perform a partial aggregation. It was introduced to do some initial aggregation operations on blocks in the mapper/reducer.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.sysds.lops.Lop

Lop.SimpleInstType, Lop.Type, Lop.VisitStatus

Field Summary

Fields inherited from class org.apache.sysds.lops.Lop

_beginColumn, _beginLine, _endColumn, _endLine, _filename, CP_CHILD_THREAD, CP_ROOT_THREAD_ID, DATATYPE_PREFIX, FILE_SEPARATOR, FRAME_VAR_NAME_PREFIX, INSTRUCTION_DELIMITOR, LITERAL_PREFIX, MATRIX_VAR_NAME_PREFIX, NAME_VALUE_SEPARATOR, OPERAND_DELIMITOR, PROCESS_PREFIX, SCALAR_VAR_NAME_PREFIX, UPDATE_INPLACE_PREFIX, VALUETYPE_PREFIX, VARIABLE_NAME_PLACEHOLDER

Constructor Summary

Constructors

Constructor	Description
PartialAggregate(Lop input, Types.AggOp op, Types.Direction direct, Types.DataType dt, Types.ValueType vt, Types.ExecType et, int k)
PartialAggregate(Lop input, Types.AggOp op, Types.Direction direct, Types.DataType dt, Types.ValueType vt, AggBinaryOp.SparkAggType aggtype, Types.ExecType et)

Method Summary

Modifier and Type	Method	Description
AggBinaryOp.SparkAggType	getAggType()	Method to get aggregate type if applicable.
Types.CorrectionLocationType	getCorrectionLocation()	This method computes the location of "correction" terms in the output produced by PartialAgg instruction.
static Types.CorrectionLocationType	getCorrectionLocation(Types.AggOp operation, Types.Direction direction)
String	getInstructions(String input1, String output)	Instruction generation for CP and Spark
static String	getOpcode(Types.AggOp op, Types.Direction dir)
void	setDimensionsBasedOnDirection(long dim1, long dim2, long blen)
static void	setDimensionsBasedOnDirection(Lop lop, long dim1, long dim2, long blen, Types.Direction dir)
String	toString()	Method to have Lops print their state.

Methods inherited from class org.apache.sysds.lops.Lop

activatePrefetch, addInput, addOutput, addToDag, createReachable, getBeginColumn, getBeginLine, getBroadcastInput, getComputeEstimate, getDataType, getEndColumn, getEndLine, getExecType, getFederatedOutput, getFilename, getID, getInput, getInputs, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getLevel, getNnz, getNumCols, getNumRows, getOutputMemoryEstimate, getOutputParameters, getOutputs, getPrivacyConstraint, getProducesIntermediateOutput, getReachable, getSimpleInstructionType, getTotalMemoryEstimate, getType, getValueType, getVisited, isAllOutputsCP, isAllOutputsGPU, isAsynchronousOp, isDataExecLocation, isExecCP, isExecGPU, isExecSpark, isVariable, isVisited, prefetchActivated, prepInputOperand, prepInputOperand, prepOperand, prepOperand, prepOutputOperand, prepOutputOperand, prepOutputOperand, prepScalarInputOperand, prepScalarInputOperand, prepScalarLabel, prepScalarOperand, printErrorLocation, removeConsumer, removeInput, removeOutput, replaceAllInputs, replaceAllOutputs, replaceInput, resetVisitStatus, setAllPositions, setAsynchronous, setBeginColumn, setBeginLine, setComputeEstimate, setConsumerCount, setDataType, setEndColumn, setEndLine, setExecType, setFederatedOutput, setFilename, setMemoryEstimates, setNewID, setPrivacyConstraint, setValueType, setVisited, setVisited

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

PartialAggregate

public PartialAggregate(Lop input,
                        Types.AggOp op,
                        Types.Direction direct,
                        Types.DataType dt,
                        Types.ValueType vt,
                        Types.ExecType et,
                        int k)

PartialAggregate

public PartialAggregate(Lop input,
                        Types.AggOp op,
                        Types.Direction direct,
                        Types.DataType dt,
                        Types.ValueType vt,
                        AggBinaryOp.SparkAggType aggtype,
                        Types.ExecType et)

Method Detail

getCorrectionLocation

public Types.CorrectionLocationType getCorrectionLocation()

This method computes the location of "correction" terms in the output produced by PartialAgg instruction. When computing the stable sum, "correction" refers to the compensation as defined by the original Kahan algorithm. When computing the stable mean, "correction" refers to two extra values (the running mean, count) produced by each Mapper i.e., by each PartialAgg instruction. This method is invoked during hop-to-lop translation, while creating the corresponding Aggregate lop Computed information is encoded in the PartialAgg instruction so that the appropriate aggregate operator is used at runtime (see: dml.runtime.matrix.operator.AggregateOperator.java and dml.runtime.matrix)

Returns:

correct location

getCorrectionLocation

public static Types.CorrectionLocationType getCorrectionLocation(Types.AggOp operation,
                                                                 Types.Direction direction)

getAggType

public AggBinaryOp.SparkAggType getAggType()

Description copied from class: Lop

Method to get aggregate type if applicable. This method is overridden by the Lops with aggregate types (e.g. MapMult)

Overrides:

getAggType in class Lop

Returns:

SparkAggType

setDimensionsBasedOnDirection

public void setDimensionsBasedOnDirection(long dim1,
                                          long dim2,
                                          long blen)

setDimensionsBasedOnDirection

public static void setDimensionsBasedOnDirection(Lop lop,
                                                 long dim1,
                                                 long dim2,
                                                 long blen,
                                                 Types.Direction dir)

toString

public String toString()

Description copied from class: Lop

Method to have Lops print their state. This is for debugging purposes.

Specified by:

toString in class Lop

getInstructions

public String getInstructions(String input1,
                              String output)

Instruction generation for CP and Spark

Overrides:

getInstructions in class Lop

Parameters:

input1 - input 1

output - output

Returns:

instructions as string

getOpcode

public static String getOpcode(Types.AggOp op,
                               Types.Direction dir)