Package org.apache.sysds.lops
Class PartialAggregate
- java.lang.Object
-
- org.apache.sysds.lops.Lop
-
- org.apache.sysds.lops.PartialAggregate
-
public class PartialAggregate extends Lop
Lop to perform a partial aggregation. It was introduced to do some initial aggregation operations on blocks in the mapper/reducer.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.sysds.lops.Lop
Lop.SimpleInstType, Lop.Type, Lop.VisitStatus
-
-
Field Summary
-
Fields inherited from class org.apache.sysds.lops.Lop
_beginColumn, _beginLine, _endColumn, _endLine, _filename, CP_CHILD_THREAD, CP_ROOT_THREAD_ID, DATATYPE_PREFIX, FILE_SEPARATOR, FRAME_VAR_NAME_PREFIX, INSTRUCTION_DELIMITOR, LITERAL_PREFIX, MATRIX_VAR_NAME_PREFIX, NAME_VALUE_SEPARATOR, OPERAND_DELIMITOR, PROCESS_PREFIX, SCALAR_VAR_NAME_PREFIX, UPDATE_INPLACE_PREFIX, VALUETYPE_PREFIX, VARIABLE_NAME_PLACEHOLDER
-
-
Constructor Summary
Constructors Constructor Description PartialAggregate(Lop input, Types.AggOp op, Types.Direction direct, Types.DataType dt, Types.ValueType vt, Types.ExecType et, int k)
PartialAggregate(Lop input, Types.AggOp op, Types.Direction direct, Types.DataType dt, Types.ValueType vt, AggBinaryOp.SparkAggType aggtype, Types.ExecType et)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description AggBinaryOp.SparkAggType
getAggType()
Method to get aggregate type if applicable.Types.CorrectionLocationType
getCorrectionLocation()
This method computes the location of "correction" terms in the output produced by PartialAgg instruction.static Types.CorrectionLocationType
getCorrectionLocation(Types.AggOp operation, Types.Direction direction)
String
getInstructions(String input1, String output)
Instruction generation for CP and Sparkstatic String
getOpcode(Types.AggOp op, Types.Direction dir)
void
setDimensionsBasedOnDirection(long dim1, long dim2, long blen)
static void
setDimensionsBasedOnDirection(Lop lop, long dim1, long dim2, long blen, Types.Direction dir)
String
toString()
Method to have Lops print their state.-
Methods inherited from class org.apache.sysds.lops.Lop
activatePrefetch, addInput, addOutput, addToDag, createReachable, getBeginColumn, getBeginLine, getBroadcastInput, getComputeEstimate, getDataType, getEndColumn, getEndLine, getExecType, getFederatedOutput, getFilename, getID, getInput, getInputs, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getLevel, getNnz, getNumCols, getNumRows, getOutputMemoryEstimate, getOutputParameters, getOutputs, getPipelineID, getProducesIntermediateOutput, getReachable, getSimpleInstructionType, getTotalMemoryEstimate, getType, getValueType, getVisited, isAllOutputsCP, isAllOutputsGPU, isAsynchronousOp, isDataExecLocation, isExecCP, isExecGPU, isExecSpark, isVariable, isVisited, prefetchActivated, prepInputOperand, prepInputOperand, prepOperand, prepOperand, prepOutputOperand, prepOutputOperand, prepOutputOperand, prepScalarInputOperand, prepScalarInputOperand, prepScalarLabel, prepScalarOperand, printErrorLocation, removeConsumer, removeInput, removeOutput, replaceAllInputs, replaceAllOutputs, replaceInput, resetVisitStatus, setAllPositions, setAsynchronous, setBeginColumn, setBeginLine, setComputeEstimate, setConsumerCount, setDataType, setEndColumn, setEndLine, setExecType, setFederatedOutput, setFilename, setMemoryEstimates, setNewID, setPipelineID, setValueType, setVisited, setVisited
-
-
-
-
Constructor Detail
-
PartialAggregate
public PartialAggregate(Lop input, Types.AggOp op, Types.Direction direct, Types.DataType dt, Types.ValueType vt, Types.ExecType et, int k)
-
PartialAggregate
public PartialAggregate(Lop input, Types.AggOp op, Types.Direction direct, Types.DataType dt, Types.ValueType vt, AggBinaryOp.SparkAggType aggtype, Types.ExecType et)
-
-
Method Detail
-
getCorrectionLocation
public Types.CorrectionLocationType getCorrectionLocation()
This method computes the location of "correction" terms in the output produced by PartialAgg instruction. When computing the stable sum, "correction" refers to the compensation as defined by the original Kahan algorithm. When computing the stable mean, "correction" refers to two extra values (the running mean, count) produced by each Mapper i.e., by each PartialAgg instruction. This method is invoked during hop-to-lop translation, while creating the corresponding Aggregate lop Computed information is encoded in the PartialAgg instruction so that the appropriate aggregate operator is used at runtime (see: dml.runtime.matrix.operator.AggregateOperator.java and dml.runtime.matrix)- Returns:
- correct location
-
getCorrectionLocation
public static Types.CorrectionLocationType getCorrectionLocation(Types.AggOp operation, Types.Direction direction)
-
getAggType
public AggBinaryOp.SparkAggType getAggType()
Description copied from class:Lop
Method to get aggregate type if applicable. This method is overridden by the Lops with aggregate types (e.g. MapMult)- Overrides:
getAggType
in classLop
- Returns:
- SparkAggType
-
setDimensionsBasedOnDirection
public void setDimensionsBasedOnDirection(long dim1, long dim2, long blen)
-
setDimensionsBasedOnDirection
public static void setDimensionsBasedOnDirection(Lop lop, long dim1, long dim2, long blen, Types.Direction dir)
-
toString
public String toString()
Description copied from class:Lop
Method to have Lops print their state. This is for debugging purposes.
-
getInstructions
public String getInstructions(String input1, String output)
Instruction generation for CP and Spark- Overrides:
getInstructions
in classLop
- Parameters:
input1
- input 1output
- output- Returns:
- instructions as string
-
getOpcode
public static String getOpcode(Types.AggOp op, Types.Direction dir)
-
-