Package org.apache.sysds.lops

Class Checkpoint

java.lang.Object

org.apache.sysds.lops.Lop

org.apache.sysds.lops.Checkpoint

public class Checkpoint
extends Lop

Lop for checkpoint operations. For example, on Spark, the semantic of a checkpoint is to persist an intermediate result into a specific storage level (e.g., mem_only). We use the name checkpoint in terms of cache/persist in Spark (not in terms of checkpoint in Spark streaming) in order to differentiate from CP caching. NOTE: since this class uses spark apis, it should only be instantiated if we are running in execution mode spark (whenever all spark libraries are available)

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.sysds.lops.Lop

Lop.SimpleInstType, Lop.Type, Lop.VisitStatus

Field Summary

Fields

Modifier and Type	Field	Description
static String	ASYNC_CP_OPCODE
static boolean	CHECKPOINT_SPARSE_CSR
static String	DEFAULT_CP_OPCODE
static org.apache.spark.storage.StorageLevel	DEFAULT_STORAGE_LEVEL
static org.apache.spark.storage.StorageLevel	SER_STORAGE_LEVEL

Fields inherited from class org.apache.sysds.lops.Lop

_beginColumn, _beginLine, _endColumn, _endLine, _filename, CP_CHILD_THREAD, CP_ROOT_THREAD_ID, DATATYPE_PREFIX, FILE_SEPARATOR, FRAME_VAR_NAME_PREFIX, INSTRUCTION_DELIMITOR, LITERAL_PREFIX, MATRIX_VAR_NAME_PREFIX, NAME_VALUE_SEPARATOR, OPERAND_DELIMITOR, PROCESS_PREFIX, SCALAR_VAR_NAME_PREFIX, UPDATE_INPLACE_PREFIX, VALUETYPE_PREFIX, VARIABLE_NAME_PLACEHOLDER

Constructor Summary

Constructors

Constructor	Description
Checkpoint(Lop input, Types.DataType dt, Types.ValueType vt, String level)
Checkpoint(Lop input, Types.DataType dt, Types.ValueType vt, String level, boolean isAsync)	TODO change string parameter storage.level to StorageLevel as soon as we can assume that Spark libraries are always available.

Method Summary

Modifier and Type	Method	Description
static String	getDefaultStorageLevelString()
String	getInstructions(String input1, String output)	Method should be overridden if needed
static String	getSerializeStorageLevelString()
org.apache.spark.storage.StorageLevel	getStorageLevel()
static String	getStorageLevelString(org.apache.spark.storage.StorageLevel level)	This is a utility method because Sparks StorageLevel.toString() is incompatible with its own fromString() method.
void	setStorageLevel(org.apache.spark.storage.StorageLevel level)
String	toString()	Method to have Lops print their state.

Methods inherited from class org.apache.sysds.lops.Lop

activatePrefetch, addInput, addOutput, addToDag, createReachable, getAggType, getBeginColumn, getBeginLine, getBroadcastInput, getComputeEstimate, getDataType, getEndColumn, getEndLine, getExecType, getFederatedOutput, getFilename, getID, getInput, getInputs, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getInstructions, getLevel, getNnz, getNumCols, getNumRows, getOutputMemoryEstimate, getOutputParameters, getOutputs, getPipelineID, getProducesIntermediateOutput, getReachable, getSimpleInstructionType, getTotalMemoryEstimate, getType, getValueType, getVisited, isAllOutputsCP, isAllOutputsGPU, isAsynchronousOp, isDataExecLocation, isExecCP, isExecGPU, isExecSpark, isVariable, isVisited, prefetchActivated, prepInputOperand, prepInputOperand, prepOperand, prepOperand, prepOutputOperand, prepOutputOperand, prepOutputOperand, prepScalarInputOperand, prepScalarInputOperand, prepScalarLabel, prepScalarOperand, printErrorLocation, removeConsumer, removeInput, removeOutput, replaceAllInputs, replaceAllOutputs, replaceInput, resetVisitStatus, setAllPositions, setAsynchronous, setBeginColumn, setBeginLine, setComputeEstimate, setConsumerCount, setDataType, setEndColumn, setEndLine, setExecType, setFederatedOutput, setFilename, setMemoryEstimates, setNewID, setPipelineID, setValueType, setVisited, setVisited

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_CP_OPCODE

public static final String DEFAULT_CP_OPCODE

See Also:

Constant Field Values

ASYNC_CP_OPCODE

public static final String ASYNC_CP_OPCODE

See Also:

Constant Field Values

DEFAULT_STORAGE_LEVEL

public static final org.apache.spark.storage.StorageLevel DEFAULT_STORAGE_LEVEL

SER_STORAGE_LEVEL

public static final org.apache.spark.storage.StorageLevel SER_STORAGE_LEVEL

CHECKPOINT_SPARSE_CSR

public static final boolean CHECKPOINT_SPARSE_CSR

See Also:

Constant Field Values

Constructor Detail

Checkpoint

public Checkpoint(Lop input,
                  Types.DataType dt,
                  Types.ValueType vt,
                  String level,
                  boolean isAsync)

TODO change string parameter storage.level to StorageLevel as soon as we can assume that Spark libraries are always available.

Parameters:

input - low-level operator

dt - data type

vt - value type

level - storage level

isAsync - true if eager and asynchronous checkpoint

Checkpoint

public Checkpoint(Lop input,
                  Types.DataType dt,
                  Types.ValueType vt,
                  String level)

Method Detail

getStorageLevel

public org.apache.spark.storage.StorageLevel getStorageLevel()

setStorageLevel

public void setStorageLevel(org.apache.spark.storage.StorageLevel level)

toString

public String toString()

Description copied from class: Lop

Method to have Lops print their state. This is for debugging purposes.

Specified by:

toString in class Lop

getInstructions

public String getInstructions(String input1,
                              String output)

Description copied from class: Lop

Method should be overridden if needed

Overrides:

getInstructions in class Lop

Parameters:

input1 - input 1

output - output

Returns:

instructions as string

getStorageLevelString

public static String getStorageLevelString(org.apache.spark.storage.StorageLevel level)

This is a utility method because Sparks StorageLevel.toString() is incompatible with its own fromString() method.

Parameters:

level - RDD storage level

Returns:

storage level as a string

getDefaultStorageLevelString

public static String getDefaultStorageLevelString()

getSerializeStorageLevelString

public static String getSerializeStorageLevelString()