Class Checkpoint

  • public class Checkpoint
    extends Lop
    Lop for checkpoint operations. For example, on Spark, the semantic of a checkpoint is to persist an intermediate result into a specific storage level (e.g., mem_only). We use the name checkpoint in terms of cache/persist in Spark (not in terms of checkpoint in Spark streaming) in order to differentiate from CP caching. NOTE: since this class uses spark apis, it should only be instantiated if we are running in execution mode spark (whenever all spark libraries are available)
    • Field Detail


        public static final DEFAULT_STORAGE_LEVEL

        public static final SER_STORAGE_LEVEL

        public static final boolean CHECKPOINT_SPARSE_CSR
        See Also:
        Constant Field Values
    • Constructor Detail

      • Checkpoint

        public Checkpoint​(Lop input,
                          Types.DataType dt,
                          Types.ValueType vt,
                          String level,
                          boolean isAsync)
        TODO change string parameter storage.level to StorageLevel as soon as we can assume that Spark libraries are always available.
        input - low-level operator
        dt - data type
        vt - value type
        level - storage level
        isAsync - true if eager and asynchronous checkpoint
    • Method Detail

      • getStorageLevel

        public getStorageLevel()
      • setStorageLevel

        public void setStorageLevel​( level)
      • toString

        public String toString()
        Description copied from class: Lop
        Method to have Lops print their state. This is for debugging purposes.
        Specified by:
        toString in class Lop
      • getInstructions

        public String getInstructions​(String input1,
                                      String output)
        Description copied from class: Lop
        Method should be overridden if needed
        getInstructions in class Lop
        input1 - input 1
        output - output
        instructions as string
      • getStorageLevelString

        public static String getStorageLevelString​( level)
        This is a utility method because Sparks StorageLevel.toString() is incompatible with its own fromString() method.
        level - RDD storage level
        storage level as a string
      • getDefaultStorageLevelString

        public static String getDefaultStorageLevelString()
      • getSerializeStorageLevelString

        public static String getSerializeStorageLevelString()