Package org.apache.sysds.hops.recompile

Class Recompiler

java.lang.Object

org.apache.sysds.hops.recompile.Recompiler

public class Recompiler
extends Object

Dynamic recompilation of hop dags to runtime instructions, which includes the following substeps: (1) deep copy hop dag, (2) refresh matrix characteristics, (3) apply dynamic rewrites, (4) refresh memory estimates, (5) construct lops (incl operator selection), and (6) generate runtime program (incl piggybacking).

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	Recompiler.ResetType

Constructor Summary

Constructors

Constructor	Description
Recompiler()

Method Summary

Modifier and Type	Method	Description
static boolean	checkCPCheckpoint(DataCharacteristics dc)
static boolean	checkCPReblock(ExecutionContext ec, String varin)	CP Reblock check for spark instructions; in contrast to MR, we can not rely on the input file sizes because inputs might be passed via rdds.
static ArrayList<Hop>	deepCopyHopsDag(List<Hop> hops)	Deep copy of hops dags for parallel recompilation.
static Hop	deepCopyHopsDag(Hop hops)	Deep copy of hops dags for parallel recompilation.
static void	executeInMemoryReblock(ExecutionContext ec, String varin, String varout)
static void	executeInMemoryReblock(ExecutionContext ec, String varin, String varout, LineageItem litem)
static void	extractDAGOutputStatistics(List<Hop> hops, LocalVariableMap vars)
static void	extractDAGOutputStatistics(List<Hop> hops, LocalVariableMap vars, boolean overwrite)
static void	extractDAGOutputStatistics(Hop hop, LocalVariableMap vars, boolean overwrite)
static void	rClearLops(Hop hop)	Clearing lops for a given hops includes to (1) remove the reference to constructed lops and (2) clear the exec type (for consistency).
static void	recompileFunctionOnceIfNeeded(boolean recompileOnce, List<ProgramBlock> childBlocks, long tid, boolean inplace, Recompiler.ResetType reset, ExecutionContext ec)
static ArrayList<Instruction>	recompileHopsDag(Hop hop, LocalVariableMap vars, RecompileStatus status, boolean inplace, boolean replaceLit, long tid)
static ArrayList<Instruction>	recompileHopsDag(StatementBlock sb, ArrayList<Hop> hops, ExecutionContext ec, RecompileStatus status, boolean inplace, boolean replaceLit, long tid)
static ArrayList<Instruction>	recompileHopsDag(StatementBlock sb, ArrayList<Hop> hops, LocalVariableMap vars, RecompileStatus status, boolean inplace, boolean replaceLit, long tid)
static ArrayList<Instruction>	recompileHopsDag2Forced(Hop hop, long tid, Types.ExecType et)
static ArrayList<Instruction>	recompileHopsDag2Forced(StatementBlock sb, ArrayList<Hop> hops, long tid, Types.ExecType et)
static ArrayList<Instruction>	recompileHopsDagInstructions(Hop hop)
static ArrayList<Instruction>	recompileHopsDagInstructions(StatementBlock sb, ArrayList<Hop> hops)
static void	recompileProgramBlockHierarchy(List<ProgramBlock> pbs, LocalVariableMap vars, long tid, boolean inplace, Recompiler.ResetType resetRecompile)
static void	recompileProgramBlockHierarchy2Forced(ArrayList<ProgramBlock> pbs, long tid, Set<String> fnStack, Types.ExecType et)	Method to recompile program block hierarchy to forced execution time.
static void	recompileProgramBlockInstructions(ProgramBlock pb)	This method does NO full program block recompile (no stats update, no rewrites, no recursion) but only regenerates lops and instructions.
static RecompileStatus	reconcileUpdatedCallVarsIf(RecompileStatus oldStatus, RecompileStatus callStatusIf, RecompileStatus callStatusElse, StatementBlock sb)
static LocalVariableMap	reconcileUpdatedCallVarsIf(LocalVariableMap oldCallVars, LocalVariableMap callVarsIf, LocalVariableMap callVarsElse, StatementBlock sb)
static boolean	reconcileUpdatedCallVarsLoops(RecompileStatus oldCallStatus, RecompileStatus callStatus, StatementBlock sb)
static boolean	reconcileUpdatedCallVarsLoops(LocalVariableMap oldCallVars, LocalVariableMap callVars, StatementBlock sb)
static void	reinitRecompiler()	Re-initializes the recompiler according to the current optimizer flags.
static void	removeUpdatedScalars(LocalVariableMap callVars, StatementBlock sb)	Remove any scalar variables from the variable map if the variable is updated in this block.
static boolean	requiresRecompilation(ArrayList<Hop> hops)
static boolean	requiresRecompilation(Hop hop)
static int	rGetMaxParallelism(List<Hop> hops)
static int	rGetMaxParallelism(Hop hop)
static void	rRecompileProgramBlock2Forced(ProgramBlock pb, long tid, Set<String> fnStack, Types.ExecType et)
static void	rReplaceLiterals(Hop hop, ExecutionContext ec, boolean scalarsOnly)	public interface to package local literal replacement
static void	rReplaceLiterals(Hop hop, LocalVariableMap vars, boolean scalarsOnly)
static void	rSetExecType(Hop hop, Types.ExecType etype)
static void	rSetMaxParallelism(List<Hop> hops, int k)
static void	rSetMaxParallelism(Hop hop, int k)
static void	rUpdateFunctionNames(Hop hop, long pid)
static void	rUpdateStatistics(Hop hop, LocalVariableMap vars)
static void	updateFunctionNames(List<Hop> hops, long pid)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Recompiler

public Recompiler()

Method Detail

reinitRecompiler

public static void reinitRecompiler()

Re-initializes the recompiler according to the current optimizer flags.

recompileHopsDag

public static ArrayList<Instruction> recompileHopsDag(StatementBlock sb,
                                                      ArrayList<Hop> hops,
                                                      ExecutionContext ec,
                                                      RecompileStatus status,
                                                      boolean inplace,
                                                      boolean replaceLit,
                                                      long tid)

recompileHopsDag

public static ArrayList<Instruction> recompileHopsDag(StatementBlock sb,
                                                      ArrayList<Hop> hops,
                                                      LocalVariableMap vars,
                                                      RecompileStatus status,
                                                      boolean inplace,
                                                      boolean replaceLit,
                                                      long tid)

recompileHopsDag

public static ArrayList<Instruction> recompileHopsDag(Hop hop,
                                                      LocalVariableMap vars,
                                                      RecompileStatus status,
                                                      boolean inplace,
                                                      boolean replaceLit,
                                                      long tid)

recompileHopsDag2Forced

public static ArrayList<Instruction> recompileHopsDag2Forced(StatementBlock sb,
                                                             ArrayList<Hop> hops,
                                                             long tid,
                                                             Types.ExecType et)

recompileHopsDag2Forced

public static ArrayList<Instruction> recompileHopsDag2Forced(Hop hop,
                                                             long tid,
                                                             Types.ExecType et)

recompileHopsDagInstructions

public static ArrayList<Instruction> recompileHopsDagInstructions(StatementBlock sb,
                                                                  ArrayList<Hop> hops)

recompileHopsDagInstructions

public static ArrayList<Instruction> recompileHopsDagInstructions(Hop hop)

recompileProgramBlockHierarchy

public static void recompileProgramBlockHierarchy(List<ProgramBlock> pbs,
                                                  LocalVariableMap vars,
                                                  long tid,
                                                  boolean inplace,
                                                  Recompiler.ResetType resetRecompile)

recompileProgramBlockHierarchy2Forced

public static void recompileProgramBlockHierarchy2Forced(ArrayList<ProgramBlock> pbs,
                                                         long tid,
                                                         Set<String> fnStack,
                                                         Types.ExecType et)

Method to recompile program block hierarchy to forced execution time. This affects also referenced functions and chains of functions. Use et==null in order to release the forced exec type.

Parameters:

pbs - list of program blocks

tid - thread id

fnStack - function stack

et - execution type

recompileProgramBlockInstructions

public static void recompileProgramBlockInstructions(ProgramBlock pb)
                                              throws IOException

This method does NO full program block recompile (no stats update, no rewrites, no recursion) but only regenerates lops and instructions. The primary use case is recompilation after are hop configuration changes which allows to preserve statistics (e.g., propagated worst case stats from other program blocks) and better performance for recompiling individual program blocks.

Parameters:

pb - program block

Throws:

IOException - if IOException occurs

requiresRecompilation

public static boolean requiresRecompilation(ArrayList<Hop> hops)

requiresRecompilation

public static boolean requiresRecompilation(Hop hop)

deepCopyHopsDag

public static ArrayList<Hop> deepCopyHopsDag(List<Hop> hops)

Deep copy of hops dags for parallel recompilation.

Parameters:

hops - list of high-level operators

Returns:

list of high-level operators

deepCopyHopsDag

public static Hop deepCopyHopsDag(Hop hops)

Deep copy of hops dags for parallel recompilation.

Parameters:

hops - high-level operator

Returns:

high-level operator

updateFunctionNames

public static void updateFunctionNames(List<Hop> hops,
                                       long pid)

rUpdateFunctionNames

public static void rUpdateFunctionNames(Hop hop,
                                        long pid)

reconcileUpdatedCallVarsLoops

public static boolean reconcileUpdatedCallVarsLoops(LocalVariableMap oldCallVars,
                                                    LocalVariableMap callVars,
                                                    StatementBlock sb)

reconcileUpdatedCallVarsLoops

public static boolean reconcileUpdatedCallVarsLoops(RecompileStatus oldCallStatus,
                                                    RecompileStatus callStatus,
                                                    StatementBlock sb)

reconcileUpdatedCallVarsIf

public static LocalVariableMap reconcileUpdatedCallVarsIf(LocalVariableMap oldCallVars,
                                                          LocalVariableMap callVarsIf,
                                                          LocalVariableMap callVarsElse,
                                                          StatementBlock sb)

reconcileUpdatedCallVarsIf

public static RecompileStatus reconcileUpdatedCallVarsIf(RecompileStatus oldStatus,
                                                         RecompileStatus callStatusIf,
                                                         RecompileStatus callStatusElse,
                                                         StatementBlock sb)

rRecompileProgramBlock2Forced

public static void rRecompileProgramBlock2Forced(ProgramBlock pb,
                                                 long tid,
                                                 Set<String> fnStack,
                                                 Types.ExecType et)

removeUpdatedScalars

public static void removeUpdatedScalars(LocalVariableMap callVars,
                                        StatementBlock sb)

Remove any scalar variables from the variable map if the variable is updated in this block.

Parameters:

callVars - Map of variables eligible for propagation.

sb - DML statement block.

extractDAGOutputStatistics

public static void extractDAGOutputStatistics(List<Hop> hops,
                                              LocalVariableMap vars)

extractDAGOutputStatistics

public static void extractDAGOutputStatistics(List<Hop> hops,
                                              LocalVariableMap vars,
                                              boolean overwrite)

extractDAGOutputStatistics

public static void extractDAGOutputStatistics(Hop hop,
                                              LocalVariableMap vars,
                                              boolean overwrite)

rClearLops

public static void rClearLops(Hop hop)

Clearing lops for a given hops includes to (1) remove the reference to constructed lops and (2) clear the exec type (for consistency). The latter is important for advanced optimizers like parfor; otherwise subtle side-effects of program recompilation and hop-lop rewrites possible (e.g., see indexingop hop-lop rewrite in combination parfor rewrite set exec type that eventuelly might lead to unnecessary remote_parfor jobs).

Parameters:

hop - high-level operator

rUpdateStatistics

public static void rUpdateStatistics(Hop hop,
                                     LocalVariableMap vars)

rReplaceLiterals

public static void rReplaceLiterals(Hop hop,
                                    ExecutionContext ec,
                                    boolean scalarsOnly)

public interface to package local literal replacement

Parameters:

hop - high-level operator

ec - Execution context

scalarsOnly - if true, replace only scalar variables but no matrix operations; if false, apply full literal replacement

rReplaceLiterals

public static void rReplaceLiterals(Hop hop,
                                    LocalVariableMap vars,
                                    boolean scalarsOnly)

rSetExecType

public static void rSetExecType(Hop hop,
                                Types.ExecType etype)

rGetMaxParallelism

public static int rGetMaxParallelism(List<Hop> hops)

rGetMaxParallelism

public static int rGetMaxParallelism(Hop hop)

rSetMaxParallelism

public static void rSetMaxParallelism(List<Hop> hops,
                                      int k)

rSetMaxParallelism

public static void rSetMaxParallelism(Hop hop,
                                      int k)

recompileFunctionOnceIfNeeded

public static void recompileFunctionOnceIfNeeded(boolean recompileOnce,
                                                 List<ProgramBlock> childBlocks,
                                                 long tid,
                                                 boolean inplace,
                                                 Recompiler.ResetType reset,
                                                 ExecutionContext ec)

checkCPReblock

public static boolean checkCPReblock(ExecutionContext ec,
                                     String varin)

CP Reblock check for spark instructions; in contrast to MR, we can not rely on the input file sizes because inputs might be passed via rdds.

Parameters:

ec - execution context

varin - variable

Returns:

true if CP reblock?

checkCPCheckpoint

public static boolean checkCPCheckpoint(DataCharacteristics dc)

executeInMemoryReblock

public static void executeInMemoryReblock(ExecutionContext ec,
                                          String varin,
                                          String varout)

executeInMemoryReblock

public static void executeInMemoryReblock(ExecutionContext ec,
                                          String varin,
                                          String varout,
                                          LineageItem litem)