Package org.apache.sysds.api.mlcontext

Class MLContextUtil

  • public final class MLContextUtil
extends Object
    Utility class containing methods for working with the MLContext API.

    • Field Detail

      • VERSION_NOT_AVAILABLE

        public static final String VERSION_NOT_AVAILABLE
        Version not available message.
        See Also:
        Constant Field Values

      • BUILD_TIME_NOT_AVAILABLE

        public static final String BUILD_TIME_NOT_AVAILABLE
        Build time not available message.
        See Also:
        Constant Field Values

      • BASIC_DATA_TYPES

        public static final Class[] BASIC_DATA_TYPES
        Basic data types supported by the MLContext API.

      • COMPLEX_DATA_TYPES

        public static final Class[] COMPLEX_DATA_TYPES
        Complex data types supported by the MLContext API.

      • ALL_SUPPORTED_DATA_TYPES

        public static final Class[] ALL_SUPPORTED_DATA_TYPES
        All data types supported by the MLContext API.

    • Constructor Detail

      • MLContextUtil

        public MLContextUtil()

    • Method Detail

      • isSparkVersionSupported

        public static boolean isSparkVersionSupported​(String sparkVersion,
                                              String minimumRecommendedSparkVersion)
        Determine whether the Spark version is supported.
        Parameters:
        sparkVersion - Spark version string (ie, "1.5.0").
        minimumRecommendedSparkVersion - Minimum recommended Spark version string (ie, "2.1.0").
        Returns:
        true if Spark version supported; otherwise false.

      • verifySparkVersionSupported

        public static void verifySparkVersionSupported​(org.apache.spark.sql.SparkSession spark)
        Check that the Spark version is supported. If it isn't supported, throw an MLContextException.
        Parameters:
        spark - SparkSession
        Throws:
        MLContextException - thrown if Spark version isn't supported

      • setDefaultConfig

        public static void setDefaultConfig()
        Set default SystemDS configuration properties.

      • setConfig

        public static void setConfig​(String configFilePath)
        Set SystemDS configuration properties based on a configuration file.
        Parameters:
        configFilePath - Path to configuration file.
        Throws:
        MLContextException - if configuration file was not found or a parse exception occurred

      • setCompilerConfig

        public static void setCompilerConfig()
        Set SystemDS compiler configuration properties for MLContext

      • checkInputValueTypes

        public static void checkInputValueTypes​(Map<String,​Object> inputs)
        Verify that the types of input values are supported.
        Parameters:
        inputs - Map of String/Object pairs
        Throws:
        MLContextException - if an input value type is not supported

      • checkInputValueType

        public static void checkInputValueType​(String name,
                                       Object value)
        Verify that the type of input value is supported.
        Parameters:
        name - The name of the input
        value - The value of the input
        Throws:
        MLContextException - if the input value type is not supported

      • checkInputParameterType

        public static void checkInputParameterType​(String parameterName,
                                           Object parameterValue)
        Verify that the type of input parameter value is supported.
        Parameters:
        parameterName - The name of the input parameter
        parameterValue - The value of the input parameter
        Throws:
        MLContextException - if the input parameter value type is not supported

      • isBasicType

        public static boolean isBasicType​(Object object)
        Is the object one of the supported basic data types? (Integer, Boolean, Double, String)
        Parameters:
        object - the object type to be examined
        Returns:
        true if type is a basic data type; otherwise false.

      • getBasicTypeString

        public static String getBasicTypeString​(Object object)
        Obtain the SystemDS scalar value type string equivalent of an accepted basic type (Integer, Boolean, Double, String)
        Parameters:
        object - the object type to be examined
        Returns:
        a String representing the type as a SystemDS scalar value type

      • isComplexType

        public static boolean isComplexType​(Object object)
        Is the object one of the supported complex data types? (JavaRDD, RDD, DataFrame, Matrix, double[][], MatrixBlock, URL)
        Parameters:
        object - the object type to be examined
        Returns:
        true if type is a complex data type; otherwise false.

      • convertInputParametersForParser

        public static Map<String,​String> convertInputParametersForParser​(Map<String,​Object> basicInputParameterMap)
        Converts non-string basic input parameter values to strings to pass to the parser.
        Parameters:
        basicInputParameterMap - map of input parameters
        Returns:
        map of String/String name/value pairs

      • convertInputType

        public static Data convertInputType​(String parameterName,
                                    Object parameterValue)
        Convert input types to internal SystemDS representations
        Parameters:
        parameterName - The name of the input parameter
        parameterValue - The value of the input parameter
        Returns:
        input in SystemDS data representation

      • convertInputType

        public static Data convertInputType​(String parameterName,
                                    Object parameterValue,
                                    Metadata metadata)
        Convert input types to internal SystemDS representations
        Parameters:
        parameterName - The name of the input parameter
        parameterValue - The value of the input parameter
        metadata - matrix/frame metadata
        Returns:
        input in SystemDS data representation

      • isCSVLineAllNumbers

        public static boolean isCSVLineAllNumbers​(String line)
        If no metadata is supplied for an RDD or JavaRDD, this method can be used to determine whether the data appears to be matrix (or a frame)
        Parameters:
        line - a line of the RDD
        Returns:
        true if all the csv-separated values are numbers, false otherwise

      • doesDataFrameLookLikeMatrix

        public static boolean doesDataFrameLookLikeMatrix​(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df)
        Examine the DataFrame schema to determine whether the data appears to be a matrix.
        Parameters:
        df - the DataFrame
        Returns:
        true if the DataFrame appears to be a matrix, false otherwise

      • quotedString

        public static String quotedString​(String str)
        Return a double-quoted string with inner single and double quotes escaped.
        Parameters:
        str - the original string
        Returns:
        double-quoted string with inner single and double quotes escaped

      • displayMap

        public static String displayMap​(String mapName,
                                Map<String,​Object> map)
        Display the keys and values in a Map
        Parameters:
        mapName - the name of the map
        map - Map of String keys and Object values
        Returns:
        the keys and values in the Map as a String

      • displaySet

        public static String displaySet​(String setName,
                                Set<String> set)
        Display the values in a Set
        Parameters:
        setName - the name of the Set
        set - Set of String values
        Returns:
        the values in the Set as a String

      • displaySymbolTable

        public static String displaySymbolTable​(String name,
                                        LocalVariableMap symbolTable)
        Display the keys and values in the symbol table
        Parameters:
        name - the name of the symbol table
        symbolTable - the LocalVariableMap
        Returns:
        the keys and values in the symbol table as a String

      • displaySymbolTable

        public static String displaySymbolTable​(LocalVariableMap symbolTable)
        Display the keys and values in the symbol table
        Parameters:
        symbolTable - the LocalVariableMap
        Returns:
        the keys and values in the symbol table as a String

      • determineOutputTypeAsString

        public static String determineOutputTypeAsString​(LocalVariableMap symbolTable,
                                                 String outputName)
        Obtain a symbol table output type as a String
        Parameters:
        symbolTable - the symbol table
        outputName - the name of the output variable
        Returns:
        the symbol table output type for a variable as a String

      • displayInputs

        public static String displayInputs​(String name,
                                   Map<String,​Object> map,
                                   LocalVariableMap symbolTable)
        Obtain a display of script inputs.
        Parameters:
        name - the title to display for the inputs
        map - the map of inputs
        symbolTable - the symbol table
        Returns:
        the script inputs represented as a String

      • displayOutputs

        public static String displayOutputs​(String name,
                                    Set<String> outputNames,
                                    LocalVariableMap symbolTable)
        Obtain a display of the script outputs.
        Parameters:
        name - the title to display for the outputs
        outputNames - the names of the output variables
        symbolTable - the symbol table
        Returns:
        the script outputs represented as a String

      • displayOutputs

        public static String displayOutputs​(Set<String> outputNames,
                                    LocalVariableMap symbolTable)
        Obtain a display of the script outputs.
        Parameters:
        outputNames - the names of the output variables
        symbolTable - the symbol table
        Returns:
        the script outputs represented as a String

      • welcomeMessage

        public static String welcomeMessage()
        The SystemDS welcome message
        Returns:
        the SystemDS welcome message

      • getSparkContext

        public static org.apache.spark.SparkContext getSparkContext​(MLContext mlContext)
        Obtain the Spark Context
        Parameters:
        mlContext - the SystemDS MLContext
        Returns:
        the Spark Context

      • getJavaSparkContext

        public static org.apache.spark.api.java.JavaSparkContext getJavaSparkContext​(MLContext mlContext)
        Obtain the Java Spark Context
        Parameters:
        mlContext - the SystemDS MLContext
        Returns:
        the Java Spark Context

      • getSparkContextFromProxy

        public static org.apache.spark.SparkContext getSparkContextFromProxy()
        Obtain the Spark Context from the MLContextProxy
        Returns:
        the Spark Context

      • getJavaSparkContextFromProxy

        public static org.apache.spark.api.java.JavaSparkContext getJavaSparkContextFromProxy()
        Obtain the Java Spark Context from the MLContextProxy
        Returns:
        the Java Spark Context

      • getSparkSessionFromProxy

        public static org.apache.spark.sql.SparkSession getSparkSessionFromProxy()
        Obtain the Spark Session from the MLContextProxy
        Returns:
        the Spark Session

      • doesSymbolTableContainFrameObject

        public static boolean doesSymbolTableContainFrameObject​(LocalVariableMap symbolTable,
                                                        String variableName)
        Determine if the symbol table contains a FrameObject with the given variable name.
        Parameters:
        symbolTable - the LocalVariableMap
        variableName - the variable name
        Returns:
        true if the variable in the symbol table is a FrameObject, false otherwise.

      • doesSymbolTableContainMatrixObject

        public static boolean doesSymbolTableContainMatrixObject​(LocalVariableMap symbolTable,
                                                         String variableName)
        Determine if the symbol table contains a MatrixObject with the given variable name.
        Parameters:
        symbolTable - the LocalVariableMap
        variableName - the variable name
        Returns:
        true if the variable in the symbol table is a MatrixObject, false otherwise.

      • deleteRemoveVariableInstructions

        public static void deleteRemoveVariableInstructions​(Program progam)
        Delete the 'remove variable' instructions from a runtime program.
        Parameters:
        progam - runtime program