Class Connection

    • Constructor Detail

      • Connection

        public Connection()
        Connection constructor, the starting point for any other JMLC API calls.
      • Connection

        public Connection​(CompilerConfig.ConfigType... cconfigs)
        Connection constructor, the starting point for any other JMLC API calls. This variant allows to enable a set of boolean compiler configurations.
        Parameters:
        cconfigs - one or many boolean compiler configurations to enable.
      • Connection

        public Connection​(DMLConfig dmlconfig,
                          CompilerConfig.ConfigType... cconfigs)
        Connection constructor, the starting point for any other JMLC API calls. This variant allows to pass a global dml configuration and enable a set of boolean compiler configurations.
        Parameters:
        dmlconfig - a dml configuration.
        cconfigs - one or many boolean compiler configurations to enable.
      • Connection

        public Connection​(DMLConfig dmlconfig)
        Connection constructor, the starting point for any other JMLC API calls. This variant allows to pass a global dml configuration.
        Parameters:
        dmlconfig - a dml configuration.
    • Method Detail

      • setConfigTypes

        public void setConfigTypes​(boolean activate,
                                   CompilerConfig.ConfigType... cconfigs)
        Sets compiler configs.
        Parameters:
        activate - activate or disable
        cconfigs - the configs to set
      • setStatistics

        public void setStatistics​(boolean stats)
        Sets a boolean flag indicating if runtime statistics should be gathered Same behavior as in "MLContext.setStatistics()"
        Parameters:
        stats - boolean value with true indicating statistics should be gathered
      • setLineage

        public void setLineage​(boolean lt)
        Sets a boolean flag indicating if lineage trace should be captured
        Parameters:
        lt - boolean value with true indicating lineage should be captured
      • gatherMemStats

        public void gatherMemStats​(boolean stats)
        Sets a boolean flag indicating if memory profiling statistics should be gathered. The option is false by default.
        Parameters:
        stats - boolean value with true indicating memory statistics should be gathered
      • prepareScript

        public PreparedScript prepareScript​(String script,
                                            String[] inputs,
                                            String[] outputs)
        Prepares (precompiles) a script and registers input and output variables.
        Parameters:
        script - string representing the DML or PyDML script
        inputs - string array of input variables to register
        outputs - string array of output variables to register
        Returns:
        PreparedScript object representing the precompiled script
      • prepareScript

        public PreparedScript prepareScript​(String script,
                                            Map<String,​String> args,
                                            String[] inputs,
                                            String[] outputs)
        Prepares (precompiles) a script, sets input parameter values, and registers input and output variables.
        Parameters:
        script - string representing the DML or PyDML script
        args - map of input parameters ($) and their values
        inputs - string array of input variables to register
        outputs - string array of output variables to register
        Returns:
        PreparedScript object representing the precompiled script
      • prepareScript

        public PreparedScript prepareScript​(String script,
                                            Map<String,​String> nsscripts,
                                            Map<String,​String> args,
                                            String[] inputs,
                                            String[] outputs)
        Prepares (precompiles) a script, sets input parameter values, and registers input and output variables.
        Parameters:
        script - string representing of the DML or PyDML script
        nsscripts - map (name, script) of the DML or PyDML namespace scripts
        args - map of input parameters ($) and their values
        inputs - string array of input variables to register
        outputs - string array of output variables to register
        Returns:
        PreparedScript object representing the precompiled script
      • close

        public void close()
        Close connection to SystemDS, which clears the thread-local DML and compiler configurations.
        Specified by:
        close in interface AutoCloseable
        Specified by:
        close in interface Closeable
      • readScript

        public String readScript​(String fname)
                          throws IOException
        Read a DML or PyDML file as a string.
        Parameters:
        fname - the filename of the script
        Returns:
        string content of the script file
        Throws:
        IOException - if IOException occurs
      • readDoubleMatrix

        public double[][] readDoubleMatrix​(String fname)
                                    throws IOException
        Reads an input matrix in arbitrary format from HDFS into a dense double array. NOTE: this call currently only supports default configurations for CSV.
        Parameters:
        fname - the filename of the input matrix
        Returns:
        matrix as a two-dimensional double array
        Throws:
        IOException - if IOException occurs
      • readDoubleMatrix

        public double[][] readDoubleMatrix​(String fname,
                                           Types.FileFormat fmt,
                                           long rows,
                                           long cols,
                                           int blen,
                                           long nnz)
                                    throws IOException
        Reads an input matrix in arbitrary format from HDFS into a dense double array. NOTE: this call currently only supports default configurations for CSV.
        Parameters:
        fname - the filename of the input matrix
        fmt - file format
        rows - number of rows in the matrix
        cols - number of columns in the matrix
        blen - block length
        nnz - number of non-zero values, -1 indicates unknown
        Returns:
        matrix as a two-dimensional double array
        Throws:
        IOException - if IOException occurs
      • convertToDoubleMatrix

        public double[][] convertToDoubleMatrix​(String input,
                                                String meta)
                                         throws IOException
        Converts an input string representation of a matrix in csv or textcell format into a dense double array. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.
        Parameters:
        input - string matrix in csv or textcell format
        meta - string representing SystemDS matrix metadata in JSON format
        Returns:
        matrix as a two-dimensional double array
        Throws:
        IOException - if IOException occurs
      • convertToDoubleMatrix

        public double[][] convertToDoubleMatrix​(String input,
                                                int rows,
                                                int cols)
                                         throws IOException
        Converts an input string representation of a matrix in textcell format into a dense double array.
        Parameters:
        input - string matrix in textcell format
        rows - number of rows in the matrix
        cols - number of columns in the matrix
        Returns:
        matrix as a two-dimensional double array
        Throws:
        IOException - if IOException occurs
      • convertToDoubleMatrix

        public double[][] convertToDoubleMatrix​(InputStream input,
                                                int rows,
                                                int cols)
                                         throws IOException
        Converts an input stream of a string matrix in textcell format into a dense double array.
        Parameters:
        input - InputStream to a string matrix in textcell format
        rows - number of rows in the matrix
        cols - number of columns in the matrix
        Returns:
        matrix as a two-dimensional double array
        Throws:
        IOException - if IOException occurs
      • convertToDoubleMatrix

        public double[][] convertToDoubleMatrix​(InputStream input,
                                                int rows,
                                                int cols,
                                                String format)
                                         throws IOException
        Converts an input stream of a string matrix in csv or textcell format into a dense double array.
        Parameters:
        input - InputStream to a string matrix in csv or textcell format
        rows - number of rows in the matrix
        cols - number of columns in the matrix
        format - input format of the given stream
        Returns:
        matrix as a two-dimensional double array
        Throws:
        IOException - if IOException occurs
      • convertToMatrix

        public MatrixBlock convertToMatrix​(String input,
                                           String meta)
                                    throws IOException
        Converts an input string representation of a matrix in csv or textcell format into a matrix block. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.
        Parameters:
        input - string matrix in csv or textcell format
        meta - string representing SystemDS matrix metadata in JSON format
        Returns:
        matrix as a matrix block
        Throws:
        IOException - if IOException occurs
      • convertToMatrix

        public MatrixBlock convertToMatrix​(InputStream input,
                                           String meta)
                                    throws IOException
        Converts an input stream of a string matrix in csv or textcell format into a matrix block. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.
        Parameters:
        input - InputStream to a string matrix in csv or textcell format
        meta - string representing SystemDS matrix metadata in JSON format
        Returns:
        matrix as a matrix block
        Throws:
        IOException - if IOException occurs
      • convertToMatrix

        public MatrixBlock convertToMatrix​(String input,
                                           int rows,
                                           int cols)
                                    throws IOException
        Converts an input string representation of a matrix in textcell format into a matrix block.
        Parameters:
        input - string matrix in textcell format
        rows - number of rows in the matrix
        cols - number of columns in the matrix
        Returns:
        matrix as a matrix block
        Throws:
        IOException - if IOException occurs
      • convertToMatrix

        public MatrixBlock convertToMatrix​(InputStream input,
                                           int rows,
                                           int cols)
                                    throws IOException
        Converts an input stream of a string matrix in text format into a matrix block.
        Parameters:
        input - InputStream to a string matrix in text format
        rows - number of rows in the matrix
        cols - number of columns in the matrix
        Returns:
        matrix as a matrix block
        Throws:
        IOException - if IOException occurs
      • convertToMatrix

        public MatrixBlock convertToMatrix​(InputStream input,
                                           int rows,
                                           int cols,
                                           String format)
                                    throws IOException
        Converts an input stream of a string matrix in csv or text format into a matrix block.
        Parameters:
        input - InputStream to a string matrix in csv or text format
        rows - number of rows in the matrix
        cols - number of columns in the matrix
        format - input format of the given stream
        Returns:
        matrix as a matrix block
        Throws:
        IOException - if IOException occurs
      • readStringFrame

        public String[][] readStringFrame​(String fname)
                                   throws IOException
        Reads an input frame in arbitrary format from HDFS into a dense string array. NOTE: this call currently only supports default configurations for CSV.
        Parameters:
        fname - the filename of the input frame
        Returns:
        frame as a two-dimensional string array
        Throws:
        IOException - if IOException occurs
      • readStringFrame

        public String[][] readStringFrame​(String fname,
                                          Types.FileFormat fmt,
                                          long rows,
                                          long cols)
                                   throws IOException
        Reads an input frame in arbitrary format from HDFS into a dense string array. NOTE: this call currently only supports default configurations for CSV.
        Parameters:
        fname - the filename of the input frame
        fmt - file format type
        rows - number of rows in the frame
        cols - number of columns in the frame
        Returns:
        frame as a two-dimensional string array
        Throws:
        IOException - if IOException occurs
      • convertToStringFrame

        public String[][] convertToStringFrame​(String input,
                                               String meta)
                                        throws IOException
        Converts an input string representation of a frame in csv or textcell format into a dense string array. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.
        Parameters:
        input - string frame in csv or textcell format
        meta - string representing SystemDS frame metadata in JSON format
        Returns:
        frame as a two-dimensional string array
        Throws:
        IOException - if IOException occurs
      • convertToStringFrame

        public String[][] convertToStringFrame​(String input,
                                               int rows,
                                               int cols)
                                        throws IOException
        Converts an input stream of a string frame in textcell format into a dense string array.
        Parameters:
        input - string frame in textcell format
        rows - number of rows in the frame
        cols - number of columns in the frame
        Returns:
        frame as a two-dimensional string array
        Throws:
        IOException - if IOException occurs
      • convertToStringFrame

        public String[][] convertToStringFrame​(InputStream input,
                                               int rows,
                                               int cols)
                                        throws IOException
        Converts an input stream of a string frame in textcell format into a dense string array.
        Parameters:
        input - InputStream to a string frame in text format
        rows - number of rows in the frame
        cols - number of columns in the frame
        Returns:
        frame as a two-dimensional string array
        Throws:
        IOException - if IOException occurs
      • convertToStringFrame

        public String[][] convertToStringFrame​(InputStream input,
                                               int rows,
                                               int cols,
                                               String format)
                                        throws IOException
        Converts an input stream of a string frame in csv or text format into a dense string array.
        Parameters:
        input - InputStream to a string frame in csv or text format
        rows - number of rows in the frame
        cols - number of columns in the frame
        format - input format of the given stream
        Returns:
        frame as a two-dimensional string array
        Throws:
        IOException - if IOException occurs
      • convertToFrame

        public FrameBlock convertToFrame​(String input,
                                         String meta)
                                  throws IOException
        Converts an input string representation of a frame in csv or textcell format into a frame block. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.
        Parameters:
        input - string frame in csv or textcell format
        meta - string representing SystemDS frame metadata in JSON format
        Returns:
        frame as a frame block
        Throws:
        IOException - if IOException occurs
      • convertToFrame

        public FrameBlock convertToFrame​(InputStream input,
                                         String meta)
                                  throws IOException
        Converts an input stream of a string frame in csv or textcell format into a frame block. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.
        Parameters:
        input - InputStream to a string frame in csv or textcell format
        meta - string representing SystemDS frame metadata in JSON format
        Returns:
        frame as a frame block
        Throws:
        IOException - if IOException occurs
      • convertToFrame

        public FrameBlock convertToFrame​(String input,
                                         int rows,
                                         int cols)
                                  throws IOException
        Converts an input string representation of a frame in textcell format into a frame block.
        Parameters:
        input - string frame in textcell format
        rows - number of rows in the frame
        cols - number of columns in the frame
        Returns:
        frame as a frame block
        Throws:
        IOException - if IOException occurs
      • convertToFrame

        public FrameBlock convertToFrame​(InputStream input,
                                         int rows,
                                         int cols)
                                  throws IOException
        Converts an input stream of a string frame in textcell format into a frame block.
        Parameters:
        input - InputStream to a string frame in textcell format
        rows - number of rows in the frame
        cols - number of columns in the frame
        Returns:
        frame as a frame block
        Throws:
        IOException - if IOException occurs
      • convertToFrame

        public FrameBlock convertToFrame​(InputStream input,
                                         int rows,
                                         int cols,
                                         String format)
                                  throws IOException
        Converts an input stream of a frame in csv or textcell format into a frame block.
        Parameters:
        input - InputStream to a string frame in csv or textcell format
        rows - number of rows in the frame
        cols - number of columns in the frame
        format - input format of the given stream
        Returns:
        frame as a frame block
        Throws:
        IOException - if IOException occurs
      • readTransformMetaDataFromFile

        public FrameBlock readTransformMetaDataFromFile​(String metapath)
                                                 throws IOException
        Reads transform meta data from an HDFS file path and converts it into an in-memory FrameBlock object. The column names in the meta data file 'column.names' is processed with default separator ','.
        Parameters:
        metapath - hdfs file path to meta data directory
        Returns:
        FrameBlock object representing transform metadata
        Throws:
        IOException - if IOException occurs
      • readTransformMetaDataFromFile

        public FrameBlock readTransformMetaDataFromFile​(String spec,
                                                        String metapath)
                                                 throws IOException
        Reads transform meta data from an HDFS file path and converts it into an in-memory FrameBlock object. The column names in the meta data file 'column.names' is processed with default separator ','.
        Parameters:
        spec - transform specification as json string
        metapath - hdfs file path to meta data directory
        Returns:
        FrameBlock object representing transform metadata
        Throws:
        IOException - if IOException occurs
      • readTransformMetaDataFromFile

        public FrameBlock readTransformMetaDataFromFile​(String spec,
                                                        String metapath,
                                                        String colDelim)
                                                 throws IOException
        Reads transform meta data from an HDFS file path and converts it into an in-memory FrameBlock object.
        Parameters:
        spec - transform specification as json string
        metapath - hdfs file path to meta data directory
        colDelim - separator for processing column names in the meta data file 'column.names'
        Returns:
        FrameBlock object representing transform metadata
        Throws:
        IOException - if IOException occurs
      • readTransformMetaDataFromPath

        public FrameBlock readTransformMetaDataFromPath​(String metapath)
                                                 throws IOException
        Reads transform meta data from the class path and converts it into an in-memory FrameBlock object. The column names in the meta data file 'column.names' is processed with default separator ','.
        Parameters:
        metapath - resource path to meta data directory
        Returns:
        FrameBlock object representing transform metadata
        Throws:
        IOException - if IOException occurs
      • readTransformMetaDataFromPath

        public FrameBlock readTransformMetaDataFromPath​(String spec,
                                                        String metapath)
                                                 throws IOException
        Reads transform meta data from the class path and converts it into an in-memory FrameBlock object. The column names in the meta data file 'column.names' is processed with default separator ','.
        Parameters:
        spec - transform specification as json string
        metapath - resource path to meta data directory
        Returns:
        FrameBlock object representing transform metadata
        Throws:
        IOException - if IOException occurs
      • readTransformMetaDataFromPath

        public FrameBlock readTransformMetaDataFromPath​(String spec,
                                                        String metapath,
                                                        String colDelim)
                                                 throws IOException
        Reads transform meta data from the class path and converts it into an in-memory FrameBlock object.
        Parameters:
        spec - transform specification as json string
        metapath - resource path to meta data directory
        colDelim - separator for processing column names in the meta data file 'column.names'
        Returns:
        FrameBlock object representing transform metadata
        Throws:
        IOException - if IOException occurs