Package org.apache.sysds.runtime.controlprogram.paramserv

Class SparkParamservUtils

java.lang.Object

org.apache.sysds.runtime.controlprogram.paramserv.SparkParamservUtils

public class SparkParamservUtils
extends Object

Constructor Summary

Constructors

Constructor	Description
SparkParamservUtils()

Method Summary

Modifier and Type	Method	Description
static org.apache.spark.api.java.JavaPairRDD<Long,scala.Tuple2<MatrixBlock,MatrixBlock>>	assembleTrainingData(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> featuresRDD, org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> labelsRDD)	Assemble the matrix of features and labels according to the rowID
static org.apache.spark.api.java.JavaPairRDD<Integer,scala.Tuple2<MatrixBlock,MatrixBlock>>	doPartitionOnSpark(SparkExecutionContext sec, MatrixObject features, MatrixObject labels, Statement.PSScheme scheme, int workerNum)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkParamservUtils

public SparkParamservUtils()

Method Detail

assembleTrainingData

public static org.apache.spark.api.java.JavaPairRDD<Long,scala.Tuple2<MatrixBlock,MatrixBlock>> assembleTrainingData(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> featuresRDD,
                                                                                                                                 org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> labelsRDD)

Assemble the matrix of features and labels according to the rowID

Parameters:

featuresRDD - indexed features matrix block

labelsRDD - indexed labels matrix block

Returns:

Assembled rdd with rowID as key while matrix of features and labels as value (rowID -> features, labels)

doPartitionOnSpark

public static org.apache.spark.api.java.JavaPairRDD<Integer,scala.Tuple2<MatrixBlock,MatrixBlock>> doPartitionOnSpark(SparkExecutionContext sec,
                                                                                                                                  MatrixObject features,
                                                                                                                                  MatrixObject labels,
                                                                                                                                  Statement.PSScheme scheme,
                                                                                                                                  int workerNum)