Package org.apache.sysds.runtime.io

Class ReaderWriterFederated

  • public class ReaderWriterFederated
extends Object
    This class serves as the reader for federated objects. To read the files a mdt file is required. The reader is different from the other readers in the since that it does not return a MatrixBlock but a Matrix Object wrapper, containing the federated Mapping. On the Matrix Object the function isFederated() will if called read in the federated locations and instantiate the map. The reading is done through this code. This means in practice that it circumvent the other reading code. See more in: org.apache.sysds.runtime.controlprogram.caching.MatrixObject.readBlobFromHDFS() org.apache.sysds.runtime.controlprogram.caching.CacheableData.isFederated()

    • Constructor Detail

      • ReaderWriterFederated

        public ReaderWriterFederated()

    • Method Detail

      • read

        public static List<org.apache.commons.lang3.tuple.Pair<FederatedRange,​FederatedData>> read​(String file,
                                                                                                 DataCharacteristics mc)
        Read a federated map from disk, It is not initialized before it is used in: org.apache.sysds.runtime.instructions.fed.InitFEDInstruction
        Parameters:
        file - The file to read (defaults to HDFS)
        mc - The data characteristics of the file, that can be read from the mtd file.
        Returns:
        A List of federatedRanges and Federated Data

      • write

        public static void write​(String file,
                         FederationMap fedMap)
        TODO add writing to each of the federated locations so that they also save their matrices. Currently this would write the federated matrix to disk only locally.
        Parameters:
        file - The file to save to, (defaults to HDFS paths)
        fedMap - The federated map to save.