Package org.apache.sysds.runtime.frame.data.columns

Class RaggedArray<T>

java.lang.Object

org.apache.sysds.runtime.frame.data.columns.Array<T>

org.apache.sysds.runtime.frame.data.columns.RaggedArray<T>

All Implemented Interfaces:

org.apache.hadoop.io.Writable

public class RaggedArray<T>
extends Array<T>

A Ragged array for a single column contains a smaller array, only containing the values of the top most part of the This makes the allocation much better in cases where only the top n rows of a m row frame are used for the specific column. It is typically used for instances of transform encode, where the transform encode return a metadata frame to enable encoding and decoding the matrix.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.sysds.runtime.frame.data.columns.Array

Array.ArrayIterator

Constructor Summary

Constructors

Constructor	Description
RaggedArray(Array<T> a, int m)	The allocation where, a's length is shorter than m, and we handle all accesses above len(a) as null.
RaggedArray(T[] a, int m)	The allocation where, a's length is shorter than m, and we handle all accesses above len(a) as null.

Method Summary

Modifier and Type	Method	Description
Pair<Types.ValueType,Boolean>	analyzeValueType(int maxCells)	Analyze the column to figure out if the value type can be refined to a better type.
void	append(String value)	Append a string value to the current Array, this should in general be avoided, and appending larger blocks at a time should be preferred.
Array<T>	append(Array<T> other)	Append other array, if the other array is fitting in current allocated size use that allocated size, otherwise allocate new array to combine the other with this.
void	append(T value)	Append a value of the same type of the Array.
Array<?>	changeTypeWithNulls(Types.ValueType t)
Array<T>	clone()	Overwrite of the java internal clone function for arrays, return a clone of underlying data that is mutable, (not immutable data.) Immutable data is dependent on the individual allocated arrays
boolean	containsNull()	analyze if the array contains null values.
boolean	equals(Array<T> other)	Equals operation on arrays.
void	fill(String val)	fill the entire array with specific value.
void	fill(T val)	fill the entire array with specific value.
Object	get()	Get the underlying array out of the column Group, it is the responsibility of the caller to know what type it is.
T	get(int index)	Get the value at a given index.
byte[]	getAsByteArray()	Return the current allocated Array as a byte[], this is used to serialize the allocated Arrays out to the PythonAPI.
double	getAsDouble(int i)	Get the index's value.
double	getAsNaNDouble(int i)	Get the index's value.
long	getExactSerializedSize()	Get the exact serialized size on disk of this array.
ArrayFactory.FrameArrayType	getFrameArrayType()	Get the internal FrameArrayType, to specify the encoding of the Types, note there are more Frame Array Types than there is ValueTypes.
long	getInMemorySize()	Get in memory size, not counting reference to this object.
Types.ValueType	getValueType()	Get the current value type of this array.
double	hashDouble(int idx)	Hash the given index of the array.
boolean	isEmpty()	Get if this array is empty, aka filled with empty values.
boolean	isNotEmpty(int i)
boolean	isShallowSerialize()	analyze if this array can be shallow serialized.
boolean	possiblyContainsNaN()
void	readFields(DataInput in)
void	reset(int size)	Reset the Array and set to a different size.
Array<T>	select(boolean[] select, int nTrue)	Slice out the true indices in the select input and return the sub array.
Array<T>	select(int[] indices)	Slice out the specified indices and return the sub array.
void	set(int index, double value)	Set index to given double value (cast to the correct type of this array)
void	set(int rl, int ru, Array<T> value)	Set range to given arrays value
void	set(int rl, int ru, Array<T> value, int rlSrc)	Set range to given arrays value with an offset into other array
void	set(int index, String value)	Set index to the given value of the string parsed.
void	set(int index, T value)	Set index to the given value of same type
void	setFromOtherType(int rl, int ru, Array<?> value)	Set range to given arrays value
void	setFromOtherTypeNz(int rl, int ru, Array<?> value)	Set non default values in the range from the value array given
void	setNz(int rl, int ru, Array<T> value)	Set non default values in the range from the value array given
Array<T>	slice(int rl, int ru)	Slice out the sub range and return new array with the specified type.
String	toString()
void	write(DataOutput out)

Methods inherited from class org.apache.sysds.runtime.frame.data.columns.Array

analyzeValueType, baseMemoryCost, changeType, changeType, changeType, changeType, changeTypeWithNulls, changeTypeWithNulls, equals, extractDouble, findEmpty, findEmptyInverse, getCache, getInternal, getIterator, getMinMaxLength, getNulls, getRecodeMap, minMax, minMax, setCache, setFromOtherTypeNz, setNz, size, statistics

Methods inherited from class java.lang.Object

getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

RaggedArray

public RaggedArray(T[] a,
                   int m)

The allocation where, a's length is shorter than m, and we handle all accesses above len(a) as null.

Parameters:

a - The underlying array that is shorter than length m

m - The overall supported length m

RaggedArray

public RaggedArray(Array<T> a,
                   int m)

The allocation where, a's length is shorter than m, and we handle all accesses above len(a) as null.

Parameters:

a - The underlying array that is shorter than length m

m - The overall supported length m

Method Detail

write

public void write(DataOutput out)
           throws IOException

Throws:

IOException

readFields

public void readFields(DataInput in)
                throws IOException

Throws:

IOException

get

public T get(int index)

Description copied from class: Array

Get the value at a given index. This method returns objects that have a high overhead in allocation. Therefore it is not as efficient as using the vectorized operations specified in the object.

Specified by:

get in class Array<T>

Parameters:

index - The index to query

Returns:

The value returned as an object

get

public Object get()

Description copied from class: Array

Get the underlying array out of the column Group, it is the responsibility of the caller to know what type it is. Also it is not guaranteed that the underlying data structure does not allocate an appropriate response to the caller. This in practice means that if called there is a possibility that the entire array is allocated again. So the method should only be used for debugging purposes not for performance.

Specified by:

get in class Array<T>

Returns:

The underlying array.

getAsDouble

public double getAsDouble(int i)

Description copied from class: Array

Get the index's value. returns 0 in case of Null.

Specified by:

getAsDouble in class Array<T>

Parameters:

i - index to get value from

Returns:

the value

getAsNaNDouble

public double getAsNaNDouble(int i)

Description copied from class: Array

Get the index's value. returns Double.NaN in case of Null.

Overrides:

getAsNaNDouble in class Array<T>

Parameters:

i - index to get value from

Returns:

the value

set

public void set(int index,
                T value)

Description copied from class: Array

Set index to the given value of same type

Specified by:

set in class Array<T>

Parameters:

index - The index to set

value - The value to assign

set

public void set(int index,
                double value)

Description copied from class: Array

Set index to given double value (cast to the correct type of this array)

Specified by:

set in class Array<T>

Parameters:

index - the index to set

value - the value to set it to (before casting to correct value type)

set

public void set(int index,
                String value)

Description copied from class: Array

Set index to the given value of the string parsed.

Specified by:

set in class Array<T>

Parameters:

index - The index to set

value - The value to assign

setFromOtherType

public void setFromOtherType(int rl,
                             int ru,
                             Array<?> value)

Description copied from class: Array

Set range to given arrays value

Specified by:

setFromOtherType in class Array<T>

Parameters:

rl - row lower

ru - row upper (inclusive)

value - value array to take values from (other type)

set

public void set(int rl,
                int ru,
                Array<T> value)

Description copied from class: Array

Set range to given arrays value

Specified by:

set in class Array<T>

Parameters:

rl - row lower

ru - row upper (inclusive)

value - value array to take values from (same type)

set

public void set(int rl,
                int ru,
                Array<T> value,
                int rlSrc)

Description copied from class: Array

Set range to given arrays value with an offset into other array

Overrides:

set in class Array<T>

Parameters:

rl - row lower

ru - row upper (inclusive)

value - value array to take values from

rlSrc - the offset into the value array to take values from

setNz

public void setNz(int rl,
                  int ru,
                  Array<T> value)

Description copied from class: Array

Set non default values in the range from the value array given

Specified by:

setNz in class Array<T>

Parameters:

rl - row start

ru - row upper inclusive

value - value array of same type

setFromOtherTypeNz

public void setFromOtherTypeNz(int rl,
                               int ru,
                               Array<?> value)

Description copied from class: Array

Set non default values in the range from the value array given

Specified by:

setFromOtherTypeNz in class Array<T>

Parameters:

rl - row start

ru - row end inclusive

value - value array of different type

append

public void append(String value)

Description copied from class: Array

Append a string value to the current Array, this should in general be avoided, and appending larger blocks at a time should be preferred.

Specified by:

append in class Array<T>

Parameters:

value - The value to append

append

public void append(T value)

Description copied from class: Array

Append a value of the same type of the Array. This should in general be avoided, and appending larger blocks at a time should be preferred.

Specified by:

append in class Array<T>

Parameters:

value - The value to append

append

public Array<T> append(Array<T> other)

Description copied from class: Array

Append other array, if the other array is fitting in current allocated size use that allocated size, otherwise allocate new array to combine the other with this. This method should use the set range function, and should be preferred over the append single values.

Specified by:

append in class Array<T>

Parameters:

other - The other array of same type to append to this.

Returns:

The combined arrays.

slice

public Array<T> slice(int rl,
                      int ru)

Description copied from class: Array

Slice out the sub range and return new array with the specified type. If the conversion fails fallback to normal slice.

Specified by:

slice in class Array<T>

Parameters:

rl - row start

ru - row end (not included)

Returns:

A new array of sub range.

reset

public void reset(int size)

Description copied from class: Array

Reset the Array and set to a different size. This method is used to reuse an already allocated Array, without extra allocation. It should only be done in cases where the Array is no longer in use in any FrameBlocks.

Specified by:

reset in class Array<T>

Parameters:

size - The size to reallocate into.

getAsByteArray

public byte[] getAsByteArray()

Description copied from class: Array

Return the current allocated Array as a byte[], this is used to serialize the allocated Arrays out to the PythonAPI.

Specified by:

getAsByteArray in class Array<T>

Returns:

The array as bytes

getValueType

public Types.ValueType getValueType()

Description copied from class: Array

Get the current value type of this array.

Specified by:

getValueType in class Array<T>

Returns:

The current value type.

analyzeValueType

public Pair<Types.ValueType,Boolean> analyzeValueType(int maxCells)

Description copied from class: Array

Analyze the column to figure out if the value type can be refined to a better type. The return is in two parts, first the type it can be, second if it contains nulls.

Specified by:

analyzeValueType in class Array<T>

Parameters:

maxCells - maximum number of cells to analyze

Returns:

A better or equivalent value type to represent the column, including null information.

getFrameArrayType

public ArrayFactory.FrameArrayType getFrameArrayType()

Description copied from class: Array

Get the internal FrameArrayType, to specify the encoding of the Types, note there are more Frame Array Types than there is ValueTypes.

Specified by:

getFrameArrayType in class Array<T>

Returns:

The FrameArrayType

getExactSerializedSize

public long getExactSerializedSize()

Description copied from class: Array

Get the exact serialized size on disk of this array.

Specified by:

getExactSerializedSize in class Array<T>

Returns:

The exact size on disk

changeTypeWithNulls

public Array<?> changeTypeWithNulls(Types.ValueType t)

Overrides:

changeTypeWithNulls in class Array<T>

fill

public void fill(String val)

Description copied from class: Array

fill the entire array with specific value.

Specified by:

fill in class Array<T>

Parameters:

val - the value to fill with.

fill

public void fill(T val)

Description copied from class: Array

fill the entire array with specific value.

Specified by:

fill in class Array<T>

Parameters:

val - the value to fill with.

isShallowSerialize

public boolean isShallowSerialize()

Description copied from class: Array

analyze if this array can be shallow serialized. to allow caching without modification.

Specified by:

isShallowSerialize in class Array<T>

Returns:

boolean saying true if shallow serialization is available

isEmpty

public boolean isEmpty()

Description copied from class: Array

Get if this array is empty, aka filled with empty values.

Specified by:

isEmpty in class Array<T>

Returns:

boolean saying true if empty

select

public Array<T> select(int[] indices)

Description copied from class: Array

Slice out the specified indices and return the sub array.

Specified by:

select in class Array<T>

Parameters:

indices - The indices to slice out

Returns:

the sliced out indices in an array format

select

public Array<T> select(boolean[] select,
                       int nTrue)

Description copied from class: Array

Slice out the true indices in the select input and return the sub array.

Specified by:

select in class Array<T>

Parameters:

select - a boolean vector specifying what to select

nTrue - number of true values inside select

Returns:

the sliced out indices in an array format

isNotEmpty

public boolean isNotEmpty(int i)

Specified by:

isNotEmpty in class Array<T>

clone

public Array<T> clone()

Description copied from class: Array

Overwrite of the java internal clone function for arrays, return a clone of underlying data that is mutable, (not immutable data.) Immutable data is dependent on the individual allocated arrays

Specified by:

clone in class Array<T>

Returns:

A clone

hashDouble

public double hashDouble(int idx)

Description copied from class: Array

Hash the given index of the array. It is allowed to return NaN on null elements.

Specified by:

hashDouble in class Array<T>

Parameters:

idx - The index to hash

Returns:

The hash value of that index.

equals

public boolean equals(Array<T> other)

Description copied from class: Array

Equals operation on arrays.

Specified by:

equals in class Array<T>

Parameters:

other - The other array to compare to.

Returns:

True if the arrays are equivalent.

getInMemorySize

public long getInMemorySize()

Description copied from class: Array

Get in memory size, not counting reference to this object.

Overrides:

getInMemorySize in class Array<T>

Returns:

the size in memory of this object.

containsNull

public boolean containsNull()

Description copied from class: Array

analyze if the array contains null values.

Overrides:

containsNull in class Array<T>

Returns:

If the array contains null.

possiblyContainsNaN

public boolean possiblyContainsNaN()

Specified by:

possiblyContainsNaN in class Array<T>

toString

public String toString()

Overrides:

toString in class Array<T>