Builtin Functions Reference

Table of Contents

• Introduction
• Built-In Construction Functions
  • tensor-Function
• DML-Bodied Built-In functions
  • confusionMatrix-Function
  • correctTypos-Function
  • cspline-Function
  • csplineCG-Function
  • csplineDS-Function
  • cvlm-Function
  • DBSCAN-Function
  • decisionTree-Function
  • discoverFD-Function
  • dist-Function
  • dmv-Function
  • ema-Function
  • ffPredict-Function
  • ffTrain-Function
  • gaussianClassifier-Function
  • glm-Function
  • gmm-Function
  • gnmf-Function
  • gridSearch-Function
  • hyperband-Function
  • img_brightness-Function
  • img_crop-Function
  • img_mirror-Function
  • impurityMeasures-Function
  • imputeByFD-Function
  • intersect-Function
  • KMeans-Function
  • KNN-function
  • lenetTrain-Function
  • lenetPredict-Function
  • lm-Function
  • lmCG-Function
  • lmDS-Function
  • lmPredict-Function
  • matrixProfile-Function
  • mdedup-Function
  • mice-Function
  • msvm-Function
  • multiLogReg-Function
  • naiveBayes-Function
  • naiveBayesPredict-Function
  • normalize-Function
  • outlier-Function
  • outlierByDB-Function
  • pnmf-Function
  • scale-Function
  • setdiff-Function
  • sherlock-Function
  • sherlockPredict-Function
  • sigmoid-Function
  • slicefinder-Function
  • smote-Function
  • steplm-Function
  • symmetricDifference-Function
  • tomekLink-Function
  • toOneHot-Function
  • tSNE-Function
  • union-Function
  • unique-Function
  • winsorize-Function
  • xgboost-Function

Introduction

The DML (Declarative Machine Learning) language has built-in functions which enable access to both low- and high-level functions to support all kinds of use cases.

A builtin is either implemented on a compiler level or as DML scripts that are loaded at compile time.

Built-In Construction Functions

There are some functions which generate an object for us. They create matrices, tensors, lists and other non-primitive objects.

tensor-Function

The tensor-function creates a tensor for us.

tensor(data, dims, byRow = TRUE)

Arguments

Note that this function is highly unstable and will be overworked and might change signature and functionality.

Returns

data-Argument

The data-argument can be a Matrix of any datatype from which the elements will be taken and placed in the tensor until filled. If given as a Tensor the same procedure takes place. We iterate through Matrix and Tensor by starting with each dimension index at 0 and then incrementing the lowest one, until we made a complete pass over the dimension, and then increasing the dimension index above. This will be done until the Tensor is completely filled.

If data is a Scalar, we fill the whole tensor with the value.

dims-Argument

The dimension of the tensor can either be given by a vector represented by either by a Matrix, Tensor, String or List. Dimensions given by a String will be expected to be concatenated by spaces.

Example

print("Dimension matrix:");
d = matrix("2 3 4", 1, 3);
print(toString(d, decimal=1))

print("Tensor A: Fillvalue=3, dims=2 3 4");
A = tensor(3, d); # fill with value, dimensions given by matrix
print(toString(A))

print("Tensor B: Reshape A, dims=4 2 3");
B = tensor(A, "4 2 3"); # reshape tensor, dimensions given by string
print(toString(B))

print("Tensor C: Reshape dimension matrix, dims=1 3");
C = tensor(d, list(1, 3)); # values given by matrix, dimensions given by list
print(toString(C, decimal=1))

print("Tensor D: Values=tst, dims=Tensor C");
D = tensor("tst", C); # fill with string, dimensions given by tensor
print(toString(D))

Note that reshape construction is not yet supported for SPARK execution.

DML-Bodied Built-In Functions

DML-bodied built-in functions are written as DML-Scripts and executed as such when called.

confusionMatrix-Function

A confusionMatrix-accepts a vector for prediction and a one-hot-encoded matrix, then it computes the max value of each vector and compare them, after which it calculates and returns the sum of classifications and the average of each true class.

Usage

confusionMatrix(P, Y)

Arguments

Returns

Example

numClasses = 1
z = rand(rows = 5, cols = 1, min = 1, max = 9)
X = round(rand(rows = 5, cols = 1, min = 1, max = numClasses))
y = toOneHot(X, numClasses)
[ConfusionSum, ConfusionAvg] = confusionMatrix(P=z, Y=y)

correctTypos-Function

The correctTypos - function tries to correct typos in a given frame. This algorithm operates on the assumption that most strings are correct and simply swaps strings that do not occur often with similar strings that occur more often. If correct is set to FALSE only prints suggested corrections without affecting the frame.

Usage

correctTypos(strings, frequency_threshold, distance_threshold, decapitalize, correct, is_verbose)

Arguments

Returns

Example

A = read(“file1”, data_type=”frame”, rows=2000, cols=1, format=”binary”)
A_corrected = correctTypos(A, 0.02, 3, FALSE, TRUE)

cspline-Function

This cspline-function solves Cubic spline interpolation. The function usages natural spline with \(q_1''(x_0) == q_n''(x_n) == 0.0\). By default, it calculates via csplineDS-function.

Algorithm reference: https://en.wikipedia.org/wiki/Spline_interpolation#Algorithm_to_find_the_interpolating_cubic_spline

Usage

[result, K] = cspline(X, Y, inp_x, tol, maxi)

Arguments

Returns

Example

num_rec = 100 # Num of records
X = matrix(seq(1,num_rec), num_rec, 1)
Y = round(rand(rows = 100, cols = 1, min = 1, max = 5))
inp_x = 4.5
tolerance = 0.000001
max_iter = num_rec
[result, K] = cspline(X=X, Y=Y, inp_x=inp_x, tol=tolerance, maxi=max_iter)

csplineCG-Function

This csplineCG-function solves Cubic spline interpolation with conjugate gradient method. Usage will be same as cspline-function.

Usage

[result, K] = csplineCG(X, Y, inp_x, tol, maxi)

Arguments

Returns

Example

num_rec = 100 # Num of records
X = matrix(seq(1,num_rec), num_rec, 1)
Y = round(rand(rows = 100, cols = 1, min = 1, max = 5))
inp_x = 4.5
tolerance = 0.000001
max_iter = num_rec
[result, K] = csplineCG(X=X, Y=Y, inp_x=inp_x, tol=tolerance, maxi=max_iter)

csplineDS-Function

This csplineDS-function solves Cubic spline interpolation with direct solver method.

Usage

[result, K] = csplineDS(X, Y, inp_x)

Arguments

Returns

Example

num_rec = 100 # Num of records
X = matrix(seq(1,num_rec), num_rec, 1)
Y = round(rand(rows = 100, cols = 1, min = 1, max = 5))
inp_x = 4.5

[result, K] = csplineDS(X=X, Y=Y, inp_x=inp_x)

cvlm-Function

The cvlm-function is used for cross-validation of the provided data model. This function follows a non-exhaustive cross validation method. It uses lm and lmPredict functions to solve the linear regression and to predict the class of a feature vector with no intercept, shifting, and rescaling.

Usage

cvlm(X, y, k)

Arguments

Returns

Example

X = rand (rows = 5, cols = 5)
y = X %*% rand(rows = ncol(X), cols = 1)
[predict, beta] = cvlm(X = X, y = y, k = 4)

decisionTree-Function

The decisionTree() implements the classification tree with both scale and categorical features.

Usage

M = decisionTree(X, Y, R);

Arguments

Returns

Example

X = matrix("4.5 4.0 3.0 2.8 3.5
            1.9 2.4 1.0 3.4 2.9
            2.0 1.1 1.0 4.9 3.4
            2.3 5.0 2.0 1.4 1.8
            2.1 1.1 3.0 1.0 1.9", rows=5, cols=5)
Y = matrix("1.0
            0.0
            0.0
            1.0
            0.0", rows=5, cols=1)
R = matrix("1.0 1.0 3.0 1.0 1.0", rows=1, cols=5)
M = decisionTree(X = X, Y = Y, R = R)

discoverFD-Function

The discoverFD-function finds the functional dependencies.

Usage

discoverFD(X, Mask, threshold)

Arguments

Returns

dist-Function

The dist-function is used to compute Euclidian distances between N d-dimensional points.

Usage

dist(X)

Arguments

Returns

Example

X = rand (rows = 5, cols = 5)
Y = dist(X)

dmv-Function

The dmv-function is used to find disguised missing values utilising syntactical pattern recognition.

Usage

dmv(X, threshold, replace)

Arguments

Returns

Example

A = read("fileA", data_type="frame", rows=10, cols=8);
Z = dmv(X=A)
Z = dmv(X=A, threshold=0.9)
Z = dmv(X=A, threshold=0.9, replace="NaN")

ffPredict-Function

The ffPredict-function computes prediction of the given thata using trained model.

It takes the list of layers returned by the ffTrain-function.

Usage

prediction = ffPredict(model, x_test, 128)

Arguments

Returns

Example

model = ffTrain(x_train, y_train, 128, 10, 0.001, ...)

prediction = ffPredict(model=model, X=x_test)

ffTrain-Function

The ffTrain-function trains simple feed-forward neural network.

Neural network trained has the following architecture:

affine1 -> relu -> dropout -> affine2 -> configurable output layer activation

Hidden layer has 128 neurons. Dropout rate is 0.35. Input and ouptut sizes are inferred from X and Y.

Usage

model = ffTrain(X=x_train, Y=y_train, out_activation="sigmoid", loss_fcn="cel")

Arguments

User can specify following output layer activations: sigmoid, relu, lrelu, tanh, softmax, logits (no activation).

User can specify following loss functions: l1, l2, log_loss, logcosh_loss, cel (cross-entropy loss).

When validation set is used function outputs validation loss to the stdout after each epoch.

Returns

Example

model = ffTrain(X=x_train, Y=y_train, batch_size=128, epochs=10, learning_rate=0.001, out_activation="sigmoid", loss_fcn="cel", shuffle=TRUE, validation_split=0.2, verbose=TRUE)

prediction = ffPredict(model=model, X=x_train)

gaussianClassifier-Function

The gaussianClassifier-function computes prior probabilities, means, determinants, and inverse covariance matrix per class.

Classification is as per \(p(C=c | x) = p(x | c) * p(c)\) Where \(p(x | c)\) is the (multivariate) Gaussian P.D.F. for class \(c\), and \(p(c)\) is the prior probability for class \(c\).

Usage

[prior, means, covs, det] = gaussianClassifier(D, C, varSmoothing)

Arguments

Returns

Example

X = rand (rows = 200, cols = 50 )
y = X %*% rand(rows = ncol(X), cols = 1)

[prior, means, covs, det] = gaussianClassifier(D=X, C=y, varSmoothing=1e-9)

glm-Function

The glm-function is a flexible generalization of ordinary linear regression that allows for response variables that have error distribution models.

Usage

glm(X,Y)

Arguments

Returns

Example

X = rand (rows = 5, cols = 5 )
y = X %*% rand(rows = ncol(X), cols = 1)
beta = glm(X=X,Y=y)

gmm-Function

The gmm-function implements builtin Gaussian Mixture Model with four different types of covariance matrices i.e., VVV, EEE, VVI, VII and two initialization methods namely “kmeans” and “random”.

Usage

gmm(X=X, n_components = 3,  model = "VVV",  init_params = "random", iter = 100, reg_covar = 0.000001, tol = 0.0001, verbose=TRUE)

Arguments

Returns

Example

X = read($1)
[labels, df, bic] = gmm(X=X, n_components = 3,  model = "VVV",  init_params = "random", iter = 100, reg_covar = 0.000001, tol = 0.0001, verbose=TRUE)

gnmf-Function

The gnmf-function does Gaussian Non-Negative Matrix Factorization. In this, a matrix X is factorized into two matrices W and H, such that all three matrices have no negative elements. This non-negativity makes the resulting matrices easier to inspect.

Usage

gnmf(X, rnk, eps = 10^-8, maxi = 10)

Arguments

Returns

Example

X = rand(rows = 50, cols = 10)
W = rand(rows = nrow(X), cols = 2, min = -0.05, max = 0.05);
H = rand(rows = 2, cols = ncol(X), min = -0.05, max = 0.05);
gnmf(X = X, rnk = 2, eps = 10^-8, maxi = 10)

gridSearch-Function

The gridSearch-function is used to find the optimal hyper-parameters of a model which results in the most accurate predictions. This function takes train and eval functions by name.

Usage

gridSearch(X, y, train, predict, params, paramValues, verbose)

Arguments

Returns

Example

X = rand (rows = 50, cols = 10)
y = X %*% rand(rows = ncol(X), cols = 1)
params = list("reg", "tol", "maxi")
paramRanges = list(10^seq(0,-4), 10^seq(-5,-9), 10^seq(1,3))
[B, opt]= gridSearch(X=X, y=y, train="lm", predict="lmPredict", params=params, paramValues=paramRanges, verbose = TRUE)

hyperband-Function

The hyperband-function is used for hyper parameter optimization and is based on multi-armed bandits and early elimination. Through multiple parallel brackets and consecutive trials it will return the hyper parameter combination which performed best on a validation dataset. A set of hyper parameter combinations is drawn from uniform distributions with given ranges; Those make up the candidates for hyperband. Notes:

• hyperband is hard-coded for lmCG, and uses lmPredict for validation
• hyperband is hard-coded to use the number of iterations as a resource
• hyperband can only optimize continuous hyperparameters

Usage

hyperband(X_train, y_train, X_val, y_val, params, paramRanges, R, eta, verbose)

Arguments

Returns

Example

X_train = rand(rows=50, cols=10);
y_train = rowSums(X_train) + rand(rows=50, cols=1);
X_val = rand(rows=50, cols=10);
y_val = rowSums(X_val) + rand(rows=50, cols=1);

params = list("reg");
paramRanges = matrix("0 20", rows=1, cols=2);

[bestWeights, optHyperParams] = hyperband(X_train=X_train, y_train=y_train, 
    X_val=X_val, y_val=y_val, params=params, paramRanges=paramRanges);

img_brightness-Function

The img_brightness-function is an image data augumentation function. It changes the brightness of the image.

Usage

img_brightness(img_in, value, channel_max)

Arguments

Returns

Example

A = rand(rows = 3, cols = 3, min = 0, max = 255)
B = img_brightness(img_in = A, value = 128, channel_max = 255)

img_crop-Function

The img_crop-function is an image data augumentation function. It cuts out a subregion of an image.

Usage

img_crop(img_in, w, h, x_offset, y_offset)

Arguments

Returns

Example

A = rand(rows = 3, cols = 3, min = 0, max = 255) 
B = img_crop(img_in = A, w = 20, h = 10, x_offset = 0, y_offset = 0)

img_mirror-Function

The img_mirror-function is an image data augumentation function. It flips an image on the X (horizontal) or Y (vertical) axis.

Usage

img_mirror(img_in, horizontal_axis)

Arguments

Returns

Example

A = rand(rows = 3, cols = 3, min = 0, max = 255)
B = img_mirror(img_in = A, horizontal_axis = TRUE)

impurityMeasures-Function

impurityMeasures() computes the measure of impurity for each feature of the given dataset based on the passed method (gini or entropy).

Usage

IM = impurityMeasures(X = X, Y = Y, R = R, n_bins = 20, method = "gini");

Arguments

Returns

Example

X = matrix("4.0 3.0 2.8 3.5
            2.4 1.0 3.4 2.9
            1.1 1.0 4.9 3.4
            5.0 2.0 1.4 1.8
            1.1 3.0 1.0 1.9", rows=5, cols=4)
Y = matrix("1.0
            0.0
            0.0
            1.0
            0.0", rows=5, cols=1)
R = matrix("1.0 2.0 1.0 1.0", rows=1, cols=4)
IM = impurityMeasures(X = X, Y = Y, R = R, method = "entropy")

imputeByFD-Function

The imputeByFD-function imputes missing values from observed values (if exist) using robust functional dependencies.

Usage

imputeByFD(X, sourceAttribute, targetAttribute, threshold)

Arguments

Returns

Example

X = matrix("1 1 1 2 4 5 5 3 3 NaN 4 5 4 1", rows=7, cols=2)
imputeByFD(X = X, source = 1, target = 2, threshold = 0.6, verbose = FALSE)

imputeEMA-Function

The imputeEMA-function imputes values with exponential moving average (single, double or triple).

Usage

ema(X, search_iterations, mode, freq, alpha, beta, gamma)

Arguments

Returns

Example

X = read("fileA", data_type="frame")
ema(X = X, search_iterations = 1, mode = "triple", freq = 4, alpha = 0.1, beta = 0.1, gamma = 0.1,)

KMeans-Function

The kmeans() implements the KMeans Clustering algorithm.

Usage

kmeans(X = X, k = 20, runs = 10, max_iter = 5000, eps = 0.000001, is_verbose = FALSE, avg_sample_size_per_centroid = 50, seed = -1)

Arguments

Returns

Example

X = rand (rows = 3972, cols = 972)
kmeans(X = X, k = 20, runs = 10, max_iter = 5000, eps = 0.000001, is_verbose = FALSE, avg_sample_size_per_centroid = 50, seed = -1)

KNN-Function

The knn() implements the KNN (K Nearest Neighbor) algorithm.

Usage

[NNR, PR, FI] = knn(Train, Test, CL, k_value)

Arguments

Returns

Example

X = rand(rows = 100, cols = 20)
T = rand(rows= 3, cols = 20) # query rows, and columns
CL = matrix(seq(1,100), 100, 1)
k = 3
[NNR, PR, FI] = knn(Train=X, Test=T, CL=CL, k_value=k, predict_con_tg=1)

lenetTrain-Function

The lenetTrain-function trains LeNet CNN. The architecture of the networks is: conv1 -> relu1 -> pool1 -> conv2 -> relu2 -> pool2 -> affine3 -> relu3 -> affine4 -> softmax

Usage

model = lenetTrain(images, labels, images_val, labels_val, C, Hin, Win, 128, 3, 0.007, 0.9, 0.95, 5e-04, TRUE, -1)

Arguments

Returns

Example

data = read(path, format="csv")
images = images = train_data[,2:ncol(data)]
labels_int = data[,1]
C = 1
Hin = 28
Win = 28

# Scale images to [-1,1], and one-hot encode the labels
n = nrow(train_data)
images = (images / 255.0) * 2 - 1
labels = table(seq(1, n), labels_int+1, n, 10)

model = lenetTrain(images, labels, images_val, labels_val, C, Hin, Win, 128, 3, 0.007, 0.9, 0.95, 5e-04, TRUE, -1)

lenetPredict-Function

The lenetPredict-function makes prediction given the data and trained LeNet model.

Usage

probs = lenetPredict(model=model, X=images, C=C, Hin=Hin, Win=Win)

Arguments

Returns

Example

model = lenetTrain(images, labels, images_val, labels_val, C, Hin, Win, 128, 3, 0.007, 0.9, 0.95, 5e-04, TRUE, -1)
probs = lenetPredict(model=model, X=images_test, C=C, Hin=Hin, Win=Win)

lm-Function

The lm-function solves linear regression using either the direct solve method or the conjugate gradient algorithm depending on the input size of the matrices (See lmDS-function and lmCG-function respectively).

Usage

lm(X, y, icpt = 0, reg = 1e-7, tol = 1e-7, maxi = 0, verbose = TRUE)

Arguments

Note that if number of features is small enough (rows of X/y < 2000), the lmDS-Function’ is called internally and parameters tol and maxi are ignored.

Returns

icpt-Argument

The icpt-argument can be set to 3 modes:

• 0 = no intercept, no shifting, no rescaling
• 1 = add intercept, but neither shift nor rescale X
• 2 = add intercept, shift & rescale X columns to mean = 0, variance = 1

Example

X = rand (rows = 50, cols = 10)
y = X %*% rand(rows = ncol(X), cols = 1)
lm(X = X, y = y)

intersect-Function

The intersect-function implements set intersection for numeric data.

Usage

intersect(X, Y)

Arguments

Returns

lmCG-Function

The lmCG-function solves linear regression using the conjugate gradient algorithm.

Usage

lmCG(X, y, icpt = 0, reg = 1e-7, tol = 1e-7, maxi = 0, verbose = TRUE)

Arguments

Returns

Example

X = rand (rows = 50, cols = 10)
y = X %*% rand(rows = ncol(X), cols = 1)
lmCG(X = X, y = y, maxi = 10)

lmDS-Function

The lmDS-function solves linear regression by directly solving the linear system.

Usage

lmDS(X, y, icpt = 0, reg = 1e-7, verbose = TRUE)

Arguments

Returns

Example

X = rand (rows = 50, cols = 10)
y = X %*% rand(rows = ncol(X), cols = 1)
lmDS(X = X, y = y)

lmPredict-Function

The lmPredict-function predicts the class of a feature vector.

Usage

lmPredict(X=X, B=w, ytest= Y)

Arguments

Returns

Example

X = rand (rows = 50, cols = 10)
y = X %*% rand(rows = ncol(X), cols = 1)
w = lm(X = X, y = y)
yp = lmPredict(X = X, B = w, ytest=matrix(0,1,1))

matrixProfile-Function

The matrixProfile-function implements the SCRIMP algorithm for efficient time-series analysis.

Usage

matrixProfile(ts, window_size, sample_percent, is_verbose)

Arguments

| Name | Type | Default | Description | | :—— | :————- | ——– | :———- | | ts | Matrix | — | Input Frame X | | window_size | Integer | 4 | Sliding window size | | sample_percent| Double | 1.0 | Degree of approximation between zero and one (1 computes the exact solution) | | verbose | Boolean | False | Print debug information |

Returns

mdedup-Function

The mdedup-function implements builtin for deduplication using matching dependencies (e.g. Street 0.95, City 0.90 -> ZIP 1.0) by Jaccard distance.

Usage

mdedup(X, Y, intercept, epsilon, lamda, maxIterations, verbose)

Arguments

Returns

Example

X = as.frame(rand(rows = 50, cols = 10))
LHSfeatures = matrix("1 3 19", 1, 2)
LHSthreshold = matrix("0.85 0.85", 1, 2)
RHSfeatures = matrix("30", 1, 1)
RHSthreshold = matrix("1.0", 1, 1)
duplicates = mdedup(X, LHSfeatures, LHSthreshold, RHSfeatures, RHSthreshold, verbose = FALSE)

mice-Function

The mice-function implements Multiple Imputation using Chained Equations (MICE) for nominal data.

Usage

mice(F, cMask, iter, complete, verbose)

Arguments

Returns

Example

F = matrix("4 3 NaN 8 7 8 5 NaN 6", rows=3, cols=3)
cMask = round(rand(rows=1,cols=ncol(F),min=0,max=1))
dataset = mice(F, cMask, iter = 3, verbose = FALSE)

msvm-Function

The msvm-function implements builtin multiclass SVM with squared slack variables It learns one-against-the-rest binary-class classifiers by making a function call to l2SVM

Usage

msvm(X, Y, intercept, epsilon, lamda, maxIterations, verbose)

Arguments

Returns

Example

X = rand(rows = 50, cols = 10)
y = round(X %*% rand(rows=ncol(X), cols=1))
model = msvm(X = X, Y = y, intercept = FALSE, epsilon = 0.005, lambda = 1.0, maxIterations = 100, verbose = FALSE)

multiLogReg-Function

The multiLogReg-function solves Multinomial Logistic Regression using Trust Region method. (See: Trust Region Newton Method for Logistic Regression, Lin, Weng and Keerthi, JMLR 9 (2008) 627-650)

Usage

multiLogReg(X, Y, icpt, reg, tol, maxi, maxii, verbose)

Arguments

Returns

Example

X = rand(rows = 50, cols = 30)
Y = X %*% rand(rows = ncol(X), cols = 1)
betas = multiLogReg(X = X, Y = Y, icpt = 2,  tol = 0.000001, reg = 1.0, maxi = 100, maxii = 20, verbose = TRUE)

naiveBayes-Function

The naiveBayes-function computes the class conditional probabilities and class priors.

Usage

naiveBayes(D, C, laplace, verbose)

Arguments

Returns

Example

D=rand(rows=10,cols=1,min=10)
C=rand(rows=10,cols=1,min=10)
[prior, classConditionals] = naiveBayes(D, C, laplace = 1, verbose = TRUE)

naiveBaysePredict-Function

The naiveBaysePredict-function predicts the scoring with a naive Bayes model.

Usage

naiveBaysePredict(X=X, P=P, C=C)

Arguments

Returns

Example

[YRaw, Y] = naiveBaysePredict(X=data, P=model_prior, C=model_conditionals)

normalize-Function

The normalize-function normalises the values of a matrix by changing the dataset to use a common scale. This is done while preserving differences in the ranges of values. The output is a matrix of values in range [0,1].

Usage

normalize(X); 

Arguments

Returns

Example

X = rand(rows = 50, cols = 10)
y = X %*% rand(rows = ncol(X), cols = 1)
y = normalize(X = X)

outlier-Function

This outlier-function takes a matrix data set as input from where it determines which point(s) have the largest difference from mean.

Usage

outlier(X, opposite)

Arguments

Returns

Example

X = rand (rows = 50, cols = 10)
outlier(X=X, opposite=1)

outlierByDB-Function

The outlierByDB-function implements an outlier prediction for a trained dbscan model. The points in the Xtest matrix are checked against the model and are considered part of the cluster if at least one member is within eps distance.

Usage

outlierByDB(X, model, eps)

Arguments

Returns

Example

eps = 1
minPts = 5
X = rand(rows=1780, cols=180, min=1, max=20)
[indices, model] = dbscan(X=X, eps=eps, minPts=minPts)
Y = rand(rows=500, cols=180, min=1, max=20)
Z = outlierByDB(Xtest=Y, clusterModel = model, eps = eps)

pnmf-Function

The pnmf-function implements Poisson Non-negative Matrix Factorization (PNMF). Matrix X is factorized into two non-negative matrices, W and H based on Poisson probabilistic assumption. This non-negativity makes the resulting matrices easier to inspect.

Usage

pnmf(X, rnk, eps = 10^-8, maxi = 10, verbose = TRUE)

Arguments

Returns

Example

X = rand(rows = 50, cols = 10)
[W, H] = pnmf(X = X, rnk = 2, eps = 10^-8, maxi = 10, verbose = TRUE)

scale-Function

The scale function is a generic function whose default method centers or scales the column of a numeric matrix.

Usage

scale(X, center=TRUE, scale=TRUE)

Arguments

Returns

Example

X = rand(rows = 20, cols = 10)
center=TRUE;
scale=TRUE;
Y= scale(X,center,scale)

setdiff-Function

The setdiff-function returns the values of X that are not in Y.

Usage

setdiff(X, Y)

Arguments

Returns

Example

X = matrix("1 2 3 4", rows = 4, cols = 1)
Y = matrix("2 3", rows = 2, cols = 1)
R = setdiff(X = X, Y = Y)

sherlock-Function

Implements training phase of Sherlock: A Deep Learning Approach to Semantic Data Type Detection

[Hulsebos, Madelon, et al. “Sherlock: A deep learning approach to semantic data type detection.” Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining., 2019]

Usage

sherlock(X_train, y_train)

Arguments

Returns