Package org.apache.datasketches.quantilescommon

Interface QuantilesAPI

  • All Known Subinterfaces:
    QuantilesDoublesAPI, QuantilesFloatsAPI, QuantilesGenericAPI<T>
    All Known Implementing Classes:
    CompactDoublesSketch, DoublesSketch, ItemsSketch, KllDoublesSketch, KllFloatsSketch, KllItemsSketch, KllSketch, ReqSketch, UpdateDoublesSketch
    public interface QuantilesAPI

    This is a stochastic streaming sketch that enables near-real time analysis of the approximate distribution of items from a very large stream in a single pass, requiring only that the items are comparable. The analysis is obtained using the getQuantile() function or the inverse functions getRank(), getPMF() (the Probability Mass Function), and getCDF() (the Cumulative Distribution Function).

    Given an input stream of N items, the natural rank of any specific item is defined as its index (1 to N) in the hypothetical sorted stream of all N input items.

    The normalized rank (rank) of any specific item is defined as its natural rank divided by N, which is a number in the interval [0.0, 1.0]. In the Javadocs for all the quantile sketches natural rank is seldom used so any reference to just rank should be interpreted as normalized rank.

    Inputs into a quantile sketch are called "items" that can be either generic or specific primitives, like float or double depending on the sketch implementation. In order to keep its size small, sketches don't retain all the items offered and retain only a small fraction of all the items, thus purging most of the items. The items retained are then sorted and associated with a rank. At this point we call the retained items quantiles. Thus, all quantiles are items, but only a few items become quantiles. Depending on the context the two terms can be interchangeable.

    All quantile sketches are configured with a parameter k, which affects the size of the sketch and its estimation error.

    In the research literature, the estimation error is commonly called epsilon (or eps) and is a fraction between zero and one. Larger sizes of k result in a smaller epsilon, but also a larger sketch. The epsilon error is always with respect to the rank domain. Estimating the confidence interval in the quantile domain can be done by first computing the error in the rank domain and then translating that to the quantile domain. The sketch provides methods to assist with that.

    The relationship between the normalized rank and the corresponding quantiles can be viewed as a two dimensional monotonic plot with the normalized rank on one axis and the corresponding quantiles on the other axis. Let q := quantile and r := rank then both q = getQuantile(r) and r = getRank(q) are monotonically increasing functions. If the y-axis is used for the rank domain and the x-axis for the quantile domain, then y = getRank(x) is also the single point Cumulative Distribution Function (CDF).

    The functions getQuantile() translate ranks into corresponding quantiles. The functions getRank(), getCDF(), and getPMF() (Probability Mass Function) perform the opposite operation and translate quantiles into ranks (or cumulative probabilities, or probability masses, depending on the context).

    As an example, consider a large stream of one million items such as packet sizes coming into a network node. The absolute rank of any specific item size is simply its index in the hypothetical sorted array of such items. The normalized rank is the natural rank divided by the stream size, or N, in this case one million. The quantile corresponding to the normalized rank of 0.5 represents the 50th percentile or median of the distribution, obtained from getQuantile(0.5). Similarly, the 95th percentile is obtained from getQuantile(0.95).

    From the min and max quantiles, for example, say 1 and 1000 bytes, you can obtain the PMF from getPMF(100, 500, 900) that will result in an array of 4 probability masses such as {.4, .3, .2, .1}, which means that

    • 40% of the mass was < 100,
    • 30% of the mass was ≥ 100 and < 500,
    • 20% of the mass was ≥ 500 and < 900, and
    • 10% of the mass was ≥ 900.
    A frequency histogram can be obtained by simply multiplying these probability masses by getN(), which is the total count of items received. The getCDF() works similarly, but produces the cumulative distribution instead.

    The accuracy of this sketch is a function of the configured k, which also affects the overall size of the sketch. Accuracy of this quantile sketch is always with respect to the normalized rank.

    The getPMF() function has about 13 to 47% worse rank error (depending on k) than the other queries because the mass of each "bin" of the PMF has "double-sided" error from the upper and lower edges of the bin as a result of a subtraction of random variables where the errors from the two edges can sometimes add.

    A getQuantile(rank) query has the following probabilistic guarantees:

    • Let q = getQuantile(r) where r is the rank between zero and one.
    • The quantile q will be a quantile from the input stream.
    • Let trueRank be the true rank of q derived from the hypothetical sorted stream of all N quantiles.
    • Let eps = getNormalizedRankError(false)[*].
    • Then r - eps ≤ trueRank ≤ r + eps. Note that the error is on the rank, not the quantile.

    A getRank(quantile) query has the following probabilistic guarantees:

    • Let r = getRank(q) where q is a quantile between the min and max quantiles of the input stream.
    • Let trueRank be the true rank of q derived from the hypothetical sorted stream of all N quantiles.
    • Let eps = getNormalizedRankError(false)[*].
    • Then r - eps ≤ trueRank ≤ r + eps.

    A getPMF() query has the following probabilistic guarantees:

    • Let {r1, r2, ..., rm+1} = getPMF(v1, v2, ..., vm) where q1, q2, ..., qm are monotonically increasing quantiles supplied by the user that are part of the monotonic sequence q0 = min, q1, q2, ..., qm, qm+1 = max, and where min and max are the actual minimum and maximum quantiles of the input stream automatically included in the sequence by the getPMF(...) function.
    • Let ri = massi = estimated mass between vi-1 and qi where q0 = min and qm+1 = max.
    • Let trueMass be the true mass between the quantiles of qi, qi+1 derived from the hypothetical sorted stream of all N quantiles.
    • Let eps = getNormalizedRankError(true)[*].
    • Then mass - eps ≤ trueMass ≤ mass + eps.
    • r1 includes the mass of all points between min = q0 and q1.
    • rm+1 includes the mass of all points between qm and max = qm+1.

    A getCDF(...) query has the following probabilistic guarantees:

    • Let {r1, r2, ..., rm+1} = getCDF(q1, q2, ..., qm) where q1, q2, ..., qm) are monotonically increasing quantiles supplied by the user that are part of the monotonic sequence {q0 = min, q1, q2, ..., qm, qm+1 = max}, and where min and max are the actual minimum and maximum quantiles of the input stream automatically included in the sequence by the getCDF(...) function.
    • Let ri = massi = estimated mass between q0 = min and qi.
    • Let trueMass be the true mass between the true ranks of qi, qi+1 derived from the hypothetical sorted stream of all N quantiles.
    • Let eps = getNormalizedRankError(true)[*].
    • then mass - eps ≤ trueMass ≤ mass + eps.
    • r1 includes the mass of all points between min = q0 and q1.
    • rm+1 includes the mass of all points between min = q0 and max = qm+1.

    Because errors are independent, we can make some estimates of the size of the confidence bounds for the quantile returned from a call to getQuantile(), but not error bounds. These confidence bounds may be quite large for certain distributions.

    • Let q = getQuantile(r), the estimated quantile of rank r.
    • Let eps = getNormalizedRankError(false)[*].
    • Let qlo = estimated quantile of rank (r - eps).
    • Let qhi = estimated quantile of rank (r + eps).
    • Then qlo ≤ q ≤ qhi.

    This sketch is order and distribution insensitive

    This algorithm intentionally inserts randomness into the sampling process for items that ultimately get retained in the sketch. Thus, the results produced by this algorithm are not deterministic. For example, if the same stream is inserted into two different instances of this sketch, the answers obtained from the two sketches should be close, but may not be be identical.

    Similarly, there may be directional inconsistencies. For example, if a quantile obtained from getQuantile(rank) is input into the reverse query getRank(quantile), the resulting rank should be close, but may not exactly equal the original rank.

    Please visit our website: DataSketches Home Page and specific Javadocs for more information.

    [*] Note that obtaining epsilon may require using a similar function but with more parameters based on the specific sketch implementation.

    Author:
    Lee Rhodes, Kevin Lang, Alexander Saydakov
    See Also:
    Sketching Quantiles and Ranks, Tutorial, QuantileSearchCriteria

    • Method Summary

      All Methods Instance Methods Abstract Methods 
      Modifier and Type Method Description
      int getK()
      Gets the user configured parameter k, which controls the accuracy of the sketch and its memory space usage.
      long getN()
      Gets the length of the input stream.
      int getNumRetained()
      Gets the number of quantiles retained by the sketch.
      double getRankLowerBound​(double rank)
      Gets the lower bound of the rank confidence interval in which the true rank of the given rank exists.
      double getRankUpperBound​(double rank)
      Gets the upper bound of the rank confidence interval in which the true rank of the given rank exists.
      boolean hasMemory()
      Returns true if this sketch's data structure is backed by Memory or WritableMemory.
      boolean isDirect()
      Returns true if this sketch's data structure is off-heap (a.k.a., Direct or Native memory).
      boolean isEmpty()
      Returns true if this sketch is empty.
      boolean isEstimationMode()
      Returns true if this sketch is in estimation mode.
      boolean isReadOnly()
      Returns true if this sketch is read only.
      void reset()
      Resets this sketch to the empty state.
      String toString()
      Returns a summary of the key parameters of the sketch.

    • Method Detail

      • getK

        int getK()
        Gets the user configured parameter k, which controls the accuracy of the sketch and its memory space usage.
        Returns:
        the user configured parameter k, which controls the accuracy of the sketch and its memory space usage.

      • getN

        long getN()
        Gets the length of the input stream.
        Returns:
        the length of the input stream.

      • getNumRetained

        int getNumRetained()
        Gets the number of quantiles retained by the sketch.
        Returns:
        the number of quantiles retained by the sketch

      • getRankLowerBound

        double getRankLowerBound​(double rank)
        Gets the lower bound of the rank confidence interval in which the true rank of the given rank exists.
        Parameters:
        rank - the given normalized rank.
        Returns:
        the lower bound of the rank confidence interval in which the true rank of the given rank exists.

      • getRankUpperBound

        double getRankUpperBound​(double rank)
        Gets the upper bound of the rank confidence interval in which the true rank of the given rank exists.
        Parameters:
        rank - the given normalized rank.
        Returns:
        the upper bound of the rank confidence interval in which the true rank of the given rank exists.

      • hasMemory

        boolean hasMemory()
        Returns true if this sketch's data structure is backed by Memory or WritableMemory.
        Returns:
        true if this sketch's data structure is backed by Memory or WritableMemory.

      • isDirect

        boolean isDirect()
        Returns true if this sketch's data structure is off-heap (a.k.a., Direct or Native memory).
        Returns:
        true if this sketch's data structure is off-heap (a.k.a., Direct or Native memory).

      • isEmpty

        boolean isEmpty()
        Returns true if this sketch is empty.
        Returns:
        true if this sketch is empty.

      • isEstimationMode

        boolean isEstimationMode()
        Returns true if this sketch is in estimation mode.
        Returns:
        true if this sketch is in estimation mode.

      • isReadOnly

        boolean isReadOnly()
        Returns true if this sketch is read only.
        Returns:
        true if this sketch is read only.

      • reset

        void reset()
        Resets this sketch to the empty state. If the sketch is read only this does nothing.

        The parameter k will not change.

      • toString

        String toString()
        Returns a summary of the key parameters of the sketch.
        Overrides:
        toString in class Object
        Returns:
        a summary of the key parameters of the sketch.