pyspark.sql.functions.theta_intersection

pyspark.sql.functions.theta_intersection(col1, col2)

Returns the intersection of two binary representations of Datasketches ThetaSketch objects, using a Datasketches Intersection object.

New in version 4.1.0.

Parameters

col1Column or column name

col2Column or column name

Returns

Column

The binary representation of the intersected ThetaSketch.

See also

pyspark.sql.functions.theta_intersection_agg()

pyspark.sql.functions.theta_sketch_agg()

pyspark.sql.functions.theta_sketch_estimate()

Examples

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([(1,1),(2,2),(3,2),(3,3)], "struct<v1:int,v2:int>")
>>> df = df.agg(
...     sf.theta_sketch_agg("v1").alias("sketch1"),
...     sf.theta_sketch_agg("v2").alias("sketch2")
... )
>>> df.select(sf.theta_sketch_estimate(sf.theta_intersection(df.sketch1, "sketch2"))).show()
+-----------------------------------------------------------+
|theta_sketch_estimate(theta_intersection(sketch1, sketch2))|
+-----------------------------------------------------------+
|                                                          3|
+-----------------------------------------------------------+