class PipelinedShuffleDependency[K, V, C] extends ShuffleDependency[K, V, C]
:: DeveloperApi :: A ShuffleDependency whose output can be read incrementally: a consumer stage may begin reading the shuffle output while the producer stage is still running, rather than waiting for the producer's full, materialized output.
This is a subtype of ShuffleDependency -- and thus, like it, a first-class dependency kind
alongside NarrowDependency under Dependency. It is intended to be the marker the
DAGScheduler will use to decide that the producer and consumer
stages connected by this edge may run concurrently (a "pipelined group"), and that the shuffle
layer should serve this shuffle with an incremental shuffle implementation. A plain
ShuffleDependency keeps the existing semantics: its output is fully materialized before any
consumer reads it.
Two behaviors are active from construction, driven by this type: shuffle registration routes to
the incremental (pipelined) shuffle manager via SparkEnv.shuffleManagerFor (a plain
ShuffleDependency goes to the blocking manager), and push-based shuffle merge is
unconditionally disabled (setShuffleMergeAllowed(false); see below) because merge exposes
output only after a post-completion finalize step and would register merge results for a
transient shuffle. Beyond those, the concurrent scheduling behavior is added separately by the
DAGScheduler components that match on this type; code that only matches the parent
ShuffleDependency still treats it as an ordinary (materialized) shuffle for everything else.
The name is *pipelined* rather than *streaming*: reading producer output as it is produced is a general execution capability (software-pipelining of dependent stages), not specific to streaming. Streaming / real-time mode is the first caller, but nothing here is streaming-specific.
Note: the parent's checksumMismatchFullRetryEnabled /
checksumMismatchQueryLevelRollbackEnabled parameters are intentionally not exposed here, so
they stay at their false defaults for a pipelined shuffle. Their checksum retry / query-level
rollback recomputes and re-runs succeeding stages after a mismatch; in a pipelined group any
failure aborts the whole group and the caller reruns from scratch, so that stage-level recompute
never fires -- the mechanism is moot by construction (it is also incompatible with a consumer
that has already read the output incrementally). Leaving the params unset keeps the idiom
unreachable.
- Annotations
- @DeveloperApi()
- Alphabetic
- By Inheritance
- PipelinedShuffleDependency
- ShuffleDependency
- Logging
- Dependency
- Serializable
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new PipelinedShuffleDependency(_rdd: RDD[_ <: Product2[K, V]], partitioner: Partitioner, serializer: Serializer = SparkEnv.get.serializer, keyOrdering: Option[Ordering[K]] = None, aggregator: Option[Aggregator[K, V, C]] = None, mapSideCombine: Boolean = false, shuffleWriterProcessor: ShuffleWriteProcessor = new ShuffleWriteProcessor, rowBasedChecksums: Array[RowBasedChecksum] = ShuffleDependency.EMPTY_ROW_BASED_CHECKSUMS)(implicit arg0: ClassTag[K], arg1: ClassTag[V], arg2: ClassTag[C])
Type Members
- implicit class LogStringContext extends AnyRef
- Definition Classes
- Logging
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def MDC(key: LogKey, value: Any): MDC
- Attributes
- protected
- Definition Classes
- Logging
- val aggregator: Option[Aggregator[K, V, C]]
- Definition Classes
- ShuffleDependency
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def checksumMismatchFullRetryEnabled: Boolean
- Definition Classes
- ShuffleDependency
- val checksumMismatchQueryLevelRollbackEnabled: Boolean
- Definition Classes
- ShuffleDependency
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def getMergerLocs: Seq[BlockManagerId]
- Definition Classes
- ShuffleDependency
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- val keyOrdering: Option[Ordering[K]]
- Definition Classes
- ShuffleDependency
- def log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- def logBasedOnLevel(level: Level)(f: => MessageWithContext): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logName: String
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- val mapSideCombine: Boolean
- Definition Classes
- ShuffleDependency
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def newShuffleMergeState(): Unit
- Definition Classes
- ShuffleDependency
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- val partitioner: Partitioner
- Definition Classes
- ShuffleDependency
- def rdd: RDD[Product2[K, V]]
- Definition Classes
- ShuffleDependency → Dependency
- val rowBasedChecksums: Array[RowBasedChecksum]
- Definition Classes
- ShuffleDependency
- val serializer: Serializer
- Definition Classes
- ShuffleDependency
- def setMergerLocs(mergerLocs: Seq[BlockManagerId]): Unit
- Definition Classes
- ShuffleDependency
- val shuffleHandle: ShuffleHandle
- Definition Classes
- ShuffleDependency
- val shuffleId: Int
- Definition Classes
- ShuffleDependency
- def shuffleMergeAllowed: Boolean
- Definition Classes
- ShuffleDependency
- def shuffleMergeEnabled: Boolean
- Definition Classes
- ShuffleDependency
- def shuffleMergeFinalized: Boolean
Returns true if push-based shuffle is disabled or if the shuffle merge for this shuffle is finalized.
Returns true if push-based shuffle is disabled or if the shuffle merge for this shuffle is finalized.
- Definition Classes
- ShuffleDependency
- def shuffleMergeId: Int
- Definition Classes
- ShuffleDependency
- val shuffleWriterProcessor: ShuffleWriteProcessor
- Definition Classes
- ShuffleDependency
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- def withLogContext(context: Map[String, String])(body: => Unit): Unit
- Attributes
- protected
- Definition Classes
- Logging
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)