New Streamlet API for Heron
As of version 0.16.0, Heron offers a new Streamlet API that you can use to write topologies in a more declarative, functional manner, without needing to specify spout and bolt logic directly. The Streamlet API is currently in beta and available for Java. The Streamlet API for Python will be available soon.
More information on the Streamlet API can be found below.
Heron topologies consist of two basic components:
- Spouts inject data into Heron topologies, potentially from external sources like pub-sub messaging systems (Apache Kafka, Apache Pulsar, etc.)
- Bolts apply user-defined processing logic to data supplied by spouts
Spouts and bolts are connected to one another via streams of data. Below is a visual illustration of a simple Heron topology:
In the diagram above, spout S1 feeds data to bolts B1 and B2 for processing; in turn, bolt B1 feeds processed data to bolts B3 and B4, while bolt B2 feeds processed data to bolt B4. This is just a simple example; you can create arbitrarily complex topologies in Heron.
There are currently two APIs available that you can use to build Heron topologies:
- The higher-level Heron Streamlet API (recommended for new topologies), which enables you to create topologies in a declarative, developer-friendly style inspired by functional programming concepts (such as map, flatMap, and filter operations)
- The lower-level topology API (not recommended for new topologies), based on the original Apache Storm API, which requires you to specify spout and bolt logic directly
- Submit the topology to the cluster. The topology is not yet processing streams but is ready to be activated.
- Activate the topology. The topology will begin processing streams in accordance with the topology architecture that you’ve created.
- Restart an active topology if, for example, you need to update the topology configuration.
- Deactivate the topology. Once deactivated, the topology will stop processing but remain running in the cluster.
- Kill a topology to completely remove it from the cluster. It is no longer known to the Heron cluster and can no longer be activated. Once killed, the only way to run that topology is to re-submit and re-activate it.
A topology’s logical plan is analagous to a database query plan in that it maps out the basic operations associated with a topology. Here’s an example logical plan for the example Streamlet API topology below:
A topology’s physical plan is related to its logical plan but with the crucial difference that a physical plan determines the “physical” execution logic of a topology, i.e. how topology processes are divided between containers. Here’s a basic visual representation of a physical plan:
Windowed computations gather results from a topology or topology component within a specified finite time frame rather than, say, on a per-tuple basis.
Here are some examples of window operations:
- Counting how many customers have purchased a product during each one-hour period in the last 24 hours.
- Determining which player in an online game has the highest score within the last 1000 computations.
Sliding windows are windows that overlap, as in this figure:
For sliding windows, you need to specify two things:
- The length or duration of the window (length if the window is a count window, duration if the window is a time window).
- The sliding interval, which determines when the window slides, i.e. at what point during the current window the new window begins.
In the figure above, the duration of the window is 10 seconds, while the sliding interval is 5 seconds. Each new window begins five seconds into the current window.
With sliding time windows, data can be processed in more than one window. Tuples 3, 4, and 5 above are processed in both window 1 and window 2 while tuples 6, 7, and 8 are processed in both window 2 and window 3.
Setting the duration of a window to 16 seconds and the sliding interval to 12 seconds would produce this window arrangement:
Here, the sliding interval determines that a new window is always created 12 seconds into the current window.
Tumbling windows are windows that don’t overlap, as in this figure:
Tumbling windows don’t overlap because a new window doesn’t begin until the current window has elapsed. For tumbling windows, you only need to specify the length or duration of the window but no sliding interval.
With tumbling windows, data are never processed in more than one window because the windows never overlap.
Count windows are specified on the basis of the number of operations rather than a time interval. A count window of 100 would mean that a window would elapse after 100 tuples have been processed, with no relation to clock time.
With count windows, this scenario (for a count window of 50) would be completely normal:
|Window||Tuples processed||Clock time|
|3||50||1 hour, 12 minutes|
Time windows differ from count windows because you need to specify a time duration (in seconds) rather than a number of tuples processed.
With time windows, this scenario (for a time window of 30 seconds) would be completely normal:
|Window||Tuples processed||Clock time|
All window types
As explained above, windows differ along two axes: sliding (overlapping) vs. tumbling (non overlapping) and count vs. time. This produces four total types:
Resource allocation with the Heron Streamlet API
When creating topologies using the Streamlet API, there are three types of resources that you can specify:
- The number of containers into which the topology’s physical plan will be split
- The total number of CPUs allocated to be used by the topology
- The total amount of RAM allocated to be used by the topology
For each topology, there are defaults for each resource type:
|Number of containers||1||1|
Allocating resources to topologies
For instructions on allocating resources to topologies, see the language-specific documentation for:
A Heron spout is a source of streams, responsible for emitting tuples into the topology. A spout may, for example, read data from a Kestrel queue or read tweets from the Twitter API and emit tuples to one or more bolts.
Information on building spouts can be found in Building Spouts.
A Heron bolt consumes streams of tuples emitted by spouts and performs some set of user-defined processing operations on those tuples, which may include performing complex stream transformations, performing storage operations, aggregating multiple streams into one, emitting tuples to other bolts within the topology, and much more.
Information on building bolts can be found in Building Bolts.
Heron’s original topology API required using a fundamentally tuple-driven data model. You can find more information in Heron’s Data Model.